Add Monolith archive and shader extraction, a build-gated compact memory layout, vertex storage writes under rasterizer discard, primitive ordering and interleaved geometry feedback. Correct native decoder/audio wait accounting and the oe calls used after movies.
Validated opening combat and movement on Mac with real audio, the full Mac/iOS shader build, GPU regressions, and the unsigned IPA archive. iOS runtime, later areas and DLC gameplay remain untested.
Compare revision shader builds and pipeline interfaces across local games,
and run available smoke routes with crash, hang, timing and image checks.
Add the developer loop and share the measuring lock with bench.py.
Keep all local games in static checks and honor --no-smoke. Include shader
bytes and fallback pairs in cache keys, recheck interfaces on reuse, and
reuse smoke baselines only for matching inputs and run settings. Detect
failed prebuilds, missing or intermittently bad screenshots, and Vulkan
build failures even when no MSL was emitted. Keep game-derived files and
writable profile copies inside each game's NSP work folder.
Validation: tool tests and the lint ratchet pass. Full shader rebuilds and
gameplay comparisons remain unverified; the 26 discovered Mac builds have
no smoke routes in this checkout.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
HAL Laboratory's engine on NintendoWare, SDK 11.4.0, NVN. Plays on the Mac at 30 fps, as on
the Switch: online and difficulty prompts, the 1080p opening movie, Point of Arrival with its
guide movies, the city reveal, the Sword ability and fights, title and Continue; a 15-minute
soak with no crash. No game code is cut; the profile is the memory layout and the shader groups.
All 27,297 shaders build for Mac and iOS; the IPA (6.4 GB) is untested on a device.
- Threads (general): a guest thread waiting for a host lock that HLE code holds while it runs
guest code is blocked, not runnable (dm_host_lock, a native HLE wait). The movie decoder held
its decode lock across SignalEvent; the signal's same-core handoff waited for the movie input
thread, which waited for that lock, and the game froze at its first guide movie. t_preempt
has the case.
Docs: the game note, games table, README gallery, threads.md, shaders.md, symptom index;
AGENTS count 73 -> 74. Checked: host tests, preempt, tool tests (253, 15 skips), docs tests;
a Mac run on chain 5cddc47253ab through the first guide movie.
Persist the interface language, translate app-owned labels and dialogs immediately, preserve raw build diagnostics, and replace unverified device and account claims with precise preparation and compatibility wording. Add English and Spanish READMEs and localization regression coverage.
Provide folder selection, build progress, process-group cancellation, and IPA/RomFS/Sideloadly buttons. Preserve command documentation separately and correct the personal author policy.
Profile 0100C1F0051B6000 (SDK 4.4.0, Retro Studios' engine, NVN): plays on the Mac from boot
through the menus and the intro cinematic into the first level at 60 fps. General changes:
- svc SetHeapSize: a request beyond the heap region returns ResultOutOfMemory, as Horizon
(the game binary-searches its heap size between 3 and 6 GB)
- shader_split.py: SNVN program records in Retro's RFRM forms (MaterialArchive.arc, the packs'
MTRL resources) and the LZSS-compressed startup material archive in main
- nvn_mtl.m: a texture in a CPU_UNCACHED pool starts with its storage at first use (title logo,
menu panels); transform feedback captures draws that rasterize too
- dmsh_build.py: transform feedback buffers holding several outputs (DK's fur simulation);
chain header regenerated
- hle_audren.c: the renderer event is the kernel event the audio service gives, not the SDK's
local one (the main thread was starved at the first level's load)
- hle_oe.c: oe::GetUserInactivityDetectionTimeExtensionMode
Profile 010057D006492000 (no virtual address memory: the 4 GB heap layout; tex_flush_upload and tex_dirty_upload for the textures Unreal writes from the CPU into uncached pools after creating them) and its note. Brought up on the Mac through the boot, the language notice, the title, the traveler selection and Ophilia's and H'aanit's openings (free play in the cathedral and in S'warkii) at 30 fps; battles, sound and the iPhone not yet tested.
General changes found on it:
- patch_nso.py: a jump table whose index is bounds-checked with a 64-bit cmp (main+0x1676f58), accepted only in the last pass so earlier rewrites are byte-identical.
- profile_guess.py: Unreal's 64 GB alias layout only with virtual address memory (this game sets a 3.08 GB heap instead).
- NVN: nvnTextureBuilderGetSparseTileLayout / nvnTextureGetSparseTileLayout (Unreal sizes a sparse texture's mip tail with it; unimplemented, it asked for 1.75 GB twice); sparse 2D textures without a packaged layout take the block height of their 64 KB tile (the world map's tiles were shuffled); every virtual pool gets an address range of its own until its first mapping decides between the alias (a whole-pool mapping, Mario's pool) and the sparse page table (Unreal's one-texture pools, mapped piece by piece: streamed textures were black and later mappings refused).
- Audio renderer: SdkMutexType is looked up only when the SDK uses it (SDK 3.5.1 takes LockMutex at +0x1f8).
README: the gallery entry (Mac screenshot of S'warkii, IPA ~3.6 GB, all 5,434 shaders built for iOS); AGENTS.md: seventy-one games.
Checked on the rebased tree: the tool tests (15 skipped), the docs tests, a Mac boot into the traveler selection; before the rebase the graphics tests (make run, exit 0).
Unity 2021.3.25f1 on SDK 16.2.0 (the sdk build of Super Mario Bros. Wonder 1.0.0).
The sdk build takes the node-area cut and SDK 20's 2 MB region alignment: Unity's
N MB + 252 KB regions each took the next size class's slot, which exhausted the
4.375 GB alias region after about six minutes. Profile with a 1.5 GB pool, note,
games table row, README gallery entry and reference entry.
Profile 010069C01AB82000 (Atlus, Unreal Engine 4.27, SDK 15.3.2, NVN). Opening gameplay
tested on the Mac: new game, classroom, saves and reload in a fresh process, Tokyo map,
Shinagawa Station and its event. iOS and combat untested.
- Memory: the FMallocBinned3 constructor cut (main+0x1f2f63c, gate +0x1f2f644) with the
measured UE4 vtable in hle_uemalloc.c; -SwitchVirtualAddressSpaceSize=2048 and a 3.75 GB
block. profile_guess.py now leaves 128 MB for the initial heap in provisional Unreal
layouts (test_profile_guess.py).
- Shaders (general, chain 86c8282f6fb4): returning IGN image atomics keep the old value with
bounded access (Ryujinx translator, t_atomic_return.m); buffer-image reflection keeps its
Metal texture type (nvn_refl.m, t_refl.m). All 19,746 shaders build for Mac and iOS.
- 60 fps: hle_fpslock.c writes the game's own Project.FrameRateLimit (main+0x6d6d5d0, which
GameUserSettings applies to t.MaxFPS) and rhi.SyncInterval, gated on main 0a4f14a5...
Mac (M4 Pro): 59.2-60.1 fps at the station, frame p95 17.5-20.3 ms; a 1 s walk ends at
the same place at 30 and 60 fps. DM_FPS_TARGET=30 keeps 30.
Docs: the game note, games table, README gallery, memory.md, shaders.md, symptom index;
AGENTS count 68 -> 69. Checked: tool tests (186, 6 skips), docs tests.
A side library of every vertex shader with stage inputs, as <key>_bv is, whose main0 / main0_z have no [[stage_in]] and read their inputs from a per-draw layout block at buffer 16: per stream the base address, stride and divisor, per NVN attribute its state word, the base instance from [[base_instance]] (so indirect draws are right too). dm_fetch decodes the NVN format at run time into the register words Maxwell's fetch gives the shader, and the input takes them with as_type, so the body runs unchanged. A pipeline built with it has no vertex descriptor and serves every vertex layout the pair is drawn with; the hybrid pipelines (next change) draw with it while a draw's own pipeline is built.
The words are bit-identical to Metal's vertex-descriptor fetch for every format nvn_mtl.m vfmt knows and to the host's CPU conversion for the integer-read-as-float formats: unorm / snorm through the unpack builtins (a plain division under fast math is a unit in the last place off for up to half the values), R11G11B10F NaNs canonical as from Metal, divisors with Metal's rule base + (instance - base) / divisor. Positions are [[invariant]] and built with preserved invariance, so a pulling pipeline and the vertex-descriptor one compute the same positions. Transform feedback, geometry capture and layer entries get no variant; shaders without stage inputs need none. The run-time decode costs GPU time: +65 % of a vertex-bound draw with 8 attributes on an M2 Max, about 1 ms per million vertices, so it is meant only to bridge the frames until the draw's own pipeline is built.
The iOS app does not use the variants yet (the Metal 4 path is off on iOS by default), so build_ipa.sh leaves the _bp libraries out of the IPA unless IOS_PULL=1 (pack_ipa.py --skip-suffix).
Chain version 8393d85a4eae (on main's 20fe3d62b9f6). Breath of the Wild's full Mac build on chain 97e2f301eb73 (25 min, a full retranslation since the code fingerprint changed): 19,592 shaders ok, 8,888 pulling libraries generated (275 MB), none failed, 43 vertex shaders without stage inputs; every existing library is byte-identical to the previous build (19,592 _b.metallib and _b.metal, 6,254 _bf, 1,070 _bv). Other profiles' shaders were not rebuilt.
Tests: t_pull.m (304 checks: the 76 formats vfmt knows, 73 comparisons with Metal's fetch at aligned, 2-byte and stride-1 placements, 24 with the CPU conversion, stride 0 and divisors 1-3 with a base instance), t_pull_inv.m (2,000 random triangles: a depth prepass with one pipeline and depth-equal with the other passes every covered pixel, both ways round) and test_dmsh_pull.py (the generator: entries, ids it adds or keeps, entries left out, shaders declined; compiles for macOS and iOS); pack_ipa.py's --skip-suffix checked by hand. The other graphics tests and the host unit tests give the results they give on main. docs/reference/rendering.md ("Pulling vertex inputs") describes the variant and its cost; AGENTS.md lists the new chain files.
- profile 01007BB017812000 (1.0.3, SDK 13.3.0): plays on the Mac at 60 fps
- provisional profiles add the subsdks the NROs import (profile_guess nro_modules)
- an NRO importing nvnBootstrapLoader makes an NVN game; NROs calling glslcCompile get the run-time GLSLC capture
- shader_split reads zip archive members (Portal's nxcontent/nvsb GLSLC outputs)
- ro::LookupModuleSymbol for a name a loaded module lacks is not red (Horizon fails it too)
- AGENTS.md: a Mac gameplay test is enough to count a game as playable (65)
Replace the separate ASTRAL CHAIN bring-up notice with a gameplay screenshot and notes. Record that passing Mac gameplay testing is sufficient for gallery inclusion, with validation scope kept in each game note.
The profile is the NSP script's provisional one with tex_flush_upload and tex_dirty_upload: RE Engine writes its UI textures (ASTC) into CPU-visible pools in place, as Monster Hunter Rise does, and without both keys the language screen's background and buttons were magenta (with tex_flush_upload alone they still were). 7,744 shaders from the PAK's SDF definitions and main; 7,739 build. The other 5 are compute shaders with atomics on a 3D image, which SPIRV-Cross indexes with an int3, as Octopath Traveler 0's 16; none was requested on the routes below. PR #138 (Nishikori-Yui) built them by flattening the coordinate in dmsh_build.py, a chain change that would rebuild every game's libraries, so it is left out until a run requests one (the game note). PR #138's profile is the same as this one (the same keys and shader tables).
Host fix: the memory line's "freed but alive" count only grew in most games (Animal Crossing: New Horizons 3,113 after 27,900 frames, Pokémon Sword 2,050, Super Smash Bros. Ultimate 1,279; Ghost Trick 125 to 146 buffers, 6.6 to 8.7 GB of zero-copy buffers, so GPU allocated read 9.9 to 11.7 GB against 2.2 GB resident). A temporary census of every host table that can hold a texture over a pool buffer found no holder: mtl_pool_del runs on the guest thread that finalizes the pool, whose implicit autorelease pool never drains, and adding the buffer to the weak table g_dead_pools left it autoreleased there. containsObject: / removeObject: on a weak NSHashTable do the same (a standalone test), and res_add / res_remove call them on guest threads too. mtl_pool_del, res_add_at and res_remove now have their own @autoreleasepool. Ghost Trick: 0 freed but alive, GPU allocated 1.7 GB instead of 8.3 GB 300 frames into the boot and 2.9 GB in Chapter 1, resident memory, frame times and pictures unchanged; Pokémon Sword: 8 to 0 at the title after 90 s, the same frame pacing.
On the Mac: the opening movies, the language screen, the first-boot notices, save creation, the Bonus Content notices and Chapter 1's opening in the junkyard (the Ghost World, the first trick), and a second boot that loads the save, at 60 fps (frame interval p99 about 17 to 19 ms), about 2.3 GB resident, no red line; the Wwise sound plays (a muted run with DM_AUDIO_DUMP). Checked: the tool tests (57, the docs check among them), t_nvn (124 checks), a boot of the rebased host. README (games gallery, a Mac picture of the junkyard), AGENTS.md's count (sixty-two: the gallery's cells), the game note, the games table, docs/reference/nvn-objects.md and the symptom index say so
Movie decoder (hle_movie.c, every game), both found on the opening movie (Movie/Cutscene01.mp4, H.264 and AAC through movie::Extractor):
- AAC decoders offer at most 32 outputs (MD_NOUT_AAC; Vorbis keeps 64, video 16). The game's movie player keeps the outputs it holds in a ring of 32 entries per track (main+0xe93d4c: 0x200 bytes) and pushes with no bounds check (main+0xe94140), so with 64 the 33rd AAC output wrote its index (0x3f) and time over the next heap block, and the game crashed in sead::ExpHeap's free a second into the movie, at the same address every boot. The console's AAC decoder cannot offer more than 32, or the game would overflow there too; Monster Train 2 peaked at 30 held.
- An AAC decoder's GetOutputBufferSize is one frame (1024 samples per channel, 2048 with SBR), what each output holds. The player sizes its own AudioOut buffers by it, decodes into them and appends each whole, advancing its audio clock by size / 4 samples (main+0xe92a74); with the 128 KB Vorbis bound every 4 KB AAC frame played as 0.68 s, the clock crawled and the movie froze on its first second. Vorbis keeps the bound (its packets vary in length).
Advanced blend modes (hle_nvn.c, nvn_obj.h, nvn_mtl.m, every game): the blend state keeps the mode, the premultiplied-source flag and the overlap (with their getters), and the pipeline blends the Porter-Duff modes (ZERO, SRC, DST, SRC/DST_OVER, _IN, _OUT, _ATOP, XOR, the PLUS modes) with the fixed-function factors they amount to under uncorrelated overlap (adv_blend); the others keep the state's functions with a red line naming the mode. The game's levels set SRC_OVER on one blend state, which the host used to ignore.
DM_PROBE_POLL takes a value filter (=<value>): every change is logged, the threads are snapshotted and dumped only at changes to that value. That is how the ring overflow was caught: DM_PROBE_WATCH on the heap link made the movie's file reads into the same page fail.
Profile: the provisional profile's guesses held (no system resource, a 3.14 GB heap in the 4 GB layout, modules main subsdk0 sdk), plus render_cond on (the game skips clears under render conditions; without it every level was black behind its HUD). 12,691 shaders (BNSH / BAHS in the zstd RomFS and main), all translated. Pad only (main imports no touch screen), the default 2D layout. A prewarm list recorded on the Mac from a fresh save over the route above (550 lines; with it nothing was built on demand). Checked: the NSP script packs it from the NSPs with the profile in dm/games; the graphics run list and the 49 tool tests pass; after rebasing on 6d34984 (spin waits) the game boots into level 1-1 as before. README (gallery, a Mac picture of level 1-1; 2D layout), AGENTS.md (57 -> 58) and docs/development.md, docs/agents 03 / 04 / 05 say so
On the Mac from update 13.0.5 (v2031616): the opening movie, the title, the first launch's DLC unlocks and Challenges board, the main menu, rulesets, stage and fighter select, 2-player battles (Frigate Orpheon, Big Blue, Great Bay, Find Mii, Prism Tower, Kongo Jungle 64) at 60 fps with about 2.7 ms of GPU per frame and 2 GB resident (1.2 GB of Metal allocations), quitting to the results and a second battle whose fighters' NROs the game loaded again; no red line in battle.
- Profile: modules main subsdk0 sdk (Catherine: Full Body's sdk and movie builds), the 5 GB guest layout, shader groups for data.arc's shader databases, effects and UI layouts, 35,012 shaders.
- patches.py: the engine's allocator (main+0x37c9a20, a static constructor) reserves 62 GB of address space with AllocateAddressRegion and aborted at main+0x37c9bd0. Booted once with a 64 GB alias region it commits from the region's start up (1.73 GB at the title, up to 2.5 GB in a battle) and keeps its page tables in the top 14 MB; all 11 instructions derived from the size are cut to a 4 GB region.
- sdk_patches.py: this SDK 8.2.1 sdk build (5fd4f531, Catherine's too) has the 1 GB AddressRegionManager node area and the 1 GB alignment of large regions (DRAGON QUEST XI S's SDK 7.3.2 layout at sdk+0x336b40 / +0x336d5c); both cut, so the region starts 64 MB into the 4.375 GB alias region.
- NVN: nvnSetExternalDebuggerEnabledNVX, fetched with nvnDeviceGetProcAddress(NULL, …) before the device and called unchecked (pc=0 from main+0x37fe92c), is a no-op in names.txt / hle_nvn.c, as docs/agents/03 already described.
- iOS NRO embedding (the part d979c34 left out): gen_embed.py packs the NROs nro.tsv lists into three sections (__TEXT,__dm_nro_x0, __DM_NRO_W,__img, __TEXT,__dm_nro_x1) with a table of their pieces (__DATA,__dm_nro_tab), since a Mach-O file has at most 255 sections and three per NRO would take 294; loader.c map_image maps an NRO from there (nro_pieces); build_ipa.sh puts nro.tsv beside the rewriter tables. The 98 NROs add 157 MB of code and 51 MB of data to the executable.
- shader_split.py reads data.arc through smash_arc.py (smash_arc_jobs / smash_arc_shaders: each current file scanned, labelled /data.arc/<format>#<hash40>); smash_arc.files(regions=True) gives every region's data of a regional file, without which 128 effect shaders were missing (34,884 of 35,012).
- Docs: README (games table, a Mac picture of Great Bay), AGENTS.md (50 → 51), docs/development.md (game notes, the NRO paragraph, nro_collect.py), docs/agents/01 (data.arc) and 03 (the 4 GB cut and its measurements, iOS NRO embedding).
Checked: the NSP script packs it from the NSPs alone, for the Mac and for iOS; tool tests (41), host unit tests and the NRO loader test pass. Played on an iPhone with this port's build from before it moved onto d979c34's NRO loader; the merged build was checked on the Mac (battles, an NRO loaded again) and builds for iOS.
Erick Sanchez