From 7deea8eedc460c654e6036bfb4bb911ae43eee32 Mon Sep 17 00:00:00 2001 From: seedds Date: Fri, 23 Oct 2026 14:13:48 +0800 Subject: [PATCH] Tears of the Kingdom: the SDK's address-region node area cut from ~1 GB to 62 MB on the sdk build it shares with Breath of the Wild 1.9.0 (patch_nso.py). The game's resource heaps are address regions in the 2.375 GB alias region; at the first descent from the Great Sky Island it was used up to +2.25 GB with 1.00 GB mapped, and the game took a memory shortage (MSF / R / Rf) and played the memory-shortage Blood Moon, which put Link back at the start (GitHub issue #70; 7 of 8 Mac runs from the issue's save). With the cut: +1.75 GB, no shortage in 3.7 minutes on the island. Probe tools: the write watch reports exclusive and atomic stores, and DM_PROBE_POLL snapshots every thread when a word changes --- AGENTS.md | 2 +- dm/games/0100F2C0115B6000/game.tsv | 3 +- dm/host/probe.c | 66 ++++++++++++++++++++++++++++++ dm/tools/patch_nso.py | 10 +++++ docs/agents/05-debugging.md | 4 ++ docs/development.md | 1 + 6 files changed, 84 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7b5cdd4..433e09d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,6 @@ # Notes for agents -- This repository runs Nintendo Switch games natively on Apple platforms: the game's own code runs, rewritten only where Apple's kernel requires it. Forty-three games are playable on the iPhone: Super Mario Bros. Wonder (update 1.2.1, profile `010015100B514000`), Fire Emblem Engage (update 2.0.0, `0100A6301214E000`), Metroid Dread (update 2.1.0, `010093801237C000`), 13 Sentinels: Aegis Rim (update 1.10, `01003FC01670C000`), Super Mario 3D World + Bowser's Fury (update 1.2.2, `010028600EBDA000`), The Legend of Zelda: Breath of the Wild (update 1.9.0, SDK 20, `01007EF00011E000`), The Legend of Zelda: Link's Awakening (update 1.1.0, SDK 8, `01006BB00C6F0000`) Super Mario Odyssey (update 1.4.1, SDK 7, `0100000000010000`), Hades (update 1.0.38233, SDK 9, `0100535012974000`), Hyrule Warriors: Age of Calamity (update 1.3.0, SDK 9.6, `01002B00111A2000`), Fire Emblem: Three Houses (update 1.2.0, SDK 6.5, `010055D009F78000`), Super Mario Galaxy (update 1.3.1, SDK 18, `010099C022B96000`) The Legend of Zelda: Tears of the Kingdom (update 1.4.3, SDK 20, `0100F2C0115B6000`) Monster Hunter Rise (update 16.0.2, SDK 11.4.3, `0100B04011742000`), Ace Attorney Investigations Collection (update 1.0.0.2, SDK 17.5.3, `010005501E68C000`), Hades II (base game, SDK 19.3.9, `0100A00019DE0000`), Unicorn Overlord (update 1.05, SDK 16.2.4, `010069401ADB8000`), Rhythm Heaven Groove (base game 1.0.0, SDK 21.4.0, `0100D9F01D474000`), Octopath Traveler 0 (update 1.0.5, SDK 19.3.13, `01005270232F2000`), Pokémon Legends: Z-A (update 2.0.2, SDK 19.3.11, `0100F43008C44000`), Cuphead (update 1.3.7, SDK 5.5.0, `0100A5C00D162000`), Super Mario Galaxy 2 (update 1.4.0, SDK 18.3.5, `0100FD8022DAA000`), FINAL FANTASY TACTICS - The Ivalice Chronicles (update 1.4.0, SDK 19.3.9, `010038B015560000`), Absolum (update 1.1.1, SDK 18.3.2, `0100A83022A22000`), Captain Toad: Treasure Tracker (update 1.4.2, SDK 8.2.0, `01009BF0072D4000`) and Brain Memory Training Academy (base game 1.0.0, SDK 16.2.4, `0100F5D01DA3A000`), The Great Ace Attorney Chronicles (base game 1.0.0, SDK 10.4.1, `010036E00FB20000`), STEINS;GATE RE:BOOT (update 1.0.3, SDK 20.5.17, `01008BC024FDE000`), Big Brain Academy: Brain vs. Brain (update 1.2.0, SDK 11.4.3, `0100620012D6E000`), Patrick's Parabox (update 1.0.1, SDK 13.3.7, `0100E44016C22000`), Celeste (update 1.4.1.1, SDK 15.3.2, `01002B30028F6000`), DRAGON QUEST XI S: Echoes of an Elusive Age – Definitive Edition (update 1.0.4, SDK 7.3.2, `01006C300E9F0000`), Monster Train 2 (update 1.1.1, SDK 18.3.5, `010051701FB46000`), Raging Loop (base game 1.0.0, SDK 7.3.2, `010020400E1C2000`), DARK SOULS: REMASTERED (update 1.0.3, SDK 4.5.0, `01004AB00A260000`), Paper Mario: The Thousand-Year Door (update 1.0.1, SDK 15.3.2, `0100ECD018EBE000`), Hollow Knight: Silksong (update 1.0.30000, SDK 20.5.8, `010013C00E930000`), Catherine: Full Body (update 1.0.1a, SDK 8.2.1, `0100BF00112C0000`), Digimon Story Cyber Sleuth: Complete Edition (update 1.0.3, SDK 6.4.0, `010014E00DB56000`), Animal Crossing: New Horizons (update 3.0.3, SDK 20.5.4, `01006F8002326000`), DRAGON QUEST VII Reimagined (update 1.2.0, SDK 19.3.1, `0100A9D01C446000`), FINAL FANTASY RESONANCE (update 1.0.1, SDK 20.5.17, `0100606026F12000`) and Luigi's Mansion 3 (update 1.4.0, SDK 7.3.2, `0100DCA0064A6000`). Hades renders with Vulkan, which the host implements on Metal (`dm/gfx/vk/hle_vk.m`; `main` imports Vulkan, not NVN). Age of Calamity is Koei Tecmo's KTGL engine: files in RDB archives, shaders as GLSLC outputs (`shader_split.py` `idrk_shaders`). Three Houses is KTGL too; 9 of its shaders exist only as compiles of the game's own GLSLC at run time, captured once on the Mac (its `main` holds GLSL sources and imports `glslcCompile`). Breath of the Wild and Tears of the Kingdom run at 60 fps through a frame step the host decides in whole vsyncs (`patch_nso.py` `FRAME_STEP`, `nvn_fstep.h`, profile keys `fps_target` and `frame_step`); Tears of the Kingdom has a 5 GB guest layout. Monster Hunter Rise is Capcom's RE Engine: files in KPKA PAK archives (`shader_split.py` `kpka_jobs`, shaders in SDF files), the base game an XCI, its GPU heap a 64 GB sparse virtual pool (`hle_nvn.c` `vpool_*`, a 4 KB page table the replay reads textures through); it runs at 60 fps (30 on the Switch). Super Mario Galaxy is the Wii game on NERD's Hagi layer (PowerPC code recompiled into `main`), renders with Vulkan, and generates its shaders at run time, which the NSP script captures on the Mac (no SPIR-V file in its RomFS); its pointer is the touch screen in handheld mode (profile key `op_mode`). Ace Attorney Investigations Collection is Unity 2022 IL2CPP: its shaders keep their parameters as blob entries of their own (`shader_split.py` `unity_code_blobs`), and its C# file API goes through the SDK's C library, whose path checks the host answers (`hle_fs.c` `IsPathMounted` and the length limits). Hades II, unlike Hades, renders with NVN; its shaders bind textures at fixed slots up to 34 (slot 34's handle lies in the driver's image table, `dmsh_build.py`), and the two long lines through Melinoë while the staff's Omega attack charges are the game's own aim lines, not a rendering bug. Unicorn Overlord is Vanillaware's engine like 13 Sentinels (shaders compiled into `main`); both need the profile key `alloc_big_align`, which aligns large allocations to 64 KB so the ring heaps the engine cuts from a memory pool fit. Rhythm Heaven Groove is Nintendo's own 2D engine on SDK 21, the newest SDK so far (its DSP command list has no magic, `hle_audren.c`); it plays handheld with Boost (profile keys `op_mode handheld`, `perf_mode boost`), which skips its TV latency calibration and keeps its 1920x1080 draw size. Octopath Traveler 0 is Unreal Engine 5: everything in one pak with an AES-encrypted index whose key is found in `main` (`dm/gfx/tools/uepak.py`), its shaders Oodle-compressed in the pak's shader code archives and decoded once on the Mac by the game's own Oodle (any game with Unreal paks, `DM_OODLE_JOB`), its allocator replaced by the host's because Unreal's reserves 37 GB of address space (`patch_nso.py` cut, `hle_uemalloc.c`), its address region capped by the game's own argument (profile key `game_args`), and the particle shaders that pick a texture slot at run time bound by slot (profile key `tex_dyn_slots`); it runs at 60 fps (30 on the Switch) because the host writes Unreal's `t.MaxFPS` and `rhi.SyncInterval` in game memory (`hle_fpslock.c`, profile key `fps_target`). Pokémon Legends: Z-A is Game Freak's Trinity engine: everything in the packs of `arc/data.trpfs` (FlatBuffers, every file Oodle-compressed), which the shader split decodes with the game's own decoder in the Mac host's tool mode (`shader_split.py` `trinity_shaders`, `hle_oodle.c` `DM_OODLE_SERVE`); its 12 GB StandardAllocator is cut to 4 GB to fit a 5 GB guest layout, and the host refuses maps past the pool as Horizon does (profile key `mem_map_limit`); its characters are skinned with transform feedback (`dmsh_build.py` `vs_tf`, `nvn_mtl.m` `tf_draw`), and its occlusion queries, which decide how characters are animated, need the draw's base instance in the driver segment (c0 + 0x644, `encode_args`). Cuphead is Unity 2017.4 IL2CPP on SDK 5.5.0: Unity 2017's serialized and Shader formats are read by `unityfs.py` / `shader_split.py`, its rewritten modules (80 MB) need the profile key `mem_used` above the default 64 MB, or its `SetHeapSize` fails, and its NVN pool allocator breaks on a zero-size block, so the host never reports a ZCull storage size of 0 (`hle_nvn.c` `zcull_size`). Super Mario Galaxy 2 is Galaxy's Hagi layer with Galaxy's `sdk` build; its movie player reserves 384 MB of virtual address memory, which fits only because a rewriter cut on that `sdk` build shrinks the SDK's address-region node area to 62 MB (`patch_nso.py`; the profile's alias region is 512 MB). FINAL FANTASY TACTICS - The Ivalice Chronicles is Square Enix's engine on NVN: its shaders are GLSLC outputs in loose technique files, which `shader_split.py` reads in any RomFS file no other reader takes; its UI textures are written by the CPU into uncached pools (profile keys `tex_flush_upload`, `tex_dirty_upload`); and two job-pool workers share one auto-clear event, so a build cut makes a worker woken in another's place pass the wake on (`patch_nso.py`). Absolum is FNA (XNA 4 in C#) on .NET NativeAOT with SDL3 and FNA3D's NVN driver: its `main` starts its text with 1.7 MB of data the rewriter leaves alone (`nso_code.py`), its shaders are GLSLC outputs in `shaders.nx.bin`, its picture is flipped by a viewport swizzle the host applies, and everything lives in virtual address memory: malloc's 40 GB StandardAllocator is cut to 2.5 GB and the SDK's node area and large-region alignment are cut (`patch_nso.py`), and the .NET GC's region range is 1.5 GB (profile key `game_env`), to fit a 5 GB guest layout. Captain Toad: Treasure Tracker is Nintendo EPD Tokyo's engine (the `al` library over sead / agl, as 3D World and Odyssey) and needed no host change: it plays handheld because its pointer is the touch screen there (the Joy-Con gyro docked; profile key `op_mode`), and agl's post-effect program that samples its own attachment reads a snapshot (profile key `feedback_snap`). Brain Memory Training Academy is Unity 2022.3 IL2CPP with no bundles: its serialized files carry no type trees and nothing lends it a `Shader` tree, so `unityfs.py` has Unity 2022.3's `Shader` layout written out (`KNOWN_TREES`), and like Ace Attorney Investigations Collection it needs the SDK's address-region node area cut on its `sdk` build (`patch_nso.py`). The Great Ace Attorney Chronicles is Capcom's MT Framework: files in ARC archives of zlib entries, its 460 shaders in one shader package inside one of them (`shader_split.py` `mtspk_shaders`); it copies its textures into a CPU-cached pool after creating them and never flushes (its `main` has no cache maintenance at all), so it needs `tex_flush_upload` and `tex_dirty_upload` like FINAL FANTASY TACTICS. STEINS;GATE RE:BOOT is M2's ADV engine (E-mote characters, PSB archives, H.264 movies through the SDK's movie library): it needed `nn::audio::SetAudioOutVolume` (`hle_audio.c`), and because it decodes every picture into one reused region of a pool that lives for good, the storage-sharing texture table now drops an entry whose storage another texture takes over (`nvn_mtl.m` `alias_take_over`); it draws 1280x720 in both modes by its own choice. Big Brain Academy: Brain vs. Brain is NintendoWare Bezel Engine 1.56: nearly all of RomFS is in Bezel Engine archives of zstd entries, its shaders in their `.bfsha` entries (`shader_split.py` `bea_members`); its `nninitStartup` sets a heap or a 60 GB virtual address region by `IsVirtualAddressMemoryEnabled`, so the host reports the NPDM's system resource (profile key `sys_resource`, 0 here); its activity previews loop the movie decoder with Flush and Start from four threads (`hle_movie.c`); and it plays handheld with Boost (profile keys `op_mode handheld`, `perf_mode boost`), because only handheld does it read the touch screen. Patrick's Parabox is Unity 2020.3 IL2CPP with no bundles: `unityfs.py` has Unity 2020.3's `Shader` layout written out (`KNOWN_TREES`), it needs the SDK's address-region node area cut on its `sdk` build like Brain Memory Training Academy (`patch_nso.py`), and Unity's player hands the shared fonts straight to FreeType, so a game that imports no `ScalableFontEngineHelper::Decode` gets the TTF itself (`hle_sdk16.c` `font_skip`). Celeste is MonoGame for the Switch, its C# compiled to C++ by Sickhead's BRUTE: its shaders are MonoGame effects (MGFX) in its XNB files and in `main` (`shader_split.py` `mgfx_effect`), and it needed no host change. DRAGON QUEST XI S is Unreal Engine 4.18: legacy paks with LZ5 blocks, decoded on the Mac by the game's own LZ5 (`uepak.py`, `hle_oodle.c`), its shaders inline in the cooked packages (`shader_split.py` `ue_legacy_jobs`), and the slice-writing geometry shader of its colour-grading LUT folded into the vertex shader (`dmsh_build.py` `gs_slice`, `nvn_mtl.m` `prog_fold_gs`); Square Enix's sound driver never waits on the renderer event, so audio catch-up frames wait for queue space (`hle_audren.c`); its movie player plays only once every track holds a decoded output, so audio movie decoders keep 32 (`hle_movie.c`); and its Unreal region is capped at 4064 MB (profile key `game_args`, MB up to 4095) with the node area and large-region alignment cut on its SDK 7.3.2 build (`patch_nso.py`) for a 5 GB guest layout; it runs at 60 fps (30 on the Switch) because the host writes Unreal's `t.MaxFPS` and `nvn.SyncInterval` in game memory (`hle_fpslock.c`, profile key `fps_target`). Monster Train 2 is Unity 2022.3 IL2CPP on Super Mario Galaxy's SDK 18.3.5 `sdk` build: Unity creates a thread only when `nn::os::GetFreeThreadCount` (`svcGetInfo` type 24, `hle_svc.c`) is at least 1; its MP4 intro needs a movie audio decoder with more output slots than a video one, because Unity's player holds about half a second of audio before it plays (`hle_movie.c` `MD_NOUT`); and its post-processing compute shaders (the colour grading LUT among them) are Unity `ComputeShader` objects, which `shader_split.py` reads (`unity_compute`) with Unity 2022.3's layout written out in `unityfs.py`. It reads no touch, so it stays docked with the 3D pad layout, and it runs at 60 fps (30 on the Switch) with the profile key `present_interval display`, as its animations follow the time that passed. Raging Loop is Unity 2018.3 IL2CPP with no bundles on DRAGON QUEST XI S's `sdk` and `subsdk0` builds: `unityfs.py` has Unity 2018.3's `Shader` layout written out (`KNOWN_TREES`, an older form than the 2020 and 2022 layouts, with the parameters in each subprogram), and it needed no host change; it reads no touch, so it stays docked with the 2D pad layout. DARK SOULS: REMASTERED is FromSoftware's engine on SDK 4.5.0, the oldest SDK so far: its shaders are GLSLC outputs in BND3 archives inside the dvdbnd archives (`shader_split.py` `bdt_shaders`), and SDK 4's GLSLC outputs are read through their section headers (`glslc_section`); it keeps 12 sampler pools with different samplers under the same IDs, so handles a shader reads from a uniform buffer take the base of the pool bound at replay (`nvn_mtl.m` `ubo_tex_id` / `ubo_samp_id`); and it runs at 60 fps (30 on the Switch) with two rewriter cuts, its frame limiter's 30 and its frame update's fixed 1/30 s step (`patch_nso.py`). Paper Mario: The Thousand-Year Door is Intelligent Systems' engine on NintendoWare, on Celeste's SDK 15.3.2 `sdk` build: nearly every RomFS file is zstd (`.zst`, which `shader_split.py` decompresses by magic); its AngelScript call bridges leave sp misaligned, which a general rewriter rule rounds up (`site_sp_align`); it runs at 60 fps (30 on the Switch) by the host's frame step (`FRAME_STEP`, with the floor of its vsync step cut from 2 to 1); and its three job workers poll the job list with no SVC, which held it at 24 fps on an iPhone until the general spin wait rule (`patch_nso.py` "Spin waits", `hle_thread.c` `dm_spin_wait`). Hollow Knight: Silksong is Unity 6 (6000.0.50f1) IL2CPP, the newest Unity so far: its bundles carry their own type trees, so `unityfs.py` needed no layout; Unity 6 polls `nn::hid::GetDebugPadStates` every frame (`hle_hid.c`), and it takes all its memory itself through `nn::os::AllocateAddressRegion` / `AllocateMemoryPages` with no heap, so it runs in the 5 GB guest layout that `profile_guess.py` now gives every game with an NPDM system resource, with the SDK's address-region node area cut on its `sdk` build (`patch_nso.py`). Catherine: Full Body is Atlus's GFD engine with CRI middleware (CPK archives, ADX2 sound, Sofdec2 movies through `movie::Decoder`, whose player reads `nv12-uv-stride`, `hle_movie.c`): its 3,128 shaders are GLSLC outputs in `data/shaders_binary`, `main` and two compiles of its own GLSLC at boot, which the host lets run by reporting at least that GLSLC's GPU code version (`hle_glslc.c`); that GLSLC writes every vertex shader as a VertexA part (shader type 0, the position) and a VertexB part, which `dmsh_build.py` merges as Ryujinx does (`vertex_ab`; the screen was black without it), and because the game reuses a copy's staging memory a frame after recording it, the host commits a segment at the first pass boundary after a buffer → texture copy (`nvn_mtl.m` `g_cut_copy`). Digimon Story Cyber Sleuth: Complete Edition is Media.Vision's MVGL engine and renders with NVIDIA's desktop OpenGL 4.5 through EGL, not NVN, which the host implements on Metal (`dm/gfx/gl/hle_gl.m`: the 13 EGL imports and the 79 GL functions its code calls of the 2,956 its glad loader fetches, counted with `dm/tools/loader_census.py`; `profile_guess.py` gives `graphics gl` when `main` imports `eglCreateWindowSurface` and names `glShaderSource`); its shaders are GLSL ES 3.20 source in its MVGL archives (`dm/gfx/tools/mvgl.py`: MDB1 with doboz entries) and in `main`, which `gl_shaders.py` translates offline with glslang and SPIRV-Cross (packing needs `brew install glslang`) and the host finds by the hash of its text. Animal Crossing: New Horizons is Nintendo EPD's engine (sead / agl / gsys) on SDK 20.5.4: its `main` links its own HIPC client for `olsc:u` (cloud backup settings), which the host answers with sessions of its own (`hle_sf.c`), its island lives in device save data (`MountDeviceSaveData`, `hle_fs.c`), it skips clears with conditional rendering (profile key `render_cond`), its 63 MB `main` needed halfword jump tables padded only to their largest entry (`patch_nso.py` `half_count`), and its `SaveProc` builds a new island in grid walks that yield once per cell, so a yield whose loop moved its callee-saved registers since the last one now returns at once instead of backing off, for every game (`hle_thread.c` `thr_yield`; the island showed after 77 s of black screen, now about 10 s). DRAGON QUEST VII Reimagined is Unreal Engine 4.27: IoStore containers beside one pak, its shaders LZ4 blocks in the pak's shader code archives, which `uepak.py` decodes itself (`lz4_shaders`), and its allocator replaced as Octopath Traveler 0's (`patch_nso.py` cut and observation gate, `hle_uemalloc.c` layout `UE4_27`) in a 2 GB engine region (profile key `game_args`); its audio renderer runs at 32 kHz, resampled by the host on the CPU path (`hle_audren.c`); and Unreal's volumetric lightmap merge sizes its indirection texture with a query on the image it writes, so a texture slot that shares its c2 word with an image slot reads the bound image (`nvn_mtl.m` `tex_slot_h`) and the translator types such a query from the image (`dmsh_xlate` `ImageTypesOf`): without them every character in the opening cutscene glowed white; it runs at 60 fps (30 on the Switch) because the host writes Unreal's `t.MaxFPS` and `rhi.SyncInterval` in game memory (`hle_fpslock.c`, profile key `fps_target`), the same two limits a 60 fps mod for the base game lifts with code patches (`UEngine::SetMaxFPS` returning at once, the name `r.VSync` broken). FINAL FANTASY RESONANCE is Unreal Engine 5.6.1, the newest Unreal so far: its shader libraries are IoStore chunks in its container, not in its pak, two layers of Oodle that `uepak.py` reads through the game's own decoder in the Mac host with no boot (`Toc`, `io_archive`, `OodleHost`); its 94 MB `main` is 95% without unwind entries, so the rewriter analyses long functions in pieces (`patch_nso.py` `pieces`; one 82 MB function took 100 GB of memory) and treats code before the first unwind entry as a function; its allocator is replaced as Octopath Traveler 0's (UE 5.6's 32-slot FMalloc, `hle_uemalloc.c` layout `UE5_6`), and its engine region is 4064 MB (profile key `game_args`), because the engine's own page allocator never gives address space back (2 GB aborted in the first battle); 2,077 particle shaders need `tex_dyn_slots`; it runs at 60 fps (30 on the Switch) by the host's `rhi.SyncInterval` write (`hle_fpslock.c`, profile key `fps_target`). Luigi's Mansion 3 is Next Level Games' own engine on NVN: its RomFS files are in the engine's `.dict` / `.data` archives, and update 1.4.0 ships `.patch` files beside the base ones, which `shader_split.py` applies as the game does (`nlg_shaders`); the game patches texture handles into its recorded command buffers in place, at the command start + 0x10 it took from `GetCommandMemoryUsed`, so single texture binds are recorded in the hardware's layout (`OP_BIND_TEX1`, every game; without it the title showed a stretched icon and a new game a white screen); its G-buffer is two integer targets, which its pixel shaders write through a second, integer-output entry point (`dmsh_build.py` `fs_uint`, every game); and it runs at 60 fps (30 on the Switch) because the host writes its frame limiter's 33.333 ms, a constant in read-only data, as 1000 / `fps_target` less half a millisecond (`hle_fpslock.c`, with `present_interval display`). Its tessellated draws are skipped (Metal 4 has no tessellation stage) and its geometry shaders do not build; neither was drawn in play so far. [README.md](README.md) is the player-facing page (games, packing, playing); [docs/development.md](docs/development.md) has the internals, the script's options, building by hand, prewarm lists and per-game notes. +- This repository runs Nintendo Switch games natively on Apple platforms: the game's own code runs, rewritten only where Apple's kernel requires it. Forty-three games are playable on the iPhone: Super Mario Bros. Wonder (update 1.2.1, profile `010015100B514000`), Fire Emblem Engage (update 2.0.0, `0100A6301214E000`), Metroid Dread (update 2.1.0, `010093801237C000`), 13 Sentinels: Aegis Rim (update 1.10, `01003FC01670C000`), Super Mario 3D World + Bowser's Fury (update 1.2.2, `010028600EBDA000`), The Legend of Zelda: Breath of the Wild (update 1.9.0, SDK 20, `01007EF00011E000`), The Legend of Zelda: Link's Awakening (update 1.1.0, SDK 8, `01006BB00C6F0000`) Super Mario Odyssey (update 1.4.1, SDK 7, `0100000000010000`), Hades (update 1.0.38233, SDK 9, `0100535012974000`), Hyrule Warriors: Age of Calamity (update 1.3.0, SDK 9.6, `01002B00111A2000`), Fire Emblem: Three Houses (update 1.2.0, SDK 6.5, `010055D009F78000`), Super Mario Galaxy (update 1.3.1, SDK 18, `010099C022B96000`) The Legend of Zelda: Tears of the Kingdom (update 1.4.3, SDK 20, `0100F2C0115B6000`) Monster Hunter Rise (update 16.0.2, SDK 11.4.3, `0100B04011742000`), Ace Attorney Investigations Collection (update 1.0.0.2, SDK 17.5.3, `010005501E68C000`), Hades II (base game, SDK 19.3.9, `0100A00019DE0000`), Unicorn Overlord (update 1.05, SDK 16.2.4, `010069401ADB8000`), Rhythm Heaven Groove (base game 1.0.0, SDK 21.4.0, `0100D9F01D474000`), Octopath Traveler 0 (update 1.0.5, SDK 19.3.13, `01005270232F2000`), Pokémon Legends: Z-A (update 2.0.2, SDK 19.3.11, `0100F43008C44000`), Cuphead (update 1.3.7, SDK 5.5.0, `0100A5C00D162000`), Super Mario Galaxy 2 (update 1.4.0, SDK 18.3.5, `0100FD8022DAA000`), FINAL FANTASY TACTICS - The Ivalice Chronicles (update 1.4.0, SDK 19.3.9, `010038B015560000`), Absolum (update 1.1.1, SDK 18.3.2, `0100A83022A22000`), Captain Toad: Treasure Tracker (update 1.4.2, SDK 8.2.0, `01009BF0072D4000`) and Brain Memory Training Academy (base game 1.0.0, SDK 16.2.4, `0100F5D01DA3A000`), The Great Ace Attorney Chronicles (base game 1.0.0, SDK 10.4.1, `010036E00FB20000`), STEINS;GATE RE:BOOT (update 1.0.3, SDK 20.5.17, `01008BC024FDE000`), Big Brain Academy: Brain vs. Brain (update 1.2.0, SDK 11.4.3, `0100620012D6E000`), Patrick's Parabox (update 1.0.1, SDK 13.3.7, `0100E44016C22000`), Celeste (update 1.4.1.1, SDK 15.3.2, `01002B30028F6000`), DRAGON QUEST XI S: Echoes of an Elusive Age – Definitive Edition (update 1.0.4, SDK 7.3.2, `01006C300E9F0000`), Monster Train 2 (update 1.1.1, SDK 18.3.5, `010051701FB46000`), Raging Loop (base game 1.0.0, SDK 7.3.2, `010020400E1C2000`), DARK SOULS: REMASTERED (update 1.0.3, SDK 4.5.0, `01004AB00A260000`), Paper Mario: The Thousand-Year Door (update 1.0.1, SDK 15.3.2, `0100ECD018EBE000`), Hollow Knight: Silksong (update 1.0.30000, SDK 20.5.8, `010013C00E930000`), Catherine: Full Body (update 1.0.1a, SDK 8.2.1, `0100BF00112C0000`), Digimon Story Cyber Sleuth: Complete Edition (update 1.0.3, SDK 6.4.0, `010014E00DB56000`), Animal Crossing: New Horizons (update 3.0.3, SDK 20.5.4, `01006F8002326000`), DRAGON QUEST VII Reimagined (update 1.2.0, SDK 19.3.1, `0100A9D01C446000`), FINAL FANTASY RESONANCE (update 1.0.1, SDK 20.5.17, `0100606026F12000`) and Luigi's Mansion 3 (update 1.4.0, SDK 7.3.2, `0100DCA0064A6000`). Hades renders with Vulkan, which the host implements on Metal (`dm/gfx/vk/hle_vk.m`; `main` imports Vulkan, not NVN). Age of Calamity is Koei Tecmo's KTGL engine: files in RDB archives, shaders as GLSLC outputs (`shader_split.py` `idrk_shaders`). Three Houses is KTGL too; 9 of its shaders exist only as compiles of the game's own GLSLC at run time, captured once on the Mac (its `main` holds GLSL sources and imports `glslcCompile`). Breath of the Wild and Tears of the Kingdom run at 60 fps through a frame step the host decides in whole vsyncs (`patch_nso.py` `FRAME_STEP`, `nvn_fstep.h`, profile keys `fps_target` and `frame_step`); Tears of the Kingdom has a 5 GB guest layout, and its `sdk` build (Breath of the Wild's) has the SDK's address-region node area cut (`patch_nso.py`): without it the resource heaps ran out of alias region at the first descent from the Great Sky Island and the game played its memory-shortage Blood Moon, which put Link back at the start (its log shows `MSF`). Monster Hunter Rise is Capcom's RE Engine: files in KPKA PAK archives (`shader_split.py` `kpka_jobs`, shaders in SDF files), the base game an XCI, its GPU heap a 64 GB sparse virtual pool (`hle_nvn.c` `vpool_*`, a 4 KB page table the replay reads textures through); it runs at 60 fps (30 on the Switch). Super Mario Galaxy is the Wii game on NERD's Hagi layer (PowerPC code recompiled into `main`), renders with Vulkan, and generates its shaders at run time, which the NSP script captures on the Mac (no SPIR-V file in its RomFS); its pointer is the touch screen in handheld mode (profile key `op_mode`). Ace Attorney Investigations Collection is Unity 2022 IL2CPP: its shaders keep their parameters as blob entries of their own (`shader_split.py` `unity_code_blobs`), and its C# file API goes through the SDK's C library, whose path checks the host answers (`hle_fs.c` `IsPathMounted` and the length limits). Hades II, unlike Hades, renders with NVN; its shaders bind textures at fixed slots up to 34 (slot 34's handle lies in the driver's image table, `dmsh_build.py`), and the two long lines through Melinoë while the staff's Omega attack charges are the game's own aim lines, not a rendering bug. Unicorn Overlord is Vanillaware's engine like 13 Sentinels (shaders compiled into `main`); both need the profile key `alloc_big_align`, which aligns large allocations to 64 KB so the ring heaps the engine cuts from a memory pool fit. Rhythm Heaven Groove is Nintendo's own 2D engine on SDK 21, the newest SDK so far (its DSP command list has no magic, `hle_audren.c`); it plays handheld with Boost (profile keys `op_mode handheld`, `perf_mode boost`), which skips its TV latency calibration and keeps its 1920x1080 draw size. Octopath Traveler 0 is Unreal Engine 5: everything in one pak with an AES-encrypted index whose key is found in `main` (`dm/gfx/tools/uepak.py`), its shaders Oodle-compressed in the pak's shader code archives and decoded once on the Mac by the game's own Oodle (any game with Unreal paks, `DM_OODLE_JOB`), its allocator replaced by the host's because Unreal's reserves 37 GB of address space (`patch_nso.py` cut, `hle_uemalloc.c`), its address region capped by the game's own argument (profile key `game_args`), and the particle shaders that pick a texture slot at run time bound by slot (profile key `tex_dyn_slots`); it runs at 60 fps (30 on the Switch) because the host writes Unreal's `t.MaxFPS` and `rhi.SyncInterval` in game memory (`hle_fpslock.c`, profile key `fps_target`). Pokémon Legends: Z-A is Game Freak's Trinity engine: everything in the packs of `arc/data.trpfs` (FlatBuffers, every file Oodle-compressed), which the shader split decodes with the game's own decoder in the Mac host's tool mode (`shader_split.py` `trinity_shaders`, `hle_oodle.c` `DM_OODLE_SERVE`); its 12 GB StandardAllocator is cut to 4 GB to fit a 5 GB guest layout, and the host refuses maps past the pool as Horizon does (profile key `mem_map_limit`); its characters are skinned with transform feedback (`dmsh_build.py` `vs_tf`, `nvn_mtl.m` `tf_draw`), and its occlusion queries, which decide how characters are animated, need the draw's base instance in the driver segment (c0 + 0x644, `encode_args`). Cuphead is Unity 2017.4 IL2CPP on SDK 5.5.0: Unity 2017's serialized and Shader formats are read by `unityfs.py` / `shader_split.py`, its rewritten modules (80 MB) need the profile key `mem_used` above the default 64 MB, or its `SetHeapSize` fails, and its NVN pool allocator breaks on a zero-size block, so the host never reports a ZCull storage size of 0 (`hle_nvn.c` `zcull_size`). Super Mario Galaxy 2 is Galaxy's Hagi layer with Galaxy's `sdk` build; its movie player reserves 384 MB of virtual address memory, which fits only because a rewriter cut on that `sdk` build shrinks the SDK's address-region node area to 62 MB (`patch_nso.py`; the profile's alias region is 512 MB). FINAL FANTASY TACTICS - The Ivalice Chronicles is Square Enix's engine on NVN: its shaders are GLSLC outputs in loose technique files, which `shader_split.py` reads in any RomFS file no other reader takes; its UI textures are written by the CPU into uncached pools (profile keys `tex_flush_upload`, `tex_dirty_upload`); and two job-pool workers share one auto-clear event, so a build cut makes a worker woken in another's place pass the wake on (`patch_nso.py`). Absolum is FNA (XNA 4 in C#) on .NET NativeAOT with SDL3 and FNA3D's NVN driver: its `main` starts its text with 1.7 MB of data the rewriter leaves alone (`nso_code.py`), its shaders are GLSLC outputs in `shaders.nx.bin`, its picture is flipped by a viewport swizzle the host applies, and everything lives in virtual address memory: malloc's 40 GB StandardAllocator is cut to 2.5 GB and the SDK's node area and large-region alignment are cut (`patch_nso.py`), and the .NET GC's region range is 1.5 GB (profile key `game_env`), to fit a 5 GB guest layout. Captain Toad: Treasure Tracker is Nintendo EPD Tokyo's engine (the `al` library over sead / agl, as 3D World and Odyssey) and needed no host change: it plays handheld because its pointer is the touch screen there (the Joy-Con gyro docked; profile key `op_mode`), and agl's post-effect program that samples its own attachment reads a snapshot (profile key `feedback_snap`). Brain Memory Training Academy is Unity 2022.3 IL2CPP with no bundles: its serialized files carry no type trees and nothing lends it a `Shader` tree, so `unityfs.py` has Unity 2022.3's `Shader` layout written out (`KNOWN_TREES`), and like Ace Attorney Investigations Collection it needs the SDK's address-region node area cut on its `sdk` build (`patch_nso.py`). The Great Ace Attorney Chronicles is Capcom's MT Framework: files in ARC archives of zlib entries, its 460 shaders in one shader package inside one of them (`shader_split.py` `mtspk_shaders`); it copies its textures into a CPU-cached pool after creating them and never flushes (its `main` has no cache maintenance at all), so it needs `tex_flush_upload` and `tex_dirty_upload` like FINAL FANTASY TACTICS. STEINS;GATE RE:BOOT is M2's ADV engine (E-mote characters, PSB archives, H.264 movies through the SDK's movie library): it needed `nn::audio::SetAudioOutVolume` (`hle_audio.c`), and because it decodes every picture into one reused region of a pool that lives for good, the storage-sharing texture table now drops an entry whose storage another texture takes over (`nvn_mtl.m` `alias_take_over`); it draws 1280x720 in both modes by its own choice. Big Brain Academy: Brain vs. Brain is NintendoWare Bezel Engine 1.56: nearly all of RomFS is in Bezel Engine archives of zstd entries, its shaders in their `.bfsha` entries (`shader_split.py` `bea_members`); its `nninitStartup` sets a heap or a 60 GB virtual address region by `IsVirtualAddressMemoryEnabled`, so the host reports the NPDM's system resource (profile key `sys_resource`, 0 here); its activity previews loop the movie decoder with Flush and Start from four threads (`hle_movie.c`); and it plays handheld with Boost (profile keys `op_mode handheld`, `perf_mode boost`), because only handheld does it read the touch screen. Patrick's Parabox is Unity 2020.3 IL2CPP with no bundles: `unityfs.py` has Unity 2020.3's `Shader` layout written out (`KNOWN_TREES`), it needs the SDK's address-region node area cut on its `sdk` build like Brain Memory Training Academy (`patch_nso.py`), and Unity's player hands the shared fonts straight to FreeType, so a game that imports no `ScalableFontEngineHelper::Decode` gets the TTF itself (`hle_sdk16.c` `font_skip`). Celeste is MonoGame for the Switch, its C# compiled to C++ by Sickhead's BRUTE: its shaders are MonoGame effects (MGFX) in its XNB files and in `main` (`shader_split.py` `mgfx_effect`), and it needed no host change. DRAGON QUEST XI S is Unreal Engine 4.18: legacy paks with LZ5 blocks, decoded on the Mac by the game's own LZ5 (`uepak.py`, `hle_oodle.c`), its shaders inline in the cooked packages (`shader_split.py` `ue_legacy_jobs`), and the slice-writing geometry shader of its colour-grading LUT folded into the vertex shader (`dmsh_build.py` `gs_slice`, `nvn_mtl.m` `prog_fold_gs`); Square Enix's sound driver never waits on the renderer event, so audio catch-up frames wait for queue space (`hle_audren.c`); its movie player plays only once every track holds a decoded output, so audio movie decoders keep 32 (`hle_movie.c`); and its Unreal region is capped at 4064 MB (profile key `game_args`, MB up to 4095) with the node area and large-region alignment cut on its SDK 7.3.2 build (`patch_nso.py`) for a 5 GB guest layout; it runs at 60 fps (30 on the Switch) because the host writes Unreal's `t.MaxFPS` and `nvn.SyncInterval` in game memory (`hle_fpslock.c`, profile key `fps_target`). Monster Train 2 is Unity 2022.3 IL2CPP on Super Mario Galaxy's SDK 18.3.5 `sdk` build: Unity creates a thread only when `nn::os::GetFreeThreadCount` (`svcGetInfo` type 24, `hle_svc.c`) is at least 1; its MP4 intro needs a movie audio decoder with more output slots than a video one, because Unity's player holds about half a second of audio before it plays (`hle_movie.c` `MD_NOUT`); and its post-processing compute shaders (the colour grading LUT among them) are Unity `ComputeShader` objects, which `shader_split.py` reads (`unity_compute`) with Unity 2022.3's layout written out in `unityfs.py`. It reads no touch, so it stays docked with the 3D pad layout, and it runs at 60 fps (30 on the Switch) with the profile key `present_interval display`, as its animations follow the time that passed. Raging Loop is Unity 2018.3 IL2CPP with no bundles on DRAGON QUEST XI S's `sdk` and `subsdk0` builds: `unityfs.py` has Unity 2018.3's `Shader` layout written out (`KNOWN_TREES`, an older form than the 2020 and 2022 layouts, with the parameters in each subprogram), and it needed no host change; it reads no touch, so it stays docked with the 2D pad layout. DARK SOULS: REMASTERED is FromSoftware's engine on SDK 4.5.0, the oldest SDK so far: its shaders are GLSLC outputs in BND3 archives inside the dvdbnd archives (`shader_split.py` `bdt_shaders`), and SDK 4's GLSLC outputs are read through their section headers (`glslc_section`); it keeps 12 sampler pools with different samplers under the same IDs, so handles a shader reads from a uniform buffer take the base of the pool bound at replay (`nvn_mtl.m` `ubo_tex_id` / `ubo_samp_id`); and it runs at 60 fps (30 on the Switch) with two rewriter cuts, its frame limiter's 30 and its frame update's fixed 1/30 s step (`patch_nso.py`). Paper Mario: The Thousand-Year Door is Intelligent Systems' engine on NintendoWare, on Celeste's SDK 15.3.2 `sdk` build: nearly every RomFS file is zstd (`.zst`, which `shader_split.py` decompresses by magic); its AngelScript call bridges leave sp misaligned, which a general rewriter rule rounds up (`site_sp_align`); it runs at 60 fps (30 on the Switch) by the host's frame step (`FRAME_STEP`, with the floor of its vsync step cut from 2 to 1); and its three job workers poll the job list with no SVC, which held it at 24 fps on an iPhone until the general spin wait rule (`patch_nso.py` "Spin waits", `hle_thread.c` `dm_spin_wait`). Hollow Knight: Silksong is Unity 6 (6000.0.50f1) IL2CPP, the newest Unity so far: its bundles carry their own type trees, so `unityfs.py` needed no layout; Unity 6 polls `nn::hid::GetDebugPadStates` every frame (`hle_hid.c`), and it takes all its memory itself through `nn::os::AllocateAddressRegion` / `AllocateMemoryPages` with no heap, so it runs in the 5 GB guest layout that `profile_guess.py` now gives every game with an NPDM system resource, with the SDK's address-region node area cut on its `sdk` build (`patch_nso.py`). Catherine: Full Body is Atlus's GFD engine with CRI middleware (CPK archives, ADX2 sound, Sofdec2 movies through `movie::Decoder`, whose player reads `nv12-uv-stride`, `hle_movie.c`): its 3,128 shaders are GLSLC outputs in `data/shaders_binary`, `main` and two compiles of its own GLSLC at boot, which the host lets run by reporting at least that GLSLC's GPU code version (`hle_glslc.c`); that GLSLC writes every vertex shader as a VertexA part (shader type 0, the position) and a VertexB part, which `dmsh_build.py` merges as Ryujinx does (`vertex_ab`; the screen was black without it), and because the game reuses a copy's staging memory a frame after recording it, the host commits a segment at the first pass boundary after a buffer → texture copy (`nvn_mtl.m` `g_cut_copy`). Digimon Story Cyber Sleuth: Complete Edition is Media.Vision's MVGL engine and renders with NVIDIA's desktop OpenGL 4.5 through EGL, not NVN, which the host implements on Metal (`dm/gfx/gl/hle_gl.m`: the 13 EGL imports and the 79 GL functions its code calls of the 2,956 its glad loader fetches, counted with `dm/tools/loader_census.py`; `profile_guess.py` gives `graphics gl` when `main` imports `eglCreateWindowSurface` and names `glShaderSource`); its shaders are GLSL ES 3.20 source in its MVGL archives (`dm/gfx/tools/mvgl.py`: MDB1 with doboz entries) and in `main`, which `gl_shaders.py` translates offline with glslang and SPIRV-Cross (packing needs `brew install glslang`) and the host finds by the hash of its text. Animal Crossing: New Horizons is Nintendo EPD's engine (sead / agl / gsys) on SDK 20.5.4: its `main` links its own HIPC client for `olsc:u` (cloud backup settings), which the host answers with sessions of its own (`hle_sf.c`), its island lives in device save data (`MountDeviceSaveData`, `hle_fs.c`), it skips clears with conditional rendering (profile key `render_cond`), its 63 MB `main` needed halfword jump tables padded only to their largest entry (`patch_nso.py` `half_count`), and its `SaveProc` builds a new island in grid walks that yield once per cell, so a yield whose loop moved its callee-saved registers since the last one now returns at once instead of backing off, for every game (`hle_thread.c` `thr_yield`; the island showed after 77 s of black screen, now about 10 s). DRAGON QUEST VII Reimagined is Unreal Engine 4.27: IoStore containers beside one pak, its shaders LZ4 blocks in the pak's shader code archives, which `uepak.py` decodes itself (`lz4_shaders`), and its allocator replaced as Octopath Traveler 0's (`patch_nso.py` cut and observation gate, `hle_uemalloc.c` layout `UE4_27`) in a 2 GB engine region (profile key `game_args`); its audio renderer runs at 32 kHz, resampled by the host on the CPU path (`hle_audren.c`); and Unreal's volumetric lightmap merge sizes its indirection texture with a query on the image it writes, so a texture slot that shares its c2 word with an image slot reads the bound image (`nvn_mtl.m` `tex_slot_h`) and the translator types such a query from the image (`dmsh_xlate` `ImageTypesOf`): without them every character in the opening cutscene glowed white; it runs at 60 fps (30 on the Switch) because the host writes Unreal's `t.MaxFPS` and `rhi.SyncInterval` in game memory (`hle_fpslock.c`, profile key `fps_target`), the same two limits a 60 fps mod for the base game lifts with code patches (`UEngine::SetMaxFPS` returning at once, the name `r.VSync` broken). FINAL FANTASY RESONANCE is Unreal Engine 5.6.1, the newest Unreal so far: its shader libraries are IoStore chunks in its container, not in its pak, two layers of Oodle that `uepak.py` reads through the game's own decoder in the Mac host with no boot (`Toc`, `io_archive`, `OodleHost`); its 94 MB `main` is 95% without unwind entries, so the rewriter analyses long functions in pieces (`patch_nso.py` `pieces`; one 82 MB function took 100 GB of memory) and treats code before the first unwind entry as a function; its allocator is replaced as Octopath Traveler 0's (UE 5.6's 32-slot FMalloc, `hle_uemalloc.c` layout `UE5_6`), and its engine region is 4064 MB (profile key `game_args`), because the engine's own page allocator never gives address space back (2 GB aborted in the first battle); 2,077 particle shaders need `tex_dyn_slots`; it runs at 60 fps (30 on the Switch) by the host's `rhi.SyncInterval` write (`hle_fpslock.c`, profile key `fps_target`). Luigi's Mansion 3 is Next Level Games' own engine on NVN: its RomFS files are in the engine's `.dict` / `.data` archives, and update 1.4.0 ships `.patch` files beside the base ones, which `shader_split.py` applies as the game does (`nlg_shaders`); the game patches texture handles into its recorded command buffers in place, at the command start + 0x10 it took from `GetCommandMemoryUsed`, so single texture binds are recorded in the hardware's layout (`OP_BIND_TEX1`, every game; without it the title showed a stretched icon and a new game a white screen); its G-buffer is two integer targets, which its pixel shaders write through a second, integer-output entry point (`dmsh_build.py` `fs_uint`, every game); and it runs at 60 fps (30 on the Switch) because the host writes its frame limiter's 33.333 ms, a constant in read-only data, as 1000 / `fps_target` less half a millisecond (`hle_fpslock.c`, with `present_interval display`). Its tessellated draws are skipped (Metal 4 has no tessellation stage) and its geometry shaders do not build; neither was drawn in play so far. [README.md](README.md) is the player-facing page (games, packing, playing); [docs/development.md](docs/development.md) has the internals, the script's options, building by hand, prewarm lists and per-game notes. - Each game's constants and shader tables form a game profile in `dm/games//`, where `` is the base game's title ID: `game.tsv` plus the tables and `dmsh_chain.h`. Mario's profile is `010015100B514000`. Tools pick the profile with `DM_GAME`; `make -C dm/host` and `build_ipa.sh` pick it with `GAME`. There is no default game: set it every time (the tools stop and list the profiles otherwise). Put per-game values in the profile, not in code ([dm/games/README.md](dm/games/README.md)). - Prefer general optimizations and fixes to game-specific rules. Performance work (frame pacing, power, caches, load times) counts here as much as correctness. First look for a change that holds for every game: Horizon / NVN / Vulkan semantics, or what the binary, its SDK version, its imports or its files show. Use a game-specific rule (a profile key, a rewriter cut, a build-ID gate, a per-game branch in the host) only when it is absolutely necessary, that is, when no general rule can do it. - Per-build knowledge is keyed by the NSO build ID: rewriter cuts in `dm/tools/patch_nso.py`, the profile's `build` rows (the builds the game was brought up on, and their display versions; the NSP script packs any other version from a derived profile, without those builds' fixes), and host behaviour that uses fixed module addresses checks `module_t.build_id` (from `fixups.tsv`; `dm_main_build()`). When a new game build appears, carry addresses over with `dm/tools/nso_fnmap.py` and check them by disassembly; never let fixed addresses run on a build they were not found in. diff --git a/dm/games/0100F2C0115B6000/game.tsv b/dm/games/0100F2C0115B6000/game.tsv index 65e82a2..5f91444 100644 --- a/dm/games/0100F2C0115B6000/game.tsv +++ b/dm/games/0100F2C0115B6000/game.tsv @@ -28,7 +28,8 @@ zip_store .zs:.bwav:.mc:.webm:.metallib # ---- Memory ---- # The NPDM declares 24 MB of system resource, so IsVirtualAddressMemoryEnabled is true and the game's memory lives in the alias region (as Engage's, see 0100A6301214E000/game.tsv). # The game also calls SetHeapSize(0x86000000) in nninitStartup (2.1 GB of the pool as its heap; a 2 GB heap region aborts there with 0xd401). -# Measured on the Mac (title, opening cave): the alias region is used up to +1.75 GB (the SDK's ~1 GB node area, then 0.5 GB mapped); what it can map is bounded by the pool less the heap (1.1 GB). +# What it can map is bounded by the pool less the heap (1.1 GB). The sdk build's node area is cut from ~1 GB to 62 MB (dm/tools/patch_nso.py): measured on the Mac, the alias region is then used +# up to +1.75 GB on the Great Sky Island with 1.0 GB mapped. Without the cut it reached +2.25 GB at the first descent and the game took a memory shortage (MSF, a Blood Moon that puts Link back, issue #70). # So 5 GB, the total proven on a phone (one reservation of at most 6 GB without the extended virtual addressing entitlement): 256 MB in front + 2.375 GB alias region + 256 MB stack region + 2.125 GB heap region mem_reserve 0x140000000 mem_alias 0x98000000 diff --git a/dm/host/probe.c b/dm/host/probe.c index 40e4d8a..9c80a69 100644 --- a/dm/host/probe.c +++ b/dm/host/probe.c @@ -249,6 +249,23 @@ static int watch_fault(void *addr, int is_write, void *ucv) { } return 1; } + // Exclusive and atomic stores (STXR / STLXR / STLR / STXP, CAS, SWP, LD) address [Xn] with no offset: one that reaches the watched bytes is reported like a store, every time + // (its new value lands when it runs on the unprotected page). Tears of the Kingdom's memory-shortage flag is set by such a store, and the page it lies on takes many of them + { uint32_t in = *(uint32_t *)pc; int ex = (in & 0x3f000000) == 0x08000000, at = (in & 0x3f200c00) == 0x38200000; // exclusive class (CAS / CASP among them); LD / SWP + if (ex || at) { + _STRUCT_ARM_THREAD_STATE64 *ss = &uc->uc_mcontext->__ss; unsigned rn = (in >> 5) & 31, sz = 1u << (in >> 30); + uint64_t base = rn == 31 ? ss->__sp : rn == 29 ? ss->__fp : rn == 30 ? ss->__lr : ss->__x[rn]; + if ((in & 0xbfa00000) == 0x88200000) sz *= 2; // STXP / STLXP: a register pair + if (base < g_wat + g_wlen && base + sz > g_wat) { + flockfile(stderr); + { extern uint64_t g_t0_ns; uint64_t now_ns(void); uint32_t thr_cur_handle(void); + fprintf(stderr, "[dm] WATCH %#llx [%.4fs] T=%x %s store of %u bytes at %#llx (instruction %08x, not emulated), value before %#llx", (unsigned long long)g_wat, (now_ns() - g_t0_ns) / 1e9, thr_cur_handle(), + ex ? "exclusive" : "atomic", sz, (unsigned long long)base, in, (unsigned long long)watch_val()); } + watch_where("pc", pc); watch_where("lr", ss->__lr); fputc('\n', stderr); + dm_backtrace(ss->__fp); + funlockfile(stderr); + } + } } { static uint64_t seen[16]; static _Atomic int ns; int k = 0, n = atomic_load(&ns); // not a store form the handler carries out: let it run on an unprotected page while (k < n && k < 16 && seen[k] != pc) k++; if (k == n && n < 16) { seen[atomic_fetch_add(&ns, 1) & 15] = pc; flockfile(stderr); fprintf(stderr, "[dm] WATCH: instruction %08x not emulated, page unprotected for it:", *(uint32_t *)pc); watch_where("pc", pc); fputc('\n', stderr); funlockfile(stderr); } } @@ -272,7 +289,56 @@ static void *watch_arm(void *p) { } return NULL; } +// DM_PROBE_POLL=
[+] (hex guest address, up to 8 bytes, 4 by default): when even the watch above misses a write (a page that atomics keep unprotected), a host thread spins on +// the bytes and, on each change, prints old → new and dumps every guest thread with its backtrace (svc_thread_dump, the first 8 changes): the writer is still inside the code that wrote. +// Tears of the Kingdom's memory-shortage flag (a word its frame check compares with 1 and clears) lies on such a page +static uint64_t g_poll, g_plen; +// All threads but this one stopped at once, then their pc, lr and 8 frame-pointer frames read and printed after they run again: the writer is caught a few microseconds after its store +static void poll_snap(void) { + enum { MT = 256, NF = 8 }; static uint64_t fr[MT][NF + 2]; static thread_act_t keep[MT]; + thread_act_array_t ths; mach_msg_type_number_t nth = 0, ns = 0; thread_act_t self = mach_thread_self(); + if (task_threads(mach_task_self(), &ths, &nth) != KERN_SUCCESS) return; + for (mach_msg_type_number_t i = 0; i < nth && ns < MT; i++) if (ths[i] != self && thread_suspend(ths[i]) == KERN_SUCCESS) keep[ns++] = ths[i]; + for (mach_msg_type_number_t i = 0; i < ns; i++) { + arm_thread_state64_t st; mach_msg_type_number_t n = ARM_THREAD_STATE64_COUNT; memset(fr[i], 0, sizeof fr[i]); + if (thread_get_state(keep[i], ARM_THREAD_STATE64, (thread_state_t)&st, &n) != KERN_SUCCESS) continue; + fr[i][0] = __darwin_arm_thread_state64_get_pc(st); fr[i][1] = __darwin_arm_thread_state64_get_lr(st); + uint64_t fp = __darwin_arm_thread_state64_get_fp(st), w[2]; vm_size_t g; + for (int k = 0; k < NF && fp > 0x100000000ull && !vm_read_overwrite(mach_task_self(), fp, 16, (vm_address_t)w, &g) && g == 16; k++) { fr[i][2 + k] = w[1]; fp = w[0]; } + } + for (mach_msg_type_number_t i = 0; i < ns; i++) thread_resume(keep[i]); + for (mach_msg_type_number_t i = 0; i < nth; i++) mach_port_deallocate(mach_task_self(), ths[i]); + vm_deallocate(mach_task_self(), (vm_address_t)ths, nth * sizeof *ths); mach_port_deallocate(mach_task_self(), self); + flockfile(stderr); + for (mach_msg_type_number_t i = 0; i < ns; i++) { // threads with guest code anywhere in their first frames + int guest = 0; for (int k = 0; k < NF + 2; k++) guest |= module_of((void *)fr[i][k]) != NULL; + if (!guest) continue; + fprintf(stderr, "[dm] POLL thread %u:", i); watch_where("pc", fr[i][0]); watch_where("lr", fr[i][1]); + for (int k = 2; k < NF + 2 && fr[i][k]; k++) watch_where("<", fr[i][k]); + fputc('\n', stderr); + } + funlockfile(stderr); +} +static void *poll_run(void *p) { + (void)p; void svc_thread_dump(void); extern uint64_t g_t0_ns; uint64_t now_ns(void); vm_size_t got; + uint64_t old = 0; // read through the kernel: the address is not readable until the game maps it (the heap before SetHeapSize) + while (vm_read_overwrite(mach_task_self(), g_poll, g_plen, (vm_address_t)&old, &got) || got != g_plen) usleep(1000); + LOG("POLL %#llx + %llu armed, value %#llx", (unsigned long long)g_poll, (unsigned long long)g_plen, (unsigned long long)old); + for (int n = 0;;) { + uint64_t v = 0; + if (vm_read_overwrite(mach_task_self(), g_poll, g_plen, (vm_address_t)&v, &got) || v == old) continue; + if (n < 8) poll_snap(); + fprintf(stderr, "[dm] POLL %#llx [%.4fs] %#llx → %#llx\n", (unsigned long long)g_poll, (now_ns() - g_t0_ns) / 1e9, (unsigned long long)old, (unsigned long long)v); + old = v; + if (n++ < 2) svc_thread_dump(); + } + return NULL; +} static void watch_install(void) { + { const char *e = getenv("DM_PROBE_POLL"); + if (e) { char *end; g_poll = strtoull(e, &end, 16); g_plen = *end == '+' ? strtoull(end + 1, 0, 16) : 4; + if (!g_poll || !g_plen || g_plen > 8) die("DM_PROBE_POLL format:
[+] (hex)"); + pthread_t t; pthread_create(&t, NULL, poll_run, NULL); pthread_detach(t); } } const char *e = getenv("DM_PROBE_WATCH"); if (!e) return; char *end; g_wat = strtoull(e, &end, 16); g_wlen = *end == '+' ? strtoull(end + 1, 0, 16) : 8; if (!g_wat || !g_wlen) die("DM_PROBE_WATCH format:
[+] (hex)"); diff --git a/dm/tools/patch_nso.py b/dm/tools/patch_nso.py index 13e039a..b789c49 100644 --- a/dm/tools/patch_nso.py +++ b/dm/tools/patch_nso.py @@ -138,6 +138,16 @@ CUTS = { (0x145bfc, '08fca752', '087ca052', 'AddressRegionManager constructor: node area limit mov w8,#0x3fe00000 → mov w8,#0x3e00000'), (0x145c68, '09fca752', '097ca052', 'AddressRegionManager constructor: length of the node area registered as an occupied region mov w9,#0x3fe00000 → mov w9,#0x3e00000'), ], + # SDK 20.5.22 sdk (The Legend of Zelda: Tears of the Kingdom 1.4.3 and Breath of the Wild 1.9.0, the same build): the same node area cut. Tears of the Kingdom's resource heaps live in + # address regions after the node area. At the first descent from the Great Sky Island the alias region's high-water mark was +2.25 GB of its 2.375 GB with 1.00 GB mapped, and the game then + # took a memory shortage: its frame check (main+0x1fd32ec) logs MSF / R / Rf and plays the memory-shortage Blood Moon, which puts Link back at the start (GitHub issue #70; 7 of 8 Mac runs from + # the issue's save). With an 8 GB alias region the mark went on to +2.50 GB, with this cut it stays at +1.75 GB, and neither showed the shortage. The constructor is at sdk+0x145cfc + # (limit = alias start + 0x3fe00000 at +0x28, a unit heap of 0x30-byte nodes, the area then registered as occupied at +0x145d78); VammManager::AllocateAddressRegion (sdk+0x145df0) aligns + # regions to 64 KB or 2 MB only, so no alignment cut is needed + '574284b40c06cf1821a44857939083be': [ + (0x145d0c, '08fca752', '087ca052', 'AddressRegionManager constructor: node area limit mov w8,#0x3fe00000 → mov w8,#0x3e00000'), + (0x145d78, '09fca752', '097ca052', 'AddressRegionManager constructor: length of the node area registered as an occupied region mov w9,#0x3fe00000 → mov w9,#0x3e00000'), + ], # Super Mario Galaxy 1.3.1 / Super Mario Galaxy 2 1.4.0 sdk (SDK 18.3.5, the same build in both): the same node area cut. Galaxy 2's movie player initializes its StandardAllocator with # virtual address memory (main+0x62c280: Initialize(nullptr, 384 MB)), which aborted at its CentralHeap::Init: a 256 MB alias region holds no region after a 1 GB node area. # The constructor is at sdk+0x13be3c (limit = alias start + 0x3fe00000 at +0x28, a unit heap of 0x30-byte nodes, the area then registered as occupied at +0x13bee0) diff --git a/docs/agents/05-debugging.md b/docs/agents/05-debugging.md index 2a455b1..e7f7b43 100644 --- a/docs/agents/05-debugging.md +++ b/docs/agents/05-debugging.md @@ -51,6 +51,10 @@ The `!xN` heap walk takes the layout of the game's sead version from `DM_PROBE_H **Who writes a word: `DM_PROBE_WATCH=
[+]`** (hex guest address, 8 bytes by default; the bytes must lie in one 16 KB host page). The host write-protects that page as soon as it is writable and keeps it protected: the fault handler carries out every store that lands on the page through a second, writable mapping of it (STR / STUR / STP of general and SIMD registers in their immediate, pre- and post-index forms, register-offset STR, DC ZVA), so the stores of a busy page all pass through it. A store that reaches the watched bytes prints `WATCH … store of N bytes at …: old → new`, the thread, pc, lr and a backtrace. Other instructions (atomics, exclusives) get a `not emulated` line once per pc and run on an unprotected page, protected again a millisecond later. A guest object sits at the same address in every run (the guest heap is laid out the same), so take the address from a probe's register dump first. STEINS;GATE RE:BOOT's render size setting (engine object + 0x6000): the static search for stores with that offset, conditional probes on 25 candidate stores and a wrapper around the host's `memcpy` / `memset` (the guest's go straight to the host's libc, `hle_table.c`, where guest probes cannot see them) all found nothing; a watch that unprotected the page for every fault missed the write among the bursts of stores to the engine object's page; this one found it on the first run (`main+0x389e90`, a setter its start-up code calls with 0). +An exclusive or atomic store (`STXR`, `STLR`, `CAS`, `SWP`, `LD`) that reaches the watched bytes prints `WATCH … exclusive store` (or `atomic store`) with its pc and backtrace, every time, though it is not emulated. A page that takes many atomics stays unprotected most of the time, so even that can miss: Tears of the Kingdom's memory-shortage flag (a word its frame check compares with 1 and clears) shares its page with atomic counters, and the watch saw neither the write nor the check's own clear. + +**When the watch misses: `DM_PROBE_POLL=
[+]`** (hex guest address, 4 bytes by default, up to 8). A host thread spins on the bytes; on each change it stops every thread at once, reads each one's pc, lr and 8 frame-pointer frames, lets them run again, then prints `POLL thread …` for the threads with guest code in those frames, the change (`POLL … old → new`) and, for the first two changes, the full guest thread dump. The snapshot comes a few microseconds after the store; a writer that went on into a wait shows only its wait (for that flag the one thread in guest code was `PlacementMgr` in `main+0x5aaac0`, both times). It costs one core while it runs. + A probe's backtrace reads each frame with `vm_read_overwrite`: code built without frame pointers keeps data in x29, and the walk used to fault in the host on such a value (x29 = 0x430 at `main+0x38921c` in the same game). Use it when lldb is unavailable: after macOS revokes the debugger authorization (`DevToolsSecurity -status` says developer mode is disabled), `lldb` launches hang waiting for a GUI approval prompt. `sudo DevToolsSecurity -enable` (run by the user) restores it. diff --git a/docs/development.md b/docs/development.md index a4de183..e11fe0d 100644 --- a/docs/development.md +++ b/docs/development.md @@ -321,6 +321,7 @@ The game's own work and its frame rate stay as they are; the host avoids work of - **60 fps at normal speed:** the game picks a present interval each frame (2 = 30 fps, 3 under load) and steps its game time by 60 / fps. Stubs in `patch_nso.py` (`FRAME_STEP`; the sites come from masagrator's FPSLocker config for this build) replace that logic with the host's frame step in whole vsyncs, as in Breath of the Wild: int fps = 60 / step. Prerendered movies hold 30. On the Mac (M4 Pro) the opening cave holds 60 at 1.000 of real time; with 12 ms added to every frame it drops to 30 after three late frames and stays at 1.000. `DM_FPS_TARGET=30` saves power, `0` gives the game's own step back (every stub then runs the original instructions). - **Before the host has decided a step** (the first frames): the stub steps by the measured frame time, capped at the target. Game time that ran ahead after a short frame is paid back by the next frames (the idea from OJZen/totk-mobile); frames longer than 1/15 s are not caught up. - **Render scale:** `DM_RENDER_SCALE=0.5`–`1.0` multiplies the size of the 3D views (1600x896 docked) and fixes it there; the game's own dynamic resolution runs only at 1.0. The HUD keeps its resolution. +- **Address space (GitHub issue #70):** the game's resource heaps are address regions in the alias region (2.375 GB in the 5 GB guest layout, 64 GB on the Switch). With the SDK's ~1 GB node area at its start, the alias region was used up to +2.25 GB at the first descent from the Great Sky Island with 1.00 GB mapped, and the game then took a memory shortage: its frame check (main+0x1fd32ec) logs `MSF`, then `R` and `Rf` (reason bits of one resource heap's object), and plays the memory-shortage Blood Moon, which puts Link back at the start. A rewriter cut on the `sdk` build (the one Breath of the Wild 1.9.0 also uses) shrinks the node area to 62 MB, as Fire Emblem Engage's, Hollow Knight: Silksong's and other games' builds have it: the high-water mark is +1.75 GB there, and no shortage came in 3.7 minutes on the island (it came within a minute in 7 of 8 runs without it). Route from the issue's save on the Mac: `DM_KEYS="20:A*6/4,f2480:LUp~5"` (title, load, jump, steer into the pond). **Monster Hunter Rise (`0100B04011742000`)** - **Engine:** Capcom's RE Engine, SDK 11.4.3, update 16.0.2 (v2228224, Sunbreak). The base game is an XCI (cartridge dump; nx_extract reads its secure partition), the update an NSP, and its 252 DLC NSPs are licences only. `main` has 125 MB of text: the rewriter puts its patch area right before the base (`BIG_TEXT`, a second pass with `DM_PATCH_FRONT`).