One translation chain version for every game: DMSH_CHAIN in dm/gfx/nvn/dmsh_chain.h replaces the 47 dm/games/<id>/dmsh_chain.h, which every chain change rewrote and some were left behind (GitHub issue #78). The version hashed each profile's shader tables and nvn_* / shader_* keys too, once the only thing that kept the host off a library built from older tables; each shader's cache key (dmsh_build.py shader_key) has covered them since, so the version now covers only the chain's tools (dmsh_chain.py, dmsh_build.py, dmsh_fetch.py, dmsh_vin.py, dmsh_geometry.py, dmsh_facts.py, the translator's source, ios_metal.py, the lock) and the options, and needs no DM_GAME. What the per-game part still did is done by the library itself: a full dmsh_build.py run stamps b-<V>/inputs.tsv (Mac and iOS) with the game and the digest of the profile's inputs (dmsh_chain.py inputs / --inputs: the four shader tables, bf_want.tsv, which the variants use but the version never counted, and the keys), starts from no shader outputs (a partial one from none of the keys it builds), so a key that left the map, a shader that no longer builds or a variant no longer made keeps no old library, refuses a directory stamped for another game, and writes the profile's c1_full.tsv and cb_extent.tsv from its Mac outputs; a partial build keeps the stamp only while it names the current inputs. c1_guard.py / cb_extent.py head their table with the library's chain version and stamp (table_head), and the host takes a table only beside that stamp, logs the stamp and reports one of another game; build_ipa.sh reads the one header and stops when the iOS libraries' stamp is not this profile's current inputs or c1_full.tsv names another. NSP script: chain_header writes the one header when stale, before any host is built; provisional and derived profiles no longer write one; chain_tables makes a table again when it does not name the reused library's stamp (False without one). Makefiles take the header beside nvn_mtl.m; Xcode projects prepared earlier point at a profile header that is gone and are told to prepare again. Tests: test_nsp_chain.py (the committed header is current, stale and missing ones are written, tables against the stamp). Checked on The Great Ace Attorney Chronicles: a full build gave 460 of 460 shaders and both tables (0 and 409 entries, as before), the Mac host booted with the stamp and both tables past frame 900; one added facts row rebuilt one shader under the same chain version with a new digest and new tables, and the host refused the old tables beside the new stamp; a full build removed planted outputs, another game's build into that directory stopped, a partial build after an input change dropped the stamp; build_ipa.sh stopped on a stale stamp, a missing one and a stale or old-form c1_full.tsv; three NSP packs built (no table lines: dmsh_build made them), reused everything, and remade deleted tables from the reused build. Graphics unit tests give the same results as before (on this Mac t_refl needs ~/dm-work's libraries and several GPU tests fail on the previous commit too). AGENTS.md's chain procedure (header once after a tool change; a full shader build makes the tables), dm/games/README.md, dm/gfx/README.md and docs/development.md say so

This commit is contained in:
seedds committed 2026-10-23 23:37:03 +08:00
1 parent f4f0a85d6e
commit 86d9ba1239
70 files changed
+248 -243

No files matched your search

+6 -7
View File
@@ -1,19 +1,18 @@
# Notes for agents
- This repository runs Nintendo Switch games natively on Apple platforms: the game's own code runs, rewritten only where Apple's kernel requires it. Forty-seven games are playable on the iPhone: Super Mario Bros. Wonder (update 1.2.1, profile `010015100B514000`), Fire Emblem Engage (update 2.0.0, `0100A6301214E000`), Metroid Dread (update 2.1.0, `010093801237C000`), 13 Sentinels: Aegis Rim (update 1.10, `01003FC01670C000`), Super Mario 3D World + Bowser's Fury (update 1.2.2, `010028600EBDA000`), The Legend of Zelda: Breath of the Wild (update 1.9.0, SDK 20, `01007EF00011E000`), The Legend of Zelda: Link's Awakening (update 1.1.0, SDK 8, `01006BB00C6F0000`) Super Mario Odyssey (update 1.4.1, SDK 7, `0100000000010000`), Hades (update 1.0.38233, SDK 9, `0100535012974000`), Hyrule Warriors: Age of Calamity (update 1.3.0, SDK 9.6, `01002B00111A2000`), Fire Emblem: Three Houses (update 1.2.0, SDK 6.5, `010055D009F78000`), Super Mario Galaxy (update 1.3.1, SDK 18, `010099C022B96000`) The Legend of Zelda: Tears of the Kingdom (update 1.4.3, SDK 20, `0100F2C0115B6000`) Monster Hunter Rise (update 16.0.2, SDK 11.4.3, `0100B04011742000`), Ace Attorney Investigations Collection (update 1.0.0.2, SDK 17.5.3, `010005501E68C000`), Hades II (base game, SDK 19.3.9, `0100A00019DE0000`), Unicorn Overlord (update 1.05, SDK 16.2.4, `010069401ADB8000`), Rhythm Heaven Groove (base game 1.0.0, SDK 21.4.0, `0100D9F01D474000`), Octopath Traveler 0 (update 1.0.5, SDK 19.3.13, `01005270232F2000`), Pokémon Legends: Z-A (update 2.0.2, SDK 19.3.11, `0100F43008C44000`), Cuphead (update 1.3.7, SDK 5.5.0, `0100A5C00D162000`), Super Mario Galaxy 2 (update 1.4.0, SDK 18.3.5, `0100FD8022DAA000`), FINAL FANTASY TACTICS - The Ivalice Chronicles (update 1.4.0, SDK 19.3.9, `010038B015560000`), Absolum (update 1.1.1, SDK 18.3.2, `0100A83022A22000`), Captain Toad: Treasure Tracker (update 1.4.2, SDK 8.2.0, `01009BF0072D4000`) and Brain Memory Training Academy (base game 1.0.0, SDK 16.2.4, `0100F5D01DA3A000`), The Great Ace Attorney Chronicles (base game 1.0.0, SDK 10.4.1, `010036E00FB20000`), STEINS;GATE RE:BOOT (update 1.0.3, SDK 20.5.17, `01008BC024FDE000`), Big Brain Academy: Brain vs. Brain (update 1.2.0, SDK 11.4.3, `0100620012D6E000`), Patrick's Parabox (update 1.0.1, SDK 13.3.7, `0100E44016C22000`), Celeste (update 1.4.1.1, SDK 15.3.2, `01002B30028F6000`), DRAGON QUEST XI S: Echoes of an Elusive Age – Definitive Edition (update 1.0.4, SDK 7.3.2, `01006C300E9F0000`), Monster Train 2 (update 1.1.1, SDK 18.3.5, `010051701FB46000`), Raging Loop (base game 1.0.0, SDK 7.3.2, `010020400E1C2000`), DARK SOULS: REMASTERED (update 1.0.3, SDK 4.5.0, `01004AB00A260000`), Paper Mario: The Thousand-Year Door (update 1.0.1, SDK 15.3.2, `0100ECD018EBE000`), Hollow Knight: Silksong (update 1.0.30000, SDK 20.5.8, `010013C00E930000`), Catherine: Full Body (update 1.0.1a, SDK 8.2.1, `0100BF00112C0000`), Digimon Story Cyber Sleuth: Complete Edition (update 1.0.3, SDK 6.4.0, `010014E00DB56000`), Animal Crossing: New Horizons (update 3.0.3, SDK 20.5.4, `01006F8002326000`), DRAGON QUEST VII Reimagined (update 1.2.0, SDK 19.3.1, `0100A9D01C446000`), FINAL FANTASY RESONANCE (update 1.0.1, SDK 20.5.17, `0100606026F12000`), Luigi's Mansion 3 (update 1.4.0, SDK 7.3.2, `0100DCA0064A6000`), AI: THE SOMNIUM FILES (update 1.0.1, SDK 6.4.2, `010089B00D09C000`), ANIMAL WELL (update 1.0.10, SDK 20.5.7, `010020D01AD24000`), AI: THE SOMNIUM FILES - nirvanA Initiative (update 1.0.2, SDK 13.3.5, `0100713016CCC000`) and Mario Kart 8 Deluxe (update 4.0.0, SDK 4.4.0, `0100152000022000`). Hades renders with Vulkan, which the host implements on Metal (`dm/gfx/vk/hle_vk.m`; `main` imports Vulkan, not NVN). Age of Calamity is Koei Tecmo's KTGL engine: files in RDB archives, shaders as GLSLC outputs (`shader_split.py` `idrk_shaders`). Three Houses is KTGL too; 9 of its shaders exist only as compiles of the game's own GLSLC at run time, captured once on the Mac (its `main` holds GLSL sources and imports `glslcCompile`). Breath of the Wild and Tears of the Kingdom run at 60 fps through a frame step the host decides in whole vsyncs (`FRAME_STEP` in their profiles' `patches.py`, `nvn_fstep.h`, profile keys `fps_target` and `frame_step`); Tears of the Kingdom has a 5 GB guest layout, and its `sdk` build (Breath of the Wild's) has the SDK's address-region node area cut (`sdk_patches.py`): without it the resource heaps ran out of alias region at the first descent from the Great Sky Island and the game played its memory-shortage Blood Moon, which put Link back at the start (its log shows `MSF`). Monster Hunter Rise is Capcom's RE Engine: files in KPKA PAK archives (`shader_split.py` `kpka_jobs`, shaders in SDF files), the base game an XCI, its GPU heap a 64 GB sparse virtual pool (`hle_nvn.c` `vpool_*`, a 4 KB page table the replay reads textures through); it runs at 60 fps (30 on the Switch). Super Mario Galaxy is the Wii game on NERD's Hagi layer (PowerPC code recompiled into `main`), renders Line truncated
- Each game's constants and shader tables form a game profile in `dm/games/<id>/`, where `<id>` is the base game's title ID: `game.tsv` plus the tables (and `patches.py` when the rewriter has patches for its builds) and `dmsh_chain.h`. Mario's profile is `010015100B514000`. Tools pick the profile with `DM_GAME`; `make -C dm/host` and `build_ipa.sh` pick it with `GAME`. There is no default game: set it every time (the tools stop and list the profiles otherwise). Put per-game values in the profile, not in code ([dm/games/README.md](dm/games/README.md)).
- Each game's constants and shader tables form a game profile in `dm/games/<id>/`, where `<id>` is the base game's title ID: `game.tsv` plus the tables (and `patches.py` when the rewriter has patches for its builds). Mario's profile is `010015100B514000`. Tools pick the profile with `DM_GAME`; `make -C dm/host` and `build_ipa.sh` pick it with `GAME`. There is no default game: set it every time (the tools stop and list the profiles otherwise). Put per-game values in the profile, not in code ([dm/games/README.md](dm/games/README.md)).
- Prefer general optimizations and fixes to game-specific rules. Performance work (frame pacing, power, caches, load times) counts here as much as correctness. First look for a change that holds for every game: Horizon / NVN / Vulkan semantics, or what the binary, its SDK version, its imports or its files show. Use a game-specific rule (a profile key, a rewriter cut, a build-ID gate, a per-game branch in the host) only when it is absolutely necessary, that is, when no general rule can do it.
- Per-build knowledge is keyed by the NSO build ID: rewriter cuts, observation gates and frame-step stubs in the profile's `patches.py` (a game's own modules) or `dm/tools/sdk_patches.py` (Nintendo's `sdk` modules, which several games ship), which `dm/tools/patch_nso.py` (the general rewriter) loads, the profile's `build` rows (the builds the game was brought up on, and their display versions; the NSP script packs any other version from a derived profile, without those builds' fixes), and host behaviour that uses fixed module addresses checks `module_t.build_id` (from `fixups.tsv`; `dm_main_build()`). When a new game build appears, carry addresses over with `dm/tools/nso_fnmap.py` and check them by disassembly; never let fixed addresses run on a build they were not found in.
- To analyse or port **another** Switch title, first run `scripts/build_ipa_from_nsp.py` on its NSP folder. With no matching profile, it writes a survey to `<NSP folder>/<title ID>_survey/survey.md`, then a provisional profile guessed from the game files (`dm/games/profile_guess.py`; `<NSP folder>/<title ID>_build/profile/<title ID>/`, found by the tools through `DM_GAMES_DIR`), with a short heap check on the Mac when the game has virtual address memory, and a best-effort IPA that may not boot. Then follow [docs/agents](docs/agents/README.md) (extraction, binary survey, host bring-up failure catalogue, graphics specifics, debugging recipes) and add a profile.
- Scripts the user runs live in `scripts/` at the repository root (now `build_ipa_from_nsp.py`). Each one runs both in Spyder (F5, configured by the settings block at its top) and from a terminal (command-line options override those settings; non-zero exit on error). Tools that only agents or other scripts call stay under `dm/`.
- Game files, keys and anything extracted from them live in the game's NSP folder, never in this repository and never in the home folder (it is nearly full): the work folder `<NSP folder>/<id>-work` for extraction, runs and analysis, beside the NSP script's `<id>_build/`. Tool venvs and other large scratch data go on that drive too. Do not print key values.
- Language: everything is in English: `README.md`, docs, code comments, log messages, script output and other files. Red log lines (something unimplemented or wrong) start with `[RED] `. Text that must match a game's own Chinese data is written as escapes (`"\xE8\xB5\xB7"`), never as characters.
- After changing anything on the shader translation chain, run the chain procedure for each affected profile. The chain covers `dm/gfx/tools/dmsh_*`, `Program.cs`, the profile's `shader_map.tsv` / `shader_facts.tsv` / `shader_pairs.tsv`, and its `shader_*` keys. The procedure:
1. Run `dmsh_chain.py --header`, which writes the profile's `dmsh_chain.h`.
2. Rebuild the shaders (Mac and, with `IOSOUT`, iOS).
3. Regenerate the profile's `c1_full.tsv` (`c1_guard.py`) and `cb_extent.tsv` (`cb_extent.py`) for Mac runs. They are made from each build's Mac outputs and git ignores them; the NSP script makes its own.
4. Rebuild the host.
- The shader translation chain has one version for every game, `DMSH_CHAIN` in `dm/gfx/nvn/dmsh_chain.h`, which the host is compiled with and loads only libraries of. It covers the chain's tools: `dm/gfx/tools/dmsh_*` (`dmsh_build.py`, `dmsh_chain.py`, `dmsh_fetch.py`, `dmsh_vin.py`, `dmsh_geometry.py`, `dmsh_facts.py`), `Program.cs`, `ios_metal.py` and `dmsh_chain.lock`. A profile's own inputs (`shader_map.tsv`, `shader_facts.tsv`, `shader_pairs.tsv`, `bf_want.tsv`, its `shader_*` / `nvn_*` keys) are not part of it: they are in each shader's cache key, and their digest stamps the libraries (`b-<V>/inputs.tsv`). The procedure:
1. After changing a chain tool, run `dmsh_chain.py --header` once and commit `dm/gfx/nvn/dmsh_chain.h` with the change (`dm/tools/tests/test_nsp_chain.py` fails on a stale one).
2. After that, or after changing a profile's inputs, rebuild that profile's shaders: a full `dmsh_build.py` run (Mac and, with `IOSOUT`, iOS) stamps its libraries and writes the profile's `c1_full.tsv` and `cb_extent.tsv` from its Mac outputs (git ignores them; the host takes them only beside the stamp they name). Give each game its own output directory.
3. Rebuild the host.
The host only loads libraries of its own chain version. The translator is `dm/gfx/tools/dmsh_xlate_bin`, which matches `dmsh_chain.lock`; a fresh `dotnet publish` does not.
The translator is `dm/gfx/tools/dmsh_xlate_bin`, which matches `dmsh_chain.lock`; a fresh `dotnet publish` does not.
- iOS builds are unsigned IPAs that the user sideloads and tests; diagnostics come back as the app's `Documents/log.txt`. Do not sign or install builds on the user's devices.
- Commit only when asked, with `git -c user.name=seedds -c user.email=seedds@gmail.com commit`.