One translation chain version for every game: DMSH_CHAIN in dm/gfx/nvn/dmsh_chain.h replaces the 47 dm/games/<id>/dmsh_chain.h, which every chain change rewrote and some were left behind (GitHub issue #78). The version hashed each profile's shader tables and nvn_* / shader_* keys too, once the only thing that kept the host off a library built from older tables; each shader's cache key (dmsh_build.py shader_key) has covered them since, so the version now covers only the chain's tools (dmsh_chain.py, dmsh_build.py, dmsh_fetch.py, dmsh_vin.py, dmsh_geometry.py, dmsh_facts.py, the translator's source, ios_metal.py, the lock) and the options, and needs no DM_GAME. What the per-game part still did is done by the library itself: a full dmsh_build.py run stamps b-<V>/inputs.tsv (Mac and iOS) with the game and the digest of the profile's inputs (dmsh_chain.py inputs / --inputs: the four shader tables, bf_want.tsv, which the variants use but the version never counted, and the keys), starts from no shader outputs (a partial one from none of the keys it builds), so a key that left the map, a shader that no longer builds or a variant no longer made keeps no old library, refuses a directory stamped for another game, and writes the profile's c1_full.tsv and cb_extent.tsv from its Mac outputs; a partial build keeps the stamp only while it names the current inputs. c1_guard.py / cb_extent.py head their table with the library's chain version and stamp (table_head), and the host takes a table only beside that stamp, logs the stamp and reports one of another game; build_ipa.sh reads the one header and stops when the iOS libraries' stamp is not this profile's current inputs or c1_full.tsv names another. NSP script: chain_header writes the one header when stale, before any host is built; provisional and derived profiles no longer write one; chain_tables makes a table again when it does not name the reused library's stamp (False without one). Makefiles take the header beside nvn_mtl.m; Xcode projects prepared earlier point at a profile header that is gone and are told to prepare again. Tests: test_nsp_chain.py (the committed header is current, stale and missing ones are written, tables against the stamp). Checked on The Great Ace Attorney Chronicles: a full build gave 460 of 460 shaders and both tables (0 and 409 entries, as before), the Mac host booted with the stamp and both tables past frame 900; one added facts row rebuilt one shader under the same chain version with a new digest and new tables, and the host refused the old tables beside the new stamp; a full build removed planted outputs, another game's build into that directory stopped, a partial build after an input change dropped the stamp; build_ipa.sh stopped on a stale stamp, a missing one and a stale or old-form c1_full.tsv; three NSP packs built (no table lines: dmsh_build made them), reused everything, and remade deleted tables from the reused build. Graphics unit tests give the same results as before (on this Mac t_refl needs ~/dm-work's libraries and several GPU tests fail on the previous commit too). AGENTS.md's chain procedure (header once after a tool change; a full shader build makes the tables), dm/games/README.md, dm/gfx/README.md and docs/development.md say so
This commit is contained in:
1 parent
f4f0a85d6e
commit
86d9ba1239
70 files changed
+248
-243
No files matched your search
@@ -1,19 +1,18 @@
|
||||
# Notes for agents
|
||||
|
||||
- This repository runs Nintendo Switch games natively on Apple platforms: the game's own code runs, rewritten only where Apple's kernel requires it. Forty-seven games are playable on the iPhone: Super Mario Bros. Wonder (update 1.2.1, profile `010015100B514000`), Fire Emblem Engage (update 2.0.0, `0100A6301214E000`), Metroid Dread (update 2.1.0, `010093801237C000`), 13 Sentinels: Aegis Rim (update 1.10, `01003FC01670C000`), Super Mario 3D World + Bowser's Fury (update 1.2.2, `010028600EBDA000`), The Legend of Zelda: Breath of the Wild (update 1.9.0, SDK 20, `01007EF00011E000`), The Legend of Zelda: Link's Awakening (update 1.1.0, SDK 8, `01006BB00C6F0000`) Super Mario Odyssey (update 1.4.1, SDK 7, `0100000000010000`), Hades (update 1.0.38233, SDK 9, `0100535012974000`), Hyrule Warriors: Age of Calamity (update 1.3.0, SDK 9.6, `01002B00111A2000`), Fire Emblem: Three Houses (update 1.2.0, SDK 6.5, `010055D009F78000`), Super Mario Galaxy (update 1.3.1, SDK 18, `010099C022B96000`) The Legend of Zelda: Tears of the Kingdom (update 1.4.3, SDK 20, `0100F2C0115B6000`) Monster Hunter Rise (update 16.0.2, SDK 11.4.3, `0100B04011742000`), Ace Attorney Investigations Collection (update 1.0.0.2, SDK 17.5.3, `010005501E68C000`), Hades II (base game, SDK 19.3.9, `0100A00019DE0000`), Unicorn Overlord (update 1.05, SDK 16.2.4, `010069401ADB8000`), Rhythm Heaven Groove (base game 1.0.0, SDK 21.4.0, `0100D9F01D474000`), Octopath Traveler 0 (update 1.0.5, SDK 19.3.13, `01005270232F2000`), Pokémon Legends: Z-A (update 2.0.2, SDK 19.3.11, `0100F43008C44000`), Cuphead (update 1.3.7, SDK 5.5.0, `0100A5C00D162000`), Super Mario Galaxy 2 (update 1.4.0, SDK 18.3.5, `0100FD8022DAA000`), FINAL FANTASY TACTICS - The Ivalice Chronicles (update 1.4.0, SDK 19.3.9, `010038B015560000`), Absolum (update 1.1.1, SDK 18.3.2, `0100A83022A22000`), Captain Toad: Treasure Tracker (update 1.4.2, SDK 8.2.0, `01009BF0072D4000`) and Brain Memory Training Academy (base game 1.0.0, SDK 16.2.4, `0100F5D01DA3A000`), The Great Ace Attorney Chronicles (base game 1.0.0, SDK 10.4.1, `010036E00FB20000`), STEINS;GATE RE:BOOT (update 1.0.3, SDK 20.5.17, `01008BC024FDE000`), Big Brain Academy: Brain vs. Brain (update 1.2.0, SDK 11.4.3, `0100620012D6E000`), Patrick's Parabox (update 1.0.1, SDK 13.3.7, `0100E44016C22000`), Celeste (update 1.4.1.1, SDK 15.3.2, `01002B30028F6000`), DRAGON QUEST XI S: Echoes of an Elusive Age – Definitive Edition (update 1.0.4, SDK 7.3.2, `01006C300E9F0000`), Monster Train 2 (update 1.1.1, SDK 18.3.5, `010051701FB46000`), Raging Loop (base game 1.0.0, SDK 7.3.2, `010020400E1C2000`), DARK SOULS: REMASTERED (update 1.0.3, SDK 4.5.0, `01004AB00A260000`), Paper Mario: The Thousand-Year Door (update 1.0.1, SDK 15.3.2, `0100ECD018EBE000`), Hollow Knight: Silksong (update 1.0.30000, SDK 20.5.8, `010013C00E930000`), Catherine: Full Body (update 1.0.1a, SDK 8.2.1, `0100BF00112C0000`), Digimon Story Cyber Sleuth: Complete Edition (update 1.0.3, SDK 6.4.0, `010014E00DB56000`), Animal Crossing: New Horizons (update 3.0.3, SDK 20.5.4, `01006F8002326000`), DRAGON QUEST VII Reimagined (update 1.2.0, SDK 19.3.1, `0100A9D01C446000`), FINAL FANTASY RESONANCE (update 1.0.1, SDK 20.5.17, `0100606026F12000`), Luigi's Mansion 3 (update 1.4.0, SDK 7.3.2, `0100DCA0064A6000`), AI: THE SOMNIUM FILES (update 1.0.1, SDK 6.4.2, `010089B00D09C000`), ANIMAL WELL (update 1.0.10, SDK 20.5.7, `010020D01AD24000`), AI: THE SOMNIUM FILES - nirvanA Initiative (update 1.0.2, SDK 13.3.5, `0100713016CCC000`) and Mario Kart 8 Deluxe (update 4.0.0, SDK 4.4.0, `0100152000022000`). Hades renders with Vulkan, which the host implements on Metal (`dm/gfx/vk/hle_vk.m`; `main` imports Vulkan, not NVN). Age of Calamity is Koei Tecmo's KTGL engine: files in RDB archives, shaders as GLSLC outputs (`shader_split.py` `idrk_shaders`). Three Houses is KTGL too; 9 of its shaders exist only as compiles of the game's own GLSLC at run time, captured once on the Mac (its `main` holds GLSL sources and imports `glslcCompile`). Breath of the Wild and Tears of the Kingdom run at 60 fps through a frame step the host decides in whole vsyncs (`FRAME_STEP` in their profiles' `patches.py`, `nvn_fstep.h`, profile keys `fps_target` and `frame_step`); Tears of the Kingdom has a 5 GB guest layout, and its `sdk` build (Breath of the Wild's) has the SDK's address-region node area cut (`sdk_patches.py`): without it the resource heaps ran out of alias region at the first descent from the Great Sky Island and the game played its memory-shortage Blood Moon, which put Link back at the start (its log shows `MSF`). Monster Hunter Rise is Capcom's RE Engine: files in KPKA PAK archives (`shader_split.py` `kpka_jobs`, shaders in SDF files), the base game an XCI, its GPU heap a 64 GB sparse virtual pool (`hle_nvn.c` `vpool_*`, a 4 KB page table the replay reads textures through); it runs at 60 fps (30 on the Switch). Super Mario Galaxy is the Wii game on NERD's Hagi layer (PowerPC code recompiled into `main`), renders Line truncated
|
||||
- Each game's constants and shader tables form a game profile in `dm/games/<id>/`, where `<id>` is the base game's title ID: `game.tsv` plus the tables (and `patches.py` when the rewriter has patches for its builds) and `dmsh_chain.h`. Mario's profile is `010015100B514000`. Tools pick the profile with `DM_GAME`; `make -C dm/host` and `build_ipa.sh` pick it with `GAME`. There is no default game: set it every time (the tools stop and list the profiles otherwise). Put per-game values in the profile, not in code ([dm/games/README.md](dm/games/README.md)).
|
||||
- Each game's constants and shader tables form a game profile in `dm/games/<id>/`, where `<id>` is the base game's title ID: `game.tsv` plus the tables (and `patches.py` when the rewriter has patches for its builds). Mario's profile is `010015100B514000`. Tools pick the profile with `DM_GAME`; `make -C dm/host` and `build_ipa.sh` pick it with `GAME`. There is no default game: set it every time (the tools stop and list the profiles otherwise). Put per-game values in the profile, not in code ([dm/games/README.md](dm/games/README.md)).
|
||||
- Prefer general optimizations and fixes to game-specific rules. Performance work (frame pacing, power, caches, load times) counts here as much as correctness. First look for a change that holds for every game: Horizon / NVN / Vulkan semantics, or what the binary, its SDK version, its imports or its files show. Use a game-specific rule (a profile key, a rewriter cut, a build-ID gate, a per-game branch in the host) only when it is absolutely necessary, that is, when no general rule can do it.
|
||||
- Per-build knowledge is keyed by the NSO build ID: rewriter cuts, observation gates and frame-step stubs in the profile's `patches.py` (a game's own modules) or `dm/tools/sdk_patches.py` (Nintendo's `sdk` modules, which several games ship), which `dm/tools/patch_nso.py` (the general rewriter) loads, the profile's `build` rows (the builds the game was brought up on, and their display versions; the NSP script packs any other version from a derived profile, without those builds' fixes), and host behaviour that uses fixed module addresses checks `module_t.build_id` (from `fixups.tsv`; `dm_main_build()`). When a new game build appears, carry addresses over with `dm/tools/nso_fnmap.py` and check them by disassembly; never let fixed addresses run on a build they were not found in.
|
||||
- To analyse or port **another** Switch title, first run `scripts/build_ipa_from_nsp.py` on its NSP folder. With no matching profile, it writes a survey to `<NSP folder>/<title ID>_survey/survey.md`, then a provisional profile guessed from the game files (`dm/games/profile_guess.py`; `<NSP folder>/<title ID>_build/profile/<title ID>/`, found by the tools through `DM_GAMES_DIR`), with a short heap check on the Mac when the game has virtual address memory, and a best-effort IPA that may not boot. Then follow [docs/agents](docs/agents/README.md) (extraction, binary survey, host bring-up failure catalogue, graphics specifics, debugging recipes) and add a profile.
|
||||
- Scripts the user runs live in `scripts/` at the repository root (now `build_ipa_from_nsp.py`). Each one runs both in Spyder (F5, configured by the settings block at its top) and from a terminal (command-line options override those settings; non-zero exit on error). Tools that only agents or other scripts call stay under `dm/`.
|
||||
- Game files, keys and anything extracted from them live in the game's NSP folder, never in this repository and never in the home folder (it is nearly full): the work folder `<NSP folder>/<id>-work` for extraction, runs and analysis, beside the NSP script's `<id>_build/`. Tool venvs and other large scratch data go on that drive too. Do not print key values.
|
||||
- Language: everything is in English: `README.md`, docs, code comments, log messages, script output and other files. Red log lines (something unimplemented or wrong) start with `[RED] `. Text that must match a game's own Chinese data is written as escapes (`"\xE8\xB5\xB7"`), never as characters.
|
||||
- After changing anything on the shader translation chain, run the chain procedure for each affected profile. The chain covers `dm/gfx/tools/dmsh_*`, `Program.cs`, the profile's `shader_map.tsv` / `shader_facts.tsv` / `shader_pairs.tsv`, and its `shader_*` keys. The procedure:
|
||||
1. Run `dmsh_chain.py --header`, which writes the profile's `dmsh_chain.h`.
|
||||
2. Rebuild the shaders (Mac and, with `IOSOUT`, iOS).
|
||||
3. Regenerate the profile's `c1_full.tsv` (`c1_guard.py`) and `cb_extent.tsv` (`cb_extent.py`) for Mac runs. They are made from each build's Mac outputs and git ignores them; the NSP script makes its own.
|
||||
4. Rebuild the host.
|
||||
- The shader translation chain has one version for every game, `DMSH_CHAIN` in `dm/gfx/nvn/dmsh_chain.h`, which the host is compiled with and loads only libraries of. It covers the chain's tools: `dm/gfx/tools/dmsh_*` (`dmsh_build.py`, `dmsh_chain.py`, `dmsh_fetch.py`, `dmsh_vin.py`, `dmsh_geometry.py`, `dmsh_facts.py`), `Program.cs`, `ios_metal.py` and `dmsh_chain.lock`. A profile's own inputs (`shader_map.tsv`, `shader_facts.tsv`, `shader_pairs.tsv`, `bf_want.tsv`, its `shader_*` / `nvn_*` keys) are not part of it: they are in each shader's cache key, and their digest stamps the libraries (`b-<V>/inputs.tsv`). The procedure:
|
||||
1. After changing a chain tool, run `dmsh_chain.py --header` once and commit `dm/gfx/nvn/dmsh_chain.h` with the change (`dm/tools/tests/test_nsp_chain.py` fails on a stale one).
|
||||
2. After that, or after changing a profile's inputs, rebuild that profile's shaders: a full `dmsh_build.py` run (Mac and, with `IOSOUT`, iOS) stamps its libraries and writes the profile's `c1_full.tsv` and `cb_extent.tsv` from its Mac outputs (git ignores them; the host takes them only beside the stamp they name). Give each game its own output directory.
|
||||
3. Rebuild the host.
|
||||
|
||||
The host only loads libraries of its own chain version. The translator is `dm/gfx/tools/dmsh_xlate_bin`, which matches `dmsh_chain.lock`; a fresh `dotnet publish` does not.
|
||||
The translator is `dm/gfx/tools/dmsh_xlate_bin`, which matches `dmsh_chain.lock`; a fresh `dotnet publish` does not.
|
||||
- iOS builds are unsigned IPAs that the user sideloads and tests; diagnostics come back as the app's `Documents/log.txt`. Do not sign or install builds on the user's devices.
|
||||
- Commit only when asked, with `git -c user.name=seedds -c user.email=seedds@gmail.com commit`.
|
||||
Reference in new issue
Block a user