okooooooo/nxnative-easy
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
86d9ba123967cfe520e556e882dcd4e0ef7ba272
nxnative-easy/docs/agents
T
History
seedds 86d9ba1239 One translation chain version for every game: DMSH_CHAIN in dm/gfx/nvn/dmsh_chain.h replaces the 47 dm/games/<id>/dmsh_chain.h, which every chain change rewrote and some were left behind (GitHub issue #78). The version hashed each profile's shader tables and nvn_* / shader_* keys too, once the only thing that kept the host off a library built from older tables; each shader's cache key (dmsh_build.py shader_key) has covered them since, so the version now covers only the chain's tools (dmsh_chain.py, dmsh_build.py, dmsh_fetch.py, dmsh_vin.py, dmsh_geometry.py, dmsh_facts.py, the translator's source, ios_metal.py, the lock) and the options, and needs no DM_GAME. What the per-game part still did is done by the library itself: a full dmsh_build.py run stamps b-<V>/inputs.tsv (Mac and iOS) with the game and the digest of the profile's inputs (dmsh_chain.py inputs / --inputs: the four shader tables, bf_want.tsv, which the variants use but the version never counted, and the keys), starts from no shader outputs (a partial one from none of the keys it builds), so a key that left the map, a shader that no longer builds or a variant no longer made keeps no old library, refuses a directory stamped for another game, and writes the profile's c1_full.tsv and cb_extent.tsv from its Mac outputs; a partial build keeps the stamp only while it names the current inputs. c1_guard.py / cb_extent.py head their table with the library's chain version and stamp (table_head), and the host takes a table only beside that stamp, logs the stamp and reports one of another game; build_ipa.sh reads the one header and stops when the iOS libraries' stamp is not this profile's current inputs or c1_full.tsv names another. NSP script: chain_header writes the one header when stale, before any host is built; provisional and derived profiles no longer write one; chain_tables makes a table again when it does not name the reused library's stamp (False without one). Makefiles take the header beside nvn_mtl.m; Xcode projects prepared earlier point at a profile header that is gone and are told to prepare again. Tests: test_nsp_chain.py (the committed header is current, stale and missing ones are written, tables against the stamp). Checked on The Great Ace Attorney Chronicles: a full build gave 460 of 460 shaders and both tables (0 and 409 entries, as before), the Mac host booted with the stamp and both tables past frame 900; one added facts row rebuilt one shader under the same chain version with a new digest and new tables, and the host refused the old tables beside the new stamp; a full build removed planted outputs, another game's build into that directory stopped, a partial build after an input change dropped the stamp; build_ipa.sh stopped on a stale stamp, a missing one and a stale or old-form c1_full.tsv; three NSP packs built (no table lines: dmsh_build made them), reused everything, and remade deleted tables from the reused build. Graphics unit tests give the same results as before (on this Mac t_refl needs ~/dm-work's libraries and several GPU tests fail on the previous commit too). AGENTS.md's chain procedure (header once after a tool change; a full shader build makes the tables), dm/games/README.md, dm/gfx/README.md and docs/development.md say so
2026-10-23 23:37:03 +08:00
..
01-extract.md
patch_nso.py split into the general rewriter and per-build patch files (GitHub issue #63): every game's bring-up edited the one file, so PRs for unrelated games conflicted. dm/tools/patch_nso.py (1,711 → about 1,100 lines) keeps the general passes and the driver; the instruction encoders move unchanged to dm/tools/nso_asm.py; a game's own patches (CUTS, OBS and the frame-step stubs, now FRAME_STEP = {build ID: stub(ctx)}) move verbatim with their comments to dm/games/<id>/patches.py (16 profiles: Tears of the Kingdom, Breath of the Wild, Age of Calamity and Paper Mario: The Thousand-Year Door with their stubs), and the cuts of Nintendo's sdk module builds, 6 of the 15 shipped by 2 to 4 games, to dm/tools/sdk_patches.py. The rewriter loads every patch file, whatever the game, and applies those of a module's build ID at the points it did before (cuts before the scan, gates after the rewrite, stubs after the gates); a build ID claimed twice stops it, and its command line needs no DM_GAME. build_src.sha256 lists the rewriter's files and every patch file (the same for every module). NSP script: the rewrite step's reuse key counts nso_asm.py and only the patch files that name one of the game's builds (patch_files), profile_files leaves patches.py out (only the rewrite step reads it), and a derived profile gets none. dm/tools/tests/test_patches.py: no build claimed twice, the entries' forms, a game's file names only its own profile's non-sdk builds, sdk_patches.py only sdk builds some profile ships, and the reuse key's patch files. Checked byte-identical: every extracted module on the game drive (131 builds) rewritten with the previous and the new rewriter gives the same outputs but build_src.sha256, 28 of the 30 patched builds among them (all four frame-step mains), and 7 fail alike (rtld, and New Super Mario Bros. U Deluxe's, which no profile rewrites); the other two (Super Mario Bros. Wonder's mains and Absolum's, no game files here) by their CUTS / OBS data, equal to the previous tables. AGENTS.md, docs, dm/games/README.md, the profiles' comments and the host comments that named a game's cut in patch_nso.py now name its patches.py or sdk_patches.py
2026-10-23 23:36:21 +08:00
02-survey.md
Digimon Story Cyber Sleuth: Complete Edition plays on the Mac: profile 010014E00DB56000 (update 1.0.3, SDK 6.4.0), an OpenGL layer on Metal (dm/gfx/gl), its GLSL translated offline (gl_shaders.py), MVGL archives (mvgl.py)
2026-10-22 20:43:37 +08:00
03-bringup.md
patch_nso.py split into the general rewriter and per-build patch files (GitHub issue #63): every game's bring-up edited the one file, so PRs for unrelated games conflicted. dm/tools/patch_nso.py (1,711 → about 1,100 lines) keeps the general passes and the driver; the instruction encoders move unchanged to dm/tools/nso_asm.py; a game's own patches (CUTS, OBS and the frame-step stubs, now FRAME_STEP = {build ID: stub(ctx)}) move verbatim with their comments to dm/games/<id>/patches.py (16 profiles: Tears of the Kingdom, Breath of the Wild, Age of Calamity and Paper Mario: The Thousand-Year Door with their stubs), and the cuts of Nintendo's sdk module builds, 6 of the 15 shipped by 2 to 4 games, to dm/tools/sdk_patches.py. The rewriter loads every patch file, whatever the game, and applies those of a module's build ID at the points it did before (cuts before the scan, gates after the rewrite, stubs after the gates); a build ID claimed twice stops it, and its command line needs no DM_GAME. build_src.sha256 lists the rewriter's files and every patch file (the same for every module). NSP script: the rewrite step's reuse key counts nso_asm.py and only the patch files that name one of the game's builds (patch_files), profile_files leaves patches.py out (only the rewrite step reads it), and a derived profile gets none. dm/tools/tests/test_patches.py: no build claimed twice, the entries' forms, a game's file names only its own profile's non-sdk builds, sdk_patches.py only sdk builds some profile ships, and the reuse key's patch files. Checked byte-identical: every extracted module on the game drive (131 builds) rewritten with the previous and the new rewriter gives the same outputs but build_src.sha256, 28 of the 30 patched builds among them (all four frame-step mains), and 7 fail alike (rtld, and New Super Mario Bros. U Deluxe's, which no profile rewrites); the other two (Super Mario Bros. Wonder's mains and Absolum's, no game files here) by their CUTS / OBS data, equal to the previous tables. AGENTS.md, docs, dm/games/README.md, the profiles' comments and the host comments that named a game's cut in patch_nso.py now name its patches.py or sdk_patches.py
2026-10-23 23:36:21 +08:00
04-graphics.md
One translation chain version for every game: DMSH_CHAIN in dm/gfx/nvn/dmsh_chain.h replaces the 47 dm/games/<id>/dmsh_chain.h, which every chain change rewrote and some were left behind (GitHub issue #78). The version hashed each profile's shader tables and nvn_* / shader_* keys too, once the only thing that kept the host off a library built from older tables; each shader's cache key (dmsh_build.py shader_key) has covered them since, so the version now covers only the chain's tools (dmsh_chain.py, dmsh_build.py, dmsh_fetch.py, dmsh_vin.py, dmsh_geometry.py, dmsh_facts.py, the translator's source, ios_metal.py, the lock) and the options, and needs no DM_GAME. What the per-game part still did is done by the library itself: a full dmsh_build.py run stamps b-<V>/inputs.tsv (Mac and iOS) with the game and the digest of the profile's inputs (dmsh_chain.py inputs / --inputs: the four shader tables, bf_want.tsv, which the variants use but the version never counted, and the keys), starts from no shader outputs (a partial one from none of the keys it builds), so a key that left the map, a shader that no longer builds or a variant no longer made keeps no old library, refuses a directory stamped for another game, and writes the profile's c1_full.tsv and cb_extent.tsv from its Mac outputs; a partial build keeps the stamp only while it names the current inputs. c1_guard.py / cb_extent.py head their table with the library's chain version and stamp (table_head), and the host takes a table only beside that stamp, logs the stamp and reports one of another game; build_ipa.sh reads the one header and stops when the iOS libraries' stamp is not this profile's current inputs or c1_full.tsv names another. NSP script: chain_header writes the one header when stale, before any host is built; provisional and derived profiles no longer write one; chain_tables makes a table again when it does not name the reused library's stamp (False without one). Makefiles take the header beside nvn_mtl.m; Xcode projects prepared earlier point at a profile header that is gone and are told to prepare again. Tests: test_nsp_chain.py (the committed header is current, stale and missing ones are written, tables against the stamp). Checked on The Great Ace Attorney Chronicles: a full build gave 460 of 460 shaders and both tables (0 and 409 entries, as before), the Mac host booted with the stamp and both tables past frame 900; one added facts row rebuilt one shader under the same chain version with a new digest and new tables, and the host refused the old tables beside the new stamp; a full build removed planted outputs, another game's build into that directory stopped, a partial build after an input change dropped the stamp; build_ipa.sh stopped on a stale stamp, a missing one and a stale or old-form c1_full.tsv; three NSP packs built (no table lines: dmsh_build made them), reused everything, and remade deleted tables from the reused build. Graphics unit tests give the same results as before (on this Mac t_refl needs ~/dm-work's libraries and several GPU tests fail on the previous commit too). AGENTS.md's chain procedure (header once after a tool change; a full shader build makes the tables), dm/games/README.md, dm/gfx/README.md and docs/development.md say so
2026-10-23 23:37:03 +08:00
05-debugging.md
patch_nso.py split into the general rewriter and per-build patch files (GitHub issue #63): every game's bring-up edited the one file, so PRs for unrelated games conflicted. dm/tools/patch_nso.py (1,711 → about 1,100 lines) keeps the general passes and the driver; the instruction encoders move unchanged to dm/tools/nso_asm.py; a game's own patches (CUTS, OBS and the frame-step stubs, now FRAME_STEP = {build ID: stub(ctx)}) move verbatim with their comments to dm/games/<id>/patches.py (16 profiles: Tears of the Kingdom, Breath of the Wild, Age of Calamity and Paper Mario: The Thousand-Year Door with their stubs), and the cuts of Nintendo's sdk module builds, 6 of the 15 shipped by 2 to 4 games, to dm/tools/sdk_patches.py. The rewriter loads every patch file, whatever the game, and applies those of a module's build ID at the points it did before (cuts before the scan, gates after the rewrite, stubs after the gates); a build ID claimed twice stops it, and its command line needs no DM_GAME. build_src.sha256 lists the rewriter's files and every patch file (the same for every module). NSP script: the rewrite step's reuse key counts nso_asm.py and only the patch files that name one of the game's builds (patch_files), profile_files leaves patches.py out (only the rewrite step reads it), and a derived profile gets none. dm/tools/tests/test_patches.py: no build claimed twice, the entries' forms, a game's file names only its own profile's non-sdk builds, sdk_patches.py only sdk builds some profile ships, and the reuse key's patch files. Checked byte-identical: every extracted module on the game drive (131 builds) rewritten with the previous and the new rewriter gives the same outputs but build_src.sha256, 28 of the 30 patched builds among them (all four frame-step mains), and 7 fail alike (rtld, and New Super Mario Bros. U Deluxe's, which no profile rewrites); the other two (Super Mario Bros. Wonder's mains and Absolum's, no game files here) by their CUTS / OBS data, equal to the previous tables. AGENTS.md, docs, dm/games/README.md, the profiles' comments and the host comments that named a game's cut in patch_nso.py now name its patches.py or sdk_patches.py
2026-10-23 23:36:21 +08:00
README.md
Remove the D2R code and the legacy frame step
2026-10-21 02:09:49 +08:00

README.md

Porting a Switch title: agent guide

These pages are for an agent asked to analyse a Switch NSP and port it with the native-execution approach of this repository and d2r-mobile: the game's own aarch64 code runs natively on Apple hardware, a rewriter patches only the instructions Apple's kernel cannot run, the host replaces Horizon system services, and NVN is reimplemented on Metal.

Everything here was learned on Diablo II: Resurrected (SDK ~12, NVN 53, the original target of d2r-mobile) and Super Mario Bros. Wonder (SDK 16.2.0, NVN 55, this repository; its update 1.2.1 uses SDK 16.2.12), then extended by Fire Emblem Engage (Unity, SDK 12.3.7), Metroid Dread (MercurySteam's engine, SDK 11.4.3) 13 Sentinels: Aegis Rim (Vanillaware's engine, SDK 11.4.3, the same sdk and movie module as Dread) Super Mario 3D World + Bowser's Fury (Nintendo EPD's sead / agl engine, SDK 10.4.0, NVN API 53) The Legend of Zelda: Breath of the Wild (EPD's sead / ksys engine, SDK 20.5.22, DT_RELR relocations, update 1.9.0), The Legend of Zelda: Link's Awakening (Grezzo's engine on NintendoWare, SDK 8.2.0, lower-left window origin) Hyrule Warriors: Age of Calamity (Koei Tecmo's KTGL, SDK 9.6.0, RDB archives, shaders as GLSLC outputs) Super Mario Galaxy (a Wii game on NERD's Hagi layer: recompiled PowerPC code, SDK 18.3.5, Vulkan with SPIR-V generated at run time, a 2.9 GB bss), Ace Attorney Investigations Collection (Unity 2022.3 IL2CPP, SDK 17.5.3: the C# file API through the SDK's C library), and the bring-up of Octopath Traveler 0 (Unreal Engine 5, SDK 19.3.13: an AES-encrypted pak, Oodle-compressed shaders that the game's own code decodes on the Mac, its allocator replaced by the host's). Where titles differed, each is given; the next title will differ again, so treat every number below as "measure it, don't assume it".

Page Covers
01-extract.md Keys, NSP → NCA → ExeFS / RomFS, update NSPs, the workspace layout
02-survey.md Static survey of a new title before touching the host: SDK version, imports vs. host hooks, SVCs, instructions to rewrite, NVN surface, data containers, Ghidra / REA setup
03-bringup.md Getting the host to boot the game: the run loop, how to read the log, and a catalogue of every failure pattern met so far with its root cause and fix
04-graphics.md NVN on Metal for a new title: object sizes, shader containers and keys, binding layouts, bindless textures, depth mode, index sizes, swizzle, transient render targets, the diagnostic switches
05-debugging.md Recipes: crash reports, lldb scripts, Ghidra headless decompilation, objdump, memory growth, render-pass and texture dumps, on-device diagnostics, offline pipeline link checks

Ground rules

  • Bring your own files. Keys, NSPs and anything extracted from them stay in the game's work folder ($W = <NSP folder>/<title>-work, next to the NSPs on the user's external drive; the home folder is nearly full), never in the repository, never in a commit, never printed. Print key names and lengths only.
  • Measure, then change. Each fix in this repository started from a concrete observation (a crash address and its decompilation, a census over all shaders, a memory profile). Write the evidence into the code comment next to the fix, the way the existing code does; the next title will need it.
  • One repository, one profile per title. nxnative (first called smbw-mobile) started as a copy of d2r-mobile's generic parts. Title- or SDK-specific constants live in the game profile dm/games/<id>/ (dm/games/README.md). That covers program ID, known builds, modules, address-space partition, NVN driver tables, the shader tables and the app identity. The code reads the profile through game_profile.h (host) or dm/games/game.py (tools). Behaviour that fits only one title's code is gated by that title's build ID (dm_main_build()), never left on for every title. Some things are still built into the translator: the texture / sampler handle bases and the depth mode (see 04). dmsh_build.py refuses a profile whose bases differ from the translator's.
  • Write code comments in English, and start log lines that mark something unimplemented or wrong with [RED] .
  • Test where it runs. The Mac host (dm/host/dmhost) is the development loop; the iOS app reuses the same host and graphics code. Never claim something works on iOS without running it there.

Checklist for a new title

  1. Run scripts/build_ipa_from_nsp.py on the title's NSP folder. With no profile for its title ID, it writes <NSP folder>/<title ID>_survey/survey.md:

    • modules and build IDs, SDK version and components, engine and graphics API markers;
    • RomFS containers;
    • the rewrite-site census and imports vs. host hooks.

    It extracts only the code modules plus a RomFS list for the survey. Unless given --survey-only, it then writes a provisional profile into <NSP folder>/<title ID>_build/profile/<title ID>/, extracts the game, boots it on the Mac until its heap request when it has virtual address memory (the heap check) and packs a best-effort IPA. That profile is the starting point for step 3: the static keys follow rules checked against every profile, the rest are guesses or defaults (dm/games/README.md); the run-time keys (render_cond, tex_flush_upload, core_preempt, …) are the bring-up's to find. For Mac runs, export DM_GAMES_DIR=<NSP folder>/<title ID>_build/profile until the profile moves into dm/games/.

  2. Extract base (and update, if in scope) — 01. Finish the survey — 02. Write down: SDK version, NVN API version, module list, imports the host lacks, SVCs, instruction census, NVN loader pattern, shader containers.

  3. Add a game profile dm/games/<title ID>/game.tsv, named after the base game's title ID (start from 010015100B514000/game.tsv: identity, build rows, modules; keep the memory / NVN values until they are measured) and build the host for it with make -C dm/host GAME=<id>.

  4. Run the rewriter and check its counts against insn_census.py.

  5. Boot on the Mac and iterate on the failure catalogue in 03 until the game reaches its frame loop.

  6. Extract and translate the shaders, find the binding layout by census, and get the first frame — 04.

  7. Check memory over a minute of running (footprint, the host's memory report line); fix growth before moving on.

  8. Input, audio, then the iOS bundle (d2r-mobile's dm/ios, RomFS inside the app).

  9. Test on a phone: an unsigned IPA the user sideloads, switches through env.txt, results through Documents/log.txt (05). Several bugs only showed there (the command-buffer byte and the audio buffer size in 03, pipeline link failures in later levels in 04).

  10. For a game update: extract it over the base game, diff the imports, carry the per-build addresses over, and rebuild the shader tables for both versions (01).

Reference in new issue View Git Blame Copy Permalink