Docs: iPhone fixes, update 1.2.1, prewarm lists; drop host README
- README: status for v0 / 1.2.1 on the iPhone, update 1.2.1 build steps, diagnostic env files, finishing the IPA by hand when Xcode uses a custom build location, app data folders after a reinstall, prewarm lists (pso_user.tsv / pso_manifest.tsv). - AGENTS.md: per-build knowledge keyed by build ID, shader_pairs.tsv on the chain, unsigned IPAs only. - docs/agents: update extraction and offset porting (01), the command-buffer byte, command memory, iOS audio, update imports and build-gated D2R code (03), vertex output padding and iOS math mode (04), on-device diagnostics and the offline link check (05). - dm/gfx README / nvn_metal_mapping (en + zh-CN): pairs table, several game versions, prewarm lists, DM_PSO_USER, command buffer notes; corrected shader key, build script and 8-bit index lines. - dm/host/README(.zh-CN).md removed: an index of d2r-mobile's D2R features; most files it listed no longer exist.
This commit is contained in:
seedds committed
1 parent
8911232933
commit
73b1722248
13 files changed
+105
-314
No files matched your search
@@ -1,8 +1,10 @@
|
||||
# Notes for agents
|
||||
|
||||
- This repository ports Super Mario Bros. Wonder (Switch, base game v0) to Apple platforms by running the game's own code natively. Start with [README.md](README.md) for status and build steps.
|
||||
- This repository ports Super Mario Bros. Wonder (Switch, base game v0 and update 1.2.1) to Apple platforms by running the game's own code natively. Start with [README.md](README.md) for status and build steps.
|
||||
- Per-build knowledge is keyed by the NSO build ID: rewriter cuts in `dm/tools/patch_nso.py`, and host behaviour that uses fixed module addresses checks `module_t.build_id` (from `fixups.tsv`). When a new game build appears, carry addresses over with `dm/tools/nso_fnmap.py` and check them by disassembly; never let fixed addresses run on a build they were not found in.
|
||||
- To analyse or port **another** Switch title, follow [docs/agents](docs/agents/README.md): extraction, binary survey, host bring-up failure catalogue, graphics specifics, debugging recipes.
|
||||
- Game files, keys and anything extracted from them live in the user's workspace (`~/smbw-work`), never in this repository. Do not print key values.
|
||||
- Code comments are bilingual (Chinese line, then English line); red log lines start with `【红】`. Keep both conventions.
|
||||
- After changing anything on the shader translation chain (`dm/gfx/tools/dmsh_*`, `Program.cs`, `shader_map.tsv`, `shader_facts.tsv`), run `dmsh_chain.py --header`, rebuild the shaders, regenerate `c1_full.tsv`, and rebuild the host: the host only loads libraries of its own chain version.
|
||||
- After changing anything on the shader translation chain (`dm/gfx/tools/dmsh_*`, `Program.cs`, `shader_map.tsv`, `shader_facts.tsv`, `shader_pairs.tsv`), run `dmsh_chain.py --header`, rebuild the shaders (Mac and, with `IOSOUT`, iOS), regenerate `c1_full.tsv`, and rebuild the host: the host only loads libraries of its own chain version.
|
||||
- iOS builds are unsigned IPAs that the user sideloads and tests; diagnostics come back as the app's `Documents/log.txt`. Do not sign or install builds on the user's devices.
|
||||
- Commit only when asked, with `git -c user.name=seedds -c user.email=ext6@xcra.ai commit`.
|
||||
@@ -1,67 +1,21 @@
|
||||
# smbw-mobile
|
||||
|
||||
Work in progress: an iOS native-execution port of **Super Mario Bros. Wonder** (Switch, base game v0), built from the generic parts of [d2r-mobile](../d2r-mobile) (MIT). Bring your own game files; nothing from the game is in this repository.
|
||||
Work in progress: an iOS native-execution port of **Super Mario Bros. Wonder** (Switch, base game v0 and update 1.2.1), built from the generic parts of [d2r-mobile](../d2r-mobile) (MIT). Bring your own game files; nothing from the game is in this repository.
|
||||
|
||||
Same approach as d2r-mobile: the game's own aarch64 code runs natively. A rewriter changes only the instructions that cannot run under Apple's kernel (x18, thread pointers, `svc`, timers). The host replaces Switch system services, and the NVN graphics API is reimplemented on Metal.
|
||||
|
||||
## Status (2026-10-13)
|
||||
## Status (2026-10-14)
|
||||
|
||||
| Step | State |
|
||||
|---|---|
|
||||
| Rewriter (`dm/tools/patch_nso.py`, `DM_RULESET=native`) | Runs unchanged on Mario `main` / `sdk` |
|
||||
| Mac host | **Plays through the menus (player count, character select) into the opening course**: the course, effects, HUD, menu and dialogue text render in Chinese or English. Title screen at 60 fps, memory flat (about 900 MB physical footprint on the title screen) |
|
||||
| Shaders | All 12,811 shaders the game ships (bfsha archives, particle effects, engine `agl` packages, UI layouts, `main`) are extracted by `shader_split.py` and translated offline; 12,809 compile. 2 `agl` post-effect compute shaders make SPIRV-Cross throw |
|
||||
| Rewriter (`dm/tools/patch_nso.py`, `DM_RULESET=native`) | Runs unchanged on Mario `main` / `sdk`, v0 and 1.2.1 |
|
||||
| Mac host | **Plays through the menus (player count, character select) into the opening course**: the course, effects and HUD render. Title screen at 60 fps, memory flat (about 900 MB physical footprint on the title screen). Menu and dialogue text currently render blank on the Mac (they are fine on the iPhone); not investigated yet |
|
||||
| Shaders | All shaders the game ships (bfsha archives, particle effects, engine `agl` packages, UI layouts, `main`) are extracted by `shader_split.py` and translated offline: 12,811 in v0, 16,268 across v0 + 1.2.1 (the update recompiled 3,451 pixel shaders and added 6); 16,266 compile. 2 `agl` post-effect compute shaders make SPIRV-Cross throw. Vertex shaders get the outputs their paired pixel shaders read (all 14,603 pairs link) |
|
||||
| Input | Scripted input `DM_KEYS="12:A,14:A,20:A,26:A,32:A"` (seconds:button) reaches the opening course; `DM_WINDOW=1` for a window with keyboard input |
|
||||
| Current frontier | **The opening course is playable with sound** (run, jump, collect coins; characters, NPCs, sky, text, button icons all render; music and effects play). Only the opening course has been played |
|
||||
| Known gaps | Partial / mid-pass clears; sparse virtual-pool mappings; conditional render; `DrawElementsIndirect`; TBDR optimizations are off by default (they don't understand bindless textures yet); resource heaps get +16 KB slack because one model's setup came up 0x70 bytes short of the size table (cause unknown) |
|
||||
| Current frontier | **Playable on the iPhone with sound through several levels**, on v0 and on 1.2.1 (run, jump, collect coins; characters, NPCs, sky, text, button icons render; music and effects play). The Mac runs are scripted and reach the opening course |
|
||||
| Known gaps | Partial / mid-pass clears; sparse virtual-pool mappings; conditional render; `DrawElementsIndirect`; TBDR optimizations are off by default (they don't understand bindless textures yet); resource heaps get +16 KB slack because one model's setup came up 0x70 bytes short of the size table (cause unknown); no amiibo (no NFC reader); the bundled prewarm list is empty, so the first run after an install stutters while pipelines compile (see "Prewarm lists" below) |
|
||||
| Audio | The game's own SDK renderer runs in-process in its CPU / manual-execution mode (`hle_audren.c`); the host renders one 5 ms frame per buffer the output hands back and sends the device-sink mix (5.1 folded to stereo) through Core Audio (`DM_AUDIO=1` on Mac, default on iOS). About 0.1 ms of CPU per frame on an M-series Mac |
|
||||
| iOS bundle | `dm/ios/tools/build_ipa.sh` builds one unsigned IPA (iPhone + iPad): game code embedded in the executable, iOS shader libraries, RomFS (3.5 GB) inside the app, standard-controller virtual gamepad and physical controllers. Compiles and links; **not yet run on a device** (this Mac has no signing identity, and the Simulator SDK has no Metal 4) |
|
||||
|
||||
## What changed relative to d2r-mobile
|
||||
|
||||
- Removed: all D2R gameplay / UI modules (`dm_*.c`), D2R diagnostics (`diag.c`), Blizzard TACT/CASC, Granny, movie, auto-targeting, and the archive validator. `host_stubs.c` holds the leftover hook points as no-ops.
|
||||
- `hle_hid.c`: rewritten as a generic standard-controller layer covering all Npad styles plus six-axis, vibration, mouse and keyboard stubs.
|
||||
- `hle_sdk16.c` (new): newer-SDK `oe` / `time` / shared font (`pl`) / vsync event / `diag` hooks.
|
||||
- `hle_offline.c` (new): offline stubs for play reports, local wireless, sockets (C++ and C APIs), SSL, nifm, nsd, friends, ngc and err.
|
||||
- `hle_fs.c`, `hle_account.c`: newer-SDK overloads (`MountRom(name)`, `ReadFile` variants, `GetDirectoryEntryCount`, user enumeration, ...).
|
||||
- `loader.c`: forced hooks that a game never references are now only logged (`DM_FORCE_STRICT=1` restores the abort).
|
||||
- `patch_nso.py`, two fixes:
|
||||
- **Exclusive-window bug, inherited from d2r-mobile.** When an ldxr/stxr window contained a branch out (`ldxr; cmp; b.ne out; stxr; cbnz`), a successful stxr fell into the branch-out exit. Every enqueue in Mario's job system retried forever and the job system hung. The fall-through exit is now emitted first. D2R's windows had no branch out, so D2R never hit it.
|
||||
- `mrs xN, ctr_el0` is rewritten to a constant (macOS traps it; SDK 16's `FlushDataCache` reads it).
|
||||
- `hle_thread.c`: SetThreadCoreMask moves the ideal core into the new mask, as Horizon does, so `GetCurrentProcessorNumber` is right for pinned threads.
|
||||
- NVN:
|
||||
- API version reported as 55.15 (`DM_NVN_API` overrides it).
|
||||
- 13 newer names were added to the table.
|
||||
- `tex_t` is packed to 0x80 bytes: Mario allocates `NVNtexture` as 0x80, and the old 0x88 record overran it and corrupted the heap.
|
||||
- `view_t` is packed to 16 bytes: SDK 16's `NVNtextureView` is 16 bytes and Mario keeps it in a 16-byte stack slot; the old 28-byte record overwrote the saved frame pointer and return address.
|
||||
- Shader key = sha256(control section + code area): Mario has many shaders with an identical control section and different code.
|
||||
- Bindless textures: 38 % of Mario's shaders read texture handles out of their own uniform buffers. New binding kinds `tex_ubo` / `samp_ubo` (`NVN_RB_TEXU` / `NVN_RB_SAMPU`) read the handle at replay time.
|
||||
- Bound texture handles live at `0x20 + 8 × slot` in NVN 55 (D2R's driver: `0x168`); images stay at `0x120 + 8 × slot`.
|
||||
- 8-bit index buffers are widened to 16-bit at replay (Metal has no 8-bit indices); Mario's full-screen passes and its final copy to the window use them.
|
||||
- Texture swizzle (`TextureBuilderSetSwizzle` / `TextureViewSetSwizzle`) is applied through Metal texture views (the font atlas is `(1, 1, 1, R)`).
|
||||
- Non-packaged textures are shared by storage (pool + offset + description): Mario rebuilds its transient render targets on new objects every frame and never finalizes them, which leaked about 13 MB of GPU memory per frame.
|
||||
- TBDR optimizations are off by default (`DM_NVN_TBDR=1` turns them on) until they recognize bindless readers.
|
||||
- Linear textures in CPU-visible pools are created directly on the pool buffer (zero copy). Mario's glyph cache (2048×8192 R8) is filled by the CPU; as a private GPU texture all menu and dialogue text was blank.
|
||||
- Host command-memory segments (lent when a command buffer outgrows the memory the game gave it) grow on demand instead of a fixed 64 × 1 MB.
|
||||
- `RGB10A2SN` vertex format; instanced `DrawArrays`.
|
||||
- Vertex formats follow the shader input type (integer inputs get integer formats of the matching sign).
|
||||
- Copies, fills and inline uniform updates batched in one Metal 4 encoder get a barrier when they touch a range an earlier one wrote (Metal 4 does not order them). Mario writes the same uniform slot twice in a row; the reversed order made the fountain's GPU wave simulation diverge and draw tall panels over the screen.
|
||||
- Scissored zero clears outside a pass are copied from a zero buffer into the scissor rectangle (they were skipped).
|
||||
- New non-packaged colour textures are zero-filled (Metal leaves them undefined; Switch memory pools start as zero), and inline uniform updates are copied out of command memory when encoded.
|
||||
- Shader chain:
|
||||
- `dm/gfx/tools/shader_split.py` (new) scans RomFS and `main` for BNSH and BAHS (agl `sharcb`) containers and writes the package directory, the shader map and the answer table (compute thread-group size from control +0x748, shared memory from +0x754).
|
||||
- The translator uses GL depth (`-1..1`) and no depth clip control, so vertex shaders remap z' = z/2 + w/2: Mario sets `nvnDeviceSetDepthMode` to near = -w, and without the remap its full-screen copy was clipped away.
|
||||
- Vertex shaders whose inputs are only used as integers are re-translated with `uint` inputs (`int_inputs`): skinned models read bone indices as raw integer bits, and the float default made every character invisible.
|
||||
- Compute code areas (magic 0x12345679) start their instructions at +0x100 (0x30 header + zero padding); `dmsh_build.py` strips all of it. Stripping only 0x30 misaligned the translator's 32-byte instruction groups: 12 particle / post-effect compute shaders became endless loops and hung the GPU at the first gameplay scene.
|
||||
- Host:
|
||||
- Guest address space repartitioned: alias region 2.875 GB, heap 2.625 GB. Mario reserves 1.75 GB of `VirtualAddressMemory` (HeapDef) that it maps into the alias region on demand; with 1.25 GB, large layout archives got no address and were skipped.
|
||||
- Threads created with ideal core -2 run on the process default core (0); `GetCurrentProcessorNumber` never returns a negative core. Mario's font engine indexes per-core state with it.
|
||||
- Shared fonts are served with a BFTTF "already decoded" header so `ScalableFontEngineHelper::Decode` leaves them alone. Type 5 (Nintendo extension) gets a generated icon font (`dm/tools/make_ext_font.py`: A/B/X/Y, L/R/ZL/ZR/SL/SR, +/−, D-pad, stick arrows at the original private-use code points); the other types share a desktop font. `DM_FONT_DIR` points at `standard.ttf` / `nintendo_ext.ttf` (the iOS app ships both).
|
||||
- Crash reports scan the stack for return addresses and dump the popped frame.
|
||||
- `DM_PROBE` (`host/probe.c`): register-relative dumps (`@xN+off`), hit filters (`?[xN+off]=val`), a `sead::ExpHeap` walker (`!xN`), thread IDs, `DM_PROBE_ALL`.
|
||||
- Audio (`hle_audren.c`): newer SDKs link the whole audio renderer into the game (`nn::audio::server` command generation and `nn::audio::dsp::CommandListProcessor`, the CPU version of the DSP). `OpenAudioRenderer` is switched to rendering device = CPU and execution mode = manual and then runs in-process; a host-started guest thread calls the game's `ExecuteAudioRendererRendering` for every buffer the output returns, the takeover of `AudioRendererImpl::ExecuteAudioRendererRendering` reads the DeviceSink inputs out of the mix buffers (DeviceSink is a no-op in the CPU processor), and an update hook renders a catch-up frame when the game sends two updates without a frame in between (otherwise newly added effects are never initialized and the light limiter dereferences null).
|
||||
- `patch_nso.py` code cuts for Mario `main` (build ID `cd6e42ae…`): resource heaps (`main+0x1e62fc`) get +16 KB.
|
||||
- `dm/tools/ghidra/Decompile.java`: headless decompile by module offset; `@off` (re)creates functions reached only through vtables and undoes Ghidra's wrong "non-returning" mark on `nn::os` imports.
|
||||
| iOS bundle | `dm/ios/tools/build_ipa.sh` builds one unsigned IPA (iPhone + iPad): game code embedded in the executable, iOS shader libraries, RomFS (3.5 GB) inside the app, standard-controller virtual gamepad and physical controllers. **Runs on an iPhone** (sideloaded): an 11-minute session through several levels at 60 fps with no crash and no audio dropouts |
|
||||
|
||||
## Build and run on Mac
|
||||
|
||||
@@ -71,7 +25,7 @@ Requires Xcode, the .NET SDK, spirv-cross and the game files extracted to `~/smb
|
||||
# 1. rewrite the code modules
|
||||
DM_RULESET=native python3 dm/tools/patch_nso.py ~/smbw-work/data/v0/ExeFS/{main,sdk} ~/smbw-work/build/patched_v0 {main,sdk}
|
||||
# 2. shaders: extract, then translate (writes <cache>/b-<chain version>/)
|
||||
<venv python with lz4> dm/gfx/tools/shader_split.py ~/smbw-work/data/v0/RomFS ~/smbw-work/data/v0/ExeFS/main ~/smbw-work/shader/pkg dm/gfx/nvn/shader_map.tsv dm/gfx/nvn/shader_facts.tsv
|
||||
<venv python with lz4> dm/gfx/tools/shader_split.py ~/smbw-work/data/v0/RomFS ~/smbw-work/data/v0/ExeFS/main ~/smbw-work/shader/pkg dm/gfx/nvn/shader_map.tsv dm/gfx/nvn/shader_facts.tsv dm/gfx/nvn/shader_pairs.tsv
|
||||
dotnet publish dm/gfx/tools/dmsh_xlate -c Release -o ~/smbw-work/build/gfx/dmsh_xlate
|
||||
XLATE=~/smbw-work/build/gfx/dmsh_xlate/dmsh_xlate python3 dm/gfx/tools/dmsh_chain.py --header
|
||||
XLATE=... python3 dm/gfx/tools/dmsh_build.py ~/smbw-work/shader/pkg ~/smbw-work/build/cache/metal_shader_cache
|
||||
@@ -83,31 +37,68 @@ python3 dm/tools/make_ext_font.py ~/smbw-work/build/font/nintendo_ext.ttf
|
||||
~/smbw-work/run/boot.sh <logname> [seconds] # sets DM_MSL_DIR to the shader cache and DM_FONT_DIR to ~/smbw-work/build/font
|
||||
```
|
||||
|
||||
### Update 1.2.1 (v327680)
|
||||
|
||||
hactool rejects the update's patch NCA ("Invalid BKTR layout!"); `dm/tools/nx_extract` (LibHac) layers it over the base game instead and takes the title keys from the NSPs' tickets. Files identical to v0 become APFS clones, so the merged RomFS adds about 114 MB.
|
||||
|
||||
```
|
||||
dotnet build dm/tools/nx_extract -c Release -o ~/smbw-work/build/nx_extract
|
||||
~/smbw-work/build/nx_extract/nx_extract <prod.keys> <v0 NSP> <v327680 NSP> ~/smbw-work/data/v327680 ~/smbw-work/data/v0/RomFS
|
||||
DM_RULESET=native python3 dm/tools/patch_nso.py ~/smbw-work/data/v327680/ExeFS/{main,sdk} ~/smbw-work/build/patched_v327680 {main,sdk}
|
||||
# shaders: list both versions, base first, so one shader build serves both (then dmsh_chain.py --header, dmsh_build.py, c1_guard.py, make as above)
|
||||
<venv python with lz4> dm/gfx/tools/shader_split.py ~/smbw-work/data/v0/RomFS,~/smbw-work/data/v327680/RomFS ~/smbw-work/data/v0/ExeFS/main,~/smbw-work/data/v327680/ExeFS/main ~/smbw-work/shader/pkg dm/gfx/nvn/shader_map.tsv dm/gfx/nvn/shader_facts.tsv dm/gfx/nvn/shader_pairs.tsv
|
||||
GV=v327680 ~/smbw-work/run/boot.sh <logname> [seconds] # GV picks data/$GV and build/patched_$GV
|
||||
PATCHED=~/smbw-work/build/patched_v327680 ROMFS=~/smbw-work/data/v327680/RomFS dm/ios/tools/build_ipa.sh
|
||||
```
|
||||
|
||||
What 1.2.1 changes for the port:
|
||||
|
||||
- `main` is a new build (`ff773e90…`, SDK 16.2.12; v0 is 16.2.0). The one per-build rewriter cut (resource heap +16 KB) was carried over with `dm/tools/nso_fnmap.py`. The rewriter now writes each module's build ID into `fixups.tsv`, and the host limits two inherited D2R behaviours to D2R's build: the 3D render scale (it writes and calls fixed `main` addresses, which in Mario are unrelated code and data) and the owned add-on content.
|
||||
- New SDK calls the host answers: `nn::nfp` (amiibo: one NFC device on the connected controller that never finds a tag), `pl::IsRunningOnOunce` (false, i.e. a Switch 1), `ec::ShowShopApplicationInformationForUpgrade` (no-op), `account::PopOpenUsers` (none, so the game opens the preselected user as in v0). `oe::GetDisplayVersion` reports 1.2.1.
|
||||
- 3,451 SecredUber pixel shaders were recompiled and 6 effect shaders added; the shader tables hold both versions.
|
||||
|
||||
Agent-facing guides for analysing and porting other titles are in [docs/agents](docs/agents/README.md).
|
||||
|
||||
## Build the iOS app (unsigned IPA)
|
||||
|
||||
Needs Xcode 27, xcodegen, fontTools (`FONT_PY=` a python that has it, if `python3` does not), and the Mac build steps above (the rewritten modules in `~/smbw-work/build/patched_v0`). The app also bundles `dm/font/standard.ttf` (copied from macOS's Arial Unicode, `STD_FONT=` to override) and the generated `nintendo_ext.ttf`.
|
||||
Needs Xcode 27, xcodegen, fontTools (`FONT_PY=` a python that has it, if `python3` does not), and the Mac build steps above (the rewritten modules in `~/smbw-work/build/patched_v0`; for 1.2.1 set `PATCHED` and `ROMFS` as in the update section). The app also bundles `dm/font/standard.ttf` (copied from macOS's Arial Unicode, `STD_FONT=` to override) and the generated `nintendo_ext.ttf`.
|
||||
|
||||
```
|
||||
# iOS shader libraries for the current chain version (writes ~/smbw-work/build/ios/dmsh_b/b-<V>/; about 15 min)
|
||||
IOSOUT=~/smbw-work/build/ios/dmsh_b XLATE=... python3 dm/gfx/tools/dmsh_build.py ~/smbw-work/shader/pkg ~/smbw-work/build/cache/metal_shader_cache
|
||||
# the IPA (default ~/Desktop/SMBWMobile.ipa)
|
||||
dm/ios/tools/build_ipa.sh [out.ipa]
|
||||
# diagnostic build: bake environment switches into the app (one KEY=VALUE per line, # comments)
|
||||
DM_ENV_FILE=diag_env.txt dm/ios/tools/build_ipa.sh
|
||||
```
|
||||
|
||||
If Xcode's Locations setting puts build products in a custom location, `xcodebuild` ignores the script's `-derivedDataPath` for the products and the script stops at `FAIL: no built app` after `BUILD SUCCEEDED`. The app is then under that location (here `~/Library/Developer/Xcode/DerivedData/Build/Products/Release-iphoneos/DmIOS.app`). Finish by hand the way the script does: copy it into `Payload/`, delete `_CodeSignature` and `embedded.mobileprovision`, `cp -cR` the RomFS to `Payload/DmIOS.app/romfs`, then `zip -qry -n .zs:.bwav:.metallib out.ipa Payload`.
|
||||
|
||||
Sign and install it with your own tool (AltStore, Sideloadly, ...), which replaces the placeholder bundle ID `org.smbwmobile.app`. The app shell is `dm/ios/Sources`:
|
||||
|
||||
- `main.m`: environment, log, boot thread.
|
||||
- `front_ios.m`: the `front.h` functions and a centred 16:9 Metal view.
|
||||
- `DmGamepad.m`: touch gamepad (left stick anywhere in the lower left, D-pad, A/B/X/Y in Nintendo positions, ZL/L, ZR/R, −/+). Hidden while a physical controller is connected; physical buttons map by position.
|
||||
|
||||
The host log goes to the app's Documents (`log.txt`, previous run `log.prev.txt`), visible in the Files app. Game files are also accepted at `Documents/romfs`.
|
||||
The host log goes to the app's Documents (`log.txt`, previous run `log.prev.txt`), visible in the Files app. Game files are also accepted at `Documents/romfs`. Environment switches can also be put in `Documents/env.txt` (same format as `DM_ENV_FILE`; it takes precedence over the bundled file), so a sideloaded app can be switched without rebuilding.
|
||||
|
||||
Everything the app writes lives in its Documents: `save/` (game saves), `pso_user.tsv` (see below), the logs. Sideloading tools may give a reinstalled app a new data folder (LiveContainer did, under its `Data/Application/<UUID>/Documents`); the old saves and prewarm list stay in the previous folder and can be copied over with the Files app while the app is closed.
|
||||
|
||||
### Prewarm lists
|
||||
|
||||
Metal compiles every combination of shaders and pipeline settings for the GPU the first time it is used (tens of milliseconds each; a burst of them on entering a level is a visible freeze). The host keeps lists of what it had to compile and rebuilds those pipelines on a background queue at the next launch:
|
||||
|
||||
- `pso_user.tsv`, next to the save directory (`Documents/pso_user.tsv` on iOS, `run/pso_user.tsv` for the Mac `boot.sh`; `DM_PSO_USER=path` overrides): appended while playing, one line per shader (`S`), render pipeline (`R`, the hex of `pso_skey` in `dm/gfx/nvn/nvn_pso_skey.h`) or compute pipeline (`C`) that had to be built on the spot.
|
||||
- `dm/gfx/nvn/pso_manifest.tsv`: the seed list bundled with the app (and read by the Mac host next to `shader_map.tsv`). Prewarmed before `pso_user.tsv`. Currently empty.
|
||||
|
||||
The lines name shaders by content key and describe the pipeline settings; they hold no compiled code and nothing personal, so a list recorded on one iPhone works on any other device (each one compiles for its own GPU and iOS caches the result). Lines a build cannot use (unknown shader, a different key layout) are skipped. To seed the bundled list, append recorded lines from one or more devices to `pso_manifest.tsv` below its two header lines (duplicates are dropped at load; keep the recorded order, it is the order of first use) and rebuild the IPA. Record on the game version you ship: pipelines of the 3,451 shaders that 1.2.1 replaced are compiled for nothing on 1.2.1.
|
||||
|
||||
## Tools (dm/tools)
|
||||
|
||||
| Tool | What it does |
|
||||
|---|---|
|
||||
| `nx_extract/` | NSP → ExeFS / RomFS with LibHac, merging an update over the base game (C#, `dotnet build`) |
|
||||
| `nso_fnmap.py` | Carries offsets from one build to another (function matching with addresses masked; globals via their references) |
|
||||
| `nso2elf.py` | Converts NSO to ELF for Ghidra / REA |
|
||||
| `insn_census.py` | Counts the instruction sites the rewriter must handle |
|
||||
| `nso_syms.py` | Compares the SDK import surface of two games |
|
||||
|
||||
+8
-5
@@ -21,7 +21,7 @@ For the build steps of the whole project see `README.md` and `setup.sh` in the r
|
||||
- **Replay layer `nvn/nvn_mtl.m`**: at submit time it replays the record stream as Metal commands.
|
||||
- Memory pools are created as zero-copy `MTLBuffer`s, with the GPU address equal to the CPU address, so only unified-memory devices are supported.
|
||||
- The visible effects of "GPU completion" (sync objects becoming ready, events writing their value, counters) happen only after the Metal command buffer has actually completed.
|
||||
- Pipeline states and depth-stencil states are cached by state hash; at startup, shaders and pipelines are pre-built in the background according to the warm-up manifest.
|
||||
- Pipeline states and depth-stencil states are cached by state hash; at startup, shaders and pipelines are pre-built in the background according to the warm-up manifests (the bundled seed `pso_manifest.tsv`, then this device's `pso_user.tsv`; see below). A pipeline that fails to build is cached as failed and reported once with its shader keys.
|
||||
- The game's packaged textures are in Tegra block-linear layout and are deswizzled before upload (see `tex/README.md`).
|
||||
- Presentation: the game's swapchain texture is drawn to a `CAMetalLayer`, and presentation timing is scheduled by the pacer in `nvn/nvn_pace.h`.
|
||||
|
||||
@@ -42,7 +42,7 @@ Without Apple's Metal compiler (Windows, Linux, a Mac without Xcode) the last st
|
||||
|
||||
The whole chain is driven by `tools/dmsh_build.py`, and `setup.sh` runs it for you. Key points:
|
||||
|
||||
- **Key**: each shader is keyed by the first 16 hexadecimal digits of the sha256 of its control section (0x880 bytes). At run time the game hands the control section to `nvnProgramSetShaders`, and the replay layer computes the same key to find the library.
|
||||
- **Key**: each shader is keyed by the first 16 hexadecimal digits of the sha256 of its control section (the first 0x880 bytes) followed by its code area (many of Mario's shaders share a control section and differ in code). At run time the game hands both to `nvnProgramSetShaders`, and the replay layer computes the same key to find the library.
|
||||
- **Build outputs**: `<key>_b.metallib` (the shader library), `<key>_b.json` (reflection used by replay: each binding's kind, slot number, index within the argument buffer and so on), `<key>_b.metal` (the MSL source). Some fragment shaders additionally have `<key>_bf.metallib`, a "readable tile" version used for pass merging (see the file header of `tools/dmsh_fetch.py`). `_b_build.json` in the directory records the success or failure of each key in this run.
|
||||
- **`nvn/shader_map.tsv`**: the shader map, one row per shader variant. At run time only its "key → stage (vertex / fragment / compute)" is used.
|
||||
- **`nvn/shader_facts.tsv`**: the answer table for translation queries. The translator needs to know some facts that cannot be read from the machine code (the dimension type of a texture, image formats, the component types of vertex attributes and fragment outputs, a compute shader's thread group size and shared memory byte count), and this table gives the answers by key. The table ships with the repository, and building the shader libraries only reads it; the table is maintained by `tools/dmsh_facts.py`, whose file header has the details.
|
||||
@@ -53,7 +53,9 @@ The whole chain is driven by `tools/dmsh_build.py`, and `setup.sh` runs it for y
|
||||
- `dmsh_chain.py --header`: rewrites `nvn/dmsh_chain.h`.
|
||||
- It is normal for the local Xcode or spirv-cross version to differ from the one registered in the repository; `setup.sh` automatically re-registers according to the local tools.
|
||||
- **`nvn/c1_full.tsv`**: the index list for the shaders' built-in constant section (c1), generated by `tools/c1_guard.py` scanning the translated MSL; it lists the shader keys whose "run-time index could not be proven bounded". Its first line records the `b-<V>` it corresponds to. For the shaders on the list the replay layer allocates a separate whole 64 KiB buffer for c1; when the list is missing or its version does not match, it reports red and conservatively treats all shaders this way.
|
||||
- **`nvn/pso_manifest.tsv`**: the warm-up seed manifest, recording the shaders and pipelines used first after startup. The row order is the pre-build order; do not sort it.
|
||||
- **`nvn/shader_pairs.tsv`**: the (vertex key, pixel key) of every BNSH variation, written by `tools/shader_split.py`. Metal only builds a pipeline when every pixel shader input is a vertex shader output, while Maxwell gives an unwritten attribute a default; `dmsh_build.py` therefore adds the outputs a vertex shader's partners read but it doesn't write, set to (0, 0, 0, 1) (`vs_pad`). The table is part of the translation chain version.
|
||||
- **Several game versions**: `shader_split.py` takes comma-separated RomFS and `main` lists (base game first) and writes the union; the map, facts and pairs tables currently cover Mario v0 and 1.2.1.
|
||||
- **`nvn/pso_manifest.tsv`**: the warm-up seed manifest bundled with the app: shaders (`S`), render pipelines (`R`, the hex of `pso_skey` in `nvn/nvn_pso_skey.h`) and compute pipelines (`C`) to pre-build at startup. The row order is the pre-build order (first use); do not sort it. Its companion is `pso_user.tsv`, which the host appends next to the save directory whenever it has to build something on the spot, and which is pre-built after the seed at the next launch. Both name shaders by key and describe pipeline settings only, so a list recorded on one device works on any other; lines a build cannot use are skipped. Recorded lines can be appended to the seed below its header (currently empty).
|
||||
- **Shaders embedded in the program itself**: the main program additionally contains 4 compute shaders that are not in the shader package; they were hand-written as MSL from the disassembly and are in `nvn/nvn_builtin.h`.
|
||||
|
||||
Binding conventions followed during translation (file header of `tools/maxwell_refs.py`):
|
||||
@@ -103,7 +105,7 @@ The scripts in `tools/`; usage is written in each file's header:
|
||||
|
||||
## Building
|
||||
|
||||
The graphics layer has no separate library. `dm/host/Makefile` compiles `dm/gfx/*.c` and the source files under `dm/gfx/nvn/` and `dm/gfx/tex/` into the host `dmhost` by wildcard; the iOS App is built by `dm/ios/run.sh build`, which also copies the map, `pso_manifest.tsv`, `c1_full.tsv` and the `b-<V>/` shader libraries into the App. `tests/` and `metal/` do not go into the host.
|
||||
The graphics layer has no separate library. `dm/host/Makefile` compiles `dm/gfx/*.c` and the source files under `dm/gfx/nvn/` and `dm/gfx/tex/` into the host `dmhost` by wildcard; the iOS App is built by `dm/ios/tools/build_ipa.sh`, which also copies the map, `pso_manifest.tsv`, `c1_full.tsv` and the `b-<V>/` shader libraries into the App. `tests/` and `metal/` do not go into the host.
|
||||
|
||||
The shader libraries are generated by `setup.sh`:
|
||||
|
||||
@@ -145,6 +147,7 @@ Resource locations:
|
||||
| `DM_MSL_DIR=directory` | The parent directory of the shader libraries; the host loads from `b-<V>/` beneath it. When running the host on a Mac, set it to `$DM_ROOT/build/cache/metal_shader_cache` |
|
||||
| `DM_PSO_MANIFEST=path` or `off` | Use a different warm-up manifest, or turn warm-up off |
|
||||
| `DM_PSO_RECORD=path` | Appends the shaders and pipelines used in this run to a file, for re-recording the warm-up manifest |
|
||||
| `DM_PSO_USER=path` | Where to keep the per-device list of shaders and pipelines built on the spot (default: `pso_user.tsv` in the parent of `DM_SAVE`) |
|
||||
|
||||
Logging and capture:
|
||||
|
||||
@@ -177,7 +180,7 @@ Comparison and testing:
|
||||
The following paths are not implemented and report red in the log when reached (`RED_ONCE` in `nvn/nvn_mtl.m`):
|
||||
|
||||
- clears in the middle of a render pass, and color clears with a partial mask or partial scissor;
|
||||
- indexed quad primitives, 8-bit indices;
|
||||
- indexed quad primitives (8-bit indices are widened to 16-bit at replay);
|
||||
- some forms of texture-to-texture copy and texture clear.
|
||||
|
||||
Functions listed in `g_nops`, such as `MultiDraw*IndirectCount`, `DispatchComputeIndirect`, `SetDepthBounds` and `DownsampleTextureView`, are empty implementations: calling them neither reports red nor has any effect.
|
||||
@@ -21,7 +21,7 @@ English: [README.md](README.md)
|
||||
- **回放层 `nvn/nvn_mtl.m`**：提交时把记录流回放成 Metal 命令。
|
||||
- 内存池建成零拷贝的 `MTLBuffer`，GPU 地址等于 CPU 地址，所以只支持统一内存设备。
|
||||
- 「GPU 完成」的可见效果（同步对象就绪、事件写值、计数器）只在 Metal 命令缓冲真正完成之后发生。
|
||||
- 管线状态、深度模板状态按状态哈希缓存；启动时按预热清单在后台预建着色器与管线。
|
||||
- 管线状态、深度模板状态按状态哈希缓存；启动时按预热清单（随包的种子清单 `pso_manifest.tsv`，再加本机的 `pso_user.tsv`，见下）在后台预建着色器与管线。建不成的管线记为失败、不再重试，按着色器键报红一次。
|
||||
- 游戏打包贴图是 Tegra 块线性排布，上传前解排布（见 `tex/README.zh-CN.md`）。
|
||||
- 出图：游戏的交换链贴图画到 `CAMetalLayer`，出图节拍由 `nvn/nvn_pace.h` 的节拍器排。
|
||||
|
||||
@@ -42,7 +42,7 @@ Switch 原版着色器包
|
||||
|
||||
整条链由 `tools/dmsh_build.py` 驱动，`setup.sh` 会替你跑。要点：
|
||||
|
||||
- **键**：每个着色器以它的控制段（0x880 字节）的 sha256 前 16 位十六进制为键。运行时游戏把控制段交给 `nvnProgramSetShaders`，回放层算出同一个键去找库。
|
||||
- **键**：每个着色器以「控制段（前 0x880 字节）+ 代码区」的 sha256 前 16 位十六进制为键（马力欧有很多着色器控制段相同、代码不同）。运行时游戏把两者交给 `nvnProgramSetShaders`，回放层算出同一个键去找库。
|
||||
- **产物**：`<键>_b.metallib`（着色器库）、`<键>_b.json`（回放用的反射：各绑定的种类、槽号、在参数缓冲里的序号等）、`<键>_b.metal`（MSL 源）。部分片元着色器另有 `<键>_bf.metallib`，是「可读 tile」版本，供通道合并用（见 `tools/dmsh_fetch.py` 文件头）。目录里的 `_b_build.json` 记本次每个键的成败。
|
||||
- **`nvn/shader_map.tsv`**：着色器映射表，每个着色器变体一行。运行时只用其中「键 → 阶段（顶点 / 片元 / 计算）」。
|
||||
- **`nvn/shader_facts.tsv`**：翻译查询的作答表。翻译器需要知道一些机器码里读不出的事实（贴图的维度类型、图像格式、顶点属性与片元输出的分量类型、计算着色器的线程组大小与共享内存字节数），这张表按键给出答案。表随仓库，出着色器库只读这张表；表由 `tools/dmsh_facts.py` 维护，文件头有说明。
|
||||
@@ -53,7 +53,9 @@ Switch 原版着色器包
|
||||
- `dmsh_chain.py --header`：重写 `nvn/dmsh_chain.h`。
|
||||
- 本机的 Xcode 或 spirv-cross 版本与仓库登记的不同是正常的，`setup.sh` 会自动按本机工具重新登记。
|
||||
- **`nvn/c1_full.tsv`**：着色器自带常数段（c1）的下标清单，由 `tools/c1_guard.py` 扫描翻译出的 MSL 生成，列出「运行时下标没能证明有界」的着色器键。首行记着它对应的 `b-<V>`。回放层对清单里的着色器给 c1 单独分配整块 64 KiB 缓冲；清单缺失或版本对不上时报红，并对全部着色器都这样保守处理。
|
||||
- **`nvn/pso_manifest.tsv`**：预热种子清单，记启动后最先用到的着色器与管线，行序就是预建顺序，不要排序。
|
||||
- **`nvn/shader_pairs.tsv`**：每个 BNSH 变体的（顶点键, 像素键），由 `tools/shader_split.py` 写出。Metal 要求像素着色器的每个输入都是顶点着色器的输出才肯建管线，而 Maxwell 对没写的属性给缺省值；所以 `dmsh_build.py` 给顶点着色器补上配对像素着色器读、它却没写的输出，值为 (0, 0, 0, 1)（`vs_pad`）。这张表算进翻译链版本。
|
||||
- **多个游戏版本**：`shader_split.py` 的 RomFS 与 `main` 参数可写成逗号分隔的列表（基础版在前），输出并集；映射表、作答表、配对表目前覆盖马力欧 v0 与 1.2.1。
|
||||
- **`nvn/pso_manifest.tsv`**：随包的预热种子清单：启动时预建的着色器（`S`）、渲染管线（`R`，即 `nvn/nvn_pso_skey.h` 里 `pso_skey` 的十六进制）与计算管线（`C`）。行序就是预建顺序（首次使用的顺序），不要排序。与它配套的是 `pso_user.tsv`：宿主每次不得不当场建着色器或管线时追加记到存档目录旁，下次启动在种子清单之后预建。两者都只以键指认着色器、只描述管线设置，所以一台设备录的清单在别的设备上同样有效；本构建用不上的行跳过。录到的行可以追加到种子清单表头之后（目前为空）。
|
||||
- **程序本体内嵌的着色器**：主程序里另有 4 个计算着色器不在着色器包里，按反汇编手写成 MSL，在 `nvn/nvn_builtin.h`。
|
||||
|
||||
翻译时遵循的绑定约定（`tools/maxwell_refs.py` 文件头）：
|
||||
@@ -103,7 +105,7 @@ Switch 原版着色器包
|
||||
|
||||
## 构建
|
||||
|
||||
图形层没有单独的库。`dm/host/Makefile` 按通配把 `dm/gfx/*.c`、`dm/gfx/nvn/` 与 `dm/gfx/tex/` 下的源文件编进宿主 `dmhost`；iOS App 由 `dm/ios/run.sh build` 构建，同时把映射表、`pso_manifest.tsv`、`c1_full.tsv` 和 `b-<V>/` 着色器库拷进 App。`tests/` 与 `metal/` 不进宿主。
|
||||
图形层没有单独的库。`dm/host/Makefile` 按通配把 `dm/gfx/*.c`、`dm/gfx/nvn/` 与 `dm/gfx/tex/` 下的源文件编进宿主 `dmhost`；iOS App 由 `dm/ios/tools/build_ipa.sh` 构建，同时把映射表、`pso_manifest.tsv`、`c1_full.tsv` 和 `b-<V>/` 着色器库拷进 App。`tests/` 与 `metal/` 不进宿主。
|
||||
|
||||
着色器库由 `setup.sh` 生成：
|
||||
|
||||
@@ -145,6 +147,7 @@ make -C dm/gfx/metal run # 离屏示例自测，见 metal/README.zh-CN.md
|
||||
| `DM_MSL_DIR=目录` | 着色器库的上级目录，宿主从其下的 `b-<V>/` 加载。在 Mac 上跑宿主时设为 `$DM_ROOT/build/cache/metal_shader_cache` |
|
||||
| `DM_PSO_MANIFEST=路径` 或 `off` | 换一份预热清单，或关掉预热 |
|
||||
| `DM_PSO_RECORD=路径` | 把本次运行用到的着色器与管线追加记到文件，用来重录预热清单 |
|
||||
| `DM_PSO_USER=路径` | 本机「当场建的着色器与管线」清单放哪里（缺省：`DM_SAVE` 父目录下的 `pso_user.tsv`） |
|
||||
|
||||
日志与抓取：
|
||||
|
||||
@@ -177,7 +180,7 @@ make -C dm/gfx/metal run # 离屏示例自测，见 metal/README.zh-CN.md
|
||||
下面这些路径没有实现，走到会在日志里报红（`nvn/nvn_mtl.m` 里的 `RED_ONCE`）：
|
||||
|
||||
- 渲染通道中途的清除、带部分掩码或部分剪裁的清色；
|
||||
- 带索引的四边形图元、8 位索引；
|
||||
- 带索引的四边形图元（8 位索引在回放时扩成 16 位）；
|
||||
- 部分形态的贴图到贴图拷贝与清贴图。
|
||||
|
||||
`MultiDraw*IndirectCount`、`DispatchComputeIndirect`、`SetDepthBounds`、`DownsampleTextureView` 等列在 `g_nops` 里的函数是空实现，调用不报红也不生效。
|
||||
@@ -22,9 +22,9 @@ Two conventions that run through the whole table:
|
||||
| TextureView | `newTextureViewWithPixelFormat:textureType:levels:slices:` | — |
|
||||
| Sampler | `MTLSamplerState` | For border color Metal has only three presets (transparent black, opaque black, opaque white); the nearest one to the border color NVN gives is chosen |
|
||||
| TexturePool / SamplerPool | A host-side table: descriptor number → texture / sampler | The value of a separate texture handle is the descriptor number |
|
||||
| Program | One `MTLFunction` per stage | Keyed by the hash of the shader's control section and loaded from the offline-translated `<key>_b.metallib`; nothing is translated at run time. For the translation chain see the README |
|
||||
| Program | One `MTLFunction` per stage | Keyed by the hash of the shader's control section plus code area and loaded from the offline-translated `<key>_b.metallib`; nothing is translated at run time. For the translation chain see the README |
|
||||
| State objects (Blend / Color / ChannelMask / Multisample / Polygon / DepthStencil / VertexAttrib / VertexStream) | Fields of the render pipeline descriptor + `MTLDepthStencilState` + encoder state | The game copies these small structures by value, and the host side packs them in place with bit fields. Render pipelines and depth-stencil states are cached by state hash |
|
||||
| CommandBuffer | A record stream recorded in the game's command memory | Replayed at submit time |
|
||||
| CommandBuffer | A record stream recorded in the game's command memory | Replayed at submit time. The host's record inside the game's `NVNcommandBuffer` leaves byte +0x10 alone: the game's inlined NVN code writes it directly. When a recording outgrows the game's memory the host lends a segment, released only when the game re-grants the memory the recording started in; `GetCommandMemoryUsed` counts the bytes in both |
|
||||
| Queue | Metal command queue | `SubmitCommands` = replay and submit |
|
||||
| Sync | The fence value of a submission | `FenceSync` records the fence value of the most recent submission; `SyncWait` lets the caller through only after the GPU has actually completed up to that value (the completed value of an `MTLSharedEvent`). Waiting on a sync object that was never fenced returns a timeout directly |
|
||||
| Event | A 32-bit value in game memory | The value is written after the GPU completes |
|
||||
|
||||
@@ -22,9 +22,9 @@ English: [nvn_metal_mapping.md](nvn_metal_mapping.md)
|
||||
| TextureView | `newTextureViewWithPixelFormat:textureType:levels:slices:` | — |
|
||||
| Sampler | `MTLSamplerState` | 边框色 Metal 只有三种预设（透明黑、不透明黑、不透明白），按 NVN 给的边框色就近取 |
|
||||
| TexturePool / SamplerPool | 宿主侧的表：描述符号 → 贴图 / 采样器 | 分离贴图句柄的值就是描述符号 |
|
||||
| Program | 每个阶段一个 `MTLFunction` | 以着色器控制段的哈希为键，从离线翻译好的 `<键>_b.metallib` 加载；运行时不翻译。翻译链见 README |
|
||||
| Program | 每个阶段一个 `MTLFunction` | 以着色器「控制段 + 代码区」的哈希为键，从离线翻译好的 `<键>_b.metallib` 加载；运行时不翻译。翻译链见 README |
|
||||
| 状态对象（Blend / Color / ChannelMask / Multisample / Polygon / DepthStencil / VertexAttrib / VertexStream） | 渲染管线描述符的字段 + `MTLDepthStencilState` + 编码器状态 | 游戏按值拷贝这些小结构，宿主侧用位域就地打包。渲染管线、深度模板状态按状态哈希缓存 |
|
||||
| CommandBuffer | 录在游戏命令内存里的记录流 | 提交时回放 |
|
||||
| CommandBuffer | 录在游戏命令内存里的记录流 | 提交时回放。宿主放在游戏 `NVNcommandBuffer` 里的记录让开 +0x10 那个字节：游戏内联的 NVN 代码直接写它。录制超出游戏给的内存时宿主借出段，只在游戏重新给出这段录制起始所在的内存时才收回；`GetCommandMemoryUsed` 两边的字节都算 |
|
||||
| Queue | Metal 命令队列 | `SubmitCommands` = 回放并提交 |
|
||||
| Sync | 提交的栅栏值 | `FenceSync` 记下最近一次提交的栅栏值；`SyncWait` 只在 GPU 真正完成到该值之后放行（`MTLSharedEvent` 的已完成值）。没围栏过的同步对象等待直接返回超时 |
|
||||
| Event | 游戏内存里的一个 32 位值 | 写值在 GPU 完成之后发生 |
|
||||
|
||||
@@ -1,118 +0,0 @@
|
||||
# dm/host: host file index
|
||||
|
||||
中文: [README.zh-CN.md](README.zh-CN.md)
|
||||
|
||||
The host does two things: it loads the rewritten game program and provides the platform services that program calls. The features added on top of the original game, and the touch adaptation, live in this directory too. For how it all works see [dm/README.md](../README.md).
|
||||
|
||||
The file name prefix tells you which part a file belongs to:
|
||||
|
||||
- No prefix (`main.c`, `loader.c`, `entry.S`, …) and `hle*.c`: loading, system calls, system services, front end and diagnostics. These are listed one by one under "Files in host/" in [dm/README.md](../README.md); this page does not repeat that and only adds what is missing there.
|
||||
- `dm_*.c`: features added on top of the original game, and the touch adaptation. That section gives only a few examples; this page lists them all by responsibility.
|
||||
|
||||
Every file opens with a comment saying what it does and what that is based on. Addresses in the feature files are all offsets into the original 3.3.94012 image.
|
||||
|
||||
## Base files not listed in that section
|
||||
|
||||
| File | What it does |
|
||||
|---|---|
|
||||
| `Makefile` | Builds the Mac host `dmhost`. `hle_*.c` and the graphics-layer sources are picked up by wildcard, so adding a file needs no change here; the three missing tables (`dm_vault_tab.h`, `dm_hud_exp.h`, `dm_bright_levels.inc`) are generated from your game files and are not in the repository |
|
||||
| `hle_target.c` | Auto-target priority for touch taps: bosses, then elites, then normal monsters. Hooks the comparison function of the original controller targeting; the original eligibility, range and line-of-sight checks still apply |
|
||||
| `hle_private_aim.c` | Aiming by dragging a skill button: converts the right stick's on-screen direction into a direction in the game world |
|
||||
| `touch_mac.c`, `touch_mac.h` | The touch front end of the Mac host: feeds touch points tick by tick from a touch script (`DM_TOUCH_SCRIPT`); the touch cases in [dm/verify](../verify/README.md) run on it |
|
||||
| `romfs_validate.h` | Interface of `romfs_validate.c`: checks that the resource archives exist and have the right size, and produces the list of what is missing for the player |
|
||||
| `stash_seed.h` | The preset content used the first time the shared stash is created. Not shipped in the public repository (it is the bytes of a save the game generated): without it nothing is seeded |
|
||||
| `dm_derived_rt.h` | Used only in the "shell" that contains no game content: the tables generated from game files are not compiled in but read from the install package at run time; see [docs/NO_XCODE.md](../../docs/NO_XCODE.md) |
|
||||
|
||||
Also, besides diagnostics, `diag.c` holds the entry hook of every server tick: work that `dm_vault.c`, `dm_pages.c` and others must do inside a server tick is called from there.
|
||||
|
||||
## In-game features (`dm_*`)
|
||||
|
||||
### Touch and UI interaction
|
||||
|
||||
| File | What it does |
|
||||
|---|---|
|
||||
| `dm_cursor.c`, `dm_cursor.h` | The layer where touch meets the game UI: turns touch points into the original cursor position and confirm presses, sends taps to the original widgets as UI messages, and decides where a drag-and-drop lands. The per-frame work of the features is also dispatched here, by touch / controller mode |
|
||||
| `dm_cursor_item.c`, `dm_cursor_item.h` | Which item the cursor is pointing at, taken from the game's own hover object rather than computed back from cell coordinates |
|
||||
| `dm_world.c`, `dm_world.h` | Tapping a target in the game world interacts with it: tap an item name to pick it up; tap an NPC, object or exit to walk over and interact. Reuses the game's own click entry point |
|
||||
| `dm_actions.c`, `dm_actions.h` | Entry points for "select first, then confirm": buying and selling at merchants, stat and skill point allocation, setting the quantity box. All through the paths of the game's own controller version |
|
||||
| `dm_textinput.c` | Connects UI text boxes (gold amount, Chronicle search and so on) to the system keyboard |
|
||||
| `dm_textbox.c`, `dm_textbox_core.c`, `dm_textbox_core.h` | Makes plain text boxes display the color codes in item annotations correctly; `_core` is pure logic that does not touch game memory and is tested directly by the unit tests |
|
||||
|
||||
### Controller and the two modes
|
||||
|
||||
| File | What it does |
|
||||
|---|---|
|
||||
| `dm_padmode.c` | The two UI layouts, controller and touch: connecting a controller brings back the original controller UI, touching the screen brings back the touch UI, with no restart |
|
||||
| `dm_modetx.h` | The mode-switch transaction: freeze input, the old side lets go, the game rebuilds its UI, the new side takes over, input is released. A pure state machine |
|
||||
| `dm_mode_items.h` | Registry of behavior that runs only in touch mode or only in controller mode; a new mode-specific behavior gets one row here |
|
||||
| `dm_padmap.h` | Maps the four face buttons of a physical controller to game buttons, by the letter printed on the key cap |
|
||||
|
||||
### Screen and picture
|
||||
|
||||
| File | What it does |
|
||||
|---|---|
|
||||
| `dm_uiscreen.c`, `dm_uiscreen.h` | On screens narrower than 16:9 (iPad), lays the UI out at 16:9 while the 3D picture still fills the whole screen |
|
||||
| `dm_bright.c` | Dynamic exposure for the 3D brightness boost: how much to brighten depends on the kind of level you are in |
|
||||
| `dm_hud.c`, `dm_hud.h` | The numbers on the bottom UI (level, experience percentage, difficulty), and the request for a difficulty step (1–8) |
|
||||
|
||||
### Text and language
|
||||
|
||||
| File | What it does |
|
||||
|---|---|
|
||||
| `dm_i18n.c`, `dm_i18n.h` | Looks up our own UI text by key; the language follows the game's language setting |
|
||||
| `dm_i18n_tab.h` | The compile-time table generated from the string table. Generated; do not edit by hand, see [dm/i18n/README.md](../i18n/README.md) |
|
||||
| `dm_update.c` | The build number and new-version notice on the character select page |
|
||||
| `dm_namegen.c` | Random names when the name field is left empty: assembled from syllables, avoiding names already taken |
|
||||
|
||||
### Endgame modes
|
||||
|
||||
| File | What it does |
|
||||
|---|---|
|
||||
| `dm_rift.c`, `dm_rift.h` | The Rift: the portal, picking maps, progress, the guardian, the timer and how many floors a clear is worth |
|
||||
| `dm_rift_ui.c` | The Rift panel, built from original widgets |
|
||||
| `dm_doorspot.h` | Where the two doors appear when a Rift floor is settled: guaranteed to be on a cell the player can walk to. Pure algorithm |
|
||||
| `dm_bossrush.c`, `dm_bossrush.h` | Boss rush |
|
||||
| `dm_elysium.c`, `dm_elysium.h` | Elysium mode: a per-character switch chosen at character creation; on entering a game it changes data tables in memory, and restores the original values on leaving |
|
||||
| `dm_kills.c` | The kill board: per character and per difficulty, the kill count, first-kill time and fastest time for each boss |
|
||||
|
||||
### Stash and items
|
||||
|
||||
| File | What it does |
|
||||
|---|---|
|
||||
| `dm_vault.c` | The server half of the Chronicle vault: depositing and withdrawing identified unique and set items |
|
||||
| `dm_vault_ui.c` | The Chronicle vault's UI: the sixth tab of the stash |
|
||||
| `dm_pages.c` | Personal stash pages 2–5: the items of the other pages are kept in a side file in the character's directory, and page turning happens inside the server tick |
|
||||
| `dm_pouch.c` | The pouch and "Deposit": runes and gems picked up in the field go into the pouch without taking inventory cells, and are stored away when you open the stash back in town |
|
||||
| `dm_runeword.c` | The runeword helper: long-press a socketed base to list the runewords it can make, and socket them in one tap once you have the runes |
|
||||
|
||||
### Convenience features
|
||||
|
||||
| File | What it does |
|
||||
|---|---|
|
||||
| `dm_potpick.c` | Picks up matching potions automatically when the belt is not full |
|
||||
| `dm_beltauto.c` | Refills the belt with potions when the inventory is opened |
|
||||
| `dm_dropfilter.c` | The light beam and landing sound for high-value drops |
|
||||
| `dm_drophaptic.c` | Haptic feedback when a high-value item lands, in three strengths |
|
||||
| `dm_freetp.c` | Free town portal: reuses the original town portal skill directly |
|
||||
| `dm_townwp.c` | Entering a game, or a quick restart, puts you next to the town waypoint |
|
||||
| `dm_shoprefresh.c` | "Refresh Stock" on the merchant trade page; calls only the original stock mechanism |
|
||||
|
||||
### Settings, saves and small shared pieces
|
||||
|
||||
| File | What it does |
|
||||
|---|---|
|
||||
| `dm_custom.c`, `dm_custom.h` | The "Options → Custom" page: all feature switches are read and written here; modules only call `dm_custom_get` |
|
||||
| `dm_cloudsync.c` | The core of iCloud save sync: comparing and moving files between the local save directory and one cloud directory, without touching the iCloud API |
|
||||
| `dm_unit.h` | A small shared helper: reads which level a unit is in |
|
||||
|
||||
## tests/
|
||||
|
||||
The host's unit tests, in one command:
|
||||
|
||||
```
|
||||
make -C dm/host/tests run
|
||||
```
|
||||
|
||||
It needs the game files and a workspace prepared by `./setup.sh`. The `t_*.c` files are per-module unit tests. Four source-only checks run with them: no writes to the original game's UI widgets (`native_write_scan.py`), registration of touch / controller mode behavior (`mode_scan.py`), the string table matching its generated header (`dm/ios/tools/dm_i18n.py --check`), and the literal gate (`dm/i18n/scan_literals.py`); so do two checks against the original program image (`movie_cut.py`, `oe_notify.py`).
|
||||
|
||||
The directory also holds a few standalone scripts that are not part of that command (`run_rule.py`, `run_manifest.py`, `move_codec.py`, `client_sta_gate.py`, `server42_rule.py`, `server_walk_request.py`); what each is for, and its limits, is in its file header.
|
||||
@@ -1,118 +0,0 @@
|
||||
# dm/host：宿主文件索引
|
||||
|
||||
English: [README.md](README.md)
|
||||
|
||||
宿主做两件事：装载改写后的游戏程序、提供它调用的平台服务；在原版之上加的功能和触摸适配也在这个目录里。整体怎么工作见 [dm/README.zh-CN.md](../README.zh-CN.md)。
|
||||
|
||||
文件名的前缀就是分工：
|
||||
|
||||
- 没有前缀的（`main.c`、`loader.c`、`entry.S`……）与 `hle*.c`：装载、系统调用、系统服务、前端与诊断。这部分在 [dm/README.zh-CN.md](../README.zh-CN.md) 的「host/ 各文件」一节已逐个列出，本页不重复，只补那里没列的。
|
||||
- `dm_*.c`：在原版之上加的功能和触摸适配。那一节只举了几个例子，本页按职责列全。
|
||||
|
||||
每个文件开头的注释说明它做什么、依据是什么；功能文件里的地址都是 3.3.94012 原映像的偏移。
|
||||
|
||||
## 基础部分里上面那一节没列的
|
||||
|
||||
| 文件 | 管什么 |
|
||||
|---|---|
|
||||
| `Makefile` | 编出 Mac 宿主 `dmhost`。`hle_*.c` 与图形层的源文件按通配纳入，新增文件不用改它；缺的三份表（`dm_vault_tab.h`、`dm_hud_exp.h`、`dm_bright_levels.inc`）从你的游戏文件现生成，不进仓库 |
|
||||
| `hle_target.c` | 触屏点按时自动选怪的先后：首领、精英、普通。接在原版手柄选目标的比较函数上，原版的资格、射程、视线检查照旧 |
|
||||
| `hle_private_aim.c` | 拖动技能键瞄准：把右摇杆在屏幕上的方向换算成游戏世界里的方向 |
|
||||
| `touch_mac.c`、`touch_mac.h` | Mac 宿主上的触摸前端：按触点脚本（`DM_TOUCH_SCRIPT`）逐拍送触点，[dm/verify](../verify/README.zh-CN.md) 的触摸功能用例靠它跑 |
|
||||
| `romfs_validate.h` | `romfs_validate.c` 的接口：核对资源档案在不在、大小对不对，并给出给玩家看的缺失清单 |
|
||||
| `stash_seed.h` | 第一次生成共享仓库时用的预置内容。公开仓库不带这个文件（它是游戏生成的一份存档的字节）：不在就不预置 |
|
||||
| `dm_derived_rt.h` | 只在不含游戏内容的「壳」里用：由游戏文件生成的几张表不编进代码，运行时从安装包里读，见 [docs/NO_XCODE.zh-CN.md](../../docs/NO_XCODE.zh-CN.md) |
|
||||
|
||||
另外，`diag.c` 除了诊断，还是每个服务端拍的入口：`dm_vault.c`、`dm_pages.c` 等要在服务端拍里做的事由它调用。
|
||||
|
||||
## 游戏内功能（`dm_*`）
|
||||
|
||||
### 触摸与界面交互
|
||||
|
||||
| 文件 | 管什么 |
|
||||
|---|---|
|
||||
| `dm_cursor.c`、`dm_cursor.h` | 触摸落到游戏界面的那一层：把触点变成原版光标位置与确认键，把点击作为界面消息发给原版控件，判定拖放的落点。各功能的每帧处理也在这里按触屏 / 手柄模式调度 |
|
||||
| `dm_cursor_item.c`、`dm_cursor_item.h` | 光标当前指着哪件物品，按游戏自己的悬停对象取，不按格子坐标反算 |
|
||||
| `dm_world.c`、`dm_world.h` | 点游戏世界里的目标就是与它交互：点物品名拾取，点 NPC、物件、出口走过去再交互。复用游戏自己的点击入口 |
|
||||
| `dm_actions.c`、`dm_actions.h` | 「先选中、再确认」的入口：商人买卖、属性与技能加点、数量框设值。全部走游戏手柄版自己的路径 |
|
||||
| `dm_textinput.c` | 界面文本框（金币数量、编年史搜索等）接系统键盘 |
|
||||
| `dm_textbox.c`、`dm_textbox_core.c`、`dm_textbox_core.h` | 让普通文字框正确显示物品注释里的颜色码；`_core` 是不碰游戏内存的纯逻辑，单测直接测它 |
|
||||
|
||||
### 手柄与两种模式
|
||||
|
||||
| 文件 | 管什么 |
|
||||
|---|---|
|
||||
| `dm_padmode.c` | 手柄 / 触屏两套界面布局：连上手柄回到原版手柄界面，碰屏幕回触屏界面，切换不重启 |
|
||||
| `dm_modetx.h` | 模式切换的事务：冻结输入、旧一侧收手、游戏重建界面、新一侧就位、放开输入。纯状态机 |
|
||||
| `dm_mode_items.h` | 只在触屏或只在手柄模式下做事的行为的登记表；新加按模式分的行为在这里登记一行 |
|
||||
| `dm_padmap.h` | 实体手柄四个面键对应到游戏键：按键帽上印的字母对应 |
|
||||
|
||||
### 屏幕与画面
|
||||
|
||||
| 文件 | 管什么 |
|
||||
|---|---|
|
||||
| `dm_uiscreen.c`、`dm_uiscreen.h` | 比 16:9 窄的屏幕（iPad）上，界面按 16:9 摆、3D 画面仍铺满整屏 |
|
||||
| `dm_bright.c` | 「3D 画面加亮」的动态曝光：按所在关卡的类型决定加亮多少 |
|
||||
| `dm_hud.c`、`dm_hud.h` | 底部界面上的数字（等级、经验百分比、难度），以及难度档位（1–8）的请求 |
|
||||
|
||||
### 文字与语言
|
||||
|
||||
| 文件 | 管什么 |
|
||||
|---|---|
|
||||
| `dm_i18n.c`、`dm_i18n.h` | 我们加的界面文字按键取串，语言跟游戏的语言设置走 |
|
||||
| `dm_i18n_tab.h` | 由中英表生成的编译期表。生成文件，不要手改，见 [dm/i18n/README.zh-CN.md](../i18n/README.zh-CN.md) |
|
||||
| `dm_update.c` | 选角页的构建号与新版本提示 |
|
||||
| `dm_namegen.c` | 起名页留空时随机取名：按音节拼，避开已有的角色名 |
|
||||
|
||||
### 终局玩法
|
||||
|
||||
| 文件 | 管什么 |
|
||||
|---|---|
|
||||
| `dm_rift.c`、`dm_rift.h` | 秘境：进门、抽图、进度、守护者、计时与层数结算 |
|
||||
| `dm_rift_ui.c` | 秘境面板，用原版控件搭 |
|
||||
| `dm_doorspot.h` | 秘境结算处两扇门落在哪：保证落在玩家走得到的格上。纯算法 |
|
||||
| `dm_bossrush.c`、`dm_bossrush.h` | 首领连战 |
|
||||
| `dm_elysium.c`、`dm_elysium.h` | 极乐世界模式：建角时勾选的按角色开关，进局时改内存里的数据表，退局写回原值 |
|
||||
| `dm_kills.c` | 刷怪积分牌：按角色、按难度记首领的击杀次数、首杀时刻与最快用时 |
|
||||
|
||||
### 仓库与物品
|
||||
|
||||
| 文件 | 管什么 |
|
||||
|---|---|
|
||||
| `dm_vault.c` | 编年史仓库的服务端一半：已鉴定的暗金与套装的存入、取出 |
|
||||
| `dm_vault_ui.c` | 编年史仓库的界面：储藏箱的第六个页签 |
|
||||
| `dm_pages.c` | 个人仓库第 2–5 页：另外几页的物品存在角色目录的侧文件里，翻页在服务端拍里做 |
|
||||
| `dm_pouch.c` | 百宝囊与「存放」：野外的符文、宝石收进囊里不占背包格，回城开储藏箱时自动存好 |
|
||||
| `dm_runeword.c` | 符文之语助手：长按带孔底材列出能做的符文之语，符文齐了一键镶嵌 |
|
||||
|
||||
### 省心功能
|
||||
|
||||
| 文件 | 管什么 |
|
||||
|---|---|
|
||||
| `dm_potpick.c` | 腰带不满时自动捡起对应的药水 |
|
||||
| `dm_beltauto.c` | 打开背包时自动把药水补进腰带 |
|
||||
| `dm_dropfilter.c` | 高价值掉落的光柱与落地音 |
|
||||
| `dm_drophaptic.c` | 高价值物品落地时的震动，分三档 |
|
||||
| `dm_freetp.c` | 免费回城：直接复用原版的回城技能 |
|
||||
| `dm_townwp.c` | 进局与一键重开时落在城镇传送小站旁 |
|
||||
| `dm_shoprefresh.c` | 商人交易页的「刷新货架」，只调原版的货单机制 |
|
||||
|
||||
### 设置、存档与公共小件
|
||||
|
||||
| 文件 | 管什么 |
|
||||
|---|---|
|
||||
| `dm_custom.c`、`dm_custom.h` | 「选项 → 自定义」页：各项功能的开关统一在这里读写，各模块只调 `dm_custom_get` |
|
||||
| `dm_cloudsync.c` | iCloud 存档同步的核心：本地存档目录与一个云端目录之间的文件对比与搬运，不碰 iCloud 接口 |
|
||||
| `dm_unit.h` | 公共小函数：从一个单位读出它所在的关卡 |
|
||||
|
||||
## tests/
|
||||
|
||||
宿主的单元测试，一条命令：
|
||||
|
||||
```
|
||||
make -C dm/host/tests run
|
||||
```
|
||||
|
||||
需要游戏文件与 `./setup.sh` 备好的工作区。`t_*.c` 是各模块的单测；一起跑的还有四条只读源码的检查：不许写原版界面控件（`native_write_scan.py`）、触屏 / 手柄模式的登记（`mode_scan.py`）、中英表与生成头文件一致（`dm/ios/tools/dm_i18n.py --check`）、字面量闸门（`dm/i18n/scan_literals.py`）；以及两条对着原版程序映像的核对（`movie_cut.py`、`oe_notify.py`）。
|
||||
|
||||
目录里另有几个不在这条命令里的独立脚本（`run_rule.py`、`run_manifest.py`、`move_codec.py`、`client_sta_gate.py`、`server42_rule.py`、`server_walk_request.py`），各自的用途与局限写在文件头。
|
||||
@@ -23,6 +23,8 @@ ht -t nca --exefsdir=$W/data/v0/ExeFS --romfsdir=$W/data/v0/RomFS $W/build/nsp/<
|
||||
|
||||
- ExeFS holds `main.npdm`, `rtld` (usually), `main`, `sdk` and sometimes `subsdk*` modules (NSO). D2R and Mario both had only `main` + `sdk` worth loading.
|
||||
- Update NSPs: the update's Program NCA is a patch NCA; extract it with `--basenca=<base program nca>` so RomFS is the merged view. Keep base and update in separate directories (`data/v0`, `data/v<ver>`); the rewriter and the host take one ExeFS at a time.
|
||||
- Newer patch NCAs (seen: Mario 1.2.1, key generation 17) make hactool stop with `Invalid BKTR layout!`. `dm/tools/nx_extract` (C#, LibHac 0.19) handles them: `nx_extract <prod.keys> <base.nsp> <update.nsp> <out dir> [<base RomFS dir>]`. It reads the NCAs straight out of both NSPs (no 3+ GB intermediate NCA), imports the common tickets itself, prints the display version and module build IDs, and with a base RomFS dir writes identical files as APFS clones.
|
||||
- After an update, find every address the host or rewriter hard-codes for the old build (`CUTS` / `OBS` / `PARAMS` in `patch_nso.py`, `->base + 0x…` in the host) and carry it over with `dm/tools/nso_fnmap.py <old.orig.bin> <new.orig.bin> 0x…`: it matches functions with address bits masked and maps globals through their `adrp` references; check each result by disassembly (`sdis.py`). Also diff the imported SDK symbols: new imports that reach a system service need host implementations, or the real SDK aborts after connecting to `sm:`.
|
||||
- Decompressed NSO images for analysis: `dm/tools/nso.py <nso> <out.bin>` (lz4; use a venv that has `lz4`). `dm/tools/nso2elf.py` makes an ELF for Ghidra / REA / objdump.
|
||||
|
||||
## Workspace layout used so far
|
||||
|
||||
@@ -39,6 +39,7 @@ Each entry: symptom → root cause → fix (where the fix lives). Check these fi
|
||||
- **First `AllocateAddressRegion` fails with 0x1803** → SDK 16's AddressRegionManager takes 0x3fe00000 at the start of the alias region as a node area → the alias region must be larger than 1 GB.
|
||||
- **Large resources silently skipped, later a null deref where the resource is used** → the game reserves "virtual address memory" (Mario: 1.75 GB in `HeapDef.byml`) and maps it into the alias region with `MapPhysicalMemory` on demand; the alias region left after the node area was too small. Also visible as `SetHeapSize` asking for much less than the pool. → repartition the guest block (`hle_svc.c` `mem_init`): Mario uses alias 2.875 GB, stacks 256 MB, heap 1.625 GB (it asks for 1.47 GB), 5 GB in total. Measure before sizing (`DM_SVC=1`: the `SetHeapSize` request and the highest `MapPhysicalMemory` address). On iOS without the extended-virtual-addressing entitlement a single 6 GB reservation succeeds and 7 GB fails, but Metal and the rest of the process need address space too; 5 GB is the total proven on a phone (D2R).
|
||||
- **`MapPhysicalMemory` faults on part of a page** → round to 16 KB host pages.
|
||||
- **Some textures are re-uploaded every frame (several MB per frame)** → the dirty-page tracker registers each memory pool rounded to 16 KB host pages, but NVN pools are only 4 KB aligned, so neighbouring pools share their first / last host page and one address can lie in several registered ranges. Taking the first range that contains an address judged textures in the second pool untracked (always dirty), released only one range on a write fault, and could unregister the neighbour when a pool was destroyed. `nvn_dirty.c` now uses the range that holds the whole span, releases every range covering a faulting page, and removes a range by the pool's exact start (`t_dirty_slots` covers it).
|
||||
- **Write through a null pointer at +0x1c0 during startup** → `GetInfo(UserExceptionContextAddress)` must return a real buffer.
|
||||
- **Heap allocations fail late in a busy scene** → the reported application pool (3285 MB, `APP_POOL`) and used memory decide the game's heap size; report Switch-accurate numbers.
|
||||
|
||||
@@ -49,7 +50,9 @@ Each entry: symptom → root cause → fix (where the fix lives). Check these fi
|
||||
|
||||
### Services and SDK
|
||||
|
||||
- **Abort inside `nn::sm::Initialize`** → an SDK path reached a system service the host doesn't replace → hook the public SDK entry point named in the backtrace (`HLE_FORCE` in the relevant `hle_*.c`).
|
||||
- **Abort inside `nn::sm::Initialize`** → an SDK path reached a system service the host doesn't replace → hook the public SDK entry point named in the backtrace (`HLE_FORCE` in the relevant `hle_*.c`). The log shows it as `SVC 0x1f` (ConnectToNamedPort, port name `sm:`) and `SVC 0x21` answered by placeholders, then `nn::diag` abort.
|
||||
- **A game update aborts at boot where the base game ran** → the update imports SDK functions the base game didn't. Diff the undefined dynamic symbols of the two `main` modules before the first run. Mario 1.2.1 added 28, among them `nn::nfp` (amiibo, initialized at boot), `pl::IsRunningOnOunce` (is this a Switch 2), `ec::ShowShopApplicationInformationForUpgrade`, `account::PopOpenUsers` and `aoc::CountAddOnContent`. Read the call sites before choosing answers: Mario only reads the count from `PopOpenUsers` and falls back to `TryOpenPreselectedUser` on 0, and only counts add-on content when running on a Switch 2. The host answers as a Switch 1 with no amiibo and no add-on content (`hle_sdk16.c`, `hle_account.c`, `hle_misc.c`).
|
||||
- **Code inherited from another title calls fixed addresses** → harmless on the build it was written for, silent memory corruption on any other. d2r-mobile's 3D render scale read a renderer pointer at `main+0x31e0848` and called `main+0x8582f0`; in Mario v0 that is the middle of a static initializer and only did nothing because the pointer happened to read 0. Such code checks `module_t.build_id` now (written into `fixups.tsv` by the rewriter) and stays off on other builds.
|
||||
- **Device / subsystem setup returns quietly and the next stage never starts** → a version or capability check failed (NVN API version, a feature query) → compare what the game queries with what the host answers.
|
||||
- **Font engine runs on garbage** → the system shared font must look like BFTTF: the SDK calls `ScalableFontEngineHelper::Decode(ptr, size)`, which XOR-decodes in place unless the first word is the "already decoded" marker (LE `0x18029a7f`) → serve `marker + bswap(size ^ 0x49621806) + TTF` (`hle_sdk16.c`).
|
||||
- **File-not-found noise** → games probe many optional paths (debug parameters, other-language voice files); with `DM_FS=1` these show as `fs … 不存在`. Only a missing file followed by a failure matters.
|
||||
@@ -60,7 +63,8 @@ Each entry: symptom → root cause → fix (where the fix lives). Check these fi
|
||||
- Fix: a registered code cut gives every resource heap +16 KB (`CUTS` in `dm/tools/patch_nso.py`, keyed by the module build ID, original bytes verified). Per-title code patches go there, with the reason.
|
||||
- Read the probe log carefully: `tryAlloc` probes show *requests*, not successes, so a running total that already exceeds the heap means an earlier request failed.
|
||||
|
||||
- **`FATAL NVN 命令内存：游戏池已空…`** → our command records are larger than real NVN commands, so command buffers the game sizes for hardware overflow; when the game's out-of-memory callback declines to add memory, the host lends a segment. Many small command buffers each overflowing once per frame exhausted a fixed pool of 64 × 1 MB. The pool now grows on demand (256 KB segments, reused after 8 frames, abort only beyond 1 GB; `hostseg_take` in `hle_nvn.c`).
|
||||
- **`FATAL NVN 命令内存：游戏池已空…`** → our command records are larger than real NVN commands, so command buffers the game sizes for hardware overflow; when the game's out-of-memory callback declines to add memory, the host lends a segment. Many small command buffers each overflowing once per frame exhausted a fixed pool of 64 × 1 MB. The pool now grows on demand (256 KB segments, abort only beyond 1 GB; `hostseg_take` in `hle_nvn.c`). A segment is released only when the game re-grants memory overlapping the guest bytes of the recording that jumped into it (`hostseg_release` in `AddCommandMemory`): the game records setup blocks once and keeps `CallCommands`-ing them for many frames, so recycling segments after a fixed number of frames would overwrite live recordings.
|
||||
- **`GetCommandMemoryUsed` must count everything the recording used**, including the parts that went into host segments (plus 16 bytes for the jump). Mario marks a recorded block as empty when the count is 0 and then skips calling it (see the next entry for how that happened).
|
||||
- **Frames stop, no crash, the main thread sits in `nvnSyncWait(sync, -1)`, and the watchdog's wait-state line shows the main fence one short (`已提交 N / 已完成 N-1`)** → a GPU command buffer never completes. Check GPU load from outside: `ioreg -r -d 1 -c IOAccelerator | grep 'Device Utilization'` at 100 % means a shader is looping forever. Find the shaders first used in the last frames (`DM_NVN_PASSDUMP`, compare the `K` keys) and look at their `.metal` for an empty `do {} while (true);` with no `return` (see [04](04-graphics.md) section 1: the compute program start).
|
||||
|
||||
- **Menus and dialogue boxes draw but contain no text; images and pictogram digits are fine** → the game rasterizes glyphs on the CPU into a *linear* texture in a CPU-visible pool (Mario: 2048×8192 R8, flags IMAGE|LINEAR). A private Metal texture never sees those writes. Linear textures in CPU-visible pools are now created on the pool buffer (`tex_linear` in `nvn_mtl.m`). `DM_NVN_TEXCPU=1` lists non-packaged textures in CPU-visible pools.
|
||||
@@ -75,6 +79,7 @@ Each entry: symptom → root cause → fix (where the fix lives). Check these fi
|
||||
- **The CPU processor drops the final mix.** `dsp::SendCommandBuffer`, `dsp::Wait` and `dsp::Signal` are stubs in the application build, and DeviceSink (command 0x13) is a no-op in `CommandListProcessor::Process`. Take the output after `Process` from the mix buffers the DeviceSink command names (layout in `hle_audren.c`).
|
||||
- **Generation needs the renderer's lock.** `AudioRenderSystem::Update` holds the SDK mutex at `AudioRenderSystem + 0x1f0`; `GenerateCommand` does not take it, so the host does (the auto-mode system thread does the same in `SendCommandToDsp`).
|
||||
- **At least one rendered frame between two updates.** An effect's "needs init" parameter state is set only in the update where the game flags it new; the next update copies the game's value over it. Mario updates twice in a row at start-up; with a real audio device the first frame came after both, and the light limiter later crashed on a null state pointer (`ApplyLightLimiterEffect`, `sdk+0x228494`). The `AudioRenderSystem::Update` hook renders a catch-up frame first when nothing was rendered since the last update. `DM_AUDREN_TRACE=1` logs every update with the number of frames rendered since the previous one.
|
||||
- **Crackling on iOS only** → RemoteIO asks for as many frames per callback as the audio session's I/O buffer duration, about 21 ms by default (1024 frames at 48 kHz), while the renderer keeps 4 blocks × 5 ms = 20 ms in flight. Every callback came up 1.3 ms short and was padded with zeros, about 47 gaps per second. Reproduced on the Mac by forcing 1024-frame callbacks (1999 underruns in 2000 callbacks). Fix: `AVAudioSession` `setPreferredIOBufferDuration:0.005` (and 48 kHz) before the game opens audio and again on return to the foreground (`front_ios_audio_session` in `front_ios.m`); the device then calls back with 240 frames and nothing underruns. Log the callback sizes and the queued blocks at each callback when checking a new device.
|
||||
- **Checking audio without listening**: `DM_AUDIO_DUMP=file` writes the PCM the host plays (16-bit, 48 kHz, stereo). RMS per second, clipping share, spectral flatness (music ≈ 0.1–0.5, white noise ≈ 0.85) and the strongest FFT peaks (they should land on musical notes) tell real music from noise or silence.
|
||||
|
||||
### NVN object sizes (memory corruption)
|
||||
@@ -86,6 +91,10 @@ The game allocates NVN objects with the sizes of *its* SDK's headers, which shra
|
||||
|
||||
Check every host object against the new SDK: `_Static_assert(sizeof(host) <= N)` next to the struct, and when a crash returns to a nonsense address, look at the 0x100 bytes below `sp` in the crash report for a zeroed frame.
|
||||
|
||||
The game may also write into an NVN object directly, without an NVN call, through code inlined from the SDK headers. Host records stored in place must leave those bytes alone:
|
||||
|
||||
- **`NVNcommandBuffer` +0x10**: Mario writes the byte 1 there (`main+0x4da5c`, `main+0xb8dec`). In the host record it was the low byte of `end`. Each write shrank the game's 512-byte material setup blocks, so their recordings overflowed into host segments; `GetCommandMemoryUsed` (then `cur − begin`) returned 0, the game marked the setups empty and skipped calling them, and the draws ran with the previous draw's vertex buffers: out-of-range vertices, NaN, a black band and flicker. It was seen on the iPhone; Mac runs of the same scene looked clean. `cmd_t` now reserves the byte (`nvn_flag10` in `nvn_obj.h`). To find such writes: put the field you suspect behind a guard value and log when it changes outside the host's own code, or force overflows on the Mac by shrinking the command memory the game gets.
|
||||
|
||||
### Game state
|
||||
|
||||
- **"Stuck" at a screen with one draw per frame** → usually not stuck: most draws are in recorded command buffers replayed later; dump the passes ([04](04-graphics.md)) before assuming the game logic is blocked.
|
||||
@@ -6,7 +6,7 @@ The graphics layer records NVN commands into its own command format (`dm/gfx/nvn
|
||||
|
||||
Do not stop at the obvious archive. In Mario the main `bfsha` held 4,888 of the 12,811 shaders the game actually uses.
|
||||
|
||||
`dm/gfx/tools/shader_split.py <RomFS> <ExeFS main> <pkg dir> <map out> <facts out>` scans **every** RomFS file (zstd-decompressed, SARC expanded recursively) and `main`, and recognizes:
|
||||
`dm/gfx/tools/shader_split.py <RomFS> <ExeFS main> <pkg dir> <map out> <facts out> [<pairs out>]` scans **every** RomFS file (zstd-decompressed, SARC expanded recursively) and `main`, and recognizes the containers below. The pairs table lists the (vertex key, pixel key) of every BNSH variation (section 6). For a base game plus an update, give RomFS and `main` as comma-separated lists, base first: keys are content hashes, so the output is the union of both versions and one shader build serves both (Mario 1.2.1 recompiled 3,451 of its pixel shaders; v0 rows keep their package indices).
|
||||
|
||||
- **BNSH** (`BNSH\0\0\0\0` + `grsc` block at +0x60): inside `bfsha`, particle effects (`esetb`, VFXB), UI layouts (`blarc`), and `main` itself. Variation table at +0x80 (count at +0x7c); binary program at variation +0x10; six stage pointers from program +0x08 (vs, hs, ds, gs, ps, cs); stage code descriptor: control ptr +0x08, code ptr +0x10, code size +0x18, control size +0x1c. All pointers relative to the BNSH start.
|
||||
- **BAHS** (agl `sharcb`, engine utility shaders): code region at header +0x1c (size +0x18). Binary entries in the table area are 16 bytes `{code offset, 0, control size (0x880/0x88c), signed offset from this entry to the control section}`. Control sections are shared between programs, so pair by entry, never by searching for code near a control.
|
||||
@@ -60,6 +60,8 @@ Consequence: any optimization that infers "which textures a draw reads" from bou
|
||||
- **New textures start as zero.** Metal leaves a new texture's content undefined; a freshly mapped Switch memory pool reads as 0, and feedback effects (such as the wave simulation above) never clear their state texture. Non-packaged colour textures are zero-filled from a shared zero buffer when created (`tex_zero_fill`, batched with the uploads).
|
||||
- **Scissored colour clears**: NVN `ClearColor` honours the scissor. Clears are normally folded into the next pass's load action, which only works for a full-target clear; Mario clears sub-rectangles of small targets to zero (pyramid levels packed into 480×272, 256×256 and 64×36 targets). A zero clear outside a pass with a partial scissor is now a copy from the shared zero buffer into that rectangle of the attachment's level / slice (`clear_rect_zero`). `DM_NVN_CLEARLOG=1` logs clears that are still skipped (mid-pass, partial channel mask or non-zero colour with a partial scissor).
|
||||
- **Inline uniform updates are snapshotted at encode time** (`mtl_sub_update`): the data is copied into a host-owned ring instead of being read from command memory when the GPU runs.
|
||||
- **Every pixel shader input must be a vertex shader output.** Maxwell attributes are untyped slots: a pixel shader that reads a slot the vertex shader never wrote gets a default. Metal refuses to build the pipeline (`Fragment input(s) user(locnN) mismatching vertex shader output type(s) or not written by vertex shader`), and the draw is skipped. Translating one shader at a time cannot know what the partner reads, so `shader_split.py` writes the pairs of each BNSH variation (`shader_pairs.tsv`) and `dmsh_build.py` gives each vertex shader the outputs its partners read but it doesn't write, as `float4` (0, 0, 0, 1) (`vs_pad`; vertex shaders with partners are finished after the pixel shaders, and the partner list is part of the shader's cache rule). In Mario 331 of 9,043 pairs failed; the opening course used none of them, later levels did (about 11,500 failed builds in one 11-minute session). Check all pairs offline after a build (recipe in [05](05-debugging.md)). The host caches failed pipelines and reports each once with its vertex / fragment keys (`【红】建渲染管线失败（顶点 … 片元 …）`); before that, every draw retried the failing compile.
|
||||
- **iOS math mode**: the iOS libraries are compiled with `-fmetal-math-mode=relaxed` (`dm/ios/tools/ios_metal.py`). The translated shaders reproduce Maxwell's NaN behaviour with `isnan` selects (8,337 of 12,809 use them), and the default fast mode allows the GPU compiler to assume there is no NaN. Relaxed keeps fast math's reassociation but not that assumption.
|
||||
|
||||
## 7. Diagnostic switches (host)
|
||||
|
||||
|
||||
@@ -87,6 +87,19 @@ Instruments' Metal Resource Events didn't record this process from the command l
|
||||
|
||||
See [04](04-graphics.md) section 7. Notes: pass dumps are large and may contain non-UTF-8 bytes; filter with `LC_ALL=C grep -a "\[pass\] F600 " | grep -av " o cmd"`. A Python one-liner with PIL (the dm venv has it) gives mean colour and non-black share of each dumped PNG, which is quicker than opening 20 images.
|
||||
|
||||
## On-device diagnostics (iOS)
|
||||
|
||||
The phone runs an unsigned IPA the user sideloads; nothing is attached to a debugger. The loop is: build an IPA with switches, the user plays and sends `Documents/log.txt` (the previous launch is `log.prev.txt`), read it, repeat.
|
||||
|
||||
- **Switches**: `DM_ENV_FILE=<file> dm/ios/tools/build_ipa.sh` bakes `KEY=VALUE` lines into the app (`dm/env.txt`); `Documents/env.txt` overrides them without a rebuild. Every applied switch is logged at boot (`boot: 环境 KEY=VALUE（来源文件）`). The app's working directory is `Documents`, so relative output paths (`DM_NVN_SHOT=3600:shot_%d.png`, `DM_NVN_TEXDUMP`) land where the Files app can reach them.
|
||||
- **Which build ran**: the first lines name the modules (`main: … patch_off …`), the shader map size (`映射表 N 条`) and, for builds that have it, the build-ID gate line. Compare them with the build you sent before reading further; users sometimes send an old log. The save path (`fs 挂载 save:`) shows the data folder, which changes when a sideloading tool gives a reinstall a new one.
|
||||
- **Reading the periodic lines**: `[长帧]` gives each long frame's submit / GPU / pipeline-compile / upload split; a burst of `新建管线` in long frames after an install is first-use compilation (see "Prewarm lists" in the README). In `显示节拍 至 N：…`, `空节拍` (refreshes with no new frame) and `合并丢弃旧帧` are totals since launch, not per window. `上屏 至帧 N` windows give dropped frames, p99 frame time and the physical footprint; `内存 至帧 N` gives resident and Metal-allocated memory. Memory that steps up at level loads and stays flat in between is not a leak.
|
||||
- **Screen recordings** from the user are useful for timing and visual symptoms; extract frames with `ffmpeg -ss <t> -i rec.mov -frames:v 1 out.png`.
|
||||
|
||||
## Offline pipeline link check
|
||||
|
||||
Pipeline link errors depend only on the two libraries, so all vertex / pixel pairs can be checked on the Mac after a shader build, without the game: for each line of `shader_pairs.tsv`, load `<vs>_b.metallib` and `<ps>_b.metallib` from `b-<V>/`, build a vertex descriptor from the vertex function's `vertexAttributes` (format from each attribute's type), set colour attachment formats from the `[[color(n)]]` outputs in the pixel shader's `_b.metal` (uint / int / float formats to match), add a depth format when it writes `[[depth(...)]]`, and call `newRenderPipelineStateWithDescriptor:`. Count failures by message. Mario: 8,712 linked / 331 failed before the vertex output padding, 9,043 / 0 after, and 14,603 / 0 for the v0 + 1.2.1 tables. Pipelines built this way have the default blend state and sample count, so this checks linking only.
|
||||
|
||||
## Intermittent rendering glitches
|
||||
|
||||
A glitch that comes and goes between frames or runs needs a detector before bisection, or "no glitch in these 16 frames" proves nothing. What worked for Mario's fountain panels:
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
These pages are for an agent asked to analyse a Switch NSP and port it with the native-execution approach of this repository and [d2r-mobile](../../../d2r-mobile): the game's own aarch64 code runs natively on Apple hardware, a rewriter patches only the instructions Apple's kernel cannot run, the host replaces Horizon system services, and NVN is reimplemented on Metal.
|
||||
|
||||
Everything here was learned on two titles: Diablo II: Resurrected (SDK ~12, NVN 53, the original target of d2r-mobile) and Super Mario Bros. Wonder (SDK 16.2.1, NVN 55, this repository). Where the two differed, both are given; a third title will differ again, so treat every number below as "measure it, don't assume it".
|
||||
Everything here was learned on two titles: Diablo II: Resurrected (SDK ~12, NVN 53, the original target of d2r-mobile) and Super Mario Bros. Wonder (SDK 16.2.0, NVN 55, this repository; its update 1.2.1 uses SDK 16.2.12). Where the two differed, both are given; a third title will differ again, so treat every number below as "measure it, don't assume it".
|
||||
|
||||
| Page | Covers |
|
||||
|---|---|
|
||||
@@ -10,7 +10,7 @@ Everything here was learned on two titles: Diablo II: Resurrected (SDK ~12, NVN
|
||||
| [02-survey.md](02-survey.md) | Static survey of a new title before touching the host: SDK version, imports vs. host hooks, SVCs, instructions to rewrite, NVN surface, data containers, Ghidra / REA setup |
|
||||
| [03-bringup.md](03-bringup.md) | Getting the host to boot the game: the run loop, how to read the log, and a catalogue of every failure pattern met so far with its root cause and fix |
|
||||
| [04-graphics.md](04-graphics.md) | NVN on Metal for a new title: object sizes, shader containers and keys, binding layouts, bindless textures, depth mode, index sizes, swizzle, transient render targets, the diagnostic switches |
|
||||
| [05-debugging.md](05-debugging.md) | Recipes: crash reports, lldb scripts, Ghidra headless decompilation, objdump, memory growth, render-pass and texture dumps |
|
||||
| [05-debugging.md](05-debugging.md) | Recipes: crash reports, lldb scripts, Ghidra headless decompilation, objdump, memory growth, render-pass and texture dumps, on-device diagnostics, offline pipeline link checks |
|
||||
|
||||
## Ground rules
|
||||
|
||||
@@ -30,3 +30,5 @@ Everything here was learned on two titles: Diablo II: Resurrected (SDK ~12, NVN
|
||||
6. Extract and translate the shaders, find the binding layout by census, and get the first frame — [04](04-graphics.md).
|
||||
7. Check memory over a minute of running (`footprint`, the host's memory report line); fix growth before moving on.
|
||||
8. Input, audio, then the iOS bundle (d2r-mobile's `dm/ios`, RomFS inside the app).
|
||||
9. Test on a phone: an unsigned IPA the user sideloads, switches through `env.txt`, results through `Documents/log.txt` ([05](05-debugging.md)). Several bugs only showed there (the command-buffer byte and the audio buffer size in [03](03-bringup.md), pipeline link failures in later levels in [04](04-graphics.md)).
|
||||
10. For a game update: extract it over the base game, diff the imports, carry the per-build addresses over, and rebuild the shader tables for both versions ([01](01-extract.md)).
|
||||
Reference in new issue
Block a user