nxnative-easy/docs/agents/03-bringup.md

03 — Host bring-up

The loop

DM_RULESET=native python3 dm/tools/patch_nso.py $W/data/v0/ExeFS/main $W/build/patched_v0 main   # and sdk
make -C dm/host dmhost
~/<title>-work/run/boot.sh <log name> [seconds] [VAR=value ...]

boot.sh runs dmhost <patched dir> with DM_ROMFS, DM_SAVE, DM_TEMP, DM_MSL_DIR set, under a perl alarm (a run that hits the time limit ends with CRASH sig=14 — that is the alarm, not a crash). Logs go to run/<name>.log. Logs may contain non-UTF-8 bytes: use grep -a / LC_ALL=C when filtering.

Reading the log

Marker	Meaning
>>> nnosInitialize, >>> nnMain	Boot stages reached
【红】…	Something unimplemented or wrong, printed once per kind. Collect them with grep -a "【红】" log | sort | uniq -c after every run
UNIMPL <name>	A forced NVN / SDK stub was called
[nvn] 帧 N：… | name=count …	Per-frame NVN call counts (calls made on the frame thread; command buffers recorded elsewhere replay later)
[长帧]	A long frame, with submit / GPU / pipeline / upload breakdown
内存 至帧 N：常驻 … 显卡已分配 …	Memory report (RSS, Metal allocated size). Printed when memory moves; steady growth here is a leak. A watchdog exits at 24 GB
CRASH sig=… pc=… lr=… + registers + bt: + 栈扫描 + thread dump	Crash report: see 05
线程 0x10xx 「name」… 等:<state> addr=…	Thread dump: what each guest thread waits on (地址 = address arbiter, 锁 = mutex with holder, 条件变量, 同步对象, 运行/未在等 = running)

Useful switches: DM_FS=1 (file opens, and not-found probes), DM_SVC=1 (memory SVCs and other syscalls), DM_THR=1 (thread SVCs), DM_WATCH=<s> (dump all threads after s seconds), DM_WATCH_STALL=<s> (dump when the frame loop stops progressing).

Failure catalogue

Each entry: symptom → root cause → fix (where the fix lives). Check these first; most new-title failures are one of them in a new form.

Rewriter

• Job system hangs; a queue claims thousands of tasks but every slot is empty → in an ldxr … cmp … b.ne out … stxr … cbnz window, the rewritten stub emitted its exits in the wrong order, so successful stores took the failure path → emit the fall-through exit first (patch_nso.py). D2R had no such windows; Mario had 82.
• SIGILL / trap on a system register read → newer SDK reads a register macOS traps (CTR_EL0 in SDK 16's FlushDataCache) → rewrite to a constant.

Memory and address space

• First AllocateAddressRegion fails with 0x1803 → SDK 16's AddressRegionManager takes 0x3fe00000 at the start of the alias region as a node area → the alias region must be larger than 1 GB.
• Large resources silently skipped, later a null deref where the resource is used → the game reserves "virtual address memory" (Mario: 1.75 GB in HeapDef.byml) and maps it into the alias region with MapPhysicalMemory on demand; the alias region left after the node area was too small. Also visible as SetHeapSize asking for much less than the pool. → repartition the guest block (hle_svc.c mem_init): Mario uses alias 2.875 GB, stacks 256 MB, heap 1.625 GB (it asks for 1.47 GB), 5 GB in total. Measure before sizing (DM_SVC=1: the SetHeapSize request and the highest MapPhysicalMemory address). On iOS without the extended-virtual-addressing entitlement a single 6 GB reservation succeeds and 7 GB fails, but Metal and the rest of the process need address space too; 5 GB is the total proven on a phone (D2R).
• MapPhysicalMemory faults on part of a page → round to 16 KB host pages.
• Some textures are re-uploaded every frame (several MB per frame) → the dirty-page tracker registers each memory pool rounded to 16 KB host pages, but NVN pools are only 4 KB aligned, so neighbouring pools share their first / last host page and one address can lie in several registered ranges. Taking the first range that contains an address judged textures in the second pool untracked (always dirty), released only one range on a write fault, and could unregister the neighbour when a pool was destroyed. nvn_dirty.c now uses the range that holds the whole span, releases every range covering a faulting page, and removes a range by the pool's exact start (t_dirty_slots covers it).
• Write through a null pointer at +0x1c0 during startup → GetInfo(UserExceptionContextAddress) must return a real buffer.
• Heap allocations fail late in a busy scene → the reported application pool (3285 MB, APP_POOL) and used memory decide the game's heap size; report Switch-accurate numbers.

Threads

• Per-core job queues never drained → a SetThreadCoreMask that pins a thread to other cores (ideal core −3 = keep) must move the ideal core into the new mask, as Horizon does, so GetCurrentProcessorNumber reports a core the thread can run on.
• Crash indexing a per-core table with a huge offset (index −2) → threads created with ideal core −2 (use process default) were recorded as core −2 and GetCurrentProcessorNumber returned −2 → resolve −2 to the process default core (0) and never return a negative core (hle_thread.c). Seen in Mario's font engine (main+0x107dbc0).

Services and SDK

• Abort inside nn::sm::Initialize → an SDK path reached a system service the host doesn't replace → hook the public SDK entry point named in the backtrace (HLE_FORCE in the relevant hle_*.c). The log shows it as SVC 0x1f (ConnectToNamedPort, port name sm:) and SVC 0x21 answered by placeholders, then nn::diag abort.

• A game update aborts at boot where the base game ran → the update imports SDK functions the base game didn't. Diff the undefined dynamic symbols of the two main modules before the first run. Mario 1.2.1 added 28, among them nn::nfp (amiibo, initialized at boot), pl::IsRunningOnOunce (is this a Switch 2), ec::ShowShopApplicationInformationForUpgrade, account::PopOpenUsers and aoc::CountAddOnContent. Read the call sites before choosing answers: Mario only reads the count from PopOpenUsers and falls back to TryOpenPreselectedUser on 0, and only counts add-on content when running on a Switch 2. The host answers as a Switch 1 with no amiibo and no add-on content (hle_sdk16.c, hle_account.c, hle_misc.c).

• Code inherited from another title calls fixed addresses → harmless on the build it was written for, silent memory corruption on any other. d2r-mobile's 3D render scale read a renderer pointer at main+0x31e0848 and called main+0x8582f0; in Mario v0 that is the middle of a static initializer and only did nothing because the pointer happened to read 0. Such code checks module_t.build_id now (written into fixups.tsv by the rewriter) and stays off on other builds.

• Device / subsystem setup returns quietly and the next stage never starts → a version or capability check failed (NVN API version, a feature query) → compare what the game queries with what the host answers.

• Font engine runs on garbage → the system shared font must look like BFTTF: the SDK calls ScalableFontEngineHelper::Decode(ptr, size), which XOR-decodes in place unless the first word is the "already decoded" marker (LE 0x18029a7f) → serve marker + bswap(size ^ 0x49621806) + TTF (hle_sdk16.c).

• File-not-found noise → games probe many optional paths (debug parameters, other-language voice files); with DM_FS=1 these show as fs … 不存在. Only a missing file followed by a failure matters.

• A buffer is created with a null pool right after a scene change → NVN virtual memory pools (flag 0x200) have no storage until nvnMemoryPoolMapVirtual maps physical pool pages in; if the host returns failure, the game treats the pool as unusable. Mario maps its single 128 MB virtual pool onto one physical pool in one request, so the host aliases it (hle_nvn.c). True sparse mappings are still unimplemented.

• A null-pool buffer even with virtual pools working → the game's own heap ran out. Trace it with probes (05): the gsys GPU allocator at main+0x4290c0 and the heap's tryAlloc (vtable slot +0x30) show which heap, its size and every allocation into it; the !xN heap walk shows the free and used blocks at the moment of failure.

  • Mario: each resource gets its own sead::ExpHeap sized resource-size-table value + 0x1008 (main+0x1e603c → ExpHeap::create main+0x4aac38). The table values are Nintendo's measured peaks with almost no margin (Yoshi's animation file uses 99 % of its budget). Model/NpcCommonerA.bfres in the opening course came up about 0x70 bytes short: its last GPU chunk (0x9000 bytes, 4 KB aligned) fit, but the chunk's 0x598-byte descriptor did not. Every allocation size before it is fixed by game code and data, so the difference from hardware is still unknown.
  • Fix: a registered code cut gives every resource heap +16 KB (CUTS in dm/tools/patch_nso.py, keyed by the module build ID, original bytes verified). Per-title code patches go there, with the reason.
  • Read the probe log carefully: tryAlloc probes show requests, not successes, so a running total that already exceeds the heap means an earlier request failed.

• FATAL NVN 命令内存：游戏池已空… → our command records are larger than real NVN commands, so command buffers the game sizes for hardware overflow; when the game's out-of-memory callback declines to add memory, the host lends a segment. Many small command buffers each overflowing once per frame exhausted a fixed pool of 64 × 1 MB. The pool now grows on demand (256 KB segments, abort only beyond 1 GB; hostseg_take in hle_nvn.c). A segment is released only when the game re-grants memory overlapping the guest bytes of the recording that jumped into it (hostseg_release in AddCommandMemory): the game records setup blocks once and keeps CallCommands-ing them for many frames, so recycling segments after a fixed number of frames would overwrite live recordings.

• GetCommandMemoryUsed must count everything the recording used, including the parts that went into host segments (plus 16 bytes for the jump). Mario marks a recorded block as empty when the count is 0 and then skips calling it (see the next entry for how that happened).

• Frames stop, no crash, the main thread sits in nvnSyncWait(sync, -1), and the watchdog's wait-state line shows the main fence one short (已提交 N / 已完成 N-1) → a GPU command buffer never completes. Check GPU load from outside: ioreg -r -d 1 -c IOAccelerator | grep 'Device Utilization' at 100 % means a shader is looping forever. Find the shaders first used in the last frames (DM_NVN_PASSDUMP, compare the K keys) and look at their .metal for an empty do {} while (true); with no return (see 04 section 1: the compute program start).

• Menus and dialogue boxes draw but contain no text; images and pictogram digits are fine → the game rasterizes glyphs on the CPU into a linear texture in a CPU-visible pool (Mario: 2048×8192 R8, flags IMAGE|LINEAR). A private Metal texture never sees those writes. Linear textures in CPU-visible pools are now created on the pool buffer (tex_linear in nvn_mtl.m). DM_NVN_TEXCPU=1 lists non-packaged textures in CPU-visible pools.

• Characters (skinned models) invisible or stretched into streaks while the level renders → integer vertex inputs (bone indices) translated as float; see 04 section 6. To find the draws: log model file ranges with DM_PROBE=main:480f0 (x1 = file data, x2 = size), match the vertex/index buffer addresses in DM_NVN_PASSDUMP_ALL against them, then DM_NVN_HILITE their fragment shader.

• Sky black → CPU-written linear colour ramps (see the linear-texture entry above).

• Controller-button icons in text show as ? → those glyphs come from the system shared font's Nintendo-extension set (private-use code points; Mario's GameMsg/Key.msbt uses U+E0E0–E0F0 for A/B/X/Y/L/R/ZL/ZR/SL/SR/D-pad/+/−, U+E0AB–E0AE for D-pad directions, U+E090–E097 for stick arrows, U+E100/E101 for sticks). dm/tools/make_ext_font.py generates a stand-in nintendo_ext.ttf with those icons; the host serves it for shared-font type 5 (DM_FONT_DIR or DM_SHARED_FONT_EXT) and a desktop font for the other types. To find which code points a title uses, scan the TXT2 sections of its MSBT files for U+E000–F8FF.

• Shared fonts on iOS: the macOS stand-in path (/System/Library/Fonts/Supplemental/Arial Unicode.ttf) does not exist on a phone, and GetSharedFontLoadState would report "not loaded". The iOS app ships dm/font/standard.ttf and dm/font/nintendo_ext.ttf (copied / generated by build_ipa.sh) and sets DM_FONT_DIR.

Audio

• Check whether the SDK carries the renderer. List the sdk module's exports for nn::audio::server:: and nn::audio::dsp::CommandListProcessor. If they are there (Mario, SDK 16), do not write a mixer: switch AudioRendererParameter to rendering device = CPU (+0x2c = 1) and execution mode = manual (+0x30 = 1) in GetAudioRendererWorkBufferSize and both OpenAudioRenderer overloads, call the SDK originals (guest_sym returns the module's own code even for a forced symbol), and drive ExecuteAudioRendererRendering from a guest thread (dm_guest_thread_spawn). The open then goes through CreateAudioRendererManagerByDfc; no IPC.
• The CPU processor drops the final mix. dsp::SendCommandBuffer, dsp::Wait and dsp::Signal are stubs in the application build, and DeviceSink (command 0x13) is a no-op in CommandListProcessor::Process. Take the output after Process from the mix buffers the DeviceSink command names (layout in hle_audren.c).
• Generation needs the renderer's lock. AudioRenderSystem::Update holds the SDK mutex at AudioRenderSystem + 0x1f0; GenerateCommand does not take it, so the host does (the auto-mode system thread does the same in SendCommandToDsp).
• At least one rendered frame between two updates. An effect's "needs init" parameter state is set only in the update where the game flags it new; the next update copies the game's value over it. Mario updates twice in a row at start-up; with a real audio device the first frame came after both, and the light limiter later crashed on a null state pointer (ApplyLightLimiterEffect, sdk+0x228494). The AudioRenderSystem::Update hook renders a catch-up frame first when nothing was rendered since the last update. DM_AUDREN_TRACE=1 logs every update with the number of frames rendered since the previous one.
• Crackling on iOS only → RemoteIO asks for as many frames per callback as the audio session's I/O buffer duration, about 21 ms by default (1024 frames at 48 kHz), while the renderer keeps 4 blocks × 5 ms = 20 ms in flight. Every callback came up 1.3 ms short and was padded with zeros, about 47 gaps per second. Reproduced on the Mac by forcing 1024-frame callbacks (1999 underruns in 2000 callbacks). Fix: AVAudioSession setPreferredIOBufferDuration:0.005 (and 48 kHz) before the game opens audio and again on return to the foreground (front_ios_audio_session in front_ios.m); the device then calls back with 240 frames and nothing underruns. Log the callback sizes and the queued blocks at each callback when checking a new device.
• Checking audio without listening: DM_AUDIO_DUMP=file writes the PCM the host plays (16-bit, 48 kHz, stereo). RMS per second, clipping share, spectral flatness (music ≈ 0.1–0.5, white noise ≈ 0.85) and the strongest FFT peaks (they should land on musical notes) tell real music from noise or silence.

NVN object sizes (memory corruption)

The game allocates NVN objects with the sizes of its SDK's headers, which shrank in newer SDKs. A host record larger than the game's allocation overwrites neighbouring memory:

• NVNtexture is 0x80 bytes in SDK 16 (heap corruption: the old 0x88 record overwrote the next heap block's links).
• NVNtextureView is 16 bytes in SDK 16 (stack smash: a 16-byte local next to the saved frame pointer; the function returned to 0x100000000).

Check every host object against the new SDK: _Static_assert(sizeof(host) <= N) next to the struct, and when a crash returns to a nonsense address, look at the 0x100 bytes below sp in the crash report for a zeroed frame.

The game may also write into an NVN object directly, without an NVN call, through code inlined from the SDK headers. Host records stored in place must leave those bytes alone:

• NVNcommandBuffer +0x10: Mario writes the byte 1 there (main+0x4da5c, main+0xb8dec). In the host record it was the low byte of end. Each write shrank the game's 512-byte material setup blocks, so their recordings overflowed into host segments; GetCommandMemoryUsed (then cur − begin) returned 0, the game marked the setups empty and skipped calling them, and the draws ran with the previous draw's vertex buffers: out-of-range vertices, NaN, a black band and flicker. It was seen on the iPhone; Mac runs of the same scene looked clean. cmd_t now reserves the byte (nvn_flag10 in nvn_obj.h). To find such writes: put the field you suspect behind a guard value and log when it changes outside the host's own code, or force overflows on the Mac by shrinking the command memory the game gets.

Game state

• "Stuck" at a screen with one draw per frame → usually not stuck: most draws are in recorded command buffers replayed later; dump the passes (04) before assuming the game logic is blocked.