nxnative-easy/docs/agents/05-debugging.md

05 — Debugging recipes

Reading a crash report

[dm] CRASH sig=11 addr=0 pc=main+0x373e04 lr=main+0x373dd8 sp=0x7066dd6370
x0=… x28=…
[dm] bt: main+0x373dd8 main+0x2ea2ac …          frame-pointer backtrace
[dm] fp=…  sp 下方 0x100 字节: …                 the 32 words below sp (the frame that was just popped)
[dm] 栈扫描 sp=…: [-72]main+0x57a14 [+24]main+0x57618 …   every word near sp that resolves to a module or host symbol
[dm] 线程 0x1001 「主线程」…                       all guest threads, what they wait on, their backtraces

• pc and lr are module-relative (main+…, sdk+…) or host symbols. Module offsets are the same as in the decompressed NSO and the ELF (Ghidra adds 0x100000).
• Look at the registers before the code: a pointer that is a valid pointer plus garbage high bits, an index of 0xfffffffe, or pc == lr == 0x100000000 each tell a different story. a fault at 0x74a740e114 = 0x70a740c570 + 0xfffffffe·4 + 0x1bac was an index of −2 widened as unsigned; pc == lr on a nonsense value is a ret through an overwritten saved LR (look at the zeroed frame below sp).
• When the frame-pointer backtrace is empty, the stack scan's negative offsets (below sp) show the return addresses of the calls the crashing function made last.

Ghidra / objdump

# decompile the functions containing these module offsets
JAVA_HOME=/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home \
  /opt/homebrew/opt/ghidra/libexec/support/analyzeHeadless ~/<title>-work/ghidra/proj <proj> -process <module>.elf \
  -noanalysis -readOnly -scriptPath <repo>/dm/tools/ghidra -postScript Decompile.java 0x373e04,0x2ea2ac

# raw disassembly (works where Ghidra defined no function)
objdump -d --start-address=0x2ea1c0 --stop-address=0x2ea2b0 ~/<title>-work/elf/<module>.elf

Prefix an offset with @ (... Decompile.java @0x49740) to (re)create the function at exactly that address. Code reached only through vtables has no function, and Ghidra's analysis marked some nn::os imports (LockMutex) as non-returning, which cut many functions short right after the first lock. The script clears that flag and the call-return overrides on such calls before decompiling.

Map GOT slots to NVN names with the survey table (census/*_nvn_slots.tsv): a call through [x8, #0x3e8] after adrp x8, 0x35ad000 is GOT entry 0x35ad3e8.

Probes without a debugger (DM_PROBE)

DM_PROBE=main:4290c0,main:9712c (module:offset, comma-separated; host/probe.c, Mac only) replaces the instruction with brk, and on every hit prints the guest thread (T=), x0–x7 (and strings they point to), x19–x23, lr, x8 and a backtrace, then runs the original instruction from a trampoline. Put probes at function entries or where x16 is dead; the probed instruction must not be PC-relative (no adrp, adr, b, bl, literal ldr: probe the next instruction instead). Output from several threads interleaves; parse it with a regex rather than by line. Probes slow the game, which can shift heap addresses between runs.

Per-probe suffixes, in this order:

Suffix	Effect
@off, @*off, @xN+off, @*xN+off	Print 16 bytes at x0+off (or xN+off); * dereferences that address first
?xN=val, ?[xN+off]=val	Only report hits where the register (or the 64-bit word at xN+off) equals val (hex); others pass silently. E.g. main:4743c0?[x0+30]=781548 = only allocations into the heap whose size field is 0x781548
!xN	Treat xN as a sead::ExpHeap and print its free and used blocks (offset+size)

DM_PROBE_ALL=1 adds x9–x18; DM_PROBE_X2=<bytes> dumps the memory x2 points to.

Use it when lldb is unavailable: after macOS revokes the debugger authorization (DevToolsSecurity -status says developer mode is disabled), lldb launches hang waiting for a GUI approval prompt. sudo DevToolsSecurity -enable (run by the user) restores it.

lldb

The host uses SIGSEGV for dirty-page tracking, so lldb must pass it through without stopping. Address randomization is off under lldb, so guest addresses repeat across runs. Template (~/<title>-work/dbg/*.py, run with perl -e 'alarm 240; exec @ARGV' lldb -b -o "command script import x.py"):

import lldb
HOST = '/path/to/dm/host/dmhost'
def __lldb_init_module(dbg, _):
    dbg.SetAsync(False); ci = dbg.GetCommandInterpreter(); res = lldb.SBCommandReturnObject()
    ci.HandleCommand('settings set platform.plugin.darwin.ignored-exceptions EXC_BAD_ACCESS|EXC_BAD_INSTRUCTION', res)
    t = dbg.CreateTarget(HOST); t.BreakpointCreateByName('run_module_inits')
    li = lldb.SBLaunchInfo(['<patched dir>']); li.SetEnvironmentEntries(['DM_ROMFS=…', 'DM_SAVE=…', 'DM_TEMP=…'], True)
    p = t.Launch(li, lldb.SBError())
    for sig in ('SIGSEGV', 'SIGBUS', 'SIGUSR1', 'SIGUSR2', 'SIGPIPE'): ci.HandleCommand('process handle %s -n false -p true -s false' % sig, res)
    base = None
    while p.GetState() == lldb.eStateStopped:
        fr = p.GetSelectedThread().GetFrameAtIndex(0)
        if base is None and fr.GetFunctionName() == 'run_module_inits':
            base = t.EvaluateExpression('(unsigned long)g_mod[0].base').GetValueAsUnsigned()   # g_mod[0] = main
            t.BreakpointCreateByAddress(base + 0x2ea228)                                       # module offset
        elif base is not None and fr.GetPC() == base + 0x2ea228:
            print('x0=%#x' % fr.FindRegister('x0').GetValueAsUnsigned(), flush=True)          # flush: a timeout kill loses buffered output
        p.Continue()
    p.Kill()

Patterns that worked: print every call of one function with its arguments; stop when a return value is null and print the backtrace; trace the branch path through one function (one-shot breakpoints on every branch site). Hardware watchpoints on words touched by ldxr/stxr may stay silent.

Memory growth

1. The host's memory report line (内存 至帧 …) every few hundred frames; footprint -p <pid> from outside.
2. vmmap -summary <pid> twice a few seconds apart: which region grows. "owned unmapped (graphics)" = private Metal resources.
3. DM_NVN_ALLOCPROF=1: growth per replayed op type and per function (tex_mtl / pass begin / pipelines / argument buffers), command allocator sizes, and a log of new textures with serial, size, format and storage. A stream of new serials with the same sizes = objects re-created every frame.

Instruments' Metal Resource Events didn't record this process from the command line (xctrace record --launch, with --env for the variables); the in-host probes were faster.

Render-pass investigation

See 04 section 7. Notes: pass dumps are large and may contain non-UTF-8 bytes; filter with LC_ALL=C grep -a "\[pass\] F600 " | grep -av " o cmd". A Python one-liner with PIL (the dm venv has it) gives mean colour and non-black share of each dumped PNG, which is quicker than opening 20 images.

On-device diagnostics (iOS)

The phone runs an unsigned IPA the user sideloads; nothing is attached to a debugger. The loop is: build an IPA with switches, the user plays and sends Documents/log.txt (the previous launch is log.prev.txt), read it, repeat.

• Switches: DM_ENV_FILE=<file> dm/ios/tools/build_ipa.sh bakes KEY=VALUE lines into the app (dm/env.txt); Documents/env.txt overrides them without a rebuild. Every applied switch is logged at boot (boot: 环境 KEY=VALUE（来源文件）). The app's working directory is Documents, so relative output paths (DM_NVN_SHOT=3600:shot_%d.png, DM_NVN_TEXDUMP) land where the Files app can reach them.
• Which build ran: the first lines name the modules (main: … patch_off …), the shader map size (映射表 N 条) and, for builds that have it, the build-ID gate line. Compare them with the build you sent before reading further; users sometimes send an old log. The save path (fs 挂载 save:) shows the data folder, which changes when a sideloading tool gives a reinstall a new one.
• Reading the periodic lines: [长帧] gives each long frame's submit / GPU / pipeline-compile / upload split; a burst of 新建管线 in long frames after an install is first-use compilation (see "Prewarm lists" in the README). In 显示节拍 至 N：…, 空节拍 (refreshes with no new frame) and 合并丢弃旧帧 are totals since launch, not per window. 上屏 至帧 N windows give dropped frames, p99 frame time and the physical footprint; 内存 至帧 N gives resident and Metal-allocated memory. Memory that steps up at level loads and stays flat in between is not a leak.
• Screen recordings from the user are useful for timing and visual symptoms; extract frames with ffmpeg -ss <t> -i rec.mov -frames:v 1 out.png.

Offline pipeline link check

Pipeline link errors depend only on the two libraries, so all vertex / pixel pairs can be checked on the Mac after a shader build, without the game: for each line of shader_pairs.tsv, load <vs>_b.metallib and <ps>_b.metallib from b-<V>/, build a vertex descriptor from the vertex function's vertexAttributes (format from each attribute's type), set colour attachment formats from the [[color(n)]] outputs in the pixel shader's _b.metal (uint / int / float formats to match), add a depth format when it writes [[depth(...)]], and call newRenderPipelineStateWithDescriptor:. Count failures by message. Mario: 8,712 linked / 331 failed before the vertex output padding, 9,043 / 0 after, and 14,603 / 0 for the v0 + 1.2.1 tables. Pipelines built this way have the default blend state and sample count, so this checks linking only.

Intermittent rendering glitches

A glitch that comes and goes between frames or runs needs a detector before bisection, or "no glitch in these 16 frames" proves nothing. What worked for Mario's fountain panels:

1. Screenshot a dense frame range (DM_NVN_SHOT=$(seq -s, 4430 10 4590):x_%d.png) and score each frame automatically. The first metric (texture busyness) missed the flat translucent variant of the glitch; a metric tuned to its shape (long vertical straight edges in a sky region) separated glitch frames (≈1.0) from clean ones (≈0.3) reliably. Look at a few frames by eye before trusting a metric.
2. Run a control with nothing changed to measure how often the glitch appears per run.
3. Bisect with DM_NVN_SKIP_FS (up to 16 fragment shader keys per run) until one draw remains, then DM_NVN_HILITE it to see its geometry.
4. Dump that draw's inputs across frames (DM_NVN_CBDUMP + DM_NVN_CBDUMP_KEY). If every input is byte-identical between good and bad frames, the difference is on the GPU side: follow the textures it samples (DM_NVN_TEXINFO, DM_NVN_TEXRAW) to whatever writes them.
5. For a GPU-side simulation, trace its state on the GPU timeline (DM_NVN_T2T_TRACE), re-implement the shader step in numpy with the logged uniform updates (the Y update_ub lines print the first four floats), and compare frame by frame. Frames that disagree show which input the GPU really used; for the fountain, a coefficient of −1 instead of 1.81 explained every bad texel and led to the missing intra-encoder barrier.