nxnative-easy/docs/agents/05-debugging.md

05 — Debugging recipes

Reading a crash report

[dm] CRASH sig=11 addr=0 pc=main+0x373e04 lr=main+0x373dd8 sp=0x7066dd6370
x0=… x28=…
[dm] bt: main+0x373dd8 main+0x2ea2ac …          frame-pointer backtrace
[dm] fp=…  0x100 bytes below sp: …                 the 32 words below sp (the frame that was just popped)
[dm] stack scan sp=…: [-72]main+0x57a14 [+24]main+0x57618 …   every word near sp that resolves to a module or host symbol
[dm] thread 0x1001 'main thread' …                  all guest threads, what they wait on, their backtraces

• pc and lr are module-relative (main+…, sdk+…) or host symbols. Module offsets are the same as in the decompressed NSO and the ELF (Ghidra adds 0x100000).
• Look at the registers before the code: a pointer that is a valid pointer plus garbage high bits, an index of 0xfffffffe, or pc == lr == 0x100000000 each tell a different story. a fault at 0x74a740e114 = 0x70a740c570 + 0xfffffffe·4 + 0x1bac was an index of −2 widened as unsigned; pc == lr on a nonsense value is a ret through an overwritten saved LR (look at the zeroed frame below sp).
• The handler takes SIGSEGV, SIGBUS, SIGILL, SIGTRAP, SIGALRM (the run's time limit) and SIGABRT (abort(): an assertion or an uncaught exception in the host or a framework; before it was handled, Big Brain Academy: Brain vs. Brain died once with exit 134, nothing in the log and no report). A run that ends with exit 137 and no CRASH line was killed with SIGKILL: macOS does that after a trap in a system library (_os_unfair_lock_corruption_abort destroying a corrupted mutex, the movie decoder entry Input indices returned by a flush while a client holds them), and its report is in ~/Library/Logs/DiagnosticReports/dmhost-*.ips, a JSON document whose faultingThread names the thread whose frames to read.
• When the frame-pointer backtrace is empty, the stack scan's negative offsets (below sp) show the return addresses of the calls the crashing function made last.
• [dm] *xN: lines show the first six words each register points at (read safely, so a bad pointer is skipped). A word inside a module prints as main+…: the first word of an object is its vtable, which names its class, and comparing it with the class's own vtable shows whether the object was destroyed (a base-class vtable) or rebuilt.

Pipeline compile waits

Render-pipeline lookahead runs after the demanded pipeline is available and cached. It admits at most two predictions per scan and at most two scheduled or running predictions overall, so speculative attachment variants cannot create an unbounded compiler backlog ahead of real draws. Exact-key results are still recorded only when used; a full pending table may discard completed unused results, never running jobs. The independent startup warmer is not subject to this limit. A new pipeline can still stall on its intrinsic first-use compilation cost; compare pipeline-miss timings and live stacks with asset reads before attributing a hitch to loading. dm/gfx/tests/t_pso_pending.m exercises the queue and demand-before-prediction ordering with a gated fake compiler.

Ghidra / objdump

# decompile the functions containing these module offsets
JAVA_HOME=/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home \
  /opt/homebrew/opt/ghidra/libexec/support/analyzeHeadless $W/ghidra/proj <proj> -process <module>.elf \
  -noanalysis -readOnly -scriptPath <repo>/dm/tools/ghidra -postScript Decompile.java 0x373e04,0x2ea2ac

# raw disassembly (works where Ghidra defined no function)
objdump -d --start-address=0x2ea1c0 --stop-address=0x2ea2b0 $W/elf/<module>.elf

Prefix an offset with @ (... Decompile.java @0x49740) to (re)create the function at exactly that address. Code reached only through vtables has no function, and Ghidra's analysis marked some nn::os imports (LockMutex) as non-returning, which cut many functions short right after the first lock. The script clears that flag and the call-return overrides on such calls before decompiling.

Map GOT slots to NVN names with the survey table (census/*_nvn_slots.tsv): a call through [x8, #0x3e8] after adrp x8, 0x35ad000 is GOT entry 0x35ad3e8.

Probes without a debugger (DM_PROBE)

DM_PROBE=main:4290c0,main:9712c (module:offset, comma-separated; host/probe.c, Mac only) replaces the instruction with brk, and on every hit prints the guest thread (T=), x0–x7 (and strings they point to), x19–x23, lr, x8 and a backtrace, then runs the original instruction from a trampoline. Put probes at function entries or where x16 is dead; the probed instruction must not be PC-relative (no adrp, adr, b, bl, literal ldr: probe the next instruction instead). Each hit prints as one block even when several threads hit at once. Probes slow the game, which can shift heap addresses between runs.

Per-probe suffixes, in this order:

Suffix	Effect
@off, @*off, @xN+off, @*xN+off, @*xN+off+add	Print 16 bytes at x0+off (or xN+off); * dereferences that address first, and +add moves on that many bytes from the pointer read (@*x0+48+8 on an al::LiveActor is its position, al::getTrans)
?xN=val, ?[xN+off]=val	Only report hits where the register (or the 64-bit word at xN+off) equals val (hex); others pass silently. E.g. main:4743c0?[x0+30]=781548 = only allocations into the heap whose size field is 0x781548
!xN	Treat xN as a sead::ExpHeap and print its free and used blocks (offset+size, p + alignment pad)

The !xN heap walk takes the layout of the game's sead version from DM_PROBE_HEAP=<name>,<start>,<size>,<free list>,<used list> (hex offsets in the heap object; default Mario's 0,28,30,b0,c8, Breath of the Wild's 28,38,40,e0,f8; find them in the heap's constructor and tryAlloc). DM_PROBE_HEAPNAME=<text> reports only hits whose heap name contains the text (resource heaps are named after their file), DM_PROBE_HEAPWORDS=1 adds the first two words of each used block (a vtable names the object), DM_PROBE_HEAPFRAME=<vtable offset in main> prints heaps of that class as frame heaps (head, tail, free), and DM_PROBE_HEAPRAW=1 dumps the heap object's first 0x130 bytes, to find another class's fields.

DM_PROBE_ALL=1 adds x9–x18; DM_PROBE_X2=<bytes> dumps the memory x2 points to.

Who writes a word: DM_PROBE_WATCH=<address>[+<length>] (hex guest address, 8 bytes by default; the bytes must lie in one 16 KB host page). The host write-protects that page as soon as it is writable and keeps it protected: the fault handler carries out every store that lands on the page through a second, writable mapping of it (STR / STUR / STP of general and SIMD registers in their immediate, pre- and post-index forms, register-offset STR, DC ZVA), so the stores of a busy page all pass through it. A store that reaches the watched bytes prints WATCH … store of N bytes at …: old → new, the thread, pc, lr and a backtrace. Other instructions (atomics, exclusives) get a not emulated line once per pc and run on an unprotected page, protected again a millisecond later. A guest object sits at the same address in every run (the guest heap is laid out the same), so take the address from a probe's register dump first. STEINS;GATE RE:BOOT's render size setting (engine object + 0x6000): the static search for stores with that offset, conditional probes on 25 candidate stores and a wrapper around the host's memcpy / memset (the guest's go straight to the host's libc, hle_table.c, where guest probes cannot see them) all found nothing; a watch that unprotected the page for every fault missed the write among the bursts of stores to the engine object's page; this one found it on the first run (main+0x389e90, a setter its start-up code calls with 0).

An exclusive or atomic store (STXR, STLR, CAS, SWP, LD<op>) that reaches the watched bytes prints WATCH … exclusive store (or atomic store) with its pc and backtrace, every time, though it is not emulated. A page that takes many atomics stays unprotected most of the time, so even that can miss: Tears of the Kingdom's memory-shortage flag (a word its frame check compares with 1 and clears) shares its page with atomic counters, and the watch saw neither the write nor the check's own clear.

When the watch misses: DM_PROBE_POLL=<address>[+<length>] (hex guest address, 4 bytes by default, up to 8). A host thread spins on the bytes; on each change it stops every thread at once, reads each one's pc, lr and 8 frame-pointer frames, lets them run again, then prints POLL thread … for the threads with guest code in those frames, the change (POLL … old → new) and, for the first two changes, the full guest thread dump. The snapshot comes a few microseconds after the store; a writer that went on into a wait shows only its wait (for that flag the one thread in guest code was PlacementMgr in main+0x5aaac0, both times). It costs one core while it runs. A word that changes thousands of times before the bad store takes a value filter, DM_PROBE_POLL=<address>+<length>=<value>: every change is still logged, but the threads are snapshotted and dumped only at changes to that value (Mario vs. Donkey Kong: a heap block's list link, rewritten by every allocation beside it, until an output index landed in it, AAC decoders offer at most 32 outputs).

A probe's backtrace reads each frame with vm_read_overwrite: code built without frame pointers keeps data in x29, and the walk used to fault in the host on such a value (x29 = 0x430 at main+0x38921c in the same game).

Use it when lldb is unavailable: after macOS revokes the debugger authorization (DevToolsSecurity -status says developer mode is disabled), lldb launches hang waiting for a GUI approval prompt. sudo DevToolsSecurity -enable (run by the user) restores it.

lldb

The host uses SIGSEGV for dirty-page tracking, so lldb must pass it through without stopping. Address randomization is off under lldb, so guest addresses repeat across runs. Template ($W/dbg/*.py, run with perl -e 'alarm 240; exec @ARGV' lldb -b -o "command script import x.py"):

import lldb
HOST = '/path/to/dm/host/dmhost'
def __lldb_init_module(dbg, _):
    dbg.SetAsync(False); ci = dbg.GetCommandInterpreter(); res = lldb.SBCommandReturnObject()
    ci.HandleCommand('settings set platform.plugin.darwin.ignored-exceptions EXC_BAD_ACCESS|EXC_BAD_INSTRUCTION', res)
    t = dbg.CreateTarget(HOST); t.BreakpointCreateByName('run_module_inits')
    li = lldb.SBLaunchInfo(['<patched dir>']); li.SetEnvironmentEntries(['DM_ROMFS=…', 'DM_SAVE=…', 'DM_TEMP=…'], True)
    p = t.Launch(li, lldb.SBError())
    for sig in ('SIGSEGV', 'SIGBUS', 'SIGUSR1', 'SIGUSR2', 'SIGPIPE'): ci.HandleCommand('process handle %s -n false -p true -s false' % sig, res)
    base = None
    while p.GetState() == lldb.eStateStopped:
        fr = p.GetSelectedThread().GetFrameAtIndex(0)
        if base is None and fr.GetFunctionName() == 'run_module_inits':
            base = t.EvaluateExpression('(unsigned long)g_mod[0].base').GetValueAsUnsigned()   # g_mod[0] = main
            t.BreakpointCreateByAddress(base + 0x2ea228)                                       # module offset
        elif base is not None and fr.GetPC() == base + 0x2ea228:
            print('x0=%#x' % fr.FindRegister('x0').GetValueAsUnsigned(), flush=True)          # flush: a timeout kill loses buffered output
        p.Continue()
    p.Kill()

Patterns that worked: print every call of one function with its arguments; stop when a return value is null and print the backtrace; trace the branch path through one function (one-shot breakpoints on every branch site). Hardware watchpoints on words touched by ldxr/stxr may stay silent.

Scripted input and thread scheduling

• DM_KEYS="<seconds>:<button>[+button][~hold],…" presses buttons from host start (hle_hid.c). <seconds>:<button>*<count>/<interval> repeats a press, e.g. 42:A*170/2 presses A every 2 s from 42 s on, 170 times; at most 1,024 steps after expansion. Steps run in time order, so repeated groups interleave (f17400:Left~0.3*120/500,f17480:A*120/500,f17560:Left~0.3*120/500,f17640:A*120/500 = Left, A, Left, A every 500 frames: Fire Emblem: Three Houses' confirm dialogs default to No, and this cycle confirms Yes whether a dialog is open or not when it starts; A, Left, A loops forever once it starts on an open dialog); a list that mixes seconds and frame numbers keeps its listed order. A long run through a game's story is then a single switch, such as Metroid Dread's title, new game and story recap. LLeft / LUp / LRight / LDown push the left stick fully (Super Mario 3D World moves on its map by stick only); Shake shakes the controller for the hold time, at least one whole shake (its six-axis sensor); combos such as L+R~0.5 press buttons together. Scripted presses are player 1's (dm_pad_set), as is the keyboard. An f before the time counts presented frames instead of seconds (f2500:A, repeat intervals in frames too): with another heavy job running, seconds drift away from the game's progress and a route stops lining up; frame numbers do not. Breath of the Wild from its title menu into the autosave: DM_KEYS=f2500:A,f2800:A,f3100:A,f3400:A (control at frame ~3900).
• DM_CTL=<file> (Mac) drives a running game step by step instead: the host polls the file every 100 ms and acts on each line appended to it (hle_hid.c ctl_main). A line is a press as in DM_KEYS (A, LDown~0.2, L+R~0.5), or shot <path> (the next presented frame as PNG), passdump <n> (the next n frames as DM_NVN_PASSDUMP), texdump <path with %d> (the next frame's colour targets as DM_NVN_TEXDUMP), texraw <serial> <path> (level 0 of the host texture behind a serial, raw, with a width height format row header), texlist <NVN target> (registered textures of that target, 2 = 3D, with storage address, size and block-linear layout), memdump <address> <bytes> <path> (guest memory to a file: a buffer texture's storage from its B line holds what the GPU wrote), peek <address> <count> (floats to the log), cvarf / cvari <main offset> <value> (an Unreal console variable's game- and render-thread copies, for a TAutoConsoleVariable found as for hle_fpslock.c), or env KEY=VALUE (for switches read at use). Look, press, look again: menus and loads whose timing moves between runs need no blind timed script. A shell helper that appends a line and waits for a shot to be written keeps the loop short (DRAGON QUEST VII Reimagined's work folder: run/ctl.sh with wait S / snap NAME, and route scripts on top of it into the opening cutscene).
• Frame numbers are a stable way to aim dumps (DM_NVN_PASSDUMP=<frame>, DM_NVN_SHOT=…) at the same scene in another run: the per-frame NVN call counts ([nvn] frame N: … calls) matched between runs of the same script. Times in seconds drift with load.
• A route through a 3D level (Captain Toad: Treasure Tracker's opening course, where Toad has to find his way up a diorama to the star): screenshots alone took a dozen runs and still lost Toad behind the scenery, and turning the camera mid-route made the same input land differently each run. What worked: read the player's position every frame with a probe and plan from the stage's own data. The guest heap is laid out the same in every run, so the player actor is at the same address. DM_PROBE=main:1fb2e0@*x0+48+8 (PlayerActor::control, the actor's pose at +0x48, the position 8 bytes into it, as al::getTrans reads it) prints x, y (height), z as three floats per frame. The stage map (StageData/<stage>.szs, a BYML) lists every actor with its Translate, including the goal and the rail the guide character walks. Stick directions are camera-relative: on the default camera two directions together (LUp+LRight) move along one world axis. Hold against walls where possible, since a wall stops the player at the same place in every run; the route then reproduced in 5 of 5 runs.
• Spin waits (patch_nso.py "Spin waits", dm_spin_wait in hle_thread.c).
  • Switches: DM_SPIN=0 makes the gate return at once, so the loops poll with no sleep; the gate itself stays. DM_SPIN_POLL=<µs> sets how long a wait polls before it yields: 2000 by default, which is where the yield back-off starts sleeping. DM_SPIN_LIST=1 makes the rewriter print every loop it found.
  • Logs: sites.tsv lists each loop as <head> S <back-edges>. In the power line's top yielders, a thread's spin passes show as spin passes N, last at main+0x…; the site is the loop head.
  • Checking a loop: to tell whether one is a real wait, disassemble from its head to its last back-edge. It must reload what it compares on every pass.
• Who runs during a hitch. Guest threads carry their guest names as host thread names (0x1001 main thread, 0x1008 Worker), so sample <pid> and xcrun xctrace record --template 'Time Profiler' --attach <pid> (no root needed) name them; the export (xctrace export --xpath '/trace-toc/run[@number="1"]/data/table[@schema="time-profile"]') gives each sample's time, thread and backtrace, and guest frames map to main+… with the module bases at the top of the log. The trace starts at the TOC's start-date; the TSV's @call_ms counts from the process start (kinfo_proc p_starttime), which lines the two up. DM_HITCH_DUMP=<ms>[:<s>] logs every guest thread once whenever no frame has been presented for <ms> (from <s> seconds after the start on, at most 24 times): what it waits on, its last SVC, and where it is, with x0–x2 and x19–x25 when it runs guest code. Breath of the Wild's CPU hitches showed up there as the main thread and two workers in one guest loop with no SVC, and the registers told which thread the loop waited for (its item index in x0; docs/development.md, Spin loops).
• DM_PREEMPT=0 / 1 disables / enables the additional check of already-runnable same-core peers after every thread SVC, overriding the profile's core_preempt (hle_thread.c). Wake-triggered handoffs apply with either setting. The thread dump reports same-core handoffs N, total T ms; these waits have no timeout that releases unfinished guest work. See Same-core wake handoffs for the runnable-state rule and regression test.

Racy crashes: load, trace gates

A crash that needs an unlucky thread interleaving (one run in three, or once in a long session on the phone) gets much more frequent when the CPU is busy, because threads are descheduled at more places. 12 busy shell loops (for i in $(seq 12); do (while :; do :; done) & done) beside a scripted Mac run made Link's Awakening's physics crash happen in every run instead of 2 of 5, and the frame rate kept switching between 60 and 30 as on the phone. That is enough to test a fix: same load, same route, a few runs.

To see the order of events, put temporary observation gates (OBS in the profile's patches.py, only at function entries or right after a bl, on position-independent instructions) on the functions involved. Their host callbacks append (time, thread, event, pointers) to an in-memory ring, and the crash handler prints the last ~150 entries. Printing each event as it happens is too slow and changes the timing. A function whose first instruction is PC-relative (an adrp, typical of short destructors) cannot take a gate: a temporary cut can put svc #0x8f at its entry and move its body into the udf padding after it (inside the same 4 KB page, an adrp keeps its encoding), with svc_host_call handling the number. Inside the host callback, the caller's return address is on the guest stack, in the frame the gate pushed (regs + 800 + 8), because the gate's own call has replaced x30. Remove all of it afterwards.

Memory growth

1. The host's memory report line (memory up to frame …) every few hundred frames; footprint -p <pid> from outside. On a phone, a crash with no log usually means headroom to the system memory limit reached 0.

2. vmmap -summary <pid> twice a few seconds apart: which region grows. "owned unmapped (graphics)" = private Metal resources.

3. DM_NVN_ALLOCPROF=1: growth per replayed op type and per function (tex_mtl / pass begin / pipelines / argument buffers), command allocator sizes, and a log of new textures with serial, size, format and storage. A stream of new serials with the same sizes = objects re-created every frame.

4. GPU allocated climbing while the residency set, shared textures and the upload counts stay flat: retired objects kept alive by an autorelease pool that never drains. Any objc call on the guest submit thread outside g_sub_pool leaks what it autoreleases (Metroid Dread; Breath of the Wild, GitHub issue #9). Build the host with the suspect wrapped in @autoreleasepool and compare.

5. serial objects climbing by the same amount every frame with GPU memory flat: entries the serial table never deletes. A census of g_by_serial by class and size in mtl_mem_stat shows which (BotW: finalized textures replay resolved again; they now go to the small table counted as finalized textures still referenced).

6. Address space rather than memory (an iPhone has about 5 GB of it without the extended virtual addressing entitlement): DM_ALIAS_WINDOWS=<MB> logs every 30 s how much of each window of that size in the alias region is mapped (windows aligned to their size in absolute addresses). For an allocator that reserves fixed windows and commits each from its start, the windows are its pools: Unreal's FMallocBinned3 in Octopath Traveler 0 showed 74 pools of 512 MB holding 732 MB together, the largest 68 MB, so no smaller uniform window fitted (memory.md). Unreal's own heap on the host malloc zone (hle_uemalloc.c) logs its peak as [mem] Unreal heap … in use.

Instruments' Metal Resource Events didn't record this process from the command line (xctrace record --launch, with --env for the variables); the in-host probes were faster.

Save failures

DM_TEST_FS_FAILAT=<point>[:<count>][,…] makes steps of the save transaction fail with ENOSPC (mountclone, reclone, guestname, renameback; dm/host/tests/t_fs.c section 6). The game must keep running: a mount that cannot clone its work directory falls back to the committed saves read-only, and the iOS front end shows a notice.

Render-pass investigation

See 04 section 7. Notes: pass dumps are large and may contain non-UTF-8 bytes; filter with LC_ALL=C grep -a "\[pass\] F600 " | grep -av " o cmd". A Python one-liner with PIL (the dm venv has it) gives mean colour and non-black share of each dumped PNG, which is quicker than opening 20 images.

On-device diagnostics (iOS)

The phone runs an unsigned IPA the user sideloads; nothing is attached to a debugger. The loop is: build an IPA with switches, the user plays and sends Documents/log.txt (the previous launch is log.prev.txt), read it, repeat.

• Switches: DM_ENV_FILE=<file> dm/ios/tools/build_ipa.sh bakes KEY=VALUE lines into the app (dm/env.txt); Documents/env.txt overrides them without a rebuild. Every applied switch is logged at boot (boot: env KEY=VALUE (source file)). The app's working directory is Documents, so relative output paths (DM_NVN_SHOT=3600:shot_%d.png, DM_NVN_TEXDUMP) land where the Files app can reach them.
• Which build ran: the first lines name the modules (main: … patch_off …), the shader map size (shader map N entries) and, for builds that have it, the build-ID gate line. Compare them with the build you sent before reading further; users sometimes send an old log. The save path (fs mount save:) shows the data folder, which changes when a sideloading tool gives a reinstall a new one.
• Reading the periodic lines: [longframe] gives each long frame's submit / GPU / pipeline-compile / upload split; a burst of new pipelines in long frames after an install is first-use compilation (see "Prewarm lists" in docs/development.md). In display link up to tick N: …, empty ticks (refreshes with no new frame) and old frames dropped by merging are totals since launch, not per window. on-screen up to frame N windows give dropped frames (a frame on screen longer than 1.5 × its own hold, so a frame-step hold of 2 vsyncs at 33.3 ms is not dropped; discarded presents count too), p99 frame time and the physical footprint; memory up to frame N gives resident and Metal-allocated memory. Memory that steps up at level loads and stays flat in between is not a leak.
• Power: power up to frame N covers the window since the previous one, for the whole process: CPU energy in watts and per frame (task_energy from TASK_POWER_INFO_V2, CPU only; compare per frame when one build holds the frame rate better), CPU time (100% = one core) and the share on performance cores, wakeups per second, the thermal state (0 nominal … 3 critical; changes are also logged as front end: thermal state → n), and the two guest threads that yielded most with how many of those yields the back-off turned into sleeps (DM_YIELD_BACKOFF, see docs/development.md), and its spin wait passes with the last loop head. A thread yielding hundreds of thousands of times per window with few back-off sleeps is a busy spin. A busy thread that never yields and is not in the list may poll in a loop the spin wait rule does not take: sample it and look at its loop. residency set up to frame N gives residency-set commits and their time, what was added by source line of nvn_mtl.m (count, MB), removals, how many draw passes began with a full barrier or without one (DM_NVN_BARRIER=game), and segment commits: a source line adding gigabytes per window is allocation churn. It ends with render-target traffic per frame: attachment bytes loaded and stored, passes that load and store everything, and the stores DM_NVN_UNWRITTEN_DC saved (the same counters as the TSV's @rt_load_mb / @rt_store_mb). threads up to frame N gives guest-thread CPU time in ms per second by Switch priority band (≤ 44, the main thread's level and above; 45–59; ≥ 60) and the three busiest threads with their priority; it shows what DM_QOS_MAP would move. On the Mac the same lines appear; energy readings vary by tens of percent between identical runs, so compare runs interleaved and over the same scene.
• Screen recordings from the user are useful for timing and visual symptoms; extract frames with ffmpeg -ss <t> -i rec.mov -frames:v 1 out.png.

Offline pipeline link check

Pipeline link errors depend only on the two libraries, so all vertex / pixel pairs can be checked on the Mac after a shader build, without the game: for each line of shader_pairs.tsv, load <vs>_b.metallib and <ps>_b.metallib from b-<V>/, build a vertex descriptor from the vertex function's vertexAttributes (format from each attribute's type), set colour attachment formats from the [[color(n)]] outputs in the pixel shader's _b.metal (uint / int / float formats to match), add a depth format when it writes [[depth(...)]], and call newRenderPipelineStateWithDescriptor:. Count failures by message. Mario: 8,712 linked / 331 failed before the vertex output padding, 9,043 / 0 after, and 14,603 / 0 for the v0 + 1.2.1 tables. Pipelines built this way have the default blend state and sample count, so this checks linking only.

Intermittent rendering glitches

A glitch that comes and goes between frames or runs needs a detector before bisection, or "no glitch in these 16 frames" proves nothing. What worked for Mario's fountain panels:

1. Screenshot a dense frame range (DM_NVN_SHOT=$(seq -s, 4430 10 4590):x_%d.png) and score each frame automatically. The first metric (texture busyness) missed the flat translucent variant of the glitch; a metric tuned to its shape (long vertical straight edges in a sky region) separated glitch frames (≈1.0) from clean ones (≈0.3) reliably. Look at a few frames by eye before trusting a metric.
2. Run a control with nothing changed to measure how often the glitch appears per run.
3. Bisect with DM_NVN_SKIP_FS (up to 16 fragment shader keys per run) until one draw remains, then DM_NVN_HILITE it to see its geometry.
4. Dump that draw's inputs across frames (DM_NVN_CBDUMP + DM_NVN_CBDUMP_KEY). If every input is byte-identical between good and bad frames, the difference is on the GPU side: follow the textures it samples (DM_NVN_TEXINFO, DM_NVN_TEXRAW) to whatever writes them.
5. For a GPU-side simulation, trace its state on the GPU timeline (DM_NVN_T2T_TRACE), re-implement the shader step in numpy with the logged uniform updates (the Y update_ub lines print the first four floats), and compare frame by frame. Frames that disagree show which input the GPU really used; for the fountain, a coefficient of −1 instead of 1.81 explained every bad texel and led to the missing intra-encoder barrier.
6. A dump read at encode, or after the submission completed, may not be what the draw read: later work of the submission can rewrite the buffer. When a draw's indices or vertices look wrong, list in command order every copy and inline update of the frame from the pass dump (DM_NVN_PASSDUMP_ALL=1, the Y copy_b2b / Y update_ub lines): a copy whose source an earlier GPU write of the same frame covers reads that write, not the bytes the CPU staged. Cuphead's broken runs had dozens of those a frame, and the clean runs none. The game's allocations overlapped, so the cause was in the game's allocator, which a host answer had broken (Sizes the hardware never reports as 0).