Ported from d2r-mobile (b94670c79, 2eb575877, 704147218), adapted:
- Save transaction: if the work directory cannot be set up at mount
(leftovers not removable, clone fails, e.g. a full disk) the host
now mounts the committed saves read-only and tells the front end
instead of exiting; a reclone after a refused commit clones beside
the work directory and swaps it in atomically, and on failure keeps
it tainted for the next try; a failed rename rollback refuses the
transaction instead of exiting. Creating an existing file or folder
on a read-only mount answers "already exists", and file / directory
calls ignore the null handle a failed open leaves. Test switch
DM_TEST_FS_FAILAT; tests/t_fs.c section 6 (68 of 68 pass). An Engage
run with the mount clone failing loads its autosave read-only and
plays on.
- iOS front end: front_save_failed shows a bilingual notice (it only
logged before).
- thread.c: each guest thread's TLS block, thread-local region and
thread object (9.5 KB) are freed at thread exit through a key of
their own (the guest overwrites the TLS / TLR key slots).
tests/t_thread_leak.c: 600 blocks leaked before, none now.
- Memory line on iOS: os_proc_available_memory, how far the app is
from being killed for memory.
- The host test Makefile gets the game profile include path; t_fs
stubs the module table. Docs: 03-bringup memory line fields,
05-debugging save failures. Mario regression t78 and the iOS compile
are clean.
- Ported from d2r-mobile ef9b1ff13, adapted to our shared-page ranges:
setting a page read-only marked it PROT before mprotect landed, so a
concurrent unprotect-before-write could set it back to NONE and then
the mprotect made it read-only anyway. A game thread writing it then
spun forever in the fault handler, and a host system call into it
failed with EFAULT. Now NONE -> BUSY -> mprotect -> PROT (and back),
waiters re-check after BUSY clears, and the wait is capped at 2 s
with a red line. Engage's movie decoder writes frames from a host
thread while replay protects pages; this is a likely cause of the
old early stalls.
- tests/t_dirty_race.c (ported) reproduces it deterministically: on
the old tracker read() into the page fails with EFAULT; it passes
now, and t_dirty_slots still passes.
- The graphics unit tests had stopped linking (prog_key moved to
hle_nvn.c) and some crashed (hand-built textures lacked the texture
magic replay now checks; t_nvn used a function that has since been
implemented as its unimplemented example and hard-coded 523 names).
Fixed: t_nvn, t_fuse_check, t_zskip, t_xv, t_ub, t_pfv, t_present_crop,
t_rt_depth_ro, t_rt_range_shape, the dirty tests and others pass.
Still failing, the same before this change: t_reorder_chain (30 of
155), t_defer_chain (1 of 59), t_mip_extent (1 of 303), t_rt_range,
t_rt_range_ro; t_refl needs D2R shader samples.
- Mario regression t77, an Engage movie / title / demo run (0 missed
dirty pages) and the iOS compile are clean.
Ported from d2r-mobile (e2a0c9a9b, 2eb575877), adapted:
- Memory pools no longer go into the residency set when created. On
iOS a no-copy buffer in the residency set is read in and wired whole,
even if the game never uses the pool (upstream measured +530 MB for
one unused 512 MB pool; a 4 GB A14 was killed at launch). A pool now
becomes resident, and the set is committed on the spot, the first
time mtl_addr / mtl_gpu_addr hands out its address. Mario keeps 163
of its 871 pools resident, Engage 18 of 30. The memory line shows
the count.
- The occlusion-query overflow buffer (one submission past VIS_SLOTS)
was cleared at the next submission without being retired, so it
stayed in the residency set; overflow also must never retire a ring
buffer still in rotation. Both fixed.
- Pipeline prediction uses the format a colour attachment is actually
attached with (the view's, when a view changes the format), shared
with attach(). Prediction used the base format, so Mario's last pass
(written through an sRGB view) predicted the wrong pipeline.
- Mario regression t76 (same scene, 60 fps), Engage movie / title /
demo run and the iOS compile are clean.
- Finalizing a memory pool also drops the cached MTLTexture (view or
base texture) of every texture ID whose registered texture lives on
that pool. The cache was released only when the ID was registered
again; a still-used ID rebuilds it on the slow path.
- The memory readout counts finalized pool buffers through a weak
table. Objects read from it come back retained and autoreleased, and
the frame hook runs on a guest thread with no draining pool, so the
enumeration is now wrapped in @autoreleasepool.
- Engage idle demo GPU memory at quiet points 1.06 to 1.12 GB (1.16 to
1.20 before); about 17 finalized 16 MB pool buffers are still held by
something else, bounded. Mario regression t74, Engage combat run and
the iOS compile are clean.
- A ClearDepthStencil inside an open pass was skipped (red line "通道中途
的深度模板清除未实现"), so later draws depth-tested against stale
depth. It now ends the pass and attaches the clear to the next one,
whose color attachments load as before, the same as a continued pass
does when a clear arrives. A scissor that does not cover the depth
target still cannot be expressed as a load-action clear; that case
keeps the red line.
- Engage hits it during boot (pass 43); its loading screens, title and
demo scenes render as before. Mario does not take this path
(regression t73 clean). The iOS project compiles.
- The storage-sharing texture table (g_alias, keyed by pool object,
offset and description) never evicted. Engage frees memory pools on
scene changes and builds new ones, so the old pools' render targets
and other private textures stayed for good, and linear textures built
on a pool buffer kept the whole buffer alive. A reused pool object
address could also hand an old texture to a new pool. Finalizing a
pool now evicts its entries (mtl_alias_drop_pool) through deferred
release.
- With 21b3bc3, Engage's 850 s idle demo stays at 1.2 to 1.4 GB of GPU
memory (5.4 GB before both fixes) and ends at 2.5 GB resident instead
of 3.6 GB; in the prologue battle GPU memory is 1.07 GB at frame
11100 (2.38 GB before).
- The memory report line adds the host objects held by serial, the
finalized pool buffers still alive, and the sharing table size with
its evictions.
- Mario regression t72 (it does finalize pools: 32 evictions; same
scene, 60 fps, GPU memory unchanged), Engage demo scenes and combat
are clean; the iOS project compiles. Docs: 04-graphics entry.
- Each memory pool is wrapped in a no-copy MTLBuffer that goes into the
residency set. nvnMemoryPoolFinalize only dropped it from g_keep; the
residency set still held it, so the buffer stayed alive around the
freed guest memory, and a new pool at the same address wrapped it
again. It now goes through deferred release (out of the residency
set once submitted work completes), like finalized textures.
- Engage's idle title demo, which cycles scenes, reached 5.4 GB of GPU
memory in 850 s; it now ends at about 2.1 GB (a slower growth
remains). In the prologue battle GPU memory at frame 9600 is 1.46 GB
instead of 2.38 GB. That matters on iPhone, where the system kills
the app at its memory limit.
- Mario regression t71 (60 fps, same scene, pools unchanged), an
Engage battle run with combat animations, and the 850 s demo run are
clean; the iOS project compiles.
- nvnDeviceGetTimestampInNanoseconds returned the counter's raw 19.2 MHz
ticks; it now converts them to ns. GetCurrentTimestampInNanoseconds
returned real ns, so a game subtracting one from the other (Engage
calls both) got garbage.
- Counter timestamps and GetCurrentTimestampInNanoseconds now come from
the guest clock (guest_ns, the source of svc GetSystemTick / cntpct),
as on hardware, where GPU timestamps and the CPU counter share one
timebase. They came from CLOCK_MONOTONIC, which keeps running while
the Mac sleeps while the guest clock does not. With ns units but the
old clock, Mario dropped to 30 fps (it compares the GPU end time with
a CPU frame-start tick at main+0x20a0394); on the guest clock it runs
at 60 fps as before.
- Mario regression t67 (60 fps, same scene; frame-time p95 matches a
baseline run of the committed code, t70) and an Engage idle run
(30 fps, same pacing as before) are clean; the iOS project compiles.
- Measured over a 12-minute run: StandardAllocator commits only 0.58 GB
of the 3.14 GB it reserves, Unity's own regions 0.97 GB. The profile
now reports a 2.5 GB pool (StandardAllocator about 2.44 GB) and uses
a 5 GB block with a 4.375 GB alias region, the total proven on a
phone. The old 6 GB block was iOS's single-reservation limit and left
Metal no address space. Reserved address space peaks at 3.47 of
4.31 GB; a 670 s battle-script run is clean.
- The early stalls (one run in four) have not come back in 12 runs
since the real thread pause for IL2CPP's GC; the docs mark them
probably fixed and keep the sampling recipe.
- Docs: sizing entry with the commit split and how to measure it,
profile notes, README Engage row (it reaches the battle map; the
scripted presses end on the save screen, not in combat).
- The prologue battle crash is marked fixed, with a pointer to the
address-space entry in 03-bringup (missed in fe9bab9).
- The early stalls get their own open entry, with the sampling recipe
and the new stderr-lock line from the crash handler.
- crash.c: before printing a crash or time-limit report, the handler
tries the stderr lock for 1 s. If another thread holds it, it writes
one line (signal, pc / lr as module offsets) with write() and exits,
instead of hanging with the process and leaving an empty log. This
is for the intermittent early stalls, where even the SIGALRM report
never appeared.
- Engage's idle attract-demo crash (about 220 s) was the same alias
address-space shortage as the battle crash: the particle index
buffers got no mapping past 2 MB. With the node-area cut (fe9bab9) an
850 s idle run is clean, with no corrupted texture objects; with the
old SDK the same run dies at 159 s in Unity's heap allocator.
- Docs: the 04-graphics entry is marked fixed with the cause; 03-bringup
and the README Engage row updated. Mario regression t65 clean; the
iOS project compiles.
- The crash about 6 minutes into the prologue battle (null write at
main+0x9a4664) was the alias region running out of address space, not
memory: Unity's low-level virtual allocator got no region from
AllocateAddressRegion and returned 0. The SDK's AddressRegionManager
keeps the first 1 GB of the alias region as its node area and aligns
each region by size; after StandardAllocator's 3.14 GB block only about
1.2 GB was left for Unity, and it was full and fragmented.
- patch_nso.py: a host-only cut for Engage's sdk build (56ec2474…)
shrinks the node area from 0x3fe00000 to 0x3e00000 (62 MB, about 1.35
million nodes). Both constructor movs change together.
- A 12-minute scripted run now plays through the battle without a crash;
reserved address space levels off at 4.17 of 5.31 GB.
- hle_svc.c: the alias-mapped total counts pages with a bitmap, so maps
of already-mapped pages and oversized unmaps no longer skew it (it went
negative). Engage really maps about 1.5 GB, not 0.5 GB.
- Docs: bring-up catalogue entry with the measuring recipe, profile
memory notes, README Engage row. Mario regression t64 clean; the iOS
project compiles.
- SetThreadActivity and GetThreadContext3 were placeholders that only
reported success. IL2CPP's garbage collector pauses each thread and
reads its registers for roots, so threads kept running and roots held
in registers were missed. A thread inside an SVC now counts as paused
at once with the gate frame as its context (thr_svc_enter / leave
around every gate SVC; it waits at the SVC exit while paused). A
thread in guest code is stopped with SIGUSR2 and parks in the handler.
The context is Horizon's 0x320-byte ThreadContext.
- nn::pctl::IsFreeCommunicationAvailable answers "not available" like
the other free-communication calls; the SDK aborted in sm when Engage
entered its first battle map.
- Memory log lines: heap growth, alias-region high-water mark and the
total mapped in the alias region (one line per 256 MB).
- Engage profile: a 6 GB reservation (5.375 GB alias region, 128 MB
heap region), iOS's single-reservation limit without the extended
virtual addressing entitlement. Engage never calls SetHeapSize; its
mappings reach +5.26 GB.
- Engage on the Mac: with scripted input and DM_KBD answering the name
keyboard, a new game reaches the prologue cutscene and first battle
map. Open: a heap-allocator crash a few minutes into that battle, and
occasional early stalls (docs/agents/04-graphics.md).
- Docs: catalogue entries, README rows. Mario 1.2.1 regression (120 s
scripted) clean; the iOS project compiles.
- nvnCommandBufferDownsample / DownsampleTextureView are recorded as a
texture copy with a host flag (DM_COPY_RESOLVE) and replayed as a
Metal multisample resolve (depth takes sample 0); equal sample counts
become a plain copy. They used to be skipped.
- A texture object that is invalid when recorded (Unity finalizes and
re-initializes the same render target between recording and
submitting) keeps the old behaviour: the record holds the object
pointer and replay reads it at submit. Snapshots made render targets
disappear in that case.
- crash.c: only the first crashing thread prints its state; later ones
wait in pause(), and a 15 s alarm ends the process if the dump stalls.
Unity's job threads crash together, and the dumps used to interleave
and leave the process hung.
- NVN render target, texture copy and clear records now carry a copy of
each texture object (REC_SNAP / tex_snap) instead of the game's
pointer, as NVN commands do not depend on the API object once
recorded. Replay rejects objects without the texture magic and sizes
beyond Metal's limits with a red line (a 1869367126-high texture used
to abort in Metal's validation).
- SignalToAddress type 2 with count <= 0 and waiters present now
subtracts 2 (mesosphere, Ryujinx and yuzu agree); it subtracted 1.
- Engage on the Mac: scripted input reaches the main menu and a new
game's appearance choice. The idle attract demo still crashes after
about 220 s (Unity's particle geometry job overruns its index buffer);
findings are in docs/agents/04-graphics.md.
- Docs: catalogue entries, README row. Mario 1.2.1 regression (120 s
scripted) clean: only the known SetRenderEnableConditional red line.
- dm/third_party/libvpx: the VP9 decoder of libvpx v1.17.0 (upstream
6df3ec3, BSD 3-Clause), unmodified: the 71 sources a VP9-decoder-only
arm64 build compiles, their headers, and the generated configuration
in config/ (one config for Mac and iOS, NEON, no run-time CPU
detection). README.md records the configure line.
- dm/host/Makefile builds it file by file into gen/vpx/libvpx.a, so host
changes do not recompile it. gen_build.py now collects sources and
header paths from every clang line of make -n (host link line first),
so the iOS project compiles the same files.
- hle_movie.c decodes VP9 packets with libvpx instead of emitting black
frames. Each decoder has its own host thread (guest threads run on
SDK stacks, too small for libvpx); the guest thread that sends a packet
or returns an output index waits for the decode and signals the
events itself. Packets are decoded in arrival order from one queue;
I420 is converted to NV12 with NEON. The destructor logs decode times.
- Engage's 94 s opening movie decodes completely (2,816 of 2,816 frames,
about 2 ms per frame on the Mac). The iOS project compiles and links
with the decoder (not yet run on a device).
- hle_movie.c implements movie::Decoder (the SDK's MediaCodec wrapper)
through hooks on every vtable method: DecoderEvents signalled with
the SDK's SignalEvent, input / output index queues, Buffer and
MediaData filled with the SDK's own setters, one NV12 frame per
packet into the buffer Unity allocates (w x h x 1.5); packets wait
while every output slot is taken. Frames are black for now (no VP9
decoding yet); the size comes from Configure or the VP9 header.
- Engage on the Mac: loading screen, the 94 s opening movie, then the
title screen and, left idle, the 3D attract demo. Not yet: input,
a crash about 221 s into the demo, an unmapped texture format, iOS.
- Docs: catalogue entries (movie protocol, shader keys, CPU-written
textures, NVNprogram size), Engage graphics notes, README row.
- Shader keys are computed inside nvnProgramSetShaders: NVN lets the
caller free the control section once it returns, and Unity does
(about 100 programs were reported missing with garbage control
sections). A stage whose code is not yet in a memory pool (Mario
sets some programs with an unmapped address first) is hashed at its
first draw. Keys live in a side table: Mario's NVNprogram is smaller
than the header's 0xC0, and growing prog_t corrupted the next object.
- Textures written with nvnTextureWriteTexels resolve like packaged
textures: uploaded from guest memory and re-uploaded on dirty pages.
Unity's video frames and the engines' 4x4 default textures (black,
white, transparent, red) used to stay zero. Mario's course frames
get slightly brighter where materials sample the default white.
- The c2 driver bank also gets the separate-sampler table
(0x568 + 8 r), which Engage's shaders read at run time.
- Mario 1.2.1 regression (120 s scripted) clean: only the known
SetRenderEnableConditional red line.
- New dm/gfx/tools/unityfs.py: UnityFS bundles (LZ4 / LZMA blocks),
serialized files (formats 19..22), objects read by their type trees
(Unity's common string table comes from the game's own main), and a
Shader object's per-platform compressed subprograms. The player's
own files have no type trees and borrow the bundles' Shader tree of
the same type hash.
- shader_split.py reads Unity Shader objects in a second pass: each
Switch subprogram is one NVN program (0x60-byte header of per-stage
offset / control size / code size, then control + code per stage);
a program's vertex / pixel stages form a pair. Engage: 9,269
shaders, 4,634 pairs. Mario's tables and package are byte-identical.
- Engage profile: shader groups engine / assets, measured NVN layout
(texture handles 0x168 + 8 r, 128 slots; separate samplers 0x568),
regenerated tables, chain a295a8035b7b. 9,268 of 9,269 compile (one
geometry shader; Metal has none).
- dmsh_build.py: the translator's built-in bases only matter with
per-slot facts, so the guard checks that; c2 reads count the sampler
table and 16-byte rows overlapping a table as fillable.
- game.py: DM_GAME_NVN_SAMP_BASE. NSP script survey note updated.
- Mario chain is now 271056163ea2 (Mac and iOS libraries rebuilt,
16,266 of 16,268 as before; c1_full regenerated).
- dm/games/fe_engage: identity and the 2.0.0 build IDs (main,
subsdk0, sdk), modules in Horizon load order, RomFS marker, empty
shader tables and their chain header. Memory: an 8 GB reservation
with a 5.5 GB alias region for its virtual address memory (Mac only
for now: iOS allows at most 6 GB without the extended-addressing
entitlement); NVN constants are Mario's until measured.
- On the Mac the host boots Engage through Unity's startup and runs to
the opening movie with nothing on screen yet: its shaders are in
UnityFS bundles that shader_split.py cannot read, and movies are not
decoded.
- README: the games table row.
- patch_nso.py: three more x18 forms from IL2CPP code: an offset jump
table (targets read at rewrite time), blr x18 (original return
address kept for C++ unwinding) and an sp write from x18 (alloca).
Mario's rewrite output stays byte-identical.
- loader.c: symbols resolve globally in load order, also for a
module's references to its own exports (local / non-default
visibility excepted). Engage's main overrides operator new / delete;
sdk bound its own copies and freed Unity's blocks into the SDK heap.
- hle_svc.c: QueryMemory reports each module's Horizon layout (code,
read-only, data, patch area) and other host memory as free, which
the SDK's C++ unwinder (nn::ro::detail::GetExceptionInfo) needs.
- New takeovers: nn::fs::IsMounted, nn::ro (hle_ro.c), prepo
Initialize, SDK 12 audio device names and OpenDefaultAudioOut, an
SDK 12 hid block (single GetNpadState, modes, touch, gestures,
six-axis details, colors, applets), movie::Decoder with no impl
(hle_movie.c) and failing nvdrv opens (hle_nvdrv.c), and NVN getters
/ no-ops Engage calls. nvn.h: TextureGetMemoryPool returns a pointer.
- Makefile: a game stamp, so changing GAME rebuilds the host.
- docs/agents/03-bringup.md: catalogue entries for all of the above.
- Mario 1.2.1 scripted run still reaches the opening course with the
same single red line.
- attach(): a render-target view that changes the format is attached
as a Metal view of that format (kept until the submission completes),
so pipeline formats follow it. Mario's last pass writes the RGBA8
swapchain texture through an RGBA8_SRGB view; with the base texture
attached, linear values were stored unencoded and the whole picture
was darker: on the Mac and the iPhone every channel equalled the
Switch value decoded from sRGB once more (0.55 -> 0.26). After the
fix the Mac title frame matches a Switch screenshot (mean RGB
0.556 / 0.621 / 0.633 vs 0.557 / 0.621 / 0.633).
- present_raw_src(): an sRGB-typed swapchain texture is sampled through
a non-sRGB view, so the stored bytes reach the screen as on the
Switch's scan-out (no change for Mario, whose swapchain is RGBA8).
t_present_crop gets an sRGB case (34/34; fails without the fix).
- docs/agents/04-graphics.md and the mapping docs: the rule and how it
was measured.
- hle_nvn.c: drop the command memory callback / grant / caller
logging, the overflow details, the +0x10 write log, the one-off
NVN table dump, and the DM_DBG_GRANTS, DM_DBG_CALLSITES,
DM_DBG_CMDMEM_SHRINK and DM_DBG_OLD_USED switches.
- nvn_mtl.m: drop the per-draw NaN probe (DM_DBG_NANPROBE), the
colour target statistics (DM_DBG_TEXSTAT) and the PASSDUMP
completion timing.
- Kept as regular diagnostics: the audio callback line every 2000
callbacks (frames per callback, queued blocks, underruns) and
DM_DBG_AU_FRAMES (docs/agents/03-bringup.md, crackling on iOS);
DM_NVN_SHOT up to 4096 frames (the 05-debugging burst recipe).
- Mac 1.2.1 scripted run reaches the opening course with the same
single red line as before.
- README: presented as Switch games running natively on Apple
platforms, with Mario as the one supported game. A games table
(Mario playable, Fire Emblem Engage surveyed), component status,
game profiles and how to add a game, the NSP script, generic
step-by-step build commands (profile, workspace and version as
variables), and a Mario section for the per-game notes (1.2.1).
- AGENTS.md: profiles, DM_GAME / GAME, workspaces, build-ID gating,
the chain rule per profile.
- docs/agents: one repository with one profile per title; survey
mode in 02-survey, including the compressed NACP title area.
- dm/gfx README (+ zh-CN): table paths in the profile.
- The game comes from the NSPs' title ID and its profile in
dm/games/; the versions to extract, the accepted build IDs, the
modules to rewrite and the tables to compare all come from the
profile. TITLE / BUILDS are gone from the script. Work folder:
<NSP folder>/<id>_build/.
- A title with no profile is surveyed instead:
<NSP folder>/<title ID>_survey/survey.md lists the NSPs, modules and
build IDs, SDK version and components, Unity version, engine / API
markers, RomFS folders, magics and extensions, insn_census counts,
imports vs host hooks, and notes. Fire Emblem Engage was surveyed
this way.
- APP_NAME setting; empty uses the game name from the base NSP's
NACP (compressed title areas decoded), then the profile's app_name.
- PACKAGE and OUT_IPA default to the profile's package and ipa.
- Tested on Mario end to end from fresh extraction: the IPA is named
"Super Mario Bros. Wonder" and carries the current chain's tables.
- dm/games/<id>/game.tsv holds what is specific to one game: title and
program ID, known builds (module build IDs, display versions),
package and shader versions, code modules, RomFS marker, zip store
list, memory split, NVN driver constants, shader split groups, app
name / bundle ID / IPA name. Mario's profile is smbw, the default.
- The shader tables, c1_full.tsv, pso_manifest.tsv, bf_want.tsv and
dmsh_chain.h move from dm/gfx/nvn to dm/games/smbw.
- dm/games/game.py loads profiles (DM_GAME, by title ID), answers
shell queries and generates the host's game_profile.h.
- Shader tools (dmsh_chain, dmsh_build, dmsh_facts, shader_split,
tbdr_budget) take tables and constants from the profile. dmsh_build
stops if the profile's texture / sampler bases differ from the
translator's built-in ones. dmsh_chain hashes the profile's nvn_* /
shader_* keys; the chain version is now 02ed42db694f. All 16,266
translated shaders are byte-identical to 5f1ec82dfcf0's, and
c1_full.tsv differs only in its header.
- Host: GAME ?= smbw in the Makefile; memory split, display versions,
NVN bases, default modules, window title and the shader map path
come from the profile. D2R-only code (Settings.json edits, the
main+0x3509a0 read) runs only on D2R's build, which also stops a
stray Settings.json in Mario's iOS save.
- iOS: build_ipa.sh, gen_build.py, project.yml and main.m take the
app name, bundle ID, category, modules, RomFS marker and tables from
the profile (GAME=<id>, workspace ~/<id>-work).
- dm/games/README(.zh-CN).md: the keys and how to add a game.
- nx_extract --manifest: write ExeFS and romfs_manifest.tsv (path,
size, first 32 bytes of every file) instead of the RomFS, so a large
title can be surveyed without writing its game files.
- nx_extract prints the build ID of every NSO module, the display
version and the game name, and saves the raw control.nacp next to
the output for naming the app.
- Game name: newer SDKs compress the NACP title area (u16 size + raw
deflate, inflating to 32 languages x 0x300; Mario 1.2.1 does this).
LibHac 0.19 does not decode it, so nx_extract does.
- nso_syms.py: '-' for the reference game compares one game's imports
with the host hooks only.
- dm/ios/tools/build_ipa_from_nsp.py (Spyder or python3): picks the
base game and update 1.2.1 by title ID (DLC is skipped), extracts
them with nx_extract, checks the main / sdk build IDs, rewrites the
modules, splits the shaders and requires the tables to match the
repo's byte for byte, builds the iOS shader libraries, then runs
build_ipa.sh. It uses only the repo, the NSP folder and prod.keys:
intermediates go to <NSP folder>/smbw_build/ (full step logs in
logs/), and the extracted game files are deleted after success.
Tested end to end: first run 33 min, re-run 10 min.
- dm/gfx/tools/dmsh_xlate_bin: the translator build that matches
dmsh_chain.lock. A fresh dotnet publish gives different dmsh_xlate
and Ryujinx dll fingerprints, which dmsh_build refuses.
- build_ipa.sh: when Xcode's Locations setting is a custom location,
take the app path from the build log instead of stopping at
"no built app".
- README: the script; the custom-location note replaces the by-hand
finish.
- DmGamepad.m: layout after a PS remote-play pad. ZL above L and ZR
above R (teardrops) in the top corners; -/+ in the bottom corners
with the D-pad (four arrow pads) and the A/B/X/Y diamond just above
them; left and right sticks next to those along the bottom edge.
Everything hugs the edges to leave the picture room. Layout
constants are at the top of the file.
- Sticks are fixed rings (tilt from the ring centre, 0.12 R dead zone)
instead of the floating left stick; the new right stick merges with
a physical pad's right stick the same way as the left one.
- One CAShapeLayer per control instead of a full-screen drawRect: a
press swaps the fill, a stick move only moves its knob. Colours
carry low alpha (fill 0.14, outline 0.38, labels 0.6) rather than a
view alpha, which would add an off-screen pass.
- front_ios.m header and the README app-shell line follow.
- README: status for v0 / 1.2.1 on the iPhone, update 1.2.1 build
steps, diagnostic env files, finishing the IPA by hand when Xcode
uses a custom build location, app data folders after a reinstall,
prewarm lists (pso_user.tsv / pso_manifest.tsv).
- AGENTS.md: per-build knowledge keyed by build ID, shader_pairs.tsv on
the chain, unsigned IPAs only.
- docs/agents: update extraction and offset porting (01), the
command-buffer byte, command memory, iOS audio, update imports and
build-gated D2R code (03), vertex output padding and iOS math mode
(04), on-device diagnostics and the offline link check (05).
- dm/gfx README / nvn_metal_mapping (en + zh-CN): pairs table, several
game versions, prewarm lists, DM_PSO_USER, command buffer notes;
corrected shader key, build script and 8-bit index lines.
- dm/host/README(.zh-CN).md removed: an index of d2r-mobile's D2R
features; most files it listed no longer exist.
- dm/tools/nx_extract (C#, LibHac 0.19): hactool stops with "Invalid
BKTR layout!" on the update's patch NCA. nx_extract reads the NCAs
straight from both NSPs, imports their common tickets, merges the
update's RomFS over the base game, prints the display version and
module build IDs, and writes files identical to a base RomFS as APFS
clones (the merged 1.2.1 RomFS adds about 114 MB).
- dm/tools/nso_fnmap.py: carries offsets between builds by matching
eh_frame functions with address bits masked; globals through their
adrp references.
- patch_nso.py: resource heap +16 KB cut for 1.2.1 main ff773e90...
(main+0x1cf9b0, mapped from v0 0x1e62fc; the function is identical
after masking). fixups.tsv now records the module build ID; the
loader keeps it in module_t.build_id.
- hle_oe.c: the 3D render scale inherited from d2r-mobile writes and
calls fixed main addresses (0x31e0848, 0x8582f0). In Mario v0
0x8582f0 is the middle of a static initializer; it only did nothing
because the pointer read 0. Now enabled only on D2R's build.
- hle_misc.c: owned add-on content defaults to none except on D2R's
build (DM_AOC_OWNED overrides).
- New imports in 1.2.1 answered as a Switch 1 without amiibo:
nn::nfp (one NFC device on the connected controller, detection never
finds a tag; nfp::Initialize at boot used to abort in nn::sm),
pl::IsRunningOnOunce (false), ec::ShowShopApplicationInformation-
ForUpgrade (no-op), account::PopOpenUsers (none: the game falls back
to TryOpenPreselectedUser as in v0). oe::GetDisplayVersion by build
(1.0.0 / 1.2.1).
Mac: 1.2.1 and v0 both run the scripted opening for 120 s with only
the known SetRenderEnableConditional red line. iPhone: 1.2.1 played
8.5 minutes with no crash.
Metal builds a pipeline only when every pixel shader input is a vertex
shader output; Maxwell gives an unwritten attribute a default. With
shaders translated one at a time, 331 of v0's 9,043 vertex / pixel
pairs failed to link and later levels lost those draws (about 11,500
failed pipeline builds in one device session).
- shader_split.py writes shader_pairs.tsv (pairs of every BNSH
variation) and takes comma-separated RomFS / main lists for several
game versions, writing their union (keys are content hashes; v0 rows
keep their package indices).
- dmsh_build.py vs_pad(): vertex shaders get the outputs their partners
read but they don't write, float4 (0, 0, 0, 1); vertex shaders with
partners are finished last and their partner list is part of the
cache rule. shader_pairs.tsv is part of the chain version.
- Tables now cover v0 and update 1.2.1: 16,268 shaders (1.2.1
recompiled 3,451 SecredUber pixel shaders and added 6 effect
shaders), 14,603 pairs. Chain 5f1ec82dfcf0: 16,266 compile (the 2
known SPIRV-Cross failures); an offline link test of all pairs gives
14,603 / 0 (v0 before the padding: 8,712 / 331). c1_full.tsv
regenerated (same 11 keys).
The v0 IPA with this padding played 11 minutes on the iPhone with no
pipeline failures.
Fixes the flicker / black band in the opening course and the crash after
the first scene on the iPhone (A20 Pro), confirmed on the device.
- nvn_obj.h: cmd_t leaves byte +0x10 of NVNcommandBuffer to the game.
Mario's inlined NVN code writes 1 there (main+0x4da5c, main+0xb8dec).
It used to hit the low byte of `end`: the game's 512-byte material
setup blocks shrank and overflowed into host segments,
GetCommandMemoryUsed (cur - begin) returned 0, the game marked the
setups empty and skipped calling them, and draws ran with the previous
vertex buffers (out-of-range vertices, NaN).
- hle_nvn.c: GetCommandMemoryUsed counts bytes across host segments plus
16 for the jump. Host segments are released only when the game
re-grants memory overlapping the recording's guest bytes (the game
keeps calling recorded setup blocks for many frames); the 8-frame
recycling is gone.
- nvn_mtl.m: failed pipelines are cached and reported once with their
vertex / fragment keys instead of being rebuilt on every draw.
- front_ios.m / main.m: the audio session asks for a 5 ms I/O buffer at
boot and on return to the foreground. The default (~21 ms, 1024
frames per callback) outran the 4 x 5 ms the renderer keeps queued:
every callback underran (crackling). Device: 240-frame callbacks,
0 underruns.
Still contains [DEBUG-flk1] instrumentation (tagged, mostly behind
DM_DBG_* switches; audio statistics every 2000 callbacks in
hle_audio.c). To be removed with one grep.
Temporary instrumentation for the A20 Pro NaN investigation; every line
is tagged [DEBUG-flk1] so it can be removed with one grep.
- DM_DBG_TEXSTAT=N with DM_NVN_TEXDUMP=<frame>:<path>: for N consecutive
frames, log one line per colour target instead of writing PNGs:
per-channel mean, NaN / Inf counts, largest finite value, black share.
- Inside the DM_NVN_PASSDUMP window: each submission's commit->completion
time and the counter values written back (timestamps / samples passed).
- DM_NVN_SHOT accepts up to 4096 frames (was 16).
Device capture used so far (unsigned IPA, DM_ENV_FILE for build_ipa.sh):
DM_NVN_TEXDUMP=3600:td_%d.png
DM_DBG_TEXSTAT=2
DM_NVN_PASSDUMP=3600-3601
DM_NVN_SHOT=3600,3601,3602,3603:shot_%d.png
DM_NVN_SHOT_OUT=960x540
Opening course on an iPhone 18 Pro Max (A20 Pro GPU): black diagonal
band, black-edged white glow at the horizon, grey striped blocks,
alternating frame by frame. The Mac (M4 Pro) never shows it, not even
with the iOS metallibs, display pacing, -Os, DM_3D_SCALE=0.75 or the
no-BC path.
On-device colour-target stats (DM_DBG_TEXSTAT, diagnostic commit that
follows): even frames' main 1920x1080 RG11B10 target has 54 NaN pixels
(Mac: 52); odd frames 727k NaN, and the 960x540 / 1100x280 / 275x70
targets downsampled from it carry it on. The NaN is already in the main
target before the first downsample (pass 47 of 80), so it comes from
the scene passes, not the post-effect chain.
The translated shaders reproduce Maxwell's NaN rules with
select(x, 0, isnan(x)) (8337 / 12809 shaders). Default fast math
records air.compile.fast_math_enable, which lets each GPU's backend
compiler assume no NaN / Inf; relaxed keeps the reassociation /
contraction but preserves NaN / Inf. Hypothesis only: chain
1b46c9a2e920 (MSL byte-identical to b1a4876b9511; c1_full.tsv lists
the same 11 shaders) still has to be run on the iPhone. If the NaN
persists, the next step is finding the producing draw in passes 12-45.
dmsh_chain.py --update-lock on a fresh Mac (Xcode 27.0, metal 32023.921
unchanged): the translator DLLs built here and Homebrew's spirv-cross
have different fingerprints. Chain b1a4876b9511; dmsh_build gives the
same result as documented (12809 / 12811 compile, the same 2 agl compute
shaders make SPIRV-Cross throw) and c1_full.tsv lists the same 11 shaders.
Another machine with other tool builds needs --update-lock + --header.
A sideloaded app cannot be given environment variables from outside.
main.m now reads KEY=VALUE lines from Documents/env.txt and from
dm/env.txt in the bundle (Documents wins, real environment wins over
both) and logs each applied line as "boot: 环境 ...".
build_ipa.sh copies DM_ENV_FILE=<file> into the bundle as dm/env.txt.
Ranges are rounded to 16 KB host pages but NVN pools are only 4 KB
aligned, so adjacent pools share their first / last page and one
address can fall in several registered ranges. Every lookup took the
first range containing the address:
- textures whose pool's first page is shared with a smaller preceding
pool were judged untracked and re-uploaded on every use (two ASTC 6x5
textures in the opening course: 12 re-uploads, about 6 MB, per frame);
- a fault or host pre-write on a shared page released only one range,
leaving the other marked protected;
- destroying a pool could unregister its neighbour (same rounded base).
Tracked / protect / since now use the range that holds the whole span;
faults and host pre-writes release every range covering the page; pool
removal matches the exact pool start. Opening course: 3600 re-uploads
per 300 frames -> 0; DM_NVN_DIRTYCHECK=1 over 3900 frames: 0 misses.
t_dirty_slots covers the shared-page case (3 checks fail on the old code).
- Audio: run the sdk's own audio renderer on the CPU in manual mode (hle_audren.c),
feed its DeviceSink output to the AudioOut sink; catch-up render on back-to-back
updates fixes the ApplyLightLimiterEffect crash at the title screen
- Button icons: make_ext_font.py generates a stand-in Nintendo extension font
(A/B/X/Y, L/R/ZL/ZR/SL/SR, +/-, D-pad, stick arrows); served as shared font type 5;
the iOS bundle now carries both shared fonts
- Fountain panels: Metal 4 does not order copies inside one encoder; add barriers
when ranges overlap, so the GPU wave simulation no longer reads a stale uniform
- Zero-fill new colour textures; scissored zero clears; uniform-update snapshots
- Docs: graphics ordering, audio, diagnostics, intermittent-glitch method
- dm/ios: lean app shell (main.m boot/env/log, front_ios.m front.h + 16:9 Metal view,
DmGamepad.m touch gamepad in Nintendo positions + GameController), xcodegen project,
tools/build_ipa.sh (embedded game code, iOS shader libraries, RomFS in the bundle)
- Guest address space back to 5 GB (heap 1.625 GB; the game asks for 1.47 GB)
- README / agent guide: iOS build steps and address-space sizing
- dmsh_build.py: vertex shaders whose inputs are only used as integer bit casts are
re-translated with uint inputs (529 shaders); bone indices were converted to floats and
every character was stretched off screen
- Host: Metal vertex formats follow the shader input type (signedness swap, 32-bit float
reinterpretation), mismatches reported
- LINEAR_RENDER_TARGET textures in CPU-visible pools are zero-copy too (sky colour ramps)
- Pass dump lists all vertex attributes; docs and README updated
- patch_nso.py: Mario main cut gives every resource heap +16 KB; Model/NpcCommonerA.bfres
came up 0x70 bytes short of its size-table budget while loading Course900
- dmsh_build.py: compute instructions start at code area +0x100, not +0x30; the misaligned
decode turned 12 particle / post-effect compute shaders into endless loops (GPU hang)
- Linear textures in CPU-visible pools are built on the pool buffer, so the CPU-rasterized
glyph cache reaches the GPU (menu and dialogue text)
- Host command-memory segments grow on demand; RGB10A2SN vertex format; instanced DrawArrays
- DM_PROBE: thread IDs, register-relative dumps, hit filters, ExpHeap walker
- dm/tools/ghidra/Decompile.java; agent guides and README updated
- DM_KEYS scripted controller input ("seconds:button[+button][~hold]")
- NVN virtual memory pools aliased onto their physical pool when every
MapVirtual request shares one base (Mario's 128 MB pool); previously
MapVirtual failed and the game treated the pool as unusable
- DM_PROBE ported from d2r-mobile (host/probe.c): brk probes with
register / backtrace output, plus DM_PROBE_X2 object dumps; used
because lldb launches now wait on a macOS authorization prompt
- diagnostics: DM_NVN_VLOG (pools, MapVirtual), DM_NVN_CBLOG (command
memory grants and callbacks), DM_NVN_ALLOCPROF
- docs: new failure patterns and the probe recipe
Open: starting a new game crashes while loading the opening course
(model resource heap overflow for NpcCommonerA.bfres).
Shaders
- shader_split.py (new): scan RomFS and main for every BNSH and agl BAHS
container (bfsha, particle effects, layouts, agl/gsys packs, main):
12,811 shaders, map + compute facts from the control section
- key = sha256(control + code): many shaders share a control section
- NVN 55 binding layout: bound textures at 0x20 + 8*slot, images 0x120
- bindless textures read from uniform buffers (tex_ubo / samp_ubo)
- GL depth mode: translator remaps z' = z/2 + w/2 in vertex shaders
- compute code areas carry a 0x30 header (magic 0x12345679)
Replay
- 8-bit index buffers widened to 16-bit
- texture swizzle through Metal swizzled views
- non-packaged textures shared by storage (fixes a ~13 MB/frame leak
from transient render targets rebuilt every frame)
- view_t packed to 16 bytes (SDK 16 NVNtextureView; fixes a stack smash)
- TBDR optimizations off by default until they know bindless readers
Host
- alias region 2.875 GB for the 1.75 GB VirtualAddressMemory reservation
- ideal core -2 resolves to the process default core
- shared font served with a BFTTF "decoded" header
- crash reports scan the stack and dump the popped frame
Docs: docs/agents (guides for porting other titles), AGENTS.md
- patch_nso.py: fix exclusive-window exit order. The fall-through stub
must come right after the copied window; it was emitted last, so a
successful stxr in 'ldxr; cmp; b.ne out; stxr; cbnz' fell into the
branch-out stub (main 0x81c40: job-system enqueue retried forever,
tail filled to capacity with no task stored). Bug inherited from
d2r-mobile, latent there (no branch-out windows).
- patch_nso.py: rewrite 'mrs xN, ctr_el0' (trapped on macOS) to a
constant with 64-byte data cache lines.
- hle_thread.c: SetThreadCoreMask moves the ideal core into the new mask
(Horizon behavior), so pinned threads report the right core.
Frames 2 and 3 now record, submit and present (~1,900 NVN calls).
Next: Mario's shaders (bfsha) are not in the translation map yet.
- BindTexture (combined handles go straight into the texture slot; the
replay already splits texture / sampler ids), BindImage,
BindTexturesDeferred, BindVertexBuffer, CallCommands
- DrawElements / BaseVertex / ArraysInstanced; viewport, scissors and
depth ranges in single / multi forms
- ReportValue via the event-write record; new OP_CLEAR_BUF record with
in-order replay (fillBuffer, or a pattern buffer for non-uniform values)
- TextureWriteTexels: deswizzle level, patch region, swizzle back, after
lifting write protection
- Command memory: when the game's callback cannot supply enough, continue
in a host segment instead of aborting
- Remaining state the replay lacks (conditional render, depth clamp,
rasterizer discard, ...) is logged once and drawing continues
The game now runs its frame without errors from these; the main thread
waits in the job system's fork-join (executors keep polling).
- hle_svc.c: guest AS 6 GB with a 1.25 GB alias region (newer SDK's
AddressRegionManager always takes ~1 GB of it as its node area),
MapPhysicalMemory rounded to 16 KB host pages, GetInfo(UserExceptionContext)
backed by a real buffer, program id now SMBW (DM_PROGRAM_ID overrides)
- hle_audren.c (new): silent AudioRenderer, event fired every 5 ms
- hle_fs.c: SetPriorityOnCurrentThread / IsEnabledAccessLog (fsp-srv)
- hle_sdk16.c: settings language comparison; diag aborts report the call
site and backtrace
- NVN: nvnProgramSetShadersExt (0x18-byte stage entries)
The game now runs ~40 threads; the main thread waits on job events while
render commands (CallCommands, BindTexture, BindVertexBuffer, DrawElements,
WriteTexels) are still unimplemented.
seedds