- Scripts the user runs now live in scripts/ at the repository root;
build_ipa_from_nsp.py moves there from dm/ios/tools (history kept).
AGENTS.md records the convention.
- It still runs in Spyder (F5, settings block, prompts for an empty
NSP folder or prod.keys). From a terminal it now takes options that
override the settings: <NSP folder>, -k prod.keys, -p version,
-o out.ipa, --app-name, --keep-extracted, --env-file (-h for help).
Without keyboard input a missing value is an error instead of a hung
prompt. Exit codes 0 / 1 / 130 in a terminal; inside IPython (Spyder)
it does not call sys.exit, so the console stays clean.
- The repository root is found by walking up to dm/tools/nx_extract
rather than a fixed depth.
- Checked: -h from another directory; error paths (missing folder, no
terminal input, empty folder, unregistered version) exit 1 with a
clear message; a CLI run on the Engage NSP folder identifies the game,
skips DLC, picks base + update, passes the tool check and stops at
the free-space check; IPython %run under the Spyder interpreter
(prompts, -h, an error) keeps the console alive.
- README, dm/games READMEs and docs/agents point at the new path.
Ported from d2r-mobile (b94670c79, 2eb575877, 704147218), adapted:
- Save transaction: if the work directory cannot be set up at mount
(leftovers not removable, clone fails, e.g. a full disk) the host
now mounts the committed saves read-only and tells the front end
instead of exiting; a reclone after a refused commit clones beside
the work directory and swaps it in atomically, and on failure keeps
it tainted for the next try; a failed rename rollback refuses the
transaction instead of exiting. Creating an existing file or folder
on a read-only mount answers "already exists", and file / directory
calls ignore the null handle a failed open leaves. Test switch
DM_TEST_FS_FAILAT; tests/t_fs.c section 6 (68 of 68 pass). An Engage
run with the mount clone failing loads its autosave read-only and
plays on.
- iOS front end: front_save_failed shows a bilingual notice (it only
logged before).
- thread.c: each guest thread's TLS block, thread-local region and
thread object (9.5 KB) are freed at thread exit through a key of
their own (the guest overwrites the TLS / TLR key slots).
tests/t_thread_leak.c: 600 blocks leaked before, none now.
- Memory line on iOS: os_proc_available_memory, how far the app is
from being killed for memory.
- The host test Makefile gets the game profile include path; t_fs
stubs the module table. Docs: 03-bringup memory line fields,
05-debugging save failures. Mario regression t78 and the iOS compile
are clean.
- Ported from d2r-mobile ef9b1ff13, adapted to our shared-page ranges:
setting a page read-only marked it PROT before mprotect landed, so a
concurrent unprotect-before-write could set it back to NONE and then
the mprotect made it read-only anyway. A game thread writing it then
spun forever in the fault handler, and a host system call into it
failed with EFAULT. Now NONE -> BUSY -> mprotect -> PROT (and back),
waiters re-check after BUSY clears, and the wait is capped at 2 s
with a red line. Engage's movie decoder writes frames from a host
thread while replay protects pages; this is a likely cause of the
old early stalls.
- tests/t_dirty_race.c (ported) reproduces it deterministically: on
the old tracker read() into the page fails with EFAULT; it passes
now, and t_dirty_slots still passes.
- The graphics unit tests had stopped linking (prog_key moved to
hle_nvn.c) and some crashed (hand-built textures lacked the texture
magic replay now checks; t_nvn used a function that has since been
implemented as its unimplemented example and hard-coded 523 names).
Fixed: t_nvn, t_fuse_check, t_zskip, t_xv, t_ub, t_pfv, t_present_crop,
t_rt_depth_ro, t_rt_range_shape, the dirty tests and others pass.
Still failing, the same before this change: t_reorder_chain (30 of
155), t_defer_chain (1 of 59), t_mip_extent (1 of 303), t_rt_range,
t_rt_range_ro; t_refl needs D2R shader samples.
- Mario regression t77, an Engage movie / title / demo run (0 missed
dirty pages) and the iOS compile are clean.
- The storage-sharing texture table (g_alias, keyed by pool object,
offset and description) never evicted. Engage frees memory pools on
scene changes and builds new ones, so the old pools' render targets
and other private textures stayed for good, and linear textures built
on a pool buffer kept the whole buffer alive. A reused pool object
address could also hand an old texture to a new pool. Finalizing a
pool now evicts its entries (mtl_alias_drop_pool) through deferred
release.
- With 21b3bc3, Engage's 850 s idle demo stays at 1.2 to 1.4 GB of GPU
memory (5.4 GB before both fixes) and ends at 2.5 GB resident instead
of 3.6 GB; in the prologue battle GPU memory is 1.07 GB at frame
11100 (2.38 GB before).
- The memory report line adds the host objects held by serial, the
finalized pool buffers still alive, and the sharing table size with
its evictions.
- Mario regression t72 (it does finalize pools: 32 evictions; same
scene, 60 fps, GPU memory unchanged), Engage demo scenes and combat
are clean; the iOS project compiles. Docs: 04-graphics entry.
- Measured over a 12-minute run: StandardAllocator commits only 0.58 GB
of the 3.14 GB it reserves, Unity's own regions 0.97 GB. The profile
now reports a 2.5 GB pool (StandardAllocator about 2.44 GB) and uses
a 5 GB block with a 4.375 GB alias region, the total proven on a
phone. The old 6 GB block was iOS's single-reservation limit and left
Metal no address space. Reserved address space peaks at 3.47 of
4.31 GB; a 670 s battle-script run is clean.
- The early stalls (one run in four) have not come back in 12 runs
since the real thread pause for IL2CPP's GC; the docs mark them
probably fixed and keep the sampling recipe.
- Docs: sizing entry with the commit split and how to measure it,
profile notes, README Engage row (it reaches the battle map; the
scripted presses end on the save screen, not in combat).
- The prologue battle crash is marked fixed, with a pointer to the
address-space entry in 03-bringup (missed in fe9bab9).
- The early stalls get their own open entry, with the sampling recipe
and the new stderr-lock line from the crash handler.
- crash.c: before printing a crash or time-limit report, the handler
tries the stderr lock for 1 s. If another thread holds it, it writes
one line (signal, pc / lr as module offsets) with write() and exits,
instead of hanging with the process and leaving an empty log. This
is for the intermittent early stalls, where even the SIGALRM report
never appeared.
- Engage's idle attract-demo crash (about 220 s) was the same alias
address-space shortage as the battle crash: the particle index
buffers got no mapping past 2 MB. With the node-area cut (fe9bab9) an
850 s idle run is clean, with no corrupted texture objects; with the
old SDK the same run dies at 159 s in Unity's heap allocator.
- Docs: the 04-graphics entry is marked fixed with the cause; 03-bringup
and the README Engage row updated. Mario regression t65 clean; the
iOS project compiles.
- The crash about 6 minutes into the prologue battle (null write at
main+0x9a4664) was the alias region running out of address space, not
memory: Unity's low-level virtual allocator got no region from
AllocateAddressRegion and returned 0. The SDK's AddressRegionManager
keeps the first 1 GB of the alias region as its node area and aligns
each region by size; after StandardAllocator's 3.14 GB block only about
1.2 GB was left for Unity, and it was full and fragmented.
- patch_nso.py: a host-only cut for Engage's sdk build (56ec2474…)
shrinks the node area from 0x3fe00000 to 0x3e00000 (62 MB, about 1.35
million nodes). Both constructor movs change together.
- A 12-minute scripted run now plays through the battle without a crash;
reserved address space levels off at 4.17 of 5.31 GB.
- hle_svc.c: the alias-mapped total counts pages with a bitmap, so maps
of already-mapped pages and oversized unmaps no longer skew it (it went
negative). Engage really maps about 1.5 GB, not 0.5 GB.
- Docs: bring-up catalogue entry with the measuring recipe, profile
memory notes, README Engage row. Mario regression t64 clean; the iOS
project compiles.
- SetThreadActivity and GetThreadContext3 were placeholders that only
reported success. IL2CPP's garbage collector pauses each thread and
reads its registers for roots, so threads kept running and roots held
in registers were missed. A thread inside an SVC now counts as paused
at once with the gate frame as its context (thr_svc_enter / leave
around every gate SVC; it waits at the SVC exit while paused). A
thread in guest code is stopped with SIGUSR2 and parks in the handler.
The context is Horizon's 0x320-byte ThreadContext.
- nn::pctl::IsFreeCommunicationAvailable answers "not available" like
the other free-communication calls; the SDK aborted in sm when Engage
entered its first battle map.
- Memory log lines: heap growth, alias-region high-water mark and the
total mapped in the alias region (one line per 256 MB).
- Engage profile: a 6 GB reservation (5.375 GB alias region, 128 MB
heap region), iOS's single-reservation limit without the extended
virtual addressing entitlement. Engage never calls SetHeapSize; its
mappings reach +5.26 GB.
- Engage on the Mac: with scripted input and DM_KBD answering the name
keyboard, a new game reaches the prologue cutscene and first battle
map. Open: a heap-allocator crash a few minutes into that battle, and
occasional early stalls (docs/agents/04-graphics.md).
- Docs: catalogue entries, README rows. Mario 1.2.1 regression (120 s
scripted) clean; the iOS project compiles.
- crash.c: only the first crashing thread prints its state; later ones
wait in pause(), and a 15 s alarm ends the process if the dump stalls.
Unity's job threads crash together, and the dumps used to interleave
and leave the process hung.
- NVN render target, texture copy and clear records now carry a copy of
each texture object (REC_SNAP / tex_snap) instead of the game's
pointer, as NVN commands do not depend on the API object once
recorded. Replay rejects objects without the texture magic and sizes
beyond Metal's limits with a red line (a 1869367126-high texture used
to abort in Metal's validation).
- SignalToAddress type 2 with count <= 0 and waiters present now
subtracts 2 (mesosphere, Ryujinx and yuzu agree); it subtracted 1.
- Engage on the Mac: scripted input reaches the main menu and a new
game's appearance choice. The idle attract demo still crashes after
about 220 s (Unity's particle geometry job overruns its index buffer);
findings are in docs/agents/04-graphics.md.
- Docs: catalogue entries, README row. Mario 1.2.1 regression (120 s
scripted) clean: only the known SetRenderEnableConditional red line.
- hle_movie.c implements movie::Decoder (the SDK's MediaCodec wrapper)
through hooks on every vtable method: DecoderEvents signalled with
the SDK's SignalEvent, input / output index queues, Buffer and
MediaData filled with the SDK's own setters, one NV12 frame per
packet into the buffer Unity allocates (w x h x 1.5); packets wait
while every output slot is taken. Frames are black for now (no VP9
decoding yet); the size comes from Configure or the VP9 header.
- Engage on the Mac: loading screen, the 94 s opening movie, then the
title screen and, left idle, the 3D attract demo. Not yet: input,
a crash about 221 s into the demo, an unmapped texture format, iOS.
- Docs: catalogue entries (movie protocol, shader keys, CPU-written
textures, NVNprogram size), Engage graphics notes, README row.
- New dm/gfx/tools/unityfs.py: UnityFS bundles (LZ4 / LZMA blocks),
serialized files (formats 19..22), objects read by their type trees
(Unity's common string table comes from the game's own main), and a
Shader object's per-platform compressed subprograms. The player's
own files have no type trees and borrow the bundles' Shader tree of
the same type hash.
- shader_split.py reads Unity Shader objects in a second pass: each
Switch subprogram is one NVN program (0x60-byte header of per-stage
offset / control size / code size, then control + code per stage);
a program's vertex / pixel stages form a pair. Engage: 9,269
shaders, 4,634 pairs. Mario's tables and package are byte-identical.
- Engage profile: shader groups engine / assets, measured NVN layout
(texture handles 0x168 + 8 r, 128 slots; separate samplers 0x568),
regenerated tables, chain a295a8035b7b. 9,268 of 9,269 compile (one
geometry shader; Metal has none).
- dmsh_build.py: the translator's built-in bases only matter with
per-slot facts, so the guard checks that; c2 reads count the sampler
table and 16-byte rows overlapping a table as fillable.
- game.py: DM_GAME_NVN_SAMP_BASE. NSP script survey note updated.
- Mario chain is now 271056163ea2 (Mac and iOS libraries rebuilt,
16,266 of 16,268 as before; c1_full regenerated).
- patch_nso.py: three more x18 forms from IL2CPP code: an offset jump
table (targets read at rewrite time), blr x18 (original return
address kept for C++ unwinding) and an sp write from x18 (alloca).
Mario's rewrite output stays byte-identical.
- loader.c: symbols resolve globally in load order, also for a
module's references to its own exports (local / non-default
visibility excepted). Engage's main overrides operator new / delete;
sdk bound its own copies and freed Unity's blocks into the SDK heap.
- hle_svc.c: QueryMemory reports each module's Horizon layout (code,
read-only, data, patch area) and other host memory as free, which
the SDK's C++ unwinder (nn::ro::detail::GetExceptionInfo) needs.
- New takeovers: nn::fs::IsMounted, nn::ro (hle_ro.c), prepo
Initialize, SDK 12 audio device names and OpenDefaultAudioOut, an
SDK 12 hid block (single GetNpadState, modes, touch, gestures,
six-axis details, colors, applets), movie::Decoder with no impl
(hle_movie.c) and failing nvdrv opens (hle_nvdrv.c), and NVN getters
/ no-ops Engage calls. nvn.h: TextureGetMemoryPool returns a pointer.
- Makefile: a game stamp, so changing GAME rebuilds the host.
- docs/agents/03-bringup.md: catalogue entries for all of the above.
- Mario 1.2.1 scripted run still reaches the opening course with the
same single red line.
- attach(): a render-target view that changes the format is attached
as a Metal view of that format (kept until the submission completes),
so pipeline formats follow it. Mario's last pass writes the RGBA8
swapchain texture through an RGBA8_SRGB view; with the base texture
attached, linear values were stored unencoded and the whole picture
was darker: on the Mac and the iPhone every channel equalled the
Switch value decoded from sRGB once more (0.55 -> 0.26). After the
fix the Mac title frame matches a Switch screenshot (mean RGB
0.556 / 0.621 / 0.633 vs 0.557 / 0.621 / 0.633).
- present_raw_src(): an sRGB-typed swapchain texture is sampled through
a non-sRGB view, so the stored bytes reach the screen as on the
Switch's scan-out (no change for Mario, whose swapchain is RGBA8).
t_present_crop gets an sRGB case (34/34; fails without the fix).
- docs/agents/04-graphics.md and the mapping docs: the rule and how it
was measured.
- hle_nvn.c: drop the command memory callback / grant / caller
logging, the overflow details, the +0x10 write log, the one-off
NVN table dump, and the DM_DBG_GRANTS, DM_DBG_CALLSITES,
DM_DBG_CMDMEM_SHRINK and DM_DBG_OLD_USED switches.
- nvn_mtl.m: drop the per-draw NaN probe (DM_DBG_NANPROBE), the
colour target statistics (DM_DBG_TEXSTAT) and the PASSDUMP
completion timing.
- Kept as regular diagnostics: the audio callback line every 2000
callbacks (frames per callback, queued blocks, underruns) and
DM_DBG_AU_FRAMES (docs/agents/03-bringup.md, crackling on iOS);
DM_NVN_SHOT up to 4096 frames (the 05-debugging burst recipe).
- Mac 1.2.1 scripted run reaches the opening course with the same
single red line as before.
- README: presented as Switch games running natively on Apple
platforms, with Mario as the one supported game. A games table
(Mario playable, Fire Emblem Engage surveyed), component status,
game profiles and how to add a game, the NSP script, generic
step-by-step build commands (profile, workspace and version as
variables), and a Mario section for the per-game notes (1.2.1).
- AGENTS.md: profiles, DM_GAME / GAME, workspaces, build-ID gating,
the chain rule per profile.
- docs/agents: one repository with one profile per title; survey
mode in 02-survey, including the compressed NACP title area.
- dm/gfx README (+ zh-CN): table paths in the profile.
- README: status for v0 / 1.2.1 on the iPhone, update 1.2.1 build
steps, diagnostic env files, finishing the IPA by hand when Xcode
uses a custom build location, app data folders after a reinstall,
prewarm lists (pso_user.tsv / pso_manifest.tsv).
- AGENTS.md: per-build knowledge keyed by build ID, shader_pairs.tsv on
the chain, unsigned IPAs only.
- docs/agents: update extraction and offset porting (01), the
command-buffer byte, command memory, iOS audio, update imports and
build-gated D2R code (03), vertex output padding and iOS math mode
(04), on-device diagnostics and the offline link check (05).
- dm/gfx README / nvn_metal_mapping (en + zh-CN): pairs table, several
game versions, prewarm lists, DM_PSO_USER, command buffer notes;
corrected shader key, build script and 8-bit index lines.
- dm/host/README(.zh-CN).md removed: an index of d2r-mobile's D2R
features; most files it listed no longer exist.
- Audio: run the sdk's own audio renderer on the CPU in manual mode (hle_audren.c),
feed its DeviceSink output to the AudioOut sink; catch-up render on back-to-back
updates fixes the ApplyLightLimiterEffect crash at the title screen
- Button icons: make_ext_font.py generates a stand-in Nintendo extension font
(A/B/X/Y, L/R/ZL/ZR/SL/SR, +/-, D-pad, stick arrows); served as shared font type 5;
the iOS bundle now carries both shared fonts
- Fountain panels: Metal 4 does not order copies inside one encoder; add barriers
when ranges overlap, so the GPU wave simulation no longer reads a stale uniform
- Zero-fill new colour textures; scissored zero clears; uniform-update snapshots
- Docs: graphics ordering, audio, diagnostics, intermittent-glitch method
- dm/ios: lean app shell (main.m boot/env/log, front_ios.m front.h + 16:9 Metal view,
DmGamepad.m touch gamepad in Nintendo positions + GameController), xcodegen project,
tools/build_ipa.sh (embedded game code, iOS shader libraries, RomFS in the bundle)
- Guest address space back to 5 GB (heap 1.625 GB; the game asks for 1.47 GB)
- README / agent guide: iOS build steps and address-space sizing
- dmsh_build.py: vertex shaders whose inputs are only used as integer bit casts are
re-translated with uint inputs (529 shaders); bone indices were converted to floats and
every character was stretched off screen
- Host: Metal vertex formats follow the shader input type (signedness swap, 32-bit float
reinterpretation), mismatches reported
- LINEAR_RENDER_TARGET textures in CPU-visible pools are zero-copy too (sky colour ramps)
- Pass dump lists all vertex attributes; docs and README updated
- patch_nso.py: Mario main cut gives every resource heap +16 KB; Model/NpcCommonerA.bfres
came up 0x70 bytes short of its size-table budget while loading Course900
- dmsh_build.py: compute instructions start at code area +0x100, not +0x30; the misaligned
decode turned 12 particle / post-effect compute shaders into endless loops (GPU hang)
- Linear textures in CPU-visible pools are built on the pool buffer, so the CPU-rasterized
glyph cache reaches the GPU (menu and dialogue text)
- Host command-memory segments grow on demand; RGB10A2SN vertex format; instanced DrawArrays
- DM_PROBE: thread IDs, register-relative dumps, hit filters, ExpHeap walker
- dm/tools/ghidra/Decompile.java; agent guides and README updated
- DM_KEYS scripted controller input ("seconds:button[+button][~hold]")
- NVN virtual memory pools aliased onto their physical pool when every
MapVirtual request shares one base (Mario's 128 MB pool); previously
MapVirtual failed and the game treated the pool as unusable
- DM_PROBE ported from d2r-mobile (host/probe.c): brk probes with
register / backtrace output, plus DM_PROBE_X2 object dumps; used
because lldb launches now wait on a macOS authorization prompt
- diagnostics: DM_NVN_VLOG (pools, MapVirtual), DM_NVN_CBLOG (command
memory grants and callbacks), DM_NVN_ALLOCPROF
- docs: new failure patterns and the probe recipe
Open: starting a new game crashes while loading the opening course
(model resource heap overflow for NpcCommonerA.bfres).
Shaders
- shader_split.py (new): scan RomFS and main for every BNSH and agl BAHS
container (bfsha, particle effects, layouts, agl/gsys packs, main):
12,811 shaders, map + compute facts from the control section
- key = sha256(control + code): many shaders share a control section
- NVN 55 binding layout: bound textures at 0x20 + 8*slot, images 0x120
- bindless textures read from uniform buffers (tex_ubo / samp_ubo)
- GL depth mode: translator remaps z' = z/2 + w/2 in vertex shaders
- compute code areas carry a 0x30 header (magic 0x12345679)
Replay
- 8-bit index buffers widened to 16-bit
- texture swizzle through Metal swizzled views
- non-packaged textures shared by storage (fixes a ~13 MB/frame leak
from transient render targets rebuilt every frame)
- view_t packed to 16 bytes (SDK 16 NVNtextureView; fixes a stack smash)
- TBDR optimizations off by default until they know bindless readers
Host
- alias region 2.875 GB for the 1.75 GB VirtualAddressMemory reservation
- ideal core -2 resolves to the process default core
- shared font served with a BFTTF "decoded" header
- crash reports scan the stack and dump the popped frame
Docs: docs/agents (guides for porting other titles), AGENTS.md
seedds