Drop unused normal-pass colour-target blend and mask state from pipeline keys.
Normalize historical prewarm rows and deduplicate their parsed keys before
compilation, preserving merged passes' logical target state.
Include production-path GPU regression coverage and the reference/game notes.
Validated on the latest main: Wonder host build, key serialization tests,
48 pending-pipeline checks, 48 archive checks, 158 prewarm GPU checks across
MTLDevice and Metal 4, and documentation checks. The regression fails on the
original base and detects wrong pixels if the merged-pass guard is removed.
Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
Exercise historical manifest rows through the production warmer, shader loader,
pipeline builders and draw-time cache. Check normal and merged passes on both
MTLDevice and Metal 4, with GPU pixels compared against the original descriptors.
The regression fails on the PR base and when the merged-pass guard is removed.
Document the shared normalization rule and keep game measurements in their notes.
Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
Return the fixture profile from the rewrite mock so the preparation pipeline keeps a valid game profile after the Portal change.
Validation: all three tests in dm/ios/tests/test_xcode_workflow.py pass; git diff --check passes.
- profile 01007BB017812000 (1.0.3, SDK 13.3.0): plays on the Mac at 60 fps
- provisional profiles add the subsdks the NROs import (profile_guess nro_modules)
- an NRO importing nvnBootstrapLoader makes an NVN game; NROs calling glslcCompile get the run-time GLSLC capture
- shader_split reads zip archive members (Portal's nxcontent/nvsb GLSLC outputs)
- ro::LookupModuleSymbol for a name a loaded module lacks is not red (Horizon fails it too)
- AGENTS.md: a Mac gameplay test is enough to count a game as playable (65)
Keep the latest upload-staging and compact-binding fixes from main. Add a real-Metal readback lifetime regression and reference documentation for all three general fixes.
Validation: nine Metal test runs pass, including direct and deferred upload-budget tests; Wonder host build, 133-file iOS compile check, and documentation checks pass.
Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
Split GPU upload batches at 128 MiB and retire overflow and CPU staging when the upload completes, rather than retaining it until the main rendering fence. Scope weak-reference diagnostics to avoid temporary staging retention.
Add a real-Metal regression covering memory bounds, upload pixels and an open render pass in direct and deferred encoding modes. Both modes pass 261 checks; documentation checks and the TotK host build pass.
Refs #147. Reporter-device validation and the full-device reboot investigation remain open.
pso_key_norm (66abbc3) drops the blend factors of targets that do not blend. It now also clears, outside a merged pass, the write mask and blend-enable bit of a colour target the pass has no format for: pso_desc reads them only for targets with a format, and they keep whatever earlier draws left. A merged pass keeps them, as its blend / mask index logical targets through lmap. MacNativeGames f3e2f66 found 35% of a Tears of the Kingdom prewarm list's render pipelines (8,479 of 24,146) differing only in such state, each compiled again.
pso_key_norm moves to nvn_pso_skey.h with the merged flag, and pso_skey_parse normalizes, so a prewarm list recorded before gives one key per pipeline; the prewarm takes each parsed key once (the line check alone let such rows through as different lines). t_pso_skey: keys round-trip normalized, two keys differing only in unused blend / mask state parse equal, a merged pass keeps the mask and blend bits. t_pso_archive gives every target a format, so its normalization checks still test the blend rule alone.
- mtl_read_texels runs on the game's own thread outside a submission, where no pool drains what Metal autoreleases (the command buffer, the blit encoder, the read-back buffer), and that thread lives for good: every GPU read-back stayed alive. It runs under an @autoreleasepool now. MacNativeGames 635fb31 measured Breath of the Wild's ~130 read-backs a second holding 3.7 GB of GPU memory after 30 minutes.
- arena_alloc's buffer out-parameter is __strong: an unqualified id * is __autoreleasing under ARC, which put the arena buffer in the autorelease pool at every call (a few per draw). The arena also keeps its length, so an allocation, arena_new and the idle trim send no message. MacNativeGames 0531f28: autorelease 2.63% -> 1.73% of replay time.
- The texture / sampler registry lock g_rmu is an os_unfair_lock, which lends a waiter's priority to its owner. Game threads (priority 46) take it to register textures and samplers, the submit thread (97) for every argument encode; preempted while holding the mutex, they kept the submit thread waiting 1.8 ms a frame in a Tears of the Kingdom shrine (MacNativeGames 5f5de5e: 155 -> ~0 ms/s of waiting, frame time unchanged there). t_stencil_view, t_view_levels and t_view_swizzle take the lock the same way.
Apply the generalized image-atomic bounds and backing checks from PR #178.
Also reject unsupported descriptor formats before integer fallback and fail
unresolved bindless atomics instead of silently deleting them. Rebuild the
translator and record chain 467efdfd0b62, with regression tests and findings.
Validation: compiler/helper and sanitizer tests, 96 Metal compile/link checks,
ten GPU probes, all 28,603 Z-A shaders for Mac and iPhone, and a 240-second
Mac station-area run reaching 6,987 frames without a freeze or GPU fault.
Reported iPhone cases and the station-to-world transition remain unverified.
Co-authored-by: b4f8abbb24dbf56a7d0280f296e16eeb <51352400+b4f8abbb24dbf56a7d0280f296e16eeb@users.noreply.github.com>
Integrate the Wonder 1.0.1 resource-heap fix and shared opt-in function matching for compatible game updates.
Validated with 95 tool tests and byte-identical Wonder 1.2.1 rewrite outputs for registered and synthetic unregistered build IDs.
Co-authored-by: Xc1Ym <29765332+Xc1Ym@users.noreply.github.com>
docs/development.md: a new section "A Mac app" after "A Mac-only build for development": what --platform mac-app builds and where the app goes (-o), the bundle's layout, APFS clones or copies, where the save, pso_user.tsv, temporary files, env.txt and the log go, signal forwarding and the alert, why the launcher runs the host as a child process (macOS privacy checks on removable volumes: a launcher that exec'ed dmhost got EPERM on every open on the external drive, signed or not), and signing (ad-hoc or --mac-sign, never the hardened runtime, so no notarization). "A Mac-only build for development" no longer says there is no Mac app. The script's examples get one for --platform mac-app --mac-sign, and a line lists the three --platform values. README.md "Pack an IPA": one line pointing at the Mac app.
Docs only; no test ran for this commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The NSP script's --platform mac-app (Spyder: PLATFORM = 'mac-app') does everything --platform mac builds (the Mac shaders, the Mac host for the profile, the button-icon font), then packs <NSP folder>/<game name>.app (-o / OUT_IPA names another path; ':' and '/' in the game name become ' - '), which boots the game when opened from Finder, from any folder, without the build folder. New dm/mac/pack_app.py packs it: the launcher dm/mac/launcher.c (compiled for arm64 at the host's deployment target) as CFBundleExecutable, the host beside it in Contents/MacOS, and in Contents/Resources what the IPA carries (build_ipa.sh's choice): romfs/, msl/<b-chain | vk | gl>/ (the Mac libraries only, not the cas/ compile cache, which the host never reads), aoc/<index>/, and dm/ with the rewritten modules and the rewriter tables the host reads (*.patched.bin, fixups, gates, cuts, obs, nro.tsv; not sites.tsv), the profile's shader_map.tsv, pso_manifest.tsv, c1_full.tsv and cb_extent.tsv when present, font/ and the --env-file as env.txt; AppIcon.icns from the game icon (sips, iconutil); Info.plist with the NACP name and display version, the IPA's bundle ID and the profile's app_category, plus DMTitleID, DMRomfsMarker and DMLanguage for the launcher. The large folders are APFS clones (cp -c) when a clonefile into the app works (the same APFS volume), plain copies otherwise after a free space check. The app is written as <app>.partial and swapped in whole.
The launcher sets the host's environment from the bundle (as the iOS app's main.m does) and runs the host as a child with posix_spawn, never exec: macOS privacy (TCC) attributes a child's file access to the app, while a launcher that hands its process over (a shell script exec'ing dmhost) got EPERM on every open on the external drive, silently, signed or not (found on Echoes of Wisdom's hand-made wrapper). Save and temporary files go to ~/Library/Application Support/nxnative/<title ID>/ (save/, temp/; pso_user.tsv lands beside save/, which is also DM_ROOT and the working folder), the host's output to ~/Library/Logs/nxnative/<title ID>.log (the previous one kept as .prev.log). Environment switches: <Application Support folder>/env.txt, then the bundle's dm/env.txt; neither replaces a variable already set. It forwards SIGTERM, SIGINT and SIGHUP to the host, returns its exit code (128 + the signal when it was killed), and shows an alert (osascript) when the bundle's game data is missing.
Signing: the inner host first, then the bundle, codesign --force --sign <identity>, ad-hoc by default, --mac-sign / MAC_SIGN names an identity; never with the hardened runtime (it refuses the executable memory the host maps for the game's code), so the app cannot be notarized. Finder information on copied files, which codesign refuses, is removed and the signing tried again.
In the script, mac_run is split: mac_host builds the host and the font (the stage "Mac host", for both platforms), mac_run makes the command of --platform mac as before, and mac_app runs pack_app.py (the stage "Mac app", its steps from pack_app.py's step lines); --platform mac-app plans the same number of stages as an IPA build.
New test dm/tools/tests/test_nsp_mac_app.py (written first; on main it fails: no dm/mac/pack_app.py, no --platform mac-app): packs an app from stand-ins with a stand-in host compiled there, and checks the layout, Info.plist, the ad-hoc signature without the runtime flag, the launcher's environment, working folder and log rotation, both env files, the exit code, SIGTERM forwarding, the alert (a stand-in osascript), plain copies, and the script's options and pack_app.py command line for an NVN and a Vulkan game. Checked: every test of dm/tools/tests, dm/ios/tests and dm/gfx/tests, make -C dm/host/tests GAME=010015100B514000 run and dm/ios/tools/compile_check.py pass. Not checked here: an app of a real game (next), a GL game, an app moved to another Mac.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the separate ASTRAL CHAIN bring-up notice with a gameplay screenshot and notes. Record that passing Mac gameplay testing is sufficient for gallery inclusion, with validation scope kept in each game note.
Support PKZ v1 archives and advertise the translator's GPU-code range independently of SDK version. Register the 1.0.1 profile, shader tables, regression checks and Mac bring-up findings. Retain 30 fps; document the unshipped 60 fps experiment and outstanding iPhone validation.
The dispatcher gained a dm_ue_heap_owns dependency in 16f4889. Supply the no-Unreal-allocator stub in both isolated test fixtures, alongside their existing host dependency stubs.
Validated the full host unit suite with the CI game profile and all nine CI CPU-only graphics tests.
Integrate the present snapshots and frame-step work from #87 together with
the encode-thread support and wait ledger from #157. Keep snapshots off
by default on both platforms pending wider device validation.
Offer Frame buffering in the iOS launcher for NVN builds. The existing
launcher remembers the choice and preserves env.txt overrides. Pass the
packer's detected graphics API to the settings generator.
Wait for snapshot capacity at acquire; a present reaching a busy slot
falls back to its original texture ticket. Drain snapshot commits and GPU
completion during suspend even when display tickets were discarded.
Regression tests cover both waits, fallback tickets and copied pixels.
Validated on M4 Pro: full gfx run list, 38 snapshot lifecycle checks,
219 checks with encoding and snapshots enabled, Mac host build, 133-file
iOS compile check, and eight launcher/docs/packing tests. The host unit
suite cannot link t_heap_size because dm_ue_heap_owns is missing, reproduced
on unchanged main cbbb600. No iPhone runtime performance claim.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
The profile is the NSP script's provisional one with tex_flush_upload and tex_dirty_upload: RE Engine writes its UI textures (ASTC) into CPU-visible pools in place, as Monster Hunter Rise does, and without both keys the language screen's background and buttons were magenta (with tex_flush_upload alone they still were). 7,744 shaders from the PAK's SDF definitions and main; 7,739 build. The other 5 are compute shaders with atomics on a 3D image, which SPIRV-Cross indexes with an int3, as Octopath Traveler 0's 16; none was requested on the routes below. PR #138 (Nishikori-Yui) built them by flattening the coordinate in dmsh_build.py, a chain change that would rebuild every game's libraries, so it is left out until a run requests one (the game note). PR #138's profile is the same as this one (the same keys and shader tables).
Host fix: the memory line's "freed but alive" count only grew in most games (Animal Crossing: New Horizons 3,113 after 27,900 frames, Pokémon Sword 2,050, Super Smash Bros. Ultimate 1,279; Ghost Trick 125 to 146 buffers, 6.6 to 8.7 GB of zero-copy buffers, so GPU allocated read 9.9 to 11.7 GB against 2.2 GB resident). A temporary census of every host table that can hold a texture over a pool buffer found no holder: mtl_pool_del runs on the guest thread that finalizes the pool, whose implicit autorelease pool never drains, and adding the buffer to the weak table g_dead_pools left it autoreleased there. containsObject: / removeObject: on a weak NSHashTable do the same (a standalone test), and res_add / res_remove call them on guest threads too. mtl_pool_del, res_add_at and res_remove now have their own @autoreleasepool. Ghost Trick: 0 freed but alive, GPU allocated 1.7 GB instead of 8.3 GB 300 frames into the boot and 2.9 GB in Chapter 1, resident memory, frame times and pictures unchanged; Pokémon Sword: 8 to 0 at the title after 90 s, the same frame pacing.
On the Mac: the opening movies, the language screen, the first-boot notices, save creation, the Bonus Content notices and Chapter 1's opening in the junkyard (the Ghost World, the first trick), and a second boot that loads the save, at 60 fps (frame interval p99 about 17 to 19 ms), about 2.3 GB resident, no red line; the Wwise sound plays (a muted run with DM_AUDIO_DUMP). Checked: the tool tests (57, the docs check among them), t_nvn (124 checks), a boot of the rebased host. README (games gallery, a Mac picture of the junkyard), AGENTS.md's count (sixty-two: the gallery's cells), the game note, the games table, docs/reference/nvn-objects.md and the symptom index say so
The Asian release has its own title ID. Its subsdk0 and sdk are the western builds, but its main (31194702…) is not the western 1.10's (4b632573…), so the NSP script's lookup by main build does not find the western profile, and its RomFS marker is ROBO_CN.CPK. The profile is the NSP script's provisional one with the western profile's alloc_big_align (the engine's ring heaps are cut from a pool at 64 KB alignment) and the same 4 GB layout. Its shader tables have the same keys as the western ones (all 751 shader_map rows; shader_pairs, shader_facts and bf_want identical), at other offsets in main.
Ported onto main with its game note (dm/games/01008D7016438000/README.md), its row in the games table and a line in the western note pointing to it. The game.tsv header is in English (the PR had the Japanese and Korean titles as characters). README's gallery and AGENTS.md's count are unchanged: it is the same game and is not tested on the iPhone.
Checked (PR #161): on the Mac the NSP script packs it from the base NSP alone and it plays at 60 fps with sound, saving and loading. Here: the tool tests in dm/tools/tests pass, test_docs among them (61 profiles), and game.py lists the profile and writes its header. Not checked: iOS (no Asian NSP here).
The memory-profile host test imports profile_guess, which imported nso_syms and required LZ4 even for plain log parsing. The host CI job has no third-party Python packages, so this broke the host suite after cae5ec2. Move the import into load() and document the standard-library-only heap probe.
Validated the exact failing target and full parallel host suite with system Python, the heap probe with site packages disabled, six related tool tests, and synthetic compressed NSO decoding.
Incorporate the active SVC improvements from PR #151 around the existing
alias bitmap and mapping registry. Serialize heap changes, physical maps,
stack ownership and memory queries with one lock; report committed alias
runs and borrowed sources; initialize reused logical pages while preserving
live neighbors and lazy first exposure. Retain PR #127's fail-fast host
protection policy and the log formats parsed by the NSP heap probe.
Game smoke tests exposed two integration requirements: Raging Loop's SDK
extends committed windows across borrowed stack sources, so allow recommit
that preserves existing bytes; Tears of the Kingdom's audio helper needs
guest-backed stacks, so allocate host-created guest-thread stacks through
SDK malloc and free them after thread destruction and copyback.
Keep SDK reservation ownership and existing patches. The unused host
reservation allocator from #151 is not included.
Validation: full host suite; focused UBSan, ASan and ThreadSanitizer tests;
real dirty-tracker concurrency tests; heap-probe parser compatibility; Mac
host builds; iOS arm64 syntax checks; all 60 profile layout checks.
Raging Loop and Tears of the Kingdom passed guest-thread self-checks and
ran to deliberate timeouts after the startup fixes. Their shader caches
are stale for this checkout, so rendering/performance are not validated.
No IPA build or iOS device validation was performed.
Co-authored-by: b4f8abbb24dbf56a7d0280f296e16eeb <51352400+b4f8abbb24dbf56a7d0280f296e16eeb@users.noreply.github.com>
Adapt PR #122: compile reusable bases with unspecialized colour state, specialize per draw state, and persist bases between launches. Keep iOS opt-in and the MTLDevice fallback.
Preserve rasterizing depth-only pipelines through the legacy path. Share in-progress base builds, including errors, while unrelated bases compile in parallel. Defer timed flushes during compaction and finish flush/cleanup together; synchronize the pending count with serializer capture.
Checked on Apple M4 Pro: t_pso_archive 48 checks with Metal validation, t_pso_pending 48, t_integer_blend 63, t_geometry 95, and t_pso_skey pass. The Breath of the Wild Mac host builds and nvn_mtl.m passes the iOS arm64 syntax check. Real-game memory usage and iPhone runtime behavior were not measured.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Apply PR #127 and require the regression test's abort handler to observe a validated production fatal path. Unrelated assertion failures can no longer pass the injected-failure cases.
Co-authored-by: nocmt <nocmtone@gmail.com>
Raise the recovery test's elapsed-time ceiling from 2.6 to 5 seconds while retaining buffer ordering, minimum playback duration, and resumed callback checks. The original CI run recovered correctly in 2686 ms and passed unchanged on rerun.
Validated with the full host unit suite and all nine CPU-only graphics tests using GAME=010015100B514000.
