Reuse unchanged IPA entries through the packing index and APFS clones,
and keep Mac app and derived-profile shader outputs while their inputs
match. Preserve main's shader regression checks and .NET discovery.
Count uncompressed bytes in streamed files so a size change aborts the
pack without replacing the previous IPA or index. Cover both deflated
and stored streams with growth and shrinkage regression tests, and
record the cause and fix in the packing reference.
Co-authored-by: Suraj Mandal <surajmandalcell@gmail.com>
(cherry picked from commit 09f3778c36)
Derive DOTNET_ROOT from dotnet's Microsoft.NETCore.App runtime listing and
export it to both explicit and inherited build environments. Preserve caller
overrides, use the build environment for discovery, and leave discovery
unchanged on probe failures instead of treating Homebrew's wrapper directory
as a runtime root.
Add regression coverage and document the packing failure and its resolution.
Based on PR #284.
Co-authored-by: Jose Rosendo <josemrb99@gmail.com>
(cherry picked from commit 89744da753)
One translation chain serves every game, so a translator or host change was checked
only on its author's games; on another game the first sign was a crash at run time
(c3a0880 broke the image atomics of SHIN MEGAMI TENSEI V, Yoshi's Crafted World and
FINAL FANTASY RESONANCE).
- CI: the tool-tests job installs .NET 10, spirv-cross and spirv-tools and points
DMSH_TEST_XLATE at the committed translator, so the translator's synthetic-shader
tests (test_image_atomic_bounds, test_dmsh_geometry) run on every change.
test_dmsh_geometry imports its sibling module when run from the repository root.
- dm/gfx/tools/shader_regressions.py: lists the shaders that translated in the
game's previous b-*/ and fail in the current chain's, from dmsh_build.py's build
reports (not part of the chain). The NSP script runs it after its shader step and
stops on a regression unless ALLOW_SHADER_REGRESSIONS / --allow-shader-regressions;
docs/development.md describes it.
- FINAL FANTASY RESONANCE: the c3a0880 regression is resolved since 2e655b7 (image
atomics with a used result): chain 39e4d9585b1d builds 73,488 of 73,496 shaders
(the 8 3D image atomics as before), shader_regressions.py finds 0 against the
earlier build, and a new game from an empty save plays the opening cutscene on
the Mac with no red line; the two shaders that failed and aborted run their
atomics on their buffer textures.
Checked: every CI test file as the job runs it (DMSH_TEST_XLATE set), the docs tests.
(cherry picked from commit b117c91317)
The Mac app packer left out shader_pairs.tsv and pso_pairs.tsv, which the
host's pipeline prebuild reads beside the shader map and build_ipa.sh already
copies into the IPA. Every NVN game's Mac app therefore built no pulling
bases, had no hybrid stand-ins, and compiled first-seen pipelines on the
submit thread. Copy both tables, and the shader split's pso_pairs.tsv for a
profile without one.
An NVN game's Mac app also carries its pipeline archive made offline for this
Mac's GPU (pso_offline.py --platform macos), handed to the host as
DM_PSO_BUNDLE as on iOS; --gpu none leaves it out. Integrated with three
fixes: the archive is validated and counted in the free-space check before
anything is copied; a plan whose pairs all need function constants packs the
app without an archive instead of aborting; and a PSO_GPUS value that does
not name this Mac's GPU is reported.
Validation: test_nsp_mac_app 22 (pair tables, split copy, bad archive, free
space, GPU note, no-archive plan; the table and space tests fail without the
fixes), the tool suite, iOS packaging tests and test_docs. A Breath of the
Wild Mac app on an M4 Pro (applegpu_g16s, 629.4 MB archive): 16 of 16 probed
bases found, no pair base built, no red line; the run stayed in the title.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Co-authored-by: blindlight86 <25404088+blindlight86@users.noreply.github.com>
(cherry picked from commit 4639c5114c)
From PR #329 (stacked on #328): a game's usage manifest, dm/games/<id>/usage.tsv,
written by its owner with regress.py usage (shader features of its chain build,
and the NVN / HLE functions, SVCs and present path of one smoke-route run;
names and counts only), and regress.py impact <base>..<head>, which maps the
functions around the changed lines through the host's callers and the rules of
dm/tools/impact_map.tsv to manifest rows. Manifests for Breath of the Wild,
Tears of the Kingdom and Hades.
Fixed on top of the PR:
- impact checked the head out as a git worktree under the harness root, so it
stopped (PermissionError on /Volumes/NX) for exactly the contributors it is
for, who have no games and no harness root. The call graph is now read from
the git object store at the head revision, and the report is printed and
written only to --out or an existing harness root.
- A hunk mapped only to the function at its first line: a hunk spanning two
functions named one. Every added line is mapped now, and a pure deletion maps
to the function around it (it was moved one line up).
- Functions defined in headers were not in the call graph: the frame step
(nvn_fstep.h, static inline, called from nvnQueuePresentTexture) reached no
entry point, so 4acde72 named no game. Headers are scanned, and their
functions are visible to every file. 4acde72 now names both Zeldas through
nvnQueuePresentTexture.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Compare revision shader builds and pipeline interfaces across local games,
and run available smoke routes with crash, hang, timing and image checks.
Add the developer loop and share the measuring lock with bench.py.
Keep all local games in static checks and honor --no-smoke. Include shader
bytes and fallback pairs in cache keys, recheck interfaces on reuse, and
reuse smoke baselines only for matching inputs and run settings. Detect
failed prebuilds, missing or intermittently bad screenshots, and Vulkan
build failures even when no MSL was emitted. Keep game-derived files and
writable profile copies inside each game's NSP work folder.
Validation: tool tests and the lint ratchet pass. Full shader rebuilds and
gameplay comparisons remain unverified; the 26 discovered Mac builds have
no smoke routes in this checkout.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Read both shader pair tables, including the generated BAHS fallback, for
padding, build order and shader cache dependencies. Include the optional
profile table in the input stamp and the generated table in step reuse;
regenerate a missing fallback before reusing the shader split.
Add native Metal and cache/order regressions, update the shared chain,
and document full validation on TotK and Odyssey. All 301 affected pairs
compile for g13s and g16s, both full Mac shader/host builds pass, all
57,039 pair interfaces match, and TotK's 24,153-base archive succeeds.
The separate Odyssey Max Unstruct ID compiler limit remains open.
Fixes#283
Refs #276, #298
Co-authored-by: marco <6410852+marcolaux@users.noreply.github.com>
Remove repetitive idle messages, permanent footers, the guide link and normal-run technical logs. Show a compact preparation form, contextual recovery instructions and a ready-to-install result. Send the prepared IPA directly to Sideloadly without signing or starting device installation. Simplify the catalog and fix missing Xcode detection.
Read all checkout game profiles, search names, versions and title IDs without case or accent sensitivity, and link each result to its game notes. Localize the catalog in English and Spanish and verify discovery against the current profiles.
Persist the interface language, translate app-owned labels and dialogs immediately, preserve raw build diagnostics, and replace unverified device and account claims with precise preparation and compatibility wording. Add English and Spanish READMEs and localization regression coverage.
Replace desktop artwork with generic SF Symbols, use explicit ad-hoc Mac signing, document provenance and project scope, and record pre-publication checks. Keep repository visibility private pending the owner decision on documentation artwork.
Provide folder selection, build progress, process-group cancellation, and IPA/RomFS/Sideloadly buttons. Preserve command documentation separately and correct the personal author policy.
Use the edited checkout for every game, persist identity, verify archives, preserve existing outputs, and reject concurrent builds. Include applicability triage for 57 open upstream PRs and 20 commits after the stable base.
Reuse unchanged IPA entries through the packing index and APFS clones,
and keep Mac app and derived-profile shader outputs while their inputs
match. Preserve main's shader regression checks and .NET discovery.
Count uncompressed bytes in streamed files so a size change aborts the
pack without replacing the previous IPA or index. Cover both deflated
and stored streams with growth and shrinkage regression tests, and
record the cause and fix in the packing reference.
Co-authored-by: Suraj Mandal <surajmandalcell@gmail.com>
Derive DOTNET_ROOT from dotnet's Microsoft.NETCore.App runtime listing and
export it to both explicit and inherited build environments. Preserve caller
overrides, use the build environment for discovery, and leave discovery
unchanged on probe failures instead of treating Homebrew's wrapper directory
as a runtime root.
Add regression coverage and document the packing failure and its resolution.
Based on PR #284.
Co-authored-by: Jose Rosendo <josemrb99@gmail.com>
One translation chain serves every game, so a translator or host change was checked
only on its author's games; on another game the first sign was a crash at run time
(c3a0880 broke the image atomics of SHIN MEGAMI TENSEI V, Yoshi's Crafted World and
FINAL FANTASY RESONANCE).
- CI: the tool-tests job installs .NET 10, spirv-cross and spirv-tools and points
DMSH_TEST_XLATE at the committed translator, so the translator's synthetic-shader
tests (test_image_atomic_bounds, test_dmsh_geometry) run on every change.
test_dmsh_geometry imports its sibling module when run from the repository root.
- dm/gfx/tools/shader_regressions.py: lists the shaders that translated in the
game's previous b-*/ and fail in the current chain's, from dmsh_build.py's build
reports (not part of the chain). The NSP script runs it after its shader step and
stops on a regression unless ALLOW_SHADER_REGRESSIONS / --allow-shader-regressions;
docs/development.md describes it.
- FINAL FANTASY RESONANCE: the c3a0880 regression is resolved since 2e655b7 (image
atomics with a used result): chain 39e4d9585b1d builds 73,488 of 73,496 shaders
(the 8 3D image atomics as before), shader_regressions.py finds 0 against the
earlier build, and a new game from an empty save plays the opening cutscene on
the Mac with no red line; the two shaders that failed and aborted run their
atomics on their buffer textures.
Checked: every CI test file as the job runs it (DMSH_TEST_XLATE set), the docs tests.
The Mac app packer left out shader_pairs.tsv and pso_pairs.tsv, which the
host's pipeline prebuild reads beside the shader map and build_ipa.sh already
copies into the IPA. Every NVN game's Mac app therefore built no pulling
bases, had no hybrid stand-ins, and compiled first-seen pipelines on the
submit thread. Copy both tables, and the shader split's pso_pairs.tsv for a
profile without one.
An NVN game's Mac app also carries its pipeline archive made offline for this
Mac's GPU (pso_offline.py --platform macos), handed to the host as
DM_PSO_BUNDLE as on iOS; --gpu none leaves it out. Integrated with three
fixes: the archive is validated and counted in the free-space check before
anything is copied; a plan whose pairs all need function constants packs the
app without an archive instead of aborting; and a PSO_GPUS value that does
not name this Mac's GPU is reported.
Validation: test_nsp_mac_app 22 (pair tables, split copy, bad archive, free
space, GPU note, no-archive plan; the table and space tests fail without the
fixes), the tool suite, iOS packaging tests and test_docs. A Breath of the
Wild Mac app on an M4 Pro (applegpu_g16s, 629.4 MB archive): 16 of 16 probed
bases found, no pair base built, no red line; the run stayed in the title.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Co-authored-by: blindlight86 <25404088+blindlight86@users.noreply.github.com>
Run profile-defined routes from independent copies of one save and report frame-time, presentation and GPU percentiles across repeated variants. Preserve the display-link behavior for targets at or below 60 fps.
Resolve relative paths before launching in the run folder, preserve the archive-disable sentinel, aggregate optional metrics from the runs that supplied them, and release the measurement lock when launch fails. Regression tests exercise an actual synthetic child host and leave the source save intact.
Validated with benchmark regressions, the CI-style Python test suite, and documentation checks. The three new benchmark regression cases reject the original PR code.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Recognize offset-table dispatches whose predecessor masks the index while
the dispatch checks a masked copy of the same source. Reject calls and
resolved table paths that bypass masking, undecodable instruction gaps or
targets, and writes that invalidate the table base. Keep the existing
recognizers ahead of this fallback.
Load build-gated patches from DM_GAMES_DIR and include them in rewrite reuse
keys. Preserve repository matching rules for derived profiles, select the
first external profile for provisional matching rules, and deduplicate
patch paths while retaining conflicts between distinct build claims.
Validation on current main: Python tool tests (156, 14 skipped), iOS tests
(6), graphics tests (44, 2 skipped), documentation index and diff checks.
Regression cases fail on the original implementation and pass with these
guards; both supported base-loading forms remain accepted. Real game
modules and gameplay were not revalidated.
Co-authored-by: Amost <6445689+Amost-@users.noreply.github.com>
Compile eligible shader-pair bases on the Mac with metal-tt, package one
archive per selected iPhone GPU, and use a matching bundle before falling
back to on-device preparation. Carry the profile depth mode through packing
and include the pulling libraries required by the bundled bases.
Integrate PR #226 with four confirmed fixes: validate archive container
bounds before invoking Metal; require at least one actual successful bundle
probe; hash consumed library bytes into offline cache keys; and emit device
remainder lists for constant-only plans instead of aborting packaging.
Keep game measurements in the game note and document the general findings.
Validation on current main: 52 real Metal bundle checks, 57 archive checks,
79 prewarm checks on each pipeline path, 73 Python regressions covering
offline tools, packaging, documentation and chain checks, and compilation
of all 133 iOS source files. No new iPhone gameplay validation. The archive guard validates
container structure, not private Metal payloads.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
nsz 4.6.1 rejects --keys and loads keys.txt beside argv[0] during import.
Run the converter in its temporary output folder with the selected key
file linked there before import, using absolute input and output paths.
Remove the temporary link on both successful and failed conversions.
Integrate PR #204 with XCZ and relative-path regressions, rename the test
flag that shadowed unittest.TestCase.fail, and qualify the documentation
for the verified nsz release. Record the packing finding and provenance.
Validation: reproduced the exact parser failure using real nsz 4.6.1;
a synthetic NSZ conversion verifies selected-key lookup and link cleanup;
52 NSP tests, four documentation tests and git diff --check pass.
The synthetic payload is unencrypted and uses dummy keys. Real-game
decryption, actual XCZ conversion and a full IPA build remain unverified.
Co-authored-by: Fernando Messias <fernandomessias95@gmail.com>
A pulling stand-in is used only when its base is made or archived, so a cold start's draws waited until the prebuild puts the base of every known pair into the archive (pso_prebuild): in the app after the prewarm, on a utility queue on half the performance cores, flushing every 1,000 bases (DM_PSO_PREBUILD=0: off), or as a tool on the Mac (DM_PSO_PREBUILD=tool: build, write, exit; DM_PSO_PREBUILD_DEPTH=zero, DM_PSO_PREBUILD_THREADS=<n>), once per chain version, depth mode and pair list (pso_archive/prebuilt.tsv). The NSP script runs the tool in a --platform mac build (pso_prebuild.log). Files that hold only bases without a vertex layout are named p<seq>.mtl4a; a base with a vertex layout is never looked up in them (each file a lookup misses in costs about 0.06 ms) and compaction leaves them alone.
The pairs: shader_pairs.tsv paired the stages of one BNSH variation and none of the pairs in agl's BAHS shader archives (.sharcb: the agl / gsys / terrain resource packs), and in Breath of the Wild 109 of the 543 pairs drawn on entering Zora's Domain came from those, so they had no stand-in. shader_split.py now reads a BAHS file's binary array (record size, kind: 0 vertex, 1 fragment, 3 compute, the entry bahs_shaders reads) and program array (record size, name length, stage mask, first binary index), and pairs the n-th vertex binary of a program with its n-th fragment binary (bahs_variations); a file whose arrays do not read back gives none. Those pairs go to a new table, pso_pairs.tsv, which only the prebuild reads (DM_PSO_PAIRS: another path): vs_pad keeps reading shader_pairs.tsv, so no shader library, inputs digest or other table changes. Breath of the Wild: 3,012 more pairs, and all 543 pairs drawn are now known; its shader_map / facts / pairs tables split again byte-identical. A profile made before the table existed still packs: the NSP script compares pso_pairs.tsv only where the profile has one, and otherwise keeps it beside the shader package and hands it to the prebuild. Seven other profiles have BAHS files (Odyssey, Wonder, Mario Kart 8 Deluxe, 3D World, Animal Crossing, Captain Toad, Tears of the Kingdom); their tables are made at their next pack.
Breath of the Wild, Mac (M2 Max), Zora's Domain save, empty Metal shader cache, no pso_user.tsv. Prebuild: 15,767 bases, 910 MB in 16 files, 187 s with 7 workers for the first 12,755 and 19 s for the 3,012 more; the files open in 2.2 s cold, 0.25 s from the file cache, and add about 850 MB of footprint. The frame entering the area: 340 ms with pso_pairs.tsv, 830 ms with shader_pairs.tsv alone, 990 ms through the MTLDevice (then frames of 738, 535 and 496 ms); draws that waited: 50 (20 ms), 102 (650 ms) and 512 (3,325 ms). The picture is the same.
Tests: t_pso_archive (41 checks: also a base without a layout flushed into a p file, and a base with a layout not looked up there); the other graphics tests, the host unit tests and test_nsp_mac / progress / chain give the results they give on main. Not run: an iPhone (the prebuild is off there with the Metal 4 path), the NSP script's pack end to end, the other BAHS profiles' splits. docs/development.md, dm/gfx/README.md and dm/games/README.md describe the prebuild and the table; Breath of the Wild's note (dm/games/01007EF00011E000/README.md, Pipelines on a cold start) has its measurements.
- profile 01007BB017812000 (1.0.3, SDK 13.3.0): plays on the Mac at 60 fps
- provisional profiles add the subsdks the NROs import (profile_guess nro_modules)
- an NRO importing nvnBootstrapLoader makes an NVN game; NROs calling glslcCompile get the run-time GLSLC capture
- shader_split reads zip archive members (Portal's nxcontent/nvsb GLSLC outputs)
- ro::LookupModuleSymbol for a name a loaded module lacks is not red (Horizon fails it too)
- AGENTS.md: a Mac gameplay test is enough to count a game as playable (65)
The NSP script's --platform mac-app (Spyder: PLATFORM = 'mac-app') does everything --platform mac builds (the Mac shaders, the Mac host for the profile, the button-icon font), then packs <NSP folder>/<game name>.app (-o / OUT_IPA names another path; ':' and '/' in the game name become ' - '), which boots the game when opened from Finder, from any folder, without the build folder. New dm/mac/pack_app.py packs it: the launcher dm/mac/launcher.c (compiled for arm64 at the host's deployment target) as CFBundleExecutable, the host beside it in Contents/MacOS, and in Contents/Resources what the IPA carries (build_ipa.sh's choice): romfs/, msl/<b-chain | vk | gl>/ (the Mac libraries only, not the cas/ compile cache, which the host never reads), aoc/<index>/, and dm/ with the rewritten modules and the rewriter tables the host reads (*.patched.bin, fixups, gates, cuts, obs, nro.tsv; not sites.tsv), the profile's shader_map.tsv, pso_manifest.tsv, c1_full.tsv and cb_extent.tsv when present, font/ and the --env-file as env.txt; AppIcon.icns from the game icon (sips, iconutil); Info.plist with the NACP name and display version, the IPA's bundle ID and the profile's app_category, plus DMTitleID, DMRomfsMarker and DMLanguage for the launcher. The large folders are APFS clones (cp -c) when a clonefile into the app works (the same APFS volume), plain copies otherwise after a free space check. The app is written as <app>.partial and swapped in whole.
The launcher sets the host's environment from the bundle (as the iOS app's main.m does) and runs the host as a child with posix_spawn, never exec: macOS privacy (TCC) attributes a child's file access to the app, while a launcher that hands its process over (a shell script exec'ing dmhost) got EPERM on every open on the external drive, silently, signed or not (found on Echoes of Wisdom's hand-made wrapper). Save and temporary files go to ~/Library/Application Support/nxnative/<title ID>/ (save/, temp/; pso_user.tsv lands beside save/, which is also DM_ROOT and the working folder), the host's output to ~/Library/Logs/nxnative/<title ID>.log (the previous one kept as .prev.log). Environment switches: <Application Support folder>/env.txt, then the bundle's dm/env.txt; neither replaces a variable already set. It forwards SIGTERM, SIGINT and SIGHUP to the host, returns its exit code (128 + the signal when it was killed), and shows an alert (osascript) when the bundle's game data is missing.
Signing: the inner host first, then the bundle, codesign --force --sign <identity>, ad-hoc by default, --mac-sign / MAC_SIGN names an identity; never with the hardened runtime (it refuses the executable memory the host maps for the game's code), so the app cannot be notarized. Finder information on copied files, which codesign refuses, is removed and the signing tried again.
In the script, mac_run is split: mac_host builds the host and the font (the stage "Mac host", for both platforms), mac_run makes the command of --platform mac as before, and mac_app runs pack_app.py (the stage "Mac app", its steps from pack_app.py's step lines); --platform mac-app plans the same number of stages as an IPA build.
New test dm/tools/tests/test_nsp_mac_app.py (written first; on main it fails: no dm/mac/pack_app.py, no --platform mac-app): packs an app from stand-ins with a stand-in host compiled there, and checks the layout, Info.plist, the ad-hoc signature without the runtime flag, the launcher's environment, working folder and log rotation, both env files, the exit code, SIGTERM forwarding, the alert (a stand-in osascript), plain copies, and the script's options and pack_app.py command line for an NVN and a Vulkan game. Checked: every test of dm/tools/tests, dm/ios/tests and dm/gfx/tests, make -C dm/host/tests GAME=010015100B514000 run and dm/ios/tools/compile_check.py pass. Not checked here: an app of a real game (next), a GL game, an app moved to another Mac.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Erick Sanchez