Remove repetitive idle messages, permanent footers, the guide link and normal-run technical logs. Show a compact preparation form, contextual recovery instructions and a ready-to-install result. Send the prepared IPA directly to Sideloadly without signing or starting device installation. Simplify the catalog and fix missing Xcode detection.
Read all checkout game profiles, search names, versions and title IDs without case or accent sensitivity, and link each result to its game notes. Localize the catalog in English and Spanish and verify discovery against the current profiles.
Persist the interface language, translate app-owned labels and dialogs immediately, preserve raw build diagnostics, and replace unverified device and account claims with precise preparation and compatibility wording. Add English and Spanish READMEs and localization regression coverage.
Replace desktop artwork with generic SF Symbols, use explicit ad-hoc Mac signing, document provenance and project scope, and record pre-publication checks. Keep repository visibility private pending the owner decision on documentation artwork.
Provide folder selection, build progress, process-group cancellation, and IPA/RomFS/Sideloadly buttons. Preserve command documentation separately and correct the personal author policy.
Use the edited checkout for every game, persist identity, verify archives, preserve existing outputs, and reject concurrent builds. Include applicability triage for 57 open upstream PRs and 20 commits after the stable base.
Run profile-defined routes from independent copies of one save and report frame-time, presentation and GPU percentiles across repeated variants. Preserve the display-link behavior for targets at or below 60 fps.
Resolve relative paths before launching in the run folder, preserve the archive-disable sentinel, aggregate optional metrics from the runs that supplied them, and release the measurement lock when launch fails. Regression tests exercise an actual synthetic child host and leave the source save intact.
Validated with benchmark regressions, the CI-style Python test suite, and documentation checks. The three new benchmark regression cases reject the original PR code.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Recognize offset-table dispatches whose predecessor masks the index while
the dispatch checks a masked copy of the same source. Reject calls and
resolved table paths that bypass masking, undecodable instruction gaps or
targets, and writes that invalidate the table base. Keep the existing
recognizers ahead of this fallback.
Load build-gated patches from DM_GAMES_DIR and include them in rewrite reuse
keys. Preserve repository matching rules for derived profiles, select the
first external profile for provisional matching rules, and deduplicate
patch paths while retaining conflicts between distinct build claims.
Validation on current main: Python tool tests (156, 14 skipped), iOS tests
(6), graphics tests (44, 2 skipped), documentation index and diff checks.
Regression cases fail on the original implementation and pass with these
guards; both supported base-loading forms remain accepted. Real game
modules and gameplay were not revalidated.
Co-authored-by: Amost <6445689+Amost-@users.noreply.github.com>
Compile eligible shader-pair bases on the Mac with metal-tt, package one
archive per selected iPhone GPU, and use a matching bundle before falling
back to on-device preparation. Carry the profile depth mode through packing
and include the pulling libraries required by the bundled bases.
Integrate PR #226 with four confirmed fixes: validate archive container
bounds before invoking Metal; require at least one actual successful bundle
probe; hash consumed library bytes into offline cache keys; and emit device
remainder lists for constant-only plans instead of aborting packaging.
Keep game measurements in the game note and document the general findings.
Validation on current main: 52 real Metal bundle checks, 57 archive checks,
79 prewarm checks on each pipeline path, 73 Python regressions covering
offline tools, packaging, documentation and chain checks, and compilation
of all 133 iOS source files. No new iPhone gameplay validation. The archive guard validates
container structure, not private Metal payloads.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
nsz 4.6.1 rejects --keys and loads keys.txt beside argv[0] during import.
Run the converter in its temporary output folder with the selected key
file linked there before import, using absolute input and output paths.
Remove the temporary link on both successful and failed conversions.
Integrate PR #204 with XCZ and relative-path regressions, rename the test
flag that shadowed unittest.TestCase.fail, and qualify the documentation
for the verified nsz release. Record the packing finding and provenance.
Validation: reproduced the exact parser failure using real nsz 4.6.1;
a synthetic NSZ conversion verifies selected-key lookup and link cleanup;
52 NSP tests, four documentation tests and git diff --check pass.
The synthetic payload is unencrypted and uses dummy keys. Real-game
decryption, actual XCZ conversion and a full IPA build remain unverified.
Co-authored-by: Fernando Messias <fernandomessias95@gmail.com>
A pulling stand-in is used only when its base is made or archived, so a cold start's draws waited until the prebuild puts the base of every known pair into the archive (pso_prebuild): in the app after the prewarm, on a utility queue on half the performance cores, flushing every 1,000 bases (DM_PSO_PREBUILD=0: off), or as a tool on the Mac (DM_PSO_PREBUILD=tool: build, write, exit; DM_PSO_PREBUILD_DEPTH=zero, DM_PSO_PREBUILD_THREADS=<n>), once per chain version, depth mode and pair list (pso_archive/prebuilt.tsv). The NSP script runs the tool in a --platform mac build (pso_prebuild.log). Files that hold only bases without a vertex layout are named p<seq>.mtl4a; a base with a vertex layout is never looked up in them (each file a lookup misses in costs about 0.06 ms) and compaction leaves them alone.
The pairs: shader_pairs.tsv paired the stages of one BNSH variation and none of the pairs in agl's BAHS shader archives (.sharcb: the agl / gsys / terrain resource packs), and in Breath of the Wild 109 of the 543 pairs drawn on entering Zora's Domain came from those, so they had no stand-in. shader_split.py now reads a BAHS file's binary array (record size, kind: 0 vertex, 1 fragment, 3 compute, the entry bahs_shaders reads) and program array (record size, name length, stage mask, first binary index), and pairs the n-th vertex binary of a program with its n-th fragment binary (bahs_variations); a file whose arrays do not read back gives none. Those pairs go to a new table, pso_pairs.tsv, which only the prebuild reads (DM_PSO_PAIRS: another path): vs_pad keeps reading shader_pairs.tsv, so no shader library, inputs digest or other table changes. Breath of the Wild: 3,012 more pairs, and all 543 pairs drawn are now known; its shader_map / facts / pairs tables split again byte-identical. A profile made before the table existed still packs: the NSP script compares pso_pairs.tsv only where the profile has one, and otherwise keeps it beside the shader package and hands it to the prebuild. Seven other profiles have BAHS files (Odyssey, Wonder, Mario Kart 8 Deluxe, 3D World, Animal Crossing, Captain Toad, Tears of the Kingdom); their tables are made at their next pack.
Breath of the Wild, Mac (M2 Max), Zora's Domain save, empty Metal shader cache, no pso_user.tsv. Prebuild: 15,767 bases, 910 MB in 16 files, 187 s with 7 workers for the first 12,755 and 19 s for the 3,012 more; the files open in 2.2 s cold, 0.25 s from the file cache, and add about 850 MB of footprint. The frame entering the area: 340 ms with pso_pairs.tsv, 830 ms with shader_pairs.tsv alone, 990 ms through the MTLDevice (then frames of 738, 535 and 496 ms); draws that waited: 50 (20 ms), 102 (650 ms) and 512 (3,325 ms). The picture is the same.
Tests: t_pso_archive (41 checks: also a base without a layout flushed into a p file, and a base with a layout not looked up there); the other graphics tests, the host unit tests and test_nsp_mac / progress / chain give the results they give on main. Not run: an iPhone (the prebuild is off there with the Metal 4 path), the NSP script's pack end to end, the other BAHS profiles' splits. docs/development.md, dm/gfx/README.md and dm/games/README.md describe the prebuild and the table; Breath of the Wild's note (dm/games/01007EF00011E000/README.md, Pipelines on a cold start) has its measurements.
- profile 01007BB017812000 (1.0.3, SDK 13.3.0): plays on the Mac at 60 fps
- provisional profiles add the subsdks the NROs import (profile_guess nro_modules)
- an NRO importing nvnBootstrapLoader makes an NVN game; NROs calling glslcCompile get the run-time GLSLC capture
- shader_split reads zip archive members (Portal's nxcontent/nvsb GLSLC outputs)
- ro::LookupModuleSymbol for a name a loaded module lacks is not red (Horizon fails it too)
- AGENTS.md: a Mac gameplay test is enough to count a game as playable (65)
The NSP script's --platform mac-app (Spyder: PLATFORM = 'mac-app') does everything --platform mac builds (the Mac shaders, the Mac host for the profile, the button-icon font), then packs <NSP folder>/<game name>.app (-o / OUT_IPA names another path; ':' and '/' in the game name become ' - '), which boots the game when opened from Finder, from any folder, without the build folder. New dm/mac/pack_app.py packs it: the launcher dm/mac/launcher.c (compiled for arm64 at the host's deployment target) as CFBundleExecutable, the host beside it in Contents/MacOS, and in Contents/Resources what the IPA carries (build_ipa.sh's choice): romfs/, msl/<b-chain | vk | gl>/ (the Mac libraries only, not the cas/ compile cache, which the host never reads), aoc/<index>/, and dm/ with the rewritten modules and the rewriter tables the host reads (*.patched.bin, fixups, gates, cuts, obs, nro.tsv; not sites.tsv), the profile's shader_map.tsv, pso_manifest.tsv, c1_full.tsv and cb_extent.tsv when present, font/ and the --env-file as env.txt; AppIcon.icns from the game icon (sips, iconutil); Info.plist with the NACP name and display version, the IPA's bundle ID and the profile's app_category, plus DMTitleID, DMRomfsMarker and DMLanguage for the launcher. The large folders are APFS clones (cp -c) when a clonefile into the app works (the same APFS volume), plain copies otherwise after a free space check. The app is written as <app>.partial and swapped in whole.
The launcher sets the host's environment from the bundle (as the iOS app's main.m does) and runs the host as a child with posix_spawn, never exec: macOS privacy (TCC) attributes a child's file access to the app, while a launcher that hands its process over (a shell script exec'ing dmhost) got EPERM on every open on the external drive, silently, signed or not (found on Echoes of Wisdom's hand-made wrapper). Save and temporary files go to ~/Library/Application Support/nxnative/<title ID>/ (save/, temp/; pso_user.tsv lands beside save/, which is also DM_ROOT and the working folder), the host's output to ~/Library/Logs/nxnative/<title ID>.log (the previous one kept as .prev.log). Environment switches: <Application Support folder>/env.txt, then the bundle's dm/env.txt; neither replaces a variable already set. It forwards SIGTERM, SIGINT and SIGHUP to the host, returns its exit code (128 + the signal when it was killed), and shows an alert (osascript) when the bundle's game data is missing.
Signing: the inner host first, then the bundle, codesign --force --sign <identity>, ad-hoc by default, --mac-sign / MAC_SIGN names an identity; never with the hardened runtime (it refuses the executable memory the host maps for the game's code), so the app cannot be notarized. Finder information on copied files, which codesign refuses, is removed and the signing tried again.
In the script, mac_run is split: mac_host builds the host and the font (the stage "Mac host", for both platforms), mac_run makes the command of --platform mac as before, and mac_app runs pack_app.py (the stage "Mac app", its steps from pack_app.py's step lines); --platform mac-app plans the same number of stages as an IPA build.
New test dm/tools/tests/test_nsp_mac_app.py (written first; on main it fails: no dm/mac/pack_app.py, no --platform mac-app): packs an app from stand-ins with a stand-in host compiled there, and checks the layout, Info.plist, the ad-hoc signature without the runtime flag, the launcher's environment, working folder and log rotation, both env files, the exit code, SIGTERM forwarding, the alert (a stand-in osascript), plain copies, and the script's options and pack_app.py command line for an NVN and a Vulkan game. Checked: every test of dm/tools/tests, dm/ios/tests and dm/gfx/tests, make -C dm/host/tests GAME=010015100B514000 run and dm/ios/tools/compile_check.py pass. Not checked here: an app of a real game (next), a GL game, an app moved to another Mac.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the Mac from update 13.0.5 (v2031616): the opening movie, the title, the first launch's DLC unlocks and Challenges board, the main menu, rulesets, stage and fighter select, 2-player battles (Frigate Orpheon, Big Blue, Great Bay, Find Mii, Prism Tower, Kongo Jungle 64) at 60 fps with about 2.7 ms of GPU per frame and 2 GB resident (1.2 GB of Metal allocations), quitting to the results and a second battle whose fighters' NROs the game loaded again; no red line in battle.
- Profile: modules main subsdk0 sdk (Catherine: Full Body's sdk and movie builds), the 5 GB guest layout, shader groups for data.arc's shader databases, effects and UI layouts, 35,012 shaders.
- patches.py: the engine's allocator (main+0x37c9a20, a static constructor) reserves 62 GB of address space with AllocateAddressRegion and aborted at main+0x37c9bd0. Booted once with a 64 GB alias region it commits from the region's start up (1.73 GB at the title, up to 2.5 GB in a battle) and keeps its page tables in the top 14 MB; all 11 instructions derived from the size are cut to a 4 GB region.
- sdk_patches.py: this SDK 8.2.1 sdk build (5fd4f531, Catherine's too) has the 1 GB AddressRegionManager node area and the 1 GB alignment of large regions (DRAGON QUEST XI S's SDK 7.3.2 layout at sdk+0x336b40 / +0x336d5c); both cut, so the region starts 64 MB into the 4.375 GB alias region.
- NVN: nvnSetExternalDebuggerEnabledNVX, fetched with nvnDeviceGetProcAddress(NULL, …) before the device and called unchecked (pc=0 from main+0x37fe92c), is a no-op in names.txt / hle_nvn.c, as docs/agents/03 already described.
- iOS NRO embedding (the part d979c34 left out): gen_embed.py packs the NROs nro.tsv lists into three sections (__TEXT,__dm_nro_x0, __DM_NRO_W,__img, __TEXT,__dm_nro_x1) with a table of their pieces (__DATA,__dm_nro_tab), since a Mach-O file has at most 255 sections and three per NRO would take 294; loader.c map_image maps an NRO from there (nro_pieces); build_ipa.sh puts nro.tsv beside the rewriter tables. The 98 NROs add 157 MB of code and 51 MB of data to the executable.
- shader_split.py reads data.arc through smash_arc.py (smash_arc_jobs / smash_arc_shaders: each current file scanned, labelled /data.arc/<format>#<hash40>); smash_arc.files(regions=True) gives every region's data of a regional file, without which 128 effect shaders were missing (34,884 of 35,012).
- Docs: README (games table, a Mac picture of Great Bay), AGENTS.md (50 → 51), docs/development.md (game notes, the NRO paragraph, nro_collect.py), docs/agents/01 (data.arc) and 03 (the 4 GB cut and its measurements, iOS NRO embedding).
Checked: the NSP script packs it from the NSPs alone, for the Mac and for iOS; tool tests (41), host unit tests and the NRO loader test pass. Played on an iPhone with this port's build from before it moved onto d979c34's NRO loader; the merged build was checked on the Mac (battles, an NRO loaded again) and builds for iOS.
Integrate PRs #75, #90 and #91 with focused correctness fixes:
- Prioritize demanded render pipelines and bound speculative compilation.
- Follow copied jump-table bases and reject W/X register clobbers.
- Collect and rewrite registered NROs, load them on the Mac, and preserve
runtime activation order through unload/reload and symbol rebinding.
- Preserve refused-only manifests and invalidate collection caches when
archive readers or symbol parsing change.
Add pipeline scheduling, register tracing, NRO packaging and loader
lifecycle regression coverage. Document that executable NRO embedding
on iOS remains outside this integration.
Validation: 41 Python tests; 48 pipeline scheduling, 102 NVN and 95
geometry checks; three pipeline-key serialization cases; full host test
suite; Mac host build; iOS loader syntax checks; git diff --check.
Gameplay and controlled performance A/B measurements were not run.
Co-authored-by: blindlight86 <25404088+blindlight86@users.noreply.github.com>
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
A development option; the Mac is the development host, not a platform for players. The same steps up to the
shaders (extraction, DLC, rewrite, run-time captures and decodes, heap check, shader split), then only the
Mac's: dmsh_build.py without IOSOUT (an inherited one is dropped), a Vulkan or GL game's library without --ios,
the Mac host for the profile into <id>_build/tools/dmhost (build_mac_host, through DM_GAMES_DIR /
DM_GAMES_FIRST for a provisional or derived profile) and the button-icon font into <id>_build/font. No iOS
shaders, Xcode project or IPA; the extracted files are kept whatever KEEP_EXTRACTED says, and the last line is
the command that runs the game (its save in <id>_build/run/save, DM_NVN_SHMAP for an NVN game, DM_AOC_DIR
with DLC, the --env-file switches). It refuses --prepare-xcode and an Intel Mac.
Reuse: the Mac libraries have a key of their own (shader/mac-b-<chain>.key beside the cache folder,
shader/vk.key / gl.key), which an iOS build writes too, so a Mac-only build after an IPA reuses its shaders;
either key only while the Mac libraries carry a full build's stamp (inputs.tsv). An iOS build after a Mac-only
one translates again (dmsh_build.py's per-shader keys include the iOS compile convention) and takes the Mac
libraries from the Metal compile cache. build_mac_host now has a reuse key too (the compile line, the sources and
headers of the folders it compiles, the generated profile header, libvpx, the third-party headers, clang and the
SDK), which also saves the rebuild for the Oodle / LZ5 decodes and the run-time captures. build_vk_shaders and
build_gl_shaders share layer_shaders. The default (ios) path builds as before.
Checked: dm/tools/tests 34 of 34 (11 in the new test_nsp_mac.py); Hades in Mac mode ran the Mac stages only,
printed the run command and exited 0, and a second run reused every stage, the host included (2 s); the host it
built boots Hades from that command. Breath of the Wild (NVN, with DLC) in Mac mode restored its 19,592 shaders
from the cache, printed a command with DM_AOC_DIR and DM_NVN_SHMAP, and the host booted it from that command into
play. Not run end to end here: a GL, provisional or derived game in Mac mode, and an IPA after a Mac-only build
(unit tests with the tools stubbed).
Co-authored-by: v0lumehi <34629096+v0lumehi@users.noreply.github.com>
- Engine: Unreal Engine 5.6.1 (Square Enix, the desktop renderer), the newest Unreal so far; everything in one IoStore container beside one pak; the sdk and subsdk0 builds are STEINS;GATE RE:BOOT's
- Rewriter, for every module: code before the first unwind entry is one function (each site there was a function of one instruction, so a byte jump table at main+0x219234 went unseen), and a function longer than 256 KB is analysed in pieces cut after a ret / br / b that no conditional branch or b within 64 KB crosses, with far branches and jump table targets checked against windows and retry blocks afterwards (this main's 82 MB without unwind entries took 100 GB of memory as one function; now 2.7 GB resident). Checked: DRAGON QUEST VII Reimagined's main byte-identical to its rewrite by the previous patch_nso.py; FINAL FANTASY RESONANCE's, Octopath Traveler 0's and DRAGON QUEST XI S's byte-identical between two different cut placements; only DQ VII's main among the 120 modules of 33 games on hand has other sites before its first unwind entry (none in a window or jump table)
- Shaders: no archive in the pak; the two libraries are IoStore shader library chunks whose tables and shader groups are Oodle-compressed, read through the game's own Oodle in the Mac host's tool mode with no boot (uepak.py Toc / io_archive / OodleHost; library names from CityHash64 of the manifest's archive names) and written as decoded version 2 archives; the NSP script uses them for any game with IoStore shader libraries. The pak's single-block files store block size 0 (Pak.entry). 73,496 unique shaders; 11 do not build (3 geometry, 8 compute with 3D image atomics), none met so far. 2,077 particle vertex shaders pick their texture slot at run time (tex_dyn_slots); CopyStencilToLightingChannelsPS writes integers to an R8Uint target (shader_facts_manual.tsv; without it the sprites were dark silhouettes)
- Host, for every game: a shader with run-time texture handles in a game without tex_dyn_slots freed its reflection twice and aborted the host (nvn_mtl.m); oe::MuteRecording / UnmuteRecording (no-ops)
- Memory: the FMallocBinned3 constructor call is cut and the host's allocator runs Unreal's heap (hle_uemalloc.c layout UE5_6: UE 5.5+ FMalloc has 32 slots, with MallocZeroed and two more TLS cache calls); the SDK's node area is cut to 62 MB. The engine's own page allocator only grows its use of the engine region (2.03 GB of address space over seven minutes, about 1 GB mapped), and a 2 GB region aborted in the first battle, so the region is 4064 MB in Engage's 5 GB block (this sdk build aligns regions to 2 MB, no alignment cut needed)
- Frame rate: 60 fps (30 on the Switch): the host writes rhi.SyncInterval 1 (hle_fpslock.c rows for this build; profile fps_target 60); the game steps by real time (DM_NVN_REGLUM traces at 30 and 60 fps match from the load through a walk)
- Mac: first-boot settings, the title, a new game, the opening cutscene, the first field and its autosave, Continue, and the first two battles at 59.7 fps, 2.3-2.6 GB resident, flat over seven minutes, no red line apart from a few texture type mismatches at one moment of the new game's load in some runs. Docs: docs/agents 01, 03, 04 and development.md
Each game gets its own project in <NSP folder>/<title ID>_build/xcode, registered in a shared NxNative.xcworkspace; a target phase copies the RomFS, shader libraries and add-on content into the app before Xcode signs it, and preparing again keeps the bundle ID and team chosen in Xcode. The unsigned IPA build is unchanged.
On top of the PR: the resource phase stops when the profile's shader chain is no longer the prepared shaders' (the host loads only its own chain), prepared projects ask for the Increased Memory Limit capability (DM_ENTITLEMENTS overrides), the source links are made again on every preparation, and scheme names keep non-ASCII letters. The PR's TOML launcher is left out.
Co-authored-by: Animax Deng <Animax.deng@gmail.com>
- Files and shaders: Unreal Engine 4.27 with IoStore containers beside one pak (version 11, its footer magic 0xcc bytes from the end, uepak.py magic_back); the 62,901 shaders of the pak's two shader code archives are LZ4 blocks, which uepak.py decodes itself (lz4_shaders), so the NSP script runs no Oodle host when no record is Oodle. One global shader writes integers into an R8Uint target (shader_facts_manual.tsv)
- Shader chain, for every game: a texture query on an image handle (c2 0x120 + 8 r: Unreal's GetDimensions on the RWTexture3D a compute shader writes) had no type in a facts table built without PC shaders, and the 2D default took a 3D image's depth from its width; the translator now translates such a shader again with the image types of its first translation (dmsh_xlate ImageTypesOf; dmsh_xlate.dll rebuilt, dmsh_chain.lock updated). Host, for every game: a texture binding of slot 32-39 reads the image bound in the image slot that shares its c2 word (nvn_mtl.m tex_slot_h; Hades II's texture slot 34 without an image still reads the texture). Unreal's volumetric lightmap indirection patch (7b958a7b21e72801) read the empty texture slot 32's 1x1 size and wrote only cell (0, 0, 0), so in the opening cutscene every character and prop sampled the atlas origin, a daytime level's bricks baked 2^15 brighter (white figures about 10^4 too bright). Monster Hunter Rise has four such queries. Every other profile's dmsh_chain.h rewritten with its c1_full / cb_extent lists carried over (the change touches no c1 or uniform read); DRAGON QUEST VII Reimagined rebuilt for Mac and iOS
- Host, for every game: stencil-mode texture builders (nvnTextureBuilderSetDepthStencilMode STENCIL) read the stencil of their depth-stencil storage (the hero was drawn as a dithered silhouette); the audio renderer at 32 kHz on the CPU path (the SDK makes its upsampler only for DSP rendering: UpsamplerManager::Allocate returns null and the host resamples 3:2); GetAudioDeviceOutputVolume; the shop and Nintendo Account calls of the title (ec Finalize / Cancel / GetErrorCode / ShowShopSubscriptionDetails, GetAuthorizationCode / GetState); kpr::GetKeyCodeMap; the movie output format's nv12-width / nv12-height beside nv12-uv-stride (Sofdec's VP9 movies were grey with bands)
- Memory: MallocBinned3 reserves 37 GB of pools in the engine region; its creation is cut and an observation gate hands the engine the host's malloc zone (patch_nso.py cut + gate OBS_UE_MALLOC, hle_uemalloc.c layout UE4_27; patch_nso.py places a gate where the scratch registers it uses are dead, scratch_dead); the region is 2 GB (game_args) with the SDK's node-area cut on this sdk build, in a 3.75 GB guest block; about 2.6 GB resident on the Mac
- Frame rate: this build's t.MaxFPS and rhi.SyncInterval in hle_fpslock.c; the profile keeps the game's 30 fps
- Diagnostics: DM_CTL (Mac) drives a running game line by line: presses, shots, pass and texture dumps, texlist (textures of a target with storage and block-linear layout), memdump, peek, Unreal console variables (cvarf / cvari), env; the pass dump prints buffer-to-texture copies and buffer textures' format, size and storage, and DM_NVN_CBDUMP saves the source bytes of 3D uploads (docs/agents 04, 05)
- Mac: the logos and opening movie, the title, a new adventure log, the opening cutscene (the well, the night scene with Maribel by the boat) into play at Pilchard Bay, and a saved log continued, at 30 fps with saves; no red line at run time apart from 62 of the 63 geometry shaders (one-pass cube shadow maps), which do not build. Unit tests: the graphics tests that build pass (t_refl's two sample libraries are not on this machine); 13 that compile nvn_mtl.m with hle_nvn.c already failed to build on main (g_ft defined in both, hle_nvn.c symbols missing from their stubs)
- The newest version in the folder is packed (-p / PACKAGE picks another). A registered version with every shader_versions version present builds from the profile as before
- Any other version (the base game alone, another update, other build IDs) builds from a profile derived from it in <id>_build/profile: the profile's keys, that version's build row, and shader tables made from its own files; a red line names the modules whose per-build fixes do not apply
- Host prewarm skips shader keys the shader map does not have (a list of another game version), counted in the "prewarm done" line
- Tested with Hollow Knight: Silksong: the base game alone (1.0.27776) packs with its own 2,244 shaders and plays the opening on the Mac; base + 1.0.30000 builds as before
- README: the update is optional; Silksong's row no longer names it
- Unity 6000.0.50f1 IL2CPP with FMOD; the bundles carry their type trees, so unityfs.py reads Unity 6's Shader objects with no layout written out (2,450 shaders, the one geometry shader does not build)
- Host: nn::hid::GetDebugPadStates, which Unity 6 polls every frame and the SDK's own aborts on (Result 0x2aca)
- Rewriter: the address-region node area cut on this sdk build (alias high-water mark +3.00 GB → +2.26 GB in the first room)
- profile_guess.py: every game with an NPDM system resource gets the 5 GB block; heapless ones (Silksong, Absolum, Pokémon Legends: Z-A) used to get a 256 MB alias region
- Prewarm list recorded from boot to the first two rooms of Moss Grotto
stdout carries one JSON object per line: stage, step, progress (done, total,
unit, est, note, frac, eta, elapsed; null while unknown), stage_end and say,
which carries every message, errors and a failed step's log tail included.
A count change goes out at most every 0.5 s, any other change at once; the
ticker now runs in json mode too and sends a count that was held back, so a
stage whose count moves rarely (the rewrite) no longer shows a stale count
until it ends. json mode never prompts; close() stops the ticker by an event
of its own and keeps the mode, so a second build in one Spyder session gets
its redraws again.
dm/tools/tests/test_nsp_progress.py covers the events, the held-back count,
the failed step's tail, the CLI error and the bar / lines / off output,
with no game files, keys or Xcode.
Co-authored-by: nocmt <nocmtone@gmail.com>
Unity 2018.3.10f1 IL2CPP with FMOD; its sdk and movie module are DRAGON QUEST XI S's builds, so
that build's rewriter cuts apply as they are, and no host change was needed. Brought up on the Mac
through the opening movie, the title, options, the tutorial and the prologue (choices, menu,
scenario chart, log) at 60 fps as on the Switch, with sound and saves; no red line at run time,
0.6 GB resident at the title and 0.77 GB after ten minutes of story.
- unityfs.py: Unity 2018.3's Shader layout (SHADER_2018_3, type hash 8db76007...). The game has no
bundles and its serialized files carry no type trees, so the shader split stopped. Written from
AssetRipper's TypeTreeDumps for 2018.3.10f1 and checked node for node against the dump (1,325
nodes with the alignment flags); all 48 Shader objects read to their exact end (450 subprograms,
452 shaders with main's GLSLC outputs). Other profiles' hashes are untouched.
- Memory: the hardware pool (3.2 GB) in Engage's 5 GB block. Unity 2018.3 reserves no address
regions of its own, so every mapped page lies in StandardAllocator's region (377 MB at the peak).
- Input: the game reads no touch, so it stays docked with the 2D pad; the left stick moves the menu
cursor.
Development notes, the graphics catalogue and the survey's list of Unity versions read.
A rerun after an interrupted IPA pack did a lot of the build again:
- The iOS shaders were keyed on the shader split's key, and that covers every dm/gfx/tools/*.py.
A pull that changed only the Unity readers (shader_split.py, unityfs.py) made Monster Hunter
Rise split again and then restore all 63,111 shaders from the cache (9 minutes). The iOS shaders
are now keyed on the shader package's content and on what dmsh_build.py loads (the repository
modules it imports, at any depth, and the chain lock). A split that runs again and writes the
same package leaves them reused; hashing Rise's 510 MB package takes 0.3 s.
- The DLC was extracted on every run (Rise: 252 DLC, 16-30 s). It is now reused while the DLC NSPs
(names, sizes, times) and nx_extract's sources are unchanged (aoc.key).
- Xcode's custom build location setting overrode -derivedDataPath, so every game's app was built
in one shared ~/Library/Developer/Xcode/DerivedData/Build: a build after another game's compiled
all 129 files, and two games built at once wrote the same app. build_ipa.sh sets SYMROOT and
OBJROOT under the game's work folder; a repeat build there compiled nothing (1 s).
Checked: Celeste with its split forced to run again (same package, iOS shaders reused), Rise's DLC
twice (15.9 s, then 0.1 s), and an Xcode build into a path with spaces and an apostrophe.
Each game's next build runs its iOS shader step once more, to write the new key.
New profile for Square Enix's Unreal Engine 4.18 game: from the opening movie and the title
through the prologue into free play at Cobblestone Tor, 30 fps with sound and saves, no red
line. An IPA builds from the NSP folder; it has not run on an iPhone yet.
- Files and shaders: Unreal 4.18 legacy (version 4) paks with LZ5 blocks, decoded on the Mac
with the game's own LZ5_decompress_safe (uepak.py, hle_oodle.c serving after nninitStartup);
38,892 shaders inline in the cooked packages as zlib-compressed LSLG records (shader_split.py
ue_legacy_jobs, ue_inline_shaders; the NSP script and profile_guess know both pak kinds).
- Memory: Unreal's address region capped at 4064 MB (game_args, MB up to 4095); SDK 7.3.2 cuts
in patch_nso.py (node area 62 MB, 32 MB alignment for large regions; the build Odyssey shares,
which boots unchanged) for Engage's 5 GB block. Rewriter: a jump table whose bounds check sits
right before the load (check_before_copy).
- Host, for every game: audio catch-up frames wait for queue space (Square Enix's sound driver
never waits on the renderer event: 11,000 updates a second before); movie decoder events read
at signal time as the SDK does, audio decoders keep 32 outputs (the player plays only once
every track holds one: the opening movie never started in half the boots), and no input after
an end-of-stream packet until Flush / Stop (a skipped movie mapped 2.25 GB of end-of-stream
samples); ec shop accessors, friends, auto-sleep, album, file data cache and movie hooks.
- Graphics, for every game: a geometry shader that only picks the layer (Unreal's
FWriteToSliceGS, the colour-grading LUT) folded into the vertex shader's layer entry points
(dmsh_build.py gs_slice / vs_layer_entry, nvn_mtl.m prog_fold_gs); cube-map array depth counted
in layer-faces in the Metal texture, the upload and the guest layout (t_nvn checks it);
nvnTextureGetTextureAddress and the command memory size / free getters.
- Shader chain changed (dmsh_build.py): DQ11 rebuilt for Mac and iOS; every other profile's
dmsh_chain.h rewritten with its c1_full / cb_extent lists carried over (no c1 or uniform read
touched). Mac checks on this host: Big Brain Academy (movie loops, 23 catch-up renders as
before), Monster Hunter Rise (movies), Super Mario Odyssey (boot on the cut sdk).
- MonoGame for the Switch, its C# and the game's compiled to C++ by Sickhead's BRUTE into main; FMOD Studio through AudioOut; only main and sdk. No host change: the logos, the title, a new file and the prologue at 60 fps with music and sound, no red line, about 1.65 GB resident, flat over five minutes. Not yet tested on an iPhone
- The shaders are MonoGame effects (MGFX version 8): 13 in Content/Effects/*.xnb and SpriteBatch's built-in one in main. shader_split.py reads them (mgfx_effect), each effect parsed to its end; a shader's bytecode is u32 control length, the control section, u32 code length, the code area. 26 shaders (20 pixel, 6 vertex), every binding decodes. A pass may name no vertex shader and keep SpriteBatch's (10 of the 13 effects), so they pair as single-stage programs. Cuphead's and 13 Sentinels' tables split again unchanged
- NSP script: a provisional profile never got the sys_resource the survey lists, so the host answered 32 MB of system resource for every new game; write_profile writes it now (Celeste: 0, as its NPDM declares)
- Survey: XNB files and the MonoGame and FMOD markers (profile_guess.py), and a note on where a MonoGame game's shaders are
- Prewarm list recorded through boot, the title, a new file and the prologue
Erick Sanchez