Commit Graph
430 Commits
Author SHA1 Message Date
seeddsandFernando Messias 62a640b037 Fix custom key loading for compressed game inputs (#204)
nsz 4.6.1 rejects --keys and loads keys.txt beside argv[0] during import.
Run the converter in its temporary output folder with the selected key
file linked there before import, using absolute input and output paths.
Remove the temporary link on both successful and failed conversions.

Integrate PR #204 with XCZ and relative-path regressions, rename the test
flag that shadowed unittest.TestCase.fail, and qualify the documentation
for the verified nsz release. Record the packing finding and provenance.

Validation: reproduced the exact parser failure using real nsz 4.6.1;
a synthetic NSZ conversion verifies selected-key lookup and link cleanup;
52 NSP tests, four documentation tests and git diff --check pass.
The synthetic payload is unencrypted and uses dummy keys. Real-game
decryption, actual XCZ conversion and a full IPA build remain unverified.

Co-authored-by: Fernando Messias <fernandomessias95@gmail.com>
2026-10-26 10:36:23 +08:00
seedds 435a231051 Preserve Horizon audio renderer work-buffer requirements
DSP callers were charged the local SDK CPU server budget, whose command representation is larger than the Horizon service representation. For Age of Calamity the service requires 0x61d000 while the local SDK requires 0x760000, exhausting the fixed sound heap.

Calculate the service requirement through renderer revision 15 and supply larger local backend storage separately, with cleanup on failed open and after renderer close. Remove the two game heap cuts and add sizing, nested-open, failure and lifetime regression coverage.

Validation: 360 seeded cases match isolated Horizon machine-code execution across REV1-REV15; the original game heaps initialize all five players and produce audio; the host suite, documentation and patch checks, SDK 8/20 smoke checks and iOS C compilation pass. iOS runtime and long battles remain untested.
2026-10-26 10:18:29 +08:00
seeddsandv0lumehi c4b74a62f9 Add thermal frame-rate cap with synchronized recovery (#220)
Hold the host frame step at 30 fps while iOS reports serious or critical
thermal pressure, and restore its configured cap after 30 calm seconds.
Keep the existing movie cap and DM_THERMAL_CAP=0 override.

Sample thermal state independently of movies and configured 30 fps targets
so heating resets recovery and cooling counts during playback. Serialize
the sample, cached cap and recovery timer across acquire/present readers.
The original PR failed five timing checks and ThreadSanitizer detected a
race through concurrent production cap getters.

Add controlled-clock production-path regressions and CI coverage, including
disabled/inactive caps and ThreadSanitizer. Record the shared mechanism
and validation limits in the frame-rate reference and BotW note.

Validated on latest main with the BotW profile: 24 thermal checks normally
and under ThreadSanitizer, 51 frame-step checks, 154 NVN checks, the
present-snapshot suite, 21 tool/docs checks, Mac host build, and iOS compile
check (133 sources). No new device gameplay or thermal recovery run.

Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
2026-10-26 10:13:42 +08:00
seedds c76f679834 Remove alpha_from_red: Super Mario Odyssey's layout captures need only the nvnTextureCompare fix. The game creates each capture with its per-format default mapping (an R8 one reads (R, 0, 0, 1)), rebuilds it on the same storage with the capture's own, (1, 1, 1, R), and replaces the registered descriptor only on a difference (main+0x982320); while the host's comparison ignored the mapping the captures kept alpha 1 (white coin cards, a dark box behind hints), which the 26-shader alpha_from_red list papered over. Checked on the Mac from the "Look Around" hint with a probe host on the comparison (a c3a0880^ host for the profile's chain dec5ee141a9f libraries): 40 mapping-only rebuilds in 11,700 frames (R8 captures 640x80 down to 48x57 to (1, 1, 1, R), a 320x180 RGBA one to (R, G, B, A)); ignoring the mapping again brings back the box (about 12,800 pixels from frame 4,200), the old list hid it by converting the 378x80 capture, and with the fix the list converted nothing and changed no pixel of frames 4,200-6,600. The comparison and the views have no platform branch, so the iPhone gets the same rebuilt mapping (not run on a device). With no profile left using it, the key, its host code (tex_alpha_red, afr_want, DM_NVN_ALPHA_RED) and its docs go; treg_drop_ar becomes treg_drop_im (the image-slot view it also drops stays). Odyssey's note has the A/B runs. Checked: the host builds for Odyssey, the iOS compile check (133 files), the tool tests (111, 14 skipped) 2026-10-26 10:01:20 +08:00
seeddsandparsifal295 2db12f7efd Fix vibration handles before controller haptics routing (#246)
GetVibrationDeviceHandles used routed haptics availability to decide whether
handles existed. Games that cache this answer at startup could retain an empty
answer after a controller connected. Return handles for a connected npad style
when rumble is enabled and an output sink is installed; check routed outputs
when values are sent, keeping the existing permission and lifecycle behavior.

Include the PR's rumble diagnostics, count unroutable values individually in
mixed batches, and protect aggregate counters across game threads. Correct the
code comment to describe Unicorn Overlord's observed empty handles and zero
values without claiming restored physical rumble.

Add regressions for cached startup handles delivering a nonzero value after
routing, diagnostic totals and concurrent sends. Include HID in the existing
ThreadSanitizer target and record the mechanism in the service reference.

Checked with GAME=010069401ADB8000: full host suite, 73 HID checks, audio and HID
ThreadSanitizer checks, docs checks, Mac host build, and iOS compilation
(133 files). Physical rumble and iOS runtime remain untested. Broader tool
validation is incomplete because Capstone is missing and iconutil failed.

Co-authored-by: parsifal295 <130349765+parsifal295@users.noreply.github.com>
2026-10-26 09:54:43 +08:00
seedds 16061e0399 Rule: fix a bug at its root cause, not around it. AGENTS.md says to find why a bug happens (the faulting instruction and its decompilation, the bad value and what wrote it, the semantics the host or translator gets wrong) and fix it there; a change that only hides the symptom (stub, skipped call, clamp, retry, sleep or timeout, cache flush, profile key that turns a feature off) is a workaround, used only when the cause lies where it cannot be fixed (the game's own code, changed deliberately by a rewriter cut, or a platform limit) or the user asks for a stopgap, and then recorded as open with the cause or what is unknown. docs/agents/README.md has the same as a ground rule. Checked: dm/tools/tests/test_docs.py passes (AGENTS.md 1,419 words) 2026-10-26 09:42:57 +08:00
seeddsandv0lumehi c13ddcf531 Add opt-in MetalFX spatial upscaling and a single-screen launcher (#215)
Scale the valid presented image with MetalFX's spatial scaler on the shared
Metal 4 path, preserving crop, letterboxing, origin and gamma-encoded colours.
Keep Standard as the default and fall back when scaling is unsupported.

Use the same path for asynchronous whole-window captures and diagnostic
screenshots, retaining resources until GPU completion. Bound scaler recreation
during resizing and report presentation GPU time separately by scaling mode.

Put all launcher settings on one screen with Game, Controls and Performance
sections, inline menus and sliders, and a reset to tested defaults. Extract the
settings model for native persistence tests, validate saved ranges, share HUD
setup, and document the measured costs without claiming an iPhone baseline.

Validated with Metal presentation, NVN and snapshot regressions; tool and iOS
tests; the Mac host; and an unsigned iOS smoke build. No new device gameplay run.

Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
2026-10-26 09:38:05 +08:00
seedds 3aa706a3fd Pokémon Shield (01008DB008C2C000, update 1.3.2 = v458752, Game Freak's engine on SDK 7.3.2): game profile, no host change. Sword's twin: the same sdk, subsdk0 (movie) and subsdk1 (GLSLC) builds, only main is its own (a1680262…), and the shader split gives Sword's 41,516 shaders with the same data rows in all three tables; the profile is Sword's with Shield's title, name, IPA name, build row and tables (prewarm list copied). The game note (dm/games/01008DB008C2C000/README.md) has what differs, the Mac route (a fresh save needs DM_KBD for the name; Sword's route out of the house works unchanged) and the shader build with DMSH_PREFILL from Sword's libraries (21,974 Mac / 21,684 iOS filled, 176 / 175 compiled). Plays on the iPhone (IPA 12.4 GB, user) and on the Mac at 30 fps with no red line from the first boot through the opening and Postwick to Hop. README gallery entry with a Mac screenshot, a row in the games table, a pointer from Sword's note, AGENTS.md's count to sixty-seven. Checked: the tool tests pass (111, 14 skipped) 2026-10-26 09:20:23 +08:00
seedds 3dd5d25a51 Add Echoes of Wisdom 1.1.0 port and correct cube fallback diagnostic 2026-10-26 09:14:43 +08:00
seeddsandv0lumehi 577df01c39 Fix iOS pulling shader packaging for profile defaults (#211)
Include pulling vertex libraries when the compiled game profile enables the
Metal 4 path, preserving explicit IOS_PULL overrides. Keep the pack options
array nonempty so inclusion also works with macOS Bash 3.2 and set -u.

Exercise the actual archive step with tiny shader fixtures for Breath of
the Wild and Wonder, checking both defaults and explicit overrides. Record
the packaging failures and fixes in the rendering reference and game note.

Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
2026-10-26 08:56:50 +08:00
v0lumehi 037ff69562 Profile keys present_snap and pso_mtl4: a game ships the presentation and pipeline settings it was tested with. present_snap on: present snapshots by default (a ring of 2 host textures; the iOS launcher's Frame buffering defaults to it; DM_PRESENT_SNAP and Documents/env.txt override); pso_mtl4 on: render pipelines through MTL4Compiler on iOS by default, with the pipeline archive, the prepare step and the hybrid pipelines (on by default on the Mac; DM_PSO_MTL4 overrides). Breath of the Wild sets both: they held a steady 60 fps on an iPhone 17 Pro Max (issues #142 / #159; without the snapshots the GPU idled between frames and iOS lowered its clock, 30 fps). README, docs/development.md and docs/reference/frame-rate.md say the new defaults. Checked: the host builds for Breath of the Wild (DM_GAME_PRESENT_SNAP 1, DM_GAME_PSO_MTL4 1) and Mario Wonder (0, 0), dm/tools/tests pass, t_present_snap and t_nvn pass with Breath of the Wild's profile, the iOS compile check passes 2026-10-26 01:43:25 +01:00
v0lumehi f6bd56f5cc The pipeline prebuild runs in the prepare step: with a window (and on the iPhone) the bases of every shader pair are made right after the prewarm, before the game's first frame, behind the progress screen ("Preparing pipelines"), once per chain version, depth mode and pair list (prebuilt.tsv). Built in the background while playing, they took an iPhone 17 Pro Max's GPU from about 13 to 15-18 ms a frame in Breath of the Wild (the compiler's load on the shared power budget: 43-95 dropped frames per 300 against 0-25 without it); without a prepare step the background prebuild stays. Tests: t_nvn 128, t_pso_archive 57, t_pull 346, iOS compile check 2026-10-26 01:43:20 +01:00
v0lumehi 1eeb109b35 docs: the prepare step (docs/development.md "The prepare step", Breath of the Wild's measurements in its note dm/games/01007EF00011E000/README.md; dm/gfx/README.md DM_PSO_PREPARE; README). Mac, M2 Max, runs started at a 1-minute load below 4, Metal cache empty, the prebuilt archive, the Zora's Domain save: with no list, 4 frames over 25 ms with new pipelines and 5 with shader library loads (75 libraries, 168 ms of that work, worst 128 ms) in the 60 s after the last menu press; with the new pso_manifest.tsv none, whether it was built in the background (14.6 and 16.0 s on one worker, done during boot) or by the prepare step (5.2 and 2.2 s on 12 workers, 3.3 s on 8, 872 bases compiled); a second start prepares it in 0.08 s and a player's 5,783-line list in 0.23 s, both with no screen. Hades (Vulkan) is unchanged: it builds its 1,856 pipelines itself while it loads 2026-10-26 01:43:20 +01:00
v0lumehi d9aa864d0d Breath of the Wild: a prewarm seed list (pso_manifest.tsv, 1,789 lines: 960 render pipelines, 819 shaders, 10 compute shaders), so a first start has something to prepare. Recorded on the Mac with prewarm off (DM_PSO_MANIFEST=off DM_PSO_RECORD) over update 1.9.0: boot, title, New Game and the Shrine of Resurrection until Link can move (1,298 lines); then boot, title, a save load into Zora's Domain and a minute of play, whose new lines are appended (491). The lines name shaders by their Switch content key and pipelines by NVN state (pso_skey), which no translation chain version changes: the save-load part was recorded on chain b7b9aa970b37 and every line resolves on 97e2f301eb73 2026-10-26 01:43:20 +01:00
v0lumehi 7a47d1c5eb The prepare step: with a window, the game's first submission waits until the prewarm lists (pso_manifest.tsv, then pso_user.tsv) are built, on every core at user-initiated QoS, behind a "Preparing shaders" screen that shows only past 0.5 s and with 0.3 s or more still to go. nvn_mtl.m: the prewarm reads its lists into tasks (warm_parse): a shader, a compute pipeline, or all render pipelines of one shader pair, which share their bases, so no two workers compile the same base; shaders go into one dictionary the workers share, and a vertex shader's pulling entry is loaded with it (as shader_prefetch_rest does), so neither a draw nor its hybrid stand-in loads a library the lists name. warm_prepare runs the tasks on hw.activecpu workers (DM_PSO_PREPARE_THREADS=<n>) from mtl_sub_begin's first call, before anything of that submission is encoded, and gives the front end done of total every 50 ms once 0.5 s have passed, unless the rate so far leaves less than 0.3 s; the shaders and pipelines go into the caches the draws use at once (warm_claim), which keep them for the run. It runs at the first submission and not before the game starts because the vertex entry each pipeline is built with follows the depth mode the game sets (nvnDeviceSetDepthMode). It runs at every start: with the Metal cache and the pipeline archive warm it takes well under a second and shows nothing, and a stamp that skipped it would skip it exactly when the system has dropped the Metal cache (an OS update, storage pressure on iOS). Without a window, or with DM_PSO_PREPARE=0, the prewarm runs in the background on one worker as before; DM_PSO_PREPARE=1 makes a windowless run wait too. The log line: pipeline prepare: N lines (shaders, render and compute pipelines) in s on N workers before the first frame, progress screen shown or not; bases compiled, from the archive, reused. front.h front_progress: front_mac.m draws the screen with AppKit views over the game's view, front_ios.m with UIKit views over the game view and the pad (a dark view, the title, a bar and a count); nvn_mtl.m keeps a weak empty one for the unit tests. NVN only: the Vulkan and GL layers keep no prewarm lists 2026-10-26 01:43:20 +01:00
v0lumehi 9607f51022 iOS app: shader_pairs.tsv and pso_pairs.tsv go into the bundle beside the shader map (build_ipa.sh), where the pipeline prebuild reads them; without them an iPhone made no pulling bases and hybrid draws never had a stand-in (Breath of the Wild's log: 'pipeline prebuild: no …/dm/shader_pairs.tsv, nothing to build', 0 draws with a hybrid pipeline in 15,600 frames) 2026-10-26 01:43:20 +01:00
v0lumehi e0ef4e707e Pipeline prebuild: the pulling base of every known shader pair into the pipeline archive, with the pairs of agl's BAHS program variations (pso_pairs.tsv)
A pulling stand-in is used only when its base is made or archived, so a cold start's draws waited until the prebuild puts the base of every known pair into the archive (pso_prebuild): in the app after the prewarm, on a utility queue on half the performance cores, flushing every 1,000 bases (DM_PSO_PREBUILD=0: off), or as a tool on the Mac (DM_PSO_PREBUILD=tool: build, write, exit; DM_PSO_PREBUILD_DEPTH=zero, DM_PSO_PREBUILD_THREADS=<n>), once per chain version, depth mode and pair list (pso_archive/prebuilt.tsv). The NSP script runs the tool in a --platform mac build (pso_prebuild.log). Files that hold only bases without a vertex layout are named p<seq>.mtl4a; a base with a vertex layout is never looked up in them (each file a lookup misses in costs about 0.06 ms) and compaction leaves them alone.

The pairs: shader_pairs.tsv paired the stages of one BNSH variation and none of the pairs in agl's BAHS shader archives (.sharcb: the agl / gsys / terrain resource packs), and in Breath of the Wild 109 of the 543 pairs drawn on entering Zora's Domain came from those, so they had no stand-in. shader_split.py now reads a BAHS file's binary array (record size, kind: 0 vertex, 1 fragment, 3 compute, the entry bahs_shaders reads) and program array (record size, name length, stage mask, first binary index), and pairs the n-th vertex binary of a program with its n-th fragment binary (bahs_variations); a file whose arrays do not read back gives none. Those pairs go to a new table, pso_pairs.tsv, which only the prebuild reads (DM_PSO_PAIRS: another path): vs_pad keeps reading shader_pairs.tsv, so no shader library, inputs digest or other table changes. Breath of the Wild: 3,012 more pairs, and all 543 pairs drawn are now known; its shader_map / facts / pairs tables split again byte-identical. A profile made before the table existed still packs: the NSP script compares pso_pairs.tsv only where the profile has one, and otherwise keeps it beside the shader package and hands it to the prebuild. Seven other profiles have BAHS files (Odyssey, Wonder, Mario Kart 8 Deluxe, 3D World, Animal Crossing, Captain Toad, Tears of the Kingdom); their tables are made at their next pack.

Breath of the Wild, Mac (M2 Max), Zora's Domain save, empty Metal shader cache, no pso_user.tsv. Prebuild: 15,767 bases, 910 MB in 16 files, 187 s with 7 workers for the first 12,755 and 19 s for the 3,012 more; the files open in 2.2 s cold, 0.25 s from the file cache, and add about 850 MB of footprint. The frame entering the area: 340 ms with pso_pairs.tsv, 830 ms with shader_pairs.tsv alone, 990 ms through the MTLDevice (then frames of 738, 535 and 496 ms); draws that waited: 50 (20 ms), 102 (650 ms) and 512 (3,325 ms). The picture is the same.

Tests: t_pso_archive (41 checks: also a base without a layout flushed into a p file, and a base with a layout not looked up there); the other graphics tests, the host unit tests and test_nsp_mac / progress / chain give the results they give on main. Not run: an iPhone (the prebuild is off there with the Metal 4 path), the NSP script's pack end to end, the other BAHS profiles' splits. docs/development.md, dm/gfx/README.md and dm/games/README.md describe the prebuild and the table; Breath of the Wild's note (dm/games/01007EF00011E000/README.md, Pipelines on a cold start) has its measurements.
2026-10-26 01:43:20 +01:00
v0lumehi 8593d85a9c docs/development.md: a diagram of the hybrid miss path (cache, base and specialization, pulling stand-in with the draw's own pipeline queued first, or a build at once with the bounded prediction) 2026-10-26 01:43:20 +01:00
v0lumehi bc76cab190 Hybrid pipelines: a draw whose pipeline is not built yet draws with its shader pair's pulling pipeline meanwhile, on main's demand-first pipeline builds
On a miss the draw first takes its pipeline without compiling (its base made or in the archive, then a specialization). Failing that it draws with the pair's pulling pipeline (<key>_bp: no vertex descriptor, the streams read through a per-draw layout block at buffer 16, pull_bind), specialized with the draw's colour state, while its own pipeline is built in the background; a later draw takes that one. Both compute the same positions (t_pull_inv), so a draw that switches does not flicker or z-fight. Only a pulling base already made or archived is used; otherwise it is built in the background for the pair's next miss and the draw waits. Merged passes, readable-tile and diagnostic fragments, and layer-writing or transform feedback vertex shaders have no stand-in. DM_PSO_HYBRID=0 turns it off; it needs the Metal 4 path, so it is on by default on the Mac only.

On main's demand-first pipelines (#75): pso_pend_kick stays the speculative admission (at most two running, at most two per prediction, the oldest finished entry reclaimed when the table is full). A draw with a stand-in queues its own pipeline before the prediction runs (pso_pend_demand): outside the speculative limit, on serial queues at utility QoS, one per two performance cores (pp_bg_queue), so a demanded build never waits behind speculation and speculation never waits behind it. The prediction spends its budget only on draws that will have no stand-in, since only those would wait. A draw that would wait for a build still queued makes the pipeline itself at once (pso_pend_take; Breath of the Wild's cold start waited 0.5-0.9 s for one pipeline behind a burst before). The first miss of a submission loads the shader libraries the rest of it binds in parallel (shader_prefetch_rest): 248 libraries in about 210 ms where one by one on the submit thread took about 1 s, in the hybrid and the waiting path alike.

Diagnostics: a "pipelines up to frame" line every 300 frames (submit-thread pipeline, library and prediction time, archive lookups, specializations, base compiles, hybrid draws, background builds, the draws that waited and why), DM_PSO_WAITLOG=1 (each wait over 20 ms with its pair), DM_PSO_PULL_ALL=1 (every draw with a pulling pipeline draws with it), and the longframe line's count of hybrid draws.

Breath of the Wild, Mac (M2 Max), steady state in Zora's Domain: mean GPU p50 over frames 2,400-5,100 was 8.98 ms with the hybrid, 8.98 ms on the Metal 4 path without it, 9.35 ms through the MTLDevice. Cold-start numbers are with the prebuild of the next change.

Tests: t_pull (346 checks, now also the layout block the host fills), t_pull_inv, t_pso_pending (48) and t_pso_archive (32) pass; the other graphics tests and the host unit tests give the results they give on main. docs/development.md ("Hybrid pipelines") and dm/gfx/README.md describe it.
2026-10-26 01:43:20 +01:00
v0lumehi e6b800a56d Pulling vertex shader variants in the translation chain (<key>_bp, dmsh_pull.py with its decoder dmsh_pull.metal)
A side library of every vertex shader with stage inputs, as <key>_bv is, whose main0 / main0_z have no [[stage_in]] and read their inputs from a per-draw layout block at buffer 16: per stream the base address, stride and divisor, per NVN attribute its state word, the base instance from [[base_instance]] (so indirect draws are right too). dm_fetch decodes the NVN format at run time into the register words Maxwell's fetch gives the shader, and the input takes them with as_type, so the body runs unchanged. A pipeline built with it has no vertex descriptor and serves every vertex layout the pair is drawn with; the hybrid pipelines (next change) draw with it while a draw's own pipeline is built.

The words are bit-identical to Metal's vertex-descriptor fetch for every format nvn_mtl.m vfmt knows and to the host's CPU conversion for the integer-read-as-float formats: unorm / snorm through the unpack builtins (a plain division under fast math is a unit in the last place off for up to half the values), R11G11B10F NaNs canonical as from Metal, divisors with Metal's rule base + (instance - base) / divisor. Positions are [[invariant]] and built with preserved invariance, so a pulling pipeline and the vertex-descriptor one compute the same positions. Transform feedback, geometry capture and layer entries get no variant; shaders without stage inputs need none. The run-time decode costs GPU time: +65 % of a vertex-bound draw with 8 attributes on an M2 Max, about 1 ms per million vertices, so it is meant only to bridge the frames until the draw's own pipeline is built.

The iOS app does not use the variants yet (the Metal 4 path is off on iOS by default), so build_ipa.sh leaves the _bp libraries out of the IPA unless IOS_PULL=1 (pack_ipa.py --skip-suffix).

Chain version 8393d85a4eae (on main's 20fe3d62b9f6). Breath of the Wild's full Mac build on chain 97e2f301eb73 (25 min, a full retranslation since the code fingerprint changed): 19,592 shaders ok, 8,888 pulling libraries generated (275 MB), none failed, 43 vertex shaders without stage inputs; every existing library is byte-identical to the previous build (19,592 _b.metallib and _b.metal, 6,254 _bf, 1,070 _bv). Other profiles' shaders were not rebuilt.

Tests: t_pull.m (304 checks: the 76 formats vfmt knows, 73 comparisons with Metal's fetch at aligned, 2-byte and stride-1 placements, 24 with the CPU conversion, stride 0 and divisors 1-3 with a base instance), t_pull_inv.m (2,000 random triangles: a depth prepass with one pipeline and depth-equal with the other passes every covered pixel, both ways round) and test_dmsh_pull.py (the generator: entries, ids it adds or keeps, entries left out, shaders declined; compiles for macOS and iOS); pack_ipa.py's --skip-suffix checked by hand. The other graphics tests and the host unit tests give the results they give on main. docs/reference/rendering.md ("Pulling vertex inputs") describes the variant and its cost; AGENTS.md lists the new chain files.
2026-10-26 01:43:14 +01:00
seeddsandAleksandr Svidchenkov 1a93086e2d Fix service-event and vsync worker lifetimes (#202)
Apply PR #202's host-thread skip and duplicate vsync suppression. Give
bindings unique lifetime IDs so a retired worker cannot signal another
service's event after guest storage and kernel handles are reused. Replace
the eight-entry worker table with allocated records so rapid replacement
does not abort while retired workers wait for their next tick.

Add deterministic regression cases through real event binding and SVCs,
and document the mechanism with an updated symptom index. Both the
capacity abort and stale signal reproduce against the original PR.

Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
2026-10-26 08:35:19 +08:00
seedds 8b6f907fc1 Merge pull request #243 from seedds/codex/thread-start-handoff
Honor same-core priority handoffs when starting threads
2026-10-26 08:31:38 +08:00
seedds 02e40ba753 Record threading audit and retain verified game workarounds 2026-10-26 08:22:50 +08:00
seedds c188131824 Honor same-core priority handoff when starting threads 2026-10-26 08:22:50 +08:00
seeddsandAleksandr Svidchenkov 4896c78d8a Fix packaged texture dimensions and linear storage sizing (#201)
Decode packaged byte 0 as separate block-height and block-depth nibbles,
including explicit zero dimensions, and size linear compressed textures
by pixel rows. Port the driver-derived rules from PR #201 and preserve
its attribution to digitalhivehub's PR #11 measurements.

Integrate the layout test with current main, correct its BC4 label, add
independently addressed CPU markers and packaged GPU mip coverage, and
record the mechanisms in the reference docs and symptom index.

Validation: 9 driver shapes, 189 CPU texture checks (also ASan/UBSan),
17 GPU deswizzle cases, 154 NVN checks, documentation checks, Mac host
build, and iOS compilation of 133 sources pass. No new gameplay or
iPhone runtime validation.

Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
2026-10-26 08:19:08 +08:00
seeddsandAleksandr Svidchenkov c1b7eff61f Fix Metal buffer alias visibility and sparse copy ordering (#200)
Carry ResourceAlias visibility on copy encoders containing buffer copies,
inline updates or fills, using the contribution from PR #200. Operations
through sparse virtual mappings also need ordering against surrounding
operations: disjoint guest addresses can name the same physical bytes,
so guest-range dependency detection alone misses their hazards.

Add a regression using distinct no-copy Metal buffers and real sparse
mappings. The original host failed 37 of 101 checks on an M4 Pro and
PR #200 alone left 24 failures; the complete fix passes all 101 checks
with both direct and deferred encoding under Metal validation. Ordinary
independent buffer operations retain range-based batching.

Validation: t_nvn passes 154 checks; the Mac host builds; all 133 iOS
source compile checks and the documentation checks pass. Monster Hunter
Rise gameplay, iPhone execution and sparse-barrier performance were not
retested.

Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
2026-10-26 07:28:09 +08:00
seedds 31b5944eb4 Port NieR Automata with 60 fps and geometry fixes 2026-10-26 07:13:22 +08:00
seeddsandAleksandr Svidchenkov a5513d4ae0 Fix Depth16 load-clears to match fragment depth writes (#199)
Apple GPUs truncate Depth16Unorm load-clears while rounding fragment depth,
so clearing to 0.5 stores 32767 instead of the fragment's 32768 and fails
an EQUAL comparison. Apply the format-specific correction to both
attachment and whole-texture clears, preserving other depth formats.

Exercise both production command paths against independent GPU fragment
writes for 1,042 float depths, including half-bin boundaries and adjacent
floats. Removing either correction produces 518 mismatches on M4 Pro;
with both corrections enabled, neither path differs. Record the mechanism
and the pending TOTK highlight and iPhone validation in the documentation.

Validated on M4 Pro with Metal validation: t_depth16_clear, t_clear_ds
(43 checks), t_clear_tex (49 checks), and the documentation tests pass.

Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
2026-10-26 07:06:32 +08:00
seedds 9a56ecf0c1 Order same-core wake handoffs and remove Dread movie gates
Use guest runnable state after a higher-priority same-core wake instead of Mach dispatch state or a 5 ms cutoff. Honour wake handoffs for every game and remove Metroid Dread request-ring gates and its core_preempt setting.

Validate event, mutex and condition-variable handoffs with delayed host dispatch; 13 valid Dread movie-transition runs passed. Host tests, docs and shader-chain checks passed, along with an iOS syntax check. Record Wonder boot comparison and pending iOS runtime validation.
2026-10-26 02:27:03 +08:00
seeddsandAleksandr Svidchenkov 9feac2f1e7 Rescue stalled display links with bounded retries (#198)
Switch active-app presents to drawable acquisition after 500 ms of display-link
silence, then retry with exponential backoff capped at 60 seconds. Preserve the
requested pacing mode, synchronize rescue state, and prevent focus timestamp
underflow from bypassing the foreground grace period.

Add CPU regression coverage in CI, a real Metal layer ownership check, and the
mechanism reference with upstream provenance and validation limits.

Validated: 43 CPU rescue checks; 47 checks with real Metal drawable acquisition;
34 present/crop and 10 pacing checks; Wonder Mac host build; all 133 iOS source
compile checks; documentation tests. Windowed gameplay recovery and iPhone
validation remain outstanding.

Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
2026-10-26 02:25:06 +08:00
seeddsandAleksandr Svidchenkov 5b0c11a4f3 Fix NVN dirty-page upload races (#197)
Take upload stamps before protection, initialize cached checks from those
stamps, and wait out pages another thread is unprotecting before retrying.

Publish page generations before advancing the clock again in both the fault
handler and batched host pre-write path. A shared stamp_page helper closes
the remaining batched publication race in the original PR.

Add deterministic publication regressions for both paths and document the
mechanism and the limits of the game-level evidence. The regressions fail
on main, and the batched case also fails with the original PR applied.

Validation: full graphics suite, Wonder Mac host build, all 133 iOS source
compile checks, and documentation tests pass. No fresh gameplay or iPhone
runtime validation.

Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
2026-10-26 02:06:27 +08:00
seeddsandAleksandr Svidchenkov 1fa521958a Merge PR #196: deduplicate normalized NVN pipeline keys
Drop unused normal-pass colour-target blend and mask state from pipeline keys.
Normalize historical prewarm rows and deduplicate their parsed keys before
compilation, preserving merged passes' logical target state.

Include production-path GPU regression coverage and the reference/game notes.
Validated on the latest main: Wonder host build, key serialization tests,
48 pending-pipeline checks, 48 archive checks, 158 prewarm GPU checks across
MTLDevice and Metal 4, and documentation checks. The regression fails on the
original base and detects wrong pixels if the merged-pass guard is removed.

Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
2026-10-26 01:54:19 +08:00
seeddsandAleksandr Svidchenkov 796d592aa8 Test and document NVN prewarm key normalization
Exercise historical manifest rows through the production warmer, shader loader,
pipeline builders and draw-time cache. Check normal and merged passes on both
MTLDevice and Metal 4, with GPU pixels compared against the original descriptors.
The regression fails on the PR base and when the merged-pass guard is removed.

Document the shared normalization rule and keep game measurements in their notes.

Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
2026-10-26 01:52:46 +08:00
seedds 6b947d9504 Fix Xcode workflow mock for the rewrite profile return
Return the fixture profile from the rewrite mock so the preparation pipeline keeps a valid game profile after the Portal change.

Validation: all three tests in dm/ios/tests/test_xcode_workflow.py pass; git diff --check passes.
2026-10-26 01:46:07 +08:00
seedds a8e76bf601 Port Portal: Source engine NROs on NVN, shaders from game.zip
- profile 01007BB017812000 (1.0.3, SDK 13.3.0): plays on the Mac at 60 fps
- provisional profiles add the subsdks the NROs import (profile_guess nro_modules)
- an NRO importing nvnBootstrapLoader makes an NVN game; NROs calling glslcCompile get the run-time GLSLC capture
- shader_split reads zip archive members (Portal's nxcontent/nvsb GLSLC outputs)
- ro::LookupModuleSymbol for a name a loaded module lacks is not red (Horizon fails it too)
- AGENTS.md: a Mac gameplay test is enough to count a game as playable (65)
2026-10-26 01:28:35 +08:00
seeddsandAleksandr Svidchenkov 6d4202b562 Merge PR #195: fix NVN autorelease ownership and registry contention
Keep the latest upload-staging and compact-binding fixes from main. Add a real-Metal readback lifetime regression and reference documentation for all three general fixes.

Validation: nine Metal test runs pass, including direct and deferred upload-budget tests; Wonder host build, 133-file iOS compile check, and documentation checks pass.

Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
2026-10-26 01:27:04 +08:00
seedds 2d344f050c Test NVN readback lifetime and document general Metal fixes 2026-10-26 01:25:40 +08:00
seedds caee62a853 Compact NVN buffer bindings and remove Wonder heap workaround 2026-10-26 01:21:10 +08:00
seedds 7dc250e2f1 Bound NVN upload staging and retire it on upload completion
Split GPU upload batches at 128 MiB and retire overflow and CPU staging when the upload completes, rather than retaining it until the main rendering fence. Scope weak-reference diagnostics to avoid temporary staging retention.

Add a real-Metal regression covering memory bounds, upload pixels and an open render pass in direct and deferred encoding modes. Both modes pass 261 checks; documentation checks and the TotK host build pass.

Refs #147. Reporter-device validation and the full-device reboot investigation remain open.
2026-10-26 01:05:44 +08:00
Aleksandr Svidchenkov 2ef0a17e15 Pipeline keys: the write mask and blend bit of colour targets a normal pass lacks leave the key too, and prewarm rows are normalized when read (general, from MacNativeGames)
pso_key_norm (66abbc3) drops the blend factors of targets that do not blend. It now also clears, outside a merged pass, the write mask and blend-enable bit of a colour target the pass has no format for: pso_desc reads them only for targets with a format, and they keep whatever earlier draws left. A merged pass keeps them, as its blend / mask index logical targets through lmap. MacNativeGames f3e2f66 found 35% of a Tears of the Kingdom prewarm list's render pipelines (8,479 of 24,146) differing only in such state, each compiled again.

pso_key_norm moves to nvn_pso_skey.h with the merged flag, and pso_skey_parse normalizes, so a prewarm list recorded before gives one key per pipeline; the prewarm takes each parsed key once (the line check alone let such rows through as different lines). t_pso_skey: keys round-trip normalized, two keys differing only in unused blend / mask state parse equal, a merged pass keeps the mask and blend bits. t_pso_archive gives every target a format, so its normalization checks still test the blend rule alone.
2026-10-25 20:18:40 +04:00
Aleksandr Svidchenkov 5a81e13f24 NVN on Metal: no autorelease in ReadTexels and arena_alloc, and a registry lock that lends priority (general, from MacNativeGames)
- mtl_read_texels runs on the game's own thread outside a submission, where no pool drains what Metal autoreleases (the command buffer, the blit encoder, the read-back buffer), and that thread lives for good: every GPU read-back stayed alive. It runs under an @autoreleasepool now. MacNativeGames 635fb31 measured Breath of the Wild's ~130 read-backs a second holding 3.7 GB of GPU memory after 30 minutes.
- arena_alloc's buffer out-parameter is __strong: an unqualified id * is __autoreleasing under ARC, which put the arena buffer in the autorelease pool at every call (a few per draw). The arena also keeps its length, so an allocation, arena_new and the idle trim send no message. MacNativeGames 0531f28: autorelease 2.63% -> 1.73% of replay time.
- The texture / sampler registry lock g_rmu is an os_unfair_lock, which lends a waiter's priority to its owner. Game threads (priority 46) take it to register textures and samplers, the submit thread (97) for every argument encode; preempted while holding the mutex, they kept the submit thread waiting 1.8 ms a frame in a Tears of the Kingdom shrine (MacNativeGames 5f5de5e: 155 -> ~0 ms/s of waiting, frame time unchanged there). t_stencil_view, t_view_levels and t_view_swizzle take the lock the same way.
2026-10-25 20:18:26 +04:00
seedds be5b197640 Port Persona 5 Royal with Needle shaders and NVN texture fixes 2026-10-25 23:49:24 +08:00
seeddsandb4f8abbb24dbf56a7d0280f296e16eeb c3a0880304 Guard Metal image atomics and reject unsupported bindings
Apply the generalized image-atomic bounds and backing checks from PR #178.
Also reject unsupported descriptor formats before integer fallback and fail
unresolved bindless atomics instead of silently deleting them. Rebuild the
translator and record chain 467efdfd0b62, with regression tests and findings.

Validation: compiler/helper and sanitizer tests, 96 Metal compile/link checks,
ten GPU probes, all 28,603 Z-A shaders for Mac and iPhone, and a 240-second
Mac station-area run reaching 6,987 frames without a freeze or GPU fault.
Reported iPhone cases and the station-to-world transition remain unverified.

Co-authored-by: b4f8abbb24dbf56a7d0280f296e16eeb <51352400+b4f8abbb24dbf56a7d0280f296e16eeb@users.noreply.github.com>
2026-10-25 23:37:28 +08:00
seeddsandXc1Ym 4f29560fac Merge PR #166: match verified game patches across updates
Integrate the Wonder 1.0.1 resource-heap fix and shared opt-in function matching for compatible game updates.

Validated with 95 tool tests and byte-identical Wonder 1.2.1 rewrite outputs for registered and synthetic unregistered build IDs.

Co-authored-by: Xc1Ym <29765332+Xc1Ym@users.noreply.github.com>
2026-10-25 23:26:44 +08:00
seedds be30c4aa55 Merge original PR 166 history into the generalized patch update 2026-10-25 23:21:58 +08:00
seedds b5c78fbddb Match verified game patches across compatible updates 2026-10-25 23:21:17 +08:00
seedds ec4ea16da0 Enable ASTRAL CHAIN 60 fps with corrected game timing 2026-10-25 20:59:04 +08:00
seedds 7a3b95873b Fix Mac app missing-module checks and quoted alerts 2026-10-25 19:49:34 +08:00
StanandClaude Opus 5.5 4213e5ea79 EoW A-2: docs for the Mac app (--platform mac-app)
docs/development.md: a new section "A Mac app" after "A Mac-only build for development": what --platform mac-app builds and where the app goes (-o), the bundle's layout, APFS clones or copies, where the save, pso_user.tsv, temporary files, env.txt and the log go, signal forwarding and the alert, why the launcher runs the host as a child process (macOS privacy checks on removable volumes: a launcher that exec'ed dmhost got EPERM on every open on the external drive, signed or not), and signing (ad-hoc or --mac-sign, never the hardened runtime, so no notarization). "A Mac-only build for development" no longer says there is no Mac app. The script's examples get one for --platform mac-app --mac-sign, and a line lists the three --platform values. README.md "Pack an IPA": one line pointing at the Mac app.

Docs only; no test ran for this commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 19:49:21 +08:00
StanandClaude Opus 5.5 ffcd910986 EoW A-1: a self-contained Mac app of a game (--platform mac-app)
The NSP script's --platform mac-app (Spyder: PLATFORM = 'mac-app') does everything --platform mac builds (the Mac shaders, the Mac host for the profile, the button-icon font), then packs <NSP folder>/<game name>.app (-o / OUT_IPA names another path; ':' and '/' in the game name become ' - '), which boots the game when opened from Finder, from any folder, without the build folder. New dm/mac/pack_app.py packs it: the launcher dm/mac/launcher.c (compiled for arm64 at the host's deployment target) as CFBundleExecutable, the host beside it in Contents/MacOS, and in Contents/Resources what the IPA carries (build_ipa.sh's choice): romfs/, msl/<b-chain | vk | gl>/ (the Mac libraries only, not the cas/ compile cache, which the host never reads), aoc/<index>/, and dm/ with the rewritten modules and the rewriter tables the host reads (*.patched.bin, fixups, gates, cuts, obs, nro.tsv; not sites.tsv), the profile's shader_map.tsv, pso_manifest.tsv, c1_full.tsv and cb_extent.tsv when present, font/ and the --env-file as env.txt; AppIcon.icns from the game icon (sips, iconutil); Info.plist with the NACP name and display version, the IPA's bundle ID and the profile's app_category, plus DMTitleID, DMRomfsMarker and DMLanguage for the launcher. The large folders are APFS clones (cp -c) when a clonefile into the app works (the same APFS volume), plain copies otherwise after a free space check. The app is written as <app>.partial and swapped in whole.

The launcher sets the host's environment from the bundle (as the iOS app's main.m does) and runs the host as a child with posix_spawn, never exec: macOS privacy (TCC) attributes a child's file access to the app, while a launcher that hands its process over (a shell script exec'ing dmhost) got EPERM on every open on the external drive, silently, signed or not (found on Echoes of Wisdom's hand-made wrapper). Save and temporary files go to ~/Library/Application Support/nxnative/<title ID>/ (save/, temp/; pso_user.tsv lands beside save/, which is also DM_ROOT and the working folder), the host's output to ~/Library/Logs/nxnative/<title ID>.log (the previous one kept as .prev.log). Environment switches: <Application Support folder>/env.txt, then the bundle's dm/env.txt; neither replaces a variable already set. It forwards SIGTERM, SIGINT and SIGHUP to the host, returns its exit code (128 + the signal when it was killed), and shows an alert (osascript) when the bundle's game data is missing.

Signing: the inner host first, then the bundle, codesign --force --sign <identity>, ad-hoc by default, --mac-sign / MAC_SIGN names an identity; never with the hardened runtime (it refuses the executable memory the host maps for the game's code), so the app cannot be notarized. Finder information on copied files, which codesign refuses, is removed and the signing tried again.

In the script, mac_run is split: mac_host builds the host and the font (the stage "Mac host", for both platforms), mac_run makes the command of --platform mac as before, and mac_app runs pack_app.py (the stage "Mac app", its steps from pack_app.py's step lines); --platform mac-app plans the same number of stages as an IPA build.

New test dm/tools/tests/test_nsp_mac_app.py (written first; on main it fails: no dm/mac/pack_app.py, no --platform mac-app): packs an app from stand-ins with a stand-in host compiled there, and checks the layout, Info.plist, the ad-hoc signature without the runtime flag, the launcher's environment, working folder and log rotation, both env files, the exit code, SIGTERM forwarding, the alert (a stand-in osascript), plain copies, and the script's options and pack_app.py command line for an NVN and a Vulkan game. Checked: every test of dm/tools/tests, dm/ios/tests and dm/gfx/tests, make -C dm/host/tests GAME=010015100B514000 run and dm/ios/tools/compile_check.py pass. Not checked here: an app of a real game (next), a GL game, an app moved to another Mac.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 19:49:21 +08:00