At -O2, CorrelatedValuePropagation and SimplifyCFG fold a translated
shader's repeated tests of one value into switches whose cases jump past
their own join. Apple's GPU backend then runs out of control-flow IDs
("Max Unstruct ID 758 ... exceeds supported limit", a note: metal-tt
still exits 0) and lowers the function a far slower way. Odyssey's
alRenderMaterialUber took 24.6 minutes and 17.3 MB of applegpu_g13s code;
at -O1 it takes 31 s and 1.3 MB.
Sampled archive bases from seven games compile to code within a few
percent of -O2. Odyssey's full g13s archive builds in 15.6 minutes
(exit 0), its Cap Kingdom frames match -O2 within 3/255 on scattered
pixels, and GPU time per frame is within run-to-run noise.
Document the mechanism, and correct the Odyssey note: the 758 line never
failed the build; the reporter's exit 1 came from the BAHS link errors
fixed in ff44c59.
Refs #276
Copy byte-aligned NVN streams into aligned submission-owned storage.
Use CPU snapshots only when host visibility is proven safe; otherwise
encode ordered GPU copies before regular, feedback and geometry draws.
Invalidate copies after overlapping writes and pool remaps while keeping
them reusable across unrelated uniform updates. Restore the encoder after
deferred replay when a stream copy split the pass.
Extend t_vbuf to cover all nonzero address residues, copy reuse, queued
copies, deferred updates, fills, shader writes, previous submissions and
pool replacement. Record the general mechanism and Arkham Knight finding.
The per-stream copy limit remains 64 MiB.
Validated on Apple M4 Pro: 84 GPU checks in each of normal, cache-disabled
and encoder-thread modes with Metal validation; full graphics and host
suites; 70 graphics Python tests; 298 tool tests; documentation tests;
iOS compile check (178 files). Full cross-game validation is incomplete:
the harness reported unsupported OpenGL chain checking and its first NVN
shader rebuild did not finish alongside other active builds. No local
Arkham Knight gameplay run.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Add Monolith archive and shader extraction, a build-gated compact memory layout, vertex storage writes under rasterizer discard, primitive ordering and interleaved geometry feedback. Correct native decoder/audio wait accounting and the oe calls used after movies.
Validated opening combat and movement on Mac with real audio, the full Mac/iOS shader build, GPU regressions, and the unsigned IPA archive. iOS runtime, later areas and DLC gameplay remain untested.
Compare revision shader builds and pipeline interfaces across local games,
and run available smoke routes with crash, hang, timing and image checks.
Add the developer loop and share the measuring lock with bench.py.
Keep all local games in static checks and honor --no-smoke. Include shader
bytes and fallback pairs in cache keys, recheck interfaces on reuse, and
reuse smoke baselines only for matching inputs and run settings. Detect
failed prebuilds, missing or intermittently bad screenshots, and Vulkan
build failures even when no MSL was emitted. Keep game-derived files and
writable profile copies inside each game's NSP work folder.
Validation: tool tests and the lint ratchet pass. Full shader rebuilds and
gameplay comparisons remain unverified; the 26 discovered Mac builds have
no smoke routes in this checkout.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Back compatible GPU-only image families at one NVN pool offset with exact-sized placement heaps. Keep descriptor dimensions, initialize fresh storage, register parent-heap residency and retire backing with its textures. Preserve ordinary allocations for single-description storage and exclude unsupported storage classes.
Add synthetic allocation, pixel, indirect-read, depth/stencil and lifetime regressions. Record the LM3 live-allocation census, measured Mac memory reduction and remaining validation limits. Refs #257 and #100.
Register GL and GLB entries in the shared HLE import table so direct imports and eglGetProcAddress reach the same implementations, retaining big-stack wrappers. Add regression and CI coverage for all 156 GL entries and document issue #228.
Display mode at 120 Hz previously waited two display-link ticks per frame
and stayed at 60 fps. Count its acquire waits in the link's own ticks so
120 fps takes one tick. Carry fractional ticks across frames so requests
such as 90 fps on a 120 Hz link wait four ticks per three frames.
Keep game-interval and frame-step pacing in 60 Hz vsync units. Extend
t_link_ticks through the production acquire path, add it to CPU-only CI,
and document the behavior in the frame-rate reference and symptom index.
Validation: 20 link-tick checks, 154 NVN integration checks, pacing and
frame-step tests, Metal snapshot and display-link rescue tests, Mac host
build, iOS source compilation, and documentation checks passed. Physical
display gameplay and iPhone gameplay were not retested.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Below MSL 3.1, SPIRV-Cross emulates an image atomic with a raw device pointer
beside the texture, laid out like a linear image. A block-linear image's Metal
texture is a deswizzled copy, so its atomics went to a scratch buffer nothing
read (Pokemon Legends: Z-A's atomic max into texture 302), and a 3D image's
pointer was indexed with the int3 coordinate itself, which does not compile
(16 shaders of Octopath Traveler 0, 5 of Ghost Trick, 8 of FINAL FANTASY
RESONANCE).
- dmsh_atomic.py: shaders with image atomics (OpImageTexelPointer) are written
at MSL 3.1, so the atomics run on the image's own texture; every other
shader keeps its MSL 2.3 source byte for byte. MSL that still holds the
pointer form fails the build.
- Translator (TranslationFlags.MetalImageAtomics, renamed from
Msl23ImageAtomics): admits 3D images and SUATOM.CAS (Metal's texture
compare-exchange); the IGN coordinate guard is unchanged.
- Host: the IMGA pointer binding, the scratch buffer and the layout admission
are gone. NVN images of R32Uint / R32Sint take MTLTextureUsageShaderAtomic
(img_usage, kept by tex_upgrade: Metal leaves that bit out of the usage
property); any other texture in an atomic slot is replaced by an
atomic-capable write sink with a red line.
- Chain a4e2088c2e69 (with main's vertex-output padding). Tested on
chain 0ce4b5e8aefb, this change before that rebase: Octopath Traveler 0
builds all 10,324 shaders, Z-A all
28,603, Ghost Trick's five 3D-atomic shaders build. Mac runs of both games
under the Metal API validation layer showed no validation error (Octopath
Traveler 0 to frame 14,400, its particle atomics on buffer texture 2049;
Z-A to frame 12,000 through the atomic max on texture 302). Synthetic
atomics and CAS run on buffer, 2D, 2D-array and 3D textures of a Mac GPU
(test_image_atomic_bounds.py, t_atomic_return.m).
GitHub issue #290: at 40b643b Octopath Traveler 0 already built 10,308 of
10,324 shaders (2e655b7 fixed the returning atomics); the 16 left were the 3D
ones, which build now. Not checked: an iPhone, FINAL FANTASY RESONANCE's
library, the picture of Z-A's atomic scene.
Read both shader pair tables, including the generated BAHS fallback, for
padding, build order and shader cache dependencies. Include the optional
profile table in the input stamp and the generated table in step reuse;
regenerate a missing fallback before reusing the shader split.
Add native Metal and cache/order regressions, update the shared chain,
and document full validation on TotK and Odyssey. All 301 affected pairs
compile for g13s and g16s, both full Mac shader/host builds pass, all
57,039 pair interfaces match, and TotK's 24,153-base archive succeeds.
The separate Odyssey Max Unstruct ID compiler limit remains open.
Fixes#283
Refs #276, #298
Co-authored-by: marco <6410852+marcolaux@users.noreply.github.com>
Use only the remaining finite timeout after the fast-event wait instead of
starting a second full wait for feedback. Preserve polling, CPU completion
ordering and the existing infinite-wait stage caps.
Add deterministic regressions against the production wait function across
all three fast-wake sites and the four fast-wake/wait-ledger combinations.
Co-authored-by: smartLanny <51014844+smartLanny@users.noreply.github.com>
One translation chain serves every game, so a translator or host change was checked
only on its author's games; on another game the first sign was a crash at run time
(c3a0880 broke the image atomics of SHIN MEGAMI TENSEI V, Yoshi's Crafted World and
FINAL FANTASY RESONANCE).
- CI: the tool-tests job installs .NET 10, spirv-cross and spirv-tools and points
DMSH_TEST_XLATE at the committed translator, so the translator's synthetic-shader
tests (test_image_atomic_bounds, test_dmsh_geometry) run on every change.
test_dmsh_geometry imports its sibling module when run from the repository root.
- dm/gfx/tools/shader_regressions.py: lists the shaders that translated in the
game's previous b-*/ and fail in the current chain's, from dmsh_build.py's build
reports (not part of the chain). The NSP script runs it after its shader step and
stops on a regression unless ALLOW_SHADER_REGRESSIONS / --allow-shader-regressions;
docs/development.md describes it.
- FINAL FANTASY RESONANCE: the c3a0880 regression is resolved since 2e655b7 (image
atomics with a used result): chain 39e4d9585b1d builds 73,488 of 73,496 shaders
(the 8 3D image atomics as before), shader_regressions.py finds 0 against the
earlier build, and a new game from an empty save plays the opening cutscene on
the Mac with no red line; the two shaders that failed and aborted run their
atomics on their buffer textures.
Checked: every CI test file as the job runs it (DMSH_TEST_XLATE set), the docs tests.
Two general causes of FINAL FANTASY RESONANCE's missing fires, smoke and muzzle
flashes, found and first fixed by b1tdr3amer24 (GitHub issue #168).
- Translator (vendored BindlessElimination.cs, TryTableHandle): a texture read
whose handle phi mixes c2 handle words with values that are not handles
(undefined, or float values the compiler kept in the register on paths where
the predicated read does not run) was removed and its result set to 0:
58,122 reads in FINAL FANTASY RESONANCE, mostly Niagara particle vertex
factories, whose meshes collapsed to a point. The read now takes the c2
handle among the inputs, or reads each candidate and picks by the phi's run-
time value; candidates read elsewhere as another texture shape are left out
(a second sweep, ResourceManager.GetTextureTypes). Only phis that
GenerateBindlessAccess refuses come here, so tex_dyn_slots arrays keep their
path: the other 69,449 shaders' SPIR-V is unchanged. Several candidates are
never used for an image store or atomic. dmsh_xlate's Log now reaches the
notes as "translator: ..." and the build summary counts those apart.
- Host (nvn_mtl.m, tbuf_shadow): a texture buffer view at an offset that is
not 16-byte aligned reads from the boundary below on an M4 Pro (t_tbuf_align:
56 of 56 views wrong; the M6 measurement in the old comment does not hold).
Such views are read through a per-submission aligned copy; aligned views are
untouched. A workaround for the Metal limit, its gaps (GPU writes in the same
submission, CPU_NO_ACCESS pools, image writes) recorded as open.
Translator rebuilt; lock and chain header updated. Checked: all FFR shaders
translated before and after; FFR on the Mac (fires and smoke column render,
steady; with the copy off they flicker); gfx, chain and docs tests.
Co-authored-by: b1tdr3amer24 <240231594+b1tdr3amer24@users.noreply.github.com>
Profile 0100C1F0051B6000 (SDK 4.4.0, Retro Studios' engine, NVN): plays on the Mac from boot
through the menus and the intro cinematic into the first level at 60 fps. General changes:
- svc SetHeapSize: a request beyond the heap region returns ResultOutOfMemory, as Horizon
(the game binary-searches its heap size between 3 and 6 GB)
- shader_split.py: SNVN program records in Retro's RFRM forms (MaterialArchive.arc, the packs'
MTRL resources) and the LZSS-compressed startup material archive in main
- nvn_mtl.m: a texture in a CPU_UNCACHED pool starts with its storage at first use (title logo,
menu panels); transform feedback captures draws that rasterize too
- dmsh_build.py: transform feedback buffers holding several outputs (DK's fur simulation);
chain header regenerated
- hle_audren.c: the renderer event is the kernel event the audio service gives, not the SDK's
local one (the main thread was starved at the first level's load)
- hle_oe.c: oe::GetUserInactivityDetectionTimeExtensionMode
- sead::ControllerBase::updateDerivativeParams_ added a constant 1.0 per frame to held counts, so menus
repeated twice as fast at 60 (0.3 s went 5 entries, 2 at 30). A FRAME_STEP stub at main+0xeb0f88
takes the VFR's ratio (+0x3c); checked in game: 0.2 / 0.3 / 0.4 s move 1 / 2 / 3 entries at 60
(MacNativeGames 866beb9).
- ga_capture: a recapture through lookup archives makes no folder at its compile, so it no longer
lists the folder (it could take another program's folder made meanwhile).
- tex_desc: a multisample texture of a sample count the GPU lacks is not made instead of being made
with fewer samples (shaders could read sample indices past it).
Profile 010057D006492000 (no virtual address memory: the 4 GB heap layout; tex_flush_upload and tex_dirty_upload for the textures Unreal writes from the CPU into uncached pools after creating them) and its note. Brought up on the Mac through the boot, the language notice, the title, the traveler selection and Ophilia's and H'aanit's openings (free play in the cathedral and in S'warkii) at 30 fps; battles, sound and the iPhone not yet tested.
General changes found on it:
- patch_nso.py: a jump table whose index is bounds-checked with a 64-bit cmp (main+0x1676f58), accepted only in the last pass so earlier rewrites are byte-identical.
- profile_guess.py: Unreal's 64 GB alias layout only with virtual address memory (this game sets a 3.08 GB heap instead).
- NVN: nvnTextureBuilderGetSparseTileLayout / nvnTextureGetSparseTileLayout (Unreal sizes a sparse texture's mip tail with it; unimplemented, it asked for 1.75 GB twice); sparse 2D textures without a packaged layout take the block height of their 64 KB tile (the world map's tiles were shuffled); every virtual pool gets an address range of its own until its first mapping decides between the alias (a whole-pool mapping, Mario's pool) and the sparse page table (Unreal's one-texture pools, mapped piece by piece: streamed textures were black and later mappings refused).
- Audio renderer: SdkMutexType is looked up only when the SDK uses it (SDK 3.5.1 takes LockMutex at +0x1f8).
README: the gallery entry (Mac screenshot of S'warkii, IPA ~3.6 GB, all 5,434 shaders built for iOS); AGENTS.md: seventy-one games.
Checked on the rebased tree: the tool tests (15 skipped), the docs tests, a Mac boot into the traveler selection; before the rebase the graphics tests (make run, exit 0).
Profile 010069C01AB82000 (Atlus, Unreal Engine 4.27, SDK 15.3.2, NVN). Opening gameplay
tested on the Mac: new game, classroom, saves and reload in a fresh process, Tokyo map,
Shinagawa Station and its event. iOS and combat untested.
- Memory: the FMallocBinned3 constructor cut (main+0x1f2f63c, gate +0x1f2f644) with the
measured UE4 vtable in hle_uemalloc.c; -SwitchVirtualAddressSpaceSize=2048 and a 3.75 GB
block. profile_guess.py now leaves 128 MB for the initial heap in provisional Unreal
layouts (test_profile_guess.py).
- Shaders (general, chain 86c8282f6fb4): returning IGN image atomics keep the old value with
bounded access (Ryujinx translator, t_atomic_return.m); buffer-image reflection keeps its
Metal texture type (nvn_refl.m, t_refl.m). All 19,746 shaders build for Mac and iOS.
- 60 fps: hle_fpslock.c writes the game's own Project.FrameRateLimit (main+0x6d6d5d0, which
GameUserSettings applies to t.MaxFPS) and rhi.SyncInterval, gated on main 0a4f14a5...
Mac (M4 Pro): 59.2-60.1 fps at the station, frame p95 17.5-20.3 ms; a 1 s walk ends at
the same place at 30 and 60 fps. DM_FPS_TARGET=30 keeps 30.
Docs: the game note, games table, README gallery, memory.md, shaders.md, symptom index;
AGENTS count 68 -> 69. Checked: tool tests (186, 6 skips), docs tests.
- Pipeline archive: the Metal 4 serializer leaves a com.apple.gpuarchiver/gpuarchiver-* folder per flush
(456 folders, 2.5 GB on this Mac). The first capture after a flush takes the one folder it made, the
flush removes it (and the one a recapture-only flush makes), exit removes an unflushed one; iOS clears
earlier runs' folders at start (MacNativeGames 1eff989, 110e6c6).
- tex_desc: levels past what the size holds, arrays of no or more than 2048 layers, sample counts the GPU
lacks no longer abort the process; copies and ReadTexels at a missing level are skipped (f864eb0).
- nvn_hm_get compares 8-byte keys as one word: 5.9 -> 1.5 ns a lookup (7431eb7).
- frame-rate.md: segment size cannot change BotW's 60 fps frame on the Mac (fence wait 0.02 ms).
#203 as submitted stopped the shader split of Tears of the Kingdom with a
TypeError: its 1,202 Cave/.../C.crbin.<hash>/*.quad files are a 4-byte hash
(the one in their folder name), then a zstd frame, so they pass the gate,
and on the length mismatch the error path took len() of None. The PR checked
Wonder and Yoshi only; of the RomFS of 39 games on hand, only those files
have a frame at +4.
- unsized: a frame whose header records a size other than the prefix is not
such a container and is read as it is, without decompressing it
(frame_size). A frame that records none (Kirby's LZ4 frames) must
decompress to exactly the prefix's size: the whole output is compared,
where the PR's max_length cut a longer frame at the prefix and let it pass.
- The dictionary handling is shared with unzs (zs_dict_id, unzs_dict). A
dictionary ZDICTS does not have is still reported, not fatal, as in the PR:
the ID may be any bytes after a magic at +4.
- unityfs.text_assets: an object shorter than the two length fields, or one
whose name padding leaves no room for the script's length, is left out
instead of reading past its end (struct.error, which stopped the split).
Checked:
- Kirby and the Forgotten Land (base and update 1.1.0): 121 shaders before,
27,288 after (27,302 with the update), no [RED] line. 5,121 zstd frames
record their size, equal to the prefix; 2,080 + 354 LZ4 frames record none
and decompress to the prefix's size. The PR's "383 in lyt/" is 378.
- Tears of the Kingdom: the full split exits 1 with #203 alone; with this
commit its package and tables are byte-identical to main's, no [RED] line.
- main's split and this branch's give byte-identical packages and tables on
Monster Train 2, Ace Attorney Investigations Collection, AI: THE SOMNIUM
FILES, nirvanA Initiative, BALL x PIT, Advance Wars 1+2, Brain Memory
Training Academy (Unity 2017.4 to 6000) and Super Mario Bros. Wonder.
- New test_size_prefixed_frames.py; test_unity_text_glslc.py covers short
objects (fails on #238's unityfs.py). All of dm/tools/tests pass.
- Not checked here: FINAL FANTASY CRYSTAL CHRONICLES (#238's game) is not on
hand; Kirby has not been run.
HAL keeps its assets in .cmp containers: a 4-byte little-endian uncompressed
size, then a frame (zstd, or LZ4 in the ubin/ map data). The split
decompressed a zstd file by its magic at offset 0, so a frame that starts at
offset 4 was read raw, and the shaders inside the BFRES archives of those
containers were never seen.
On Kirby and the Forgotten Land this is the difference between 121 shaders
found and 27,288, and between a game that plays and one that shows a black
screen: the game asked for 900 shaders the map did not have, and the log
filled with "[RED] shader ... is not in the shader map".
The split now opens a size-prefixed frame (unzs_bytes for zstd, unlz4_bytes
for LZ4). The size prefix is checked against the decompressed length, so a
file that is not a container is left alone, and a container that does not
open is reported instead of being passed over in silence.
- Found on: Kirby and the Forgotten Land (01004D300C5AE000, in bring-up):
7,201 .cmp files, 5,121 zstd frames with 19,364 BNSH inside (11.2 GB
decompressed) and 2,080 LZ4 frames (0.3 GB, no BNSH). The split now finds
27,288 shaders: 18,476 in gfx/, 8,313 in ptcl/, 383 in lyt/, 67 in the
loose shader files, 54 in main.
- No other game on hand has such a file: Super Mario Bros. Wonder (v0 and
v65536) and Yoshi's Crafted World (v1.0.1) have none.
- The chain version is unchanged: shader_split.py is not one of the chain's
tools (dm/gfx/nvn/dmsh_chain.h needs no rewrite).
- The docs entry is in docs/reference/shaders.md; the symptom index is
regenerated. All 15 test files in dm/tools/tests pass.
Co-Authored-By: Claude Code <noreply@anthropic.com>
FINAL FANTASY CRYSTAL CHRONICLES Remastered Edition (Unity 2018.4.2f1) draws its
maps and effects with programs outside Unity's Shader objects: the TextAsset
shader.switch.vfxsb in Data/resources.assets holds 1,160 whole GLSLC outputs
(1,207 distinct shaders); without them its terrain drew with black holes.
unityfs.text_assets reads TextAsset names and bytes without a type tree (the
tables now parsed by _tables, shared with serialized_objects); unity_text_glslc
passes those holding the GLSLC magic to glslc_shaders. Files without the magic are
not decoded, so other games' tables are unchanged. Documented in
docs/reference/shaders.md; test_unity_text_glslc.py covers it with a synthetic
serialized file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hoist block-linear row and slice addressing out of segment copies, and
index HLE, forced-takeover and missing-import names while retaining
existing duplicate precedence and stub identity.
Add byte-address differential tests and direct collision, duplicate,
growth and reuse checks. Correct the ASTC-tail and loader-serialization
comments and document independent CPU microbenchmarks. Keep TotK
prewarm data in a separate change.
Validation: 5,334 texture checks; symbol-index and NRO lifecycle tests;
AddressSanitizer/UBSan; documentation checks; Mac host build and changed
iOS source compilation. No new gameplay validation.
Co-authored-by: draco-cong <71372970+draco-cong@users.noreply.github.com>
Co-authored-by: laobeibei <laobeibei@users.noreply.github.com>
Keep the proven frame cost and GPU reference for each cap, and probe the
shorter hold when a movie or thermal cap ends. Otherwise a GPU-bound game
can fall straight back to 30 fps without a usable recovery reference.
Save an uninterrupted sustained run when a cap begins, even if no earlier
hitch populated cref. Keep the earlier reference through short runs and
update it after another sustained run. Count cap-release probes and give
them all 30 grace frames: the completed frame at the old cap consumes none.
Add movie and thermal recovery regressions, including first movies,
repeated caps, full grace and failed-probe backoff. Document the mechanism
and link it from both Zelda game notes.
Validated: 64/64 frame-step checks, including ASan/UBSan; six checks fail
against the original PR. Production thermal tests and ThreadSanitizer,
docs tests and diff checks pass; the BotW Mac host builds. Device clock
recovery remains modelled, without a new gameplay observation.
Co-authored-by: RAHUL REX <75500427+RahulRex0@users.noreply.github.com>
present_now cycles three Metal 4 command allocators by ticket, and before resetting one it waited for g_pq_done to reach the ticket that last used it. g_pq_done is not GPU completion: every display-link callback sets it to ticket - 1 before presenting (the previous present was committed, not run), and a present that was not submitted sets it to its own ticket. So a display-link present never waited there: with three presents in flight, or two after a dropped frame, the allocator was reset and its command buffer and UV buffer encoded again under a present the GPU had not run. Each allocator now keeps the newest ticket its own commit feedback reported (g_present_alloc_done), and its reuse waits for that. The display-link callback can now block in present_now until the GPU has run that allocator's previous present; it still moves g_pq_done before it waits, so the game's swapchain acquire is unchanged. General: every game presents through this path, display-link mode is the default.
t_present_alloc holds the Metal 4 queue behind a shared event and calls the display-link callback by hand, each present with a drawable of its own offscreen layer: a dropped frame (tickets 1, 3, then 4 on allocator 1 again) and one frame per tick (5, 6, 7, then 8 on allocator 2 again). Checked on an Apple M6 at 6062518: without the production change 4 of 13 checks fail in 10 of 10 runs, the earlier drawable reads back 0 in all 64 pixels, and every run logs [RED] display command buffer error with kIOGPUCommandBufferCallbackErrorPageFault; with it 13 of 13 pass in 20 of 20 runs under MTL_DEBUG_LAYER=1 with no red line. The whole dm/gfx/tests run list passes, dmhost builds for Mario, docs tests pass. Not checked: a game run that hits it, the iPhone, the not-submitted path. Docs: the frame-rate reference entry (with the page-fault symptom) and the symptom index.
Co-Authored-By: RAHUL REX <75500427+RahulRex0@users.noreply.github.com>
Integrate fast shared-event wakeups, per-segment publication, array-slice
pass merging, and independent compute dispatch hoisting from the PRs.
Serialize CPU completion effects before publishing completed fences and
validate recycled post records. Preserve submission order for early waits.
Guard scheduling against attachment aliases, first-draw and bindless
reads, indirect ranges, stencil writes, and explicit barriers. Add direct
Metal regression coverage and a gated CPU completion race test with TSAN.
Record final BotW and Metroid Dread gameplay validation with optimizations
on and off. BotW retains fewer render passes, but the measured 60 Hz M4
Pro runs do not establish a timing speedup. Keep the original higher-load
and iOS performance claims explicitly unverified.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Add periodic encoder profiles with attachment traffic and formats, alternating A/B measurement phases, bounded Metal trace capture, and gpuprof.py log summaries. Preserve main's offline pipeline-archive integration while adding the capture build setting.
Close periodic samples at frame boundaries through the last committed shared-event fence, so every:1 reports without mixing frames or reusing an unresolved heap. Add 28 CPU checks of production profiling code and include them in CI; document the mechanism and update the symptom index.
Validated with all 31 CI-style Python test files, the profiling regression, a Mac host build, and all 133 iOS source compiles. Profiling completion and heap ownership were checked with synthetic counter/event objects; no new real-GPU or gameplay validation.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Compile eligible shader-pair bases on the Mac with metal-tt, package one
archive per selected iPhone GPU, and use a matching bundle before falling
back to on-device preparation. Carry the profile depth mode through packing
and include the pulling libraries required by the bundled bases.
Integrate PR #226 with four confirmed fixes: validate archive container
bounds before invoking Metal; require at least one actual successful bundle
probe; hash consumed library bytes into offline cache keys; and emit device
remainder lists for constant-only plans instead of aborting packaging.
Keep game measurements in the game note and document the general findings.
Validation on current main: 52 real Metal bundle checks, 57 archive checks,
79 prewarm checks on each pipeline path, 73 Python regressions covering
offline tools, packaging, documentation and chain checks, and compilation
of all 133 iOS source files. No new iPhone gameplay validation. The archive guard validates
container structure, not private Metal payloads.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Hold the host frame step at 30 fps while iOS reports serious or critical
thermal pressure, and restore its configured cap after 30 calm seconds.
Keep the existing movie cap and DM_THERMAL_CAP=0 override.
Sample thermal state independently of movies and configured 30 fps targets
so heating resets recovery and cooling counts during playback. Serialize
the sample, cached cap and recovery timer across acquire/present readers.
The original PR failed five timing checks and ThreadSanitizer detected a
race through concurrent production cap getters.
Add controlled-clock production-path regressions and CI coverage, including
disabled/inactive caps and ThreadSanitizer. Record the shared mechanism
and validation limits in the frame-rate reference and BotW note.
Validated on latest main with the BotW profile: 24 thermal checks normally
and under ThreadSanitizer, 51 frame-step checks, 154 NVN checks, the
present-snapshot suite, 21 tool/docs checks, Mac host build, and iOS compile
check (133 sources). No new device gameplay or thermal recovery run.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Scale the valid presented image with MetalFX's spatial scaler on the shared
Metal 4 path, preserving crop, letterboxing, origin and gamma-encoded colours.
Keep Standard as the default and fall back when scaling is unsupported.
Use the same path for asynchronous whole-window captures and diagnostic
screenshots, retaining resources until GPU completion. Bound scaler recreation
during resizing and report presentation GPU time separately by scaling mode.
Put all launcher settings on one screen with Game, Controls and Performance
sections, inline menus and sliders, and a reset to tested defaults. Extract the
settings model for native persistence tests, validate saved ranges, share HUD
setup, and document the measured costs without claiming an iPhone baseline.
Validated with Metal presentation, NVN and snapshot regressions; tool and iOS
tests; the Mac host; and an unsigned iOS smoke build. No new device gameplay run.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
A pulling stand-in is used only when its base is made or archived, so a cold start's draws waited until the prebuild puts the base of every known pair into the archive (pso_prebuild): in the app after the prewarm, on a utility queue on half the performance cores, flushing every 1,000 bases (DM_PSO_PREBUILD=0: off), or as a tool on the Mac (DM_PSO_PREBUILD=tool: build, write, exit; DM_PSO_PREBUILD_DEPTH=zero, DM_PSO_PREBUILD_THREADS=<n>), once per chain version, depth mode and pair list (pso_archive/prebuilt.tsv). The NSP script runs the tool in a --platform mac build (pso_prebuild.log). Files that hold only bases without a vertex layout are named p<seq>.mtl4a; a base with a vertex layout is never looked up in them (each file a lookup misses in costs about 0.06 ms) and compaction leaves them alone.
The pairs: shader_pairs.tsv paired the stages of one BNSH variation and none of the pairs in agl's BAHS shader archives (.sharcb: the agl / gsys / terrain resource packs), and in Breath of the Wild 109 of the 543 pairs drawn on entering Zora's Domain came from those, so they had no stand-in. shader_split.py now reads a BAHS file's binary array (record size, kind: 0 vertex, 1 fragment, 3 compute, the entry bahs_shaders reads) and program array (record size, name length, stage mask, first binary index), and pairs the n-th vertex binary of a program with its n-th fragment binary (bahs_variations); a file whose arrays do not read back gives none. Those pairs go to a new table, pso_pairs.tsv, which only the prebuild reads (DM_PSO_PAIRS: another path): vs_pad keeps reading shader_pairs.tsv, so no shader library, inputs digest or other table changes. Breath of the Wild: 3,012 more pairs, and all 543 pairs drawn are now known; its shader_map / facts / pairs tables split again byte-identical. A profile made before the table existed still packs: the NSP script compares pso_pairs.tsv only where the profile has one, and otherwise keeps it beside the shader package and hands it to the prebuild. Seven other profiles have BAHS files (Odyssey, Wonder, Mario Kart 8 Deluxe, 3D World, Animal Crossing, Captain Toad, Tears of the Kingdom); their tables are made at their next pack.
Breath of the Wild, Mac (M2 Max), Zora's Domain save, empty Metal shader cache, no pso_user.tsv. Prebuild: 15,767 bases, 910 MB in 16 files, 187 s with 7 workers for the first 12,755 and 19 s for the 3,012 more; the files open in 2.2 s cold, 0.25 s from the file cache, and add about 850 MB of footprint. The frame entering the area: 340 ms with pso_pairs.tsv, 830 ms with shader_pairs.tsv alone, 990 ms through the MTLDevice (then frames of 738, 535 and 496 ms); draws that waited: 50 (20 ms), 102 (650 ms) and 512 (3,325 ms). The picture is the same.
Tests: t_pso_archive (41 checks: also a base without a layout flushed into a p file, and a base with a layout not looked up there); the other graphics tests, the host unit tests and test_nsp_mac / progress / chain give the results they give on main. Not run: an iPhone (the prebuild is off there with the Metal 4 path), the NSP script's pack end to end, the other BAHS profiles' splits. docs/development.md, dm/gfx/README.md and dm/games/README.md describe the prebuild and the table; Breath of the Wild's note (dm/games/01007EF00011E000/README.md, Pipelines on a cold start) has its measurements.
On a miss the draw first takes its pipeline without compiling (its base made or in the archive, then a specialization). Failing that it draws with the pair's pulling pipeline (<key>_bp: no vertex descriptor, the streams read through a per-draw layout block at buffer 16, pull_bind), specialized with the draw's colour state, while its own pipeline is built in the background; a later draw takes that one. Both compute the same positions (t_pull_inv), so a draw that switches does not flicker or z-fight. Only a pulling base already made or archived is used; otherwise it is built in the background for the pair's next miss and the draw waits. Merged passes, readable-tile and diagnostic fragments, and layer-writing or transform feedback vertex shaders have no stand-in. DM_PSO_HYBRID=0 turns it off; it needs the Metal 4 path, so it is on by default on the Mac only.
On main's demand-first pipelines (#75): pso_pend_kick stays the speculative admission (at most two running, at most two per prediction, the oldest finished entry reclaimed when the table is full). A draw with a stand-in queues its own pipeline before the prediction runs (pso_pend_demand): outside the speculative limit, on serial queues at utility QoS, one per two performance cores (pp_bg_queue), so a demanded build never waits behind speculation and speculation never waits behind it. The prediction spends its budget only on draws that will have no stand-in, since only those would wait. A draw that would wait for a build still queued makes the pipeline itself at once (pso_pend_take; Breath of the Wild's cold start waited 0.5-0.9 s for one pipeline behind a burst before). The first miss of a submission loads the shader libraries the rest of it binds in parallel (shader_prefetch_rest): 248 libraries in about 210 ms where one by one on the submit thread took about 1 s, in the hybrid and the waiting path alike.
Diagnostics: a "pipelines up to frame" line every 300 frames (submit-thread pipeline, library and prediction time, archive lookups, specializations, base compiles, hybrid draws, background builds, the draws that waited and why), DM_PSO_WAITLOG=1 (each wait over 20 ms with its pair), DM_PSO_PULL_ALL=1 (every draw with a pulling pipeline draws with it), and the longframe line's count of hybrid draws.
Breath of the Wild, Mac (M2 Max), steady state in Zora's Domain: mean GPU p50 over frames 2,400-5,100 was 8.98 ms with the hybrid, 8.98 ms on the Metal 4 path without it, 9.35 ms through the MTLDevice. Cold-start numbers are with the prebuild of the next change.
Tests: t_pull (346 checks, now also the layout block the host fills), t_pull_inv, t_pso_pending (48) and t_pso_archive (32) pass; the other graphics tests and the host unit tests give the results they give on main. docs/development.md ("Hybrid pipelines") and dm/gfx/README.md describe it.
A side library of every vertex shader with stage inputs, as <key>_bv is, whose main0 / main0_z have no [[stage_in]] and read their inputs from a per-draw layout block at buffer 16: per stream the base address, stride and divisor, per NVN attribute its state word, the base instance from [[base_instance]] (so indirect draws are right too). dm_fetch decodes the NVN format at run time into the register words Maxwell's fetch gives the shader, and the input takes them with as_type, so the body runs unchanged. A pipeline built with it has no vertex descriptor and serves every vertex layout the pair is drawn with; the hybrid pipelines (next change) draw with it while a draw's own pipeline is built.
The words are bit-identical to Metal's vertex-descriptor fetch for every format nvn_mtl.m vfmt knows and to the host's CPU conversion for the integer-read-as-float formats: unorm / snorm through the unpack builtins (a plain division under fast math is a unit in the last place off for up to half the values), R11G11B10F NaNs canonical as from Metal, divisors with Metal's rule base + (instance - base) / divisor. Positions are [[invariant]] and built with preserved invariance, so a pulling pipeline and the vertex-descriptor one compute the same positions. Transform feedback, geometry capture and layer entries get no variant; shaders without stage inputs need none. The run-time decode costs GPU time: +65 % of a vertex-bound draw with 8 attributes on an M2 Max, about 1 ms per million vertices, so it is meant only to bridge the frames until the draw's own pipeline is built.
The iOS app does not use the variants yet (the Metal 4 path is off on iOS by default), so build_ipa.sh leaves the _bp libraries out of the IPA unless IOS_PULL=1 (pack_ipa.py --skip-suffix).
Chain version 8393d85a4eae (on main's 20fe3d62b9f6). Breath of the Wild's full Mac build on chain 97e2f301eb73 (25 min, a full retranslation since the code fingerprint changed): 19,592 shaders ok, 8,888 pulling libraries generated (275 MB), none failed, 43 vertex shaders without stage inputs; every existing library is byte-identical to the previous build (19,592 _b.metallib and _b.metal, 6,254 _bf, 1,070 _bv). Other profiles' shaders were not rebuilt.
Tests: t_pull.m (304 checks: the 76 formats vfmt knows, 73 comparisons with Metal's fetch at aligned, 2-byte and stride-1 placements, 24 with the CPU conversion, stride 0 and divisors 1-3 with a base instance), t_pull_inv.m (2,000 random triangles: a depth prepass with one pipeline and depth-equal with the other passes every covered pixel, both ways round) and test_dmsh_pull.py (the generator: entries, ids it adds or keeps, entries left out, shaders declined; compiles for macOS and iOS); pack_ipa.py's --skip-suffix checked by hand. The other graphics tests and the host unit tests give the results they give on main. docs/reference/rendering.md ("Pulling vertex inputs") describes the variant and its cost; AGENTS.md lists the new chain files.
Decode packaged byte 0 as separate block-height and block-depth nibbles,
including explicit zero dimensions, and size linear compressed textures
by pixel rows. Port the driver-derived rules from PR #201 and preserve
its attribution to digitalhivehub's PR #11 measurements.
Integrate the layout test with current main, correct its BC4 label, add
independently addressed CPU markers and packaged GPU mip coverage, and
record the mechanisms in the reference docs and symptom index.
Validation: 9 driver shapes, 189 CPU texture checks (also ASan/UBSan),
17 GPU deswizzle cases, 154 NVN checks, documentation checks, Mac host
build, and iOS compilation of 133 sources pass. No new gameplay or
iPhone runtime validation.
Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
Carry ResourceAlias visibility on copy encoders containing buffer copies,
inline updates or fills, using the contribution from PR #200. Operations
through sparse virtual mappings also need ordering against surrounding
operations: disjoint guest addresses can name the same physical bytes,
so guest-range dependency detection alone misses their hazards.
Add a regression using distinct no-copy Metal buffers and real sparse
mappings. The original host failed 37 of 101 checks on an M4 Pro and
PR #200 alone left 24 failures; the complete fix passes all 101 checks
with both direct and deferred encoding under Metal validation. Ordinary
independent buffer operations retain range-based batching.
Validation: t_nvn passes 154 checks; the Mac host builds; all 133 iOS
source compile checks and the documentation checks pass. Monster Hunter
Rise gameplay, iPhone execution and sparse-barrier performance were not
retested.
Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
Apple GPUs truncate Depth16Unorm load-clears while rounding fragment depth,
so clearing to 0.5 stores 32767 instead of the fragment's 32768 and fails
an EQUAL comparison. Apply the format-specific correction to both
attachment and whole-texture clears, preserving other depth formats.
Exercise both production command paths against independent GPU fragment
writes for 1,042 float depths, including half-bin boundaries and adjacent
floats. Removing either correction produces 518 mismatches on M4 Pro;
with both corrections enabled, neither path differs. Record the mechanism
and the pending TOTK highlight and iPhone validation in the documentation.
Validated on M4 Pro with Metal validation: t_depth16_clear, t_clear_ds
(43 checks), t_clear_tex (49 checks), and the documentation tests pass.
Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
Switch active-app presents to drawable acquisition after 500 ms of display-link
silence, then retry with exponential backoff capped at 60 seconds. Preserve the
requested pacing mode, synchronize rescue state, and prevent focus timestamp
underflow from bypassing the foreground grace period.
Add CPU regression coverage in CI, a real Metal layer ownership check, and the
mechanism reference with upstream provenance and validation limits.
Validated: 43 CPU rescue checks; 47 checks with real Metal drawable acquisition;
34 present/crop and 10 pacing checks; Wonder Mac host build; all 133 iOS source
compile checks; documentation tests. Windowed gameplay recovery and iPhone
validation remain outstanding.
Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
Take upload stamps before protection, initialize cached checks from those
stamps, and wait out pages another thread is unprotecting before retrying.
Publish page generations before advancing the clock again in both the fault
handler and batched host pre-write path. A shared stamp_page helper closes
the remaining batched publication race in the original PR.
Add deterministic publication regressions for both paths and document the
mechanism and the limits of the game-level evidence. The regressions fail
on main, and the batched case also fails with the original PR applied.
Validation: full graphics suite, Wonder Mac host build, all 133 iOS source
compile checks, and documentation tests pass. No fresh gameplay or iPhone
runtime validation.
Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
Drop unused normal-pass colour-target blend and mask state from pipeline keys.
Normalize historical prewarm rows and deduplicate their parsed keys before
compilation, preserving merged passes' logical target state.
Include production-path GPU regression coverage and the reference/game notes.
Validated on the latest main: Wonder host build, key serialization tests,
48 pending-pipeline checks, 48 archive checks, 158 prewarm GPU checks across
MTLDevice and Metal 4, and documentation checks. The regression fails on the
original base and detects wrong pixels if the merged-pass guard is removed.
Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
Erick Sanchez