At -O2, CorrelatedValuePropagation and SimplifyCFG fold a translated
shader's repeated tests of one value into switches whose cases jump past
their own join. Apple's GPU backend then runs out of control-flow IDs
("Max Unstruct ID 758 ... exceeds supported limit", a note: metal-tt
still exits 0) and lowers the function a far slower way. Odyssey's
alRenderMaterialUber took 24.6 minutes and 17.3 MB of applegpu_g13s code;
at -O1 it takes 31 s and 1.3 MB.
Sampled archive bases from seven games compile to code within a few
percent of -O2. Odyssey's full g13s archive builds in 15.6 minutes
(exit 0), its Cap Kingdom frames match -O2 within 3/255 on scattered
pixels, and GPU time per frame is within run-to-run noise.
Document the mechanism, and correct the Odyssey note: the 758 line never
failed the build; the reporter's exit 1 came from the BAHS link errors
fixed in ff44c59.
Refs #276
Add Monolith archive and shader extraction, a build-gated compact memory layout, vertex storage writes under rasterizer discard, primitive ordering and interleaved geometry feedback. Correct native decoder/audio wait accounting and the oe calls used after movies.
Validated opening combat and movement on Mac with real audio, the full Mac/iOS shader build, GPU regressions, and the unsigned IPA archive. iOS runtime, later areas and DLC gameplay remain untested.
Below MSL 3.1, SPIRV-Cross emulates an image atomic with a raw device pointer
beside the texture, laid out like a linear image. A block-linear image's Metal
texture is a deswizzled copy, so its atomics went to a scratch buffer nothing
read (Pokemon Legends: Z-A's atomic max into texture 302), and a 3D image's
pointer was indexed with the int3 coordinate itself, which does not compile
(16 shaders of Octopath Traveler 0, 5 of Ghost Trick, 8 of FINAL FANTASY
RESONANCE).
- dmsh_atomic.py: shaders with image atomics (OpImageTexelPointer) are written
at MSL 3.1, so the atomics run on the image's own texture; every other
shader keeps its MSL 2.3 source byte for byte. MSL that still holds the
pointer form fails the build.
- Translator (TranslationFlags.MetalImageAtomics, renamed from
Msl23ImageAtomics): admits 3D images and SUATOM.CAS (Metal's texture
compare-exchange); the IGN coordinate guard is unchanged.
- Host: the IMGA pointer binding, the scratch buffer and the layout admission
are gone. NVN images of R32Uint / R32Sint take MTLTextureUsageShaderAtomic
(img_usage, kept by tex_upgrade: Metal leaves that bit out of the usage
property); any other texture in an atomic slot is replaced by an
atomic-capable write sink with a red line.
- Chain a4e2088c2e69 (with main's vertex-output padding). Tested on
chain 0ce4b5e8aefb, this change before that rebase: Octopath Traveler 0
builds all 10,324 shaders, Z-A all
28,603, Ghost Trick's five 3D-atomic shaders build. Mac runs of both games
under the Metal API validation layer showed no validation error (Octopath
Traveler 0 to frame 14,400, its particle atomics on buffer texture 2049;
Z-A to frame 12,000 through the atomic max on texture 302). Synthetic
atomics and CAS run on buffer, 2D, 2D-array and 3D textures of a Mac GPU
(test_image_atomic_bounds.py, t_atomic_return.m).
GitHub issue #290: at 40b643b Octopath Traveler 0 already built 10,308 of
10,324 shaders (2e655b7 fixed the returning atomics); the 16 left were the 3D
ones, which build now. Not checked: an iPhone, FINAL FANTASY RESONANCE's
library, the picture of Z-A's atomic scene.
HAL Laboratory's engine on NintendoWare, SDK 11.4.0, NVN. Plays on the Mac at 30 fps, as on
the Switch: online and difficulty prompts, the 1080p opening movie, Point of Arrival with its
guide movies, the city reveal, the Sword ability and fights, title and Continue; a 15-minute
soak with no crash. No game code is cut; the profile is the memory layout and the shader groups.
All 27,297 shaders build for Mac and iOS; the IPA (6.4 GB) is untested on a device.
- Threads (general): a guest thread waiting for a host lock that HLE code holds while it runs
guest code is blocked, not runnable (dm_host_lock, a native HLE wait). The movie decoder held
its decode lock across SignalEvent; the signal's same-core handoff waited for the movie input
thread, which waited for that lock, and the game froze at its first guide movie. t_preempt
has the case.
Docs: the game note, games table, README gallery, threads.md, shaders.md, symptom index;
AGENTS count 73 -> 74. Checked: host tests, preempt, tool tests (253, 15 skips), docs tests;
a Mac run on chain 5cddc47253ab through the first guide movie.
Read both shader pair tables, including the generated BAHS fallback, for
padding, build order and shader cache dependencies. Include the optional
profile table in the input stamp and the generated table in step reuse;
regenerate a missing fallback before reusing the shader split.
Add native Metal and cache/order regressions, update the shared chain,
and document full validation on TotK and Odyssey. All 301 affected pairs
compile for g13s and g16s, both full Mac shader/host builds pass, all
57,039 pair interfaces match, and TotK's 24,153-base archive succeeds.
The separate Odyssey Max Unstruct ID compiler limit remains open.
Fixes#283
Refs #276, #298
Co-authored-by: marco <6410852+marcolaux@users.noreply.github.com>
Two general causes of FINAL FANTASY RESONANCE's missing fires, smoke and muzzle
flashes, found and first fixed by b1tdr3amer24 (GitHub issue #168).
- Translator (vendored BindlessElimination.cs, TryTableHandle): a texture read
whose handle phi mixes c2 handle words with values that are not handles
(undefined, or float values the compiler kept in the register on paths where
the predicated read does not run) was removed and its result set to 0:
58,122 reads in FINAL FANTASY RESONANCE, mostly Niagara particle vertex
factories, whose meshes collapsed to a point. The read now takes the c2
handle among the inputs, or reads each candidate and picks by the phi's run-
time value; candidates read elsewhere as another texture shape are left out
(a second sweep, ResourceManager.GetTextureTypes). Only phis that
GenerateBindlessAccess refuses come here, so tex_dyn_slots arrays keep their
path: the other 69,449 shaders' SPIR-V is unchanged. Several candidates are
never used for an image store or atomic. dmsh_xlate's Log now reaches the
notes as "translator: ..." and the build summary counts those apart.
- Host (nvn_mtl.m, tbuf_shadow): a texture buffer view at an offset that is
not 16-byte aligned reads from the boundary below on an M4 Pro (t_tbuf_align:
56 of 56 views wrong; the M6 measurement in the old comment does not hold).
Such views are read through a per-submission aligned copy; aligned views are
untouched. A workaround for the Metal limit, its gaps (GPU writes in the same
submission, CPU_NO_ACCESS pools, image writes) recorded as open.
Translator rebuilt; lock and chain header updated. Checked: all FFR shaders
translated before and after; FFR on the Mac (fires and smoke column render,
steady; with the copy off they flicker); gfx, chain and docs tests.
Co-authored-by: b1tdr3amer24 <240231594+b1tdr3amer24@users.noreply.github.com>
Profile 0100C1F0051B6000 (SDK 4.4.0, Retro Studios' engine, NVN): plays on the Mac from boot
through the menus and the intro cinematic into the first level at 60 fps. General changes:
- svc SetHeapSize: a request beyond the heap region returns ResultOutOfMemory, as Horizon
(the game binary-searches its heap size between 3 and 6 GB)
- shader_split.py: SNVN program records in Retro's RFRM forms (MaterialArchive.arc, the packs'
MTRL resources) and the LZSS-compressed startup material archive in main
- nvn_mtl.m: a texture in a CPU_UNCACHED pool starts with its storage at first use (title logo,
menu panels); transform feedback captures draws that rasterize too
- dmsh_build.py: transform feedback buffers holding several outputs (DK's fur simulation);
chain header regenerated
- hle_audren.c: the renderer event is the kernel event the audio service gives, not the SDK's
local one (the main thread was starved at the first level's load)
- hle_oe.c: oe::GetUserInactivityDetectionTimeExtensionMode
Profile 010069C01AB82000 (Atlus, Unreal Engine 4.27, SDK 15.3.2, NVN). Opening gameplay
tested on the Mac: new game, classroom, saves and reload in a fresh process, Tokyo map,
Shinagawa Station and its event. iOS and combat untested.
- Memory: the FMallocBinned3 constructor cut (main+0x1f2f63c, gate +0x1f2f644) with the
measured UE4 vtable in hle_uemalloc.c; -SwitchVirtualAddressSpaceSize=2048 and a 3.75 GB
block. profile_guess.py now leaves 128 MB for the initial heap in provisional Unreal
layouts (test_profile_guess.py).
- Shaders (general, chain 86c8282f6fb4): returning IGN image atomics keep the old value with
bounded access (Ryujinx translator, t_atomic_return.m); buffer-image reflection keeps its
Metal texture type (nvn_refl.m, t_refl.m). All 19,746 shaders build for Mac and iOS.
- 60 fps: hle_fpslock.c writes the game's own Project.FrameRateLimit (main+0x6d6d5d0, which
GameUserSettings applies to t.MaxFPS) and rhi.SyncInterval, gated on main 0a4f14a5...
Mac (M4 Pro): 59.2-60.1 fps at the station, frame p95 17.5-20.3 ms; a 1 s walk ends at
the same place at 30 and 60 fps. DM_FPS_TARGET=30 keeps 30.
Docs: the game note, games table, README gallery, memory.md, shaders.md, symptom index;
AGENTS count 68 -> 69. Checked: tool tests (186, 6 skips), docs tests.
- Pipeline archive: the Metal 4 serializer leaves a com.apple.gpuarchiver/gpuarchiver-* folder per flush
(456 folders, 2.5 GB on this Mac). The first capture after a flush takes the one folder it made, the
flush removes it (and the one a recapture-only flush makes), exit removes an unflushed one; iOS clears
earlier runs' folders at start (MacNativeGames 1eff989, 110e6c6).
- tex_desc: levels past what the size holds, arrays of no or more than 2048 layers, sample counts the GPU
lacks no longer abort the process; copies and ReadTexels at a missing level are skipped (f864eb0).
- nvn_hm_get compares 8-byte keys as one word: 5.9 -> 1.5 ns a lookup (7431eb7).
- frame-rate.md: segment size cannot change BotW's 60 fps frame on the Mac (fence wait 0.02 ms).
#203 as submitted stopped the shader split of Tears of the Kingdom with a
TypeError: its 1,202 Cave/.../C.crbin.<hash>/*.quad files are a 4-byte hash
(the one in their folder name), then a zstd frame, so they pass the gate,
and on the length mismatch the error path took len() of None. The PR checked
Wonder and Yoshi only; of the RomFS of 39 games on hand, only those files
have a frame at +4.
- unsized: a frame whose header records a size other than the prefix is not
such a container and is read as it is, without decompressing it
(frame_size). A frame that records none (Kirby's LZ4 frames) must
decompress to exactly the prefix's size: the whole output is compared,
where the PR's max_length cut a longer frame at the prefix and let it pass.
- The dictionary handling is shared with unzs (zs_dict_id, unzs_dict). A
dictionary ZDICTS does not have is still reported, not fatal, as in the PR:
the ID may be any bytes after a magic at +4.
- unityfs.text_assets: an object shorter than the two length fields, or one
whose name padding leaves no room for the script's length, is left out
instead of reading past its end (struct.error, which stopped the split).
Checked:
- Kirby and the Forgotten Land (base and update 1.1.0): 121 shaders before,
27,288 after (27,302 with the update), no [RED] line. 5,121 zstd frames
record their size, equal to the prefix; 2,080 + 354 LZ4 frames record none
and decompress to the prefix's size. The PR's "383 in lyt/" is 378.
- Tears of the Kingdom: the full split exits 1 with #203 alone; with this
commit its package and tables are byte-identical to main's, no [RED] line.
- main's split and this branch's give byte-identical packages and tables on
Monster Train 2, Ace Attorney Investigations Collection, AI: THE SOMNIUM
FILES, nirvanA Initiative, BALL x PIT, Advance Wars 1+2, Brain Memory
Training Academy (Unity 2017.4 to 6000) and Super Mario Bros. Wonder.
- New test_size_prefixed_frames.py; test_unity_text_glslc.py covers short
objects (fails on #238's unityfs.py). All of dm/tools/tests pass.
- Not checked here: FINAL FANTASY CRYSTAL CHRONICLES (#238's game) is not on
hand; Kirby has not been run.
HAL keeps its assets in .cmp containers: a 4-byte little-endian uncompressed
size, then a frame (zstd, or LZ4 in the ubin/ map data). The split
decompressed a zstd file by its magic at offset 0, so a frame that starts at
offset 4 was read raw, and the shaders inside the BFRES archives of those
containers were never seen.
On Kirby and the Forgotten Land this is the difference between 121 shaders
found and 27,288, and between a game that plays and one that shows a black
screen: the game asked for 900 shaders the map did not have, and the log
filled with "[RED] shader ... is not in the shader map".
The split now opens a size-prefixed frame (unzs_bytes for zstd, unlz4_bytes
for LZ4). The size prefix is checked against the decompressed length, so a
file that is not a container is left alone, and a container that does not
open is reported instead of being passed over in silence.
- Found on: Kirby and the Forgotten Land (01004D300C5AE000, in bring-up):
7,201 .cmp files, 5,121 zstd frames with 19,364 BNSH inside (11.2 GB
decompressed) and 2,080 LZ4 frames (0.3 GB, no BNSH). The split now finds
27,288 shaders: 18,476 in gfx/, 8,313 in ptcl/, 383 in lyt/, 67 in the
loose shader files, 54 in main.
- No other game on hand has such a file: Super Mario Bros. Wonder (v0 and
v65536) and Yoshi's Crafted World (v1.0.1) have none.
- The chain version is unchanged: shader_split.py is not one of the chain's
tools (dm/gfx/nvn/dmsh_chain.h needs no rewrite).
- The docs entry is in docs/reference/shaders.md; the symptom index is
regenerated. All 15 test files in dm/tools/tests pass.
Co-Authored-By: Claude Code <noreply@anthropic.com>
FINAL FANTASY CRYSTAL CHRONICLES Remastered Edition (Unity 2018.4.2f1) draws its
maps and effects with programs outside Unity's Shader objects: the TextAsset
shader.switch.vfxsb in Data/resources.assets holds 1,160 whole GLSLC outputs
(1,207 distinct shaders); without them its terrain drew with black holes.
unityfs.text_assets reads TextAsset names and bytes without a type tree (the
tables now parsed by _tables, shared with serialized_objects); unity_text_glslc
passes those holding the GLSLC magic to glslc_shaders. Files without the magic are
not decoded, so other games' tables are unchanged. Documented in
docs/reference/shaders.md; test_unity_text_glslc.py covers it with a synthetic
serialized file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Mac app packer left out shader_pairs.tsv and pso_pairs.tsv, which the
host's pipeline prebuild reads beside the shader map and build_ipa.sh already
copies into the IPA. Every NVN game's Mac app therefore built no pulling
bases, had no hybrid stand-ins, and compiled first-seen pipelines on the
submit thread. Copy both tables, and the shader split's pso_pairs.tsv for a
profile without one.
An NVN game's Mac app also carries its pipeline archive made offline for this
Mac's GPU (pso_offline.py --platform macos), handed to the host as
DM_PSO_BUNDLE as on iOS; --gpu none leaves it out. Integrated with three
fixes: the archive is validated and counted in the free-space check before
anything is copied; a plan whose pairs all need function constants packs the
app without an archive instead of aborting; and a PSO_GPUS value that does
not name this Mac's GPU is reported.
Validation: test_nsp_mac_app 22 (pair tables, split copy, bad archive, free
space, GPU note, no-archive plan; the table and space tests fail without the
fixes), the tool suite, iOS packaging tests and test_docs. A Breath of the
Wild Mac app on an M4 Pro (applegpu_g16s, 629.4 MB archive): 16 of 16 probed
bases found, no pair base built, no red line; the run stayed in the title.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Co-authored-by: blindlight86 <25404088+blindlight86@users.noreply.github.com>
Add periodic encoder profiles with attachment traffic and formats, alternating A/B measurement phases, bounded Metal trace capture, and gpuprof.py log summaries. Preserve main's offline pipeline-archive integration while adding the capture build setting.
Close periodic samples at frame boundaries through the last committed shared-event fence, so every:1 reports without mixing frames or reusing an unresolved heap. Add 28 CPU checks of production profiling code and include them in CI; document the mechanism and update the symptom index.
Validated with all 31 CI-style Python test files, the profiling regression, a Mac host build, and all 133 iOS source compiles. Profiling completion and heap ownership were checked with synthetic counter/event objects; no new real-GPU or gameplay validation.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Compile eligible shader-pair bases on the Mac with metal-tt, package one
archive per selected iPhone GPU, and use a matching bundle before falling
back to on-device preparation. Carry the profile depth mode through packing
and include the pulling libraries required by the bundled bases.
Integrate PR #226 with four confirmed fixes: validate archive container
bounds before invoking Metal; require at least one actual successful bundle
probe; hash consumed library bytes into offline cache keys; and emit device
remainder lists for constant-only plans instead of aborting packaging.
Keep game measurements in the game note and document the general findings.
Validation on current main: 52 real Metal bundle checks, 57 archive checks,
79 prewarm checks on each pipeline path, 73 Python regressions covering
offline tools, packaging, documentation and chain checks, and compilation
of all 133 iOS source files. No new iPhone gameplay validation. The archive guard validates
container structure, not private Metal payloads.
Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
Drop unused normal-pass colour-target blend and mask state from pipeline keys.
Normalize historical prewarm rows and deduplicate their parsed keys before
compilation, preserving merged passes' logical target state.
Include production-path GPU regression coverage and the reference/game notes.
Validated on the latest main: Wonder host build, key serialization tests,
48 pending-pipeline checks, 48 archive checks, 158 prewarm GPU checks across
MTLDevice and Metal 4, and documentation checks. The regression fails on the
original base and detects wrong pixels if the merged-pass guard is removed.
Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
Exercise historical manifest rows through the production warmer, shader loader,
pipeline builders and draw-time cache. Check normal and merged passes on both
MTLDevice and Metal 4, with GPU pixels compared against the original descriptors.
The regression fails on the PR base and when the merged-pass guard is removed.
Document the shared normalization rule and keep game measurements in their notes.
Co-authored-by: Aleksandr Svidchenkov <a.svidchenkov@gmail.com>
- profile 01007BB017812000 (1.0.3, SDK 13.3.0): plays on the Mac at 60 fps
- provisional profiles add the subsdks the NROs import (profile_guess nro_modules)
- an NRO importing nvnBootstrapLoader makes an NVN game; NROs calling glslcCompile get the run-time GLSLC capture
- shader_split reads zip archive members (Portal's nxcontent/nvsb GLSLC outputs)
- ro::LookupModuleSymbol for a name a loaded module lacks is not red (Horizon fails it too)
- AGENTS.md: a Mac gameplay test is enough to count a game as playable (65)
Apply the generalized image-atomic bounds and backing checks from PR #178.
Also reject unsupported descriptor formats before integer fallback and fail
unresolved bindless atomics instead of silently deleting them. Rebuild the
translator and record chain 467efdfd0b62, with regression tests and findings.
Validation: compiler/helper and sanitizer tests, 96 Metal compile/link checks,
ten GPU probes, all 28,603 Z-A shaders for Mac and iPhone, and a 240-second
Mac station-area run reaching 6,987 frames without a freeze or GPU fault.
Reported iPhone cases and the station-to-world transition remain unverified.
Co-authored-by: b4f8abbb24dbf56a7d0280f296e16eeb <51352400+b4f8abbb24dbf56a7d0280f296e16eeb@users.noreply.github.com>
Support PKZ v1 archives and advertise the translator's GPU-code range independently of SDK version. Register the 1.0.1 profile, shader tables, regression checks and Mac bring-up findings. Retain 30 fps; document the unshipped 60 fps experiment and outstanding iPhone validation.
seedds