main
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 seedds
|
f8642c480f |
Run image atomics as native Metal texture atomics, 3D images and CAS included
Below MSL 3.1, SPIRV-Cross emulates an image atomic with a raw device pointer beside the texture, laid out like a linear image. A block-linear image's Metal texture is a deswizzled copy, so its atomics went to a scratch buffer nothing read (Pokemon Legends: Z-A's atomic max into texture 302), and a 3D image's pointer was indexed with the int3 coordinate itself, which does not compile (16 shaders of Octopath Traveler 0, 5 of Ghost Trick, 8 of FINAL FANTASY RESONANCE). - dmsh_atomic.py: shaders with image atomics (OpImageTexelPointer) are written at MSL 3.1, so the atomics run on the image's own texture; every other shader keeps its MSL 2.3 source byte for byte. MSL that still holds the pointer form fails the build. - Translator (TranslationFlags.MetalImageAtomics, renamed from Msl23ImageAtomics): admits 3D images and SUATOM.CAS (Metal's texture compare-exchange); the IGN coordinate guard is unchanged. - Host: the IMGA pointer binding, the scratch buffer and the layout admission are gone. NVN images of R32Uint / R32Sint take MTLTextureUsageShaderAtomic (img_usage, kept by tex_upgrade: Metal leaves that bit out of the usage property); any other texture in an atomic slot is replaced by an atomic-capable write sink with a red line. - Chain a4e2088c2e69 (with main's vertex-output padding). Tested on chain 0ce4b5e8aefb, this change before that rebase: Octopath Traveler 0 builds all 10,324 shaders, Z-A all 28,603, Ghost Trick's five 3D-atomic shaders build. Mac runs of both games under the Metal API validation layer showed no validation error (Octopath Traveler 0 to frame 14,400, its particle atomics on buffer texture 2049; Z-A to frame 12,000 through the atomic max on texture 302). Synthetic atomics and CAS run on buffer, 2D, 2D-array and 3D textures of a Mac GPU (test_image_atomic_bounds.py, t_atomic_return.m). GitHub issue #290: at |
||
|
seedds
|
b117c91317 |
Check translation chain changes on games their author does not own (#272)
One translation chain serves every game, so a translator or host change was checked only on its author's games; on another game the first sign was a crash at run time ( |
||
|
seeddsandb1tdr3amer24
|
99d0aae38f |
Keep bindless reads through mixed handle phis; read misaligned buffer textures through an aligned copy (#168)
Two general causes of FINAL FANTASY RESONANCE's missing fires, smoke and muzzle flashes, found and first fixed by b1tdr3amer24 (GitHub issue #168). - Translator (vendored BindlessElimination.cs, TryTableHandle): a texture read whose handle phi mixes c2 handle words with values that are not handles (undefined, or float values the compiler kept in the register on paths where the predicated read does not run) was removed and its result set to 0: 58,122 reads in FINAL FANTASY RESONANCE, mostly Niagara particle vertex factories, whose meshes collapsed to a point. The read now takes the c2 handle among the inputs, or reads each candidate and picks by the phi's run- time value; candidates read elsewhere as another texture shape are left out (a second sweep, ResourceManager.GetTextureTypes). Only phis that GenerateBindlessAccess refuses come here, so tex_dyn_slots arrays keep their path: the other 69,449 shaders' SPIR-V is unchanged. Several candidates are never used for an image store or atomic. dmsh_xlate's Log now reaches the notes as "translator: ..." and the build summary counts those apart. - Host (nvn_mtl.m, tbuf_shadow): a texture buffer view at an offset that is not 16-byte aligned reads from the boundary below on an M4 Pro (t_tbuf_align: 56 of 56 views wrong; the M6 measurement in the old comment does not hold). Such views are read through a per-submission aligned copy; aligned views are untouched. A workaround for the Metal limit, its gaps (GPU writes in the same submission, CPU_NO_ACCESS pools, image writes) recorded as open. Translator rebuilt; lock and chain header updated. Checked: all FFR shaders translated before and after; FFR on the Mac (fires and smoke column render, steady; with the copy off they flicker); gfx, chain and docs tests. Co-authored-by: b1tdr3amer24 <240231594+b1tdr3amer24@users.noreply.github.com> |
||
|
seedds
|
3a993fa3fe |
Subsystem references and a symptom index: the failure catalogue of docs/agents/03-bringup.md and the per-title material of docs/agents/04-graphics.md (section 1's container list, section 6 and its "Seen in <game>" subsections) move into docs/reference/, one heading per mechanism: rewriter.md, memory.md, threads.md, services.md, movies.md, audio.md, frame-rate.md, nvn-objects.md, rendering.md (with the Vulkan and OpenGL layers), shaders.md and game-state.md, indexed by docs/reference/README.md. Each entry gives the symptom, the cause, the fix (file and function), the games it was found on (linked to their notes in dm/games/<id>/README.md) and any check, caveat or later change; one mechanism met on several games is one entry. docs/agents/symptoms.md lists every entry's symptom (an assertion, a [RED] line, an abort or crash site, a visible result) with a link to it, 350 lines, generated from the Symptom bullets. 03 keeps the loop and reading the log, and its Failure catalogue section now points to the references, the index and where a new finding goes; 04 keeps sections 1-5 and 7 (the container list is shaders.md's Containers by engine), and section 6 says where each kind of finding is. Facts in the old sections about one game only were checked against that game's note, where they already were. Statements the code no longer matches are marked as history beside the current value (movie output slots: MD_NOUT 64, MD_NOUT_AAC 32, MD_NOUT_VIDEO 16; APP_POOL is the profile key app_pool; Engage's layout is 5 / 4.375 GB; inline uniform updates are deferred by default; scaled copies are drawn), and Tears of the Kingdom joins the node-area entry's build list (issue #70). The game notes, docs/agents/README.md, 02 and 05 link to the exact entries, and so do the comments in nvn_mtl.m (nvnTextureCompare) and hle_audio.c (the RemoteIO buffer) and the provisional-profile note in profile_guess.py. 03 goes from 29,600 to 470 words, 04 from 34,400 to 3,500. Checked: every backticked span, module address, issue / PR number and 16-digit hash of AGENTS.md, development.md, dm/games/README.md and docs/agents/01-05 at 6a06877 (4,600 items) still appears in the repository's Markdown (the same five file patterns as before appear as the full names they stood for), every relative link and heading anchor in the repository's Markdown resolves, and the tool tests pass (the four that need capstone or lz4 in the T7 venv)
|
||
|
seedds
|
846c54c63c |
Per-game notes beside each profile: dm/games/<id>/README.md for all 58 games, and the games table in dm/games/README.md. Each note says what the profile needs and why (a table of its game.tsv keys, patches.py cuts and frame-step stubs, sdk_patches.py cuts for its sdk build, manual shader rows), the game's own findings by topic (frame rate, memory, rendering, audio, movies, services, files and shaders, input, Mac routes), each marked current or resolved, a "Found on this game" list of the general host changes first needed there (one line each, linked to where they are explained), and what is still open, including stale profile comments and figures the sources disagree on. The notes gather what was spread over AGENTS.md, docs/development.md (Game notes and Power), docs/agents/01-05 and the profiles' own comments. The Game notes section of docs/development.md (523 lines, 53 games) is now a pointer to the table; development.md goes from 33,300 to 8,600 words. README.md, AGENTS.md and the Breath of the Wild link in 04-graphics.md point to the notes. A README.md in a profile folder changes no shader cache key (dmsh_chain.inputs hashes only the named tables and keys) and no profile listing (game.py lists folders with a game.tsv). Checked: every backticked span, module address, issue / PR number and 16-digit hash of AGENTS.md, development.md, dm/games/README.md and docs/agents/01-05 at 6a06877 (4,600 items) still appears in the repository's Markdown (five file patterns now appear as the full names they stood for), and every relative link and heading anchor in the repository's Markdown resolves
|
||
|
seedds
|
86d9ba1239 | One translation chain version for every game: DMSH_CHAIN in dm/gfx/nvn/dmsh_chain.h replaces the 47 dm/games/<id>/dmsh_chain.h, which every chain change rewrote and some were left behind (GitHub issue #78). The version hashed each profile's shader tables and nvn_* / shader_* keys too, once the only thing that kept the host off a library built from older tables; each shader's cache key (dmsh_build.py shader_key) has covered them since, so the version now covers only the chain's tools (dmsh_chain.py, dmsh_build.py, dmsh_fetch.py, dmsh_vin.py, dmsh_geometry.py, dmsh_facts.py, the translator's source, ios_metal.py, the lock) and the options, and needs no DM_GAME. What the per-game part still did is done by the library itself: a full dmsh_build.py run stamps b-<V>/inputs.tsv (Mac and iOS) with the game and the digest of the profile's inputs (dmsh_chain.py inputs / --inputs: the four shader tables, bf_want.tsv, which the variants use but the version never counted, and the keys), starts from no shader outputs (a partial one from none of the keys it builds), so a key that left the map, a shader that no longer builds or a variant no longer made keeps no old library, refuses a directory stamped for another game, and writes the profile's c1_full.tsv and cb_extent.tsv from its Mac outputs; a partial build keeps the stamp only while it names the current inputs. c1_guard.py / cb_extent.py head their table with the library's chain version and stamp (table_head), and the host takes a table only beside that stamp, logs the stamp and reports one of another game; build_ipa.sh reads the one header and stops when the iOS libraries' stamp is not this profile's current inputs or c1_full.tsv names another. NSP script: chain_header writes the one header when stale, before any host is built; provisional and derived profiles no longer write one; chain_tables makes a table again when it does not name the reused library's stamp (False without one). Makefiles take the header beside nvn_mtl.m; Xcode projects prepared earlier point at a profile header that is gone and are told to prepare again. Tests: test_nsp_chain.py (the committed header is current, stale and missing ones are written, tables against the stamp). Checked on The Great Ace Attorney Chronicles: a full build gave 460 of 460 shaders and both tables (0 and 409 entries, as before), the Mac host booted with the stamp and both tables past frame 900; one added facts row rebuilt one shader under the same chain version with a new digest and new tables, and the host refused the old tables beside the new stamp; a full build removed planted outputs, another game's build into that directory stopped, a partial build after an input change dropped the stamp; build_ipa.sh stopped on a stale stamp, a missing one and a stale or old-form c1_full.tsv; three NSP packs built (no table lines: dmsh_build made them), reused everything, and remade deleted tables from the reused build. Graphics unit tests give the same results as before (on this Mac t_refl needs ~/dm-work's libraries and several GPU tests fail on the previous commit too). AGENTS.md's chain procedure (header once after a tool change; a full shader build makes the tables), dm/games/README.md, dm/gfx/README.md and docs/development.md say so | ||
|
seedds
|
f4f0a85d6e | patch_nso.py split into the general rewriter and per-build patch files (GitHub issue #63): every game's bring-up edited the one file, so PRs for unrelated games conflicted. dm/tools/patch_nso.py (1,711 → about 1,100 lines) keeps the general passes and the driver; the instruction encoders move unchanged to dm/tools/nso_asm.py; a game's own patches (CUTS, OBS and the frame-step stubs, now FRAME_STEP = {build ID: stub(ctx)}) move verbatim with their comments to dm/games/<id>/patches.py (16 profiles: Tears of the Kingdom, Breath of the Wild, Age of Calamity and Paper Mario: The Thousand-Year Door with their stubs), and the cuts of Nintendo's sdk module builds, 6 of the 15 shipped by 2 to 4 games, to dm/tools/sdk_patches.py. The rewriter loads every patch file, whatever the game, and applies those of a module's build ID at the points it did before (cuts before the scan, gates after the rewrite, stubs after the gates); a build ID claimed twice stops it, and its command line needs no DM_GAME. build_src.sha256 lists the rewriter's files and every patch file (the same for every module). NSP script: the rewrite step's reuse key counts nso_asm.py and only the patch files that name one of the game's builds (patch_files), profile_files leaves patches.py out (only the rewrite step reads it), and a derived profile gets none. dm/tools/tests/test_patches.py: no build claimed twice, the entries' forms, a game's file names only its own profile's non-sdk builds, sdk_patches.py only sdk builds some profile ships, and the reuse key's patch files. Checked byte-identical: every extracted module on the game drive (131 builds) rewritten with the previous and the new rewriter gives the same outputs but build_src.sha256, 28 of the 30 patched builds among them (all four frame-step mains), and 7 fail alike (rtld, and New Super Mario Bros. U Deluxe's, which no profile rewrites); the other two (Super Mario Bros. Wonder's mains and Absolum's, no game files here) by their CUTS / OBS data, equal to the previous tables. AGENTS.md, docs, dm/games/README.md, the profiles' comments and the host comments that named a game's cut in patch_nso.py now name its patches.py or sdk_patches.py | ||
|
seedds
|
4c7adf4619 |
c1_full.tsv and cb_extent.tsv are no longer committed: each checkout makes them from its own shader build's Mac outputs (git ignores them in dm/games/*/), as the NSP script does after its iOS shader build. They follow from the MSL this Mac built, every chain change rewrote about 90 of them, and 56704a0 could only carry most over unchecked; 81 files, 18.7 MB. The NSP script makes a missing one as a stale one (chain_tables now says whether both name the chain: with no Mac outputs a reuse builds again and a build stops), and no step's reuse key counts them (profile_files), so the run after they were made does not split again. build_ipa.sh stops for an NVN game whose c1_full.tsv is missing or names another chain, with the c1_guard.py command, before generating anything (the host would give every c1 its own 64 KiB). Checked on The Great Ace Attorney Chronicles from a checkout without them: the first pack made both (0 and 409 entries, as committed), the second reused the split and the iOS shaders with no table lines, a third with them deleted made them from the reused build's Mac outputs; build_ipa.sh stops on a missing and a stale c1_full.tsv. AGENTS.md's chain procedure, dm/games/README.md, dm/gfx/README.md and docs/development.md say so
|
||
|
seedds
|
56704a0be1 |
Super Mario Galaxy 2's half-white Mario on the iPhone (GitHub issue 72): min / max / clamp with a NaN operand give the other operand on every GPU, in the Vulkan, GL and NVN shader chains
- Cause: Galaxy 2's GX vertex lighting (Hagi's SPIR-V) normalizes the light directions the game leaves at (0, 0, 0) for its characters' key and fill lights and takes fast::max(0.0, dot(ldir, NaN)) as the spot term. fast:: leaves a NaN operand's result to the GPU: the Switch's and the M4 Pro's return 0, so the attenuation is 1; the A20 Pro's did not, and the half of Mario that the fill light faces came out white (the fill light sits at the camera's left in view space, so its terminator runs down the middle of a character facing the camera). Found on the Mac by dumping the light uniforms of the character draws (every one has both directions at zero) and a Metal test of the expression on the M4 Pro. - vk_shaders.py, gl_shaders.py and dmsh_build.py (precise): spirv-cross's fast::min / max / clamp are written as precise:: (IEEE fmin / fmax, which give Maxwell's FMNMX and saturation results); the iOS Vulkan and GL libraries are compiled with -fmetal-math-mode=relaxed, as the NVN chain's are. Plain or fast:: min / max / clamp compile to the fast_ intrinsics in every Metal math mode, so relaxed alone does not do it. The NVN chain had the same calls, unguarded by its isnan selects (Paper Mario: The Thousand-Year Door: 124,000 fast::max, 98,000 fast::clamp, 41,000 fast::min). - dmsh_geometry.py capture: clip distances as spirv-cross writes them (gl_ClipDistance [[clip_distance]] [8], and gl_ClipDistance_N [[user(clipN)]] for the pixel stage); since |
||
|
seedds
|
626e6a6188 |
Luigi's Mansion 3 plays on the Mac at 60 fps: profile 0100DCA0064A6000 (update 1.4.0, SDK 7.3.2)
- Files and shaders: Next Level Games' engine keeps every RomFS file but the Wwise banks in archives of its own, <name>.dict indexing zlib chunks in <name>.data; update 1.4.0 ships <name>.patch beside the unchanged base files, a variant bitmask and per-section records that rebuild each section from base and patch bytes (the game's algorithm, main+0x1c8d0 / 0x1cbd0). shader_split.py nlg_shaders reads both: 5,317 shaders (5,149 in the base game), all in init, control sections back to back in one section and their code areas in the same order in another, one NVN program per stage (SINGLE_STAGE '#NLG'); 46 of the first 103 the title uses exist only in the update (docs/agents 01)
- Shader chain, for every game: a texture array over the driver table c2 (Ryujinx's texture array from a buffer, fp_t2das_cb2_0: 586 shadow-cascade pixel shaders indexed by (0x80 + cascade offset) / 8) is named tex_c2a / samp_c2a with its c2 offset (dmsh_build.py c2_array), and the replay fills element k from the slot whose c2 word lies at that offset + 8k (nvn_refl NVN_RB_TEXA / NVN_RB_SAMPA, nvn_mtl.m c2_handle). A pixel shader that writes an output from integer bits, or nothing but constants, gets a second entry point main0_u whose colour outputs are the unsigned vectors of the same bits (dmsh_build.py fs_uint); a pipeline whose written colour attachments are all unsigned integer formats takes it (pso_all_uint). The G-buffer is R32Uint + RGBA8Uint: about 1,400 pipelines failed to build and every scene draw was skipped. Every other profile's dmsh_chain.h rewritten with its c1_full / cb_extent lists carried over (the change touches no c1 or uniform read); Luigi's Mansion 3 built for Mac and iOS
- Host, for every game: the game patches texture handles into its recorded command buffers in place, at the command start + 0x10 it took from GetCommandMemoryUsed before nvnCommandBufferBindTexture (main+0x7c8964; the store at main+0xb7d0d4), so a single texture bind is recorded in the hardware's layout, {stage, slot, handle at +0x10, texture / sampler pool bases} (OP_BIND_TEX1, nvn_obj.h bind_rec decodes it at every replay site); the host's own layout put the patched handle over the bind count, the post passes read stale handles (a stretched icon over the title, a white screen in a new game) and a corrupted record now gets a red line. t_nvn: the handle patched at the command start + 0x10 is the one replayed (102 checks)
- Host, for every game: a depth texture drawn through a colour view of the same pixel size (a D16 1024x1024 texture through an R16 view; Metal has no colour view of a depth texture, and the boot aborted in pipeline validation) draws into a colour shadow, copied in before and back after the pass as a compressed target is (nvn_mtl.m shrt_block, D16 and D32F); nvnCommandBufferSetDebugLabel; nvnBlendStateSetAdvancedMode (red line for a mode other than none); a draw whose program has a geometry shader with no library gets a red line naming it (it is drawn without that stage, with the vertex shader's positions as clip coordinates)
- Frame rate: 60 fps (30 on the Switch). The frame limiter (main+0x7a3e70, a frame callback called with the milliseconds since the frame began, main+0xb7c308) runs jobs or sleeps until 33.333 ms have passed, a float in read-only data (main+0x13c94dc), and the game steps by the time that really passed: the same 2 s walk and 1.5 s turn from the same save end at the same spot at 30 and 60 fps, and the title's animation matches at time scale 1.00 (DM_NVN_REGLUM). hle_fpslock.c gets rows with no pointer (the field at main + offset; the loader leaves read-only data writable, the image's data block on iOS too) and a frame-time kind: 1000 / fps_target, at 60 less 0.5 ms, since the work after the limiter made an exact 16.667 ms 59.6 fps with a frame shown twice every 2.5 s; slightly shorter, the presents set the pace (lobby 60.0 fps, display interval p99 18.69 → 16.67 ms). Profile fps_target 60 with present_interval display (the game also asks for interval 2); DM_FPS_TARGET=30 keeps the game's 33.333 ms
- Not done: the tessellation stages (15 hull, 18 domain shaders: two programs draw twice a frame for about 30 frames, in the title scene behind the file select as it slides out before a new game; skipped, since Metal 4 has no tessellation stage) and 10 geometry shaders (none drawn from the title to the lobby). Vertex shaders that feed them can be found from the shader headers (docs/agents 04)
- Mac: the boot notices, the title, a new game, the opening cutscene (the bus ride, the arrival at the hotel) and the lobby in play at 60 fps, about 3.7 GB phys_footprint (3.0 GB of Metal allocations); red lines at run time: the geometry shader that does not build, the conditional render the host draws anyway, primitive restart 0xff (no 8-bit strip seen). Diagnostics: DM_NVN_TREG_ID=n logs that texture ID's registrations; the pass dump prints the handles of texture binds
|
||
|
seedds
|
2000f9274a |
FINAL FANTASY RESONANCE plays on the Mac at 60 fps: profile 0100606026F12000 (update 1.0.1, SDK 20.5.17)
- Engine: Unreal Engine 5.6.1 (Square Enix, the desktop renderer), the newest Unreal so far; everything in one IoStore container beside one pak; the sdk and subsdk0 builds are STEINS;GATE RE:BOOT's - Rewriter, for every module: code before the first unwind entry is one function (each site there was a function of one instruction, so a byte jump table at main+0x219234 went unseen), and a function longer than 256 KB is analysed in pieces cut after a ret / br / b that no conditional branch or b within 64 KB crosses, with far branches and jump table targets checked against windows and retry blocks afterwards (this main's 82 MB without unwind entries took 100 GB of memory as one function; now 2.7 GB resident). Checked: DRAGON QUEST VII Reimagined's main byte-identical to its rewrite by the previous patch_nso.py; FINAL FANTASY RESONANCE's, Octopath Traveler 0's and DRAGON QUEST XI S's byte-identical between two different cut placements; only DQ VII's main among the 120 modules of 33 games on hand has other sites before its first unwind entry (none in a window or jump table) - Shaders: no archive in the pak; the two libraries are IoStore shader library chunks whose tables and shader groups are Oodle-compressed, read through the game's own Oodle in the Mac host's tool mode with no boot (uepak.py Toc / io_archive / OodleHost; library names from CityHash64 of the manifest's archive names) and written as decoded version 2 archives; the NSP script uses them for any game with IoStore shader libraries. The pak's single-block files store block size 0 (Pak.entry). 73,496 unique shaders; 11 do not build (3 geometry, 8 compute with 3D image atomics), none met so far. 2,077 particle vertex shaders pick their texture slot at run time (tex_dyn_slots); CopyStencilToLightingChannelsPS writes integers to an R8Uint target (shader_facts_manual.tsv; without it the sprites were dark silhouettes) - Host, for every game: a shader with run-time texture handles in a game without tex_dyn_slots freed its reflection twice and aborted the host (nvn_mtl.m); oe::MuteRecording / UnmuteRecording (no-ops) - Memory: the FMallocBinned3 constructor call is cut and the host's allocator runs Unreal's heap (hle_uemalloc.c layout UE5_6: UE 5.5+ FMalloc has 32 slots, with MallocZeroed and two more TLS cache calls); the SDK's node area is cut to 62 MB. The engine's own page allocator only grows its use of the engine region (2.03 GB of address space over seven minutes, about 1 GB mapped), and a 2 GB region aborted in the first battle, so the region is 4064 MB in Engage's 5 GB block (this sdk build aligns regions to 2 MB, no alignment cut needed) - Frame rate: 60 fps (30 on the Switch): the host writes rhi.SyncInterval 1 (hle_fpslock.c rows for this build; profile fps_target 60); the game steps by real time (DM_NVN_REGLUM traces at 30 and 60 fps match from the load through a walk) - Mac: first-boot settings, the title, a new game, the opening cutscene, the first field and its autosave, Continue, and the first two battles at 59.7 fps, 2.3-2.6 GB resident, flat over seven minutes, no red line apart from a few texture type mismatches at one moment of the new game's load in some runs. Docs: docs/agents 01, 03, 04 and development.md |