diff --git a/AGENTS.md b/AGENTS.md index ca880c6..97c58be 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,19 +1,18 @@ # Notes for agents - This repository runs Nintendo Switch games natively on Apple platforms: the game's own code runs, rewritten only where Apple's kernel requires it. Forty-seven games are playable on the iPhone: Super Mario Bros. Wonder (update 1.2.1, profile `010015100B514000`), Fire Emblem Engage (update 2.0.0, `0100A6301214E000`), Metroid Dread (update 2.1.0, `010093801237C000`), 13 Sentinels: Aegis Rim (update 1.10, `01003FC01670C000`), Super Mario 3D World + Bowser's Fury (update 1.2.2, `010028600EBDA000`), The Legend of Zelda: Breath of the Wild (update 1.9.0, SDK 20, `01007EF00011E000`), The Legend of Zelda: Link's Awakening (update 1.1.0, SDK 8, `01006BB00C6F0000`) Super Mario Odyssey (update 1.4.1, SDK 7, `0100000000010000`), Hades (update 1.0.38233, SDK 9, `0100535012974000`), Hyrule Warriors: Age of Calamity (update 1.3.0, SDK 9.6, `01002B00111A2000`), Fire Emblem: Three Houses (update 1.2.0, SDK 6.5, `010055D009F78000`), Super Mario Galaxy (update 1.3.1, SDK 18, `010099C022B96000`) The Legend of Zelda: Tears of the Kingdom (update 1.4.3, SDK 20, `0100F2C0115B6000`) Monster Hunter Rise (update 16.0.2, SDK 11.4.3, `0100B04011742000`), Ace Attorney Investigations Collection (update 1.0.0.2, SDK 17.5.3, `010005501E68C000`), Hades II (base game, SDK 19.3.9, `0100A00019DE0000`), Unicorn Overlord (update 1.05, SDK 16.2.4, `010069401ADB8000`), Rhythm Heaven Groove (base game 1.0.0, SDK 21.4.0, `0100D9F01D474000`), Octopath Traveler 0 (update 1.0.5, SDK 19.3.13, `01005270232F2000`), Pokémon Legends: Z-A (update 2.0.2, SDK 19.3.11, `0100F43008C44000`), Cuphead (update 1.3.7, SDK 5.5.0, `0100A5C00D162000`), Super Mario Galaxy 2 (update 1.4.0, SDK 18.3.5, `0100FD8022DAA000`), FINAL FANTASY TACTICS - The Ivalice Chronicles (update 1.4.0, SDK 19.3.9, `010038B015560000`), Absolum (update 1.1.1, SDK 18.3.2, `0100A83022A22000`), Captain Toad: Treasure Tracker (update 1.4.2, SDK 8.2.0, `01009BF0072D4000`) and Brain Memory Training Academy (base game 1.0.0, SDK 16.2.4, `0100F5D01DA3A000`), The Great Ace Attorney Chronicles (base game 1.0.0, SDK 10.4.1, `010036E00FB20000`), STEINS;GATE RE:BOOT (update 1.0.3, SDK 20.5.17, `01008BC024FDE000`), Big Brain Academy: Brain vs. Brain (update 1.2.0, SDK 11.4.3, `0100620012D6E000`), Patrick's Parabox (update 1.0.1, SDK 13.3.7, `0100E44016C22000`), Celeste (update 1.4.1.1, SDK 15.3.2, `01002B30028F6000`), DRAGON QUEST XI S: Echoes of an Elusive Age – Definitive Edition (update 1.0.4, SDK 7.3.2, `01006C300E9F0000`), Monster Train 2 (update 1.1.1, SDK 18.3.5, `010051701FB46000`), Raging Loop (base game 1.0.0, SDK 7.3.2, `010020400E1C2000`), DARK SOULS: REMASTERED (update 1.0.3, SDK 4.5.0, `01004AB00A260000`), Paper Mario: The Thousand-Year Door (update 1.0.1, SDK 15.3.2, `0100ECD018EBE000`), Hollow Knight: Silksong (update 1.0.30000, SDK 20.5.8, `010013C00E930000`), Catherine: Full Body (update 1.0.1a, SDK 8.2.1, `0100BF00112C0000`), Digimon Story Cyber Sleuth: Complete Edition (update 1.0.3, SDK 6.4.0, `010014E00DB56000`), Animal Crossing: New Horizons (update 3.0.3, SDK 20.5.4, `01006F8002326000`), DRAGON QUEST VII Reimagined (update 1.2.0, SDK 19.3.1, `0100A9D01C446000`), FINAL FANTASY RESONANCE (update 1.0.1, SDK 20.5.17, `0100606026F12000`), Luigi's Mansion 3 (update 1.4.0, SDK 7.3.2, `0100DCA0064A6000`), AI: THE SOMNIUM FILES (update 1.0.1, SDK 6.4.2, `010089B00D09C000`), ANIMAL WELL (update 1.0.10, SDK 20.5.7, `010020D01AD24000`), AI: THE SOMNIUM FILES - nirvanA Initiative (update 1.0.2, SDK 13.3.5, `0100713016CCC000`) and Mario Kart 8 Deluxe (update 4.0.0, SDK 4.4.0, `0100152000022000`). Hades renders with Vulkan, which the host implements on Metal (`dm/gfx/vk/hle_vk.m`; `main` imports Vulkan, not NVN). Age of Calamity is Koei Tecmo's KTGL engine: files in RDB archives, shaders as GLSLC outputs (`shader_split.py` `idrk_shaders`). Three Houses is KTGL too; 9 of its shaders exist only as compiles of the game's own GLSLC at run time, captured once on the Mac (its `main` holds GLSL sources and imports `glslcCompile`). Breath of the Wild and Tears of the Kingdom run at 60 fps through a frame step the host decides in whole vsyncs (`FRAME_STEP` in their profiles' `patches.py`, `nvn_fstep.h`, profile keys `fps_target` and `frame_step`); Tears of the Kingdom has a 5 GB guest layout, and its `sdk` build (Breath of the Wild's) has the SDK's address-region node area cut (`sdk_patches.py`): without it the resource heaps ran out of alias region at the first descent from the Great Sky Island and the game played its memory-shortage Blood Moon, which put Link back at the start (its log shows `MSF`). Monster Hunter Rise is Capcom's RE Engine: files in KPKA PAK archives (`shader_split.py` `kpka_jobs`, shaders in SDF files), the base game an XCI, its GPU heap a 64 GB sparse virtual pool (`hle_nvn.c` `vpool_*`, a 4 KB page table the replay reads textures through); it runs at 60 fps (30 on the Switch). Super Mario Galaxy is the Wii game on NERD's Hagi layer (PowerPC code recompiled into `main`), renders with Vulkan, and generates its shaders at run time, which the NSP script captures on the Mac (no SPIR-V file in its RomFS); its pointer is the touch screen in handheld mode (profile key `op_mode`). Ace Attorney Investigations Collection is Unity 2022 IL2CPP: its shaders keep their parameters as blob entries of their own (`shader_split.py` `unity_code_blobs`), and its C# file API goes through the SDK's C library, whose path checks the host answers (`hle_fs.c` `IsPathMounted` and the length limits). Hades II, unlike Hades, renders with NVN; its shaders bind textures at fixed slots up to 34 (slot 34's handle lies in the driver's image table, `dmsh_build.py`), and the two long lines through Melinoë while the staff's Omega attack charges are the game's own aim lines, not a rendering bug. Unicorn Overlord is Vanillaware's engine like 13 Sentinels (shaders compiled into `main`); both need the profile key `alloc_big_align`, which aligns large allocations to 64 KB so the ring heaps the engine cuts from a memory pool fit. Rhythm Heaven Groove is Nintendo's own 2D engine on SDK 21, the newest SDK so far (its DSP command list has no magic, `hle_audren.c`); it plays handheld with Boost (profile keys `op_mode handheld`, `perf_mode boost`), which skips its TV latency calibration and keeps its 1920x1080 draw size. Octopath Traveler 0 is Unreal Engine 5: everything in one pak with an AES-encrypted index whose key is found in `main` (`dm/gfx/tools/uepak.py`), its shaders Oodle-compressed in the pak's shader code archives and decoded once on the Mac by the game's own Oodle (any game with Unreal paks, `DM_OODLE_JOB`), its allocator replaced by the host's because Unreal's reserves 37 GB of address space (a cut in its `patches.py`, `hle_uemalloc.c`), its address region capped by the game's own argument (profile key `game_args`), and the particle shaders that pick a texture slot at run time bound by slot (profile key `tex_dyn_slots`); it runs at 60 fps (30 on the Switch) because the host writes Unreal's `t.MaxFPS` and `rhi.SyncInterval` in game memory (`hle_fpslock.c`, profile key `fps_target`). Pokémon Legends: Z-A is Game Freak's Trinity engine: everything in the packs of `arc/data.trpfs` (FlatBuffers, every file Oodle-compressed), which the shader split decodes with the game's own decoder in the Mac host's tool mode (`shader_split.py` `trinity_shaders`, `hle_oodle.c` `DM_OODLE_SERVE`); its 12 GB StandardAllocator is cut to 4 GB to fit a 5 GB guest layout, and the host refuses maps past the pool as Horizon does (profile key `mem_map_limit`); its characters are skinned with transform feedback (`dmsh_build.py` `vs_tf`, `nvn_mtl.m` `tf_draw`), and its occlusion queries, which decide how characters are animated, need the draw's base instance in the driver segment (c0 + 0x644, `encode_args`). Cuphead is Unity 2017.4 IL2CPP on SDK 5.5.0: Unity 2017's serialized and Shader formats are read by `unityfs.py` / `shader_split.py`, its rewritten modules (80 MB) need the profile key `mem_used` above the default 64 MB, or its `SetHeapSize` fails, and its NVN pool allocator breaks on a zero-size block, so the host never reports a ZCull storage size of 0 (`hle_nvn.c` `zcull_size`). Super Mario Galaxy 2 is Galaxy's Hagi layer with Galaxy's `sdk` build; its movie player reserves 384 MB of virtual address memory, which fits only because a rewriter cut on that `sdk` build shrinks the SDK's address-region node area to 62 MB (`sdk_patches.py`; the profile's alias region is 512 MB). FINAL FANTASY TACTICS - The Ivalice Chronicles is Square Enix's engine on NVN: its shaders are GLSLC outputs in loose technique files, which `shader_split.py` reads in any RomFS file no other reader takes; its UI textures are written by the CPU into uncached pools (profile keys `tex_flush_upload`, `tex_dirty_upload`); and two job-pool workers share one auto-clear event, so a build cut makes a worker woken in another's place pass the wake on (its `patches.py`). Absolum is FNA (XNA 4 in C#) on .NET NativeAOT with SDL3 and FNA3D's NVN driver: its `main` starts its text with 1.7 MB of data the rewriter leaves alone (`nso_code.py`), its shaders are GLSLC outputs in `shaders.nx.bin`, its picture is flipped by a viewport swizzle the host applies, and everything lives in virtual address memory: malloc's 40 GB StandardAllocator is cut to 2.5 GB and the SDK's node area and large-region alignment are cut (its `patches.py` and `sdk_patches.py`), and the .NET GC's region range is 1.5 GB (profile key `game_env`), to fit a 5 GB guest layout. Captain Toad: Treasure Tracker is Nintendo EPD Tokyo's engine (the `al` library over sead / agl, as 3D World and Odyssey) and needed no host change: it plays handheld because its pointer is the touch screen there (the Joy-Con gyro docked; profile key `op_mode`), and agl's post-effect program that samples its own attachment reads a snapshot (profile key `feedback_snap`). Brain Memory Training Academy is Unity 2022.3 IL2CPP with no bundles: its serialized files carry no type trees and nothing lends it a `Shader` tree, so `unityfs.py` has Unity 2022.3's `Shader` layout written out (`KNOWN_TREES`), and like Ace Attorney Investigations Collection it needs the SDK's address-region node area cut on its `sdk` build (`sdk_patches.py`). The Great Ace Attorney Chronicles is Capcom's MT Framework: files in ARC archives of zlib entries, its 460 shaders in one shader package inside one of them (`shader_split.py` `mtspk_shaders`); it copies its textures into a CPU-cached pool after creating them and never flushes (its `main` has no cache maintenance at all), so it needs `tex_flush_upload` and `tex_dirty_upload` like FINAL FANTASY TACTICS. STEINS;GATE RE:BOOT is M2's ADV engine (E-mote characters, PSB archives, H.264 movies through the SDK's movie library): it needed `nn::audio::SetAudioOutVolume` (`hle_audio.c`), and because it decodes every picture into one reused region of a pool that lives for good, the storage-sharing texture table now drops an entry whose storage another texture takes over (`nvn_mtl.m` `alias_take_over`); it draws 1280x720 in both modes by its own choice. Big Brain Academy: Brain vs. Brain is NintendoWare Bezel Engine 1.56: nearly all of RomFS is in Bezel Engine archives of zstd entries, its shaders in their `.bfsha` entries (`shader_split.py` `bea_members`); its `nninitStartup` sets a heap or a 60 GB virtual address region by `IsVirtualAddressMemoryEnabled`, so the host reports the NPDM's system resource (profile key `sys_resource`, 0 here); its activity previews loop the movie decoder with Flush and Start from four threads (`hle_movie.c`); and it plays handheld with Boost (profile keys `op_mode handheld`, `perf_mode boost`), because only handheld does it read the touch screen. Patrick's Parabox is Unity 2020.3 IL2CPP with no bundles: `unityfs.py` has Unity 2020.3's `Shader` layout written out (`KNOWN_TREES`), it needs the SDK's address-region node area cut on its `sdk` build like Brain Memory Training Academy (`sdk_patches.py`), and Unity's player hands the shared fonts straight to FreeType, so a game that imports no `ScalableFontEngineHelper::Decode` gets the TTF itself (`hle_sdk16.c` `font_skip`). Celeste is MonoGame for the Switch, its C# compiled to C++ by Sickhead's BRUTE: its shaders are MonoGame effects (MGFX) in its XNB files and in `main` (`shader_split.py` `mgfx_effect`), and it needed no host change. DRAGON QUEST XI S is Unreal Engine 4.18: legacy paks with LZ5 blocks, decoded on the Mac by the game's own LZ5 (`uepak.py`, `hle_oodle.c`), its shaders inline in the cooked packages (`shader_split.py` `ue_legacy_jobs`), and the slice-writing geometry shader of its colour-grading LUT folded into the vertex shader (`dmsh_build.py` `gs_slice`, `nvn_mtl.m` `prog_fold_gs`); Square Enix's sound driver never waits on the renderer event, so audio catch-up frames wait for queue space (`hle_audren.c`); its movie player plays only once every track holds a decoded output, so audio movie decoders keep 32 (`hle_movie.c`); and its Unreal region is capped at 4064 MB (profile key `game_args`, MB up to 4095) with the node area and large-region alignment cut on its SDK 7.3.2 build (`sdk_patches.py`) for a 5 GB guest layout; it runs at 60 fps (30 on the Switch) because the host writes Unreal's `t.MaxFPS` and `nvn.SyncInterval` in game memory (`hle_fpslock.c`, profile key `fps_target`). Monster Train 2 is Unity 2022.3 IL2CPP on Super Mario Galaxy's SDK 18.3.5 `sdk` build: Unity creates a thread only when `nn::os::GetFreeThreadCount` (`svcGetInfo` type 24, `hle_svc.c`) is at least 1; its MP4 intro needs a movie audio decoder with more output slots than a video one, because Unity's player holds about half a second of audio before it plays (`hle_movie.c` `MD_NOUT`); and its post-processing compute shaders (the colour grading LUT among them) are Unity `ComputeShader` objects, which `shader_split.py` reads (`unity_compute`) with Unity 2022.3's layout written out in `unityfs.py`. It reads no touch, so it stays docked with the 3D pad layout, and it runs at 60 fps (30 on the Switch) with the profile key `present_interval display`, as its animations follow the time that passed. Raging Loop is Unity 2018.3 IL2CPP with no bundles on DRAGON QUEST XI S's `sdk` and `subsdk0` builds: `unityfs.py` has Unity 2018.3's `Shader` layout written out (`KNOWN_TREES`, an older form than the 2020 and 2022 layouts, with the parameters in each subprogram), and it needed no host change; it reads no touch, so it stays docked with the 2D pad layout. DARK SOULS: REMASTERED is FromSoftware's engine on SDK 4.5.0, the oldest SDK so far: its shaders are GLSLC outputs in BND3 archives inside the dvdbnd archives (`shader_split.py` `bdt_shaders`), and SDK 4's GLSLC outputs are read through their section headers (`glslc_section`); it keeps 12 sampler pools with different samplers under the same IDs, so handles a shader reads from a uniform buffer take the base of the pool bound at replay (`nvn_mtl.m` `ubo_tex_id` / `ubo_samp_id`); and it runs at 60 fps (30 on the Switch) with two rewriter cuts, its frame limiter's 30 and its frame update's fixed 1/30 s step (its `patches.py`). Paper Mario: The Thousand-Year Door is Intelligent Systems' engine on NintendoWare, on Celeste's SDK 15.3.2 `sdk` build: nearly every RomFS file is zstd (`.zst`, which `shader_split.py` decompresses by magic); its AngelScript call bridges leave sp misaligned, which a general rewriter rule rounds up (`site_sp_align`); it runs at 60 fps (30 on the Switch) by the host's frame step (`FRAME_STEP`, with the floor of its vsync step cut from 2 to 1); and its three job workers poll the job list with no SVC, which held it at 24 fps on an iPhone until the general spin wait rule (`patch_nso.py` "Spin waits", `hle_thread.c` `dm_spin_wait`). Hollow Knight: Silksong is Unity 6 (6000.0.50f1) IL2CPP, the newest Unity so far: its bundles carry their own type trees, so `unityfs.py` needed no layout; Unity 6 polls `nn::hid::GetDebugPadStates` every frame (`hle_hid.c`), and it takes all its memory itself through `nn::os::AllocateAddressRegion` / `AllocateMemoryPages` with no heap, so it runs in the 5 GB guest layout that `profile_guess.py` now gives every game with an NPDM system resource, with the SDK's address-region node area cut on its `sdk` build (`sdk_patches.py`). Catherine: Full Body is Atlus's GFD engine with CRI middleware (CPK archives, ADX2 sound, Sofdec2 movies through `movie::Decoder`, whose player reads `nv12-uv-stride`, `hle_movie.c`): its 3,128 shaders are GLSLC outputs in `data/shaders_binary`, `main` and two compiles of its own GLSLC at boot, which the host lets run by reporting at least that GLSLC's GPU code version (`hle_glslc.c`); that GLSLC writes every vertex shader as a VertexA part (shader type 0, the position) and a VertexB part, which `dmsh_build.py` merges as Ryujinx does (`vertex_ab`; the screen was black without it), and because the game reuses a copy's staging memory a frame after recording it, the host commits a segment at the first pass boundary after a buffer → texture copy (`nvn_mtl.m` `g_cut_copy`). Digimon Story Cyber Sleuth: Complete Edition is Media.Vision's MVGL engine and renders with NVIDIA's desktop OpenGL 4.5 through EGL, not NVN, which the host implements on Metal (`dm/gfx/gl/hle_gl.m`: the 13 EGL imports and the 79 GL functions its code calls of the 2,956 its glad loader fetches, counted with `dm/tools/loader_census.py`; `profile_guess.py` gives `graphics gl` when `main` imports `eglCreateWindowSurface` and names `glShaderSource`); its shaders are GLSL ES 3.20 source in its MVGL archives (`dm/gfx/tools/mvgl.py`: MDB1 with doboz entries) and in `main`, which `gl_shaders.py` translates offline with glslang and SPIRV-Cross (packing needs `brew install glslang`) and the host finds by the hash of its text. Animal Crossing: New Horizons is Nintendo EPD's engine (sead / agl / gsys) on SDK 20.5.4: its `main` links its own HIPC client for `olsc:u` (cloud backup settings), which the host answers with sessions of its own (`hle_sf.c`), its island lives in device save data (`MountDeviceSaveData`, `hle_fs.c`), it skips clears with conditional rendering (profile key `render_cond`), its 63 MB `main` needed halfword jump tables padded only to their largest entry (`patch_nso.py` `half_count`), and its `SaveProc` builds a new island in grid walks that yield once per cell, so a yield whose loop moved its callee-saved registers since the last one now returns at once instead of backing off, for every game (`hle_thread.c` `thr_yield`; the island showed after 77 s of black screen, now about 10 s). DRAGON QUEST VII Reimagined is Unreal Engine 4.27: IoStore containers beside one pak, its shaders LZ4 blocks in the pak's shader code archives, which `uepak.py` decodes itself (`lz4_shaders`), and its allocator replaced as Octopath Traveler 0's (a cut and an observation gate in its `patches.py`, `hle_uemalloc.c` layout `UE4_27`) in a 2 GB engine region (profile key `game_args`); its audio renderer runs at 32 kHz, resampled by the host on the CPU path (`hle_audren.c`); and Unreal's volumetric lightmap merge sizes its indirection texture with a query on the image it writes, so a texture slot that shares its c2 word with an image slot reads the bound image (`nvn_mtl.m` `tex_slot_h`) and the translator types such a query from the image (`dmsh_xlate` `ImageTypesOf`): without them every character in the opening cutscene glowed white; it runs at 60 fps (30 on the Switch) because the host writes Unreal's `t.MaxFPS` and `rhi.SyncInterval` in game memory (`hle_fpslock.c`, profile key `fps_target`), the same two limits a 60 fps mod for the base game lifts with code patches (`UEngine::SetMaxFPS` returning at once, the name `r.VSync` broken). FINAL FANTASY RESONANCE is Unreal Engine 5.6.1, the newest Unreal so far: its shader libraries are IoStore chunks in its container, not in its pak, two layers of Oodle that `uepak.py` reads through the game's own decoder in the Mac host with no boot (`Toc`, `io_archive`, `OodleHost`); its 94 MB `main` is 95% without unwind entries, so the rewriter analyses long functions in pieces (`patch_nso.py` `pieces`; one 82 MB function took 100 GB of memory) and treats code before the first unwind entry as a function; its allocator is replaced as Octopath Traveler 0's (UE 5.6's 32-slot FMalloc, `hle_uemalloc.c` layout `UE5_6`), and its engine region is 4064 MB (profile key `game_args`), because the engine's own page allocator never gives address space back (2 GB aborted in the first battle); 2,077 particle shaders need `tex_dyn_slots`; it runs at 60 fps (30 on the Switch) by the host's `rhi.SyncInterval` write (`hle_fpslock.c`, profile key `fps_target`). Luigi's Mansion 3 is Next Level Games' own engine on NVN: its RomFS files are in the engine's `.dict` / `.data` archives, and update 1.4.0 ships `.patch` files beside the base ones, which `shader_split.py` applies as the game does (`nlg_shaders`); the game patches texture handles into its recorded command buffers in place, at the command start + 0x10 it took from `GetCommandMemoryUsed`, so single texture binds are recorded in the hardware's layout (`OP_BIND_TEX1`, every game; without it the title showed a stretched icon and a new game a white screen); its G-buffer is two integer targets, which its pixel shaders write through a second, integer-output entry point (`dmsh_build.py` `fs_uint`, every game); and it runs at 60 fps (30 on the Switch) because the host writes its frame limiter's 33.333 ms, a constant in read-only data, as 1000 / `fps_target` less half a millisecond (`hle_fpslock.c`, with `present_interval display`). Its tessellated draws are skipped (Metal 4 has no tessellation stage) and its geometry shaders do not build; neither was drawn in play so far. AI: THE SOMNIUM FILES is Unity 2017.4 IL2CPP (Cuphead's Unity layout) with FMOD: its 168 movies are H.264 MP4s that Unity's player copies into NV12 texture pairs, and its Spike Chunsoft logo movie held the game before the title until a released output slot also went on with the H.264 decoder's drain at the end of a stream (`hle_movie.c`: Unity holds all 16 video outputs by then); it binds its vertex and fragment programs apart, and the precompile prediction reused the ring slot of the combined program the replay had saved, so a draw ran with another pair's shaders until each restore of the replay state took the combination again from the per-stage bindings (`nvn_mtl.m` `prog_resolve`, every game); it plays docked with the 3D pad layout (the investigation's view is on the right stick). ANIMAL WELL embeds all its assets and 46 BNSH shaders in `main`; its base and update Program NCAs have no data section, which `nx_extract` represents as an empty RomFS directory. It needed no host change; its profile uses the 2D pad layout. AI: THE SOMNIUM FILES - nirvanA Initiative is Unity 2020.3 IL2CPP (Patrick's Parabox's Unity) with Addressables bundles and FMOD, and needed no host change: Unity 2020.3 keys a compute shader kernel's variants by keyword set (`shader_split.py` `unity_compute`, `variantMap`); its `nninitStartup` hands pool − used to StandardAllocator as virtual address memory and Unity reserves its own regions beside it, so it has the SDK's address-region node area cut on its `sdk` build (`sdk_patches.py`) and a 1 GB pool, which the NSP script's heap check now sets for such a game (`profile_guess.py` `heap_signals`: the alias region nearly full within seconds and no heap); it plays docked with the 3D pad layout (the investigation's view is on the right stick). Mario Kart 8 Deluxe is Nintendo's sead / agl / gsys engine on SDK 4.4.0, the oldest SDK so far: it lists the six default Miis at boot and builds Mii heads from the console's MiiModel system data, so the host answers the Mii database itself (`hle_mii.c`: the six default Miis, none of the console's own) and mounts a stand-in of that archive with every part empty (`dm_mii_standin`, `hle_fs.c` `MountSystemData`; the firmware is not the host's to ship, so Mii heads draw nothing); its DLC licence is decrypted in a 0x3130-byte heap that also holds the mount cache, now sized as Horizon's RomFS working memory (`romfs_work_size`); it mounts device and account save data, which are now two host saves for a game whose `main` imports both (at boot it deletes `CommonSave:/userdata.dat`); SDK 4's DeviceSink is command 0xf of 0x130 bytes, and the audio renderer's update lock is read off each SDK's own `AudioRenderSystem::Update` (`hle_audren.c` `update_lock`: it was SDK 16's for every SDK before 20); it evaluates its conditional rendering (profile key `render_cond`) and runs at 60 fps. [README.md](README.md) is the player-facing page (games, packing, playing); [docs/development.md](docs/development.md) has the internals, the script's options, building by hand, prewarm lists and per-game notes. -- Each game's constants and shader tables form a game profile in `dm/games//`, where `` is the base game's title ID: `game.tsv` plus the tables (and `patches.py` when the rewriter has patches for its builds) and `dmsh_chain.h`. Mario's profile is `010015100B514000`. Tools pick the profile with `DM_GAME`; `make -C dm/host` and `build_ipa.sh` pick it with `GAME`. There is no default game: set it every time (the tools stop and list the profiles otherwise). Put per-game values in the profile, not in code ([dm/games/README.md](dm/games/README.md)). +- Each game's constants and shader tables form a game profile in `dm/games//`, where `` is the base game's title ID: `game.tsv` plus the tables (and `patches.py` when the rewriter has patches for its builds). Mario's profile is `010015100B514000`. Tools pick the profile with `DM_GAME`; `make -C dm/host` and `build_ipa.sh` pick it with `GAME`. There is no default game: set it every time (the tools stop and list the profiles otherwise). Put per-game values in the profile, not in code ([dm/games/README.md](dm/games/README.md)). - Prefer general optimizations and fixes to game-specific rules. Performance work (frame pacing, power, caches, load times) counts here as much as correctness. First look for a change that holds for every game: Horizon / NVN / Vulkan semantics, or what the binary, its SDK version, its imports or its files show. Use a game-specific rule (a profile key, a rewriter cut, a build-ID gate, a per-game branch in the host) only when it is absolutely necessary, that is, when no general rule can do it. - Per-build knowledge is keyed by the NSO build ID: rewriter cuts, observation gates and frame-step stubs in the profile's `patches.py` (a game's own modules) or `dm/tools/sdk_patches.py` (Nintendo's `sdk` modules, which several games ship), which `dm/tools/patch_nso.py` (the general rewriter) loads, the profile's `build` rows (the builds the game was brought up on, and their display versions; the NSP script packs any other version from a derived profile, without those builds' fixes), and host behaviour that uses fixed module addresses checks `module_t.build_id` (from `fixups.tsv`; `dm_main_build()`). When a new game build appears, carry addresses over with `dm/tools/nso_fnmap.py` and check them by disassembly; never let fixed addresses run on a build they were not found in. - To analyse or port **another** Switch title, first run `scripts/build_ipa_from_nsp.py` on its NSP folder. With no matching profile, it writes a survey to `/_survey/survey.md`, then a provisional profile guessed from the game files (`dm/games/profile_guess.py`; `<NSP folder>/<title ID>_build/profile/<title ID>/`, found by the tools through `DM_GAMES_DIR`), with a short heap check on the Mac when the game has virtual address memory, and a best-effort IPA that may not boot. Then follow [docs/agents](docs/agents/README.md) (extraction, binary survey, host bring-up failure catalogue, graphics specifics, debugging recipes) and add a profile. - Scripts the user runs live in `scripts/` at the repository root (now `build_ipa_from_nsp.py`). Each one runs both in Spyder (F5, configured by the settings block at its top) and from a terminal (command-line options override those settings; non-zero exit on error). Tools that only agents or other scripts call stay under `dm/`. - Game files, keys and anything extracted from them live in the game's NSP folder, never in this repository and never in the home folder (it is nearly full): the work folder `<NSP folder>/<id>-work` for extraction, runs and analysis, beside the NSP script's `<id>_build/`. Tool venvs and other large scratch data go on that drive too. Do not print key values. - Language: everything is in English: `README.md`, docs, code comments, log messages, script output and other files. Red log lines (something unimplemented or wrong) start with `[RED] `. Text that must match a game's own Chinese data is written as escapes (`"\xE8\xB5\xB7"`), never as characters. -- After changing anything on the shader translation chain, run the chain procedure for each affected profile. The chain covers `dm/gfx/tools/dmsh_*`, `Program.cs`, the profile's `shader_map.tsv` / `shader_facts.tsv` / `shader_pairs.tsv`, and its `shader_*` keys. The procedure: - 1. Run `dmsh_chain.py --header`, which writes the profile's `dmsh_chain.h`. - 2. Rebuild the shaders (Mac and, with `IOSOUT`, iOS). - 3. Regenerate the profile's `c1_full.tsv` (`c1_guard.py`) and `cb_extent.tsv` (`cb_extent.py`) for Mac runs. They are made from each build's Mac outputs and git ignores them; the NSP script makes its own. - 4. Rebuild the host. +- The shader translation chain has one version for every game, `DMSH_CHAIN` in `dm/gfx/nvn/dmsh_chain.h`, which the host is compiled with and loads only libraries of. It covers the chain's tools: `dm/gfx/tools/dmsh_*` (`dmsh_build.py`, `dmsh_chain.py`, `dmsh_fetch.py`, `dmsh_vin.py`, `dmsh_geometry.py`, `dmsh_facts.py`), `Program.cs`, `ios_metal.py` and `dmsh_chain.lock`. A profile's own inputs (`shader_map.tsv`, `shader_facts.tsv`, `shader_pairs.tsv`, `bf_want.tsv`, its `shader_*` / `nvn_*` keys) are not part of it: they are in each shader's cache key, and their digest stamps the libraries (`b-<V>/inputs.tsv`). The procedure: + 1. After changing a chain tool, run `dmsh_chain.py --header` once and commit `dm/gfx/nvn/dmsh_chain.h` with the change (`dm/tools/tests/test_nsp_chain.py` fails on a stale one). + 2. After that, or after changing a profile's inputs, rebuild that profile's shaders: a full `dmsh_build.py` run (Mac and, with `IOSOUT`, iOS) stamps its libraries and writes the profile's `c1_full.tsv` and `cb_extent.tsv` from its Mac outputs (git ignores them; the host takes them only beside the stamp they name). Give each game its own output directory. + 3. Rebuild the host. - The host only loads libraries of its own chain version. The translator is `dm/gfx/tools/dmsh_xlate_bin`, which matches `dmsh_chain.lock`; a fresh `dotnet publish` does not. + The translator is `dm/gfx/tools/dmsh_xlate_bin`, which matches `dmsh_chain.lock`; a fresh `dotnet publish` does not. - iOS builds are unsigned IPAs that the user sideloads and tests; diagnostics come back as the app's `Documents/log.txt`. Do not sign or install builds on the user's devices. - Commit only when asked, with `git -c user.name=seedds -c user.email=seedds@gmail.com commit`. diff --git a/dm/games/0100000000010000/dmsh_chain.h b/dm/games/0100000000010000/dmsh_chain.h deleted file mode 100644 index bf3d48e..0000000 --- a/dm/games/0100000000010000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100000000010000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "9172252ffef7" diff --git a/dm/games/010005501E68C000/dmsh_chain.h b/dm/games/010005501E68C000/dmsh_chain.h deleted file mode 100644 index 829d0f2..0000000 --- a/dm/games/010005501E68C000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010005501E68C000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "7cb671cb35dd" diff --git a/dm/games/010013C00E930000/dmsh_chain.h b/dm/games/010013C00E930000/dmsh_chain.h deleted file mode 100644 index 5c9678d..0000000 --- a/dm/games/010013C00E930000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010013C00E930000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "f06d2f4504e6" diff --git a/dm/games/010014E00DB56000/dmsh_chain.h b/dm/games/010014E00DB56000/dmsh_chain.h deleted file mode 100644 index 3b2611f..0000000 --- a/dm/games/010014E00DB56000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010014E00DB56000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "517b0349bc5f" diff --git a/dm/games/010015100B514000/dmsh_chain.h b/dm/games/010015100B514000/dmsh_chain.h deleted file mode 100644 index 5606299..0000000 --- a/dm/games/010015100B514000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010015100B514000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "0476ee28f82f" diff --git a/dm/games/010015100B514000/game.tsv b/dm/games/010015100B514000/game.tsv index eda8c1f..fd229a7 100644 --- a/dm/games/010015100B514000/game.tsv +++ b/dm/games/010015100B514000/game.tsv @@ -1,4 +1,4 @@ -# Game profile of Super Mario Bros. Wonder (Switch, base game v0 and update 1.2.1). This folder also holds the shader tables (generated by shader_split.py), c1_full.tsv, pso_manifest.tsv and the chain version header dmsh_chain.h +# Game profile of Super Mario Bros. Wonder (Switch, base game v0 and update 1.2.1). This folder also holds the shader tables (generated by shader_split.py), pso_manifest.tsv, patches.py (the rewriter's patches of its builds) and, not committed, c1_full.tsv and cb_extent.tsv # Format: key <TAB> value; several values of one key are separated by spaces; one build row per version; comments must be on their own line. Rebuild the host after changing memory / NVN constants; after changing shader-related keys follow the chain procedure in AGENTS.md # ---- Identity ---- diff --git a/dm/games/0100152000022000/dmsh_chain.h b/dm/games/0100152000022000/dmsh_chain.h deleted file mode 100644 index 3e26045..0000000 --- a/dm/games/0100152000022000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100152000022000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "ae313b896ff8" diff --git a/dm/games/010020400E1C2000/dmsh_chain.h b/dm/games/010020400E1C2000/dmsh_chain.h deleted file mode 100644 index 3e7161e..0000000 --- a/dm/games/010020400E1C2000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010020400E1C2000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "9481e9cb1e53" diff --git a/dm/games/010020D01AD24000/dmsh_chain.h b/dm/games/010020D01AD24000/dmsh_chain.h deleted file mode 100644 index c0b60f9..0000000 --- a/dm/games/010020D01AD24000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010020D01AD24000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "6a4fd7111a5f" diff --git a/dm/games/010028600EBDA000/dmsh_chain.h b/dm/games/010028600EBDA000/dmsh_chain.h deleted file mode 100644 index 488715f..0000000 --- a/dm/games/010028600EBDA000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010028600EBDA000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "4cf9d18f6cda" diff --git a/dm/games/01002B00111A2000/dmsh_chain.h b/dm/games/01002B00111A2000/dmsh_chain.h deleted file mode 100644 index b011412..0000000 --- a/dm/games/01002B00111A2000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 01002B00111A2000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "4c902904ace7" diff --git a/dm/games/01002B30028F6000/dmsh_chain.h b/dm/games/01002B30028F6000/dmsh_chain.h deleted file mode 100644 index 84d27d4..0000000 --- a/dm/games/01002B30028F6000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 01002B30028F6000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "7a4400bf4aa7" diff --git a/dm/games/010036E00FB20000/dmsh_chain.h b/dm/games/010036E00FB20000/dmsh_chain.h deleted file mode 100644 index 1de3103..0000000 --- a/dm/games/010036E00FB20000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010036E00FB20000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "fdf58b5e09d1" diff --git a/dm/games/010038B015560000/dmsh_chain.h b/dm/games/010038B015560000/dmsh_chain.h deleted file mode 100644 index 72cda12..0000000 --- a/dm/games/010038B015560000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010038B015560000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "ece8c845aa92" diff --git a/dm/games/01003FC01670C000/dmsh_chain.h b/dm/games/01003FC01670C000/dmsh_chain.h deleted file mode 100644 index 21ef4bc..0000000 --- a/dm/games/01003FC01670C000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 01003FC01670C000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "b38ccc4d4afd" diff --git a/dm/games/01004AB00A260000/dmsh_chain.h b/dm/games/01004AB00A260000/dmsh_chain.h deleted file mode 100644 index 8772ade..0000000 --- a/dm/games/01004AB00A260000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 01004AB00A260000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "8b36975c9b36" diff --git a/dm/games/010051701FB46000/dmsh_chain.h b/dm/games/010051701FB46000/dmsh_chain.h deleted file mode 100644 index 572f1d9..0000000 --- a/dm/games/010051701FB46000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010051701FB46000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "3d0ce4077073" diff --git a/dm/games/01005270232F2000/dmsh_chain.h b/dm/games/01005270232F2000/dmsh_chain.h deleted file mode 100644 index 940e74b..0000000 --- a/dm/games/01005270232F2000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 01005270232F2000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "9fa1cfbe11a5" diff --git a/dm/games/0100535012974000/dmsh_chain.h b/dm/games/0100535012974000/dmsh_chain.h deleted file mode 100644 index 619e979..0000000 --- a/dm/games/0100535012974000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100535012974000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "806c0cc34311" diff --git a/dm/games/010055D009F78000/dmsh_chain.h b/dm/games/010055D009F78000/dmsh_chain.h deleted file mode 100644 index a6037f8..0000000 --- a/dm/games/010055D009F78000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010055D009F78000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "9474997db7fd" diff --git a/dm/games/0100606026F12000/dmsh_chain.h b/dm/games/0100606026F12000/dmsh_chain.h deleted file mode 100644 index 8a6155f..0000000 --- a/dm/games/0100606026F12000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100606026F12000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "e3afba9007bc" diff --git a/dm/games/0100620012D6E000/dmsh_chain.h b/dm/games/0100620012D6E000/dmsh_chain.h deleted file mode 100644 index 84dc4ec..0000000 --- a/dm/games/0100620012D6E000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100620012D6E000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "ea02d29cc857" diff --git a/dm/games/010069401ADB8000/dmsh_chain.h b/dm/games/010069401ADB8000/dmsh_chain.h deleted file mode 100644 index 6fc8eaa..0000000 --- a/dm/games/010069401ADB8000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010069401ADB8000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "a79f94f0e4a1" diff --git a/dm/games/01006BB00C6F0000/dmsh_chain.h b/dm/games/01006BB00C6F0000/dmsh_chain.h deleted file mode 100644 index a54584e..0000000 --- a/dm/games/01006BB00C6F0000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 01006BB00C6F0000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "fbc8f074f058" diff --git a/dm/games/01006C300E9F0000/dmsh_chain.h b/dm/games/01006C300E9F0000/dmsh_chain.h deleted file mode 100644 index d917ff3..0000000 --- a/dm/games/01006C300E9F0000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 01006C300E9F0000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "130bc16dd3e5" diff --git a/dm/games/01006F8002326000/dmsh_chain.h b/dm/games/01006F8002326000/dmsh_chain.h deleted file mode 100644 index 9d88c8b..0000000 --- a/dm/games/01006F8002326000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 01006F8002326000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "de566a052820" diff --git a/dm/games/0100713016CCC000/dmsh_chain.h b/dm/games/0100713016CCC000/dmsh_chain.h deleted file mode 100644 index 918eaec..0000000 --- a/dm/games/0100713016CCC000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100713016CCC000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "4963f4147256" diff --git a/dm/games/01007EF00011E000/dmsh_chain.h b/dm/games/01007EF00011E000/dmsh_chain.h deleted file mode 100644 index 62eedc2..0000000 --- a/dm/games/01007EF00011E000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 01007EF00011E000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "19aa554b98df" diff --git a/dm/games/010089B00D09C000/dmsh_chain.h b/dm/games/010089B00D09C000/dmsh_chain.h deleted file mode 100644 index 69957a2..0000000 --- a/dm/games/010089B00D09C000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010089B00D09C000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "fb800e90de17" diff --git a/dm/games/01008BC024FDE000/dmsh_chain.h b/dm/games/01008BC024FDE000/dmsh_chain.h deleted file mode 100644 index 7a6df0b..0000000 --- a/dm/games/01008BC024FDE000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 01008BC024FDE000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "be987aacfc19" diff --git a/dm/games/010093801237C000/dmsh_chain.h b/dm/games/010093801237C000/dmsh_chain.h deleted file mode 100644 index 10b448a..0000000 --- a/dm/games/010093801237C000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010093801237C000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "0df8d3b148bd" diff --git a/dm/games/010099C022B96000/dmsh_chain.h b/dm/games/010099C022B96000/dmsh_chain.h deleted file mode 100644 index 1b52c30..0000000 --- a/dm/games/010099C022B96000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010099C022B96000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "d24bb153d3d1" diff --git a/dm/games/01009BF0072D4000/dmsh_chain.h b/dm/games/01009BF0072D4000/dmsh_chain.h deleted file mode 100644 index 735f0d5..0000000 --- a/dm/games/01009BF0072D4000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 01009BF0072D4000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "01c8f27d203f" diff --git a/dm/games/0100A00019DE0000/dmsh_chain.h b/dm/games/0100A00019DE0000/dmsh_chain.h deleted file mode 100644 index cd271fc..0000000 --- a/dm/games/0100A00019DE0000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100A00019DE0000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "c79f78cffe09" diff --git a/dm/games/0100A5C00D162000/dmsh_chain.h b/dm/games/0100A5C00D162000/dmsh_chain.h deleted file mode 100644 index ea62ddc..0000000 --- a/dm/games/0100A5C00D162000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100A5C00D162000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "73dd6f20c76d" diff --git a/dm/games/0100A6301214E000/dmsh_chain.h b/dm/games/0100A6301214E000/dmsh_chain.h deleted file mode 100644 index 7d64b26..0000000 --- a/dm/games/0100A6301214E000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100A6301214E000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "e37b6b7ffb86" diff --git a/dm/games/0100A83022A22000/dmsh_chain.h b/dm/games/0100A83022A22000/dmsh_chain.h deleted file mode 100644 index c019b9b..0000000 --- a/dm/games/0100A83022A22000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100A83022A22000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "3a3a10378f1a" diff --git a/dm/games/0100A9D01C446000/dmsh_chain.h b/dm/games/0100A9D01C446000/dmsh_chain.h deleted file mode 100644 index 5b82053..0000000 --- a/dm/games/0100A9D01C446000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100A9D01C446000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "94031cb2e33a" diff --git a/dm/games/0100B04011742000/dmsh_chain.h b/dm/games/0100B04011742000/dmsh_chain.h deleted file mode 100644 index 22416ff..0000000 --- a/dm/games/0100B04011742000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100B04011742000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "8f5abdfbe523" diff --git a/dm/games/0100BF00112C0000/dmsh_chain.h b/dm/games/0100BF00112C0000/dmsh_chain.h deleted file mode 100644 index 7cb85a5..0000000 --- a/dm/games/0100BF00112C0000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100BF00112C0000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "d93ccdee9f2b" diff --git a/dm/games/0100D9F01D474000/dmsh_chain.h b/dm/games/0100D9F01D474000/dmsh_chain.h deleted file mode 100644 index 6cf249f..0000000 --- a/dm/games/0100D9F01D474000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100D9F01D474000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "c54437137182" diff --git a/dm/games/0100DCA0064A6000/dmsh_chain.h b/dm/games/0100DCA0064A6000/dmsh_chain.h deleted file mode 100644 index 13dba7d..0000000 --- a/dm/games/0100DCA0064A6000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100DCA0064A6000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "21015a145194" diff --git a/dm/games/0100E44016C22000/dmsh_chain.h b/dm/games/0100E44016C22000/dmsh_chain.h deleted file mode 100644 index d7366c1..0000000 --- a/dm/games/0100E44016C22000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100E44016C22000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "832e3f9025e7" diff --git a/dm/games/0100ECD018EBE000/dmsh_chain.h b/dm/games/0100ECD018EBE000/dmsh_chain.h deleted file mode 100644 index f5f2fee..0000000 --- a/dm/games/0100ECD018EBE000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100ECD018EBE000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "476fbfed4a48" diff --git a/dm/games/0100F2C0115B6000/dmsh_chain.h b/dm/games/0100F2C0115B6000/dmsh_chain.h deleted file mode 100644 index 265d493..0000000 --- a/dm/games/0100F2C0115B6000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100F2C0115B6000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "1f5406b48926" diff --git a/dm/games/0100F43008C44000/dmsh_chain.h b/dm/games/0100F43008C44000/dmsh_chain.h deleted file mode 100644 index 6821987..0000000 --- a/dm/games/0100F43008C44000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100F43008C44000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "67a0957df269" diff --git a/dm/games/0100F5D01DA3A000/dmsh_chain.h b/dm/games/0100F5D01DA3A000/dmsh_chain.h deleted file mode 100644 index 15b5ef4..0000000 --- a/dm/games/0100F5D01DA3A000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100F5D01DA3A000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "577add4eb8d5" diff --git a/dm/games/0100FD8022DAA000/dmsh_chain.h b/dm/games/0100FD8022DAA000/dmsh_chain.h deleted file mode 100644 index 0d3c670..0000000 --- a/dm/games/0100FD8022DAA000/dmsh_chain.h +++ /dev/null @@ -1,2 +0,0 @@ -// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 0100FD8022DAA000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/ -#define DMSH_CHAIN "19f3169e6337" diff --git a/dm/games/README.md b/dm/games/README.md index 46756bc..aa0edfb 100644 --- a/dm/games/README.md +++ b/dm/games/README.md @@ -9,9 +9,8 @@ Each game the port knows has a folder `dm/games/<id>/`, named after the base gam - **`vk_capture.tsv`** (a Vulkan game that generates its shaders at run time, Super Mario Galaxy): the keys of the SPIR-V modules it generates. - **`bf_want.tsv`**. - **`patches.py`** (optional): the rewriter's patches of the game's own builds (`main`), keyed by NSO build ID: `CUTS` (byte cuts, each original asserted), `OBS` (observation gates) and `FRAME_STEP` (frame-step stubs). `dm/tools/patch_nso.py` loads every profile's file and applies those of a module's build, wherever it comes from; a build claimed twice stops the rewrite. Cuts of Nintendo's `sdk` modules, which several games ship, are in `dm/tools/sdk_patches.py`. A game without patches has no file, and a derived profile never has one. `dm/tools/tests/test_patches.py` checks that a file names only its own profile's builds. -- **`dmsh_chain.h`**: the chain version, written by `dmsh_chain.py --header`. -Two more tables live there but are not committed (git ignores them). Each is made from the Mac outputs of a shader build, so each checkout makes its own; the NSP script makes them after its iOS shader build. The host takes a table only when its header names the chain version. +The chain version is not in the profile: one version serves every game (`dm/gfx/nvn/dmsh_chain.h`, written by `dmsh_chain.py --header`). Two more tables live in the profile folder but are not committed (git ignores them). A full `dmsh_build.py` run makes both from its Mac outputs, so each checkout makes its own, and stamps its libraries with the digest of the profile's inputs (`b-<V>/inputs.tsv`, `dmsh_chain.py --inputs`). The host takes a table only when its header names its library's chain version and that digest. - **`c1_full.tsv`**: written by `c1_guard.py`. Without it every shader's c1 gets its own 64 KiB. - **`cb_extent.tsv`**: how far into each uniform slot every shader can read, written by `cb_extent.py`. Without it a uniform block the game updates in part inside a render pass is copied whole. @@ -98,12 +97,9 @@ These are not profile keys: the tools read them from the game files each time, b No short run decides `render_cond`, `tex_flush_upload`, `tex_dirty_upload`, `core_preempt`, `gpu_timestamps`, `feedback_snap`, `alpha_from_red`, `vconv_range` or `present_interval`, nor `op_mode` / `pad_layout` outside the presets: they keep their defaults for a developer. A minute's boot without input counted Wonder's clears under render conditions (65,536 or more; it needs no `render_cond`) as readily as Tears of the Kingdom's, saw Breath of the Wild flush 32 CPU-filled textures with data (it needs no `tex_flush_upload`) and Age of Calamity none (its movie frames come later), and Metroid Dread's movie crash shows in about half the runs. On the iPhone, `Documents/env.txt` can still set `DM_OPMODE`, `DM_NVN_GPU_TS` and `DM_NVN_FB_SNAP`, and the pad layout switches in game. - The shader tables, `dmsh_chain.h`, `c1_full.tsv` and `cb_extent.tsv` are written into the same folder. A second run reuses that `game.tsv` (delete it to survey again), so it can be edited by hand. `--provisional` builds from a provisional profile even for a game that has one in `dm/games`, to test the guesses: `DM_GAMES_FIRST=1` makes every tool search `DM_GAMES_DIR` first, and the IPA gets `-provisional` in its name. Tools find profiles outside `dm/games` through `DM_GAMES_DIR` (folders separated by `:`, searched after `dm/games`); `make -C dm/host` reaches such a profile through the symlink `dm/host/gen/<id>/profile`. To keep a game, move the folder into `dm/games/` and bring it up as below. + The shader tables, `c1_full.tsv` and `cb_extent.tsv` are written into the same folder. A second run reuses that `game.tsv` (delete it to survey again), so it can be edited by hand. `--provisional` builds from a provisional profile even for a game that has one in `dm/games`, to test the guesses: `DM_GAMES_FIRST=1` makes every tool search `DM_GAMES_DIR` first, and the IPA gets `-provisional` in its name. Tools find profiles outside `dm/games` through `DM_GAMES_DIR` (folders separated by `:`, searched after `dm/games`); `make -C dm/host` reaches such a profile through the symlink `dm/host/gen/<id>/profile`. To keep a game, move the folder into `dm/games/` and bring it up as below. 2. **Profile.** Copy `010015100B514000/game.tsv` (or the provisional profile) to `dm/games/<title ID>/game.tsv` (the base game's title ID, as the folder name) and fill in the identity, `build` rows and `modules`. Leave the memory values as Mario's until the bring-up measures them ([docs/agents](../../docs/agents/README.md)). 3. **Shader tables.** Run `DM_GAME=<id> shader_split.py … dm/games/<id>/shader_map.tsv dm/games/<id>/shader_facts.tsv dm/games/<id>/shader_pairs.tsv`. Add an empty `shader_map.tsv.conflicts.tsv`, `bf_want.tsv` and `pso_manifest.tsv` (copy Mario's headers). -4. **Chain.** - 1. `DM_GAME=<id> dmsh_chain.py --header` - 2. `DM_GAME=<id> [IOSOUT=…] dmsh_build.py <pkg> <cache>` - 3. `c1_guard.py <cache>/b-<V> dm/games/<id>/c1_full.tsv` and `cb_extent.py <cache>/b-<V> dm/games/<id>/cb_extent.tsv` (local files, not committed) +4. **Shaders.** `DM_GAME=<id> [IOSOUT=…] dmsh_build.py <pkg> <cache>`, with a cache directory of the game's own. A full build also writes `c1_full.tsv` and `cb_extent.tsv` (local files, not committed). The chain version needs nothing: it is the same for every game. 5. **Host bring-up.** Run `make -C dm/host GAME=<id>`, then work through the bring-up loop in docs/agents. Code that only fits one game is gated by its build ID (`dm_main_build()`), never left to run on every game. 6. **IPA.** Use `GAME=<id> dm/ios/tools/build_ipa.sh`, or the NSP script, which now finds the profile. diff --git a/dm/games/game.py b/dm/games/game.py index 52509f1..e3dddbf 100644 --- a/dm/games/game.py +++ b/dm/games/game.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Game profiles: dm/games/<base game title ID>/game.tsv (key <TAB> values..., lines starting with # are comments) + the shader tables and dmsh_chain.h in the same folder. The shader tools, the host build, iOS packaging and the NSP script take each game's constants from here. +# Game profiles: dm/games/<base game title ID>/game.tsv (key <TAB> values..., lines starting with # are comments) + the shader tables (and patches.py, the rewriter's patches of the game's builds) in the same folder. The shader tools, the host build, iOS packaging and the NSP script take each game's constants from here. # Usage: game.py list (list all profiles); game.py <id> <key> (print that key's value, for shell); game.py header <id> <out.h> (generate the host's game_profile.h); # game.py dir <id> (print the profile folder); game.py link <id> <link> (point a symlink at the profile folder: make cannot take paths with spaces) # From Python: import game; g = game.load() (takes DM_GAME; there is no default game: unset is an error that lists the profiles); game.by_title(title ID); g.get / g.words / g.int / g.builds() / g.table(file name) diff --git a/dm/gfx/README.md b/dm/gfx/README.md index 282b3c4..1422e54 100644 --- a/dm/gfx/README.md +++ b/dm/gfx/README.md @@ -42,15 +42,16 @@ The whole chain is driven by `tools/dmsh_build.py`; `scripts/build_ipa_from_nsp. - **Build outputs**: `<key>_b.metallib` (the shader library), `<key>_b.json` (reflection used by replay: each binding's kind, slot number, index within the argument buffer and so on), `<key>_b.metal` (the MSL source). Some fragment shaders additionally have `<key>_bf.metallib`, a "readable tile" version used for pass merging (see the file header of `tools/dmsh_fetch.py`). `_b_build.json` in the directory records the success or failure of each key in this run. - **`dm/games/<id>/shader_map.tsv`**: the shader map, one row per shader variant. At run time only its "key → stage (vertex / fragment / compute)" is used. - **`dm/games/<id>/shader_facts.tsv`**: the answer table for translation queries. The translator needs to know some facts that cannot be read from the machine code (the dimension type of a texture, image formats, the component types of vertex attributes and fragment outputs, a compute shader's thread group size and shared memory byte count), and this table gives the answers by key. The table ships with the repository, and building the shader libraries only reads it; the table is maintained by `tools/dmsh_facts.py`, whose file header has the details. -- **Translation chain version and the `b-<V>/` directory**: the version V is a string of 12 hexadecimal digits, obtained by hashing together the contents of the source files involved in the translation chain, the translation options and the tool fingerprints (`tools/dmsh_chain.lock`); `tools/dmsh_chain.py` computes it. Shader libraries are written into `<output directory>/b-<V>/`, and every reflection json also records V. When the host is compiled it takes V from `dm/games/<id>/dmsh_chain.h`, loads only from `b-<V>/`, and checks the version in each json one by one; on a mismatch it reports red and does not load. So after changing any file on the chain, or changing a tool, the libraries must be rebuilt and the host recompiled. +- **Translation chain version and the `b-<V>/` directory**: the version V is a string of 12 hexadecimal digits, obtained by hashing together the contents of the source files involved in the translation chain, the translation options and the tool fingerprints (`tools/dmsh_chain.lock`); `tools/dmsh_chain.py` computes it. Shader libraries are written into `<output directory>/b-<V>/`, and every reflection json also records V. When the host is compiled it takes V from `nvn/dmsh_chain.h`, one version for every game, loads only from `b-<V>/`, and checks the version in each json one by one; on a mismatch it reports red and does not load. So after changing any file on the chain, or changing a tool, the libraries must be rebuilt and the host recompiled. A profile's inputs (its shader tables and `shader_*` / `nvn_*` keys) are not part of V: they are in each shader's cache key, so a change rebuilds the shaders it affects, and a full build stamps the directory with the game and their digest (`b-<V>/inputs.tsv`, `dmsh_chain.py inputs`). A full build starts from no shader outputs and a partial one from none of the keys it builds, so a directory holds only what its last build made; it refuses a directory stamped for another game, so each game needs its own output directory. - `dmsh_chain.py`: prints V. - `dmsh_chain.py --check-tools`: checks that the local tools match the lock file. - `dmsh_chain.py --update-lock`: rewrites the lock file according to the local tools. - - `dmsh_chain.py --header`: rewrites `dm/games/<id>/dmsh_chain.h`. + - `dmsh_chain.py --header`: rewrites `nvn/dmsh_chain.h` (commit it with the chain change; `dm/tools/tests/test_nsp_chain.py` fails on a stale one). + - `dmsh_chain.py --inputs`: prints the digest of the current profile's inputs (`DM_GAME`), as a full build stamps it. - It is normal for the local Xcode or spirv-cross version to differ from the one registered in the repository; `dmsh_chain.py --update-lock` re-registers the lock file with the local tools (commit it afterwards). - - `tools/own_compilers.py <dmsh_build.py arguments>` runs `dmsh_build.py` with the local Metal compiler and spirv-cross even when they differ from the lock, printing one `note:` line that names them; the translator must still match. Neither the lock nor V changes, so the committed `dmsh_chain.h` headers stay valid. The NSP script builds the iOS shaders this way, so a player with another Xcode or a newer spirv-cross needs no `--update-lock`. -- **`dm/games/<id>/c1_full.tsv`**: the index list for the shaders' built-in constant section (c1), generated by `tools/c1_guard.py` scanning the translated MSL; it lists the shader keys whose "run-time index could not be proven bounded". An index counts as proven only when interval arithmetic over the whole expression (every variable followed to its single definition; literals, `& K`, `% K`, shifts, `min`, `max`, `clamp` and casts bound it) keeps it non-negative and inside the span a shared bank gives that shader: its c1 data rounded up to 256 bytes plus the 256-byte zero gap. Every stage counts (`vp_`, `fp_`, `cp_` …). Its first line records the `b-<V>` it corresponds to. For the shaders on the list the replay layer allocates a separate whole 64 KiB buffer for c1; when the list is missing or its version does not match, it reports red and conservatively treats all shaders this way. The list is not committed (git ignores it): each checkout makes it from its own build's Mac outputs, and the NSP script makes it after its iOS shader build. -- **`dm/games/<id>/cb_extent.tsv`**: how far into each uniform slot (cbuf binding) every shader can read, generated by `tools/cb_extent.py` from the translated MSL with `c1_guard.py`'s interval arithmetic: per shader key, `reg:bytes` for each slot whose reads are all proven to stay inside its first bytes, and `s<reg>` for each storage slot the shader never writes (SPIRV-Cross types every storage buffer Ryujinx emits as writable, so the host counts the others as written). When the game updates part of a bound uniform block inside a render pass, the replay layer copies only that much of the binding to give the draws after the update their data without ending the pass (`nvn_mtl.m` `ubp_get`; Breath of the Wild binds a 64 KiB block and its vertex shaders read at most 544 bytes of it). Its first line records the `b-<V>` it corresponds to; when it is missing or stale, bindings are copied whole (up to 64 KiB). Like `c1_full.tsv`, it is not committed. + - `tools/own_compilers.py <dmsh_build.py arguments>` runs `dmsh_build.py` with the local Metal compiler and spirv-cross even when they differ from the lock, printing one `note:` line that names them; the translator must still match. Neither the lock nor V changes, so the committed `dmsh_chain.h` stays valid. The NSP script builds the iOS shaders this way, so a player with another Xcode or a newer spirv-cross needs no `--update-lock`. +- **`dm/games/<id>/c1_full.tsv`**: the index list for the shaders' built-in constant section (c1), generated by `tools/c1_guard.py` scanning the translated MSL; it lists the shader keys whose "run-time index could not be proven bounded". An index counts as proven only when interval arithmetic over the whole expression (every variable followed to its single definition; literals, `& K`, `% K`, shifts, `min`, `max`, `clamp` and casts bound it) keeps it non-negative and inside the span a shared bank gives that shader: its c1 data rounded up to 256 bytes plus the 256-byte zero gap. Every stage counts (`vp_`, `fp_`, `cp_` …). Its first line records the `b-<V>` and the inputs stamp of the library it was made from. For the shaders on the list the replay layer allocates a separate whole 64 KiB buffer for c1; when the list is missing or does not name the loaded library's version and stamp, it reports red and conservatively treats all shaders this way. The list is not committed (git ignores it): each checkout makes it from its own build's Mac outputs (a full `dmsh_build.py` run does). +- **`dm/games/<id>/cb_extent.tsv`**: how far into each uniform slot (cbuf binding) every shader can read, generated by `tools/cb_extent.py` from the translated MSL with `c1_guard.py`'s interval arithmetic: per shader key, `reg:bytes` for each slot whose reads are all proven to stay inside its first bytes, and `s<reg>` for each storage slot the shader never writes (SPIRV-Cross types every storage buffer Ryujinx emits as writable, so the host counts the others as written). When the game updates part of a bound uniform block inside a render pass, the replay layer copies only that much of the binding to give the draws after the update their data without ending the pass (`nvn_mtl.m` `ubp_get`; Breath of the Wild binds a 64 KiB block and its vertex shaders read at most 544 bytes of it). Its first line records the `b-<V>` and inputs stamp, like `c1_full.tsv`'s; when it is missing or stale, bindings are copied whole (up to 64 KiB). Like `c1_full.tsv`, it is not committed, and a full `dmsh_build.py` run makes it. - **`dm/games/<id>/shader_pairs.tsv`**: the (vertex key, pixel key) of every BNSH variation (Unity: of every program), written by `tools/shader_split.py`. Metal only builds a pipeline when every pixel shader input is a vertex shader output, while Maxwell gives an unwritten attribute a default; `dmsh_build.py` therefore adds the outputs a vertex shader's partners read but it doesn't write, set to (0, 0, 0, 1) (`vs_pad`). The table is part of the translation chain version. - **Several game versions**: `shader_split.py` takes comma-separated RomFS and `main` lists (base game first) and writes the union; the map, facts and pairs tables currently cover Mario v0 and 1.2.1. - **`dm/games/<id>/pso_manifest.tsv`**: the warm-up seed manifest bundled with the app: shaders (`S`), render pipelines (`R`, the hex of `pso_skey` in `nvn/nvn_pso_skey.h`) and compute pipelines (`C`) to pre-build at startup. The row order is the pre-build order (first use); do not sort it. Its companion is `pso_user.tsv`, which the host appends next to the save directory whenever it has to build something on the spot, and which is pre-built after the seed at the next launch. Both name shaders by key and describe pipeline settings only, so a list recorded on one device works on any other; lines a build cannot use are skipped. Recorded lines can be appended to the seed below its header (currently empty). @@ -77,7 +78,7 @@ Binding conventions followed during translation (the NVN driver's layout, the sa | `nvn/nvn_hmap.h`, `nvn/nvn_cover.h`, `nvn/nvn_pace.h`, `nvn/nvn_pso_skey.h` | Hash table, geometric proof of full-coverage draws, presentation pacer, pipeline manifest key | | `nvn/nvn_builtin.h` | Hand-written MSL for the 4 compute shaders embedded in the main program | | `nvn/include/nvn.h` | NVN interface declarations (third party; for the license see the file header) | -| `nvn/names.txt` | The NVN API name list. The per-game tables described above and `dmsh_chain.h` live in the game profile `dm/games/<id>/` (named after the base game's title ID; tools pick it with `DM_GAME`, make with `GAME`; there is no default game) | +| `nvn/names.txt` | The NVN API name list. The per-game tables described above live in the game profile `dm/games/<id>/` (named after the base game's title ID; tools pick it with `DM_GAME`, make with `GAME`; there is no default game) | | `tex/` | Texture format mapping, deswizzling, BC decompression, see [tex/README.md](tex/README.md) | | `vk/hle_vk.m` | Vulkan on Metal, for games that render with Vulkan (Hades, Super Mario Galaxy); their SPIR-V shaders are translated by `tools/vk_shaders.py` | | `gl/hle_gl.m` | OpenGL through EGL on Metal, for games that render with NVIDIA's GL driver (Digimon Story Cyber Sleuth: Complete Edition); their GLSL sources are translated by `tools/gl_shaders.py` (glslang, then SPIRV-Cross) | @@ -111,9 +112,9 @@ The shader libraries are generated by `tools/dmsh_build.py` (the NSP script runs - the Mac version is in `$DM_ROOT/build/cache/metal_shader_cache/b-<V>/`; - the iOS version is in `$DM_ROOT/build/ios/dmsh_b/b-<V>/`. -After changing a file on the translation chain, the order for rebuilding manually is: `dmsh_chain.py --header` → `dmsh_build.py` → `c1_guard.py` and `cb_extent.py` (local tables, not committed) → recompile the host. +After changing a file on the translation chain, the order for rebuilding manually is: `dmsh_chain.py --header` (once, for every game) → `dmsh_build.py` (a full build also makes `c1_full.tsv` and `cb_extent.tsv`, local tables, not committed) → recompile the host. -The per-shader cache key in `dmsh_build.py` (`shader_key`) covers the profile's `shader_*` rows and the shader's pairs, and a paired vertex shader's key also takes its pixel shaders' keys (`pair_keyed`), since `vs_pad` reads their MSL. Before, these were only in the chain version, so a build after changing only them hit the old per-shader entries and filed them under the new version. `$DM_ROOT` defaults to `~/dm-work`. +The per-shader cache key in `dmsh_build.py` (`shader_key`) covers the profile's `shader_*` rows and the shader's pairs, and a paired vertex shader's key also takes its pixel shaders' keys (`pair_keyed`), since `vs_pad` reads their MSL. Before, these were only in the chain version, so a build after changing only them hit the old per-shader entries and filed them under the new version; now the chain version leaves them out altogether. `$DM_ROOT` defaults to `~/dm-work`. ## Tests diff --git a/dm/gfx/nvn/dmsh_chain.h b/dm/gfx/nvn/dmsh_chain.h new file mode 100644 index 0000000..ba7a955 --- /dev/null +++ b/dm/gfx/nvn/dmsh_chain.h @@ -0,0 +1,3 @@ +// Generated file (dm/gfx/tools/dmsh_chain.py --header): the B shader library translation chain version this source tree expects (default options), one for every game; +// the host loads only <library dir>/b-<V>/. Run dmsh_chain.py --header after changing the chain's tools and commit this file with the change +#define DMSH_CHAIN "3f4cd4aacd1d" diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index fc4c997..6d26f8d 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -23,7 +23,7 @@ #include "nvn_dirty.h" #include "nvn_refl.h" #include "nvn_cover.h" -#include "dmsh_chain.h" // DMSH_CHAIN: the translation chain version of the B shader library expected by this source tree (in the game profile folder, -I from the Makefile) +#include "dmsh_chain.h" // DMSH_CHAIN: the translation chain version of the B shader library expected by this source tree, one for every game (beside this file, tools/dmsh_chain.py --header) #include "game_profile.h" // game profile constants (profile id, the profile's switches) // Texture slot table per stage: 0-127 the combined handles (nvnCommandBufferBindTexture), 128-255 the separate texture binds (NVN_SEPTEX_SLOT0, nvn_obj.h) #define NTEX 256 @@ -2769,15 +2769,25 @@ static void shmap_load(void) { } g_shmap_path = path; const char *m = getenv("DM_MSL_DIR"); - // Library directory = <DM_MSL_DIR or default cache>/b-<translation chain version>: the version comes from the game profile's dmsh_chain.h (tools/dmsh_chain.py); a host built from an old source tree only accepts libraries of the old version + // Library directory = <DM_MSL_DIR or default cache>/b-<translation chain version>: the version comes from dm/gfx/nvn/dmsh_chain.h (tools/dmsh_chain.py), one for every game; a host built from an old source tree only accepts libraries of the old version g_msl_dir = [NSString stringWithFormat:@"%@/b-%s", m ? @(m) : [NSString stringWithFormat:@"%s%s/build/cache/metal_shader_cache", getenv("DM_ROOT") ?: getenv("HOME"), getenv("DM_ROOT") ? "" : "/dm-work"], DMSH_CHAIN]; if (![NSFileManager.defaultManager fileExistsAtPath:g_msl_dir]) MLOG("[RED] shader library directory %s does not exist: translation chain version %s of this source tree has not been built yet (dm/gfx/tools/dmsh_build.py)", g_msl_dir.UTF8String, DMSH_CHAIN); + // The library's stamp (inputs.tsv, written by a full dmsh_build.py run): the game and the digest of the profile inputs it was built from. The chain version is the same for every + // game and every state of a profile's tables, so the stamp is what ties c1_full.tsv / cb_extent.tsv to these libraries: each names the inputs of the library it was made from + NSString *stamp = [NSString stringWithContentsOfFile:[g_msl_dir stringByAppendingPathComponent:@"inputs.tsv"] encoding:NSUTF8StringEncoding error:nil], *st_game = nil, *st_inputs = nil; + for (NSString *ln in [stamp componentsSeparatedByString:@"\n"]) { + NSArray *c = [ln componentsSeparatedByString:@"\t"]; + if (c.count == 2 && [c[0] isEqual:@"game"]) st_game = c[1]; else if (c.count == 2 && [c[0] isEqual:@"inputs"]) st_inputs = c[1]; + } + if (!st_inputs) MLOG("[RED] shader library %s has no stamp (inputs.tsv): no full dmsh_build.py run made it as it is, so c1_full.tsv and cb_extent.tsv are not taken", g_msl_dir.UTF8String); + else if (st_game && ![st_game isEqual:@DM_GAME_ID]) MLOG("[RED] shader library %s was built for game %s, not " DM_GAME_ID, g_msl_dir.UTF8String, st_game.UTF8String); + else MLOG("shader library stamp: game %s, inputs %s", st_game.UTF8String, st_inputs.UTF8String); NSString *cp = [[path stringByDeletingLastPathComponent] stringByAppendingPathComponent:@"c1_full.tsv"]; // Same directory as the mapping table (Mac: dm/games/<id>; iOS: dm/ inside the App, copied along by build_ipa.sh) NSString *ct = cp ? [NSString stringWithContentsOfFile:cp encoding:NSUTF8StringEncoding error:nil] : nil; NSString *head = ct ? [[ct componentsSeparatedByString:@"\n"] firstObject] : nil; - NSString *want = [NSString stringWithFormat:@"# chain b-%s", DMSH_CHAIN]; + NSString *want = [NSString stringWithFormat:@"# chain b-%s inputs %@ ", DMSH_CHAIN, st_inputs ?: @"-"]; // a library without a stamp matches no table if (!head || ![head hasPrefix:want]) { - MLOG("[RED] c1_full.tsv is stale or missing; every c1 gets its own 64 KiB"); + MLOG("[RED] c1_full.tsv is stale or missing (its header must name this library's chain and inputs; a full dmsh_build.py run makes it); every c1 gets its own 64 KiB"); } else { g_c1_full_valid = 1; g_c1_full = [NSMutableSet set]; for (NSString *line in [ct componentsSeparatedByString:@"\n"]) if (line.length && ![line hasPrefix:@"#"]) [g_c1_full addObject:[line stringByTrimmingCharactersInSet:NSCharacterSet.whitespaceAndNewlineCharacterSet]]; diff --git a/dm/gfx/tests/Makefile b/dm/gfx/tests/Makefile index 7f6fc71..94f1f33 100644 --- a/dm/gfx/tests/Makefile +++ b/dm/gfx/tests/Makefile @@ -1,6 +1,6 @@ # Unit tests: make compiles and runs them (seconds) # Source files are included by wildcard, same convention as GFX in the host Makefile (hle_nvn.c is #included directly by t_nvn.c, so it is excluded); adding new source files needs no change here -# Game profile (same convention as the host Makefile: GAME=base game title ID, required: there is no default game): game_profile.h is generated by the host Makefile, dmsh_chain.h lives in the profile folder +# Game profile (same convention as the host Makefile: GAME=base game title ID, required: there is no default game): game_profile.h is generated by the host Makefile; dmsh_chain.h (one for every game) lives beside nvn_mtl.m ifeq ($(strip $(GAME)),) ifneq ($(filter-out clean,$(or $(MAKECMDGOALS),t_nvn)),) $(error GAME is not set: make GAME=<base game title ID>: $(shell python3 ../../games/game.py list | awk -F'\t' '{printf "%s (%s) ", $$1, $$3}')) @@ -8,9 +8,9 @@ endif endif GAME_DIR = ../../games/$(GAME) GAME_H = ../../host/gen/$(GAME)/game_profile.h -CFLAGS=-O1 -g -Wall -Wno-unused-function -I../nvn/include -I$(dir $(GAME_H)) -I$(GAME_DIR) +CFLAGS=-O1 -g -Wall -Wno-unused-function -I../nvn/include -I$(dir $(GAME_H)) GFX=$(filter-out ../nvn/hle_nvn.c,$(sort $(wildcard ../nvn/*.c ../nvn/*.m ../nvn/*.S ../tex/*.c ../tex/*.m))) -GFX_H=$(wildcard ../nvn/*.h ../nvn/*.inc ../nvn/include/*.h ../tex/*.h) $(GAME_H) $(GAME_DIR)/dmsh_chain.h +GFX_H=$(wildcard ../nvn/*.h ../nvn/*.inc ../nvn/include/*.h ../tex/*.h) $(GAME_H) .DEFAULT_GOAL := t_nvn # the profile header rule comes first, so keep the default goal as before $(GAME_H): $(MAKE) -C ../../host game_profile GAME=$(GAME) @@ -33,8 +33,8 @@ t_tbuf2d: t_tbuf2d.m ../nvn/nvn_tex.c ../nvn/nvn_tex.h clang -O1 -g -Wall -fobjc-arc -o $@ t_tbuf2d.m ../nvn/nvn_tex.c -framework Metal -framework Foundation t_deswz_gpu: t_deswz_gpu.m ../tex/nvn_deswizzle.c ../tex/nvn_deswizzle_gpu.m ../tex/nvn_deswizzle.h ../tex/nvn_deswizzle_gpu.h ../nvn/nvn_tex.c ../nvn/nvn_tex.h clang $(CFLAGS) -fobjc-arc -o $@ t_deswz_gpu.m ../tex/nvn_deswizzle.c ../tex/nvn_deswizzle_gpu.m ../nvn/nvn_tex.c -framework Metal -framework Foundation -t_refl: t_refl.m ../nvn/nvn_refl.m ../nvn/nvn_refl.h $(GAME_DIR)/dmsh_chain.h - clang -O1 -g -Wall -I$(GAME_DIR) -fobjc-arc -o $@ t_refl.m ../nvn/nvn_refl.m -framework Metal -framework Foundation +t_refl: t_refl.m ../nvn/nvn_refl.m ../nvn/nvn_refl.h ../nvn/dmsh_chain.h + clang -O1 -g -Wall -I../nvn -fobjc-arc -o $@ t_refl.m ../nvn/nvn_refl.m -framework Metal -framework Foundation t_hmap: t_hmap.c ../nvn/nvn_hmap.h clang -O1 -Wall -o $@ t_hmap.c t_cover: t_cover.c ../nvn/nvn_cover.h diff --git a/dm/gfx/tests/t_refl.m b/dm/gfx/tests/t_refl.m index fc6a1af..e7ddc26 100644 --- a/dm/gfx/tests/t_refl.m +++ b/dm/gfx/tests/t_refl.m @@ -4,7 +4,7 @@ #import <Foundation/Foundation.h> #include <stdio.h> #include <string.h> -#include "dmsh_chain.h" // the game profile folder (-I from the Makefile) +#include "dmsh_chain.h" // dm/gfx/nvn (-I from the Makefile) enum { C = NVN_RB_CBUF, S = NVN_RB_SAMP, T = NVN_RB_TEX, I = NVN_RB_IMG, TB = NVN_RB_TBUF }; enum { F = NVN_DPF_FLOAT, U = NVN_DPF_UINT, D = NVN_DPF_DEPTH }; diff --git a/dm/gfx/tools/c1_guard.py b/dm/gfx/tools/c1_guard.py index f0f9339..e7a50ab 100755 --- a/dm/gfx/tools/c1_guard.py +++ b/dm/gfx/tools/c1_guard.py @@ -1,7 +1,8 @@ #!/usr/bin/env python3 # c1_guard.py <dmsh_b version dir> <output tsv>: lists the shaders whose reads of c1 (the shader's embedded constant segment) are not proven to stay inside the span the host's # shared c1 banks give them. nvn_mtl.m c1_store packs the c1 of unlisted shaders into 4 MB banks: the data rounded up to 256 bytes, then a 256-byte zero gap, then the next shader's data. -# A listed shader gets an exclusive 64 KiB of zeros instead (reads past the data return 0, as on the Switch). The host loads the list only when its header names its chain version. +# A listed shader gets an exclusive 64 KiB of zeros instead (reads past the data return 0, as on the Switch). The host loads the list only when its header names its library's chain version +# and inputs stamp (dmsh_chain.py table_head). dmsh_build.py runs it after each full build. # An index is proven when interval arithmetic over the whole expression, following each variable's definition, gives 0 <= index and 16 * (index + 1) <= that span (_m0 is float4[4096]). # Literals, & K, % K, >> K, min, max, clamp and casts bound a value; inputs, buffer reads and anything not understood are unbounded. Every stage's c1 counts (vp_, fp_, cp_, gp_ ...). # The first version followed only the last variable of an index and took any mask, min or clamp it met as proof: `_1 + (_2 & 3u)` with _1 read from an input passed, @@ -12,6 +13,9 @@ import os import re import sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import dmsh_chain # noqa: E402 + ACCESS = re.compile(r"\b[a-z]+_c1\)\._m0\[([^\]]+)\]") DEFINITION = re.compile(r"\b(u?int|u?short|u?long|bool)\s+(_\d+)\s*=\s*([^;]+);") @@ -218,7 +222,7 @@ def main(): os.makedirs(os.path.dirname(output), exist_ok=True) with open(output, "w", encoding="utf-8", newline="") as stream: stamp = datetime.datetime.now().astimezone().isoformat(timespec="seconds") - stream.write(f"# chain {os.path.basename(version_dir)} {stamp}\n") + stream.write(f"{dmsh_chain.table_head(version_dir)} {stamp}\n") # the library's chain version and the profile inputs it was built from for key in keys: stream.write(key + "\n") print(f"not proven bounded: {len(keys)} shaders") diff --git a/dm/gfx/tools/cb_extent.py b/dm/gfx/tools/cb_extent.py index f591f13..8ac5274 100644 --- a/dm/gfx/tools/cb_extent.py +++ b/dm/gfx/tools/cb_extent.py @@ -5,7 +5,7 @@ # `s<reg>` marks a storage slot the shader never writes (unwritten); the host notes GPU writes for every other storage binding (nvn_mtl.m sbw_note). # The host (nvn_mtl.m cb_ext) reads only that much of a binding when it has to copy one: a uniform block the game updates in part inside a render pass (ubp_get). Breath of the Wild binds # a 64 KiB block to the vertex stage and updates its first 64 bytes 221 times a frame, while its vertex shaders read at most 544 bytes of it. -# The host loads the table only when its header names its chain version (like c1_full.tsv); regenerate it after each shader build +# The host loads the table only when its header names its library's chain version and inputs stamp (like c1_full.tsv); dmsh_build.py makes it after each full build import datetime import json import multiprocessing @@ -15,6 +15,7 @@ import sys sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) from c1_guard import Interval, definitions_of, known # noqa: E402 +import dmsh_chain # noqa: E402 MAX = 65536 @@ -89,7 +90,7 @@ def main(): os.makedirs(os.path.dirname(output), exist_ok=True) with open(output, "w", encoding="utf-8", newline="") as stream: stamp = datetime.datetime.now().astimezone().isoformat(timespec="seconds") - stream.write(f"# chain {os.path.basename(version_dir)} {stamp}\n") + stream.write(f"{dmsh_chain.table_head(version_dir)} {stamp}\n") # the library's chain version and the profile inputs it was built from stream.write("# shader key, then reg:bytes for each uniform slot whose reads stay inside its first bytes, and s<reg> for each storage slot it never writes (cb_extent.py)\n") for line in rows: stream.write(line + "\n") print(f"bounded uniform reads: {slots} slots; storage slots never written: {ro}; {len(rows)} shaders") diff --git a/dm/gfx/tools/dmsh_build.py b/dm/gfx/tools/dmsh_build.py index e2b217a..cc1cd73 100644 --- a/dm/gfx/tools/dmsh_build.py +++ b/dm/gfx/tools/dmsh_build.py @@ -2,7 +2,9 @@ # Route B offline build: original Switch shaders → SPIR-V (dmsh_xlate, the Ryujinx translator) → spirv-cross MSL → metallib + replay reflection JSON. # No translation at runtime (iOS has no JIT): build output goes into the shader cache directory, file name = Switch control segment key (dm_ctl_sha16, same as column 4 of shader_map.tsv) + "_b". # Source of query answers: the answer table in the game profile, dm/games/<id>/shader_facts.tsv (dmsh_facts.py) - texture types, vertex attribute / fragment output component types, compute thread group size and shared memory. -# Usage: dmsh_build.py <Switch package dir> <output cache dir> [key...] (no keys = all); build output is actually written to <output cache dir>/b-<chain version>/ (same for IOSOUT), for the chain version see dmsh_chain.py Environment: DM_GAME=game profile (required, dm/games/<id>), XLATE=path to the translation tool (default the in-repo dmsh_xlate_bin), DMSH_CAS=content-addressed cache dir (default <output cache dir>/cas), DMSH_PREFILL=Mac library root[=iOS library root] (separated by :, their b-*/ go into the MSL-keyed compile cache first; sibling b-*/ always do), J=concurrency +# Usage: dmsh_build.py <Switch package dir> <output cache dir> [key...] (no keys = all); build output is actually written to <output cache dir>/b-<chain version>/ (same for IOSOUT), for the chain version see dmsh_chain.py. +# A full build (no keys) also stamps both directories with the game and the digest of its profile inputs (inputs.tsv) and writes the profile's c1_full.tsv and cb_extent.tsv (c1_guard.py, cb_extent.py) +# from its Mac outputs. One output directory per game: the chain version is the same for every game Environment: DM_GAME=game profile (required, dm/games/<id>), XLATE=path to the translation tool (default the in-repo dmsh_xlate_bin), DMSH_CAS=content-addressed cache dir (default <output cache dir>/cas), DMSH_PREFILL=Mac library root[=iOS library root] (separated by :, their b-*/ go into the MSL-keyed compile cache first; sibling b-*/ always do), J=concurrency # Build output: <key>_b.metallib, <key>_b.json (replay reflection, including drv_reads, the runtime read offsets into driver segments c0 / c2), <key>_b.metal (MSL) # Environment IOSOUT=dir: the same MSL also goes through dm/ios/tools/ios_metal.compile to produce an iphoneos metallib + reflection (chain_metal records the iOS compile convention) # Final summary: success / failure counts and the distribution of reasons; distribution of driver segment read offsets (those outside the range the replay layer can fill are listed separately) @@ -15,8 +17,9 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))); import dmsh_chai _bad = dmsh_chain.check_tools() if _bad: sys.exit('\n'.join('[RED] translation chain tools do not match the lock file: ' + b for b in _bad) + '\n(if the tools changed, run dmsh_chain.py --update-lock and commit the lock file; if the translator just was not rebuilt, rebuild it)') CHAIN = dmsh_chain.version(); out = os.path.join(out, 'b-' + CHAIN) -GAME = dmsh_chain.GAME # the current game profile (DM_GAME): the tables come from here -if f'"{CHAIN}"' not in open(dmsh_chain.HEADER).read(): sys.exit(f'[RED] {dmsh_chain.HEADER} does not hold the current translation chain version {CHAIN}: run dmsh_chain.py --header first (otherwise the host cannot find these libraries)') +GAME = dmsh_chain.load_game() # the current game profile (DM_GAME): the tables come from here +if f'"{CHAIN}"' not in open(dmsh_chain.HEADER).read(): sys.exit(f'[RED] {dmsh_chain.HEADER} does not hold the current translation chain version {CHAIN}: run dmsh_chain.py --header first and commit it (otherwise the host cannot find these libraries)') +INPUTS = dmsh_chain.inputs(GAME) # the digest of the profile's inputs, which a full build stamps on its libraries (b-<V>/inputs.tsv) XLATE = os.environ.get('XLATE', os.path.join(dmsh_chain.XLATE_DIR, 'dmsh_xlate')) FACTS = dmsh_facts.load() if not os.path.exists(dmsh_facts.TABLE): sys.exit(f'[RED] no answer table {dmsh_facts.TABLE} (it comes with the game profile; an empty table = every query answered with defaults)') @@ -24,6 +27,9 @@ IOSOUT = os.environ.get('IOSOUT') # if set, also build the iOS version (iphone if IOSOUT: sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '../../ios/tools')); import ios_metal IOSOUT = os.path.join(IOSOUT, 'b-' + CHAIN); os.makedirs(IOSOUT, exist_ok=True) +for _d in [out] + ([IOSOUT] if IOSOUT else []): # one chain version serves every game, so b-<V>/ alone does not keep two games' libraries apart + _g = dmsh_chain.read_stamp(_d).get('game') + if _g and _g != GAME.id: sys.exit(f'[RED] {_d} holds the shader libraries of game {_g}, not {GAME.id}: give each game its own output directory') MAP = str(GAME.table('shader_map.tsv')) BF_WANT_FILE = str(GAME.table('bf_want.tsv')) BF_WANT = {line.strip() for line in open(BF_WANT_FILE) if line.strip() and not line.lstrip().startswith('#')} if os.path.exists(BF_WANT_FILE) else set() @@ -87,8 +93,8 @@ def tool_fingerprint(): '\0xlate\0' + dmsh_chain.xlate_source_sha() + '\0spirv-cross\0' + sc.stdout + sc.stderr + '\0xc\0' + repr(XC) + '\0mac\0' + MAC_RECORD + '\0ios\0' + (ios_metal.RECORD if IOSOUT else '')) -# The profile keys that change how a shader is built (nvn_*, shader_*), as dmsh_chain.version takes them: part of every shader's cache key -PROFILE_ROWS = ''.join(f'{k}\t{chr(9).join(v)}\n' for k, v in GAME.rows if k.startswith(('nvn_', 'shader_'))).encode() +# The profile keys that change how a shader is built (nvn_*, shader_*), as dmsh_chain.inputs takes them: part of every shader's cache key +PROFILE_ROWS = dmsh_chain.profile_rows(GAME) def shader_key(key, name, rec): """The per-shader cache key. It covers the profile's build keys and the shader's pairs; a vertex shader with pairs also takes its pixel shaders' keys (pair_keyed), since vs_pad reads their MSL. Before, these were only in the chain version, so a build after changing only them hit the old entries and filed them under the new version (with an empty DMSH_CAS as the workaround)""" @@ -856,6 +862,16 @@ def finish(key, w, j, c1): keys = sorted(want) if want else sorted(loc) NJ = int(os.environ.get('J', '8')) os.makedirs(out, exist_ok=True) +# A full build starts from no shader outputs, a partial one from none of the keys it builds, so what a run leaves is what it built: every output is restored from the cache anyway. +# The chain version is the same whatever the profile's tables, so a key that left the map, a shader that no longer builds or a variant no longer made would keep an old library. +# The stamp goes too, and a full build writes it at the end; a partial build keeps it only while it names the profile's current inputs +OUT_KEY = re.compile(r'([0-9a-f]{16})_b') +STAMP_BEFORE = dmsh_chain.read_stamp(out).get('inputs') +for _d in [out] + ([IOSOUT] if IOSOUT else []): + for _n in os.listdir(_d): + _m = OUT_KEY.match(_n) + if _m and (not want or _m.group(1) in want): os.unlink(os.path.join(_d, _n)) + if (not want or STAMP_BEFORE != INPUTS) and os.path.exists(os.path.join(_d, dmsh_chain.STAMP)): os.unlink(os.path.join(_d, dmsh_chain.STAMP)) w = tempfile.mkdtemp(prefix='dmsh_', dir=os.environ.get('WORK')) MTL = dmsh_mtlcache.Cache(os.path.join(CAS, 'metallib'), {'mac': ('macosx', MAC_RECORD, mac_compile), **({'ios': ('iphoneos', ios_metal.RECORD, ios_metal.compile)} if IOSOUT else {})}, w) # first take the libraries earlier chain versions left on disk: the b-*/ beside the output (iOS: the folder of the same name) and the roots DMSH_PREFILL lists @@ -933,3 +949,11 @@ for e, n in collections.Counter(f"{b['kind']} {b['msl_name'][:28]}" for x in nor nt = collections.Counter(nn.split(' (')[0] for x in ok for nn in x['notes']) for e, n in nt.most_common(8): print(f' queries without an answer {n:5d} {e}') json.dump([{'key': r[0], 'st': r[1], 'err': r[2]} for r in res], open(os.path.join(out, '_b_build.json'), 'w'), ensure_ascii=False) +if not want: # a full build: the stamp, then the c1 guard and uniform read extent tables from these Mac outputs, into the profile folder (the host takes them only beside this stamp) + for _d in [out] + ([IOSOUT] if IOSOUT else []): dmsh_chain.write_stamp(_d, GAME.id, INPUTS) + print(f'library stamp: game {GAME.id}, inputs {INPUTS}', flush=True) + for _tool, _table in (('c1_guard.py', 'c1_full.tsv'), ('cb_extent.py', 'cb_extent.tsv')): + _r = subprocess.run([sys.executable, os.path.join(os.path.dirname(os.path.abspath(__file__)), _tool), out, str(GAME.table(_table))], capture_output=True, text=True) + if _r.returncode: sys.exit(f'[RED] {_tool} failed ({_r.returncode}): {_r.stderr.strip()[-500:]}') + print(f'{_table}: {(_r.stdout.strip().splitlines() or [""])[-1]}', flush=True) +elif STAMP_BEFORE != INPUTS: print(f'no library stamp: the profile\'s inputs changed since the last full build ({STAMP_BEFORE} → {INPUTS}), and a partial build does not rebuild the rest', flush=True) diff --git a/dm/gfx/tools/dmsh_chain.py b/dm/gfx/tools/dmsh_chain.py index 636f51a..99455e3 100644 --- a/dm/gfx/tools/dmsh_chain.py +++ b/dm/gfx/tools/dmsh_chain.py @@ -1,22 +1,24 @@ #!/usr/bin/env python3 -# Translation chain version of the B shader library (the build result must be determined by source alone). -# Chain version V = first 12 digits of the sha256 of the contents of the following in-repo files + the translation options: this script, dmsh_build.py, dmsh_fetch.py (readable-tile version), dmsh_vin.py (shadow vertex inputs), the dmsh_xlate sources (Program.cs, csproj), -# from the current game profile (dm/games/<id>/, chosen by DM_GAME) the mapping table (including the conflict table), the vertex / pixel pair table (shader_pairs.tsv, used by vs_pad), the answer table for translation queries (dmsh_facts.py and shader_facts.tsv), -# the keys of the profile's game.tsv that affect the build (nvn_*, shader_*), the iOS compile convention ios_metal.py, and the tool lock dmsh_chain.lock (measured fingerprints of the Ryujinx shader dll / spirv-cross / metal compiler). The tables differ per game, so V does too. -# The build (dmsh_build.py) writes into <cache dir>/b-<V>/ and the reflection json records chain = V; at compile time the host takes V from the game profile's dmsh_chain.h, loads only from b-<V>/ and verifies each json. +# Translation chain version of the B shader library: one version for every game, which the host is compiled with (dm/gfx/nvn/dmsh_chain.h). The build result must be determined by source alone. +# Chain version V = first 12 digits of the sha256 of the contents of the following in-repo files + the translation options: this script, dmsh_build.py, dmsh_fetch.py (readable-tile version), dmsh_vin.py (shadow vertex inputs), +# dmsh_geometry.py, dmsh_facts.py, the dmsh_xlate sources (Program.cs, csproj), the iOS compile convention ios_metal.py, and the tool lock dmsh_chain.lock (measured fingerprints of the Ryujinx shader dll / spirv-cross / metal compiler). +# A game's own inputs are not part of V: its profile's shader tables and nvn_* / shader_* keys are in each shader's cache key (dmsh_build.py shader_key), so a change rebuilds the shaders it affects, +# and their digest (inputs below) stamps the library: <cache dir>/b-<V>/inputs.tsv says which game and which inputs it was built from. c1_full.tsv / cb_extent.tsv name V and that digest in their +# header (table_head), and the host takes them only when both match its library; build_ipa.sh packs a library only when its digest is the profile's. +# The build (dmsh_build.py) writes into <cache dir>/b-<V>/ and the reflection json records chain = V; at compile time the host takes V from dm/gfx/nvn/dmsh_chain.h, loads only from b-<V>/ and verifies each json. # Usage: dmsh_chain.py print V (default options) -# dmsh_chain.py --header rewrite the game profile's dmsh_chain.h (run after changing any file listed above) +# dmsh_chain.py --header rewrite dm/gfx/nvn/dmsh_chain.h (run after changing any file listed above, and commit it with the change) +# dmsh_chain.py --inputs print the digest of the current game profile's inputs (DM_GAME), as b-<V>/inputs.tsv records it # dmsh_chain.py --check-tools verify that the actual tools match the lock file; on mismatch print [RED] lines and exit 1 # dmsh_chain.py --update-lock rewrite the lock file from the actual tools (run after changing Ryujinx / spirv-cross / Xcode; committed to the repo) # Environment: XLATE_DEBUG / XLATE_REDUCED (same meaning as in dmsh_build.py, default 1 / 1; non-default options yield a different V, which the host will not load) import os, sys, hashlib, subprocess DM_ROOT = os.environ.get('DM_ROOT') or os.path.expanduser('~/dm-work') # Workspace root (game files, build outputs and run artifacts are all under it) HERE = os.path.dirname(os.path.abspath(__file__)); GFX = os.path.dirname(HERE); HOME = os.path.expanduser('~') -sys.path.insert(0, os.path.join(GFX, '../games')); import game as _game # game profiles -GAME = _game.load() FILES = ['tools/dmsh_chain.py', 'tools/dmsh_build.py', 'tools/dmsh_fetch.py', 'tools/dmsh_vin.py', 'tools/dmsh_geometry.py', 'tools/dmsh_facts.py', 'tools/dmsh_xlate/Program.cs', 'tools/dmsh_xlate/dmsh_xlate.csproj', '../ios/tools/ios_metal.py', 'tools/dmsh_chain.lock'] -TABLES = ['shader_facts.tsv', 'shader_map.tsv', 'shader_map.tsv.conflicts.tsv', 'shader_pairs.tsv'] # in the game profile folder -LOCK = os.path.join(HERE, 'dmsh_chain.lock'); HEADER = str(GAME.table('dmsh_chain.h')) +INPUT_TABLES = ['shader_facts.tsv', 'shader_map.tsv', 'shader_map.tsv.conflicts.tsv', 'shader_pairs.tsv', 'bf_want.tsv'] # in the game profile folder +LOCK = os.path.join(HERE, 'dmsh_chain.lock'); HEADER = os.path.join(GFX, 'nvn/dmsh_chain.h') +STAMP = 'inputs.tsv' # in the library directory b-<V>/ # the translator defaults to the in-repo build that matches the lock file (a rebuilt dll has different fingerprints) XLATE_DIR = os.path.dirname(os.environ.get('XLATE', os.path.join(HERE, 'dmsh_xlate_bin/dmsh_xlate'))) @@ -27,12 +29,37 @@ def version(): h = hashlib.sha256() for f in FILES: h.update(f.encode() + b'\0' + open(os.path.join(GFX, f), 'rb').read() + b'\0') - for f in TABLES: - h.update(f'game/{f}'.encode() + b'\0' + GAME.table(f).read_bytes() + b'\0') - h.update(''.join(f'{k}\t{chr(9).join(v)}\n' for k, v in GAME.rows if k.startswith(('nvn_', 'shader_'))).encode()) # the profile keys that affect the build h.update(options().encode()) return h.hexdigest()[:12] +def load_game(): # the current game profile (DM_GAME), only for what needs one: the version itself does not + sys.path.insert(0, os.path.join(GFX, '../games')); import game + return game.load() + +def profile_rows(g): # the profile keys that change how a shader is built (nvn_*, shader_*) + return ''.join(f'{k}\t{chr(9).join(v)}\n' for k, v in g.rows if k.startswith(('nvn_', 'shader_'))).encode() + +def inputs(g): + """Digest of a game profile's inputs to its shader library: the shader tables (a missing one counts as such) and the nvn_* / shader_* keys""" + h = hashlib.sha256() + for f in INPUT_TABLES: + p = g.table(f) + h.update(f'game/{f}'.encode() + b'\0' + (p.read_bytes() if p.exists() else b'missing') + b'\0') + h.update(profile_rows(g)) + return h.hexdigest()[:12] + +def read_stamp(lib): # {'game': title ID, 'inputs': digest} of a library directory; {} when it has none + try: return dict(l.rstrip('\n').split('\t', 1) for l in open(os.path.join(lib, STAMP)) if '\t' in l and not l.startswith('#')) + except OSError: return {} + +def write_stamp(lib, game_id, digest): + tmp = os.path.join(lib, STAMP + '.tmp') + with open(tmp, 'w') as f: f.write(f'# the game and the profile inputs this library was built from (dm/gfx/tools/dmsh_chain.py inputs)\ngame\t{game_id}\ninputs\t{digest}\n') + os.replace(tmp, os.path.join(lib, STAMP)) + +def table_head(lib): # the header c1_full.tsv / cb_extent.tsv start with: the library's chain version and its inputs ("none" when it has no stamp, which nothing takes) + return f'# chain {os.path.basename(os.path.normpath(lib))} inputs {read_stamp(lib).get("inputs", "none")}' + def sha16(p): return hashlib.sha256(open(p, 'rb').read()).hexdigest()[:16] def tools_now(): @@ -56,6 +83,11 @@ def check_tools(): if now['xlate_source'] != xlate_source_sha(): bad.append(f'the translator was not built from the in-repo source: embedded source {now["xlate_source"]} / in repo {xlate_source_sha()} (rebuild dmsh_xlate)') return bad +def header_text(v): + return (f'// Generated file (dm/gfx/tools/dmsh_chain.py --header): the B shader library translation chain version this source tree expects (default options), one for every game;\n' + f'// the host loads only <library dir>/b-<V>/. Run dmsh_chain.py --header after changing the chain\'s tools and commit this file with the change\n' + f'#define DMSH_CHAIN "{v}"\n') + if __name__ == '__main__': a = sys.argv[1:] if a == ['--update-lock']: @@ -68,8 +100,8 @@ if __name__ == '__main__': for b in bad: print('[RED] translation chain tools do not match the lock file: ' + b) sys.exit(1 if bad else 0) elif a == ['--header']: - open(HEADER, 'w').write(f'// Generated file (dm/gfx/tools/dmsh_chain.py --header, game {GAME.id}): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/\n' - f'#define DMSH_CHAIN "{version()}"\n') + open(HEADER, 'w').write(header_text(version())) print(version()) + elif a == ['--inputs']: print(inputs(load_game())) elif not a: print(version()) else: sys.exit(__doc__ if __doc__ else 'usage: see the file header') diff --git a/dm/gfx/tools/own_compilers.py b/dm/gfx/tools/own_compilers.py index 145b1ff..41de2c1 100644 --- a/dm/gfx/tools/own_compilers.py +++ b/dm/gfx/tools/own_compilers.py @@ -2,7 +2,7 @@ # dmsh_build.py with this Mac's own compilers: the NSP script builds the iOS shader libraries through this file. dmsh_build.py stops when any tool differs from dmsh_chain.lock; # here spirv-cross and the Metal compiler (macOS and iOS) may differ, with one note line naming them, while the translator (dmsh_xlate_bin) must still be the locked build. # Neither the chain version nor any repository file changes: a player builds the shader libraries and the host on the same Mac, so the two agree whatever the compilers, -# and an Xcode or spirv-cross other than the maintainer's (a beta, a brew upgrade) no longer needs dmsh_chain.py --update-lock, which changes the chain version of every profile. +# and an Xcode or spirv-cross other than the maintainer's (a beta, a brew upgrade) no longer needs dmsh_chain.py --update-lock, which changes the chain version (one for every game). # The lock stays the record of the compilers the libraries were tested with; on the maintainer's Mac dmsh_build.py keeps the strict check. # The per-shader cache takes spirv-cross's version into its keys but the Metal compiler only through the lock, so after a Metal compiler change the shaders already in the cache # keep the libraries the earlier compiler made and only new ones are compiled with this Mac's (the Metal compile cache, dmsh_mtlcache.py, is keyed by the compiler itself). diff --git a/dm/gfx/tools/tbdr_budget.py b/dm/gfx/tools/tbdr_budget.py index 2a6fb3d..f9ea5a7 100644 --- a/dm/gfx/tools/tbdr_budget.py +++ b/dm/gfx/tools/tbdr_budget.py @@ -19,12 +19,12 @@ TBPP = {10: 1, 20: 2, 25: 2, 30: 2, 53: 4, 55: 4, 60: 4, 65: 8, 70: 4, 71: 4, 80 texinfo = collections.defaultdict(dict); sampled = collections.defaultdict(set); tgt = {}; maxvc = collections.defaultdict(int); ndraw = collections.defaultdict(int); fskeys = collections.defaultdict(set); fused = set(); att_desc = collections.defaultdict(list) import os, json; sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))); import dmsh_chain DM_ROOT = os.environ.get('DM_ROOT') or os.path.expanduser('~/dm-work') # Workspace root (game files, build outputs and run artifacts are all under it) -LIB = os.path.join(DM_ROOT, 'build/cache/metal_shader_cache/b-{dmsh_chain.version()}') +LIB = os.path.join(DM_ROOT, f'build/cache/metal_shader_cache/b-{dmsh_chain.version()}') def tex_slots(key): try: return sum(1 for b in json.load(open(f'{LIB}/{key}_b.json'))['bindings'] if b.get('kind') in ('tex', 'samp_tex')) except Exception: return 4 nm = {} -for ln in open(dmsh_chain.GAME.table('shader_map.tsv')): # the current game profile's map +for ln in open(dmsh_chain.load_game().table('shader_map.tsv')): # the current game profile's map (DM_GAME) c = ln.rstrip('\n').split('\t') if len(c) > 4 and c[0].endswith('.bin'): nm.setdefault(c[3], c[0][:-4]) for l in open(log, errors='replace'): diff --git a/dm/host/Makefile b/dm/host/Makefile index 85dc1bd..1821db9 100644 --- a/dm/host/Makefile +++ b/dm/host/Makefile @@ -1,4 +1,4 @@ -# Game profile (dm/games/<id>/): the host is built for it (make GAME=base game title ID, required: there is no default game). game_profile.h is generated from game.tsv into gen/<id>/, dmsh_chain.h lives in the profile folder +# Game profile (dm/games/<id>/): the host is built for it (make GAME=base game title ID, required: there is no default game). game_profile.h is generated from game.tsv into gen/<id>/; the chain version header dmsh_chain.h (one for every game) lives in dm/gfx/nvn # without GAME only clean passes; anything else (a bare make too, whose default goal is the host) stops and lists the profiles ifeq ($(strip $(GAME)),) ifneq ($(filter-out clean,$(or $(MAKECMDGOALS),dmhost)),) @@ -50,8 +50,8 @@ endif endif .PHONY: FORCE FORCE: -dmhost: $(SRC) $(GFX) ../third_party/stb_vorbis/stb_vorbis.c dm.h hle.h dm_input.h front.h $(GFX_H) $(GAME_H) $(GAME_DIR)/dmsh_chain.h $(GAME_STAMP) $(VPX_LIB) - clang $(CFLAGS) -I../gfx/nvn/include -I../third_party/vulkan-headers/include -I$(dir $(GAME_H)) -I$(GAME_DIR) -I$(VPX) -I../third_party/stb_vorbis -fobjc-arc -o $@ $(SRC) $(GFX) $(VPX_LIB) -framework Cocoa -framework QuartzCore -framework Metal -framework GameController -framework AudioToolbox -framework VideoToolbox -framework CoreMedia -framework CoreVideo -framework ImageIO -framework UniformTypeIdentifiers +dmhost: $(SRC) $(GFX) ../third_party/stb_vorbis/stb_vorbis.c dm.h hle.h dm_input.h front.h $(GFX_H) $(GAME_H) $(GAME_STAMP) $(VPX_LIB) + clang $(CFLAGS) -I../gfx/nvn/include -I../third_party/vulkan-headers/include -I$(dir $(GAME_H)) -I$(VPX) -I../third_party/stb_vorbis -fobjc-arc -o $@ $(SRC) $(GFX) $(VPX_LIB) -framework Cocoa -framework QuartzCore -framework Metal -framework GameController -framework AudioToolbox -framework VideoToolbox -framework CoreMedia -framework CoreVideo -framework ImageIO -framework UniformTypeIdentifiers clean: # the host and everything make generates for it (gen/: the profile headers, the game stamp and the libvpx build; dmhost_* and *.dSYM beside them, as in .gitignore) rm -rf gen dmhost dmhost_* *.dSYM .PHONY: clean diff --git a/dm/ios/tools/build_ipa.sh b/dm/ios/tools/build_ipa.sh index a5aa405..3bdc6e8 100755 --- a/dm/ios/tools/build_ipa.sh +++ b/dm/ios/tools/build_ipa.sh @@ -49,7 +49,7 @@ else DM_SCHEME_NAME=DmIOS; fi export DM_SCHEME_NAME for f in $MODS; do [ -f "$PATCHED/$f.patched.bin" ] && [ -f "$PATCHED/$f.fixups.tsv" ] || die "no rewritten module $PATCHED/$f (run dm/tools/patch_nso.py first)"; done [ -e "$ROMFS/$MARKER" ] || die "no RomFS at $ROMFS (missing $MARKER)" -V=$(sed -n 's/^#define DMSH_CHAIN "\(.*\)"/\1/p' "$GD/dmsh_chain.h") +V=$(sed -n 's/^#define DMSH_CHAIN "\(.*\)"/\1/p' "$DM/gfx/nvn/dmsh_chain.h") # the chain version the host is compiled with, one for every game # A Vulkan game (Hades): the shaders are the game's SPIR-V, translated by dm/gfx/tools/vk_shaders.py into $MSL_IOS/vk; an OpenGL game (Digimon Story Cyber Sleuth): its GLSL, translated by # dm/gfx/tools/gl_shaders.py into $MSL_IOS/gl; neither has NVN chain libraries. GFXAPI=nvn / vulkan / gl says which (the NSP script passes what main imports); without it, the shader # libraries in $MSL_IOS tell: the chain's b-$V, else vk, else gl @@ -58,8 +58,12 @@ if [ -z "${GFXAPI:-}" ]; then if ls "$MSL_IOS/b-$V" 2> /dev/null | grep -q '_b.m if [ "$GFXAPI" = vulkan ]; then [ -f "$MSL_IOS/vk/vk_shaders.metallib" ] || die "no iOS Vulkan shader library $MSL_IOS/vk (dm/gfx/tools/vk_shaders.py \"$ROMFS\" <Mac dir> --ios \"$MSL_IOS\")" elif [ "$GFXAPI" = gl ]; then [ -f "$MSL_IOS/gl/gl_shaders.metallib" ] || die "no iOS GL shader library $MSL_IOS/gl (dm/gfx/tools/gl_shaders.py \"$ROMFS\" <main> <Mac dir> --ios \"$MSL_IOS\")" else [ -n "$V" ] && ls "$MSL_IOS/b-$V" 2> /dev/null | grep -q '_b.metallib$' || die "no iOS shader libraries $MSL_IOS/b-$V (IOSOUT=$MSL_IOS dmsh_build.py ...)" - # c1_full.tsv is not in the repository: it is made from the Mac outputs of the same shader build (the NSP script makes it, and cb_extent.tsv); the host would ignore a stale one - head -1 "$GD/c1_full.tsv" 2> /dev/null | grep -q "^# chain b-$V" || die "$GD/c1_full.tsv is missing or not made for chain $V: dm/gfx/tools/c1_guard.py <Mac shader dir>/b-$V \"$GD/c1_full.tsv\" (and cb_extent.py the same way for cb_extent.tsv)"; fi + # One chain version serves every game and every state of a profile's tables: the libraries' stamp (b-$V/inputs.tsv, written by a full dmsh_build.py run) must name this game and the + # digest of its profile's current inputs, and c1_full.tsv (not in the repository: the same build makes it, and cb_extent.tsv, from its Mac outputs) must name that digest, or the host ignores it + IN=$(DM_GAME="$GAME" python3 "$DM/gfx/tools/dmsh_chain.py" --inputs) || die "dmsh_chain.py --inputs (game profile $GD)" + grep -qxF "$(printf 'game\t%s' "$GAME")" "$MSL_IOS/b-$V/inputs.tsv" 2> /dev/null && grep -qxF "$(printf 'inputs\t%s' "$IN")" "$MSL_IOS/b-$V/inputs.tsv" || + die "the iOS shader libraries $MSL_IOS/b-$V are not a full build from this profile's current tables (inputs $IN; their stamp: $([ -f "$MSL_IOS/b-$V/inputs.tsv" ] && grep -v '^#' "$MSL_IOS/b-$V/inputs.tsv" | tr '\n\t' ' ' || echo none)): build them again (IOSOUT=$MSL_IOS dmsh_build.py ...)" + head -1 "$GD/c1_full.tsv" 2> /dev/null | grep -q "^# chain b-$V inputs $IN " || die "$GD/c1_full.tsv is missing or not made from these libraries (chain $V, inputs $IN): a full dmsh_build.py run makes it and cb_extent.tsv"; fi t "app: $DM_DISPLAY_NAME $DM_APP_VERSION ($DM_BUNDLE_ID), language $DM_LANG$([ $DM_SHOW_FPS = YES ] && echo ', frame-rate counter on')" t "generate Gen/ (host sources, embedded modules, tables)" @@ -102,7 +106,7 @@ if [ "$PREPARE_XCODE" = YES ] && [ -n "${DM_ENTITLEMENTS-default}" ]; then t "entitlements: $(plutil -convert json -o - Gen/DmIOS.entitlements)" fi xcodegen generate -q > /dev/null || die "xcodegen" -python3 tools/gen_resources.py "$IOS" "$ROMFS" "$MARKER" "$MSL_IOS/$MSLDIR" "$MSLDIR" "$AOC" "$XCODE_DD" "$GD/dmsh_chain.h" || die "generate Xcode resource paths" +python3 tools/gen_resources.py "$IOS" "$ROMFS" "$MARKER" "$MSL_IOS/$MSLDIR" "$MSLDIR" "$AOC" "$XCODE_DD" "$DM/gfx/nvn/dmsh_chain.h" || die "generate Xcode resource paths" if [ "$PREPARE_XCODE" = YES ]; then t "Xcode project: $IOS/DmIOS.xcodeproj" t "Xcode derived data: $XCODE_DD" diff --git a/dm/ios/tools/gen_resources.py b/dm/ios/tools/gen_resources.py index 29d8935..4824557 100644 --- a/dm/ios/tools/gen_resources.py +++ b/dm/ios/tools/gen_resources.py @@ -33,7 +33,7 @@ def main(): 'DM_RESOURCE_SHADERS': str(Path(shaders).resolve()), 'DM_RESOURCE_MSLDIR': msldir, 'DM_RESOURCE_AOC': str(Path(aoc).resolve()) if aoc and Path(aoc).is_dir() else '', - 'DM_RESOURCE_CHAIN_H': str(Path(chain_h).resolve()), # the profile's dmsh_chain.h, which the host is compiled with + 'DM_RESOURCE_CHAIN_H': str(Path(chain_h).resolve()), # dm/gfx/nvn/dmsh_chain.h, which the host is compiled with (one chain version for every game) } (ios / 'Gen/resources.env').write_text( '# Generated by build_ipa.sh; regenerate when the game or its drive changes.\n' diff --git a/dm/ios/tools/stage_resources.sh b/dm/ios/tools/stage_resources.sh index 0f4b1e2..76f5124 100644 --- a/dm/ios/tools/stage_resources.sh +++ b/dm/ios/tools/stage_resources.sh @@ -15,7 +15,7 @@ case "$APP" in /*.app) ;; *) die "Unexpected app resource destination: $APP";; e [ -z "$DM_RESOURCE_AOC" ] || [ -d "$DM_RESOURCE_AOC" ] || die "Add-on content unavailable: $DM_RESOURCE_AOC" # The host is compiled from the repository's current sources and loads only msl/b-<its chain version>: after a chain change (a pull, dmsh_chain.py --header) the prepared shaders are another chain's case "$DM_RESOURCE_MSLDIR" in b-*) - [ -f "$DM_RESOURCE_CHAIN_H" ] || die "No game profile chain header: $DM_RESOURCE_CHAIN_H" + [ -f "$DM_RESOURCE_CHAIN_H" ] || die "No chain version header: $DM_RESOURCE_CHAIN_H (a project prepared before the chain version moved to dm/gfx/nvn: prepare the game again, build_ipa_from_nsp.py --prepare-xcode)" V=$(sed -n 's/^#define DMSH_CHAIN "\(.*\)"/\1/p' "$DM_RESOURCE_CHAIN_H") [ "$DM_RESOURCE_MSLDIR" = "b-$V" ] || die "The prepared shaders are chain $DM_RESOURCE_MSLDIR, the sources are now at b-$V: prepare the game again (build_ipa_from_nsp.py --prepare-xcode)";; esac diff --git a/dm/tools/tests/test_nsp_chain.py b/dm/tools/tests/test_nsp_chain.py index a567273..13272f9 100644 --- a/dm/tools/tests/test_nsp_chain.py +++ b/dm/tools/tests/test_nsp_chain.py @@ -1,10 +1,12 @@ -"""NSP script chain files (scripts/build_ipa_from_nsp.py chain_header, chain_tables): a registered profile's stale chain version header is written again and a current one left alone, -only the c1 / uniform extent tables that are missing (they are not committed) or name another chain are made (GitHub issue #78), and no step's reuse key counts them; -no game files, keys or Xcode needed (temporary files only in a temporary folder).""" +"""NSP script chain files (scripts/build_ipa_from_nsp.py chain_header, chain_tables): the one chain version header (dm/gfx/nvn/dmsh_chain.h) is written again when stale and left +alone when current, and the committed one is current; only the c1 / uniform extent tables that are missing (they are not committed) or do not name the library's stamp (its chain version +and profile inputs) are made (GitHub issue #78), none without a stamped library, and no step's reuse key counts them; no game files, keys or Xcode needed (temporary files only in a +temporary folder).""" import importlib.util import os from pathlib import Path import shutil +import subprocess import sys import tempfile import types @@ -15,7 +17,6 @@ REPO = Path(__file__).resolve().parents[3] SCRIPT = REPO / 'scripts/build_ipa_from_nsp.py' spec = importlib.util.spec_from_file_location('build_ipa_from_nsp', SCRIPT) nsp = importlib.util.module_from_spec(spec); spec.loader.exec_module(nsp) -GAME = '010014E00DB56000' # a small profile (Digimon Story Cyber Sleuth): dmsh_chain.py hashes its tables class Base(unittest.TestCase): @@ -27,75 +28,92 @@ class Base(unittest.TestCase): class Header(Base): - def setUp(self): + def setUp(self): # a repository whose chain tools are the real ones (linked) and whose header is its own super().setUp() - self.dir = self.tmp / 'profiles' / GAME - shutil.copytree(REPO / 'dm/games' / GAME, self.dir) - self.env = dict(os.environ, DM_GAME=GAME, DM_GAMES_DIR=str(self.dir.parent), DM_GAMES_FIRST='1') - self.g = types.SimpleNamespace(table=lambda n: self.dir / n) + self.repo = self.tmp / 'repo' + (self.repo / 'dm/gfx/nvn').mkdir(parents=True) + (self.repo / 'dm/gfx/tools').symlink_to(REPO / 'dm/gfx/tools'); (self.repo / 'dm/ios').symlink_to(REPO / 'dm/ios') + self.h = self.repo / nsp.CHAIN_H + shutil.copy(REPO / nsp.CHAIN_H, self.h) + self.env = {k: v for k, v in os.environ.items() if k != 'DM_GAME'} # the version needs no game def header(self): - nsp.chain_header(sys.executable, REPO, self.env, self.tmp / 'logs', self.g) + nsp.chain_header(sys.executable, self.repo, self.env, self.tmp / 'logs') + + def test_committed_current(self): # a chain change committed without dmsh_chain.py --header + r = subprocess.run([sys.executable, str(REPO / 'dm/gfx/tools/dmsh_chain.py')], env=self.env, capture_output=True, text=True, check=True) + self.assertEqual(nsp.chain_of(REPO), r.stdout.split()[-1], 'dm/gfx/nvn/dmsh_chain.h is stale: run dm/gfx/tools/dmsh_chain.py --header and commit it') def test_stale_rewritten(self): - want = nsp.chain_of(self.g) - (self.dir / 'dmsh_chain.h').write_text('#define DMSH_CHAIN "000000000000"\n') + want = nsp.chain_of(REPO) + self.h.write_text('#define DMSH_CHAIN "000000000000"\n') self.header() - self.assertEqual(nsp.chain_of(self.g), want) + self.assertEqual(nsp.chain_of(self.repo), want) self.assertEqual(len(self.said), 1); self.assertIn('held 000000000000', self.said[0]) def test_missing_written(self): - want = nsp.chain_of(self.g) - (self.dir / 'dmsh_chain.h').unlink() + want = nsp.chain_of(REPO) + self.h.unlink() self.header() - self.assertEqual(nsp.chain_of(self.g), want) + self.assertEqual(nsp.chain_of(self.repo), want) self.assertIn('held no version', self.said[0]) def test_current_untouched(self): - p = self.dir / 'dmsh_chain.h'; os.utime(p, (1, 1)) + os.utime(self.h, (1, 1)) self.header() - self.assertEqual(p.stat().st_mtime, 1); self.assertEqual(self.said, []) + self.assertEqual(self.h.stat().st_mtime, 1); self.assertEqual(self.said, []) class Tables(Base): - CHAIN = '0123456789ab' + CHAIN, INPUTS = '0123456789ab', 'aaaaaaaaaaaa' def setUp(self): super().setUp() - self.sh = self.tmp / 'shader'; (self.sh / 'cache' / f'b-{self.CHAIN}').mkdir(parents=True) + self.sh = self.tmp / 'shader'; self.lib = self.sh / 'cache' / f'b-{self.CHAIN}'; self.lib.mkdir(parents=True) + (self.lib / 'inputs.tsv').write_text(f'# stamp\ngame\t010014E00DB56000\ninputs\t{self.INPUTS}\n') self.g = types.SimpleNamespace(table=lambda n: self.tmp / n) self.ran = [] cm = mock.patch.object(nsp, 'run', lambda cmd, log, env=None: self.ran.append(Path(cmd[1]).name)); cm.__enter__(); self.addCleanup(cm.__exit__, None, None, None) - def tables(self, c1, cb, every=False): - for n, chain in (('c1_full.tsv', c1), ('cb_extent.tsv', cb)): - if chain: (self.tmp / n).write_text(f'# chain b-{chain} 2026-10-23T00:00:00+00:00\n') - return nsp.chain_tables(sys.executable, REPO, self.sh, {}, self.tmp / 'logs', self.g, self.CHAIN, every) + def tables(self, c1, cb): # each a header line's "b-<chain> inputs <digest>" part (or any text), None for a missing table + for n, head in (('c1_full.tsv', c1), ('cb_extent.tsv', cb)): + if head: (self.tmp / n).write_text(f'# chain {head} 2026-10-23T00:00:00+00:00\n') + return nsp.chain_tables(sys.executable, REPO, self.sh, {}, self.tmp / 'logs', self.g, self.CHAIN) + + def current(self): return f'b-{self.CHAIN} inputs {self.INPUTS}' def test_current_kept(self): - self.assertTrue(self.tables(self.CHAIN, self.CHAIN)) + self.assertTrue(self.tables(self.current(), self.current())) self.assertEqual(self.ran, []); self.assertEqual(self.said, []) - def test_stale_made_again(self): - self.assertTrue(self.tables(self.CHAIN, 'fc6c0368c7b9')) - self.assertEqual(self.ran, ['cb_extent.py']); self.assertIn('named b-fc6c0368c7b9', self.said[0]) + def test_other_inputs_made_again(self): # the profile's tables changed and a full build stamped the library again + self.assertTrue(self.tables(self.current(), f'b-{self.CHAIN} inputs bbbbbbbbbbbb')) + self.assertEqual(self.ran, ['cb_extent.py']); self.assertIn(f'named b-{self.CHAIN} inputs bbbbbbbbbbbb', self.said[0]) + + def test_other_chain_made_again(self): + self.assertTrue(self.tables(f'b-fc6c0368c7b9 inputs {self.INPUTS}', self.current())) + self.assertEqual(self.ran, ['c1_guard.py']) + + def test_old_header_made_again(self): # a table from before the stamps names only the chain + self.assertTrue(self.tables(f'b-{self.CHAIN}', self.current())) + self.assertEqual(self.ran, ['c1_guard.py']) def test_missing_made(self): # a checkout's first pack: the tables are not in the repository self.assertTrue(self.tables(None, None)) self.assertEqual(self.ran, ['c1_guard.py', 'cb_extent.py']); self.assertIn('c1_full.tsv missing', self.said[0]) - def test_every(self): # a provisional or derived profile: both, current or not - self.assertTrue(self.tables(self.CHAIN, self.CHAIN, every=True)) - self.assertEqual(self.ran, ['c1_guard.py', 'cb_extent.py']); self.assertEqual(self.said, []) - - def test_no_mac_outputs(self): # build_shaders then builds again (reuse) or stops (after a build) - shutil.rmtree(self.sh / 'cache') - self.assertFalse(self.tables('fc6c0368c7b9', self.CHAIN)) + def test_no_stamp(self): # no full build made these Mac outputs: build_shaders then builds again (reuse) or stops (after a build) + (self.lib / 'inputs.tsv').unlink() + self.assertFalse(self.tables(self.current(), self.current())) self.assertEqual(self.ran, []) - self.assertTrue(self.tables(self.CHAIN, self.CHAIN)) # nothing to make: no Mac outputs needed - def test_not_profile_inputs(self): # the steps' reuse keys leave out what the iOS shader step makes - for n in ('game.tsv', 'shader_map.tsv', 'c1_full.tsv', 'cb_extent.tsv'): (self.tmp / n).write_text('') + def test_no_mac_outputs(self): + shutil.rmtree(self.sh / 'cache') + self.assertFalse(self.tables(None, None)) + self.assertEqual(self.ran, []) + + def test_not_profile_inputs(self): # the steps' reuse keys leave out what the iOS shader step makes, and the rewriter's patches (the rewrite step's own key) + for n in ('game.tsv', 'shader_map.tsv', 'c1_full.tsv', 'cb_extent.tsv', 'patches.py'): (self.tmp / n).write_text('') self.assertEqual([p.name for p in nsp.profile_files(types.SimpleNamespace(dir=self.tmp))], ['game.tsv', 'shader_map.tsv']) diff --git a/docs/agents/04-graphics.md b/docs/agents/04-graphics.md index e37319a..b58a3dc 100644 --- a/docs/agents/04-graphics.md +++ b/docs/agents/04-graphics.md @@ -137,7 +137,7 @@ So the translator leaves z alone (`QueryGraphicsState` depth mode false in `Prog ### Seen in 13 Sentinels - **Single-stage programs**: the game builds one NVN program per GLSLC output (838 `ProgramSetShaders` at startup) and combines vertex and pixel programs per draw with `BindProgram` stage bits, so the composite-program path from Metroid Dread does all the pairing. The real pairs exist only at draw time. Every pixel shader input is a plain `float4` at locations 1–9 (the SPH input map, words 6–13, matches the translated MSL inputs for all 375 pixel shaders), so `shader_split.py` pairs every vertex shader with a *cover*: the fewest pixel shaders whose inputs together are every input any pixel shader reads (here one shader). `vs_pad` then gives each vertex shader all of them, and any vertex / pixel combination links. -- **`dmsh_build.py`'s shader cache is keyed without the pairs**: a vertex shader whose pairs changed is restored unpadded from `cas/`. After changing a profile's pairs, delete that profile's cache (`build/cache/metal_shader_cache/cas` and `b-*`) and rebuild. The NSP script always starts from its own cache, which has the same problem only when a profile's pairs change between two of its runs. +- **`dmsh_build.py`'s shader cache was keyed without the pairs**: a vertex shader whose pairs changed was restored unpadded from `cas/`. The per-shader key now covers the pairs, and a paired vertex shader's also its pixel shaders' keys (`shader_key`, `pair_keyed`), so a change to a profile's pairs rebuilds what it affects. - **Triangle fans and indexed quads**: Metal has neither. Non-indexed fans draw through a shared `(0, i, i+1)` index buffer with `first` as the base vertex; indexed fans and quads are expanded on the CPU into a 32-bit triangle list in the frame arena at replay (`nvn_mtl.m`, as for 8-bit indices). - **Vertex streams 14 and 15**: the host bound NVN stream *n* to Metal vertex buffer 16 + *n*, but Metal has slots 0–30 only (30 is the host's constant fallback), and stream 15 hit the assertion `buffer index (31) must be < 31` a minute and a half into the prologue. Translated shaders use only `buffer(0)`, so streams 14 and 15 now go to slots 14 and 15 (`VB_SLOT`). - **Packaged texture data from CPU memory**: every texture is created with `TextureBuilderSetPackagedTextureData`; NVN copies that data into the texture storage at `TextureInitialize` (a compressible pool cannot be written by the CPU). The host copies it and marks the texture CPU-written, so it is de-laid-out and uploaded like a packaged texture. Linear textures (the save thumbnails) are excluded: in a CPU-visible pool they are read straight from the pool buffer. diff --git a/docs/development.md b/docs/development.md index be77cce..9c3bc44 100644 --- a/docs/development.md +++ b/docs/development.md @@ -107,8 +107,10 @@ The unsigned IPA build still uses the repository's `dm/ios/DmIOS.xcodeproj`, ski Each game has a profile in `dm/games/<id>/`: - `game.tsv` holds the title ID, known builds, code modules, RomFS marker, memory split, app name and bundle ID; -- the shader tables: `shader_map.tsv`, `shader_facts.tsv`, `shader_pairs.tsv`, `pso_manifest.tsv`, and `c1_full.tsv` / `cb_extent.tsv`, which each checkout makes from its own shader build (not committed); -- `dmsh_chain.h` holds the chain version. +- the shader tables: `shader_map.tsv`, `shader_facts.tsv`, `shader_pairs.tsv`, `pso_manifest.tsv`, and `c1_full.tsv` / `cb_extent.tsv`, which each checkout's full shader build makes (not committed); +- `patches.py` (when there are any): the rewriter's patches of the game's own builds. + +The shader translation chain has one version for every game, in `dm/gfx/nvn/dmsh_chain.h`. The shader tools pick a profile with `DM_GAME=<id>`; `make -C dm/host` and `build_ipa.sh` pick it with `GAME=<id>`. Profiles are named after the base game's title ID. There is no default game: without `GAME` / `DM_GAME` the tools stop and list the profiles. `python3 dm/games/game.py list` lists the profiles. Host code that relies on one build's fixed addresses checks that build's ID. @@ -134,10 +136,7 @@ $W/build/nx_extract/nx_extract <prod.keys> <base NSP> <update NSP> $W/data/$VER for m in $(python3 dm/games/game.py $G modules); do python3 dm/tools/patch_nso.py $W/data/$VER/ExeFS/$m $W/build/patched_$VER $m; done # 3. shaders: extract from every version in the profile's shader_versions (comma-separated, base first), then translate into <cache>/b-<chain version>/ DM_GAME=$G python3 dm/gfx/tools/shader_split.py $W/data/v0/RomFS,$W/data/$VER/RomFS $W/data/v0/ExeFS/main,$W/data/$VER/ExeFS/main $W/shader/pkg dm/games/$G/shader_map.tsv dm/games/$G/shader_facts.tsv dm/games/$G/shader_pairs.tsv -DM_GAME=$G python3 dm/gfx/tools/dmsh_chain.py --header -DM_GAME=$G python3 dm/gfx/tools/dmsh_build.py $W/shader/pkg $W/build/cache/metal_shader_cache -python3 dm/gfx/tools/c1_guard.py $W/build/cache/metal_shader_cache/b-<chain version> dm/games/$G/c1_full.tsv -python3 dm/gfx/tools/cb_extent.py $W/build/cache/metal_shader_cache/b-<chain version> dm/games/$G/cb_extent.tsv +DM_GAME=$G python3 dm/gfx/tools/dmsh_build.py $W/shader/pkg $W/build/cache/metal_shader_cache # also writes dm/games/$G/c1_full.tsv and cb_extent.tsv # 4. the host, and the button-icon font for the Nintendo-extension shared font make -C dm/host GAME=$G dmhost python3 dm/tools/make_ext_font.py $W/build/font/nintendo_ext.ttf diff --git a/scripts/build_ipa_from_nsp.py b/scripts/build_ipa_from_nsp.py index ad5d937..db82a37 100755 --- a/scripts/build_ipa_from_nsp.py +++ b/scripts/build_ipa_from_nsp.py @@ -15,9 +15,9 @@ # Every intermediate file is under <id>_build/ in the NSP folder: extract/ (the extracted game files, kept for the next build unless KEEP_EXTRACTED / --no-keep-extracted), aoc/ (each DLC's RomFS by index), tools/ (nx_extract, and the Mac host for a run-time shader capture), patched_*/, shader/ (shader package and caches, kept so the next build is fast), logs/ (each step's full output), tmp/, ipa/. The finished IPA also goes to the NSP folder by default. # The DLC, the rewrite, the shader split and the iOS (or Vulkan) shaders are reused when all their inputs are as at their last successful run (the game files, the tools, the profile, Xcode and spirv-cross: fingerprint() keys beside their output); # the iOS shaders' inputs are the shader package's content and the chain's own tools, so a split that runs again and writes the same package leaves them reused. -# A registered profile's chain version header (dmsh_chain.h) is written again when a chain change left it behind (chain_header: it follows from the source alone), so a run may leave it changed -# in the profile folder. The c1 / uniform extent tables (c1_full.tsv, cb_extent.tsv) are not in the repository: the iOS shader step makes them in the profile folder from this build's shaders -# (chain_tables), where git ignores them. +# The chain version header dm/gfx/nvn/dmsh_chain.h (one version for every game) is written again when a chain change was committed without it (chain_header: it follows from the source alone), +# so a run may leave it changed. The c1 / uniform extent tables (c1_full.tsv, cb_extent.tsv) are not in the repository: the iOS shader step's dmsh_build.py makes them in the profile folder from +# this build's shaders, where git ignores them, and a reused step makes them again when they do not name its libraries' stamp (chain_tables). # The shader translator is the in-repo dm/gfx/tools/dmsh_xlate_bin (the build whose fingerprints match dmsh_chain.lock; a rebuilt dll has different fingerprints and dmsh_build refuses it; its launcher is universal, so an Intel Mac runs it too). The Metal compiler and spirv-cross may be other than the lock's: the iOS shaders # are built through dm/gfx/tools/own_compilers.py, which says so in one line and leaves the lock and the chain version as they are. # The Python running this file needs lz4, capstone and fonttools (and backports.zstd before Python 3.14); also needed: Xcode, xcodegen, .NET SDK 10, zstd, spirv-cross. A first run takes 2 (Metroid Dread) to 10 minutes (Breath of the Wild: 5 of them the shader split), longer when shader libraries must be compiled; a repeat run with unchanged inputs takes about 20 s (Metroid Dread), mostly packing the IPA. Key values are never printed. @@ -548,7 +548,7 @@ def ipa_name(name, app): # "The Legend of Zelda: Link's Awakening" → TheLege return (''.join(words)[:48] or f'{app:016X}') + '.ipa' -EMPTY_TABLES = { # the shader tables before (or, for a Vulkan game, instead of) a shader split: the chain version (dmsh_chain.py) and build_ipa.sh read them +EMPTY_TABLES = { # the shader tables before (or, for a Vulkan game, instead of) a shader split: the profile inputs' digest (dmsh_chain.py inputs) and build_ipa.sh read them 'shader_map.tsv': '# kind: evidence provisional profile: filled by the shader split (a Vulkan game\'s SPIR-V shaders go through the Vulkan layer and leave it empty)\npackage\tindex\tstage\tdm_ctl_sha16\tsource\n', 'shader_facts.tsv': '# Answer table for translation queries (generated by dm/gfx/tools/shader_split.py, do not edit): key <TAB> fact JSON. Provisional profile: filled by the shader split\n', 'shader_pairs.tsv': '# kind: evidence vertex / pixel shader pairs within BNSH variations (generated by dm/gfx/tools/shader_split.py, do not edit): vertex key <TAB> pixel key\n', @@ -598,7 +598,7 @@ def write_profile(pdir, app, f): # ---- Derived profile (a version the game's profile was not brought up on) ---- -DERIVED_SKIP = {'game.tsv', 'shader_map.tsv', 'shader_facts.tsv', 'shader_pairs.tsv', 'c1_full.tsv', 'cb_extent.tsv', 'dmsh_chain.h', 'vk_capture.tsv', 'patches.py'} # made again from the packed version's own files (patches.py: the rewriter's patches of the registered builds) +DERIVED_SKIP = {'game.tsv', 'shader_map.tsv', 'shader_facts.tsv', 'shader_pairs.tsv', 'c1_full.tsv', 'cb_extent.tsv', 'vk_capture.tsv', 'patches.py'} # made again from the packed version's own files (patches.py: the rewriter's patches of the registered builds) def derive_profile(g, pdir, ver, d, mods): @@ -1028,48 +1028,59 @@ def set_keys(g, changes, section): def provisional_prep(py, repo, env, logs, g): - """What a provisional profile needs before a Mac host can be built for it: the shader tables (empty until a shader split fills them; an older provisional profile may lack some) - and the chain version header (the host and build_ipa.sh read it; a Vulkan game never runs the NVN chain that writes it)""" + """What a provisional profile needs before a Mac host can be built for it: the shader tables (empty until a shader split fills them; an older provisional profile may lack some), + which the host and build_ipa.sh read""" for n, text in EMPTY_TABLES.items(): if not g.table(n).exists(): g.table(n).write_text(text) - run([py, repo / 'dm/gfx/tools/dmsh_chain.py', '--header'], logs / 'dmsh_chain.log', env=env) -def chain_of(g): # the chain version the profile's dmsh_chain.h holds (None when it has none) - try: m = re.search(r'#define DMSH_CHAIN "(.*)"', g.table('dmsh_chain.h').read_text()) +CHAIN_H = 'dm/gfx/nvn/dmsh_chain.h' # the chain version the host is compiled with, one for every game + + +def chain_of(repo): # the chain version dm/gfx/nvn/dmsh_chain.h holds (None when it has none) + try: m = re.search(r'#define DMSH_CHAIN "(.*)"', (repo / CHAIN_H).read_text()) except OSError: return None return m.group(1) if m else None -def chain_header(py, repo, env, logs, g): - """A registered profile's chain version header, written again when it no longer holds the version the source tree gives: dmsh_build.py stops on a stale one, and the host and build_ipa.sh - read it. The version follows from the source alone (dmsh_chain.py), so a chain change that rewrote only some profiles' headers stops no build (GitHub issue #78: Breath of the Wild after bb585cd)""" +def chain_header(py, repo, env, logs): + """The chain version header, written again when it no longer holds the version the source tree gives (a chain change committed without dmsh_chain.py --header): dmsh_build.py stops + on a stale one, and the host and build_ipa.sh read it. The version follows from the source alone (dmsh_chain.py), one for every game (GitHub issue #78: Breath of the Wild after bb585cd)""" r = subprocess.run([str(py), str(repo / 'dm/gfx/tools/dmsh_chain.py')], env=env, capture_output=True, text=True) if r.returncode or not r.stdout.split(): raise BuildError(f'dmsh_chain.py exit code {r.returncode}: {(r.stderr.strip() or "no version")[-500:]}') - want, have = r.stdout.split()[-1], chain_of(g) + want, have = r.stdout.split()[-1], chain_of(repo) if have == want: return run([py, repo / 'dm/gfx/tools/dmsh_chain.py', '--header'], logs / 'dmsh_chain.log', env=env) - say(f'chain version header rewritten: {g.table("dmsh_chain.h")} held {have or "no version"}, the source tree gives {want}') + say(f'chain version header rewritten: {repo / CHAIN_H} held {have or "no version"}, the source tree gives {want}') -CHAIN_TABLES = (('c1_full.tsv', 'c1_guard.py', 'c1 guard'), ('cb_extent.tsv', 'cb_extent.py', 'uniform read extents')) # made from the shader build's Mac outputs: the host takes each only when its header names its chain +CHAIN_TABLES = (('c1_full.tsv', 'c1_guard.py', 'c1 guard'), ('cb_extent.tsv', 'cb_extent.py', 'uniform read extents')) # made from the shader build's Mac outputs: the host takes each only when its header names its library's stamp -def chain_tables(py, repo, sh, env, logs, g, chain, every=False): +def lib_head(lib): + """What c1_full.tsv / cb_extent.tsv start with to be taken beside the library directory lib (dmsh_chain.py table_head): its chain version and the profile inputs its stamp (inputs.tsv, + written by a full dmsh_build.py run) names. None when it has no stamp""" + try: stamp = dict(l.split('\t', 1) for l in (lib / 'inputs.tsv').read_text().splitlines() if '\t' in l and not l.startswith('#')) + except OSError: return None + return f'# chain {lib.name} inputs {stamp["inputs"]}' if stamp.get('inputs') else None + + +def chain_tables(py, repo, sh, env, logs, g, chain): """c1_full.tsv (c1_guard.py: shaders whose c1 reads are not proven bounded) and cb_extent.tsv (cb_extent.py: how far each shader reads into its uniform blocks) from the Mac outputs of the shader build (sh/cache/b-<chain>/), into the profile folder. They are not in the repository (git ignores them): they follow from the shaders this Mac built, so every pack makes its own. - every: both (a provisional or derived profile's tables come from its own build); else only those missing or whose header names another chain, which the host would ignore (every c1 its - own 64 KiB, whole uniform bindings copied; GitHub issue #78). True when both name the chain; False when one does not and there are no Mac outputs to make it from""" + A full dmsh_build.py run makes both; this makes those missing or not naming the library's stamp again, which the host would ignore (every c1 its own 64 KiB, whole uniform bindings + copied; GitHub issue #78). True when both name it; False when the library has no stamp (no full build's Mac outputs to make them from)""" lib = sh / 'cache' / f'b-{chain}' + want = lib_head(lib) + if not want: return False for name, tool, step in CHAIN_TABLES: p = g.table(name) try: head = p.read_text(errors='replace').partition('\n')[0] except OSError: head = '' - if not every and head.startswith(f'# chain b-{chain}'): continue - if not lib.is_dir(): return False + if head.startswith(want + ' '): continue PROG.set(step=step) run([py, repo / 'dm/gfx/tools' / tool, lib, p], logs / f'{tool[:-3]}.log', env=env) - if not every: say(f'{name} ' + (f'named {head[len("# chain "):].split(" ")[0]}' if head.startswith('# chain ') else 'names no chain' if head else 'missing') + f': made for chain {chain} ({p})') + say(f'{name} ' + (f'named {" ".join(head.split()[2:5])}' if head.startswith('# chain ') else 'names no library' if head else 'missing') + f': made for {want[2:]} ({p})') return True @@ -1162,7 +1173,7 @@ class ShaderWatch: the cache check (a hit restores <key>_b.json into the iOS output b-<chain>/, a miss writes <key>.bin into its temporary folder dmsh_* under WORK), the translation of the misses (<key>.spv in the temporary folder), and the compile, which takes the keys in a fixed order (sorted, the vertex shaders with pixel pairs last) and writes <key>_b.json again for every miss and every pixel shader: the furthest key written so far tells how far it got""" - def __init__(self, g, ios, tmp): + def __init__(self, g, out, tmp): # out: the iOS library directory b-<chain>/ keys = set() for ln in open(g.table('shader_map.tsv'), errors='replace'): # the keys dmsh_build.py builds: column 4 of the rows from a shader package c = ln.rstrip('\n').split('\t') @@ -1170,7 +1181,7 @@ class ShaderWatch: pairs = {ln.split()[0] for ln in open(g.table('shader_pairs.tsv'), errors='replace') if ln.strip() and not ln.startswith('#')} order = sorted(keys - pairs) + sorted(keys & pairs) self.pos, self.n = {k: i for i, k in enumerate(order)}, len(order) - self.out, self.tmp, self.t0, self.t1, self.w = ios / f'b-{chain_of(g)}', tmp, time.time(), None, None + self.out, self.tmp, self.t0, self.t1, self.w = out, tmp, time.time(), None, None self.hits, self.fin, self.top = set(), set(), -1 def written(self, keys, since): # those of keys whose <key>_b.json in the iOS output was written after since @@ -1204,30 +1215,27 @@ def toolchain(): # the compilers' versions, part of the shader stages' keys return ('cmd', ['xcodebuild', '-version']), ('cmd', ['spirv-cross', '--revision']) -def build_shaders(py, repo, sh, xlate, env, logs, g, provisional=False, pkg_key=None): +def build_shaders(py, repo, sh, xlate, env, logs, g, pkg_key=None): """The iOS shader libraries (dmsh_build.py). Reused when the shader package (pkg_key, its content; None never reuses) and the chain's tools, translator binaries, profile and compilers are as at the last successful build: dmsh_build.py would only restore every shader from its cache (63,000 for Breath of the Wild: about 100 s)""" ios, log = sh / 'ios', logs / 'dmsh_build.log' PROG.stage('iOS shaders') - if provisional: - PROG.set(step='chain header') - run([py, repo / 'dm/gfx/tools/dmsh_chain.py', '--header'], logs / 'dmsh_chain.log', env=env) # the profile's chain version header (dmsh_build checks it) - chain = chain_of(g) + chain = chain_of(repo) libs = ios / f'b-{chain}'; kf = libs.with_name(libs.name + '.key') if pkg_key: PROG.set(step='checking the inputs') key = fingerprint(pkg_key, ('files', chain_files(repo)), ('stat', xlate.parent), ('files', profile_files(g)), *toolchain(), str(xlate), {k: env.get(k) for k in ('XLATE_DEBUG', 'XLATE_REDUCED')}, env=env) - # the libraries are the last build's, so are tables that name the chain; a missing or stale one is made from that build's Mac outputs, and without them the step runs again + # the libraries are the last build's, so are tables that name their stamp; a missing or stale one is made from that build's Mac outputs, and without them the step runs again if next(libs.glob('*_b.metallib'), None) and (note := reused(kf, key)) is not None and chain_tables(py, repo, sh, env, logs, g, chain): PROG.end(f'{note}, reused (inputs unchanged)'); return ios else: key = None rmtree(kf) - try: PROG.watch(ShaderWatch(g, ios, Path(env['WORK']))) + try: PROG.watch(ShaderWatch(g, libs, Path(env['WORK']))) except Exception: pass # progress never stops the build (a missing table is dmsh_build.py's to report) run([py, repo / 'dm/gfx/tools/own_compilers.py', sh / 'pkg', sh / 'cache'], log, env=dict(env, XLATE=str(xlate), IOSOUT=str(ios))) # dmsh_build.py with this Mac's Metal compiler and spirv-cross PROG.watch(None) - if not chain_tables(py, repo, sh, env, logs, g, chain, every=provisional): raise BuildError(f'no Mac shader outputs in {sh / "cache" / f"b-{chain}"} to make c1_full.tsv and cb_extent.tsv from') + if not chain_tables(py, repo, sh, env, logs, g, chain): raise BuildError(f'no stamped Mac shader outputs in {sh / "cache" / f"b-{chain}"} (inputs.tsv): dmsh_build.py did not finish a full build') n = sum(1 for _ in libs.glob('*_b.metallib')) if libs.is_dir() else 0 if n == 0: raise BuildError(f'no iOS shader libraries in {libs}') say(f'{n} iOS shader libraries ({libs})') @@ -1532,7 +1540,7 @@ def main(args=None): patched = work / f'patched_{pkg_ver}' rewrite(py, repo, game / 'ExeFS', patched, env, logs, g) - if not (provisional or derived): chain_header(py, repo, env, logs, g) # before any host is built for the profile + chain_header(py, repo, env, logs) # before any host is built if provisional or derived: # the keys that need the rewritten modules, then (provisional) the heap check (both only change keys the host is built with, not the rewrite or the shader steps) provisional_prep(py, repo, env, logs, g) g = refine_after_rewrite(g, game, patched) @@ -1552,7 +1560,7 @@ def main(args=None): oodle = (lambda: {'DM_OODLE_HOST': str(build_mac_host(repo, work / 'tools/dmhost', env, logs, g)), 'DM_OODLE_MODULES': str(patched)}) if src['oodle_packs'][0] or src['ue_legacy'][0] else None own = bool(provisional or derived) # the shader tables come from this build's split (the profile's are checked otherwise) pkey = split_shaders(py, repo, [dirs[v] / 'RomFS' for v in sv], [dirs[v] / 'ExeFS/main' for v in sv], sh / 'pkg', env, logs, g, own, oodle) - ios = build_shaders(py, repo, sh, xlate, env, logs, g, own, pkey) + ios = build_shaders(py, repo, sh, xlate, env, logs, g, pkey) if not prepare_xcode and free_gb(out_ipa.parent) < 5: raise BuildError(f'less than 5 GB free on the volume of {out_ipa.parent}') say('preparing the Xcode project' if prepare_xcode else f'building the IPA → {out_ipa}')