diff --git a/dm/games/010036E00FB20000/README.md b/dm/games/010036E00FB20000/README.md index b5ebd0b..00200e4 100644 --- a/dm/games/010036E00FB20000/README.md +++ b/dm/games/010036E00FB20000/README.md @@ -10,6 +10,7 @@ Base game 1.0.0 (v0), with its DLC `010036E00FB21001`. SDK 10.4.1. Capcom's MT F | 4 GB layout: `mem_reserve 0x100000000`, `mem_alias 0x10000000`, `mem_stack 0x10000000`, `app_pool 0xCD500000` | `game.tsv` | the NPDM declares no system resource (META +0x14 = 0): `nninitStartup` sets a heap of the pool − 64 MB (3.14 GB); `main`'s bss is 9.2 MB | | Node area cut 1 GB → 62 MB, 1 GB alignment → 32 MB | `dm/tools/sdk_patches.py`, `sdk` build `d9add76f…` | applies to this build by its ID; made for Advance Wars 1+2: Re-Boot Camp, which shares it. This game has no virtual address memory and never makes an address region | | `pad_layout` not set (the default `simple`, 2D) | `game.tsv` | the gamepad starts in the 2D layout (the player-facing README lists it among the games that do) | +| `depth_mode zero` | `game.tsv` | the game sets the near = 0 depth mode, so the pipeline bases made before it runs (the NSP script's Mac prebuild, and a [pipeline archive made offline](../../../docs/development.md#the-pipeline-archive-made-offline) for the iPhone when one is built with `DM_PSO_GPUS`) take its vertex shaders' `main0_z` entries (the host itself follows the mode the game sets) | ## Rendering diff --git a/dm/games/010036E00FB20000/game.tsv b/dm/games/010036E00FB20000/game.tsv index 6fbeb4e..51b8e7f 100644 --- a/dm/games/010036E00FB20000/game.tsv +++ b/dm/games/010036E00FB20000/game.tsv @@ -45,3 +45,5 @@ tex_dirty_upload on # ---- Shader split (shader_split.py; MT Framework SPK in an ARC archive) ---- shader_groups /nativeNXx64/=nativenxx64 shader_note The Great Ace Attorney Chronicles shaders +# the NVN depth mode the game sets (nvnDeviceSetDepthMode near = 0): the pipeline archive made offline holds the main0_z vertex entries (dm/gfx/tools/pso_offline.py) +depth_mode zero diff --git a/dm/games/01003FC01670C000/README.md b/dm/games/01003FC01670C000/README.md index dc7b0cd..ec4d420 100644 --- a/dm/games/01003FC01670C000/README.md +++ b/dm/games/01003FC01670C000/README.md @@ -10,6 +10,7 @@ The Asian release (Japanese, Korean, Chinese) has its own title ID and `main` bu |---|---|---| | `alloc_big_align on` | [`game.tsv`](game.tsv) | the engine cuts ring heaps out of a memory pool at 64 KB GPU alignment, and the pool's storage comes from malloc: without 64 KB-aligned large allocations the third ring heap keeps base 0 and the prologue's first battle writes through a null pointer; see [Memory](#memory) | | 4 GB layout: `mem_reserve 0x100000000`, `mem_alias 0x10000000`, `mem_stack 0x10000000`, `app_pool 0xCD500000` | `game.tsv` | 256 MB in front + 256 MB alias region + 256 MB stack region + 3.25 GB heap region. No system resource: the game sets a heap | +| `depth_mode zero` | `game.tsv` | the game sets the near = 0 depth mode, so the pipeline bases made before it runs (the NSP script's Mac prebuild, and a [pipeline archive made offline](../../../docs/development.md#the-pipeline-archive-made-offline) for the iPhone when one is built with `DM_PSO_GPUS`) take its vertex shaders' `main0_z` entries (the host itself follows the mode the game sets) | No `patches.py`, no `sdk_patches.py` cut for its `sdk` build (`2d369d99…`), no `pad_layout` (the iPhone pad starts in the 2D layout). diff --git a/dm/games/01003FC01670C000/game.tsv b/dm/games/01003FC01670C000/game.tsv index aa65dc4..40f2cc3 100644 --- a/dm/games/01003FC01670C000/game.tsv +++ b/dm/games/01003FC01670C000/game.tsv @@ -38,3 +38,5 @@ alloc_big_align on # ---- Shader split (shader_split.py: GLSLC outputs in main, each a single-stage program, 838 ProgramSetShaders at startup; BindProgram's stage bits combine vertex and pixel per draw) ---- shader_groups ROBO=cpk shader_note 13 Sentinels: Aegis Rim shaders +# the NVN depth mode the game sets (nvnDeviceSetDepthMode near = 0): the pipeline archive made offline holds the main0_z vertex entries (dm/gfx/tools/pso_offline.py) +depth_mode zero diff --git a/dm/games/0100620012D6E000/README.md b/dm/games/0100620012D6E000/README.md index 71cb9e5..08c496e 100644 --- a/dm/games/0100620012D6E000/README.md +++ b/dm/games/0100620012D6E000/README.md @@ -12,6 +12,7 @@ Brought up on the Mac: the title, Solo, enrolling a new student (birthday, profe | `op_mode handheld` | `game.tsv` | handheld, the game reads the touch screen (menus, every activity, Party's two-player touch battles); docked it does not | | `perf_mode boost` | `game.tsv` | handheld Normal draws 1280x720, handheld with Boost 1920x1080 (the window crop goes from 1280x720 to 1920x1080 once the game sees Boost) | | 4 GB layout: `mem_reserve 0x100000000`, `mem_alias 0x10000000`, `mem_stack 0x10000000`, `app_pool 0xCD500000` | `game.tsv` | a 3328 MB heap region: `SetHeapSize` asks for 3.14 GB | +| `depth_mode zero` | `game.tsv` | the game sets the near = 0 depth mode, so the pipeline bases made before it runs (the NSP script's Mac prebuild, and a [pipeline archive made offline](../../../docs/development.md#the-pipeline-archive-made-offline) for the iPhone when one is built with `DM_PSO_GPUS`) take its vertex shaders' `main0_z` entries (the host itself follows the mode the game sets) | No `patches.py`, and `dm/tools/sdk_patches.py` has no cut for its `sdk` build (`2d369d9999e3e10e665106f840ffff9b`). diff --git a/dm/games/0100620012D6E000/game.tsv b/dm/games/0100620012D6E000/game.tsv index 6be0255..077dbbb 100644 --- a/dm/games/0100620012D6E000/game.tsv +++ b/dm/games/0100620012D6E000/game.tsv @@ -44,3 +44,5 @@ shader_note Big Brain Academy: Brain vs. Brain shaders # handheld with Boost 1920x1080 (the window crop goes from 1280x720 to 1920x1080 once the game sees Boost) op_mode handheld perf_mode boost +# the NVN depth mode the game sets (nvnDeviceSetDepthMode near = 0): the pipeline archive made offline holds the main0_z vertex entries (dm/gfx/tools/pso_offline.py) +depth_mode zero diff --git a/dm/games/01006BB00C6F0000/README.md b/dm/games/01006BB00C6F0000/README.md index 4444f2d..48ba15d 100644 --- a/dm/games/01006BB00C6F0000/README.md +++ b/dm/games/01006BB00C6F0000/README.md @@ -9,6 +9,7 @@ Update 1.1.0 (v131072) over the base game, SDK 8.2.0. Grezzo's engine on Nintend | Physics completion task `release` → `ret` | [`patches.py`](patches.py) `CUTS`, `main` `909e904a…` at `main+0x10cd070` | a late destructor wiped the rebuilt task and crashed the physics workers; see [Physics](#physics) | | Float vertex inputs 1–5 for `147c829e79fb322f`, `03c43da3e3f46da8`, `079700f3577e311f`, `15c02683a10f8e9d`, `2344874045b1e758` | [`shader_facts_manual.tsv`](shader_facts_manual.tsv) | the integer-input guess declared them `uint4` while the game feeds 16-bit or 8-bit normalized formats; see [Rendering](#rendering) | | 4 GB layout: `mem_reserve 0x100000000`, `mem_alias 0x10000000`, `mem_stack 0x10000000`, `app_pool 0xCD500000` | [`game.tsv`](game.tsv) | no system resource in the NPDM: the game uses `SetHeapSize`. 13 Sentinels' partition (256 MB in front + 256 MB alias + 256 MB stack + 3.25 GB heap), "until measured" | +| `depth_mode zero` | `game.tsv` | the game sets the near = 0 depth mode, so the pipeline bases made before it runs (the NSP script's Mac prebuild, and a [pipeline archive made offline](../../../docs/development.md#the-pipeline-archive-made-offline) for the iPhone when one is built with `DM_PSO_GPUS`) take its vertex shaders' `main0_z` entries (the host itself follows the mode the game sets) | No `dm/tools/sdk_patches.py` cut for its `sdk` build (`3ae34e68…`). No `pad_layout`: the virtual gamepad starts in the 2D layout. diff --git a/dm/games/01006BB00C6F0000/game.tsv b/dm/games/01006BB00C6F0000/game.tsv index 1da091d..283ad5b 100644 --- a/dm/games/01006BB00C6F0000/game.tsv +++ b/dm/games/01006BB00C6F0000/game.tsv @@ -35,3 +35,5 @@ app_pool 0xCD500000 # ---- Shader split (shader_split.py) ---- shader_groups /region_common/shader/=shader /region_common/effect/=effect /ui/=layout shader_note Link's Awakening shaders +# the NVN depth mode the game sets (nvnDeviceSetDepthMode near = 0): the pipeline archive made offline holds the main0_z vertex entries (dm/gfx/tools/pso_offline.py) +depth_mode zero diff --git a/dm/games/01007300020FA000/README.md b/dm/games/01007300020FA000/README.md index 884cda7..93f54bb 100644 --- a/dm/games/01007300020FA000/README.md +++ b/dm/games/01007300020FA000/README.md @@ -10,6 +10,7 @@ Update 1.0.1 (v65536) over the base game. PlatinumGames' engine, SDK 6.4.0, NVN, | `sys_resource 0x0` | `game.tsv` | the NPDM declares no system resource; no virtual address memory | | `pad_layout full` | `game.tsv` | movement and camera use both sticks | | `fps_target 60` | `game.tsv`, [`patches.py`](patches.py) | host-paced gameplay with fractional timer carry, integer input windows and camera smoothing; only the verified 1.0.1 main build | +| `depth_mode zero` | `game.tsv` | the game sets the near = 0 depth mode, so the pipeline bases made before it runs (the NSP script's Mac prebuild, and a [pipeline archive made offline](../../../docs/development.md#the-pipeline-archive-made-offline) for the iPhone when one is built with `DM_PSO_GPUS`) take its vertex shaders' `main0_z` entries (the host itself follows the mode the game sets) | ## Files and shaders diff --git a/dm/games/01007300020FA000/game.tsv b/dm/games/01007300020FA000/game.tsv index 94b51e9..3ba2cb8 100644 --- a/dm/games/01007300020FA000/game.tsv +++ b/dm/games/01007300020FA000/game.tsv @@ -37,3 +37,5 @@ sys_resource 0x0 # ---- Shader split ---- shader_groups /sound/=sound /mv/=mv /miscTex/=misctex /movie/=movie /effect/=effect /Text/=text /ba/=ba /bg/=bg /bh/=bh /bv/=bv /core/=core /em/=em /es/=es /et/=et /event/=event /font/=font /it/=it /ph1/=ph1 /ph2/=ph2 /ph3/=ph3 /ph4/=ph4 /ph5/=ph5 /ph6/=ph6 /ph7/=ph7 shader_note ASTRAL CHAIN shaders +# the NVN depth mode the game sets (nvnDeviceSetDepthMode near = 0): the pipeline archive made offline holds the main0_z vertex entries (dm/gfx/tools/pso_offline.py) +depth_mode zero diff --git a/dm/games/01007EF00011E000/README.md b/dm/games/01007EF00011E000/README.md index 5bdfdff..f15f4c9 100644 --- a/dm/games/01007EF00011E000/README.md +++ b/dm/games/01007EF00011E000/README.md @@ -141,6 +141,8 @@ Every kind of cold start also shows the game's own 130–220 ms frames every 9 | Second start (Metal cache and archive warm) | 0.08 s, no screen | 0 / 0 | | A player's list (5,783 lines with the manifest), warm | 0.23 s, no screen (2.7 s in the background) | 0 / 0 | +- **The pipeline archive made offline** ([development.md](../../../docs/development.md#the-pipeline-archive-made-offline)): every one of the 15,767 pairs makes a base (none left to the device, none left out); `metal-tt` built the iPhone 17 Pro Max's archive (`applegpu_g18p`, iOS 27.0) in 252 s on an M2 Max: 653 MB, 219 MB zipped. On that iPhone (iOS 27.0.1) a test app found all 200 bases of a 200-pair archive, at 0.06–0.3 ms each. On the Mac the same build for the M2 Max (`applegpu_g14s`, 615 MB, 253 s) held all 15,767 bases: the prebuild tool with it opened it in 1.8 s from the external drive, found 16 of 16 probed bases in 57 ms and built nothing, where the device's own prebuild took 187 + 19 s on 7 workers. The process's footprint rose by about 615 MB with the archive open. Resolved on the iPhone 17 Pro Max (iOS 27.0.1): the app with the archive (622.8 MB) opened it in 555 ms, found 16 of 16 probed bases in 17 ms, and its prepare step took 0.53 s with no base compiled (1,722 bases from the archives), where the device's own prebuild took 7–8 minutes on a first start; footprint 1,424 MiB at frame 300 and 2,579–2,679 MiB in play to frame 2,100, with the device's own earlier prebuild (932.7 MB in 30 files of `pso_archive/`, from before the archive was shipped) still open beside it. + On this Mac the background prewarm also ends before the save loads, so the step changes the first minute only where the prewarm is slower than the way into play: a slower device, fewer cores, a longer list, a player who skips through the title. Frames over 25 ms without pipeline work (20–71 per run) are the game's own and vary with the machine's other load. ## Mac routes @@ -164,10 +166,14 @@ General host changes first needed here; each applies to every game: - **`DT_RELR`** (`relr_apply`) and **halfword jump tables** (`half_count`): [Packed relative relocations (DT_RELR)](../../../docs/reference/rewriter.md#packed-relative-relocations-dt_relr), [Halfword jump table landing pads sized to the largest entry](../../../docs/reference/rewriter.md#halfword-jump-table-landing-pads-sized-to-the-largest-entry). **NVN calls on the big host stack** (`nvn_big_entry`): [NVN entry points on the big host stack](../../../docs/reference/nvn-objects.md#nvn-entry-points-on-the-big-host-stack). - **SDK 20 service calls** and **record sizes against `GetCommandMemoryUsed`** (`draw_rec`): [SDK 20 six-axis sensor and amiibo (`nfp`) calls](../../../docs/reference/services.md#sdk-20-six-axis-sensor-and-amiibo-nfp-calls), [Applet resource user ID, launch parameters and VR mode](../../../docs/reference/services.md#applet-resource-user-id-launch-parameters-and-vr-mode), [Shared font load request](../../../docs/reference/services.md#shared-font-load-request), [Model heaps sized by the host's command record sizes](../../../docs/reference/memory.md#model-heaps-sized-by-the-hosts-command-record-sizes). **SDK 20's audio renderer layout** (`render_once`): [SDK 20's `AudioRenderSystem` layout](../../../docs/reference/audio.md#sdk-20s-audiorendersystem-layout). **Running a 30 fps game at 60**: [sead's dynamic present interval, `ksys::VFR` and the host's frame step](../../../docs/reference/frame-rate.md#seads-dynamic-present-interval-ksysvfr-and-the-hosts-frame-step). - **Pipeline bases, hybrid pipelines, the prebuild and the prepare step** (`pso_build_ex`, `hybrid_get`, `pso_prebuild`, `warm_prepare`): [development.md, Hybrid pipelines](../../../docs/development.md#hybrid-pipelines), [Pulling vertex inputs](../../../docs/reference/rendering.md#pulling-vertex-inputs), [the prepare step](../../../docs/development.md#the-prepare-step); measured under [Pipelines on a cold start](#pipelines-on-a-cold-start). +- **The pipeline archive made offline** (`dm/gfx/tools/pso_offline.py`, `pso_bundle_open`, `pso_bundle_covers`): [development.md, The pipeline archive made offline](../../../docs/development.md#the-pipeline-archive-made-offline); measured under [Pipelines on a cold start](#pipelines-on-a-cold-start). +- **Resolved in synthetic archive review checks:** truncated archive containers, zero-probe coverage, stale archives after library bytes change, and plans whose pairs all require device specialization: [Offline pipeline archive reuse and fallback](../../../docs/reference/shaders.md#offline-pipeline-archive-reuse-and-fallback). Checked with generated libraries on the Mac; no new gameplay or iPhone validation. - **Hitch dumps and guest thread names** (`DM_HITCH_DUMP`), **heap probes** (`DM_PROBE_HEAP*`), **autorelease leaks**: [05, Scripted input and thread scheduling](../../../docs/agents/05-debugging.md#scripted-input-and-thread-scheduling), [05, Memory growth](../../../docs/agents/05-debugging.md#memory-growth). ## Open +- The shipped pipeline archive without the device's own earlier prebuild files (the test iPhone's `pso_archive/` still held them, [Pipelines on a cold start](#pipelines-on-a-cold-start)): the footprint against the app's memory limit and the frame times after it. + - The GPU particles' compute dispatches and the G-buffer decals still split passes; about 800 MB of attachments a frame outdoors. - `DM_RENDER_SCALE` does not change the Mac's GPU time; the sources give no iPhone measurement. - The VFR override's `the game fixes its own step` path has not been reached on the Mac. diff --git a/dm/games/01008D7016438000/README.md b/dm/games/01008D7016438000/README.md index 89bcde2..cf3e3c5 100644 --- a/dm/games/01008D7016438000/README.md +++ b/dm/games/01008D7016438000/README.md @@ -9,6 +9,7 @@ Base game 1.00 (v0), SDK 11.4.3: the Japanese / Korean / Chinese cartridge and e | `alloc_big_align on` | [`game.tsv`](game.tsv) | the engine's ring heaps are cut from a memory pool at 64 KB alignment; without it they do not fit the pool ([the western note, Memory](../01003FC01670C000/README.md#memory)) | | 4 GB layout: `mem_reserve 0x100000000`, `mem_alias 0x10000000`, `mem_stack 0x10000000`, `app_pool 0xCD500000` | `game.tsv` | as the western profile: no system resource in the NPDM, the game sets a heap | | `sys_resource 0x0` | `game.tsv` | the NPDM declares no system resource (the NSP script writes it from the NPDM) | +| `depth_mode zero` | `game.tsv` | the game sets the near = 0 depth mode, so the pipeline bases made before it runs (the NSP script's Mac prebuild, and a [pipeline archive made offline](../../../docs/development.md#the-pipeline-archive-made-offline) for the iPhone when one is built with `DM_PSO_GPUS`) take its vertex shaders' `main0_z` entries (the host itself follows the mode the game sets) | No `patches.py`, no `sdk_patches.py` cut for its `sdk` build (`2d369d99…`), no `pad_layout` (the iPhone pad starts in the 2D layout). The rest of the profile is the NSP script's provisional one. diff --git a/dm/games/01008D7016438000/game.tsv b/dm/games/01008D7016438000/game.tsv index 4fc7cf3..304e71c 100644 --- a/dm/games/01008D7016438000/game.tsv +++ b/dm/games/01008D7016438000/game.tsv @@ -35,3 +35,5 @@ shader_groups /=romfs shader_note 13 Sentinels: Aegis Rim (Asia) shaders # from the western 13 Sentinels profile: Vanillaware engine needs 64 KB aligned big allocations alloc_big_align on +# the NVN depth mode the game sets (nvnDeviceSetDepthMode near = 0): the pipeline archive made offline holds the main0_z vertex entries (dm/gfx/tools/pso_offline.py) +depth_mode zero diff --git a/dm/games/0100A6301214E000/README.md b/dm/games/0100A6301214E000/README.md index 3861bf7..2ead93d 100644 --- a/dm/games/0100A6301214E000/README.md +++ b/dm/games/0100A6301214E000/README.md @@ -10,6 +10,7 @@ Update 2.0.0 (v327680) over the base game, SDK 12.3.7. Unity 2020.3.18f1 IL2CPP | Node area cut 1 GB → 62 MB | `dm/tools/sdk_patches.py`, `sdk` build `56ec2474…` (`sdk+0x346430`, `sdk+0x3464cc`) | the alias region ran out of address space minutes into play; see [Memory](#memory) | | 5 GB guest layout: `mem_reserve 0x140000000`, `mem_alias 0x118000000`, `mem_stack 0x10000000`, `app_pool 0xA0000000` | [`game.tsv`](game.tsv) | 256 MB in front + 4.375 GB alias region + 256 MB stack region + 128 MB heap region, with a 2.5 GB pool reported. 5 GB is the total proven on a phone | | `shader_groups` `engine` / `assets` | `game.tsv` | `data.unity3d` and `Resources/` are the engine's built-ins, `StreamingAssets/` the Addressables bundles | +| `depth_mode zero` | `game.tsv` | the game sets the near = 0 depth mode, so the pipeline bases made before it runs (the NSP script's Mac prebuild, and a [pipeline archive made offline](../../../docs/development.md#the-pipeline-archive-made-offline) for the iPhone when one is built with `DM_PSO_GPUS`) take its vertex shaders' `main0_z` entries (the host itself follows the mode the game sets) | No `pad_layout`: the virtual gamepad starts in the 2D layout. `present_interval` stays `game`. diff --git a/dm/games/0100A6301214E000/game.tsv b/dm/games/0100A6301214E000/game.tsv index 65eb426..e1b1cfd 100644 --- a/dm/games/0100A6301214E000/game.tsv +++ b/dm/games/0100A6301214E000/game.tsv @@ -45,3 +45,5 @@ app_pool 0xA0000000 # engine built-ins (data.unity3d, Resources/unity default resources) go to engine, the Addressables bundles to assets shader_groups /data.unity3d=engine /Resources/=engine /StreamingAssets/=assets shader_note Fire Emblem Engage shaders +# the NVN depth mode the game sets (nvnDeviceSetDepthMode near = 0): the pipeline archive made offline holds the main0_z vertex entries (dm/gfx/tools/pso_offline.py) +depth_mode zero diff --git a/dm/games/README.md b/dm/games/README.md index 8e6a24e..edbe90a 100644 --- a/dm/games/README.md +++ b/dm/games/README.md @@ -128,13 +128,14 @@ One row per profile; the notes are each profile's `README.md`. | `mem_map_limit` | `on` / `off` (the default): `MapPhysicalMemory` fails with ResultLimitReached when it would take the process past the application pool (the pool less the used memory `GetInfo` reports), as on Horizon (`hle_svc.c`). Pokémon Legends: Z-A allocates and frees everything it can at startup and the SDK keeps the freed pages mapped until a map fails, then unmaps them and retries; without the limit they stay mapped and the game maps past the pool. Off by default: Ace Attorney Investigations Collection maps a little more than the 1 GB pool its profile reports | | `present_interval` | `game` (the default): frames are paced by the present interval the game sets, as on the Switch (a frame presented with interval 2 stays on screen for two display ticks; a game that keeps asking for 2 for a second gets the 30 fps target, and on iOS the display link then runs at 30 Hz; Mario 1.2.1 switches to 2 for single frames after a slow one; Engage asks for 2 on the Switch, and a rewriter cut on its 2.0.0 `main` makes it ask for 1, see `0100A6301214E000/game.tsv`). `display`: one frame per display tick whatever the game asks, which is only right for games whose logic does not count frames (D2R's runs on its own 25 Hz clock; Monster Train 2 asks for 2 and steps by Unity's frame time, so it runs at 60 fps: screenshots at the same moments match at 30 and 60 fps). `DM_PRESENT_INTERVAL=game` / `display` overrides it, also from `Documents/env.txt` | | `present_snap` | `on` / `off` (the default): present snapshots by default (`nvn_mtl.m` `mtl_snap_on`, a ring of 2 host textures): each present is copied and the game's swapchain texture is free at once, so the GPU does not idle between frames ([Present snapshots](../../docs/reference/frame-rate.md#present-snapshots-the-games-swapchain-texture-free-at-once)). A setting the game was tested with: the iOS launcher's **Frame buffering** defaults to it, and `DM_PRESENT_SNAP` / `Documents/env.txt` override it. Breath of the Wild sets it on | -| `pso_mtl4` | `on` / `off` (the default): on iOS, render pipelines through MTL4Compiler by default, with the pipeline archive, the prepare step, the hybrid pipelines and the prebuild (on by default on the Mac; `DM_PSO_MTL4=1` / `0` overrides). A setting the game was tested with on an iPhone. Breath of the Wild sets it on | +| `pso_mtl4` | `on` / `off` (the default): on iOS, render pipelines through MTL4Compiler by default, with the pipeline archive, the prepare step, the hybrid pipelines and the prebuild (on by default on the Mac; `DM_PSO_MTL4=1` / `0` overrides). A setting the game was tested with on an iPhone. It also gives the app the [pipeline archive made offline](../../docs/development.md#the-pipeline-archive-made-offline) (`build_ipa.sh` builds it for `g18p` unless `DM_PSO_GPUS` says otherwise; without the key, none) and the pulling libraries (`_bp`). Breath of the Wild sets it on | | `fps_target` | Frame rate cap of the rewriter's frame step, for builds that have one (`FRAME_STEP` in the profile's `patches.py`: Tears of the Kingdom 1.4.3, Breath of the Wild 1.9.0, Hyrule Warriors: Age of Calamity 1.3.0, Paper Mario: The Thousand-Year Door 1.0.1, ASTRAL CHAIN 1.0.1), or the engine's own frame limit the host writes into game memory (`hle_fpslock.c`: Monster Hunter Rise 16.0.2, which also needs `present_interval display`; Octopath Traveler 0 1.0.5, Unreal's `t.MaxFPS` and `rhi.SyncInterval`, so the game asks for present interval 1 itself; Luigi's Mansion 3 1.4.0, its frame limiter's milliseconds per frame, with `present_interval display`). `0` (the default): the game's own step. `30` / `60`: the host decides each frame's step in whole vsyncs and keeps the frame on screen that long (`hle_nvn.c` `fstep_tick`, `nvn_fstep.h`): 1 at a 60 cap, 2 when 60 does not hold or at a 30 cap (and while a prerendered movie plays), plus one vsync while the game is behind real time after a late frame. Presents follow that hold instead of the interval the game sets. `DM_FPS_TARGET` overrides it, also from `Documents/env.txt`; the log line `frame step: game time … of real time` checks the game's speed | | `fps_hold` | `on` (the default): with `fps_target`, three late frames in a row raise the hold above the cap's vsyncs (30 or 20 fps), and steady frames that fit try the lower hold again. `off`: frames are never held longer than the cap's vsyncs; each is shown as soon as it is ready. For a game whose updates do not wait for the presents and step by the cap whatever the hold (Age of Calamity): there the hold only paces the screen. `DM_FPS_HOLD=on` / `off` overrides it, also from `Documents/env.txt` Paper Mario: The Thousand-Year Door sets it off too: its frame waits for three render worker threads, and holding frames left them idle, so a frame cost more the longer the hold (the hold stuck at 3 vsyncs, 20 fps, on a loaded Mac; 32–58 fps without holds) | | `core_preempt` | `on` / `off` (the default): additionally check higher-priority runnable threads pinned to the caller's core after every thread SVC (`hle_thread.c`). `DM_PREEMPT=0` / `1` overrides it. Wake-triggered same-core handoffs apply to every game with either setting and wait until the peer blocks, with no wall-clock cap. Metroid Dread no longer needs this key or its movie request-ring gates; see [Same-core wake handoffs](../../docs/reference/threads.md#same-core-wake-handoffs) | | `alloc_big_align` | `on` / `off` (the default): `nn::mem::StandardAllocator::Allocate` aligns blocks of 64 KB or more to 64 KB (`hle_os.c`). Vanillaware's engine (13 Sentinels: Aegis Rim, Unicorn Overlord) takes a memory pool's storage from malloc and cuts ring heaps out of it at 64 KB alignment relative to the pool's GPU address. On the Switch that address is big-page aligned; the host's is the CPU address, only 4 KB aligned, so the last ring heap does not fit and keeps base 0, and the game writes through a null pointer (13 Sentinels entering its first battle, Unicorn Overlord past the title screen) | | `shared_font` | Optional, decided by the game files without it (the SDK version the host reads from the `sdk` module, `loader.c` `dm_sdk_major`, and `main`'s imports, `dm_main_imports`): what the shared-font address points at. `bfttf` (SDK 11 and later, when `main` imports `ScalableFontEngineHelper::Decode`) = the 8-byte BFTTF header (with the "already decoded" marker) followed by the TTF, which Mario passes to `Decode`; `ttf` = the TTF itself, which SDK 10 and older games hand straight to `ScalableFontEngine::LoadFont` (Super Mario 3D World + Bowser's Fury: with the header the engine fails with error 403 and the game crashes looking up a glyph; Odyssey, Link's Awakening, Age of Calamity and Three Houses alike) and Unity's player to FreeType (Patrick's Parabox: with the header its Japanese, Korean and Chinese text drew nothing). No profile sets it | | `device_api` | Optional, decided by the SDK version without it: the NVN API version `major.minor` that `nvnDeviceGetInteger` reports, 55.15 for SDK 11 and later (what Mario requires) and 53.313 for SDK 10 and older (`hle_nvn.c` `nvn_api`). Games check it and skip creating their device on a mismatch: Super Mario 3D World + Bowser's Fury (SDK 10) requires major 53 and minor ≥ 313. No profile sets it. `DM_NVN_API` overrides. Not part of the shader chain version (only `shader_*` keys are) | +| `depth_mode` | `gl` (the default, NVN's near = −w) or `zero` (near = 0): the depth mode the game sets with `nvnDeviceSetDepthMode`, for the pipeline bases made before the game runs: the [pipeline archive made offline](../../docs/development.md#the-pipeline-archive-made-offline) for the iPhone (`main0` or `main0_z` vertex entries; built for a game with `pso_mtl4 on` or with `DM_PSO_GPUS`) and the NSP script's Mac prebuild (`DM_PSO_PREBUILD_DEPTH`). The host itself follows the mode the game sets; a wrong value only costs the device its own prebuild. Great Ace Attorney Chronicles, 13 Sentinels (both releases), Big Brain Academy, Link's Awakening, Engage and ASTRAL CHAIN set `zero` | | `op_mode` | The operation mode the game sees at start: `console` (the default, docked) or `handheld`. Super Mario Galaxy and Captain Toad: Treasure Tracker use `handheld`: docked, their pointer is the Joy-Con gyro, which the host does not have; handheld, it is the touch screen, which the iPhone front end feeds with the touches no gamepad control takes (in the 3D layout, outside the right third of the screen, which is the right stick; `DmGamepad.m`). Handheld also renders at 720p. `DM_OPMODE=0` / `1` overrides, also from `Documents/env.txt` | | `perf_mode` | `follow` (the default): the performance mode follows the operation mode (Boost docked, Normal handheld); `boost`: Boost whatever the operation mode (`hle_oe.c`), a combination a Switch 1 never reports, so a profile choice. Rhythm Heaven Groove plays handheld with Boost: docked means a TV to the game, which then asks for a latency calibration, and only Boost gives its 1920x1080 draw size (handheld Normal draws 1280x720) | | `pad_layout` | The iPhone virtual gamepad's layout on first launch: `simple` (the default) = 2D (left stick, A/B/X/Y, shoulders, − / +), `full` = 3D (adds a D-pad, a right stick and L3 / R3; the right stick is not drawn: a touch landing in the right third of the screen on no control holds it, centred where it lands), `touch` = Tap (no control but the layout button: every touch goes to the game's touch screen; needs `op_mode handheld`, Balatro). The layout button at the top centre goes to the next layout in game (Tap only in handheld mode) and the app remembers the player's choice (`DmGamepad.m`) | diff --git a/dm/games/game.py b/dm/games/game.py index fc99a2f..5527a66 100644 --- a/dm/games/game.py +++ b/dm/games/game.py @@ -1,6 +1,6 @@ #!/usr/bin/env python3 # Game profiles: dm/games//game.tsv (key values..., lines starting with # are comments) + the shader tables (and patches.py, the rewriter's patches of the game's builds) in the same folder. The shader tools, the host build, iOS packaging and the NSP script take each game's constants from here. -# Usage: game.py list (list all profiles); game.py (print that key's value, for shell); game.py header (generate the host's game_profile.h); +# Usage: game.py list (list all profiles); game.py [] (print that key's value, for shell; without a default a missing key is an error); game.py header (generate the host's game_profile.h); # game.py dir (print the profile folder); game.py link (point a symlink at the profile folder: make cannot take paths with spaces) # From Python: import game; g = game.load() (takes DM_GAME; there is no default game: unset is an error that lists the profiles); game.by_title(title ID); g.get / g.words / g.int / g.builds() / g.table(file name) # DM_GAMES_DIR (folders separated by ':') adds profile folders outside the repository, searched after dm/games: the NSP script writes a provisional profile for a game without one into @@ -132,6 +132,8 @@ def header(g): if pl == 'touch' and om != 'handheld': sys.exit(f'{g.id}: pad_layout touch needs op_mode handheld (the touch screen reaches the game only in handheld mode)') pm = g.get('perf_mode', 'follow') # the performance mode reported (hle_oe.c): follow = the operation mode's (Boost docked, Normal handheld), boost = Boost always if pm not in ('follow', 'boost'): sys.exit(f'{g.id}: perf_mode must be follow or boost: {pm}') + dmo = g.get('depth_mode', 'gl') # the NVN depth mode the game sets, for the pipeline archive made offline (dm/gfx/tools/pso_offline.py; the host follows the game at run time): gl / zero + if dmo not in ('gl', 'zero'): sys.exit(f'{g.id}: depth_mode must be gl or zero: {dmo}') api = g.get('device_api', '55.15') # the NVN API major.minor version nvnDeviceGetInteger reports (hle_nvn.c) if not re.fullmatch(r'\d+\.\d+', api): sys.exit(f'{g.id}: device_api must be major.minor: {api}') src = f'dm/games/{g.id}' if g.dir.parent == HERE else str(g.dir) @@ -191,7 +193,7 @@ if __name__ == '__main__': if not (ln.is_symlink() and Path(os.readlink(ln)) == d): if ln.is_symlink() or ln.exists(): ln.unlink() ln.symlink_to(d) - elif len(a) == 2: - print(load(a[0]).get(a[1])) + elif len(a) in (2, 3): + print(load(a[0]).get(*a[1:])) else: - sys.exit(__doc__ or 'usage: game.py list | game.py | game.py header | game.py dir | game.py link ') + sys.exit(__doc__ or 'usage: game.py list | game.py [] | game.py header | game.py dir | game.py link ') diff --git a/dm/gfx/README.md b/dm/gfx/README.md index d5924d4..8feec8d 100644 --- a/dm/gfx/README.md +++ b/dm/gfx/README.md @@ -158,6 +158,7 @@ Resource locations: | `DM_PSO_HYBRID=0` | A draw whose pipeline is not built yet waits for it, instead of drawing with the pair's pulling pipeline meanwhile (docs/development.md, "Hybrid pipelines") | | `DM_PSO_PREBUILD=0` or `tool` | No prebuild of the pulling bases into the archive (in the prepare step, or in the background without one); `tool`: run the prebuild alone and exit (`DM_PSO_PREBUILD_DEPTH=zero`, `DM_PSO_PREBUILD_THREADS=`) | | `DM_PSO_PAIRS=path` | The prebuild's second pair table (default: `pso_pairs.tsv` beside the shader map) | +| `DM_PSO_BUNDLE=folder` or `off` | The pipeline archive made offline (`dm/gfx/tools/pso_offline.py`): `.mtl4a` and `prebuilt.tsv`, read only; when it covers the shader pairs the device builds no prebuild bases. The iOS app sets it to its `pso/` folder when it has one (docs/development.md, "The pipeline archive made offline") | | `DM_PSO_WAITLOG=1` | Log each draw that waited more than 20 ms for its pipeline, with why no stand-in served it | | `DM_PSO_PULL_ALL=1` | Diagnostic: every draw that has a pulling pipeline draws with it | diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index 0fc551c..b2ad664 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -52,6 +52,8 @@ static uint64_t c2_handle(int stage, unsigned b); // the handle c2 holds at by #import #include #include +#include +#include #include #include #include @@ -4458,8 +4460,8 @@ static MTLRenderPipelineDescriptor *pso_desc(const pso_key *k, NvShader *vs, NvS // taken from an archive costs about 0.06 ms, against 10-80 ms to build one. // DM_PSO_MTL4=0: the MTLDevice path as before (no bases, no archive). DM_PSO_ARCHIVE=off: no archive; DM_PSO_ARCHIVE=: the archive in that folder. // On by default on the Mac; on iOS DM_PSO_MTL4=1, or the profile key pso_mtl4 on, turns it on, and with it the hybrid pipelines and the pipeline prebuild. The archive's footprint -// (with the prebuild, about 850 MB more for Breath of the Wild) is not measured against an iPhone's limit yet, and an iPhone cannot be given an archive made offline (metal-tt -// rejects the Metal 4 serializer's pipeline script) +// (with the prebuild, about 850 MB more for Breath of the Wild) is not measured against an iPhone's limit yet. The prebuild's bases can also come made offline, bundled with the +// app (DM_PSO_BUNDLE, pso_bundle_open) static int pso4_on(void) { static int v = -1; if (v < 0) { @@ -4501,6 +4503,93 @@ static nvn_hmap g_pbase = {.klen = sizeof(pbase_key)}; static nvn_hmap g_pbase_pending = {.klen = sizeof(pbase_key)}; static os_unfair_lock g_pbase_lock = OS_UNFAIR_LOCK_INIT; static NSString *pso_user_path(void); +// The pipeline archive made offline (dm/gfx/tools/pso_offline.py, metal-tt) and bundled with the app: DM_PSO_BUNDLE= (the iOS app sets it to its pso/ folder when it has one; +// off: not used) holds .mtl4a per architecture it was built for and prebuilt.tsv, what it covers (pso_prebuild). The file named after this device's architecture is +// opened read only and looked up first among the files of bases without a vertex layout; it is never written or compacted. The device's own archive (pso_archive/) so takes only +// what it lacks: a draw, the prewarm and the prebuild compile a base, and the serializer captures it, only after the bundle missed (it holds no base with a vertex layout). A system +// that cannot open it (one older than the archive's minimum, or an OS update that changed the GPU compiler) or a GPU it was not built for gets an info line, and the device builds +// the bases itself as before (the on-device prebuild is the fallback) +static id g_arch_bundle; static NSString *g_bundle_dir; +// Metal's loader crashes on malformed containers rather than returning an error. Check the GPU image's load commands and referenced file ranges, including every slice of a +// universal archive, before giving it to Metal. This checks the container, not the compiler's private payload: Metal still decides whether this system can use its binaries. +static int arch_macho_ok(int fd, uint64_t off, uint64_t size) { + struct mach_header_64 h; + if (size < sizeof h || pread(fd, &h, sizeof h, (off_t)off) != sizeof h || h.magic != MH_MAGIC_64 || h.filetype != MH_GPU_EXECUTE || + !h.ncmds || h.sizeofcmds > size - sizeof h || h.ncmds > h.sizeofcmds / sizeof(struct load_command)) return 0; + uint64_t pos = sizeof h, end = pos + h.sizeofcmds; int pipeline = 0; + for (uint32_t i = 0; i < h.ncmds; i++) { + struct load_command lc; + if (end - pos < sizeof lc || pread(fd, &lc, sizeof lc, (off_t)(off + pos)) != sizeof lc || lc.cmdsize < sizeof lc || lc.cmdsize > end - pos) return 0; + if (lc.cmd == LC_SEGMENT_64) { + struct segment_command_64 s; + if (lc.cmdsize < sizeof s || pread(fd, &s, sizeof s, (off_t)(off + pos)) != sizeof s || s.fileoff > size || s.filesize > size - s.fileoff || + s.nsects > (lc.cmdsize - sizeof s) / sizeof(struct section_64)) return 0; + } else if (lc.cmd == LC_NOTE) { + struct note_command n; + if (lc.cmdsize < sizeof n || pread(fd, &n, sizeof n, (off_t)(off + pos)) != sizeof n || n.offset > size || n.size > size - n.offset) return 0; + if (n.size && !memcmp(n.data_owner, "AIR_PIPELINE", sizeof "AIR_PIPELINE")) pipeline = 1; + } + pos += lc.cmdsize; + } + return pos == end && pipeline; +} +static int arch_container_ok(int fd, uint64_t size) { + struct fat_header h; + if (size < sizeof h || pread(fd, &h, sizeof h, 0) != sizeof h) return 0; + if (h.magic == MH_MAGIC_64) return arch_macho_ok(fd, 0, size); + // The serializer's 0xcbfebabe wrapper uses fat_arch entries and includes a Metal library slice beside the GPU images. + int metal = h.magic == OSSwapBigToHostInt32(0xcbfebabe), wide = h.magic == OSSwapBigToHostInt32(FAT_MAGIC_64); + if (!metal && !wide && h.magic != OSSwapBigToHostInt32(FAT_MAGIC)) return 0; + uint32_t n = OSSwapBigToHostInt32(h.nfat_arch); uint64_t ent = wide ? sizeof(struct fat_arch_64) : sizeof(struct fat_arch); + if (!n || n > (size - sizeof h) / ent) return 0; + uint64_t table = sizeof h + n * ent; int gpu = 0; + for (uint32_t i = 0; i < n; i++) { + uint64_t off, len; + if (wide) { + struct fat_arch_64 a; if (pread(fd, &a, sizeof a, (off_t)(sizeof h + i * ent)) != sizeof a) return 0; + off = OSSwapBigToHostInt64(a.offset); len = OSSwapBigToHostInt64(a.size); + } else { + struct fat_arch a; if (pread(fd, &a, sizeof a, (off_t)(sizeof h + i * ent)) != sizeof a) return 0; + off = OSSwapBigToHostInt32(a.offset); len = OSSwapBigToHostInt32(a.size); + } + if (off < table || off > size || len > size - off) return 0; + uint32_t magic; + if (len < sizeof magic || pread(fd, &magic, sizeof magic, (off_t)off) != sizeof magic) return 0; + if (metal && magic == 0x424c544d) { // MTLB: its header declares the full slice length and four section ranges + uint64_t lib[11]; + if (len < sizeof lib || pread(fd, lib, sizeof lib, (off_t)off) != sizeof lib || lib[2] != len) return 0; + for (int j = 3; j < 11; j += 2) if (lib[j] > len || lib[j + 1] > len - lib[j]) return 0; + } else { + if (!arch_macho_ok(fd, off, len)) return 0; + gpu = 1; + } + } + return gpu; +} +static int arch_file_ok(NSString *path) { + int fd = open(path.fileSystemRepresentation, O_RDONLY); if (fd < 0) return 0; + struct stat st; int ok = !fstat(fd, &st) && S_ISREG(st.st_mode) && st.st_size > 0 && arch_container_ok(fd, (uint64_t)st.st_size); + close(fd); return ok; +} +static void pso_bundle_open(NSMutableArray> *archp) { + const char *e = getenv("DM_PSO_BUNDLE"); + if (!e || !*e || !strcmp(e, "off")) return; + NSString *dir = @(e), *gpu = [(id)g_dev respondsToSelector:@selector(architecture)] ? g_dev.architecture.name : nil; + NSString *p = gpu ? [dir stringByAppendingPathComponent:[gpu stringByAppendingPathExtension:@"mtl4a"]] : nil; + if (!p || ![NSFileManager.defaultManager fileExistsAtPath:p]) { + NSArray *have = [[NSFileManager.defaultManager contentsOfDirectoryAtPath:dir error:nil] filteredArrayUsingPredicate:[NSPredicate predicateWithFormat:@"self ENDSWITH '.mtl4a'"]]; + MLOG("pipeline archive bundled with the app: none for this GPU (%s; %s has %s): the bases are built on the device", gpu ? gpu.UTF8String : "architecture unknown", e, + have.count ? [have componentsJoinedByString:@", "].UTF8String : "none"); + return; + } + uint64_t t0 = bp_now(); NSError *ae = nil; + if (!arch_file_ok(p)) { MLOG("[RED] pipeline archive bundled with the app: %s is not a pipeline archive: the bases are built on the device", p.UTF8String); return; } + id a = [g_dev newArchiveWithURL:[NSURL fileURLWithPath:p] error:&ae]; + if (!a) { MLOG("pipeline archive bundled with the app: this system cannot open %s (%s; made for another system version, or an OS update changed the GPU compiler): the bases are built on the device", + p.UTF8String, ae.localizedDescription.UTF8String); return; } + g_arch_bundle = a; g_bundle_dir = dir; [archp addObject:a]; + MLOG("pipeline archive bundled with the app: %s, %.1f MB, opened in %.1f ms", p.UTF8String, [[NSFileManager.defaultManager attributesOfItemAtPath:p error:nil] fileSize] / 1048576.0, (bp_now() - t0) / 1e6); +} static void pso4_init(void) { static dispatch_once_t once; dispatch_once(&once, ^{ @@ -4508,27 +4597,29 @@ static void pso4_init(void) { const char *e = getenv("DM_PSO_ARCHIVE"); NSString *user = pso_user_path(); g_arch_dir = e && !strcmp(e, "off") ? nil : e && *e ? @(e) : user ? [user.stringByDeletingLastPathComponent stringByAppendingPathComponent:@"pso_archive"] : nil; id ser = nil; + NSMutableArray> *arch = [NSMutableArray array], *archp = [NSMutableArray array]; + pso_bundle_open(archp); // first: it holds every pair's base if (g_arch_dir) { g_arch_q = dispatch_queue_create("dm.nvn.pso-archive", dispatch_queue_attr_make_with_qos_class(DISPATCH_QUEUE_SERIAL, QOS_CLASS_UTILITY, 0)); MTL4PipelineDataSetSerializerDescriptor *sd = [MTL4PipelineDataSetSerializerDescriptor new]; sd.configuration = MTL4PipelineDataSetSerializerConfigurationCaptureBinaries; ser = [g_dev newPipelineDataSetSerializerWithDescriptor:sd]; - NSMutableArray> *arch = [NSMutableArray array], *archp = [NSMutableArray array]; uint64_t t0 = bp_now(); unsigned long long bytes = 0; + uint64_t t0 = bp_now(); unsigned long long bytes = 0; NSMutableArray *runf = [NSMutableArray array]; NSArray *names = [[[NSFileManager.defaultManager contentsOfDirectoryAtPath:g_arch_dir error:nil] filteredArrayUsingPredicate:[NSPredicate predicateWithFormat:@"self ENDSWITH '.mtl4a'"]] sortedArrayUsingSelector:@selector(compare:)]; for (NSString *n in names.reverseObjectEnumerator) { // newest first NSString *p = [g_arch_dir stringByAppendingPathComponent:n]; NSError *ae = nil; - id a = [g_dev newArchiveWithURL:[NSURL fileURLWithPath:p] error:&ae]; + id a = arch_file_ok(p) ? [g_dev newArchiveWithURL:[NSURL fileURLWithPath:p] error:&ae] : nil; int pf = [n hasPrefix:@"p"]; if (a) { [pf ? archp : arch addObject:a]; if (!pf) [runf addObject:p]; bytes += [[NSFileManager.defaultManager attributesOfItemAtPath:p error:nil] fileSize]; } - else MLOG("[RED] pipeline archive file %s not usable (%s): its bases are built again", p.UTF8String, ae.localizedDescription.UTF8String); + else MLOG("[RED] pipeline archive file %s not usable (%s): its bases are built again", p.UTF8String, ae ? ae.localizedDescription.UTF8String : "not a pipeline archive"); g_arch_seq = MAX(g_arch_seq, [n substringFromIndex:pf].intValue + 1); } - g_arch4 = arch; g_arch4p = archp; if (runf.count > PSO_ARCH_COMPACT) { g_compact_opts = [MTL4CompilerTaskOptions new]; g_compact_opts.lookupArchives = arch; g_compact_files = runf; } - MLOG("pipeline archive %s: %lu files (%lu of bases without a vertex layout), %.1f MB, opened in %.1f ms", g_arch_dir.UTF8String, (unsigned long)(arch.count + archp.count), - (unsigned long)archp.count, bytes / 1048576.0, (bp_now() - t0) / 1e6); + MLOG("pipeline archive %s: %lu files (%lu of bases without a vertex layout), %.1f MB, opened in %.1f ms", g_arch_dir.UTF8String, (unsigned long)(arch.count + archp.count - (g_arch_bundle != nil)), + (unsigned long)archp.count - (g_arch_bundle != nil), bytes / 1048576.0, (bp_now() - t0) / 1e6); } + g_arch4 = arch; g_arch4p = archp; MTL4CompilerDescriptor *cd = [MTL4CompilerDescriptor new]; cd.pipelineDataSetSerializer = ser; NSError *ce = nil; g_comp4 = [g_dev newCompilerWithDescriptor:cd error:&ce]; if (!g_comp4) MLOG("[RED] no Metal 4 compiler (%s): render pipelines are built through the MTLDevice", ce.localizedDescription.UTF8String); @@ -4837,7 +4928,8 @@ void mtl_pso_line(char *o, size_t n) { // there (the archive matches libraries by content). Bases already in the archive are skipped; what is built goes into the archive, flushed every 1,000 bases (the serializer holds // what it captured until then). Runs once per chain version, depth mode and pair list (pso_archive/prebuilt.tsv): in the app after the prewarm, on a background queue at utility QoS // (DM_PSO_PREBUILD=0: not at all), or as a tool on the Mac (DM_PSO_PREBUILD=tool: build, write, exit). Vertex shaders that write the layer, pairs whose vertex shader has stage -// inputs but no pulling library, and integer-output or multisampled bases are left to the draws +// inputs but no pulling library, and integer-output or multisampled bases are left to the draws. An app that bundles these bases made offline (DM_PSO_BUNDLE, pso_bundle_covers) +// builds none of them, only the pairs that archive left to the device (rest_pairs.tsv beside it: functions with constants) static _Atomic int g_pre_running; static _Atomic int g_pre_done, g_pre_built, g_pre_archived, g_pre_skipped, g_pre_failed; static MTL4FunctionDescriptor *pre_fn(id lib, NSString *name) { // as shader_load notes them (fn4_note): constants specialized with an empty set id f = [lib newFunctionWithName:name]; if (!f) return nil; @@ -4846,36 +4938,84 @@ static MTL4FunctionDescriptor *pre_fn(id lib, NSString *name) { // MTL4SpecializedFunctionDescriptor *sd = [MTL4SpecializedFunctionDescriptor new]; sd.functionDescriptor = ld; sd.constantValues = [MTLFunctionConstantValues new]; return sd; } static NSDictionary *pre_json(NSString *key) { NSData *d = [NSData dataWithContentsOfFile:[NSString stringWithFormat:@"%@/%@_b.json", g_msl_dir, key]]; return d ? [NSJSONSerialization JSONObjectWithData:d options:0 error:nil] : nil; } +// The base of one pair as the draws make it (pso_desc_pull / pso_desc), or nil when the pair is left to the draws. Call inside an autorelease pool +static MTL4RenderPipelineDescriptor *pre_desc(NSString *vk, NSString *fk, int zero) { + NSDictionary *vj = pre_json(vk), *fj = pre_json(fk); + if (!vj || !fj || ![vj[@"chain"] isEqual:@DMSH_CHAIN] || ![fj[@"chain"] isEqual:@DMSH_CHAIN]) return nil; + NSString *ventry = zero && vj[@"entry_z"] ? vj[@"entry_z"] : vj[@"entry"], *fentry = fj[@"entry"] ?: @"main0"; + int pull = [vj[@"pull"] isKindOfClass:[NSDictionary class]]; + if (!pull && [vj[@"vertex_inputs"] count]) return nil; // stage inputs and no pulling library: its pipelines need the draw's layout + NSError *e = nil; + id vl = lib_from_file([NSString stringWithFormat:@"%@/%@%@.metallib", g_msl_dir, vk, pull ? @"_bp" : @"_b"], &e); + if (!vl || t_lib_layer) return nil; // writes the layer: its pipelines take an input primitive class (pso_desc), no pulling pipeline + id fl = lib_from_file([NSString stringWithFormat:@"%@/%@_b.metallib", g_msl_dir, fk], &e); + MTL4FunctionDescriptor *vf = pre_fn(vl, ventry), *ff = fl ? pre_fn(fl, fentry) : nil; + if (!vf || !ff) return nil; + MTL4RenderPipelineDescriptor *m = [MTL4RenderPipelineDescriptor new]; + m.vertexFunctionDescriptor = vf; m.fragmentFunctionDescriptor = ff; m.vertexDescriptor = [MTLVertexDescriptor vertexDescriptor]; m.rasterSampleCount = 1; + for (int i = 0; i < 8; i++) { MTL4RenderPipelineColorAttachmentDescriptor *a = m.colorAttachments[i]; + a.pixelFormat = MTLPixelFormatUnspecialized; a.writeMask = MTLColorWriteMaskUnspecialized; a.blendingState = MTL4BlendStateUnspecialized; + a.sourceRGBBlendFactor = a.destinationRGBBlendFactor = a.sourceAlphaBlendFactor = a.destinationAlphaBlendFactor = MTLBlendFactorUnspecialized; + a.rgbBlendOperation = a.alphaBlendOperation = MTLBlendOperationUnspecialized; } + return m; +} static void pre_pair(NSString *vk, NSString *fk, int zero) { @autoreleasepool { - NSDictionary *vj = pre_json(vk), *fj = pre_json(fk); - if (!vj || !fj || ![vj[@"chain"] isEqual:@DMSH_CHAIN] || ![fj[@"chain"] isEqual:@DMSH_CHAIN]) { g_pre_skipped++; return; } - NSString *ventry = zero && vj[@"entry_z"] ? vj[@"entry_z"] : vj[@"entry"], *fentry = fj[@"entry"] ?: @"main0"; - int pull = [vj[@"pull"] isKindOfClass:[NSDictionary class]]; - if (!pull && [vj[@"vertex_inputs"] count]) { g_pre_skipped++; return; } // stage inputs and no pulling library: its pipelines need the draw's layout - NSError *e = nil; - id vl = lib_from_file([NSString stringWithFormat:@"%@/%@%@.metallib", g_msl_dir, vk, pull ? @"_bp" : @"_b"], &e); - if (!vl || t_lib_layer) { g_pre_skipped++; return; } // writes the layer: its pipelines take an input primitive class (pso_desc), no pulling pipeline - id fl = lib_from_file([NSString stringWithFormat:@"%@/%@_b.metallib", g_msl_dir, fk], &e); - MTL4FunctionDescriptor *vf = pre_fn(vl, ventry), *ff = fl ? pre_fn(fl, fentry) : nil; - if (!vf || !ff) { g_pre_skipped++; return; } - MTL4RenderPipelineDescriptor *m = [MTL4RenderPipelineDescriptor new]; - m.vertexFunctionDescriptor = vf; m.fragmentFunctionDescriptor = ff; m.vertexDescriptor = [MTLVertexDescriptor vertexDescriptor]; m.rasterSampleCount = 1; - for (int i = 0; i < 8; i++) { MTL4RenderPipelineColorAttachmentDescriptor *a = m.colorAttachments[i]; - a.pixelFormat = MTLPixelFormatUnspecialized; a.writeMask = MTLColorWriteMaskUnspecialized; a.blendingState = MTL4BlendStateUnspecialized; - a.sourceRGBBlendFactor = a.destinationRGBBlendFactor = a.sourceAlphaBlendFactor = a.destinationAlphaBlendFactor = MTLBlendFactorUnspecialized; - a.rgbBlendOperation = a.alphaBlendOperation = MTLBlendOperationUnspecialized; } + MTL4RenderPipelineDescriptor *m = pre_desc(vk, fk, zero); + if (!m) { g_pre_skipped++; return; } for (id a in g_arch4p) if ([a newRenderPipelineStateWithDescriptor:m error:nil]) { g_pre_archived++; return; } for (id a in g_arch4) if ([a newRenderPipelineStateWithDescriptor:m error:nil]) { g_pre_archived++; return; } + NSError *e = nil; pthread_rwlock_rdlock(&g_arch_rw); id ps = [g_comp4 newRenderPipelineStateWithDescriptor:m compilerTaskOptions:nil error:&e]; pthread_rwlock_unlock(&g_arch_rw); if (ps) { g_pre_built++; atomic_fetch_add(&g_arch_new, 1); } else { if (g_pre_failed++ < 4) MLOG("[RED] pipeline prebuild: the base of %s + %s failed: %s", vk.UTF8String, fk.UTF8String, e.localizedDescription.UTF8String); } } } +// What a prebuild covers (prebuilt.tsv): the chain version, the depth mode, and the pair lists' length in bytes and SHA-256 (first 16 hex digits). dm/gfx/tools/pso_offline.py writes +// the same for the archive it makes offline, from the same files +static NSString *pre_stamp(NSString *txt, int zero) { + NSData *d = [txt dataUsingEncoding:NSUTF8StringEncoding]; uint8_t h[CC_SHA256_DIGEST_LENGTH]; CC_SHA256(d.bytes, (CC_LONG)d.length, h); + return [NSString stringWithFormat:@"chain\t%s\ndepth\t%s\npairs\t%lu\t%02x%02x%02x%02x%02x%02x%02x%02x\n", DMSH_CHAIN, zero ? "near=0" : "near=-w", (unsigned long)d.length, + h[0], h[1], h[2], h[3], h[4], h[5], h[6], h[7]]; +} +static NSArray *> *pre_parse(NSString *txt) { // (vertex key, pixel key) per line, without empty lines, comments and repeated lines + NSMutableArray *> *pairs = [NSMutableArray array]; NSMutableSet *seen = [NSMutableSet set]; + for (NSString *l in [txt componentsSeparatedByString:@"\n"]) { if (!l.length || [l hasPrefix:@"#"] || [seen containsObject:l]) continue; [seen addObject:l]; NSArray *c = [l componentsSeparatedByString:@"\t"]; if (c.count >= 2) [pairs addObject:c]; } + return pairs; +} +// Whether the archive bundled with the app (pso_bundle_open) holds this run's prebuild: its prebuilt.tsv is the stamp the device would write for these pairs and this depth mode, +// and a probe of up to 16 pairs spread over the list (not those it left to the device, rest) finds every base it makes in it (an archive that opens but whose binaries this system +// does not take finds none) +enum { PRE_PROBE = 16 }; +static const char *stamp_line(NSString *st) { // a stamp on one line, for the log + return [[[st stringByTrimmingCharactersInSet:NSCharacterSet.newlineCharacterSet] stringByReplacingOccurrencesOfString:@"\n" withString:@"; "] stringByReplacingOccurrencesOfString:@"\t" withString:@" "].UTF8String; +} +static int pso_bundle_covers(NSArray *> *pairs, NSString *want, int zero, NSString *rest) { + NSString *have = [NSString stringWithContentsOfFile:[g_bundle_dir stringByAppendingPathComponent:@"prebuilt.tsv"] encoding:NSUTF8StringEncoding error:nil]; + if (![have isEqualToString:want]) { + MLOG("pipeline archive bundled with the app: made for %s, this run needs %s: the device builds the bases it does not hold%s", have ? stamp_line(have) : "(no prebuilt.tsv)", stamp_line(want), + have && [have rangeOfString:zero ? @"depth\tnear=-w" : @"depth\tnear=0"].location != NSNotFound ? " (the game's depth mode is not the one built for: the profile key depth_mode)" : ""); + return 0; + } + NSMutableSet *> *left = [NSMutableSet setWithArray:pre_parse(rest ?: @"")]; + NSMutableArray *> *cand = [NSMutableArray array]; + for (NSArray *p in pairs) if (![left containsObject:@[p[0], p[1]]]) [cand addObject:p]; + uint64_t t0 = bp_now(); int probed = 0, found = 0; NSUInteger n = cand.count; + for (NSUInteger k = 0; k < PRE_PROBE && k < n; k++) @autoreleasepool { + NSArray *p = cand[k * n / MIN(n, (NSUInteger)PRE_PROBE)]; + MTL4RenderPipelineDescriptor *m = pre_desc(p[0], p[1], zero); + if (!m) continue; + probed++; found += [g_arch_bundle newRenderPipelineStateWithDescriptor:m error:nil] != nil; + } + int ok = probed > 0 && found == probed; // skipped samples give no evidence that this system can take a base from the archive + MLOG("pipeline archive bundled with the app: %s (%d of %d probed bases found, %.1f ms)%s", ok ? "holds the bases of every shader pair" : "this system does not take its bases", + found, probed, (bp_now() - t0) / 1e6, ok ? "" : ": the device builds them"); + return ok; +} // threads: workers; wait: run on the calling thread until done (the tool), else in the background. Returns the number of pairs, -1 when there is nothing to do static int pso_prebuild(int threads, int wait, dispatch_qos_class_t qos) { shmap_load(); pso4_init(); - if (!g_comp4 || !g_ser4 || !g_arch_dir || atomic_exchange(&g_pre_running, 1)) return -1; + if (!g_comp4 || (!g_arch_bundle && (!g_ser4 || !g_arch_dir)) || atomic_exchange(&g_pre_running, 1)) return -1; // The pairs: shader_pairs.tsv (BNSH variations and the other readers' pairs) and pso_pairs.tsv (agl BAHS program variations, which vs_pad does not read; DM_PSO_PAIRS: another path), // both beside the shader map NSString *pp = [g_shmap_path.stringByDeletingLastPathComponent stringByAppendingPathComponent:@"shader_pairs.tsv"]; @@ -4885,12 +5025,16 @@ static int pso_prebuild(int threads, int wait, dispatch_qos_class_t qos) { if (!txt) { MLOG("pipeline prebuild: no %s, nothing to build", pp.UTF8String); g_pre_running = 0; return -1; } if (txt2) txt = [txt stringByAppendingString:txt2]; int zero = atomic_load(&g_depth_zero); + NSString *want = pre_stamp(txt, zero); + NSArray *> *pairs = pre_parse(txt); + NSString *rp = g_arch_bundle ? [g_bundle_dir stringByAppendingPathComponent:@"rest_pairs.tsv"] : nil, *rest = rp ? [NSString stringWithContentsOfFile:rp encoding:NSUTF8StringEncoding error:nil] : nil; + if (g_arch_bundle && pso_bundle_covers(pairs, want, zero, rest)) { // the archive made offline holds them: only the pairs it left to the device (function constants) are built here + if (!rest.length) { g_pre_running = 0; return -1; } + txt = rest; want = pre_stamp(rest, zero); pairs = pre_parse(rest); pp = rp; txt2 = nil; + } + if (!g_ser4 || !g_arch_dir) { g_pre_running = 0; return -1; } NSString *stamp = [g_arch_dir stringByAppendingPathComponent:@"prebuilt.tsv"]; - NSString *want = [NSString stringWithFormat:@"chain\t%s\ndepth\t%s\npairs\t%lu\t%lx\n", DMSH_CHAIN, zero ? "near=0" : "near=-w", (unsigned long)txt.length, (unsigned long)txt.hash]; if ([[NSString stringWithContentsOfFile:stamp encoding:NSUTF8StringEncoding error:nil] isEqualToString:want]) { g_pre_running = 0; return -1; } - NSMutableArray *> *pairs = [NSMutableArray array]; - NSMutableSet *seen = [NSMutableSet set]; - for (NSString *l in [txt componentsSeparatedByString:@"\n"]) { if (!l.length || [l hasPrefix:@"#"] || [seen containsObject:l]) continue; [seen addObject:l]; NSArray *c = [l componentsSeparatedByString:@"\t"]; if (c.count >= 2) [pairs addObject:c]; } int n = (int)pairs.count; uint64_t t0 = bp_now(); MLOG("pipeline prebuild: %d shader pairs of %s%s%s, %d workers (%s)", n, pp.UTF8String, txt2 ? " and " : "", txt2 ? pp2.UTF8String : "", threads, wait ? "tool" : prepare_on() ? "prepare step, before the first frame" : "background"); __block _Atomic int next = 0; @@ -4924,7 +5068,7 @@ int mtl_pso_prebuild_tool(void) { mtl_init(); const char *z = getenv("DM_PSO_PREBUILD_DEPTH"); if (z && !strcmp(z, "zero")) atomic_store(&g_depth_zero, 1); int threads = getenv("DM_PSO_PREBUILD_THREADS") ? atoi(getenv("DM_PSO_PREBUILD_THREADS")) : 0; if (threads <= 0) { int pc = 0; size_t sz = sizeof pc; if (sysctlbyname("hw.perflevel0.physicalcpu", &pc, &sz, NULL, 0) || pc <= 1) pc = 4; threads = pc - 1; } - if (pso_prebuild(threads, 1, QOS_CLASS_USER_INITIATED) < 0) MLOG("pipeline prebuild: nothing to do (no archive, no shader_pairs.tsv, or prebuilt.tsv says it is done)"); + if (pso_prebuild(threads, 1, QOS_CLASS_USER_INITIATED) < 0) MLOG("pipeline prebuild: nothing to do (no archive, no shader_pairs.tsv, prebuilt.tsv says it is done, or the archive bundled with the app holds every base)"); return 1; } static int prepare_on(void); diff --git a/dm/gfx/tests/.gitignore b/dm/gfx/tests/.gitignore index 120757a..8b2d51d 100644 --- a/dm/gfx/tests/.gitignore +++ b/dm/gfx/tests/.gitignore @@ -97,7 +97,9 @@ t_upgrade_alias.dSYM/ t_stencil_view t_stencil_view.dSYM/ t_pso_archive +t_pso_bundle t_pso_archive.dSYM/ +t_pso_bundle.dSYM/ t_pso_warm t_pso_warm.dSYM/ t_depth16_clear @@ -108,3 +110,5 @@ t_pull t_pull.dSYM/ t_pull_inv t_pull_inv.dSYM/ +t_present_snap +t_present_snap.dSYM/ diff --git a/dm/gfx/tests/Makefile b/dm/gfx/tests/Makefile index 2923ff1..9b41c38 100644 --- a/dm/gfx/tests/Makefile +++ b/dm/gfx/tests/Makefile @@ -140,6 +140,8 @@ t_pull_inv: t_pull_inv.m ../tools/dmsh_pull.py ../tools/dmsh_pull.metal clang -O1 -g -Wall -fobjc-arc -o $@ t_pull_inv.m -framework Metal -framework Foundation t_pso_archive: t_pso_archive.m $(MTL_T_DEP) clang $(CFLAGS) -fobjc-arc -o $@ t_pso_archive.m $(MTL_T_SRC) $(FW) +t_pso_bundle: t_pso_bundle.m ../tools/pso_offline.py $(MTL_T_DEP) + clang $(CFLAGS) -fobjc-arc -o $@ t_pso_bundle.m $(MTL_T_SRC) $(FW) t_pso_warm: t_pso_warm.m $(MTL_T_DEP) clang $(CFLAGS) -fobjc-arc -o $@ t_pso_warm.m $(MTL_T_SRC) $(FW) .PHONY: check_t_pso_warm @@ -196,7 +198,7 @@ t_stencil_view: t_stencil_view.m $(MTL_T_DEP) # The Vulkan layer (#included) over the whole NVN layer, whose device and present path it uses t_vk_desc: t_vk_desc.m ../vk/hle_vk.m $(MTL_T_DEP) clang $(CFLAGS) -I../../third_party/vulkan-headers/include -fobjc-arc -o $@ t_vk_desc.m $(MTL_T_SRC) ../nvn/nvn_mtl.m $(FW) -run: t_nvn t_dirty_slots t_dirty_race t_tex_layout t_builtin t_vin t_tbuf2d t_deswz_gpu t_pace t_timestamp t_fstep t_refl t_hmap t_cover t_fuse_check t_present_crop t_reorder_chain t_defer_chain t_zskip t_xv t_ub t_vbuf t_geometry t_mip_extent t_rt_range_shape t_rt_range_ro t_rt_range t_rt_depth_ro t_pso_skey t_pso_archive t_pull t_pull_inv t_pso_pending t_pfv t_view_levels t_view_swizzle t_vk_desc t_integer_blend t_primitive_restart t_clear_ds t_depth16_clear t_upgrade_alias t_stencil_view +run: t_nvn t_dirty_slots t_dirty_race t_tex_layout t_builtin t_vin t_tbuf2d t_deswz_gpu t_pace t_timestamp t_fstep t_refl t_hmap t_cover t_fuse_check t_present_crop t_reorder_chain t_defer_chain t_zskip t_xv t_ub t_vbuf t_geometry t_mip_extent t_rt_range_shape t_rt_range_ro t_rt_range t_rt_depth_ro t_pso_skey t_pso_archive t_pso_bundle t_pull t_pull_inv t_pso_pending t_pfv t_view_levels t_view_swizzle t_vk_desc t_integer_blend t_primitive_restart t_clear_ds t_depth16_clear t_upgrade_alias t_stencil_view ./t_nvn ./t_dirty_slots ./t_dirty_race @@ -234,6 +236,7 @@ run: t_nvn t_dirty_slots t_dirty_race t_tex_layout t_builtin t_vin t_tbuf2d t_de ./t_pso_skey ./t_pso_pending MTL_DEBUG_LAYER=1 ./t_pso_archive + MTL_DEBUG_LAYER=1 ./t_pso_bundle MTL_DEBUG_LAYER=1 ./t_pull MTL_DEBUG_LAYER=1 ./t_pull_inv python3 test_dmsh_pull.py diff --git a/dm/gfx/tests/t_pso_bundle.m b/dm/gfx/tests/t_pso_bundle.m new file mode 100644 index 0000000..45d47b3 --- /dev/null +++ b/dm/gfx/tests/t_pso_bundle.m @@ -0,0 +1,171 @@ +// The pipeline archive made offline and bundled with the app (DM_PSO_BUNDLE, pso_bundle_open / pso_bundle_covers in nvn_mtl.m), made here as build_ipa.sh makes it: tiny shader +// libraries and their reflection, a pair list, then dm/gfx/tools/pso_offline.py (metal-tt) for this Mac's GPU. Checks: the archive for this GPU opens and comes first among the files of +// bases without a vertex layout; with a matching stamp the prebuild builds only the pairs left to the device (a function with constants) and takes nothing from the compiler otherwise, +// and a draw of a covered pair takes its base from the bundle and captures nothing for the device's own archive; +// a stamp of another depth mode or other pairs runs the device's prebuild, which still takes the bundled bases it finds; an archive whose bases the probe does not find is not trusted; +// a folder without this GPU's file and a file that is no archive leave no archive. Without metal-tt (no Metal toolchain) it says so and passes +#include "../nvn/nvn_mtl.m" +uint64_t g_t0_ns; +int dm_guest_as_contains(uintptr_t address) { (void)address; return 0; } +void *front_metal_layer(void) { return NULL; } +module_t *module_of(const void *p) { (void)p; return NULL; } +module_t g_mod[MAX_MOD]; int g_nmod; +void svc_thread_dump(void) {} +uint64_t now_ns(void) { return bp_now(); } +static int checks, failed; +#define EXPECT(name, want, ...) do { uint64_t w = (want), g = (__VA_ARGS__); checks++; if (w != g) { fprintf(stderr, "%s: expected %llu, got %llu\n", name, (unsigned long long)w, (unsigned long long)g); failed++; } } while (0) + +static int sh(NSString *cmd) { int r = system(cmd.UTF8String); return WIFEXITED(r) ? WEXITSTATUS(r) : -1; } +static void put(NSString *path, NSString *text) { [NSFileManager.defaultManager createDirectoryAtPath:path.stringByDeletingLastPathComponent withIntermediateDirectories:YES attributes:nil error:nil]; [text writeToFile:path atomically:YES encoding:NSUTF8StringEncoding error:nil]; } +static int lib(NSString *dir, NSString *name, NSString *src) { // /.metallib from MSL, with the SDK's compiler + put([NSString stringWithFormat:@"%@/%@.metal", dir, name], src); + return sh([NSString stringWithFormat:@"cd %@ && xcrun -sdk macosx metal -c %@.metal -o %@.air && xcrun -sdk macosx metallib %@.air -o %@.metallib && rm %@.metal %@.air", dir, name, name, name, name, name, name]); +} +static void reset_counts(void) { g_pre_done = g_pre_built = g_pre_archived = g_pre_skipped = g_pre_failed = 0; } + +int main(void) { @autoreleasepool { + char tmp[] = "/tmp/t_pso_bundle.XXXXXX"; if (!mkdtemp(tmp)) abort(); + NSString *root = @(tmp), *msl = [root stringByAppendingFormat:@"/msl/b-%s", DMSH_CHAIN], *tables = [root stringByAppendingPathComponent:@"tables"]; + NSString *bundle = [root stringByAppendingPathComponent:@"bundle"], *arch = [root stringByAppendingPathComponent:@"archive"]; + // A magic number alone cannot establish an archive: these are the known crash inputs, checked without calling Metal. + NSString *bad = [root stringByAppendingPathComponent:@"bad.mtl4a"]; + const uint8_t magics[][4] = {{0xcf, 0xfa, 0xed, 0xfe}, {0xcb, 0xfe, 0xba, 0xbe}, {0xca, 0xfe, 0xba, 0xbe}, {0xca, 0xfe, 0xba, 0xbf}}; + for (unsigned i = 0; i < sizeof magics / sizeof magics[0]; i++) { + [[NSData dataWithBytes:magics[i] length:4] writeToFile:bad atomically:YES]; + EXPECT("archive guard: bare magic refused", 0, arch_file_ok(bad)); + NSMutableData *header = [NSMutableData dataWithLength:32]; memcpy(header.mutableBytes, magics[i], 4); [header writeToFile:bad atomically:YES]; + EXPECT("archive guard: bare header refused", 0, arch_file_ok(bad)); + } + NSString *v = @"#include \nusing namespace metal;\nstruct V { float4 p [[position]]; };\n" + "vertex V main0(uint id [[vertex_id]]) { V o; float2 uv = float2((id << 1) & 2, id & 2); o.p = float4(uv * 2 - 1, 0, 1); return o; }\n" + "vertex V main0_z(uint id [[vertex_id]]) { V o; float2 uv = float2((id << 1) & 2, id & 2); o.p = float4(uv * 2 - 1, 0.5, 1); return o; }\n"; + NSString *f = @"#include \nusing namespace metal;\nfragment float4 main0() { return float4(%d, %d, 0, 1); }\n"; + NSString *fc = @"#include \nusing namespace metal;\nconstant bool k [[function_constant(0)]];\nfragment float4 main0() { return is_function_constant_defined(k) ? float4(1) : float4(0); }\n"; + [NSFileManager.defaultManager createDirectoryAtPath:msl withIntermediateDirectories:YES attributes:nil error:nil]; + if (lib(msl, @"v1_bp", v) || lib(msl, @"f1_b", [NSString stringWithFormat:f, 1, 0]) || lib(msl, @"f2_b", [NSString stringWithFormat:f, 0, 1]) || lib(msl, @"fc_b", fc)) { + printf("t_pso_bundle: no Metal compiler (xcrun metal), GPU checks not run; %d file checks failed\n", failed); return failed != 0; } + put([msl stringByAppendingPathComponent:@"v1_b.json"], [NSString stringWithFormat:@"{\"chain\": \"%s\", \"entry\": \"main0\", \"entry_z\": \"main0_z\", \"pull\": {}, \"vertex_inputs\": []}", DMSH_CHAIN]); + for (NSString *k in @[@"f1", @"f2", @"fc"]) put([msl stringByAppendingFormat:@"/%@_b.json", k], [NSString stringWithFormat:@"{\"chain\": \"%s\", \"entry\": \"main0\"}", DMSH_CHAIN]); + put([tables stringByAppendingPathComponent:@"shader_map.tsv"], @"# empty\n"); + put([tables stringByAppendingPathComponent:@"shader_pairs.tsv"], @"# pairs\nv1\tf1\nv1\tf2\nv1\tfc\n"); + setenv("DM_NVN_SHMAP", [tables stringByAppendingPathComponent:@"shader_map.tsv"].UTF8String, 1); + setenv("DM_MSL_DIR", [root stringByAppendingPathComponent:@"msl"].UTF8String, 1); + setenv("DM_PSO_ARCHIVE", arch.UTF8String, 1); setenv("DM_PSO_MANIFEST", "off", 1); + mtl_init(); + NSString *gpu = g_dev.architecture.name; + // the archive as build_ipa.sh makes it (the tool's own folder: this file is dm/gfx/tests/t_pso_bundle.m) + NSString *tool = [[@(__FILE__) stringByDeletingLastPathComponent] stringByAppendingPathComponent:@"../tools/pso_offline.py"]; + if (sh([NSString stringWithFormat:@"python3 %@ --platform macos --gpu %@ --msl %@ --pairs %@/shader_pairs.tsv --out %@ --work %@/work > %@/pso_offline.log 2>&1", + tool, gpu, msl, tables, bundle, root, root])) { + NSString *log = [NSString stringWithContentsOfFile:[root stringByAppendingPathComponent:@"pso_offline.log"] encoding:NSUTF8StringEncoding error:nil]; + if ([log containsString:@"metal-tt is not available"]) { printf("t_pso_bundle: no metal-tt (xcodebuild -downloadComponent MetalToolchain), GPU checks not run; %d file checks failed\n", failed); return failed != 0; } + fprintf(stderr, "pso_offline.py failed:\n%s\n", log.UTF8String); return 1; + } + EXPECT("bundle: this GPU's archive", 1, [NSFileManager.defaultManager fileExistsAtPath:[bundle stringByAppendingFormat:@"/%@.mtl4a", gpu]]); + NSData *archiveBytes = [NSData dataWithContentsOfFile:[bundle stringByAppendingFormat:@"/%@.mtl4a", gpu]]; + EXPECT("archive guard: real offline archive accepted", 1, arch_file_ok([bundle stringByAppendingFormat:@"/%@.mtl4a", gpu])); + for (NSUInteger len = 32; len < archiveBytes.length; len = MIN(len * 2, archiveBytes.length)) { + [[archiveBytes subdataWithRange:NSMakeRange(0, len)] writeToFile:bad atomically:YES]; + EXPECT("archive guard: truncated offline archive refused", 0, arch_file_ok(bad)); + } + // Both standard universal wrappers must retain their slice bounds, including 64-bit offsets that cannot fit in the file. + for (int wide = 0; wide < 2; wide++) { + NSMutableData *wrapped = [NSMutableData dataWithLength:64]; + struct fat_header *fh = wrapped.mutableBytes; fh->magic = OSSwapHostToBigInt32(wide ? FAT_MAGIC_64 : FAT_MAGIC); fh->nfat_arch = OSSwapHostToBigInt32(1); + if (wide) { + struct fat_arch_64 *a = (void *)((uint8_t *)wrapped.mutableBytes + sizeof *fh); + a->offset = OSSwapHostToBigInt64(64); a->size = OSSwapHostToBigInt64(archiveBytes.length); + } else { + struct fat_arch *a = (void *)((uint8_t *)wrapped.mutableBytes + sizeof *fh); + a->offset = OSSwapHostToBigInt32(64); a->size = OSSwapHostToBigInt32((uint32_t)archiveBytes.length); + } + [wrapped appendData:archiveBytes]; [wrapped writeToFile:bad atomically:YES]; + EXPECT("archive guard: universal GPU image accepted", 1, arch_file_ok(bad)); + if (wide) ((struct fat_arch_64 *)((uint8_t *)wrapped.mutableBytes + sizeof *fh))->offset = UINT64_MAX; + else ((struct fat_arch *)((uint8_t *)wrapped.mutableBytes + sizeof *fh))->size = UINT32_MAX; + [wrapped writeToFile:bad atomically:YES]; + EXPECT("archive guard: universal slice outside file refused", 0, arch_file_ok(bad)); + } + NSMutableData *broken = [archiveBytes mutableCopy]; + struct mach_header_64 *mh = broken.mutableBytes; + EXPECT("archive fixture: one GPU image", MH_MAGIC_64, mh->magic); + ((struct load_command *)((uint8_t *)broken.mutableBytes + sizeof *mh))->cmdsize = 0; + [broken writeToFile:bad atomically:YES]; + EXPECT("archive guard: invalid load command refused", 0, arch_file_ok(bad)); + NSString *rest = [NSString stringWithContentsOfFile:[bundle stringByAppendingPathComponent:@"rest_pairs.tsv"] encoding:NSUTF8StringEncoding error:nil]; + EXPECT("bundle: one pair left to the device", 1, [rest isEqualToString:@"v1\tfc\n"]); + setenv("DM_PSO_BUNDLE", bundle.UTF8String, 1); + shmap_load(); pso4_init(); + EXPECT("bundle: opened", 1, g_arch_bundle != nil); + EXPECT("bundle: first among the files without a vertex layout", 1, g_arch4p.count && g_arch4p[0] == g_arch_bundle); + // its stamp matches: only the pair with function constants is built here + reset_counts(); + EXPECT("covered: one pair built on the device", 1, pso_prebuild(1, 1, QOS_CLASS_USER_INITIATED)); + EXPECT("covered: its base compiled", 1, g_pre_built); + EXPECT("covered: nothing else", 0, g_pre_archived + g_pre_skipped + g_pre_failed); + EXPECT("covered: again, nothing to do", -1, pso_prebuild(1, 1, QOS_CLASS_USER_INITIATED)); + // a draw of a covered pair (its pulling pipeline, as pso_desc_pull makes it) takes the base from the bundle: nothing compiled, nothing captured for the device's own archive + @autoreleasepool { + id vl = lib_from_file([msl stringByAppendingPathComponent:@"v1_bp.metallib"], NULL), fl = lib_from_file([msl stringByAppendingPathComponent:@"f1_b.metallib"], NULL); + MTLRenderPipelineDescriptor *d = [MTLRenderPipelineDescriptor new]; + d.vertexFunction = fn4_note([vl newFunctionWithName:@"main0"], vl, @"main0", nil); d.fragmentFunction = fn4_note([fl newFunctionWithName:@"main0"], fl, @"main0", nil); + d.vertexDescriptor = [MTLVertexDescriptor vertexDescriptor]; d.colorAttachments[0].pixelFormat = MTLPixelFormatRGBA8Unorm; + uint64_t built0 = g_pb_built, arch0 = g_pb_archived; int new0 = g_arch_new; NSError *err = nil; + EXPECT("covered draw: a pipeline", 1, pso_build(d, &err) != nil); + EXPECT("covered draw: its base from the bundle", arch0 + 1, g_pb_archived); + EXPECT("covered draw: none compiled", built0, g_pb_built); + EXPECT("covered draw: nothing captured for the device's archive", new0, g_arch_new); + } + // the near = 0 depth mode: not what the archive was made for, so the device's prebuild runs, and its main0_z bases are not in the archive + [NSFileManager.defaultManager removeItemAtPath:[arch stringByAppendingPathComponent:@"prebuilt.tsv"] error:nil]; + reset_counts(); atomic_store(&g_depth_zero, 1); + EXPECT("other depth mode: every pair", 3, pso_prebuild(1, 1, QOS_CLASS_USER_INITIATED)); + EXPECT("other depth mode: every base compiled", 3, g_pre_built); + atomic_store(&g_depth_zero, 0); + // other pairs (a second list): the device's prebuild runs and takes the bundled bases it finds + [NSFileManager.defaultManager removeItemAtPath:[arch stringByAppendingPathComponent:@"prebuilt.tsv"] error:nil]; + put([tables stringByAppendingPathComponent:@"more.tsv"], @"v1\tf2\nv1\tfc\n"); setenv("DM_PSO_PAIRS", [tables stringByAppendingPathComponent:@"more.tsv"].UTF8String, 1); + reset_counts(); + EXPECT("other pairs: every pair", 3, pso_prebuild(1, 1, QOS_CLASS_USER_INITIATED)); + EXPECT("other pairs: two bases from the bundle", 2, g_pre_archived); + EXPECT("other pairs: one compiled", 1, g_pre_built); + unsetenv("DM_PSO_PAIRS"); + // a matching stamp but an archive without the bases (here: one of the device's own files): the probe finds none, so it is not trusted + @autoreleasepool { + NSString *txt = [NSString stringWithContentsOfFile:[tables stringByAppendingPathComponent:@"shader_pairs.tsv"] encoding:NSUTF8StringEncoding error:nil]; + NSString *own = [arch stringByAppendingPathComponent:[[[NSFileManager.defaultManager contentsOfDirectoryAtPath:arch error:nil] filteredArrayUsingPredicate:[NSPredicate predicateWithFormat:@"self BEGINSWITH 'p' AND self ENDSWITH '.mtl4a'"]] firstObject]]; + id real = g_arch_bundle; NSError *e = nil; + g_arch_bundle = [g_dev newArchiveWithURL:[NSURL fileURLWithPath:own] error:&e]; + EXPECT("archive guard: real serialized archive accepted", 1, arch_file_ok(own)); + EXPECT("probe: another archive opened", 1, g_arch_bundle != nil); + EXPECT("probe: its bases are not this run's", 0, pso_bundle_covers(pre_parse(txt), pre_stamp(txt, 0), 0, rest)); + g_arch_bundle = real; + EXPECT("probe: the bundled archive holds them", 1, pso_bundle_covers(pre_parse(txt), pre_stamp(txt, 0), 0, rest)); + EXPECT("probe: not for the other depth mode", 0, pso_bundle_covers(pre_parse(txt), pre_stamp(txt, 1), 1, rest)); + // Raw-pair sampling can hit only ineligible pairs: 0 of 0 lookups must not suppress the device prebuild. + NSMutableString *sparse = [NSMutableString string]; + for (int i = 0; i < PRE_PROBE; i++) [sparse appendFormat:@"missing%d\tf1\n", i]; + [sparse appendString:@"v1\tf1\n"]; + put([bundle stringByAppendingPathComponent:@"prebuilt.tsv"], pre_stamp(sparse, 0)); + EXPECT("probe: no eligible sample cannot establish coverage", 0, pso_bundle_covers(pre_parse(sparse), pre_stamp(sparse, 0), 0, nil)); + put([bundle stringByAppendingPathComponent:@"prebuilt.tsv"], pre_stamp(txt, 0)); + } + // a folder without this GPU's archive, and a file that is no archive (Metal crashed on one, and on a bare Mach-O header too): no archive, no crash + @autoreleasepool { + id real = g_arch_bundle; NSString *other = [root stringByAppendingPathComponent:@"other"]; + put([other stringByAppendingPathComponent:@"applegpu_g99x.mtl4a"], @"x"); + NSMutableArray *list = [NSMutableArray array]; g_arch_bundle = nil; + setenv("DM_PSO_BUNDLE", other.UTF8String, 1); pso_bundle_open(list); + EXPECT("no file for this GPU: none opened", 0, list.count + (g_arch_bundle != nil)); + put([other stringByAppendingFormat:@"/%@.mtl4a", gpu], @"not an archive"); // Metal's own loadFromURL crashed on it: never handed to Metal + pso_bundle_open(list); + EXPECT("not an archive: none opened", 0, list.count + (g_arch_bundle != nil)); + EXPECT("not an archive: refused before Metal", 0, arch_file_ok([other stringByAppendingFormat:@"/%@.mtl4a", gpu])); + setenv("DM_PSO_BUNDLE", "off", 1); pso_bundle_open(list); + EXPECT("off: none opened", 0, list.count); + g_arch_bundle = real; + } + [NSFileManager.defaultManager removeItemAtPath:root error:nil]; + printf("t_pso_bundle: %d checks, %d failed\n", checks, failed); + return failed != 0; +} } diff --git a/dm/gfx/tools/pso_offline.py b/dm/gfx/tools/pso_offline.py new file mode 100644 index 0000000..9363f13 --- /dev/null +++ b/dm/gfx/tools/pso_offline.py @@ -0,0 +1,229 @@ +#!/usr/bin/env python3 +# The pipeline archive made offline: the base of every shader pair, as the host's pipeline prebuild makes it on the device (pre_desc / pre_pair in dm/gfx/nvn/nvn_mtl.m), compiled on the Mac +# by metal-tt into one Metal 4 pipeline archive per GPU architecture, so an iPhone takes the bases from the app bundle instead of compiling them on its first start. +# The host matches libraries by content, so the archive serves the libraries the app ships (msl/b-); measurements are in dm/games/01007EF00011E000/README.md. +# Per pair (vertex key, pixel key) of the profile's shader_pairs.tsv and pso_pairs.tsv, read in that order with repeated lines dropped, as the host reads them: +# · the vertex shader's pulling library _bp (a vertex shader without stage inputs: its own _b), entry `entry`, or `entry_z` for the near = 0 depth mode, from its reflection +# _b.json; the pixel shader's _b, entry `entry` (main0); no vertex descriptor, one sample, every colour attachment unspecialized +# · left out as the host leaves them out (to the draws): a reflection of another chain version or none, a vertex shader with stage inputs and no pulling library, a vertex library that +# writes the layer (render_target_array_index), a missing library or entry point +# · a pair whose functions declare function constants is left to the device (rest_pairs.tsv): the host specializes such functions with an empty set of values, and metal-tt makes +# no specialized function from a library (applegpu-nt: "cannot find private metadata"), nor the function unspecialized ("cannot lower module with unresolved function constants") +# Output folder (what the app bundles as pso/, the host's DM_PSO_BUNDLE): +# .mtl4a one archive per GPU architecture (applegpu_g18p = A19 / A19 Pro); the host opens the one named after its device's architecture +# prebuilt.tsv what the archives cover, in the host's stamp format (pso_prebuild): chain version, depth mode, the pair lists' size and SHA-256. The host skips its prebuild when +# it matches and a probe of a few bases finds them in the archive +# rest_pairs.tsv the pairs the device still builds (only when there are any) +# archive.tsv what was built: architectures, OS minimum, toolchain, bases (for people and logs; the host does not read it) +# Each archive is made again only when its key changes (key. in the work folder): this file, the chain version, the libraries' bytes and stamp (inputs.tsv), the stamp above, +# the architecture, the platform and minimum OS, and metal-tt's version. +# Usage: pso_offline.py --msl folder> --out --work [--game ] [--pairs [--pairs ]] [--gpu g18p[,g17p]] +# [--depth gl|zero] [--platform ios|macos] [--os-min 27.0] [--script-only] +# --game: the profile (DM_GAMES_DIR as for game.py) whose shader_pairs.tsv and pso_pairs.tsv are the pairs, and whose depth_mode key the depth mode, unless --pairs / --depth say; +# --gpu: architectures, with or without the applegpu_ prefix (default g18p); --depth: the game's NVN depth mode (gl = near -w, NVN's default; zero = near 0, which the game sets +# with nvnDeviceSetDepthMode: the host knows it only at run time, so the profile says it; a wrong one costs the device its own prebuild, as without the archive); +# --os-min: the archives' minimum OS (default: the SDK's major version, e.g. 27.0; a device on an older system cannot open them and builds the bases itself) +import argparse, hashlib, json, os, shutil, struct, subprocess, sys, time +from pathlib import Path + +LAYER = b'render_target_array_index' +COLOUR = ('pixel_format', 'write_mask', 'blending_state', 'alpha_blend_operation', 'rgb_blend_operation', 'destination_alpha_blend_factor', 'destination_rgb_blend_factor', + 'source_alpha_blend_factor', 'source_rgb_blend_factor') + + +def arch_name(a): # 'g18p' → 'applegpu_g18p' + a = a.strip() + return a if a.startswith('applegpu_') else f'applegpu_{a}' + + +def pair_text(paths): + """The pair lists as the host reads them: their bytes one after the other (shader_pairs.tsv, then pso_pairs.tsv when it exists)""" + return b''.join(Path(p).read_bytes() for p in paths if p and Path(p).is_file()) + + +def read_pairs(text): + """(vertex key, pixel key) per line, in order, without empty lines, comments and repeated lines (pso_prebuild's reading)""" + seen, out = set(), [] + for line in text.decode('utf-8').split('\n'): + if not line or line.startswith('#') or line in seen: continue + seen.add(line); c = line.split('\t') + if len(c) >= 2: out.append((c[0], c[1])) + return out + + +def stamp(chain, zero, text): + """The host's prebuild stamp (pso_prebuild in nvn_mtl.m): chain version, depth mode, and the pair lists' length in bytes and SHA-256 (first 16 hex digits)""" + return f'chain\t{chain}\ndepth\t{"near=0" if zero else "near=-w"}\npairs\t{len(text)}\t{hashlib.sha256(text).hexdigest()[:16]}\n' + + +def functions(path): + """{entry name: declares function constants} of a .metallib, from its header: the function list (NAME tags) and each function's public metadata (a CNST tag lists its constants). + Raises ValueError on a file it cannot read""" + with open(path, 'rb') as f: b = f.read(1 << 20) + if b[:4] != b'MTLB' or len(b) < 0x58: raise ValueError(f'{path}: not a Metal library') + fo, _, po, ps = struct.unpack_from(' folder') + archs = sorted({arch_name(x) for x in a.gpu.split(',') if x.strip()}) + if not archs: sys.exit('[RED] pso_offline: no GPU architecture (--gpu)') + text = pair_text(a.pairs) + if not text: sys.exit(f'[RED] pso_offline: no pair list ({", ".join(a.pairs)})') + zero = a.depth == 'zero' + t0 = time.time() + bases, rest, left = plan(lib_dir, chain, read_pairs(text), zero) + work.mkdir(parents=True, exist_ok=True); out.mkdir(parents=True, exist_ok=True) + js = work / 'bases.mtl4-json' # metal-tt takes the format from the extension: .mtl4-json is Metal 4's (.mtlp-json is the Metal 3 format) + body = json.dumps(script(lib_dir, bases), indent=1).encode() + if not js.is_file() or js.read_bytes() != body: js.write_bytes(body) + st = stamp(chain, zero, text) + print(f'pso_offline: {len(read_pairs(text))} pairs: {len(bases)} bases, {len(rest)} left to the device (function constants), left out as the host leaves them: ' + + ', '.join(f'{k} {v}' for k, v in left.items()) + f' ({time.time() - t0:.1f} s)') + if a.script_only: (out / 'prebuilt.tsv').write_text(st); return 0 + if not bases and not rest: sys.exit('[RED] pso_offline: no base to build') + if not bases: archs = [] # every eligible pair needs device specialization; clear old archives and still write the remainder list + osmin = a.os_min or os_min_default(a.platform); tv = tool_version(a.platform) if bases else 'not used (function constants only)' + inputs = (lib_dir / 'inputs.tsv').read_text() if (lib_dir / 'inputs.tsv').is_file() else '' + libraries = library_digest(lib_dir, bases) + for arch in archs: + key = hashlib.sha256('\0'.join([Path(__file__).read_text(), chain, inputs, libraries, st, arch, a.platform, osmin, tv, hashlib.sha256(body).hexdigest()]).encode()).hexdigest()[:16] + dst, kf = out / f'{arch}.mtl4a', work / f'key.{arch}' + if dst.is_file() and kf.is_file() and kf.read_text().strip() == key: print(f'pso_offline: {arch}: reused ({dst.stat().st_size / 2**20:.0f} MB)'); continue + kf.unlink(missing_ok=True); dst.unlink(missing_ok=True) + part = work / f'{arch}.mtl4a.partial'; part.unlink(missing_ok=True) + cmd = ['xcrun', '-sdk', sdk(a.platform), 'metal-tt', '-arch', arch, f'-{a.platform}-version-min', osmin, str(js), '-o', str(part)] + t1 = time.time(); print(f'pso_offline: {arch}: metal-tt, {len(bases)} bases (minutes) ...', flush=True) + with open(work / f'metal-tt.{arch}.log', 'w') as log: + r = subprocess.run(cmd, stdout=log, stderr=subprocess.STDOUT, env=dict(os.environ, TMPDIR=str(work))) # its temporary files on the work folder's volume + if r.returncode or not part.is_file(): + sys.exit(f'[RED] pso_offline: metal-tt failed for {arch} (exit {r.returncode}): {(work / f"metal-tt.{arch}.log").read_text()[-2000:]}') + os.replace(part, dst); kf.write_text(key + '\n') + print(f'pso_offline: {arch}: {dst.stat().st_size / 2**20:.0f} MB in {time.time() - t1:.0f} s') + for f in out.glob('*.mtl4a'): # architectures no longer asked for + if f.stem not in archs: f.unlink() + (out / 'prebuilt.tsv').write_text(st) + rp = out / 'rest_pairs.tsv' + if rest: rp.write_text(''.join(f'{v}\t{f}\n' for v, f in rest)) + else: rp.unlink(missing_ok=True) + (out / 'archive.tsv').write_text(f'# the pipeline archive made offline (dm/gfx/tools/pso_offline.py); the host reads prebuilt.tsv, not this file\narch\t{" ".join(archs)}\n' + f'platform\t{a.platform}\t{osmin}\ntoolchain\t{tv}\nbases\t{len(bases)}\nrest\t{len(rest)}\n') + return 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/dm/ios/Sources/main.m b/dm/ios/Sources/main.m index 070c4dc..1f05105 100644 --- a/dm/ios/Sources/main.m +++ b/dm/ios/Sources/main.m @@ -67,6 +67,8 @@ static int boot_env(void) { setenv("DM_FONT_DIR", [dm stringByAppendingPathComponent:@"font"].fileSystemRepresentation, 0); // Shared-font stand-ins (standard.ttf / nintendo_ext.ttf): iOS lacks the macOS font path NSString *aoc = [res stringByAppendingPathComponent:@"aoc"]; // add-on content: build_ipa.sh puts the RomFS extracted from the player's DLC NSPs at /aoc// if ([fm fileExistsAtPath:aoc]) setenv("DM_AOC_DIR", aoc.fileSystemRepresentation, 0); + NSString *pso = [res stringByAppendingPathComponent:@"pso"]; // the pipeline archive made offline (build_ipa.sh, dm/gfx/tools/pso_offline.py): the shader pairs' bases, which the device then does not build + if ([fm fileExistsAtPath:pso]) setenv("DM_PSO_BUNDLE", pso.fileSystemRepresentation, 0); chdir(doc.fileSystemRepresentation); g_argv[0] = strdup("dmhost"); g_argv[1] = strdup(dm.fileSystemRepresentation); g_argc = 2; dm_host_preload_start(g_argc, g_argv); // Module tables / relocation / takeover run early in the background, in parallel with UIKit startup (and the launcher: the settings it sets are read when the game starts) diff --git a/dm/ios/tests/test_profile_pack.py b/dm/ios/tests/test_profile_pack.py index 6d6ec7c..c4b762f 100644 --- a/dm/ios/tests/test_profile_pack.py +++ b/dm/ios/tests/test_profile_pack.py @@ -14,10 +14,12 @@ import game class ProfilePackTests(unittest.TestCase): def test_pulling_libraries_follow_compiled_profile_and_explicit_override(self): - cases = [('01007EF00011E000', None, True), ('010015100B514000', None, False), - ('01007EF00011E000', '0', False), ('010015100B514000', '1', True)] - for gid, override, included in cases: - with self.subTest(game=gid, IOS_PULL=override), tempfile.TemporaryDirectory() as temporary: + # (game, IOS_PULL, with a pipeline archive, _bp libraries packed): an archive's bases are made of the pulling libraries, so an IPA with one carries them whatever IOS_PULL says + cases = [('01007EF00011E000', None, False, True), ('010015100B514000', None, False, False), + ('01007EF00011E000', '0', False, False), ('010015100B514000', '1', False, True), + ('010015100B514000', None, True, True), ('01007EF00011E000', '0', True, True)] + for gid, override, with_pso, included in cases: + with self.subTest(game=gid, IOS_PULL=override, archive=with_pso), tempfile.TemporaryDirectory() as temporary: root = Path(temporary) dm = root / 'dm' header = dm / 'host/gen' / gid / 'game_profile.h' @@ -35,6 +37,9 @@ class ProfilePackTests(unittest.TestCase): env.update(GAME=gid, DM=str(dm), APP=str(app), MSL_IOS=str(shaders), MSLDIR='b-fixture', AOC='', ROMFS=str(romfs), OUT_IPA=str(out), PROFILE_TOOL=str(REPO / 'dm/games/game.py')) if override is not None: env['IOS_PULL'] = override + if with_pso: + pso = root / 'pso'; pso.mkdir(); (pso / 'prebuilt.tsv').write_text('fixture') + env['PSO'] = str(pso) # Exercise the actual archive step; only the preceding Xcode build is bypassed. script = (REPO / 'dm/ios/tools/build_ipa.sh').read_text() packing = script[script.index('\nP=Payload/DmIOS.app;'):] @@ -49,6 +54,7 @@ class ProfilePackTests(unittest.TestCase): for name in ('fixture_b.metallib', 'fixture_bv.metallib', 'fixture_b.json'): self.assertIn(prefix + name, names) self.assertEqual(archive.read('Payload/DmIOS.app/romfs/marker'), b'romfs') + self.assertEqual('Payload/DmIOS.app/pso/prebuilt.tsv' in names, with_pso) if __name__ == '__main__': diff --git a/dm/ios/tests/test_xcode_workflow.py b/dm/ios/tests/test_xcode_workflow.py index 7e2f784..425770e 100644 --- a/dm/ios/tests/test_xcode_workflow.py +++ b/dm/ios/tests/test_xcode_workflow.py @@ -164,6 +164,32 @@ class ResourceStagingTests(unittest.TestCase): result = subprocess.run(['bash', str(REPO / 'dm/ios/tools/stage_resources.sh')], env=environment) self.assertEqual(result.returncode, 0) + def test_pipeline_archive(self): # the archive made offline goes to /pso when build_ipa.sh names one, and leaves when it no longer does + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary).resolve() + ios, romfs, shader, pso = root / 'xcode', root / 'romfs', root / 'shaders/b-fixture', root / 'pso' + for folder in (ios / 'Gen', romfs, shader, pso): + folder.mkdir(parents=True) + (ios / 'Gen/dm.xcconfig').touch(); (romfs / 'marker').write_text('game'); (shader / 'k_bp.metallib').write_text('lib') + (pso / 'prebuilt.tsv').write_text('chain\tfixture\n'); (pso / 'applegpu_g18p.mtl4a').write_text('archive') + chain_h = root / 'dmsh_chain.h'; chain_h.write_text('#define DMSH_CHAIN "fixture"\n') + app = root / 'products/DmIOS.app' + environment = dict(os.environ, SRCROOT=str(ios), TARGET_BUILD_DIR=str(app.parent), UNLOCALIZED_RESOURCES_FOLDER_PATH=app.name, PLATFORM_NAME='iphoneos') + def stage(*pso_arg): + with patch.object(sys, 'argv', ['gen_resources.py', str(ios), str(romfs), 'marker', str(shader), 'b-fixture', '', str(root / 'dd'), str(chain_h), *pso_arg]): + gen_resources.main() + return subprocess.run(['bash', str(REPO / 'dm/ios/tools/stage_resources.sh')], env=environment, capture_output=True, text=True) + result = stage(str(pso)) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual((app / 'pso/applegpu_g18p.mtl4a').read_text(), 'archive') + self.assertTrue((app / 'msl/b-fixture/k_bp.metallib').exists()) # the pulling libraries the bases are made of + (pso / 'prebuilt.tsv').unlink() # an archive folder without its stamp: stop before copying + self.assertNotEqual(stage(str(pso)).returncode, 0) + self.assertTrue((app / 'pso/applegpu_g18p.mtl4a').exists()) + result = stage('') # built without an archive (DM_PSO_GPUS=none) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertFalse((app / 'pso').exists()) + if __name__ == '__main__': unittest.main() diff --git a/dm/ios/tools/build_ipa.sh b/dm/ios/tools/build_ipa.sh index 3f61738..f6e39e6 100755 --- a/dm/ios/tools/build_ipa.sh +++ b/dm/ios/tools/build_ipa.sh @@ -78,6 +78,20 @@ else cp "$GD/shader_map.tsv" "$GD/pso_manifest.tsv" "$GD/c1_full.tsv" Gen/dm/ || if [ -f "$GD/cb_extent.tsv" ]; then cp "$GD/cb_extent.tsv" Gen/dm/ || die "copy cb_extent.tsv"; else t "no $GD/cb_extent.tsv: the host copies whole uniform bindings"; fi # the shader pairs, beside the shader map where the pipeline prebuild reads them (nvn_mtl.m pso_prebuild): without them the app makes no pulling bases, and a hybrid draw has no stand-in for f in shader_pairs.tsv pso_pairs.tsv; do if [ -f "$GD/$f" ]; then cp "$GD/$f" Gen/dm/ || die "copy $f"; fi; done; fi # optional +# The pipeline archive made offline (dm/gfx/tools/pso_offline.py): the base of every shader pair, compiled here by metal-tt for each GPU architecture in DM_PSO_GPUS (comma-separated: +# g18p = A19 / A19 Pro; none = no archive), so the device does not build them on its first start (Breath of the Wild: 15,767 bases, 7-8 minutes on an iPhone 17 Pro Max). +# Only the Metal 4 path reads it, which iOS turns on for a game whose profile sets pso_mtl4 on: the default is g18p for such a game and none for the others (their app would carry +# it unread). It goes into the app as pso/ (main.m hands it to the host as DM_PSO_BUNDLE), with the pulling vertex libraries the bases are made of. Each architecture adds about +# 650 MB to the app (Breath of the Wild; 220 MB in the IPA) and 4 minutes to the first build; made again only when its inputs change (the libraries, the pairs, the profile's depth_mode, metal-tt) +PSO=; P4=$(python3 "$DM/games/game.py" "$GAME" pso_mtl4 off) || die "game profile $GD: pso_mtl4" +if [ -n "${DM_PSO_GPUS+set}" ]; then GPUS=$DM_PSO_GPUS; elif [ "$P4" = on ]; then GPUS=g18p; else GPUS=none; fi +if [ "$GFXAPI" = nvn ] && [ -n "$GPUS" ] && [ "$GPUS" != none ] && [ -f "$GD/shader_pairs.tsv" ]; then + PSO=$W/build/ios/pso/bundle + t "pipeline archive for $GPUS (metal-tt; minutes when its inputs changed)" + python3 "$DM/gfx/tools/pso_offline.py" --game "$GAME" --msl "$MSL_IOS/b-$V" --out "$PSO" --work "$W/build/ios/pso/work" --gpu "$GPUS" ${DM_PSO_IOS_MIN:+--os-min "$DM_PSO_IOS_MIN"} \ + | while IFS= read -r ln; do t "$ln"; done + [ "${PIPESTATUS[0]}" = 0 ] || die "pipeline archive (dm/gfx/tools/pso_offline.py; DM_PSO_GPUS=none builds the app without it)" +else [ "$GFXAPI" != nvn ] || t "no pipeline archive ($([ -n "${DM_PSO_GPUS+set}" ] && echo "DM_PSO_GPUS=${GPUS:-empty}" || echo "the profile does not set pso_mtl4 on; DM_PSO_GPUS=g18p builds one")$([ -f "$GD/shader_pairs.tsv" ] || echo ', no shader_pairs.tsv'))"; fi # The launcher's settings for this game (Launcher.swift): dm/games/settings.tsv filtered by the profile, main's rewriter table and the game's languages (dm/games/settings.py) L=$(GFXAPI="$GFXAPI" python3 "$DM/games/settings.py" "$GAME" "$PATCHED" "$VNACP" Gen/dm/launcher.json) || die "launcher settings (dm/games/settings.py)"; t "launcher: $L" # Diagnostic build: DM_ENV_FILE=file → dm/env.txt in the bundle (main.m sets one environment variable per line at boot) @@ -109,7 +123,7 @@ if [ "$PREPARE_XCODE" = YES ] && [ -n "${DM_ENTITLEMENTS-default}" ]; then t "entitlements: $(plutil -convert json -o - Gen/DmIOS.entitlements)" fi xcodegen generate -q > /dev/null || die "xcodegen" -python3 tools/gen_resources.py "$IOS" "$ROMFS" "$MARKER" "$MSL_IOS/$MSLDIR" "$MSLDIR" "$AOC" "$XCODE_DD" "$DM/gfx/nvn/dmsh_chain.h" || die "generate Xcode resource paths" +python3 tools/gen_resources.py "$IOS" "$ROMFS" "$MARKER" "$MSL_IOS/$MSLDIR" "$MSLDIR" "$AOC" "$XCODE_DD" "$DM/gfx/nvn/dmsh_chain.h" "$PSO" || die "generate Xcode resource paths" if [ "$PREPARE_XCODE" = YES ]; then t "Xcode project: $IOS/DmIOS.xcodeproj" t "Xcode derived data: $XCODE_DD" @@ -134,12 +148,14 @@ APP=$DD/Build/Products/Release-iphoneos/DmIOS.app rm -rf "$APP/msl" "$O/Payload" "$O/out.ipa" # (the last two: what the copy-and-zip packing left behind) P=Payload/DmIOS.app; MAPS=("$P=$APP" "$P/msl/$MSLDIR=$MSL_IOS/$MSLDIR") if [ -n "$AOC" ] && [ -d "$AOC" ]; then MAPS+=("$P/aoc=$AOC"); t "add-on content: $(ls "$AOC" | tr '\n' ' ')"; fi +if [ -n "${PSO:-}" ]; then MAPS+=("$P/pso=$PSO"); fi MAPS+=("$P/romfs=$ROMFS") t "pack IPA (already-compressed game files stored, everything else deflated)" # The pulling vertex libraries (_bp, dm/gfx/tools/dmsh_pull.py) serve the hybrid pipelines. Default to the same pso_mtl4 setting compiled by gen_build.py; IOS_PULL=0 / 1 overrides it. +# The pipeline archive's bases are made of them, so an IPA with an archive (pso/) always carries them, whatever IOS_PULL says (an Xcode build copies them always). PULL=$(sed -n 's/^#define DM_GAME_PSO_MTL4 //p' "$DM/host/gen/$GAME/game_profile.h") || die "read game profile header" case "$PULL" in 0|1) ;; *) die "no DM_GAME_PSO_MTL4 in generated game profile";; esac # Keep this array nonempty: macOS Bash 3.2 treats an empty array as unset under set -u. -PACK_OPTS=(--store "$(gp zip_store)"); [ "${IOS_PULL:-$PULL}" = 1 ] || PACK_OPTS+=(--skip-suffix _bp.metallib) +PACK_OPTS=(--store "$(gp zip_store)"); [ -n "${PSO:-}" ] || [ "${IOS_PULL:-$PULL}" = 1 ] || PACK_OPTS+=(--skip-suffix _bp.metallib) python3 tools/pack_ipa.py "$OUT_IPA" "${PACK_OPTS[@]}" --skip $P/_CodeSignature --skip $P/embedded.mobileprovision --skip $P/msl --skip $P/aoc --skip $P/romfs "${MAPS[@]}" || die "pack_ipa" t "done: $OUT_IPA ($(du -h "$OUT_IPA" | cut -f1))" diff --git a/dm/ios/tools/gen_resources.py b/dm/ios/tools/gen_resources.py index 4824557..8849820 100644 --- a/dm/ios/tools/gen_resources.py +++ b/dm/ios/tools/gen_resources.py @@ -25,7 +25,8 @@ def workspace_settings(workspace, derived): def main(): - ios, romfs, marker, shaders, msldir, aoc, derived, chain_h = sys.argv[1:] + ios, romfs, marker, shaders, msldir, aoc, derived, chain_h, *rest = sys.argv[1:] + pso = rest[0] if rest else '' # the pipeline archive made offline (build_ipa.sh, dm/gfx/tools/pso_offline.py), or none ios = Path(ios) paths = { 'DM_RESOURCE_ROMFS': str(Path(romfs).resolve()), @@ -34,6 +35,7 @@ def main(): 'DM_RESOURCE_MSLDIR': msldir, 'DM_RESOURCE_AOC': str(Path(aoc).resolve()) if aoc and Path(aoc).is_dir() else '', 'DM_RESOURCE_CHAIN_H': str(Path(chain_h).resolve()), # dm/gfx/nvn/dmsh_chain.h, which the host is compiled with (one chain version for every game) + 'DM_RESOURCE_PSO': str(Path(pso).resolve()) if pso and Path(pso).is_dir() else '', # → /pso, which the app hands the host as DM_PSO_BUNDLE } (ios / 'Gen/resources.env').write_text( '# Generated by build_ipa.sh; regenerate when the game or its drive changes.\n' diff --git a/dm/ios/tools/stage_resources.sh b/dm/ios/tools/stage_resources.sh index 76f5124..00bfa7f 100644 --- a/dm/ios/tools/stage_resources.sh +++ b/dm/ios/tools/stage_resources.sh @@ -13,6 +13,8 @@ case "$APP" in /*.app) ;; *) die "Unexpected app resource destination: $APP";; e [ -e "$DM_RESOURCE_ROMFS/$DM_RESOURCE_MARKER" ] || die "Game files unavailable: $DM_RESOURCE_ROMFS/$DM_RESOURCE_MARKER" [ -d "$DM_RESOURCE_SHADERS" ] || die "Shader libraries unavailable: $DM_RESOURCE_SHADERS" [ -z "$DM_RESOURCE_AOC" ] || [ -d "$DM_RESOURCE_AOC" ] || die "Add-on content unavailable: $DM_RESOURCE_AOC" +DM_RESOURCE_PSO=${DM_RESOURCE_PSO:-} # (a project prepared before the pipeline archive has none) +[ -z "$DM_RESOURCE_PSO" ] || [ -f "$DM_RESOURCE_PSO/prebuilt.tsv" ] || die "Pipeline archive unavailable: $DM_RESOURCE_PSO" # The host is compiled from the repository's current sources and loads only msl/b-: after a chain change (a pull, dmsh_chain.py --header) the prepared shaders are another chain's case "$DM_RESOURCE_MSLDIR" in b-*) [ -f "$DM_RESOURCE_CHAIN_H" ] || die "No chain version header: $DM_RESOURCE_CHAIN_H (a project prepared before the chain version moved to dm/gfx/nvn: prepare the game again, build_ipa_from_nsp.py --prepare-xcode)" @@ -32,3 +34,10 @@ if [ -n "$DM_RESOURCE_AOC" ]; then else rm -rf "$APP/aoc" fi +# The pipeline archive made offline (the shader pairs' bases, build_ipa.sh): the app hands /pso to the host as DM_PSO_BUNDLE +if [ -n "$DM_RESOURCE_PSO" ]; then + mkdir -p "$APP/pso" + /usr/bin/rsync -a --delete -- "$DM_RESOURCE_PSO/" "$APP/pso/" +else + rm -rf "$APP/pso" +fi diff --git a/dm/tools/tests/test_pso_offline.py b/dm/tools/tests/test_pso_offline.py new file mode 100644 index 0000000..60621cb --- /dev/null +++ b/dm/tools/tests/test_pso_offline.py @@ -0,0 +1,252 @@ +"""The pipeline archive made offline (dm/gfx/tools/pso_offline.py): the pairs as the host reads them, the stamp the host compares (pso_prebuild's pre_stamp), the function list of a +.metallib header, the bases the host's prebuild would make (pre_desc) and those it leaves out, the pipelines script, and the per-architecture reuse of metal-tt's archives. +No game files, Xcode or GPU needed (metal-tt is replaced; temporary files only in a temporary folder).""" +import contextlib +import hashlib +import importlib.util +import io +import json +import os +from pathlib import Path +import shutil +import struct +import subprocess +import sys +import tempfile +import unittest +from unittest import mock + +REPO = Path(__file__).resolve().parents[3] +spec = importlib.util.spec_from_file_location('pso_offline', REPO / 'dm/gfx/tools/pso_offline.py') +po = importlib.util.module_from_spec(spec); spec.loader.exec_module(po) +CHAIN = '0123456789ab' + + +def tag(name, data): return name + struct.pack('\nusing namespace metal;\nconstant bool k [[function_constant(0)]];\nstruct V { float4 p [[position]]; };\n' + 'vertex V vs(uint i [[vertex_id]]) { V o; o.p = float4(i); return o; }\nfragment float4 fs() { return is_function_constant_defined(k) ? 1 : 0; }\n') + with tempfile.TemporaryDirectory() as t: + m, air, lib = Path(t, 'a.metal'), Path(t, 'a.air'), Path(t, 'a.metallib'); m.write_text(src) + r = subprocess.run(['xcrun', '-sdk', 'iphoneos', 'metal', '-c', m, '-o', air], capture_output=True) + if r.returncode: self.skipTest('no iOS Metal compiler') + subprocess.run(['xcrun', '-sdk', 'iphoneos', 'metallib', air, '-o', lib], check=True, capture_output=True) + self.assertEqual(po.functions(lib), {'vs': False, 'fs': True}) + + +class Plan(unittest.TestCase): + def setUp(self): + self.d = Path(tempfile.mkdtemp()) / f'b-{CHAIN}'; self.d.mkdir(); self.addCleanup(shutil.rmtree, self.d.parent) + + def shader(self, key, j, libs): # reflection _b.json and libraries {suffix: funcs or (funcs, layer)} + (self.d / f'{key}_b.json').write_text(json.dumps(dict({'chain': CHAIN, 'entry': 'main0'}, **j))) + for suf, f in libs.items(): + funcs, layer = f if isinstance(f, tuple) else (f, False) + (self.d / f'{key}{suf}.metallib').write_bytes(metallib(funcs, layer)) + + def test_bases_rest_and_left_out(self): + vz = [('main0', False), ('main0_z', False)] + self.shader('pull', {'entry_z': 'main0_z', 'pull': {'entry': 'main0'}, 'vertex_inputs': [{'location': 0}]}, {'_b': vz, '_bp': vz}) + self.shader('plain', {'entry_z': 'main0_z', 'vertex_inputs': []}, {'_b': vz}) + self.shader('inputs', {'vertex_inputs': [{'location': 0}]}, {'_b': vz}) + self.shader('layer', {'pull': {}}, {'_b': vz, '_bp': (vz, True)}) + self.shader('other', {'chain': 'fedcba987654'}, {'_b': vz}) + self.shader('nolib', {'pull': {}}, {'_b': vz}) # no _bp + self.shader('fs', {}, {'_b': [('main0', False)]}) + self.shader('fsc', {}, {'_b': [('main0', True)]}) + self.shader('fsx', {'entry': 'main1'}, {'_b': [('main0', False)]}) + pairs = [('pull', 'fs'), ('plain', 'fs'), ('inputs', 'fs'), ('layer', 'fs'), ('other', 'fs'), ('nolib', 'fs'), ('pull', 'fsc'), ('pull', 'fsx'), ('pull', 'gone'), ('pull', 'fs')] + bases, rest, left = po.plan(self.d, CHAIN, pairs, False) + self.assertEqual(bases, [('pull_bp', 'main0', 'fs_b', 'main0'), ('plain_b', 'main0', 'fs_b', 'main0')]) + self.assertEqual(rest, [('pull', 'fsc')]) + self.assertEqual(left, {'reflection': 2, 'stage inputs': 1, 'library': 1, 'layer': 1, 'entry': 1}) + bases, _, _ = po.plan(self.d, CHAIN, pairs[:2], True) # near = 0: the entry_z entries + self.assertEqual(bases, [('pull_bp', 'main0_z', 'fs_b', 'main0'), ('plain_b', 'main0_z', 'fs_b', 'main0')]) + + def test_script(self): + j = po.script(self.d, [('v_bp', 'main0', 'f_b', 'main0'), ('w_bp', 'main0_z', 'f_b', 'main0')]) + self.assertEqual([l['label'] for l in j['libraries']], ['v_bp', 'f_b', 'w_bp']) + self.assertEqual(j['libraries'][0]['path'], str(self.d.resolve() / 'v_bp.metallib')) + fns = j['function_descriptors']['library_function_descriptors'] + self.assertEqual([(f['label'], f['library'], f['name']) for f in fns], + [('v_bp.main0', 'lib:v_bp', 'main0'), ('f_b.main0', 'lib:f_b', 'main0'), ('w_bp.main0_z', 'lib:w_bp', 'main0_z')]) + p = j['pipeline_descriptors']['render_pipeline_descriptors'] + self.assertEqual([(x['vertex_function_descriptor'], x['fragment_function_descriptor']) for x in p], + [('fnd:v_bp.main0', 'fnd:f_b.main0'), ('fnd:w_bp.main0_z', 'fnd:f_b.main0')]) + self.assertEqual(set(p[0]), {'vertex_function_descriptor', 'fragment_function_descriptor', 'color_attachments'}) # no vertex descriptor, one sample: the defaults + self.assertEqual(len(p[0]['color_attachments']), 8) + self.assertTrue(all(v == 'Unspecialized' for a in p[0]['color_attachments'] for v in a.values())) + self.assertEqual(len(p[0]['color_attachments'][0]), 9) + + +class Build(unittest.TestCase): + """main: one archive per architecture, made again only when its key changes; the stamp, rest_pairs.tsv and archive.tsv beside them""" + def setUp(self): + self.t = Path(tempfile.mkdtemp()); self.addCleanup(shutil.rmtree, self.t) + self.lib = self.t / f'b-{CHAIN}'; self.lib.mkdir() + for k, j, funcs in (('v', {'pull': {}, 'entry_z': 'main0_z'}, [('main0', False), ('main0_z', False)]), ('f', {}, [('main0', False)]), ('c', {}, [('main0', True)])): + (self.lib / f'{k}_b.json').write_text(json.dumps(dict({'chain': CHAIN, 'entry': 'main0'}, **j))) + (self.lib / f'{k}_b.metallib').write_bytes(metallib(funcs)) + (self.lib / 'v_bp.metallib').write_bytes(metallib([('main0', False), ('main0_z', False)])) + (self.lib / 'inputs.tsv').write_text('game\tX\ninputs\tY\n') + self.prof = self.t / 'games/0100000000001000'; self.prof.mkdir(parents=True) + (self.prof / 'game.tsv').write_text('title_id\t0100000000001000\nname\tTest\ndepth_mode\tzero\n') + (self.prof / 'shader_pairs.tsv').write_text('# pairs\nv\tf\nv\tc\n') + self.out, self.work, self.calls = self.t / 'out', self.t / 'work', [] + + def fake_run(self, cmd, **kw): + cmd = [str(c) for c in cmd] + if '-version' in cmd: return subprocess.CompletedProcess(cmd, 0, 'AIR-TT 1.0\n', '') + if '--show-sdk-version' in cmd: return subprocess.CompletedProcess(cmd, 0, '27.1\n', '') + self.calls.append(cmd); Path(cmd[cmd.index('-o') + 1]).write_bytes(b'archive ' + cmd[cmd.index('-arch') + 1].encode()) + return subprocess.CompletedProcess(cmd, 0) + + def main(self, *extra): + args = ['--game', '0100000000001000', '--msl', str(self.lib), '--out', str(self.out), '--work', str(self.work), *extra] + with mock.patch.object(po.subprocess, 'run', self.fake_run), mock.patch.dict(os.environ, DM_GAMES_DIR=str(self.prof.parent)), contextlib.redirect_stdout(io.StringIO()): + return po.main(args) + + def test_build_reuse_and_stamp(self): + self.assertEqual(self.main('--gpu', 'g18p'), 0) + self.assertEqual(len(self.calls), 1) + c = self.calls[0] + self.assertEqual(c[:8], ['xcrun', '-sdk', 'iphoneos', 'metal-tt', '-arch', 'applegpu_g18p', '-ios-version-min', '27.0']) + self.assertTrue(c[8].endswith('.mtl4-json')) # the extension metal-tt takes the Metal 4 format from + script = json.loads(Path(c[8]).read_text()) + self.assertEqual(script['pipeline_descriptors']['render_pipeline_descriptors'][0]['vertex_function_descriptor'], 'fnd:v_bp.main0_z') # the profile's depth_mode zero + self.assertEqual((self.out / 'applegpu_g18p.mtl4a').read_bytes(), b'archive applegpu_g18p') + text = (self.prof / 'shader_pairs.tsv').read_bytes() + self.assertEqual((self.out / 'prebuilt.tsv').read_text(), po.stamp(CHAIN, True, text)) + self.assertEqual((self.out / 'rest_pairs.tsv').read_text(), 'v\tc\n') + self.assertIn('bases\t1\n', (self.out / 'archive.tsv').read_text()) + self.assertEqual(self.main('--gpu', 'g18p'), 0); self.assertEqual(len(self.calls), 1) # unchanged: reused + self.assertEqual(self.main('--gpu', 'g18p,applegpu_g17p'), 0) # another architecture: only it is built + self.assertEqual([x[5] for x in self.calls[1:]], ['applegpu_g17p']) + self.assertEqual(self.main('--gpu', 'g17p', '--os-min', '26.0'), 0) # another minimum: built again; g18p no longer asked for: removed + self.assertEqual(self.calls[-1][7], '26.0') + self.assertEqual(sorted(p.name for p in self.out.glob('*.mtl4a')), ['applegpu_g17p.mtl4a']) + (self.prof / 'shader_pairs.tsv').write_text('v\tf\n') # other pairs: built again, no pair left to the device + self.assertEqual(self.main('--gpu', 'g17p', '--os-min', '26.0'), 0) + self.assertEqual(len(self.calls), 4) + self.assertFalse((self.out / 'rest_pairs.tsv').exists()) + + def test_depth_override_and_bad_key(self): + self.main('--depth', 'gl', '--script-only') + self.assertIn('depth\tnear=-w\n', (self.out / 'prebuilt.tsv').read_text()); self.assertEqual(self.calls, []) + (self.prof / 'game.tsv').write_text('title_id\t0100000000001000\nname\tTest\ndepth_mode\tnear0\n') + with self.assertRaises(SystemExit): self.main('--script-only') + + def test_library_content_invalidates_archive(self): + self.main() + for name in ('v_bp', 'f_b'): + with self.subTest(library=name): + lib = self.lib / f'{name}.metallib' + lib.write_bytes(lib.read_bytes() + b'changed GPU program') # same entries, reflection and profile inputs + before = len(self.calls) + self.main() + self.assertEqual(len(self.calls), before + 1) + self.main() + self.assertEqual(len(self.calls), before + 1) # unchanged bytes still reuse the archive + + def test_constant_only_pairs_are_left_to_device(self): + self.main() # remove the previous archive when no pair can be built offline anymore + (self.prof / 'shader_pairs.tsv').write_text('v\tc\n') + before = len(self.calls) + self.assertEqual(self.main(), 0) + self.assertEqual(len(self.calls), before) + self.assertEqual(list(self.out.glob('*.mtl4a')), []) + self.assertEqual((self.out / 'rest_pairs.tsv').read_text(), 'v\tc\n') + self.assertEqual((self.out / 'prebuilt.tsv').read_text(), po.stamp(CHAIN, True, b'v\tc\n')) + self.assertIn('bases\t0\n', (self.out / 'archive.tsv').read_text()) + + def test_metal_tt_failure_leaves_no_archive(self): + def fail(cmd, **kw): + cmd = [str(c) for c in cmd] + if '-version' in cmd or '--show-sdk-version' in cmd: return self.fake_run(cmd) + return subprocess.CompletedProcess(cmd, 1) + with mock.patch.object(po.subprocess, 'run', fail), mock.patch.dict(os.environ, DM_GAMES_DIR=str(self.prof.parent)), contextlib.redirect_stdout(io.StringIO()), self.assertRaises(SystemExit): + po.main(['--game', '0100000000001000', '--msl', str(self.lib), '--out', str(self.out), '--work', str(self.work)]) + self.assertEqual(list(self.out.glob('*.mtl4a')), []) + + +class Default(unittest.TestCase): + """Which games get an archive without DM_PSO_GPUS: build_ipa.sh's own lines, run with game.py's default form, so only a profile with pso_mtl4 on (the Metal 4 path, which alone + reads the archive on iOS) gets g18p; an explicit DM_PSO_GPUS wins, also when empty""" + def gpus(self, profile, **env): + sh = (REPO / 'dm/ios/tools/build_ipa.sh').read_text().splitlines() + i = next(n for n, ln in enumerate(sh) if ln.startswith('PSO=; P4=')) + with tempfile.TemporaryDirectory() as d: + for k, v in profile.items(): Path(d, k).mkdir(); Path(d, k, 'game.tsv').write_text(v) + e = {k: v for k, v in os.environ.items() if k != 'DM_PSO_GPUS'} + e.update(env, DM=str(REPO / 'dm'), DM_GAMES_DIR=d, GD=d) + out = {} + for gid in profile: + r = subprocess.run(['bash', '-c', 'die() { echo "FAIL: $*"; exit 1; }\n' + '\n'.join(sh[i:i + 2]) + '\necho "$P4 $GPUS"'], env=dict(e, GAME=gid), + capture_output=True, text=True) + out[gid] = r.stdout.strip() if r.returncode == 0 else 'fail' + return out + + def test_by_profile(self): + games = {'0100000000F0F000': 'title_id\t0100000000F0F000\npso_mtl4\ton\n', '0100000000F0F001': 'title_id\t0100000000F0F001\n', + '0100000000F0F002': 'title_id\t0100000000F0F002\npso_mtl4\toff\n'} + self.assertEqual(self.gpus(games), {'0100000000F0F000': 'on g18p', '0100000000F0F001': 'off none', '0100000000F0F002': 'off none'}) + self.assertEqual(self.gpus(games, DM_PSO_GPUS='g17p'), {'0100000000F0F000': 'on g17p', '0100000000F0F001': 'off g17p', '0100000000F0F002': 'off g17p'}) + self.assertEqual(self.gpus(games, DM_PSO_GPUS='none')['0100000000F0F000'], 'on none') + self.assertEqual(self.gpus(games, DM_PSO_GPUS='')['0100000000F0F000'], 'on') + + def test_breath_of_the_wild(self): # the profile that turns the Metal 4 path on: its app gets the archive + r = subprocess.run([sys.executable, str(REPO / 'dm/games/game.py'), '01007EF00011E000', 'pso_mtl4', 'off'], capture_output=True, text=True) + self.assertEqual(r.stdout.strip(), 'on') + + +if __name__ == '__main__': + unittest.main() diff --git a/docs/agents/symptoms.md b/docs/agents/symptoms.md index 7daff48..d9dd9f9 100644 --- a/docs/agents/symptoms.md +++ b/docs/agents/symptoms.md @@ -417,6 +417,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - Metal shader validation reports invalid image-atomic stores even with an image-sized scratch buffer; without validation, GPU page faults can freeze rendering. → [Bounded MSL 2.3 atomic emulation](../reference/shaders.md#bounded-msl-23-atomic-emulation) - shaders missing from the map that no game file holds. → [Built-in shaders compiled with the game's own GLSLC](../reference/shaders.md#built-in-shaders-compiled-with-the-games-own-glslc) - a null read at `main+0x3316dc` in `nnMain`, right after the window setup (the program object's interface pointers at +0xd0 / +0xd8 both null). → [GLSLC GPU code version against the driver's range](../reference/shaders.md#glslc-gpu-code-version-against-the-drivers-range) +- a truncated `.mtl4a` reaches `MTL4ArchiveImpl::loadFromURL`; the bundle reports coverage with `0 of 0 probed bases found`; rebuilt shader libraries reuse an archive of their old binaries; or packing fails with `pso_offline: no base to build` when every eligible pair has function constants. → [Offline pipeline archive reuse and fallback](../reference/shaders.md#offline-pipeline-archive-reuse-and-fallback) - identical render pipelines compile repeatedly under different keys, lengthening startup prewarm and adding pipeline misses during play. → [Pipeline keys omit unused colour state](../reference/shaders.md#pipeline-keys-omit-unused-colour-state) - a vertex shader whose pairs changed was restored unpadded from `cas/`. → [Shader cache key covering the pairs](../reference/shaders.md#shader-cache-key-covering-the-pairs) - `fragmentFunction main0 cannot be used to build a pipeline state` (a Metal validation assertion). → [Function constants specialized at load](../reference/shaders.md#function-constants-specialized-at-load) diff --git a/docs/development.md b/docs/development.md index b85f3ea..f8798a9 100644 --- a/docs/development.md +++ b/docs/development.md @@ -103,6 +103,9 @@ python3 scripts/build_ipa_from_nsp.py ~/Games/NewGame -k ~/keys/prod.keys # test the guesses on a game that has a profile: build from a provisional one (the IPA is named -provisional.ipa) python3 scripts/build_ipa_from_nsp.py ~/Games/MarioGalaxy -k ~/keys/prod.keys --provisional --allow-damaged +# the pipeline archive for another GPU too, beside the A19 / A19 Pro (applegpu_g18p) that a game whose profile sets pso_mtl4 on gets by default; --gpu none leaves it out +python3 scripts/build_ipa_from_nsp.py ~/Games/BreathOfTheWild -k ~/keys/prod.keys --gpu g18p,g17p + # development: build only what running the game on this Mac needs (no iOS shaders, Xcode project or IPA) and print the command that runs it python3 scripts/build_ipa_from_nsp.py ~/Games/MarioWonder -k ~/keys/prod.keys --platform mac @@ -112,7 +115,7 @@ python3 scripts/build_ipa_from_nsp.py ~/Games/MarioWonder -k ~/keys/prod.keys -- `--platform` takes `ios` (the default: the IPA, or the Xcode project with `--prepare-xcode`), `mac` ([A Mac-only build for development](#a-mac-only-build-for-development)) and `mac-app` ([A Mac app](#a-mac-app)). -In Spyder, open the script, fill in the settings at the top (`NSP_DIR`, `PROD_KEYS`, and optionally `PACKAGE`, `OUT_IPA`, `APP_NAME`, `BUNDLE_ID`, `LANGUAGE`, `KEEP_EXTRACTED`, `SHOW_FPS`, `PLATFORM`; for a game without a profile `HEAP_CHECK`, `HEAP_CHECK_SECS`, `PROVISIONAL`), and press F5. Empty paths are asked for at run time. +In Spyder, open the script, fill in the settings at the top (`NSP_DIR`, `PROD_KEYS`, and optionally `PACKAGE`, `OUT_IPA`, `APP_NAME`, `BUNDLE_ID`, `LANGUAGE`, `KEEP_EXTRACTED`, `SHOW_FPS`, `PSO_GPUS`, `PLATFORM`; for a game without a profile `HEAP_CHECK`, `HEAP_CHECK_SECS`, `PROVISIONAL`), and press F5. Empty paths are asked for at run time. What the script does: 1. It finds the game profile from the title ID (the `[title ID]` in the file names, or the NSP's ticket; the update by its `[vN]` tag). The update is optional: it packs the newest version in the folder (`-p` / `PACKAGE` picks another). @@ -316,7 +319,7 @@ The bases are kept in a per-game pipeline archive in `pso_archive/` beside `pso_ Every start adds a file or more: Breath of the Wild wrote 5–7 per start, and each file a base is not in costs a lookup about 0.06 ms. With more than 8 files (`PSO_ARCH_COMPACT`) the archive is compacted: the bases the prewarm takes from the old files are taken again through the compiler, with those files as its lookup archives, which captures them; the flush after the prewarm writes them into one file, and the old files are removed (`arch_compact_finish`). Bases the prewarm did not take (no `pso_user.tsv` line names them) go with the old files and are built again when drawn. The log says `pipeline archive compacted: N files gave M bases …`. -The log shows `pipeline archive …: N files, … MB, opened in … ms` at start, `pipeline archive: N new pipeline bases written to …` after each flush, and after the prewarm a line counting the bases compiled, taken from the archive and reused. `DM_PSO_ARCHIVE=off` turns the archive off; `DM_PSO_ARCHIVE=` puts it elsewhere. `DM_PSO_MTL4=0` builds every pipeline through the `MTLDevice` as before, with no bases and no archive. The Metal 4 path, with the hybrid pipelines and the prebuild, is on by default on the Mac. On iOS it is on for a game whose profile sets `pso_mtl4 on` (Breath of the Wild, tested on an iPhone 17 Pro Max) or with `DM_PSO_MTL4=1`, and off otherwise: the archive's memory is not measured against an iPhone's limit yet, and an archive cannot be made offline for an iPhone, because `metal-tt` rejects the pipeline script the Metal 4 serializer writes. iOS is a later step. Pipelines whose functions have no Metal 4 descriptor (the diagnostic `DM_NVN_HILITE` and `DM_NVN_MAGSCAN` overrides) always take the `MTLDevice` path. +The log shows `pipeline archive …: N files, … MB, opened in … ms` at start, `pipeline archive: N new pipeline bases written to …` after each flush, and after the prewarm a line counting the bases compiled, taken from the archive and reused. `DM_PSO_ARCHIVE=off` turns the archive off; `DM_PSO_ARCHIVE=` puts it elsewhere. `DM_PSO_MTL4=0` builds every pipeline through the `MTLDevice` as before, with no bases and no archive. The Metal 4 path, with the hybrid pipelines and the prebuild, is on by default on the Mac. On iOS it is on for a game whose profile sets `pso_mtl4 on` (Breath of the Wild, tested on an iPhone 17 Pro Max) or with `DM_PSO_MTL4=1`, and off otherwise: the archive's memory is not measured against an iPhone's limit yet. For such a game the standard on iOS is the prebuild's bases made on the Mac and shipped in the app ([The pipeline archive made offline](#the-pipeline-archive-made-offline)); the device builds them itself only as a fallback. Pipelines whose functions have no Metal 4 descriptor (the diagnostic `DM_NVN_HILITE` and `DM_NVN_MAGSCAN` overrides) always take the `MTLDevice` path. During compaction, timed flushes defer while prewarm runs. Once it finishes, one archive-queue task flushes the recaptured bases and removes the old files. The pending count and serializer capture are protected by the same compile/flush lock, so a flush consumes the count for the bases it writes. @@ -347,10 +350,26 @@ Every 300 frames the log has a `pipelines up to frame` line. It gives the submit The **prebuild** puts the pulling base of every known shader pair into the archive (`pso_prebuild`), so a cold start's draws have stand-ins. The pairs come from the profile's `shader_pairs.tsv` and `pso_pairs.tsv`. `shader_pairs.tsv` pairs the stages of one BNSH variation (and the other readers' programs). It held none of the pairs from agl's BAHS shader archives (`.sharcb`, the agl / gsys / terrain resource packs), and in Breath of the Wild 109 of the 543 pairs drawn on entering Zora's Domain came from those. `shader_split.py` now reads the binary and program arrays of a BAHS file (`bahs_variations`) and pairs the n-th vertex binary of a program with its n-th fragment binary. Those pairs go to `pso_pairs.tsv`, which only the prebuild reads. `vs_pad` keeps reading `shader_pairs.tsv`, so the shader libraries, the inputs digest and the other profiles' tables are unchanged. For Breath of the Wild that is 3,012 more pairs, and all 543 pairs drawn are now known. A profile made before the table existed still packs: the NSP script then keeps the table beside the shader package and hands it to the prebuild (`DM_PSO_PAIRS`). Super Mario Odyssey, Mario Bros. Wonder, Mario Kart 8 Deluxe, 3D World, Animal Crossing, Captain Toad and Tears of the Kingdom have BAHS files too; their tables are made at their next pack. -With the prepare step (a window, the iPhone) the prebuild runs right after it, before the game's first frame, behind the same progress screen ("Preparing pipelines"), on every core, flushing every 1,000 bases; a first start waits for it once. Built in the background while playing instead, they took an iPhone 17 Pro Max's GPU from about 13 to 15–18 ms a frame in Breath of the Wild, through the compiler's load on the shared power budget. Without a prepare step it still runs in the background after the prewarm, on a utility queue on half the performance cores (`DM_PSO_PREBUILD=0` turns it off either way). On the Mac it also runs as a tool, `DM_PSO_PREBUILD=tool`: build, write, exit. The NSP script runs the tool in a `--platform mac` build, into `_build/run/pso_archive` (log `pso_prebuild.log`). It runs once per chain version, depth mode and pair list (`pso_archive/prebuilt.tsv`). `DM_PSO_PREBUILD_DEPTH=zero` serves a game that sets the near=0 depth mode; `DM_PSO_PREBUILD_THREADS=` sets the workers. Its files hold bases without a vertex layout only, and are named `pNNNNN.mtl4a`; a base with a vertex layout is never looked up in them, and compaction leaves them alone. Breath of the Wild on an M2 Max: 15,767 bases, 910 MB in 16 files, 187 s with 7 workers for the first 12,755 and 19 s for the 3,012 more. Opening the files took 2.2 s on the first start (0.25 s from the file cache). The host's footprint is about 850 MB larger with them open (2,067 against 1,215 MiB at the title), which is one reason the Metal 4 path stays off on iOS unless a game's profile was tested with it (`pso_mtl4`). iOS archives cannot be made offline either: `metal-tt` rejects the pipeline script the Metal 4 serializer writes (format 0.1: "expecting [ instead got {"). On iOS this is a later step. +With the prepare step (a window, the iPhone) the prebuild runs right after it when the bases are not there yet, before the game's first frame, behind the same progress screen ("Preparing pipelines"), on every core, flushing every 1,000 bases; a first start waits for it once. Built in the background while playing instead, they took an iPhone 17 Pro Max's GPU from about 13 to 15–18 ms a frame in Breath of the Wild, through the compiler's load on the shared power budget. Without a prepare step it still runs in the background after the prewarm, on a utility queue on half the performance cores (`DM_PSO_PREBUILD=0` turns it off either way). On the Mac it also runs as a tool, `DM_PSO_PREBUILD=tool`: build, write, exit. The NSP script runs the tool in a `--platform mac` build, into `_build/run/pso_archive` (log `pso_prebuild.log`): this is how the Mac gets its archive. It runs once per chain version, depth mode and pair list (`pso_archive/prebuilt.tsv`: the pair lists' size and SHA-256). `DM_PSO_PREBUILD_DEPTH=zero` serves a game that sets the near=0 depth mode (the NSP script sets it from the profile key `depth_mode`); `DM_PSO_PREBUILD_THREADS=` sets the workers. Its files hold bases without a vertex layout only, and are named `pNNNNN.mtl4a`; a base with a vertex layout is never looked up in them, and compaction leaves them alone. Breath of the Wild on an M2 Max: 15,767 bases, 910 MB in 16 files, 187 s with 7 workers for the first 12,755 and 19 s for the 3,012 more. Opening the files took 2.2 s on the first start (0.25 s from the file cache). The host's footprint is about 850 MB larger with them open (2,067 against 1,215 MiB at the title), which is one reason the Metal 4 path stays off on iOS unless a game's profile was tested with it (`pso_mtl4`). On iOS the standard is the archive made on the Mac and shipped in the app (next section); the device's own prebuild is the fallback, for an app without a usable archive. Breath of the Wild's cold starts with and without the prebuild and `pso_pairs.tsv`: [its note](../dm/games/01007EF00011E000/README.md#pipelines-on-a-cold-start). +### The pipeline archive made offline + +On iOS the standard is the pipeline archive made on the Mac and shipped in the app. Otherwise the device builds the bases on first start behind the "Preparing pipelines" screen; [Breath of the Wild's note](../dm/games/01007EF00011E000/README.md#pipelines-on-a-cold-start) records the build times, sizes and startup measurements. `build_ipa.sh` makes them on the Mac and puts them into the app (`dm/gfx/tools/pso_offline.py`). Only the Metal 4 path reads them, so by default it does this for a game whose profile sets `pso_mtl4 on` (the path is on for it on iOS), and for no other: +1. It writes a Metal 4 pipelines script (`metal4-pipelines-script(5)`) with the base of every pair in `shader_pairs.tsv` and `pso_pairs.tsv`, by the prebuild's rules (`pre_desc` in `nvn_mtl.m`): the pulling vertex library `_bp` (a vertex shader without stage inputs: its own `_b`), the vertex entry for the depth mode, the pixel shader's entry, no vertex descriptor, one sample, every colour attachment unspecialized. Pairs the prebuild leaves to the draws are left out the same way. A pair whose functions declare function constants is left to the device (`rest_pairs.tsv`): `metal-tt` makes no specialized function from a library ("cannot find private metadata"), and the host specializes such functions with an empty set. If every eligible pair needs this specialization, the tool writes the remainder list without running `metal-tt` or shipping an archive, and the device builds them all. +2. `metal-tt` compiles the script into one archive per GPU architecture: `xcrun -sdk iphoneos metal-tt -arch applegpu_g18p -ios-version-min 27.0 bases.mtl4-json -o applegpu_g18p.mtl4a`. The script needs the `.mtl4-json` extension, from which `metal-tt` takes the format: as `.mtlp-json` (Metal 3's format) it rejected even the Metal 4 serializer's own capture ("expecting [ instead got {"), which it compiles under the right extension. The archive refers to the libraries by content, not by path, so the app's own copies (`msl/b-`) find their bases. +3. The archives go into the app as `pso/` with `prebuilt.tsv`, the stamp the device's prebuild would write for the same chain version, depth mode and pair lists. With an archive, or for a game whose profile sets `pso_mtl4 on` (its hybrid pipelines draw with them), the IPA also packs the pulling libraries (`_bp`), which the bases are made of; an Xcode build copies them anyway. + +`main.m` hands `pso/` to the host as `DM_PSO_BUNDLE` (`off` ignores it). The host opens the file named after the device's GPU architecture (`MTLDevice.architecture`, e.g. `applegpu_g18p.mtl4a`) read only and looks it up first among the files of bases without a vertex layout (`pso_bundle_open`); it never writes or compacts it. Before the prebuild, `pso_bundle_covers` compares the bundled `prebuilt.tsv` with this run's stamp, then looks up the bases of 16 pairs spread over the list. Coverage needs at least one actual lookup and no misses; ineligible sampled pairs supply no evidence. If both hold, the device builds no base, only the pairs in `rest_pairs.tsv`; otherwise it runs its own prebuild as before, which still takes from the bundle every base it finds there. An archive for another GPU, one the system cannot open (made for a newer system, or an OS update changed the GPU compiler), a probe that finds nothing, or a stamp of another depth mode each give an info line (`pipeline archive bundled with the app: …`), not a red one. The on-device prebuild is the fallback: it runs when the app has no archive for the device's GPU, when the system cannot open the archive or take its bases (an iOS update that changed the GPU compiler, a system older than its minimum), or when its stamp is not this run's (another depth mode, other pairs). The device's own archive (`pso_archive/`) gets only what the bundle lacks: a draw, the prewarm and the prebuild compile a base, which the serializer then captures, only after the bundle missed it, and bases with a vertex layout, which the bundle never holds (`t_pso_bundle` checks a covered pair's draw: no compile, nothing captured). The Mac builds its own archive as before, with the NSP script's prebuild tool ([Hybrid pipelines](#hybrid-pipelines)). Metal crashed (`MTL4ArchiveImpl::loadFromURL`) when handed a file that was no archive, a text file or a bare Mach-O header, so `arch_file_ok` checks the Mach-O load commands and payload ranges, universal slice bounds and the serializer's bundled Metal library ranges before opening a file; this holds for the device's own files too. It checks the container structure; the compiler's private payload remains Metal's responsibility. + +The settings: +- `DM_PSO_GPUS` (`build_ipa.sh`; the NSP script's `--gpu` / `PSO_GPUS`): the architectures, comma-separated, with or without the `applegpu_` prefix. Unset (or `PSO_GPUS` empty), it follows the profile: `g18p`, the A19 and A19 Pro of the iPhone 17 family, for a game that sets `pso_mtl4 on`, and `none` for the others, whose app would carry an archive it never reads (a test with `DM_PSO_MTL4=1` in `Documents/env.txt` sets it explicitly). Another device's name is its `MTLDevice.architecture`, which its log names when the app has no archive for it (`pipeline archive bundled with the app: none for this GPU (…)`). `none` builds no archive. Each architecture increases build time and app size; see the measurements in the game note linked above. +- `DM_PSO_IOS_MIN`: the archives' minimum iOS version. The default is the SDK's major version (`27.0` with the iOS 27 SDK); a device on an older system cannot open them and builds the bases itself. +- The profile key `depth_mode` (`gl`, the default, or `zero`) picks the vertex entry (`main0` or `main0_z`). The host learns the game's depth mode only when the game sets it, so the build cannot derive it. A wrong value costs only the device's own prebuild, as without the archive. + +An archive is made again only when its inputs change: the consumed libraries' bytes and stamp (`inputs.tsv`), the stamp above, the architecture, the minimum OS, `metal-tt`'s version and `pso_offline.py` itself (keys in `$W/build/ios/pso/work`). A shader rebuild with another Metal compiler can change the library bytes without changing the profile or chain stamp, so the stamp alone is insufficient. Archive validation and fallback regressions are recorded under [Offline pipeline archive reuse and fallback](reference/shaders.md#offline-pipeline-archive-reuse-and-fallback). + ## Power The game's own work and its frame rate stay as they are; the host avoids work of its own that the Switch never did. Every 300 frames the log has a `power` line (CPU energy in watts and per frame, CPU time, wakeups, thermal state, the threads that yield most) and a `residency set` line; [05](agents/05-debugging.md) explains them. diff --git a/docs/reference/rendering.md b/docs/reference/rendering.md index 81eba8d..776b448 100644 --- a/docs/reference/rendering.md +++ b/docs/reference/rendering.md @@ -363,8 +363,8 @@ The replay on Metal (`dm/gfx/nvn/nvn_mtl.m`) and the Vulkan and OpenGL layers. E - **Note:** the run-time format decode costs GPU time: on an M2 Max, 8 attributes in a vertex-bound draw of tiny visible triangles took 65 % longer (about 1.0 ms per million vertices) than with the vertex descriptor. With the formats as compile-time constants it was 9 %, and with the whole layout compiled in, 0. So these pipelines only stand in until the fast one is built. Other entry points (transform feedback, geometry capture, layer) get no variant; shaders without stage inputs need none, since their pipelines have no vertex descriptor. - **Symptom:** an unsigned iOS build reports `[RED] pulling library ... not usable (no library)` and its draws wait for pipelines even with the profile's `pso_mtl4 on`, or packaging with `IOS_PULL=1` fails with `SKIPS[@]: unbound variable`. - **Cause:** the IPA packer used to strip `_bp.metallib` unless `IOS_PULL=1`, independently of the runtime's profile default. The inclusion path expanded an empty array under `set -u`, which macOS Bash 3.2 treats as unset. -- **Fix:** `build_ipa.sh` takes the default library inclusion from the generated `DM_GAME_PSO_MTL4` compiled into the host; `IOS_PULL=0` / `1` still overrides it. Packing options always contain `--store`, so both inclusion and exclusion work under Bash 3.2. A runtime `DM_PSO_MTL4=1` override for a profile with the path off needs an IPA built with `IOS_PULL=1` too. -- **Check:** `dm/gfx/tests/t_pull.m` (the words against Metal's fetch and the CPU conversion), `t_pull_inv.m` (a depth prepass with one pipeline and depth-equal with the other), `test_dmsh_pull.py`; `dm/ios/tests/test_profile_pack.py` packs tiny fixtures through the actual archive step and checks profile defaults and explicit overrides in the ZIP. +- **Fix:** `build_ipa.sh` takes the default library inclusion from the generated `DM_GAME_PSO_MTL4` compiled into the host; `IOS_PULL=0` / `1` still overrides it. Packing options always contain `--store`, so both inclusion and exclusion work under Bash 3.2. A runtime `DM_PSO_MTL4=1` override for a profile with the path off needs an IPA built with `IOS_PULL=1` too. An IPA that packs a [pipeline archive made offline](../development.md#the-pipeline-archive-made-offline) (`pso/`) always carries the `_bp` libraries, whatever `IOS_PULL` says, since the archive's bases are made of them. +- **Check:** `dm/gfx/tests/t_pull.m` (the words against Metal's fetch and the CPU conversion), `t_pull_inv.m` (a depth prepass with one pipeline and depth-equal with the other), `test_dmsh_pull.py`; `dm/ios/tests/test_profile_pack.py` packs tiny fixtures through the actual archive step and checks profile defaults, explicit overrides and an IPA with a pipeline archive in the ZIP. - **Found on:** [The Legend of Zelda: Breath of the Wild](../../dm/games/01007EF00011E000/README.md): draws that waited for their pipeline on a cold start. ### CPU conversion of vertex attributes the input type cannot take diff --git a/docs/reference/shaders.md b/docs/reference/shaders.md index b153c09..123fe8c 100644 --- a/docs/reference/shaders.md +++ b/docs/reference/shaders.md @@ -696,6 +696,14 @@ shader on a GPU. ## Pipelines and caches +### Offline pipeline archive reuse and fallback + +- **Symptom:** a truncated `.mtl4a` reaches `MTL4ArchiveImpl::loadFromURL`; the bundle reports coverage with `0 of 0 probed bases found`; rebuilt shader libraries reuse an archive of their old binaries; or packing fails with `pso_offline: no base to build` when every eligible pair has function constants. +- **Cause:** the archive guard checked only four magic bytes; coverage accepted `found == probed` even when every sampled pair was ineligible; the offline cache key included library paths and profile stamps but omitted library bytes; and a zero-base plan aborted before writing its device remainder list. +- **Fix:** `arch_file_ok` checks the GPU Mach-O load commands and payload ranges, universal slice bounds and the serializer's bundled Metal library ranges before opening the archive. It does not validate the compiler's private payload. `pso_bundle_covers` requires at least one actual lookup and no misses. `pso_offline.py` hashes every consumed library's bytes into the cache key; when all eligible pairs need device specialization, it removes old archives and writes `rest_pairs.tsv` without invoking `metal-tt`. +- **Found on:** [The Legend of Zelda: Breath of the Wild](../../dm/games/01007EF00011E000/README.md#found-on-this-game), during review of its offline archive path with synthetic libraries and pair lists; no new gameplay or iPhone run. +- **Check:** `dm/tools/tests/test_pso_offline.py` changes vertex and fragment library bytes without changing entries or stamps, and exercises the all-constant plan. `t_pso_bundle` rejects bare headers, truncated archives, malformed load commands and out-of-range universal slices, accepts real offline and serialized archives, and rejects zero-probe coverage. + ### Pipeline keys omit unused colour state - **Symptom:** identical render pipelines compile repeatedly under different keys, lengthening startup prewarm and adding pipeline misses during play. diff --git a/scripts/build_ipa_from_nsp.py b/scripts/build_ipa_from_nsp.py index b024610..0944c3c 100755 --- a/scripts/build_ipa_from_nsp.py +++ b/scripts/build_ipa_from_nsp.py @@ -48,6 +48,9 @@ LANGUAGE = '' # game language, a code or a name: 'ja', # DM_LANG= in the app's Documents/env.txt changes it without a rebuild KEEP_EXTRACTED = True # keep the extracted game files after the IPA is built (saves extraction next time; as big as the game files); False deletes them DM_ENV_FILE = '' # optional: environment switches for a diagnostic build (one KEY=VALUE per line) +PSO_GPUS = '' # NVN games: GPU architectures the app gets the shader pairs' pipeline bases for, made on this Mac (metal-tt, comma-separated: 'g18p' = A19 / A19 Pro of the iPhone 17 family; + # another device's name is in its log when the app has none for it), so the device does not build them on its first start; each adds about 650 MB to the app (Breath of the Wild) and minutes to a first build; 'none' = no archive; + # empty = by the game's profile: 'g18p' when it sets pso_mtl4 on (the Metal 4 path, the only one that reads the archive on iOS, is on for it), else 'none' SHOW_FPS = True # True shows the game's frame rate on the iPhone (frames on screen per second, top left); DM_SHOW_FPS=0 / 1 in the app's Documents/env.txt changes it without a rebuild SURVEY_ONLY = False # a game with no profile in dm/games: only write the survey, no provisional profile and no IPA PROFILE_OUT = '' # where a provisional profile goes (empty = /_build/profile; the profile is <this>/<title ID>/) @@ -1471,6 +1474,7 @@ class IpaWatch: s = re.sub(r'^\[[\d:]+\] ', '', line) with self.lock: # a poll sees a step change whole if s.startswith('generate Gen/'): self.state = 'gen'; PROG.set(step='generating sources') + elif s.startswith('pipeline archive for'): PROG.set(step='pipeline archive (metal-tt)') elif s.startswith('xcodebuild'): units = len(os.listdir(self.ios / 'Gen/src')) + 2 + sum(1 for p in (self.ios / 'Sources').iterdir() if p.suffix in ('.c', '.m')) # + Gen/main/main.c and Gen/embed.S self.state, self.t = 'xcode', time.time(); PROG.set(step='xcodebuild', done=0, total=units, unit='files', est=True) # an incremental build compiles fewer @@ -1548,9 +1552,10 @@ def mac_run(work, game, patched, host, font, msl, aoc, gfx, lang, env, logs, g, if not g.table('pso_pairs.tsv').is_file() and (work / 'shader/pso_pairs.tsv').is_file(): renv['DM_PSO_PAIRS'] = work / 'shader/pso_pairs.tsv' # split_shaders' copy # The pipeline prebuild (docs/development.md, Hybrid pipelines): the pulling base of every shader pair into run/pso_archive, so the first run's draws have stand-ins instead # of waiting for the Metal compiler. The host's tool mode skips it when pso_archive/prebuilt.tsv matches (chain version, depth mode, pair list): Breath of the Wild's - # 12,755 pairs took about 3 minutes on an M2 Max the first time. It assumes the GL depth mode; a game that sets near=0 builds its bases again in the background in the app + # 12,755 pairs took about 3 minutes on an M2 Max the first time. It takes the depth mode from the profile's depth_mode key (default gl, NVN's); with a wrong one the game builds its bases again in the app PROG.add(1); PROG.stage('Pipeline prebuild', step='building') penv = dict(env, **{k: str(v) for k, v in renv.items()}, DM_PSO_PREBUILD='tool') + if g.get('depth_mode', 'gl') == 'zero': penv['DM_PSO_PREBUILD_DEPTH'] = 'zero' # a game that sets the near = 0 depth mode (the profile says so: the host learns it only at run time) run([host], logs / 'pso_prebuild.log', env=penv, cwd=rd) PROG.end() q = lambda v: shlex.quote(str(v)) @@ -1610,6 +1615,8 @@ def parse_args(argv): 'it must be one the game supports, which the console lists') ap.add_argument('--keep-extracted', action=argparse.BooleanOptionalAction, help='keep the extracted game files after the IPA is built, for the next build (default: the KEEP_EXTRACTED setting, on)') ap.add_argument('--env-file', help='environment switches for a diagnostic build (one KEY=VALUE per line)') + ap.add_argument('--gpu', help='NVN games: GPU architectures the app gets its pipeline archive for, made on this Mac with metal-tt, comma-separated: g18p (A19 / A19 Pro), or the name a device\'s log gives when the app has no archive for its GPU; ' + 'none = no archive, the device builds the bases on its first start (default: the PSO_GPUS setting; when that is empty, g18p for a game whose profile sets pso_mtl4 on, else none)') ap.add_argument('--show-fps', action=argparse.BooleanOptionalAction, help='show the game\'s frame rate on the iPhone, frames on screen per second at the top left (default: the SHOW_FPS setting, on)') ap.add_argument('--survey-only', action='store_true', help='a game with no profile: only the survey, no provisional profile and no IPA') ap.add_argument('--allow-damaged', action='store_true', help='a damaged NSP (files failing the NCA hash check): write them as read and go on instead of stopping') @@ -1853,6 +1860,11 @@ def main(args=None): if env_file: ipa_env['DM_ENV_FILE'] = str(Path(os.path.expanduser(env_file)).resolve()) show_fps = SHOW_FPS if args.show_fps is None else args.show_fps ipa_env['DM_SHOW_FPS'] = '1' if show_fps else '0' + gpus = (args.gpu or PSO_GPUS).strip() + if gpus and not re.fullmatch(r'none|(applegpu_)?g\w+(,(applegpu_)?g\w+)*', gpus): raise BuildError(f'--gpu / PSO_GPUS takes GPU architectures such as g18p,g17p, or none: {gpus}') + if not gpus: gpus = 'g18p' if g.get('pso_mtl4', 'off') == 'on' else 'none' # by the profile, as build_ipa.sh decides without DM_PSO_GPUS: only the Metal 4 path reads the archive on iOS + ipa_env['DM_PSO_GPUS'] = gpus + if gfx == 'nvn': say(f'pipeline archive: {"none, the device builds the bases on its first start if it takes the Metal 4 path (--gpu g18p builds one)" if gpus == "none" else gpus + " (made on this Mac with metal-tt; --gpu none leaves it out)"}') if show_fps: say('frame-rate counter: on (DM_SHOW_FPS=0 in the app\'s Documents/env.txt turns it off)') if prepare_xcode: PROG.stage('prepare Xcode')