Build and ship offline Metal pipeline archives (#226)

Compile eligible shader-pair bases on the Mac with metal-tt, package one
archive per selected iPhone GPU, and use a matching bundle before falling
back to on-device preparation. Carry the profile depth mode through packing
and include the pulling libraries required by the bundled bases.

Integrate PR #226 with four confirmed fixes: validate archive container
bounds before invoking Metal; require at least one actual successful bundle
probe; hash consumed library bytes into offline cache keys; and emit device
remainder lists for constant-only plans instead of aborting packaging.
Keep game measurements in the game note and document the general findings.

Validation on current main: 52 real Metal bundle checks, 57 archive checks,
79 prewarm checks on each pipeline path, 73 Python regressions covering
offline tools, packaging, documentation and chain checks, and compilation
of all 133 iOS source files. No new iPhone gameplay validation. The archive guard validates
container structure, not private Metal payloads.

Co-authored-by: v0lumehi <aljic.zijad@gmail.com>
This commit is contained in:
seeddsandv0lumehi committed 2026-10-26 10:52:50 +08:00
1 parent c6c4fac5c3
commit 92fc2f2711
35 files changed
+986 -51

No files matched your search

+2 -2
View File
@@ -363,8 +363,8 @@ The replay on Metal (`dm/gfx/nvn/nvn_mtl.m`) and the Vulkan and OpenGL layers. E
- **Note:** the run-time format decode costs GPU time: on an M2 Max, 8 attributes in a vertex-bound draw of tiny visible triangles took 65 % longer (about 1.0 ms per million vertices) than with the vertex descriptor. With the formats as compile-time constants it was 9 %, and with the whole layout compiled in, 0. So these pipelines only stand in until the fast one is built. Other entry points (transform feedback, geometry capture, layer) get no variant; shaders without stage inputs need none, since their pipelines have no vertex descriptor.
- **Symptom:** an unsigned iOS build reports `[RED] pulling library ... not usable (no library)` and its draws wait for pipelines even with the profile's `pso_mtl4 on`, or packaging with `IOS_PULL=1` fails with `SKIPS[@]: unbound variable`.
- **Cause:** the IPA packer used to strip `_bp.metallib` unless `IOS_PULL=1`, independently of the runtime's profile default. The inclusion path expanded an empty array under `set -u`, which macOS Bash 3.2 treats as unset.
- **Fix:** `build_ipa.sh` takes the default library inclusion from the generated `DM_GAME_PSO_MTL4` compiled into the host; `IOS_PULL=0` / `1` still overrides it. Packing options always contain `--store`, so both inclusion and exclusion work under Bash 3.2. A runtime `DM_PSO_MTL4=1` override for a profile with the path off needs an IPA built with `IOS_PULL=1` too.
- **Check:** `dm/gfx/tests/t_pull.m` (the words against Metal's fetch and the CPU conversion), `t_pull_inv.m` (a depth prepass with one pipeline and depth-equal with the other), `test_dmsh_pull.py`; `dm/ios/tests/test_profile_pack.py` packs tiny fixtures through the actual archive step and checks profile defaults and explicit overrides in the ZIP.
- **Fix:** `build_ipa.sh` takes the default library inclusion from the generated `DM_GAME_PSO_MTL4` compiled into the host; `IOS_PULL=0` / `1` still overrides it. Packing options always contain `--store`, so both inclusion and exclusion work under Bash 3.2. A runtime `DM_PSO_MTL4=1` override for a profile with the path off needs an IPA built with `IOS_PULL=1` too. An IPA that packs a [pipeline archive made offline](../development.md#the-pipeline-archive-made-offline) (`pso/`) always carries the `_bp` libraries, whatever `IOS_PULL` says, since the archive's bases are made of them.
- **Check:** `dm/gfx/tests/t_pull.m` (the words against Metal's fetch and the CPU conversion), `t_pull_inv.m` (a depth prepass with one pipeline and depth-equal with the other), `test_dmsh_pull.py`; `dm/ios/tests/test_profile_pack.py` packs tiny fixtures through the actual archive step and checks profile defaults, explicit overrides and an IPA with a pipeline archive in the ZIP.
- **Found on:** [The Legend of Zelda: Breath of the Wild](../../dm/games/01007EF00011E000/README.md): draws that waited for their pipeline on a cold start.
### CPU conversion of vertex attributes the input type cannot take
+8
View File
@@ -696,6 +696,14 @@ shader on a GPU.
## Pipelines and caches
### Offline pipeline archive reuse and fallback
- **Symptom:** a truncated `.mtl4a` reaches `MTL4ArchiveImpl::loadFromURL`; the bundle reports coverage with `0 of 0 probed bases found`; rebuilt shader libraries reuse an archive of their old binaries; or packing fails with `pso_offline: no base to build` when every eligible pair has function constants.
- **Cause:** the archive guard checked only four magic bytes; coverage accepted `found == probed` even when every sampled pair was ineligible; the offline cache key included library paths and profile stamps but omitted library bytes; and a zero-base plan aborted before writing its device remainder list.
- **Fix:** `arch_file_ok` checks the GPU Mach-O load commands and payload ranges, universal slice bounds and the serializer's bundled Metal library ranges before opening the archive. It does not validate the compiler's private payload. `pso_bundle_covers` requires at least one actual lookup and no misses. `pso_offline.py` hashes every consumed library's bytes into the cache key; when all eligible pairs need device specialization, it removes old archives and writes `rest_pairs.tsv` without invoking `metal-tt`.
- **Found on:** [The Legend of Zelda: Breath of the Wild](../../dm/games/01007EF00011E000/README.md#found-on-this-game), during review of its offline archive path with synthetic libraries and pair lists; no new gameplay or iPhone run.
- **Check:** `dm/tools/tests/test_pso_offline.py` changes vertex and fragment library bytes without changing entries or stamps, and exercises the all-constant plan. `t_pso_bundle` rejects bare headers, truncated archives, malformed load commands and out-of-range universal slices, accepts real offline and serialized archives, and rejects zero-probe coverage.
### Pipeline keys omit unused colour state
- **Symptom:** identical render pipelines compile repeatedly under different keys, lengthening startup prewarm and adding pipeline misses during play.