Fix iOS pulling shader packaging for profile defaults (#211)

Include pulling vertex libraries when the compiled game profile enables the
Metal 4 path, preserving explicit IOS_PULL overrides. Keep the pack options
array nonempty so inclusion also works with macOS Bash 3.2 and set -u.

Exercise the actual archive step with tiny shader fixtures for Breath of
the Wild and Wonder, checking both defaults and explicit overrides. Record
the packaging failures and fixes in the rendering reference and game note.

Co-authored-by: v0lumehi <aljic.zijad@gmail.com>

dm/games/01007EF00011E000/README.md

@@ -137,6 +137,7 @@ On this Mac the background prewarm also ends before the save loads, so the step
 
 General host changes first needed here; each applies to every game:
 
+- **Resolved: the IPA omitted pulling libraries with `pso_mtl4 on`, and explicitly including them failed under macOS Bash 3.2.** See [Pulling vertex inputs](../../../docs/reference/rendering.md#pulling-vertex-inputs).
 - **Pipeline keys omit disabled blend factors** (`pso_key_norm`): [Pipeline keys omit unused colour state](../../../docs/reference/shaders.md#pipeline-keys-omit-unused-colour-state).
 - **Resolved: GPU readback temporaries retained by a game thread's outer pool** (`mtl_read_texels`): [GPU readback without an autorelease pool](../../../docs/reference/memory.md#gpu-readback-without-an-autorelease-pool). Ported from MacNativeGames; checked here with a synthetic Metal readback lifetime test, without a new gameplay memory measurement.
 - **The host frame step in whole vsyncs** (`nvn_fstep.h`, `fstep_tick`, `FRAME_STEP` stubs, `fps_target`), explained in full under [Frame rate](#the-frame-step-current); its probes (`nvn_fstep_gpu_slowed`, `nvn_fstep_same`): [development.md, Power](../../../docs/development.md#power).

dm/ios/tests/test_profile_pack.py

@@ -0,0 +1,55 @@
+"""Pack tiny shader fixtures through build_ipa.sh's archive step, without Xcode or game files."""
+import os
+from pathlib import Path
+import subprocess
+import sys
+import tempfile
+import unittest
+import zipfile
+
+REPO = Path(__file__).resolve().parents[3]
+sys.path.insert(0, str(REPO / 'dm/games'))
+import game
+
+
+class ProfilePackTests(unittest.TestCase):
+    def test_pulling_libraries_follow_compiled_profile_and_explicit_override(self):
+        cases = [('01007EF00011E000', None, True), ('010015100B514000', None, False),
+                 ('01007EF00011E000', '0', False), ('010015100B514000', '1', True)]
+        for gid, override, included in cases:
+            with self.subTest(game=gid, IOS_PULL=override), tempfile.TemporaryDirectory() as temporary:
+                root = Path(temporary)
+                dm = root / 'dm'
+                header = dm / 'host/gen' / gid / 'game_profile.h'
+                header.parent.mkdir(parents=True)
+                header.write_text(game.header(game.Game(REPO / 'dm/games' / gid)))
+                app, shaders, romfs = [root / name for name in ('DmIOS.app', 'shaders', 'romfs')]
+                for folder in (app, shaders / 'b-fixture', romfs): folder.mkdir(parents=True)
+                for name in ('fixture_b.metallib', 'fixture_bp.metallib', 'fixture_bv.metallib', 'fixture_b.json'):
+                    (shaders / 'b-fixture' / name).write_text(name)
+                (app / 'executable').write_text('app')
+                (romfs / 'marker').write_text('romfs')
+                (root / 'tools').symlink_to(REPO / 'dm/ios/tools', target_is_directory=True)
+                out = root / 'fixture.ipa'
+                env = {k: v for k, v in os.environ.items() if k not in ('IOS_PULL', 'DM_GAMES_DIR', 'DM_GAMES_FIRST')}
+                env.update(GAME=gid, DM=str(dm), APP=str(app), MSL_IOS=str(shaders), MSLDIR='b-fixture',
+                           AOC='', ROMFS=str(romfs), OUT_IPA=str(out), PROFILE_TOOL=str(REPO / 'dm/games/game.py'))
+                if override is not None: env['IOS_PULL'] = override
+                # Exercise the actual archive step; only the preceding Xcode build is bypassed.
+                script = (REPO / 'dm/ios/tools/build_ipa.sh').read_text()
+                packing = script[script.index('\nP=Payload/DmIOS.app;'):]
+                result = subprocess.run(['/bin/bash', '-c', 'set -u\nt() { :; }; die() { echo "$*" >&2; exit 1; }\n'
+                                         'gp() { python3 "$PROFILE_TOOL" "$GAME" "$1"; }\n' + packing],
+                                        cwd=root, env=env, capture_output=True, text=True)
+                self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+                with zipfile.ZipFile(out) as archive:
+                    names = set(archive.namelist())
+                    prefix = 'Payload/DmIOS.app/msl/b-fixture/'
+                    self.assertEqual(prefix + 'fixture_bp.metallib' in names, included)
+                    for name in ('fixture_b.metallib', 'fixture_bv.metallib', 'fixture_b.json'):
+                        self.assertIn(prefix + name, names)
+                    self.assertEqual(archive.read('Payload/DmIOS.app/romfs/marker'), b'romfs')
+
+
+if __name__ == '__main__':
+    unittest.main()

dm/ios/tools/build_ipa.sh

@@ -136,7 +136,10 @@ P=Payload/DmIOS.app; MAPS=("$P=$APP" "$P/msl/$MSLDIR=$MSL_IOS/$MSLDIR")
 if [ -n "$AOC" ] && [ -d "$AOC" ]; then MAPS+=("$P/aoc=$AOC"); t "add-on content: $(ls "$AOC" | tr '\n' ' ')"; fi
 MAPS+=("$P/romfs=$ROMFS")
 t "pack IPA (already-compressed game files stored, everything else deflated)"
-# The pulling vertex libraries (<key>_bp, dm/gfx/tools/dmsh_pull.py) serve the hybrid pipelines, which are off on iOS (DM_PSO_MTL4=1 turns them on): left out unless IOS_PULL=1
-SKIPS=(); [ "${IOS_PULL:-0}" = 1 ] || SKIPS=(--skip-suffix _bp.metallib)
-python3 tools/pack_ipa.py "$OUT_IPA" --store "$(gp zip_store)" "${SKIPS[@]}" --skip $P/_CodeSignature --skip $P/embedded.mobileprovision --skip $P/msl --skip $P/aoc --skip $P/romfs "${MAPS[@]}" || die "pack_ipa"
+# The pulling vertex libraries (<key>_bp, dm/gfx/tools/dmsh_pull.py) serve the hybrid pipelines. Default to the same pso_mtl4 setting compiled by gen_build.py; IOS_PULL=0 / 1 overrides it.
+PULL=$(sed -n 's/^#define DM_GAME_PSO_MTL4 //p' "$DM/host/gen/$GAME/game_profile.h") || die "read game profile header"
+case "$PULL" in 0|1) ;; *) die "no DM_GAME_PSO_MTL4 in generated game profile";; esac
+# Keep this array nonempty: macOS Bash 3.2 treats an empty array as unset under set -u.
+PACK_OPTS=(--store "$(gp zip_store)"); [ "${IOS_PULL:-$PULL}" = 1 ] || PACK_OPTS+=(--skip-suffix _bp.metallib)
+python3 tools/pack_ipa.py "$OUT_IPA" "${PACK_OPTS[@]}" --skip $P/_CodeSignature --skip $P/embedded.mobileprovision --skip $P/msl --skip $P/aoc --skip $P/romfs "${MAPS[@]}" || die "pack_ipa"
 t "done: $OUT_IPA ($(du -h "$OUT_IPA" | cut -f1))"

dm/ios/tools/pack_ipa.py

@@ -6,7 +6,7 @@
 # bigger files are streamed here, their CRC patched into the local header afterwards (no data descriptors). Zip64 fields wherever a size, an offset or the entry count passes its 32 / 16-bit field.
 # Usage: pack_ipa.py <out.ipa> [--store .a:.b] [--skip <archive path>]... <archive path>=<source folder>...
 #   --skip: an archive path left out of the walk of the folder that holds it (folders a stale app build still has, the signature); a mapping of its own still adds it
-#   --skip-suffix: files whose names end so are left out of every walk (build_ipa.sh: the pulling vertex libraries _bp.metallib, which the iOS app does not use by default)
+#   --skip-suffix: files whose names end so are left out of every walk (build_ipa.sh: the pulling vertex libraries _bp.metallib when the profile's pso_mtl4 is off, unless IOS_PULL overrides it)
 #   Writes <out.ipa>.partial and renames it at the end; prints "pack: N entries, X bytes" first (the NSP script's progress total, X = the files' bytes) and a summary line last
 import argparse, collections, concurrent.futures as cf, os, stat, struct, sys, time, zlib
 

docs/agents/symptoms.md

@@ -310,6 +310,7 @@ One line per symptom in the [subsystem references](../reference/): what you see
 - a character drawn as a dithered silhouette inside a dithered square (DRAGON QUEST VII Reimagined's hero). → [Stencil-mode textures from the texture builder](../reference/rendering.md#stencil-mode-textures-from-the-texture-builder)
 - characters (skinned models) invisible or stretched into streaks while the level renders; `vertex attribute format … does not match shader input type …` for a remaining mismatch. → [Integer vertex inputs](../reference/rendering.md#integer-vertex-inputs)
 - red lines 89 and 96 (`vconv_bind`, the CPU conversion every draw); about a quarter of Age of Calamity's replay thread in battle. → [Shadow vertex inputs](../reference/rendering.md#shadow-vertex-inputs)
+- an unsigned iOS build reports `[RED] pulling library ... not usable (no library)` and its draws wait for pipelines even with the profile's `pso_mtl4 on`, or packaging with `IOS_PULL=1` fails with `SKIPS[@]: unbound variable`. → [Pulling vertex inputs](../reference/rendering.md#pulling-vertex-inputs)
 - `Vertex attribute m_36(5) of type uint4 cannot be read using MTLAttributeFormatHalf2` (draws dropped: black silhouettes); UVs collapse onto one texel (flat tan characters); vertices far out of view (negative 16-bit values widened with zeros). → [CPU conversion of vertex attributes the input type cannot take](../reference/rendering.md#cpu-conversion-of-vertex-attributes-the-input-type-cannot-take)
 - `vertex attribute format … not mapped` (printed once). → [Vertex formats Metal lacks](../reference/rendering.md#vertex-formats-metal-lacks)
 - the assertion `buffer index (31) must be < 31`. → [Vertex streams 14 and 15 at Metal slots 14 and 15](../reference/rendering.md#vertex-streams-14-and-15-at-metal-slots-14-and-15)

docs/reference/rendering.md

@@ -351,8 +351,11 @@ The replay on Metal (`dm/gfx/nvn/nvn_mtl.m`) and the Vulkan and OpenGL layers. E
 - **Cause:** a pipeline's vertex descriptor fixes the vertex layout it is drawn with, so a shader pair drawn with a layout it has no pipeline for waits until one is compiled.
 - **Fix:** pulling inputs (`<key>_bp`, `dm/gfx/tools/dmsh_pull.py`, decoder `dm/gfx/tools/dmsh_pull.metal`): a side library of every vertex shader with stage inputs, whose `main0` / `main0_z` have no `[[stage_in]]`. They read their inputs themselves from a per-draw layout block at buffer 16 (`dm_vlayout`): per stream the base address, stride and divisor, and per NVN attribute the raw attribute state word. The base instance comes from `[[base_instance]]`, so indirect draws are right too. `dm_fetch` decodes the NVN format at run time into the register words Maxwell's fetch gives the shader, and the input takes them with `as_type`. A pipeline made with it has no vertex descriptor, so it serves every vertex layout a shader pair is drawn with; it is meant for drawing while the pipeline for the draw's own layout is still being built.
 - **Note:** the words are bit-identical to Metal's vertex-descriptor fetch for every format `vfmt` knows, and to the host's CPU conversion for the integer-read-as-float formats. unorm / snorm go through the `unpack_*` builtins, because a plain division under fast math is off by one unit in the last place for some values; NaNs of R11G11B10F come out canonical, as from Metal's fetch. With a divisor the element is `base instance + (instance - base instance) / divisor`, Metal's own rule. Positions match the vertex-descriptor pipeline's (`[[invariant]]`, preserved invariance).
-- **Note:** the run-time format decode costs GPU time: on an M2 Max, 8 attributes in a vertex-bound draw of tiny visible triangles took 65 % longer (about 1.0 ms per million vertices) than with the vertex descriptor. With the formats as compile-time constants it was 9 %, and with the whole layout compiled in, 0. So these pipelines only stand in until the fast one is built. The iOS app does not use them yet (the Metal 4 path is off on iOS by default), so `build_ipa.sh` leaves the `_bp` libraries out of the IPA unless `IOS_PULL=1` (`pack_ipa.py --skip-suffix`). Other entry points (transform feedback, geometry capture, layer) get no variant; shaders without stage inputs need none, since their pipelines have no vertex descriptor.
-- **Check:** `dm/gfx/tests/t_pull.m` (the words against Metal's fetch and the CPU conversion), `t_pull_inv.m` (a depth prepass with one pipeline and depth-equal with the other), `test_dmsh_pull.py`.
+- **Note:** the run-time format decode costs GPU time: on an M2 Max, 8 attributes in a vertex-bound draw of tiny visible triangles took 65 % longer (about 1.0 ms per million vertices) than with the vertex descriptor. With the formats as compile-time constants it was 9 %, and with the whole layout compiled in, 0. So these pipelines only stand in until the fast one is built. Other entry points (transform feedback, geometry capture, layer) get no variant; shaders without stage inputs need none, since their pipelines have no vertex descriptor.
+- **Symptom:** an unsigned iOS build reports `[RED] pulling library ... not usable (no library)` and its draws wait for pipelines even with the profile's `pso_mtl4 on`, or packaging with `IOS_PULL=1` fails with `SKIPS[@]: unbound variable`.
+- **Cause:** the IPA packer used to strip `_bp.metallib` unless `IOS_PULL=1`, independently of the runtime's profile default. The inclusion path expanded an empty array under `set -u`, which macOS Bash 3.2 treats as unset.
+- **Fix:** `build_ipa.sh` takes the default library inclusion from the generated `DM_GAME_PSO_MTL4` compiled into the host; `IOS_PULL=0` / `1` still overrides it. Packing options always contain `--store`, so both inclusion and exclusion work under Bash 3.2. A runtime `DM_PSO_MTL4=1` override for a profile with the path off needs an IPA built with `IOS_PULL=1` too.
+- **Check:** `dm/gfx/tests/t_pull.m` (the words against Metal's fetch and the CPU conversion), `t_pull_inv.m` (a depth prepass with one pipeline and depth-equal with the other), `test_dmsh_pull.py`; `dm/ios/tests/test_profile_pack.py` packs tiny fixtures through the actual archive step and checks profile defaults and explicit overrides in the ZIP.
 - **Found on:** [The Legend of Zelda: Breath of the Wild](../../dm/games/01007EF00011E000/README.md): draws that waited for their pipeline on a cold start.
 
 ### CPU conversion of vertex attributes the input type cannot take