Pad vertex outputs for every pipeline prebuild pair

Read both shader pair tables, including the generated BAHS fallback, for
padding, build order and shader cache dependencies. Include the optional
profile table in the input stamp and the generated table in step reuse;
regenerate a missing fallback before reusing the shader split.

Add native Metal and cache/order regressions, update the shared chain,
and document full validation on TotK and Odyssey. All 301 affected pairs
compile for g13s and g16s, both full Mac shader/host builds pass, all
57,039 pair interfaces match, and TotK's 24,153-base archive succeeds.
The separate Odyssey Max Unstruct ID compiler limit remains open.

Fixes #283
Refs #276, #298

Co-authored-by: marco <6410852+marcolaux@users.noreply.github.com>

dm/gfx/README.md

@@ -53,7 +53,7 @@ The whole chain is driven by `tools/dmsh_build.py`; `scripts/build_ipa_from_nsp.
   - `tools/own_compilers.py <dmsh_build.py arguments>` runs `dmsh_build.py` with the local Metal compiler and spirv-cross even when they differ from the lock, printing one `note:` line that names them; the translator must still match. Neither the lock nor V changes, so the committed `dmsh_chain.h` stays valid. The NSP script builds the iOS shaders this way, so a player with another Xcode or a newer spirv-cross needs no `--update-lock`.
 - **`dm/games/<id>/c1_full.tsv`**: the index list for the shaders' built-in constant section (c1), generated by `tools/c1_guard.py` scanning the translated MSL; it lists the shader keys whose "run-time index could not be proven bounded". An index counts as proven only when interval arithmetic over the whole expression (every variable followed to its single definition; literals, `& K`, `% K`, shifts, `min`, `max`, `clamp` and casts bound it) keeps it non-negative and inside the span a shared bank gives that shader: its c1 data rounded up to 256 bytes plus the 256-byte zero gap. Every stage counts (`vp_`, `fp_`, `cp_` …). Its first line records the `b-<V>` and the inputs stamp of the library it was made from. For the shaders on the list the replay layer allocates a separate whole 64 KiB buffer for c1; when the list is missing or does not name the loaded library's version and stamp, it reports red and conservatively treats all shaders this way. The list is not committed (git ignores it): each checkout makes it from its own build's Mac outputs (a full `dmsh_build.py` run does).
 - **`dm/games/<id>/cb_extent.tsv`**: how far into each uniform slot (cbuf binding) every shader can read, generated by `tools/cb_extent.py` from the translated MSL with `c1_guard.py`'s interval arithmetic: per shader key, `reg:bytes` for each slot whose reads are all proven to stay inside its first bytes, and `s<reg>` for each storage slot the shader never writes (SPIRV-Cross types every storage buffer Ryujinx emits as writable, so the host counts the others as written). When the game updates part of a bound uniform block inside a render pass, the replay layer copies only that much of the binding to give the draws after the update their data without ending the pass (`nvn_mtl.m` `ubp_get`; Breath of the Wild binds a 64 KiB block and its vertex shaders read at most 544 bytes of it). Its first line records the `b-<V>` and inputs stamp, like `c1_full.tsv`'s; when it is missing or stale, bindings are copied whole (up to 64 KiB). Like `c1_full.tsv`, it is not committed, and a full `dmsh_build.py` run makes it.
-- **`dm/games/<id>/shader_pairs.tsv`**: the (vertex key, pixel key) of every BNSH variation (Unity: of every program), written by `tools/shader_split.py`. Metal only builds a pipeline when every pixel shader input is a vertex shader output, while Maxwell gives an unwritten attribute a default; `dmsh_build.py` therefore adds the outputs a vertex shader's partners read but it doesn't write, set to (0, 0, 0, 1) (`vs_pad`). The table is part of the translation chain version.
+- **`dm/games/<id>/shader_pairs.tsv` and `pso_pairs.tsv`**: the (vertex key, pixel key) of every BNSH variation (Unity: of every program) and agl BAHS program variation, written by `tools/shader_split.py`. Metal only builds a pipeline when every pixel shader input is a vertex shader output, while Maxwell gives an unwritten attribute a default; `dmsh_build.py` therefore adds the outputs a vertex shader's partners read but it doesn't write, set to (0, 0, 0, 1) (`vs_pad`). The profile tables are part of its inputs digest and the per-shader cache keys. A profile without `pso_pairs.tsv` takes the generated table beside the shader package.
 - **Several game versions**: `shader_split.py` takes comma-separated RomFS and `main` lists (base game first) and writes the union; the map, facts and pairs tables currently cover Mario v0 and 1.2.1.
 - **`dm/games/<id>/pso_manifest.tsv`**: the warm-up seed manifest bundled with the app: shaders (`S`), render pipelines (`R`, the hex of `pso_skey` in `nvn/nvn_pso_skey.h`) and compute pipelines (`C`) to pre-build at startup. The row order is the pre-build order (first use); do not sort it. Its companion is `pso_user.tsv`, which the host appends next to the save directory whenever it has to build something on the spot, and which is pre-built after the seed at the next launch. Both name shaders by key and describe pipeline settings only, so a list recorded on one device works on any other; lines a build cannot use are skipped. Recorded lines can be appended to the seed below its header (currently empty).
 - **Shaders embedded in the program itself**: the main program additionally contains 4 compute shaders that are not in the shader package; they were hand-written as MSL from the disassembly and are in `nvn/nvn_builtin.h`.

dm/gfx/nvn/dmsh_chain.h

@@ -1,3 +1,3 @@
 // Generated file (dm/gfx/tools/dmsh_chain.py --header): the B shader library translation chain version this source tree expects (default options), one for every game;
 // the host loads only <library dir>/b-<V>/. Run dmsh_chain.py --header after changing the chain's tools and commit this file with the change
-#define DMSH_CHAIN "5cddc47253ab"
+#define DMSH_CHAIN "7776d7a9719d"

dm/gfx/nvn/nvn_mtl.m

@@ -5376,7 +5376,7 @@ static int pso_bundle_covers(NSArray<NSArray<NSString *> *> *pairs, NSString *wa
 static int pso_prebuild(int threads, int wait, dispatch_qos_class_t qos) {
     shmap_load(); pso4_init();
     if (!g_comp4 || (!g_arch_bundle && (!g_ser4 || !g_arch_dir)) || atomic_exchange(&g_pre_running, 1)) return -1;
-    // The pairs: shader_pairs.tsv (BNSH variations and the other readers' pairs) and pso_pairs.tsv (agl BAHS program variations, which vs_pad does not read; DM_PSO_PAIRS: another path),
+    // The pairs: shader_pairs.tsv (BNSH variations and the other readers' pairs) and pso_pairs.tsv (agl BAHS program variations, also read by vs_pad; DM_PSO_PAIRS: another path),
     // both beside the shader map
     NSString *pp = [g_shmap_path.stringByDeletingLastPathComponent stringByAppendingPathComponent:@"shader_pairs.tsv"];
     const char *ev = getenv("DM_PSO_PAIRS");

dm/gfx/tests/test_dmsh_pairs.py

@@ -0,0 +1,149 @@
+"""Both shader-pair tables must feed vertex padding, its cache dependencies and every vertex variant (#283, #276)."""
+import ast
+import collections
+import hashlib
+import json
+import os
+from pathlib import Path
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import types
+import unittest
+
+TOOLS = Path(__file__).resolve().parents[1] / 'tools'
+sys.path.insert(0, str(TOOLS))
+import dmsh_chain
+import dmsh_pull
+import pso_offline
+
+
+def builder():
+    # Use production functions without the CLI's translator/profile initialization.
+    names = {'load_pairs', 'vs_pad', 'vs_depth', 'shader_key', 'pair_keyed'}
+    tree = ast.parse((TOOLS / 'dmsh_build.py').read_text())
+    ns = dict(collections=collections, os=os, re=re, sys=sys, hashlib=hashlib)
+    exec(compile(ast.Module(body=[n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name in names],
+                            type_ignores=[]), str(TOOLS / 'dmsh_build.py'), 'exec'), ns)
+    return ns
+
+
+VS = '''#include <metal_stdlib>
+using namespace metal;
+struct main0_out
+{
+    float4 gl_Position [[position]];
+    float4 v0 [[user(locn0)]];
+};
+struct main0_in
+{
+    float4 m_0 [[attribute(0)]];
+};
+vertex main0_out main0(main0_in in [[stage_in]], uint vid [[vertex_id]])
+{
+    main0_out out = {};
+    out.gl_Position = float4(float2((vid << 1) & 2, vid & 2) * 2.0 - 1.0, 0.0, 1.0);
+    out.v0 = in.m_0;
+    return out;
+}
+'''
+
+
+def pixel(loc):
+    return f'''#include <metal_stdlib>
+using namespace metal;
+struct main0_in
+{{
+    float4 v0 [[user(locn0)]];
+    float4 missing [[user(locn{loc})]];
+}};
+fragment float4 main0(main0_in in [[stage_in]]) {{ return in.v0 + in.missing; }}
+'''
+
+
+class Pairing(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory(); self.addCleanup(self.tmp.cleanup)
+        self.root = Path(self.tmp.name)
+        self.profile = self.root / 'profile'; self.profile.mkdir()
+        self.package = self.root / 'shader/pkg'; self.package.mkdir(parents=True)
+        self.g = types.SimpleNamespace(table=lambda n: self.profile / n, rows=[])
+        self.g.table('shader_pairs.tsv').write_text('# BNSH\nv\tf0\n')
+        self.extra = self.package.parent / 'pso_pairs.tsv'
+        self.extra.write_text('# BAHS\nv\tf1\nv\tf6\nv\tf1\n\n')
+        self.ns = builder()
+        self.ns.update(out=str(self.root), PAIRS=self.ns['load_pairs'](self.g, self.package))
+        for key, loc in [('f0', 0), ('f1', 1), ('f6', 6)]:
+            src = pixel(loc)
+            if not loc: src = src.replace('    float4 missing [[user(locn0)]];\n', '').replace('in.v0 + in.missing', 'in.v0')
+            (self.root / f'{key}_b.metal').write_text(src)
+
+    def test_split_fallback_union_and_defaults(self):
+        self.assertEqual(self.ns['PAIRS'], {'v': {'f0', 'f1', 'f6'}})
+        src, pad = self.ns['vs_pad']('v', VS)
+        self.assertEqual(pad, [1, 6])
+        for loc in pad:
+            self.assertIn(f'float4 dm_pad{loc} [[user(locn{loc})]];', src)
+            self.assertIn(f'out.dm_pad{loc} = float4(0.0, 0.0, 0.0, 1.0);', src)
+        self.assertEqual(src.count('float4 v0 [[user(locn0)]];'), 1)
+        self.assertEqual(self.ns['vs_pad']('v', src), (src, []))
+
+    def test_profile_table_wins_and_optional_table(self):
+        self.g.table('pso_pairs.tsv').write_text('v\tf6\n')
+        self.assertEqual(self.ns['load_pairs'](self.g, self.package), {'v': {'f0', 'f6'}})
+        self.g.table('pso_pairs.tsv').unlink(); self.extra.unlink()
+        self.assertEqual(self.ns['load_pairs'](self.g, self.package), {'v': {'f0'}})
+        self.assertEqual(self.ns['vs_pad']('unpaired', VS), (VS, []))
+
+    def test_profile_input_stamp_changes(self):
+        missing = dmsh_chain.inputs(self.g)
+        self.g.table('pso_pairs.tsv').write_text('v\tf6\n')
+        present = dmsh_chain.inputs(self.g)
+        self.g.table('pso_pairs.tsv').write_text('v\tf1\n')
+        self.assertNotEqual(missing, present)
+        self.assertNotEqual(present, dmsh_chain.inputs(self.g))
+
+    def test_vertex_cache_depends_on_extra_pairs_and_pixel_code(self):
+        self.ns.update(FACTS={}, MAP_ROWS={}, MAP_CONFLICT_ROWS={}, TOOL_FP=b'tools', CODE_FP='code',
+                       PROFILE_ROWS=b'', shader_rule_source=lambda name: b'')
+        key = self.ns['shader_key']
+        v1 = key('v', 'pkg.bin', b'vertex')
+        self.ns['PAIRS']['v'].remove('f6')
+        self.assertNotEqual(v1, key('v', 'pkg.bin', b'vertex'))
+        f1, f2 = key('f1', 'pkg.bin', b'pixel'), key('f1', 'pkg.bin', b'changed pixel')
+        self.assertNotEqual(self.ns['pair_keyed'](v1, [f1]), self.ns['pair_keyed'](v1, [f2]))
+
+    @unittest.skipUnless(shutil.which('xcrun'), 'needs the Metal compiler')
+    def test_metal_archive_with_padded_regular_and_pulling_entries(self):
+        probe = subprocess.run(['xcrun', '-sdk', 'macosx', 'metal-tt', '-version'], capture_output=True)
+        if probe.returncode: self.skipTest('no Metal toolchain')
+        src, _ = self.ns['vs_pad']('v', VS)
+        _, full = self.ns['vs_depth'](src)
+        pull, meta = dmsh_pull.variant(full)
+        self.assertIsNotNone(pull, meta)
+        for name, text in [('v_b', full), ('v_bp', pull), ('f1_b', pixel(1)), ('f6_b', pixel(6))]:
+            msl, air, lib = [self.root / (name + ext) for ext in ('.metal', '.air', '.metallib')]
+            msl.write_text(text)
+            for sdk in ('iphoneos', 'macosx'):
+                r = subprocess.run(['xcrun', '-sdk', sdk, 'metal', '-c', '-fpreserve-invariance', '-fmodules-cache-path=' + str(self.root / 'modules'), msl, '-o', air], capture_output=True, text=True)
+                self.assertEqual(r.returncode, 0, r.stderr)
+            # The archive below consumes the Mac libraries.
+            subprocess.run(['xcrun', '-sdk', 'macosx', 'metallib', air, '-o', lib], check=True, capture_output=True)
+        bases = [(v, entry, f, 'main0') for v in ('v_b', 'v_bp') for entry in ('main0', 'main0_z') for f in ('f1_b', 'f6_b')]
+        plan = pso_offline.script(self.root, bases)
+        for base, desc in zip(bases, plan['pipeline_descriptors']['render_pipeline_descriptors']):
+            if base[0] == 'v_b':
+                desc['vertex_descriptor'] = {'attributes': [{'format': 'Float4', 'offset': 0, 'buffer_index': 0}],
+                                             'layouts': [{'stride': 16, 'step_function': 'PerVertex', 'step_rate': 1}]}
+        script = self.root / 'bases.mtl4-json'; script.write_text(json.dumps(plan))
+        archive = self.root / 'pairs.mtl4a'
+        osmin = pso_offline.os_min_default('macos')
+        r = subprocess.run(['xcrun', '-sdk', 'macosx', 'metal-tt', '-arch', 'applegpu_g13s', '-macos-version-min', osmin, script, '-o', archive],
+                           env=dict(os.environ, TMPDIR=str(self.root)), capture_output=True, text=True)
+        self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
+        self.assertTrue(archive.is_file())
+
+
+if __name__ == '__main__': unittest.main()

dm/gfx/tools/dmsh_build.py

@@ -595,10 +595,20 @@ def int_inputs(src):
         if not bad and nint: out[str(loc_)] = 'uint'
     return out
 
-# vertex / pixel pairs (taken from BNSH variations by shader_split.py): vertex key → paired pixel keys
-PAIRS = collections.defaultdict(set)
-for _ln in open(GAME.table('shader_pairs.tsv')):
-    if _ln.strip() and not _ln.startswith('#'): _v, _p = _ln.split(); PAIRS[_v].add(_p)
+# Both pair tables feed pipelines, so both must feed padding, build order and shader cache keys.
+def load_pairs(game, package):
+    pairs = collections.defaultdict(set)
+    for table in ('shader_pairs.tsv', 'pso_pairs.tsv'):
+        path = str(game.table(table))
+        if table == 'pso_pairs.tsv' and not os.path.isfile(path):
+            path = os.path.join(os.path.dirname(os.path.abspath(package)), table)   # the split's copy for profiles predating this table
+        if table == 'pso_pairs.tsv' and not os.path.isfile(path): continue
+        with open(path) as f:
+            for ln in f:
+                if ln.strip() and not ln.startswith('#'): v, p = ln.split(); pairs[v].add(p)
+    return pairs
+
+PAIRS = load_pairs(GAME, dm_dir)
 
 def vs_pad(key, src):
     """Give a vertex shader the outputs its paired pixel shaders read but it does not write, with the value (0,0,0,1).
@@ -608,7 +618,8 @@ def vs_pad(key, src):
     want = set()
     for ps in PAIRS.get(key, ()):
         fn = os.path.join(out, ps + '_b.metal')
-        if os.path.exists(fn): want |= {int(x) for x in re.findall(r'\[\[user\(locn(\d+)\)', open(fn).read())}
+        if os.path.exists(fn):
+            with open(fn) as f: want |= {int(x) for x in re.findall(r'\[\[user\(locn(\d+)\)', f.read())}
     miss = sorted(want - {int(x) for x in re.findall(r'\[\[user\(locn(\d+)\)', src)})
     i = src.find('struct main0_out\n{\n'); j = src.find('\n};', i); k = src.find('main0_out out = {};')
     if not miss or i < 0 or j < 0 or k < 0: return src, []

dm/gfx/tools/dmsh_chain.py

@@ -23,7 +23,7 @@ FILES += ['../third_party/ryujinx/src/Ryujinx.Graphics.Shader/' + f for f in (
     'Translation/TranslationFlags.cs', 'Translation/HostCapabilities.cs',
     'Translation/TranslatorContext.cs', 'Translation/EmitterContext.cs', 'Translation/ShaderDefinitions.cs', 'CodeGen/Spirv/Instructions.cs',
     'StructuredIr/ShaderProperties.cs', 'Translation/Optimizations/BindlessElimination.cs')]
-INPUT_TABLES = ['shader_facts.tsv', 'shader_map.tsv', 'shader_map.tsv.conflicts.tsv', 'shader_pairs.tsv', 'bf_want.tsv']   # in the game profile folder
+INPUT_TABLES = ['shader_facts.tsv', 'shader_map.tsv', 'shader_map.tsv.conflicts.tsv', 'shader_pairs.tsv', 'pso_pairs.tsv', 'bf_want.tsv']   # in the game profile folder
 LOCK = os.path.join(HERE, 'dmsh_chain.lock'); HEADER = os.path.join(GFX, 'nvn/dmsh_chain.h')
 STAMP = 'inputs.tsv'   # in the library directory b-<V>/
 # the translator defaults to the in-repo build that matches the lock file (a rebuilt dll has different fingerprints)

dm/gfx/tools/shader_split.py

@@ -1800,8 +1800,8 @@ def ue_shaders(d, label):
     return out
 
 def var_pairs(found):
-    """(vertex key, pixel key) within the same BNSH variation; (vertex key, pixel key, 'prog') within the same BAHS program variation (bahs_variations), which go to the pipeline
-    pair table only (main: shader_pairs.tsv, which vs_pad reads, keeps the BNSH pairs)"""
+    """(vertex key, pixel key) within the same BNSH variation; (vertex key, pixel key, 'prog') within the same BAHS program variation (bahs_variations).
+    The tables stay separate; dmsh_build.py pads against both, as the pipeline prebuild uses both."""
     var = collections.defaultdict(dict)
     for where, st, ctl, code in found:
         g = where if '#var' in where else BAHS_VAR.get(where)
@@ -2021,7 +2021,7 @@ def main():
             for a, b in sorted(pairs): f.write(f'{a}\t{b}\n')
     if psopairsout:
         with open(psopairsout, 'w') as f:
-            f.write('# kind: evidence  vertex / pixel shader pairs of agl program variations (BAHS) that shader_pairs.tsv does not hold, for the pipeline prebuild only (generated by dm/gfx/tools/shader_split.py, do not edit): vertex key <TAB> pixel key\n')
+            f.write('# kind: evidence  vertex / pixel shader pairs of agl program variations (BAHS) that shader_pairs.tsv does not hold, for vertex output padding and the pipeline prebuild (generated by dm/gfx/tools/shader_split.py, do not edit): vertex key <TAB> pixel key\n')
             for a, b in sorted(prog - pairs): f.write(f'{a}\t{b}\n')
     print(f'shaders: {len(rows)} (' + ', '.join(f'{STAGE_NAME[s]} {n}' for s, n in sorted(stat.items())) + '); sources ' + ', '.join(f'{g} {n}' for g, n in per_src.most_common()) + f'; compute shader answers {len(facts)} rows' + (f'; variation pairs {len(pairs)}' if pairsout else ''))