shader_split.py reads GLSLC outputs of several stages: Baba Is You (01002CD00A51C000, no profile) keeps its shaders as 120 GLSLC outputs in RomFS/Assets.dat, each a pixel and a vertex shader compiled together, then a reflection section, and glslc_shaders took only outputs of one control section, so the NSP script's split found 0 shaders and the pack stopped one step later with "no iOS shader libraries". An output with more than one control section is now read by its section headers (glslc_section returns every GPU code section, glslc_program checks each stage and needs distinct stages); its stages are labelled #GLSLC@<offset>#var, so var_pairs pairs them with each other and separable_pairs leaves them out of the single-stage cover. The NSP script stops at the split when it finds no shader, with the reason, instead of at the iOS shader step. Checked: Baba Is You splits into 131 shaders (115 pixel, 16 vertex) and 115 pairs, and its pack built BabaIsYou.ipa from the provisional profile with 131 of 131 shaders ready (not run on an iPhone); the split's tables for 13 Sentinels, AI: THE SOMNIUM FILES, Raging Loop, Metroid Dread and Age of Calamity are byte-identical to their profiles', and no main on the game drive holds an output of several stages. docs/agents/04-graphics.md says so

dm/gfx/tools/shader_split.py

@@ -146,9 +146,9 @@ def glslc_stage(ctl, code):
     return 'cs' if code[:4] == CS_MAGIC or headerless_cs else sph_stage(code) if code[:4] == GFX_MAGIC and len(code) >= 0x34 else None
 
 def glslc_section(d, h, size):
-    """The GPU code section of the GLSLC output at h, from its section headers: output +0x4c = section count, section i's header at +0x90 + 0x90 i = {u32 size, u32 data offset (from the
+    """The GPU code sections of the GLSLC output at h, from its section headers: output +0x4c = section count, section i's header at +0x90 + 0x90 i = {u32 size, u32 data offset (from the
     output), u32 type (0 = GPU code)}; a GPU code header adds +0x30 control offset, +0x34 code offset (both from the section's data), +0x38 code length, +0x3c control length.
-    → (control, code) of its only GPU code section, or None"""
+    → [(control, code)] of each GPU code section, or None"""
     n, secs = struct.unpack_from('<I', d, h + 0x4c)[0], []
     if not 0 < n <= 16 or h + 0x90 * (n + 1) > len(d): return None
     for i in range(n):
@@ -158,10 +158,19 @@ def glslc_section(d, h, size):
         co, ko, kn, cn = struct.unpack_from('<4I', d, s + 0x30)
         if soff + ssz > size or co + cn > ssz or ko + kn > ssz: return None
         secs.append((d[h + soff + co:h + soff + co + cn], d[h + soff + ko:h + soff + ko + kn]))
-    return secs[0] if len(secs) == 1 else None
+    return secs
+
+def glslc_program(d, h, size, label):
+    """A GLSLC output of several stages, compiled together as one program (Baba Is You: each of the 120 outputs in Assets.dat holds a pixel and a vertex shader, then a reflection section): every
+    GPU code section by the section headers. Its stages pair with each other (var_pairs), not as single-stage programs (separable_pairs). [] when a section fails the triple check or two share a stage"""
+    secs = glslc_section(d, h, size) or []
+    sts = [glslc_stage(ctl, code) for ctl, code in secs]
+    if len(secs) < 2 or len(set(sts)) < len(sts) or any(not st or check(label, st, ctl, code) for st, (ctl, code) in zip(sts, secs)): return []
+    return [(f'{label}#GLSLC@{h:#x}#var', st, ctl, code) for st, (ctl, code) in zip(sts, secs)]
 
 def glslc_shaders(d, label, need=True):
-    """GLSLC compiler outputs (838 in 13 Sentinels' main read-only segment, one stage each): from the magic 0x19866891, +0x44 = the whole output's length; each output holds exactly one control section with the code area right after it (control +0x20 = control length).
+    """GLSLC compiler outputs (838 in 13 Sentinels' main read-only segment, one stage each): from the magic 0x19866891, +0x44 = the whole output's length; an output of one stage holds exactly one control section with the code area right after it (control +0x20 = control length),
+    one of several stages is read by its section headers (glslc_program).
     The magic is only 4 bytes, so only outputs that pass the triple check are kept. Every game's main is searched: Unity's player has 11 of its own (Fire Emblem Engage, Ace Attorney Investigations
     Collection) beside the game's shaders in its bundles, KTGL and Vanillaware's engine compile theirs into main (13 Sentinels: 838), the others have none.
     need: stop when there is none (a capture file); a Koei Tecmo G2S container (idrk_shaders) or a main may hold none.
@@ -172,7 +181,9 @@ def glslc_shaders(d, label, need=True):
     for m in re.finditer(b'\x91\x68\x86\x19', d):
         h = m.start(); size = struct.unpack_from('<I', d, h + 0x44)[0]
         c = d.find(CTL_MAGIC, h, h + size) if 0x760 < size <= len(d) - h else -1
-        if c < 0 or d.find(CTL_MAGIC, c + 4, h + size) >= 0: continue
+        if c < 0: continue
+        if d.find(CTL_MAGIC, c + 4, h + size) >= 0:
+            out += glslc_program(d, h, size, label); continue
         nctl, ncode = struct.unpack_from('<I', d, c + 0x20)[0], struct.unpack_from('<I', d, c + 0x704)[0]
         ctl, code = d[c:c + nctl], d[c + nctl:c + nctl + ncode]
         st = glslc_stage(ctl, code) if nctl >= 0x760 and not nctl & 3 and c + nctl + ncode <= h + size else None
@@ -181,8 +192,8 @@ def glslc_shaders(d, label, need=True):
             # 12 bytes short of their length, so the code area is not right after it; the output's own section header says where both are (glslc_section). Only outputs the rule
             # above rejects are read this way: on every output it keeps (1,073 in 13 Sentinels, Three Houses, FINAL FANTASY TACTICS, Engage and Age of Calamity) both give the same key
             sec = glslc_section(d, h, size)
-            if not sec: continue
-            ctl, code = sec; st = glslc_stage(ctl, code)
+            if not sec or len(sec) != 1: continue
+            (ctl, code), = sec; st = glslc_stage(ctl, code)
             if not st or check(label, st, ctl, code): continue
         out.append((f'{label}#GLSLC@{h:#x}', st, ctl, code))
     if not out and need: red(f'{label}: no GLSLC output')
@@ -926,7 +937,7 @@ def separable_pairs(keys):
     from the pipeline state) have their real pairs only at draw time. vs_pad uses only the union of the locations the paired pixel shaders read, so every single-stage vertex shader is paired with
     a "cover": the fewest pixel shaders whose read locations together equal those of all pixel shaders (greedy, ties go to the smaller key); padded that way, a vertex shader links with any pixel shader.
     All pairs would be 376 × 375 rows; the cover needs a few rows × 376. No single-stage vertex shader, no cover"""
-    single = [v for v in keys if keys[v][1] == 'vs' and any(s in keys[v][0] for s in SINGLE_STAGE)]
+    single = [v for v in keys if keys[v][1] == 'vs' and any(s in keys[v][0] for s in SINGLE_STAGE) and '#var' not in keys[v][0]]   # '#var': a GLSLC output of several stages (glslc_program)
     if not single: return set(), []
     ins = {k: ps_inputs(v[3]) for k, v in keys.items() if v[1] == 'ps'}
     left, cover = set().union(*ins.values()) if ins else set(), []

docs/agents/04-graphics.md

@@ -15,7 +15,7 @@ Do not stop at the obvious archive. In Mario the main `bfsha` held 4,888 of the
 - **Unity ComputeShader objects** (Monster Train 2, Unity 2022.3.54f1) → the 3D battle scene was black behind a correct UI, with `[RED] shader 854a8afc9d07f88d is not in the shader map` and `the dispatched program has no compute shader`. `DM_NVN_SHDUMP` saved the program; its code bytes were in `Data/sharedassets2.assets`, uncompressed, inside a `ComputeShader` (class 72): the post-processing stack's colour grading LUT baker (`Lut3DBaker#KGenLut3D_AcesTonemap`), so the LUT the scene is graded through stayed black. The split read only `Shader` objects. A `ComputeShader` holds per platform variant a list of kernels, each with its unique variants; a variant's `code` is the program code of a `Shader` subprogram (the 0x60-byte stage header, only the compute block present) and its `threadGroupSize` matches control +0x748. `unity_compute` in `shader_split.py` reads them beside the `Shader` objects of every serialized file; the game has no bundle with a class 72 tree, so `unityfs.py` carries Unity 2022.3.54f1's layout (`COMPUTE_2022_3`, written from AssetRipper's TypeTreeDumps for that version; all 12 objects decode, `resourcesResolved` true at each end). 85 compute shaders came in (86 with the player's own). A file whose `ComputeShader` has no tree still goes to the second pass; if it has none there either, the split prints a `[RED]` line and leaves its kernels out instead of stopping, so profiles whose compute shaders were never read keep their tables: Cuphead (2) and Fire Emblem Engage (4) have `ComputeShader` objects only in files without type trees, of hashes no layout covers. So does Raging Loop (Unity 2018.3: the two built-ins in `unity default resources`, whose 2018 layout lists kernels with no unique variants), which dispatches no compute.
 - **MSHD** (Metroid Dread, MercurySteam's engine: `system/shd/*.bshdat`, the only place its shaders live; nothing in `main` or the `.pkg` packs): each stage is a control section, whose length is at control +0x20, sometimes followed by a small compiler-option table (`fastmath true`), then the code area at the next code header. Every file holds vertex / pixel stages in alternating order (1, 2 or 4 programs) and is paired two by two. The control section carries NV assembly text after its first 0x880 bytes, and the SASS disassembly follows each code area, so the binding layout can be read straight from the files (`TEX` / `TEXS` handle immediates). 478 files, 1,342 stages, 1,010 unique keys (359 vertex, 651 pixel), 659 pairs.
 - **Unity 6 Shader objects** (Hollow Knight: Silksong, Unity 6000.0.50f1) → read with no change: its 2,068 Addressables bundles carry their type trees (they report the engine version `0.0.0`, Unity's stripped version string), so no layout had to be written out. 2,450 stage shaders: 1,189 vertex, 1,193 pixel, 67 compute and one geometry shader, which does not build; 11 of them are the player's own in `main`.
-- **GLSLC outputs in `main`** (13 Sentinels, Vanillaware's engine: no shader file in RomFS, whose six CRI CPK archives hold textures, scripts, CRI audio and USM movies): 838 GLSLC compiler outputs sit in `main`'s read-only segment, each referenced from a `{pointer, size, 0}` table in `.data`. An output starts with the magic `0x19866891`, its total size is at +0x44, and it holds exactly one control section with the code area right after it (control +0x20 = control length); a debug section with the NV assembly and the GLSL source follows. Engage's `main` also has 11 of them (Unity's own), and `shader_split.py` reads them in every game's `main`, accepting only outputs that pass the control / code / length check. The stage comes from SPH word 0, as for BAHS. Loose RomFS files no other reader takes are searched the same way: FINAL FANTASY TACTICS - The Ivalice Chronicles keeps 257 whole outputs in its technique files (`*.tec`, below), Absolum 163 in `shaders.nx.bin`, Catherine: Full Body one per file in `data/shaders_binary` (3,050, its vertex shaders in two parts, below); in the other profiles' files only Metroid Dread's MSHD files hold any, which `mshd_shaders` reads.
+- **GLSLC outputs in `main`** (13 Sentinels, Vanillaware's engine: no shader file in RomFS, whose six CRI CPK archives hold textures, scripts, CRI audio and USM movies): 838 GLSLC compiler outputs sit in `main`'s read-only segment, each referenced from a `{pointer, size, 0}` table in `.data`. An output starts with the magic `0x19866891`, its total size is at +0x44, and it holds exactly one control section with the code area right after it (control +0x20 = control length); a debug section with the NV assembly and the GLSL source follows. Engage's `main` also has 11 of them (Unity's own), and `shader_split.py` reads them in every game's `main`, accepting only outputs that pass the control / code / length check. The stage comes from SPH word 0, as for BAHS. Loose RomFS files no other reader takes are searched the same way: FINAL FANTASY TACTICS - The Ivalice Chronicles keeps 257 whole outputs in its technique files (`*.tec`, below), Absolum 163 in `shaders.nx.bin`, Catherine: Full Body one per file in `data/shaders_binary` (3,050, its vertex shaders in two parts, below); in the other profiles' files only Metroid Dread's MSHD files hold any, which `mshd_shaders` reads. An output may also hold several stages compiled together as one program (Baba Is You: the 120 outputs in `Assets.dat`, each a pixel and a vertex shader, then a reflection section): such an output is read by its section headers (+0x4c section count, section *i*'s header at +0x90 + 0x90 *i*, type 0 = GPU code; `glslc_program`), and its stages are paired with each other, not as single-stage programs. Before that reader the split found no shader in the game, and the NSP script now stops at the split when it finds none.
 - **G2S in RDB archives** (Hyrule Warriors: Age of Calamity, Koei Tecmo's KTGL): each material's shaders are a G2S container (`_S2G3000`, type KTID `0x7bcd279f`; 23,108 entries in `asset/system.rdb.bin*`, 2.3 GB inflated) of GLSLC outputs, as in 13 Sentinels' `main`, inside zlib-chunked IDRK entries ([01](01-extract.md)); one `_DOK0000` container (type `0x20a6a0bb`, the engine's system resources) holds 151 more bare GLSLC outputs, 21 of them drawn in the first battle. `shader_split.py` maps the RDB files and inflates only entries that start with those two magics (`idrk_shaders`), and the 117 outputs in `main` as in every game. Many outputs carry 0x850-byte control sections and the compute ones have no code header, so `glslc_shaders` accepts controls from 0x760 bytes and takes the stage of headerless code from the thread group size at control +0x748. 40,726 shaders (37,043 pixel, 3,440 vertex, 241 compute, 2 geometry), Mario's texture handle layout (`0x20 + 8 × slot`); pairs come from the single-stage cover (`separable_pairs`). The older KTGL of Three Houses packs the same G2S containers differently (next item).
 - **SPKG in LINKDATA** (Fire Emblem: Three Houses, the older KTGL): shader packages (`SPKG`: `patch*/nx/shader/*.spkg` and 132 entries of `DATA1.bin`) of G2S containers as above, one `_SPK4100` package without G2S (the post effects, `DATA1.bin` entry 549), and map and model archives carrying packages of their own. `kt_shaders` inflates every LINKDATA entry and loose `*.gz` / `*.spkg` file ([01](01-extract.md)), takes every GLSLC output that passes the triple check, and scans its SARCs for BNSH; `DATA1.bin` is split into 64 scan jobs by entry range (`linkdata_jobs`). Its GLSLC (output version 0x1d.2) writes the 0x848 / 0x850-byte controls and headerless compute code described above, which `glslc_shaders` used to drop: 24 shaders before, 1,355 after. A few outputs of version 0x1d.0 keep something else at control +0x20 (odd values such as 0x136b), so a control length must be a multiple of 4. Nine more shaders exist only at run time (Seen in Fire Emblem: Three Houses, below).
 - **SDF in RE Engine PAKs** (Monster Hunter Rise): shader definitions (`SDF\0`, 2,685 files in the PAK view, [01](01-extract.md)): u16 version 1, u16 entry count, u64, then 0x108-byte entries from +0x10. Entry +0x10 holds six u64 offsets (vertex, hull, domain, geometry, pixel, compute; 0 = none), each to `{u32 control length, u32 code length}` followed by the control section (0x880 bytes) and the code area; compute code has no header. An entry's stages are one program, so its vertex / pixel stages are a pair. Empty definitions are 16 bytes of version 0. `kpka_jobs` maps the PAKs, takes the newest copy of each path, peeks at 4 decompressed bytes and inflates only SDF entries (`sdf_shaders`): 63,111 shaders (30,233 vertex, 21,475 pixel, 11,271 compute, 53 geometry, 33 hull, 46 domain) and 102,345 pairs, in 5 s with 12 jobs.

scripts/build_ipa_from_nsp.py

@@ -1155,6 +1155,8 @@ def split_shaders(py, repo, roms, mains, pkg, env, logs, g, provisional=False, o
     tmp = pkg.with_name(pkg.name + '.partial'); rmtree(tmp); (tmp / 'tables').mkdir(parents=True)
     run([py, repo / 'dm/gfx/tools/shader_split.py', ','.join(map(str, roms)), ','.join(map(str, mains)), tmp / 'pkg'] + [tmp / 'tables' / n for n in names], logs / 'shader_split.log', env=env)
     m = re.search(r'shaders: (\d+) \(', (logs / 'shader_split.log').read_text(errors='replace'))
+    if m and m.group(1) == '0':   # dmsh_build.py would make no library, and the iOS shader step stop with no word of why
+        raise BuildError('the shader split found no shaders in main or the RomFS: the game keeps them in a form shader_split.py does not read; locate them (docs/agents) and add a reader')
     if provisional:   # a provisional profile takes the tables as they come
         for n in names: shutil.copyfile(tmp / 'tables' / n, g.table(n))
     else: