Pad vertex outputs for every pipeline prebuild pair
Read both shader pair tables, including the generated BAHS fallback, for padding, build order and shader cache dependencies. Include the optional profile table in the input stamp and the generated table in step reuse; regenerate a missing fallback before reusing the shader split. Add native Metal and cache/order regressions, update the shared chain, and document full validation on TotK and Odyssey. All 301 affected pairs compile for g13s and g16s, both full Mac shader/host builds pass, all 57,039 pair interfaces match, and TotK's 24,153-base archive succeeds. The separate Odyssey Max Unstruct ID compiler limit remains open. Fixes #283 Refs #276, #298 Co-authored-by: marco <6410852+marcolaux@users.noreply.github.com>
This commit is contained in:
1 parent
de13572121
commit
ff44c59e12
16 files changed
+231
-29
No files matched your search
@@ -41,6 +41,7 @@ No `patches.py`. `op_mode` stays docked: Odyssey only takes its motion moves in
|
|||||||
|
|
||||||
## Files and shaders
|
## Files and shaders
|
||||||
|
|
||||||
|
- **Resolved in the shader builder (#276): BAHS pipeline interfaces.** The local 1.4.1 libraries have 290 pairs with missing vertex outputs, involving 29 vertex shaders. All are among the split's 12,124 BAHS pairs; the profile's 20,740 other pairs have no missing outputs. The shared fix makes padding read both tables; see [Pixel shader inputs the vertex shader does not write](../../../docs/reference/shaders.md#pixel-shader-inputs-the-vertex-shader-does-not-write). The report's additional `Max Unstruct ID 758 ... exceeds supported limit` compiler error is separate and remains open; fixing missing outputs alone does not establish a successful full archive build. Targeted validation: the affected vertex sources compile for Mac and iOS, and all 290 pairs build into `g13s` and `g16s` Metal archives with both depth entry points (Metal 32023.921). Full Mac rebuild: 27,984 of 27,984 shaders successful, no binding or pulling-library failures, no regressions, and no interface mismatches across all 32,864 pairs. The Mac host also rebuilt with the regenerated uniform tables. No new gameplay or iPhone run.
|
||||||
- 27,984 shaders: agl `sharcb` and `bfsha` in `ShaderData`, particles in `EffectData`; groups per RomFS folder in `game.tsv`.
|
- 27,984 shaders: agl `sharcb` and `bfsha` in `ShaderData`, particles in `EffectData`; groups per RomFS folder in `game.tsv`.
|
||||||
- The spin-wait rewrite leaves its modules byte for byte as before the lock and store rules.
|
- The spin-wait rewrite leaves its modules byte for byte as before the lock and store rules.
|
||||||
|
|
||||||
@@ -54,6 +55,8 @@ No `patches.py`. `op_mode` stays docked: Odyssey only takes its motion moves in
|
|||||||
|
|
||||||
General host and tool changes first needed here; each applies to every game:
|
General host and tool changes first needed here; each applies to every game:
|
||||||
|
|
||||||
|
- **BAHS pairs omitted from vertex padding:** [Pixel shader inputs the vertex shader does not write](../../../docs/reference/shaders.md#pixel-shader-inputs-the-vertex-shader-does-not-write).
|
||||||
|
|
||||||
- **Resolved: NSZ decompression rejected `--keys`** when packing v0: [NSZ decompression rejects `--keys`](../../../docs/reference/packing.md#nsz-decompression-rejects---keys).
|
- **Resolved: NSZ decompression rejected `--keys`** when packing v0: [NSZ decompression rejects `--keys`](../../../docs/reference/packing.md#nsz-decompression-rejects---keys).
|
||||||
|
|
||||||
- **Largest point size at `c0 + 0x610`**: [04, section 3](../../../docs/agents/04-graphics.md#3-find-the-binding-layout-by-census) and [Graphics `c0 + 0x610`: the largest point size](../../../docs/reference/shaders.md#graphics-c0--0x610-the-largest-point-size).
|
- **Largest point size at `c0 + 0x610`**: [04, section 3](../../../docs/agents/04-graphics.md#3-find-the-binding-layout-by-census) and [Graphics `c0 + 0x610`: the largest point size](../../../docs/reference/shaders.md#graphics-c0--0x610-the-largest-point-size).
|
||||||
|
|||||||
@@ -59,6 +59,7 @@ Game-specific findings. The general host changes they led to are listed under [F
|
|||||||
|
|
||||||
## Files and shaders
|
## Files and shaders
|
||||||
|
|
||||||
|
- **Resolved in the shader builder (#283): BAHS pipeline interfaces.** The local 1.4.3 libraries reproduce all 11 reported pairs: four vertex shaders omit `locn1` or `locn6`, which their pixel partners read. Every failing pair is in the split's 4,159 BAHS pairs; none of the profile's 20,016 other pairs has a missing output. The shared fix makes padding read both tables; see [Pixel shader inputs the vertex shader does not write](../../../docs/reference/shaders.md#pixel-shader-inputs-the-vertex-shader-does-not-write). Targeted validation: the affected vertex sources compile for Mac and iOS, and every affected pair builds into `g13s` and `g16s` Metal archives with both depth entry points. This uses Metal 32023.921; the original TotK sources reproduce the reported rejection on that toolchain too. Full Mac rebuild: 28,669 of 28,669 shaders successful, no binding or pulling-library failures, no regressions, and no interface mismatches across all 24,175 pairs. The Mac host also rebuilt with the regenerated uniform tables. The complete `g16s` archive also succeeds: all 24,153 eligible bases, 1,306 MiB, Metal 32023.921 (2,494 s in the compiler with other builds running). The compiler peaked around 36 GB; a late sample was releasing `AIRNTDispose` / `LLVMContext` contexts. No new gameplay or iPhone run.
|
||||||
- **Current: prewarm capture reported in [PR #261](https://github.com/seedds/nxnative/pull/261).** The seed in `pso_manifest.tsv` contains 1,537 shader, 2,512 render-pipeline and 43 compute-pipeline rows, recorded on an M2 iPad Pro with update 1.4.3 on 2026-10-25 over about 33 minutes: boot, title, a save load and overworld exploration. The author reports no pipeline compilations in the second session's long frames with this list. This covers the recorded route, not the whole game; no new gameplay or cold-install validation here. The capture is separate from the shared deswizzle and symbol-index optimizations.
|
- **Current: prewarm capture reported in [PR #261](https://github.com/seedds/nxnative/pull/261).** The seed in `pso_manifest.tsv` contains 1,537 shader, 2,512 render-pipeline and 43 compute-pipeline rows, recorded on an M2 iPad Pro with update 1.4.3 on 2026-10-25 over about 33 minutes: boot, title, a save load and overworld exploration. The author reports no pipeline compilations in the second session's long frames with this list. This covers the recorded route, not the whole game; no new gameplay or cold-install validation here. The capture is separate from the shared deswizzle and symbol-index optimizations.
|
||||||
- Most RomFS files are zstd with dictionaries (`*.zs`); the dictionaries are `Pack/ZsDic.pack.zs`, which is also the profile's `romfs_marker`.
|
- Most RomFS files are zstd with dictionaries (`*.zs`); the dictionaries are `Pack/ZsDic.pack.zs`, which is also the profile's `romfs_marker`.
|
||||||
- Shaders: `Shader/*.bfsha.zs` (BNSH; `material.bfsha` holds most) and the layout shaders in `Shader/ApplicationPackage.Nin_NX_NVN.release.sarc.zs`.
|
- Shaders: `Shader/*.bfsha.zs` (BNSH; `material.bfsha` holds most) and the layout shaders in `Shader/ApplicationPackage.Nin_NX_NVN.release.sarc.zs`.
|
||||||
@@ -74,6 +75,8 @@ Game-specific findings. The general host changes they led to are listed under [F
|
|||||||
|
|
||||||
General host changes first needed here; each applies to every game:
|
General host changes first needed here; each applies to every game:
|
||||||
|
|
||||||
|
- **BAHS pairs omitted from vertex padding:** [Pixel shader inputs the vertex shader does not write](../../../docs/reference/shaders.md#pixel-shader-inputs-the-vertex-shader-does-not-write).
|
||||||
|
|
||||||
- **Resolved in the host: frame-step recovery after a 30 fps movie cap**: [The frame step after a 30 fps cap ends](../../../docs/reference/frame-rate.md#the-frame-step-after-a-30-fps-cap-ends). Modelled with the movie flag; no new gameplay recovery observation is claimed.
|
- **Resolved in the host: frame-step recovery after a 30 fps movie cap**: [The frame step after a 30 fps cap ends](../../../docs/reference/frame-rate.md#the-frame-step-after-a-30-fps-cap-ends). Modelled with the movie flag; no new gameplay recovery observation is claimed.
|
||||||
|
|
||||||
- **Packaged texture block dimensions and linear storage rows:** [Packaged texture block dimensions and linear storage rows](../../../docs/reference/nvn-objects.md#packaged-texture-block-dimensions-and-linear-storage-rows).
|
- **Packaged texture block dimensions and linear storage rows:** [Packaged texture block dimensions and linear storage rows](../../../docs/reference/nvn-objects.md#packaged-texture-block-dimensions-and-linear-storage-rows).
|
||||||
|
|||||||
+1
-1
@@ -3,7 +3,7 @@
|
|||||||
Each game the port knows has a folder `dm/games/<id>/`, named after the base game's title ID: Mario's is `010015100B514000`, Engage's `0100A6301214E000`. The folder holds:
|
Each game the port knows has a folder `dm/games/<id>/`, named after the base game's title ID: Mario's is `010015100B514000`, Engage's `0100A6301214E000`. The folder holds:
|
||||||
|
|
||||||
- **`game.tsv`**: the game's constants, one `key<TAB>value` row each, with comments on their own lines.
|
- **`game.tsv`**: the game's constants, one `key<TAB>value` row each, with comments on their own lines.
|
||||||
- **The shader tables** that `shader_split.py` writes: `shader_map.tsv` (+ `.conflicts.tsv`), `shader_facts.tsv`, `shader_pairs.tsv`, and `pso_pairs.tsv` (the pairs of agl BAHS program variations, read only by the pipeline prebuild; a profile without it still packs).
|
- **The shader tables** that `shader_split.py` writes: `shader_map.tsv` (+ `.conflicts.tsv`), `shader_facts.tsv`, `shader_pairs.tsv`, and `pso_pairs.tsv` (agl BAHS program variations). Both pair tables feed vertex output padding and the pipeline prebuild; for a profile without `pso_pairs.tsv`, the NSP script keeps the generated table beside the shader package.
|
||||||
- **`shader_facts_manual.tsv`** (optional): hand-written translation facts in `shader_facts.tsv`'s row format, for what the machine code cannot tell (Link's Awakening: a vertex input the integer-input guess gets wrong; Breath of the Wild and Tears of the Kingdom: integer fragment outputs, which a game without a PC version has no other source for). `shader_split.py` merges it over the generated rows; since the table is on the chain, changing it changes only that profile's chain version.
|
- **`shader_facts_manual.tsv`** (optional): hand-written translation facts in `shader_facts.tsv`'s row format, for what the machine code cannot tell (Link's Awakening: a vertex input the integer-input guess gets wrong; Breath of the Wild and Tears of the Kingdom: integer fragment outputs, which a game without a PC version has no other source for). `shader_split.py` merges it over the generated rows; since the table is on the chain, changing it changes only that profile's chain version.
|
||||||
- **`pso_manifest.tsv`**: the prewarm seed list.
|
- **`pso_manifest.tsv`**: the prewarm seed list.
|
||||||
- **`bench.tsv`** (optional): benchmark routes for `scripts/bench.py` ([A benchmark on the Mac](../../docs/development.md#a-benchmark-on-the-mac)): a name, the scripted input (`DM_KEYS`), the measured window and a description, one route per line.
|
- **`bench.tsv`** (optional): benchmark routes for `scripts/bench.py` ([A benchmark on the Mac](../../docs/development.md#a-benchmark-on-the-mac)): a name, the scripted input (`DM_KEYS`), the measured window and a description, one route per line.
|
||||||
|
|||||||
+1
-1
@@ -53,7 +53,7 @@ The whole chain is driven by `tools/dmsh_build.py`; `scripts/build_ipa_from_nsp.
|
|||||||
- `tools/own_compilers.py <dmsh_build.py arguments>` runs `dmsh_build.py` with the local Metal compiler and spirv-cross even when they differ from the lock, printing one `note:` line that names them; the translator must still match. Neither the lock nor V changes, so the committed `dmsh_chain.h` stays valid. The NSP script builds the iOS shaders this way, so a player with another Xcode or a newer spirv-cross needs no `--update-lock`.
|
- `tools/own_compilers.py <dmsh_build.py arguments>` runs `dmsh_build.py` with the local Metal compiler and spirv-cross even when they differ from the lock, printing one `note:` line that names them; the translator must still match. Neither the lock nor V changes, so the committed `dmsh_chain.h` stays valid. The NSP script builds the iOS shaders this way, so a player with another Xcode or a newer spirv-cross needs no `--update-lock`.
|
||||||
- **`dm/games/<id>/c1_full.tsv`**: the index list for the shaders' built-in constant section (c1), generated by `tools/c1_guard.py` scanning the translated MSL; it lists the shader keys whose "run-time index could not be proven bounded". An index counts as proven only when interval arithmetic over the whole expression (every variable followed to its single definition; literals, `& K`, `% K`, shifts, `min`, `max`, `clamp` and casts bound it) keeps it non-negative and inside the span a shared bank gives that shader: its c1 data rounded up to 256 bytes plus the 256-byte zero gap. Every stage counts (`vp_`, `fp_`, `cp_` …). Its first line records the `b-<V>` and the inputs stamp of the library it was made from. For the shaders on the list the replay layer allocates a separate whole 64 KiB buffer for c1; when the list is missing or does not name the loaded library's version and stamp, it reports red and conservatively treats all shaders this way. The list is not committed (git ignores it): each checkout makes it from its own build's Mac outputs (a full `dmsh_build.py` run does).
|
- **`dm/games/<id>/c1_full.tsv`**: the index list for the shaders' built-in constant section (c1), generated by `tools/c1_guard.py` scanning the translated MSL; it lists the shader keys whose "run-time index could not be proven bounded". An index counts as proven only when interval arithmetic over the whole expression (every variable followed to its single definition; literals, `& K`, `% K`, shifts, `min`, `max`, `clamp` and casts bound it) keeps it non-negative and inside the span a shared bank gives that shader: its c1 data rounded up to 256 bytes plus the 256-byte zero gap. Every stage counts (`vp_`, `fp_`, `cp_` …). Its first line records the `b-<V>` and the inputs stamp of the library it was made from. For the shaders on the list the replay layer allocates a separate whole 64 KiB buffer for c1; when the list is missing or does not name the loaded library's version and stamp, it reports red and conservatively treats all shaders this way. The list is not committed (git ignores it): each checkout makes it from its own build's Mac outputs (a full `dmsh_build.py` run does).
|
||||||
- **`dm/games/<id>/cb_extent.tsv`**: how far into each uniform slot (cbuf binding) every shader can read, generated by `tools/cb_extent.py` from the translated MSL with `c1_guard.py`'s interval arithmetic: per shader key, `reg:bytes` for each slot whose reads are all proven to stay inside its first bytes, and `s<reg>` for each storage slot the shader never writes (SPIRV-Cross types every storage buffer Ryujinx emits as writable, so the host counts the others as written). When the game updates part of a bound uniform block inside a render pass, the replay layer copies only that much of the binding to give the draws after the update their data without ending the pass (`nvn_mtl.m` `ubp_get`; Breath of the Wild binds a 64 KiB block and its vertex shaders read at most 544 bytes of it). Its first line records the `b-<V>` and inputs stamp, like `c1_full.tsv`'s; when it is missing or stale, bindings are copied whole (up to 64 KiB). Like `c1_full.tsv`, it is not committed, and a full `dmsh_build.py` run makes it.
|
- **`dm/games/<id>/cb_extent.tsv`**: how far into each uniform slot (cbuf binding) every shader can read, generated by `tools/cb_extent.py` from the translated MSL with `c1_guard.py`'s interval arithmetic: per shader key, `reg:bytes` for each slot whose reads are all proven to stay inside its first bytes, and `s<reg>` for each storage slot the shader never writes (SPIRV-Cross types every storage buffer Ryujinx emits as writable, so the host counts the others as written). When the game updates part of a bound uniform block inside a render pass, the replay layer copies only that much of the binding to give the draws after the update their data without ending the pass (`nvn_mtl.m` `ubp_get`; Breath of the Wild binds a 64 KiB block and its vertex shaders read at most 544 bytes of it). Its first line records the `b-<V>` and inputs stamp, like `c1_full.tsv`'s; when it is missing or stale, bindings are copied whole (up to 64 KiB). Like `c1_full.tsv`, it is not committed, and a full `dmsh_build.py` run makes it.
|
||||||
- **`dm/games/<id>/shader_pairs.tsv`**: the (vertex key, pixel key) of every BNSH variation (Unity: of every program), written by `tools/shader_split.py`. Metal only builds a pipeline when every pixel shader input is a vertex shader output, while Maxwell gives an unwritten attribute a default; `dmsh_build.py` therefore adds the outputs a vertex shader's partners read but it doesn't write, set to (0, 0, 0, 1) (`vs_pad`). The table is part of the translation chain version.
|
- **`dm/games/<id>/shader_pairs.tsv` and `pso_pairs.tsv`**: the (vertex key, pixel key) of every BNSH variation (Unity: of every program) and agl BAHS program variation, written by `tools/shader_split.py`. Metal only builds a pipeline when every pixel shader input is a vertex shader output, while Maxwell gives an unwritten attribute a default; `dmsh_build.py` therefore adds the outputs a vertex shader's partners read but it doesn't write, set to (0, 0, 0, 1) (`vs_pad`). The profile tables are part of its inputs digest and the per-shader cache keys. A profile without `pso_pairs.tsv` takes the generated table beside the shader package.
|
||||||
- **Several game versions**: `shader_split.py` takes comma-separated RomFS and `main` lists (base game first) and writes the union; the map, facts and pairs tables currently cover Mario v0 and 1.2.1.
|
- **Several game versions**: `shader_split.py` takes comma-separated RomFS and `main` lists (base game first) and writes the union; the map, facts and pairs tables currently cover Mario v0 and 1.2.1.
|
||||||
- **`dm/games/<id>/pso_manifest.tsv`**: the warm-up seed manifest bundled with the app: shaders (`S`), render pipelines (`R`, the hex of `pso_skey` in `nvn/nvn_pso_skey.h`) and compute pipelines (`C`) to pre-build at startup. The row order is the pre-build order (first use); do not sort it. Its companion is `pso_user.tsv`, which the host appends next to the save directory whenever it has to build something on the spot, and which is pre-built after the seed at the next launch. Both name shaders by key and describe pipeline settings only, so a list recorded on one device works on any other; lines a build cannot use are skipped. Recorded lines can be appended to the seed below its header (currently empty).
|
- **`dm/games/<id>/pso_manifest.tsv`**: the warm-up seed manifest bundled with the app: shaders (`S`), render pipelines (`R`, the hex of `pso_skey` in `nvn/nvn_pso_skey.h`) and compute pipelines (`C`) to pre-build at startup. The row order is the pre-build order (first use); do not sort it. Its companion is `pso_user.tsv`, which the host appends next to the save directory whenever it has to build something on the spot, and which is pre-built after the seed at the next launch. Both name shaders by key and describe pipeline settings only, so a list recorded on one device works on any other; lines a build cannot use are skipped. Recorded lines can be appended to the seed below its header (currently empty).
|
||||||
- **Shaders embedded in the program itself**: the main program additionally contains 4 compute shaders that are not in the shader package; they were hand-written as MSL from the disassembly and are in `nvn/nvn_builtin.h`.
|
- **Shaders embedded in the program itself**: the main program additionally contains 4 compute shaders that are not in the shader package; they were hand-written as MSL from the disassembly and are in `nvn/nvn_builtin.h`.
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
// Generated file (dm/gfx/tools/dmsh_chain.py --header): the B shader library translation chain version this source tree expects (default options), one for every game;
|
// Generated file (dm/gfx/tools/dmsh_chain.py --header): the B shader library translation chain version this source tree expects (default options), one for every game;
|
||||||
// the host loads only <library dir>/b-<V>/. Run dmsh_chain.py --header after changing the chain's tools and commit this file with the change
|
// the host loads only <library dir>/b-<V>/. Run dmsh_chain.py --header after changing the chain's tools and commit this file with the change
|
||||||
#define DMSH_CHAIN "5cddc47253ab"
|
#define DMSH_CHAIN "7776d7a9719d"
|
||||||
@@ -5376,7 +5376,7 @@ static int pso_bundle_covers(NSArray<NSArray<NSString *> *> *pairs, NSString *wa
|
|||||||
static int pso_prebuild(int threads, int wait, dispatch_qos_class_t qos) {
|
static int pso_prebuild(int threads, int wait, dispatch_qos_class_t qos) {
|
||||||
shmap_load(); pso4_init();
|
shmap_load(); pso4_init();
|
||||||
if (!g_comp4 || (!g_arch_bundle && (!g_ser4 || !g_arch_dir)) || atomic_exchange(&g_pre_running, 1)) return -1;
|
if (!g_comp4 || (!g_arch_bundle && (!g_ser4 || !g_arch_dir)) || atomic_exchange(&g_pre_running, 1)) return -1;
|
||||||
// The pairs: shader_pairs.tsv (BNSH variations and the other readers' pairs) and pso_pairs.tsv (agl BAHS program variations, which vs_pad does not read; DM_PSO_PAIRS: another path),
|
// The pairs: shader_pairs.tsv (BNSH variations and the other readers' pairs) and pso_pairs.tsv (agl BAHS program variations, also read by vs_pad; DM_PSO_PAIRS: another path),
|
||||||
// both beside the shader map
|
// both beside the shader map
|
||||||
NSString *pp = [g_shmap_path.stringByDeletingLastPathComponent stringByAppendingPathComponent:@"shader_pairs.tsv"];
|
NSString *pp = [g_shmap_path.stringByDeletingLastPathComponent stringByAppendingPathComponent:@"shader_pairs.tsv"];
|
||||||
const char *ev = getenv("DM_PSO_PAIRS");
|
const char *ev = getenv("DM_PSO_PAIRS");
|
||||||
|
|||||||
@@ -0,0 +1,149 @@
|
|||||||
|
"""Both shader-pair tables must feed vertex padding, its cache dependencies and every vertex variant (#283, #276)."""
|
||||||
|
import ast
|
||||||
|
import collections
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import types
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
TOOLS = Path(__file__).resolve().parents[1] / 'tools'
|
||||||
|
sys.path.insert(0, str(TOOLS))
|
||||||
|
import dmsh_chain
|
||||||
|
import dmsh_pull
|
||||||
|
import pso_offline
|
||||||
|
|
||||||
|
|
||||||
|
def builder():
|
||||||
|
# Use production functions without the CLI's translator/profile initialization.
|
||||||
|
names = {'load_pairs', 'vs_pad', 'vs_depth', 'shader_key', 'pair_keyed'}
|
||||||
|
tree = ast.parse((TOOLS / 'dmsh_build.py').read_text())
|
||||||
|
ns = dict(collections=collections, os=os, re=re, sys=sys, hashlib=hashlib)
|
||||||
|
exec(compile(ast.Module(body=[n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name in names],
|
||||||
|
type_ignores=[]), str(TOOLS / 'dmsh_build.py'), 'exec'), ns)
|
||||||
|
return ns
|
||||||
|
|
||||||
|
|
||||||
|
VS = '''#include <metal_stdlib>
|
||||||
|
using namespace metal;
|
||||||
|
struct main0_out
|
||||||
|
{
|
||||||
|
float4 gl_Position [[position]];
|
||||||
|
float4 v0 [[user(locn0)]];
|
||||||
|
};
|
||||||
|
struct main0_in
|
||||||
|
{
|
||||||
|
float4 m_0 [[attribute(0)]];
|
||||||
|
};
|
||||||
|
vertex main0_out main0(main0_in in [[stage_in]], uint vid [[vertex_id]])
|
||||||
|
{
|
||||||
|
main0_out out = {};
|
||||||
|
out.gl_Position = float4(float2((vid << 1) & 2, vid & 2) * 2.0 - 1.0, 0.0, 1.0);
|
||||||
|
out.v0 = in.m_0;
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
'''
|
||||||
|
|
||||||
|
|
||||||
|
def pixel(loc):
|
||||||
|
return f'''#include <metal_stdlib>
|
||||||
|
using namespace metal;
|
||||||
|
struct main0_in
|
||||||
|
{{
|
||||||
|
float4 v0 [[user(locn0)]];
|
||||||
|
float4 missing [[user(locn{loc})]];
|
||||||
|
}};
|
||||||
|
fragment float4 main0(main0_in in [[stage_in]]) {{ return in.v0 + in.missing; }}
|
||||||
|
'''
|
||||||
|
|
||||||
|
|
||||||
|
class Pairing(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.TemporaryDirectory(); self.addCleanup(self.tmp.cleanup)
|
||||||
|
self.root = Path(self.tmp.name)
|
||||||
|
self.profile = self.root / 'profile'; self.profile.mkdir()
|
||||||
|
self.package = self.root / 'shader/pkg'; self.package.mkdir(parents=True)
|
||||||
|
self.g = types.SimpleNamespace(table=lambda n: self.profile / n, rows=[])
|
||||||
|
self.g.table('shader_pairs.tsv').write_text('# BNSH\nv\tf0\n')
|
||||||
|
self.extra = self.package.parent / 'pso_pairs.tsv'
|
||||||
|
self.extra.write_text('# BAHS\nv\tf1\nv\tf6\nv\tf1\n\n')
|
||||||
|
self.ns = builder()
|
||||||
|
self.ns.update(out=str(self.root), PAIRS=self.ns['load_pairs'](self.g, self.package))
|
||||||
|
for key, loc in [('f0', 0), ('f1', 1), ('f6', 6)]:
|
||||||
|
src = pixel(loc)
|
||||||
|
if not loc: src = src.replace(' float4 missing [[user(locn0)]];\n', '').replace('in.v0 + in.missing', 'in.v0')
|
||||||
|
(self.root / f'{key}_b.metal').write_text(src)
|
||||||
|
|
||||||
|
def test_split_fallback_union_and_defaults(self):
|
||||||
|
self.assertEqual(self.ns['PAIRS'], {'v': {'f0', 'f1', 'f6'}})
|
||||||
|
src, pad = self.ns['vs_pad']('v', VS)
|
||||||
|
self.assertEqual(pad, [1, 6])
|
||||||
|
for loc in pad:
|
||||||
|
self.assertIn(f'float4 dm_pad{loc} [[user(locn{loc})]];', src)
|
||||||
|
self.assertIn(f'out.dm_pad{loc} = float4(0.0, 0.0, 0.0, 1.0);', src)
|
||||||
|
self.assertEqual(src.count('float4 v0 [[user(locn0)]];'), 1)
|
||||||
|
self.assertEqual(self.ns['vs_pad']('v', src), (src, []))
|
||||||
|
|
||||||
|
def test_profile_table_wins_and_optional_table(self):
|
||||||
|
self.g.table('pso_pairs.tsv').write_text('v\tf6\n')
|
||||||
|
self.assertEqual(self.ns['load_pairs'](self.g, self.package), {'v': {'f0', 'f6'}})
|
||||||
|
self.g.table('pso_pairs.tsv').unlink(); self.extra.unlink()
|
||||||
|
self.assertEqual(self.ns['load_pairs'](self.g, self.package), {'v': {'f0'}})
|
||||||
|
self.assertEqual(self.ns['vs_pad']('unpaired', VS), (VS, []))
|
||||||
|
|
||||||
|
def test_profile_input_stamp_changes(self):
|
||||||
|
missing = dmsh_chain.inputs(self.g)
|
||||||
|
self.g.table('pso_pairs.tsv').write_text('v\tf6\n')
|
||||||
|
present = dmsh_chain.inputs(self.g)
|
||||||
|
self.g.table('pso_pairs.tsv').write_text('v\tf1\n')
|
||||||
|
self.assertNotEqual(missing, present)
|
||||||
|
self.assertNotEqual(present, dmsh_chain.inputs(self.g))
|
||||||
|
|
||||||
|
def test_vertex_cache_depends_on_extra_pairs_and_pixel_code(self):
|
||||||
|
self.ns.update(FACTS={}, MAP_ROWS={}, MAP_CONFLICT_ROWS={}, TOOL_FP=b'tools', CODE_FP='code',
|
||||||
|
PROFILE_ROWS=b'', shader_rule_source=lambda name: b'')
|
||||||
|
key = self.ns['shader_key']
|
||||||
|
v1 = key('v', 'pkg.bin', b'vertex')
|
||||||
|
self.ns['PAIRS']['v'].remove('f6')
|
||||||
|
self.assertNotEqual(v1, key('v', 'pkg.bin', b'vertex'))
|
||||||
|
f1, f2 = key('f1', 'pkg.bin', b'pixel'), key('f1', 'pkg.bin', b'changed pixel')
|
||||||
|
self.assertNotEqual(self.ns['pair_keyed'](v1, [f1]), self.ns['pair_keyed'](v1, [f2]))
|
||||||
|
|
||||||
|
@unittest.skipUnless(shutil.which('xcrun'), 'needs the Metal compiler')
|
||||||
|
def test_metal_archive_with_padded_regular_and_pulling_entries(self):
|
||||||
|
probe = subprocess.run(['xcrun', '-sdk', 'macosx', 'metal-tt', '-version'], capture_output=True)
|
||||||
|
if probe.returncode: self.skipTest('no Metal toolchain')
|
||||||
|
src, _ = self.ns['vs_pad']('v', VS)
|
||||||
|
_, full = self.ns['vs_depth'](src)
|
||||||
|
pull, meta = dmsh_pull.variant(full)
|
||||||
|
self.assertIsNotNone(pull, meta)
|
||||||
|
for name, text in [('v_b', full), ('v_bp', pull), ('f1_b', pixel(1)), ('f6_b', pixel(6))]:
|
||||||
|
msl, air, lib = [self.root / (name + ext) for ext in ('.metal', '.air', '.metallib')]
|
||||||
|
msl.write_text(text)
|
||||||
|
for sdk in ('iphoneos', 'macosx'):
|
||||||
|
r = subprocess.run(['xcrun', '-sdk', sdk, 'metal', '-c', '-fpreserve-invariance', '-fmodules-cache-path=' + str(self.root / 'modules'), msl, '-o', air], capture_output=True, text=True)
|
||||||
|
self.assertEqual(r.returncode, 0, r.stderr)
|
||||||
|
# The archive below consumes the Mac libraries.
|
||||||
|
subprocess.run(['xcrun', '-sdk', 'macosx', 'metallib', air, '-o', lib], check=True, capture_output=True)
|
||||||
|
bases = [(v, entry, f, 'main0') for v in ('v_b', 'v_bp') for entry in ('main0', 'main0_z') for f in ('f1_b', 'f6_b')]
|
||||||
|
plan = pso_offline.script(self.root, bases)
|
||||||
|
for base, desc in zip(bases, plan['pipeline_descriptors']['render_pipeline_descriptors']):
|
||||||
|
if base[0] == 'v_b':
|
||||||
|
desc['vertex_descriptor'] = {'attributes': [{'format': 'Float4', 'offset': 0, 'buffer_index': 0}],
|
||||||
|
'layouts': [{'stride': 16, 'step_function': 'PerVertex', 'step_rate': 1}]}
|
||||||
|
script = self.root / 'bases.mtl4-json'; script.write_text(json.dumps(plan))
|
||||||
|
archive = self.root / 'pairs.mtl4a'
|
||||||
|
osmin = pso_offline.os_min_default('macos')
|
||||||
|
r = subprocess.run(['xcrun', '-sdk', 'macosx', 'metal-tt', '-arch', 'applegpu_g13s', '-macos-version-min', osmin, script, '-o', archive],
|
||||||
|
env=dict(os.environ, TMPDIR=str(self.root)), capture_output=True, text=True)
|
||||||
|
self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
|
||||||
|
self.assertTrue(archive.is_file())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__': unittest.main()
|
||||||
@@ -595,10 +595,20 @@ def int_inputs(src):
|
|||||||
if not bad and nint: out[str(loc_)] = 'uint'
|
if not bad and nint: out[str(loc_)] = 'uint'
|
||||||
return out
|
return out
|
||||||
|
|
||||||
# vertex / pixel pairs (taken from BNSH variations by shader_split.py): vertex key → paired pixel keys
|
# Both pair tables feed pipelines, so both must feed padding, build order and shader cache keys.
|
||||||
PAIRS = collections.defaultdict(set)
|
def load_pairs(game, package):
|
||||||
for _ln in open(GAME.table('shader_pairs.tsv')):
|
pairs = collections.defaultdict(set)
|
||||||
if _ln.strip() and not _ln.startswith('#'): _v, _p = _ln.split(); PAIRS[_v].add(_p)
|
for table in ('shader_pairs.tsv', 'pso_pairs.tsv'):
|
||||||
|
path = str(game.table(table))
|
||||||
|
if table == 'pso_pairs.tsv' and not os.path.isfile(path):
|
||||||
|
path = os.path.join(os.path.dirname(os.path.abspath(package)), table) # the split's copy for profiles predating this table
|
||||||
|
if table == 'pso_pairs.tsv' and not os.path.isfile(path): continue
|
||||||
|
with open(path) as f:
|
||||||
|
for ln in f:
|
||||||
|
if ln.strip() and not ln.startswith('#'): v, p = ln.split(); pairs[v].add(p)
|
||||||
|
return pairs
|
||||||
|
|
||||||
|
PAIRS = load_pairs(GAME, dm_dir)
|
||||||
|
|
||||||
def vs_pad(key, src):
|
def vs_pad(key, src):
|
||||||
"""Give a vertex shader the outputs its paired pixel shaders read but it does not write, with the value (0,0,0,1).
|
"""Give a vertex shader the outputs its paired pixel shaders read but it does not write, with the value (0,0,0,1).
|
||||||
@@ -608,7 +618,8 @@ def vs_pad(key, src):
|
|||||||
want = set()
|
want = set()
|
||||||
for ps in PAIRS.get(key, ()):
|
for ps in PAIRS.get(key, ()):
|
||||||
fn = os.path.join(out, ps + '_b.metal')
|
fn = os.path.join(out, ps + '_b.metal')
|
||||||
if os.path.exists(fn): want |= {int(x) for x in re.findall(r'\[\[user\(locn(\d+)\)', open(fn).read())}
|
if os.path.exists(fn):
|
||||||
|
with open(fn) as f: want |= {int(x) for x in re.findall(r'\[\[user\(locn(\d+)\)', f.read())}
|
||||||
miss = sorted(want - {int(x) for x in re.findall(r'\[\[user\(locn(\d+)\)', src)})
|
miss = sorted(want - {int(x) for x in re.findall(r'\[\[user\(locn(\d+)\)', src)})
|
||||||
i = src.find('struct main0_out\n{\n'); j = src.find('\n};', i); k = src.find('main0_out out = {};')
|
i = src.find('struct main0_out\n{\n'); j = src.find('\n};', i); k = src.find('main0_out out = {};')
|
||||||
if not miss or i < 0 or j < 0 or k < 0: return src, []
|
if not miss or i < 0 or j < 0 or k < 0: return src, []
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ FILES += ['../third_party/ryujinx/src/Ryujinx.Graphics.Shader/' + f for f in (
|
|||||||
'Translation/TranslationFlags.cs', 'Translation/HostCapabilities.cs',
|
'Translation/TranslationFlags.cs', 'Translation/HostCapabilities.cs',
|
||||||
'Translation/TranslatorContext.cs', 'Translation/EmitterContext.cs', 'Translation/ShaderDefinitions.cs', 'CodeGen/Spirv/Instructions.cs',
|
'Translation/TranslatorContext.cs', 'Translation/EmitterContext.cs', 'Translation/ShaderDefinitions.cs', 'CodeGen/Spirv/Instructions.cs',
|
||||||
'StructuredIr/ShaderProperties.cs', 'Translation/Optimizations/BindlessElimination.cs')]
|
'StructuredIr/ShaderProperties.cs', 'Translation/Optimizations/BindlessElimination.cs')]
|
||||||
INPUT_TABLES = ['shader_facts.tsv', 'shader_map.tsv', 'shader_map.tsv.conflicts.tsv', 'shader_pairs.tsv', 'bf_want.tsv'] # in the game profile folder
|
INPUT_TABLES = ['shader_facts.tsv', 'shader_map.tsv', 'shader_map.tsv.conflicts.tsv', 'shader_pairs.tsv', 'pso_pairs.tsv', 'bf_want.tsv'] # in the game profile folder
|
||||||
LOCK = os.path.join(HERE, 'dmsh_chain.lock'); HEADER = os.path.join(GFX, 'nvn/dmsh_chain.h')
|
LOCK = os.path.join(HERE, 'dmsh_chain.lock'); HEADER = os.path.join(GFX, 'nvn/dmsh_chain.h')
|
||||||
STAMP = 'inputs.tsv' # in the library directory b-<V>/
|
STAMP = 'inputs.tsv' # in the library directory b-<V>/
|
||||||
# the translator defaults to the in-repo build that matches the lock file (a rebuilt dll has different fingerprints)
|
# the translator defaults to the in-repo build that matches the lock file (a rebuilt dll has different fingerprints)
|
||||||
|
|||||||
@@ -1800,8 +1800,8 @@ def ue_shaders(d, label):
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
def var_pairs(found):
|
def var_pairs(found):
|
||||||
"""(vertex key, pixel key) within the same BNSH variation; (vertex key, pixel key, 'prog') within the same BAHS program variation (bahs_variations), which go to the pipeline
|
"""(vertex key, pixel key) within the same BNSH variation; (vertex key, pixel key, 'prog') within the same BAHS program variation (bahs_variations).
|
||||||
pair table only (main: shader_pairs.tsv, which vs_pad reads, keeps the BNSH pairs)"""
|
The tables stay separate; dmsh_build.py pads against both, as the pipeline prebuild uses both."""
|
||||||
var = collections.defaultdict(dict)
|
var = collections.defaultdict(dict)
|
||||||
for where, st, ctl, code in found:
|
for where, st, ctl, code in found:
|
||||||
g = where if '#var' in where else BAHS_VAR.get(where)
|
g = where if '#var' in where else BAHS_VAR.get(where)
|
||||||
@@ -2021,7 +2021,7 @@ def main():
|
|||||||
for a, b in sorted(pairs): f.write(f'{a}\t{b}\n')
|
for a, b in sorted(pairs): f.write(f'{a}\t{b}\n')
|
||||||
if psopairsout:
|
if psopairsout:
|
||||||
with open(psopairsout, 'w') as f:
|
with open(psopairsout, 'w') as f:
|
||||||
f.write('# kind: evidence vertex / pixel shader pairs of agl program variations (BAHS) that shader_pairs.tsv does not hold, for the pipeline prebuild only (generated by dm/gfx/tools/shader_split.py, do not edit): vertex key <TAB> pixel key\n')
|
f.write('# kind: evidence vertex / pixel shader pairs of agl program variations (BAHS) that shader_pairs.tsv does not hold, for vertex output padding and the pipeline prebuild (generated by dm/gfx/tools/shader_split.py, do not edit): vertex key <TAB> pixel key\n')
|
||||||
for a, b in sorted(prog - pairs): f.write(f'{a}\t{b}\n')
|
for a, b in sorted(prog - pairs): f.write(f'{a}\t{b}\n')
|
||||||
print(f'shaders: {len(rows)} (' + ', '.join(f'{STAGE_NAME[s]} {n}' for s, n in sorted(stat.items())) + '); sources ' + ', '.join(f'{g} {n}' for g, n in per_src.most_common()) + f'; compute shader answers {len(facts)} rows' + (f'; variation pairs {len(pairs)}' if pairsout else ''))
|
print(f'shaders: {len(rows)} (' + ', '.join(f'{STAGE_NAME[s]} {n}' for s, n in sorted(stat.items())) + '); sources ' + ', '.join(f'{g} {n}' for g, n in per_src.most_common()) + f'; compute shader answers {len(facts)} rows' + (f'; variation pairs {len(pairs)}' if pairsout else ''))
|
||||||
|
|
||||||
|
|||||||
@@ -338,6 +338,15 @@ class SplitShaders(Base):
|
|||||||
self.assertEqual(self.splits, 2)
|
self.assertEqual(self.splits, 2)
|
||||||
self.go(g); self.assertEqual(self.splits, 3) # a damaged run leaves no key behind
|
self.go(g); self.assertEqual(self.splits, 3) # a damaged run leaves no key behind
|
||||||
|
|
||||||
|
def test_missing_split_pairs_are_generated_again(self):
|
||||||
|
for n in self.NAMES[:3]: (self.src / n).write_text(self.table(n))
|
||||||
|
g = game.Game(self.src)
|
||||||
|
self.go(g, own=False)
|
||||||
|
pairs = self.pkg.with_name('pso_pairs.tsv'); pairs.unlink()
|
||||||
|
self.go(g, own=False)
|
||||||
|
self.assertEqual(self.splits, 2)
|
||||||
|
self.assertEqual(pairs.read_text(), self.table('pso_pairs.tsv'))
|
||||||
|
|
||||||
def test_derived_rerun_is_incremental(self): # the whole derived path of a second run: derive, prepare, rewrite keys, split
|
def test_derived_rerun_is_incremental(self): # the whole derived path of a second run: derive, prepare, rewrite keys, split
|
||||||
g = self.own(); g = nsp.set_keys(g, {'mem_used': ('0x4800000', '')}, 'After the rewrite'); k = self.go(g)
|
g = self.own(); g = nsp.set_keys(g, {'mem_used': ('0x4800000', '')}, 'After the rewrite'); k = self.go(g)
|
||||||
g = self.own(); self.assertEqual(g.get('mem_used'), '0x4800000')
|
g = self.own(); self.assertEqual(g.get('mem_used'), '0x4800000')
|
||||||
|
|||||||
@@ -110,6 +110,27 @@ class Nvn(Base):
|
|||||||
(self.tmp / 'cb_extent.tsv').write_text(f'# chain b-{CHAIN} inputs bbbbbbbbbbbb 2026-10-23T00:00:00+00:00\n')
|
(self.tmp / 'cb_extent.tsv').write_text(f'# chain b-{CHAIN} inputs bbbbbbbbbbbb 2026-10-23T00:00:00+00:00\n')
|
||||||
self.assertEqual(self.build(True), self.sh / 'cache'); self.assertEqual([Path(c[1]).name for c, _ in self.ran], ['cb_extent.py'])
|
self.assertEqual(self.build(True), self.sh / 'cache'); self.assertEqual([Path(c[1]).name for c, _ in self.ran], ['cb_extent.py'])
|
||||||
|
|
||||||
|
def test_split_pairs_invalidate_shader_reuse(self):
|
||||||
|
# A registered profile may have no pso_pairs.tsv: the split's copy feeds vs_pad.
|
||||||
|
def fingerprint(*args, **kwargs):
|
||||||
|
path = next(x[1][0] for x in args if isinstance(x, tuple) and x[0] == 'files' and x[1] and x[1][0].name == 'pso_pairs.tsv')
|
||||||
|
return str(path.read_bytes() if path.is_file() else b'missing')
|
||||||
|
with mock.patch.object(nsp, 'fingerprint', side_effect=fingerprint):
|
||||||
|
self.build(True); self.assertEqual(len(self.ran), 1)
|
||||||
|
self.build(True); self.assertEqual(self.ran, [])
|
||||||
|
(self.sh / 'pso_pairs.tsv').write_text('v\tf1\n')
|
||||||
|
self.build(True); self.assertEqual(len(self.ran), 1)
|
||||||
|
(self.sh / 'pso_pairs.tsv').write_text('v\tf2\n')
|
||||||
|
self.build(True); self.assertEqual(len(self.ran), 1)
|
||||||
|
self.build(True); self.assertEqual(self.ran, [])
|
||||||
|
|
||||||
|
def test_shader_watch_finishes_extra_pairs_last(self):
|
||||||
|
(self.tmp / 'shader_map.tsv').write_text('p.bin\t0\tvs\ta\tsrc\np.bin\t1\tps\tb\tsrc\np.bin\t2\tvs\tc\tsrc\n')
|
||||||
|
(self.tmp / 'shader_pairs.tsv').write_text('c\tb\n')
|
||||||
|
pso = self.tmp / 'pso_pairs.tsv'; pso.write_text('a\tb\n')
|
||||||
|
watch = nsp.ShaderWatch(self.g, self.tmp, self.tmp, pso)
|
||||||
|
self.assertEqual(list(watch.pos), ['b', 'a', 'c'])
|
||||||
|
|
||||||
|
|
||||||
class EnvFile(Base):
|
class EnvFile(Base):
|
||||||
def test_pairs(self):
|
def test_pairs(self):
|
||||||
|
|||||||
@@ -411,7 +411,7 @@ One line per symptom in the [subsystem references](../reference/): what you see
|
|||||||
- GLSLC outputs fail the reader's check and are dropped (44 of DARK SOULS: REMASTERED's `main`'s 72 outputs and most of its archives' pixel shaders). → [GLSLC outputs read through their section headers](../reference/shaders.md#glslc-outputs-read-through-their-section-headers)
|
- GLSLC outputs fail the reader's check and are dropped (44 of DARK SOULS: REMASTERED's `main`'s 72 outputs and most of its archives' pixel shaders). → [GLSLC outputs read through their section headers](../reference/shaders.md#glslc-outputs-read-through-their-section-headers)
|
||||||
- `shader_split.py` finds no stage for vertex shaders and stops at the first captured output (`no GLSLC output`). Translated, every post pass and the window draw one texel's colour (a black screen with sound); every varying at (0, 0, 0, 1). → [VertexA and VertexB parts merged into one vertex shader](../reference/shaders.md#vertexa-and-vertexb-parts-merged-into-one-vertex-shader)
|
- `shader_split.py` finds no stage for vertex shaders and stops at the first captured output (`no GLSLC output`). Translated, every post pass and the window draw one texel's colour (a black screen with sound); every varying at (0, 0, 0, 1). → [VertexA and VertexB parts merged into one vertex shader](../reference/shaders.md#vertexa-and-vertexb-parts-merged-into-one-vertex-shader)
|
||||||
- SPIRV-Cross can't parse the SPIR-V (`nullptr` exception). → [SPIRV-Cross parse failures](../reference/shaders.md#spirv-cross-parse-failures)
|
- SPIRV-Cross can't parse the SPIR-V (`nullptr` exception). → [SPIRV-Cross parse failures](../reference/shaders.md#spirv-cross-parse-failures)
|
||||||
- `Fragment input(s) user(locnN) mismatching vertex shader output type(s) or not written by vertex shader` and the draw is skipped; the host reports `[RED] failed to create render pipeline (vertex … fragment …)`. → [Pixel shader inputs the vertex shader does not write](../reference/shaders.md#pixel-shader-inputs-the-vertex-shader-does-not-write)
|
- `Fragment input(s) user(locnN) mismatching vertex shader output type(s) or not written by vertex shader` and the draw is skipped; the host reports `[RED] failed to create render pipeline (vertex … fragment …)`. Offline packing fails with `pso_offline: metal-tt failed` for the same reason. → [Pixel shader inputs the vertex shader does not write](../reference/shaders.md#pixel-shader-inputs-the-vertex-shader-does-not-write)
|
||||||
- outdoors dim and nothing casts a sun shadow (every shadow-map lookup fails). → [Unwritten components of a written output start as (0, 0, 0, 1)](../reference/shaders.md#unwritten-components-of-a-written-output-start-as-0-0-0-1)
|
- outdoors dim and nothing casts a sun shadow (every shadow-map lookup fails). → [Unwritten components of a written output start as (0, 0, 0, 1)](../reference/shaders.md#unwritten-components-of-a-written-output-start-as-0-0-0-1)
|
||||||
- the ground kept stale 32×32 blocks where the G-buffer draw failed the depth test: blocks of a stale image where a depth pre-pass and a later pass with other vertex shaders meet (Odyssey: 32×32 blocks on every fourth frame). → [Position invariance across shaders](../reference/shaders.md#position-invariance-across-shaders)
|
- the ground kept stale 32×32 blocks where the G-buffer draw failed the depth test: blocks of a stale image where a depth pre-pass and a later pass with other vertex shaders meet (Odyssey: 32×32 blocks on every fourth frame). → [Position invariance across shaders](../reference/shaders.md#position-invariance-across-shaders)
|
||||||
- the half of Mario facing the camera's left was white (GitHub issue 72, on an iPhone). → [min, max and clamp with a NaN operand](../reference/shaders.md#min-max-and-clamp-with-a-nan-operand)
|
- the half of Mario facing the camera's left was white (GitHub issue 72, on an iPhone). → [min, max and clamp with a NaN operand](../reference/shaders.md#min-max-and-clamp-with-a-nan-operand)
|
||||||
|
|||||||
+2
-2
@@ -256,7 +256,7 @@ The unsigned IPA build still uses the repository's `dm/ios/DmIOS.xcodeproj`, ski
|
|||||||
|
|
||||||
Each game has a profile in `dm/games/<id>/`:
|
Each game has a profile in `dm/games/<id>/`:
|
||||||
- `game.tsv` holds the title ID, known builds, code modules, RomFS marker, memory split, app name and bundle ID;
|
- `game.tsv` holds the title ID, known builds, code modules, RomFS marker, memory split, app name and bundle ID;
|
||||||
- the shader tables: `shader_map.tsv`, `shader_facts.tsv`, `shader_pairs.tsv`, `pso_pairs.tsv` (pairs only the pipeline prebuild reads), `pso_manifest.tsv`, and `c1_full.tsv` / `cb_extent.tsv`, which each checkout's full shader build makes (not committed);
|
- the shader tables: `shader_map.tsv`, `shader_facts.tsv`, `shader_pairs.tsv`, `pso_pairs.tsv` (pairs used by vertex output padding and the pipeline prebuild), `pso_manifest.tsv`, and `c1_full.tsv` / `cb_extent.tsv`, which each checkout's full shader build makes (not committed);
|
||||||
- `patches.py` (when there are any): the rewriter's patches of the game's own builds.
|
- `patches.py` (when there are any): the rewriter's patches of the game's own builds.
|
||||||
|
|
||||||
The shader translation chain has one version for every game, in `dm/gfx/nvn/dmsh_chain.h`.
|
The shader translation chain has one version for every game, in `dm/gfx/nvn/dmsh_chain.h`.
|
||||||
@@ -375,7 +375,7 @@ flowchart TD
|
|||||||
|
|
||||||
Every 300 frames the log has a `pipelines up to frame` line. It gives the submit thread's pipeline, library and prediction time; archive lookups that found the base or none; specializations and base compiles on all threads; draws with a hybrid pipeline; background builds; and the draws that waited, by why no stand-in served them. `DM_PSO_WAITLOG=1` logs every wait over 20 ms with its shader pair. `DM_PSO_PULL_ALL=1` (diagnostic) draws every draw that has a pulling pipeline with it. In steady state the GPU time is the same with and without the hybrid: in Breath of the Wild, mean GPU p50 over frames 2,400–5,100 was 8.98 ms with it, 8.98 ms on the Metal 4 path without it, and 9.35 ms through the `MTLDevice`.
|
Every 300 frames the log has a `pipelines up to frame` line. It gives the submit thread's pipeline, library and prediction time; archive lookups that found the base or none; specializations and base compiles on all threads; draws with a hybrid pipeline; background builds; and the draws that waited, by why no stand-in served them. `DM_PSO_WAITLOG=1` logs every wait over 20 ms with its shader pair. `DM_PSO_PULL_ALL=1` (diagnostic) draws every draw that has a pulling pipeline with it. In steady state the GPU time is the same with and without the hybrid: in Breath of the Wild, mean GPU p50 over frames 2,400–5,100 was 8.98 ms with it, 8.98 ms on the Metal 4 path without it, and 9.35 ms through the `MTLDevice`.
|
||||||
|
|
||||||
The **prebuild** puts the pulling base of every known shader pair into the archive (`pso_prebuild`), so a cold start's draws have stand-ins. The pairs come from the profile's `shader_pairs.tsv` and `pso_pairs.tsv`. `shader_pairs.tsv` pairs the stages of one BNSH variation (and the other readers' programs). It held none of the pairs from agl's BAHS shader archives (`.sharcb`, the agl / gsys / terrain resource packs), and in Breath of the Wild 109 of the 543 pairs drawn on entering Zora's Domain came from those. `shader_split.py` now reads the binary and program arrays of a BAHS file (`bahs_variations`) and pairs the n-th vertex binary of a program with its n-th fragment binary. Those pairs go to `pso_pairs.tsv`, which only the prebuild reads. `vs_pad` keeps reading `shader_pairs.tsv`, so the shader libraries, the inputs digest and the other profiles' tables are unchanged. For Breath of the Wild that is 3,012 more pairs, and all 543 pairs drawn are now known. A profile made before the table existed still packs: the NSP script then keeps the table beside the shader package and hands it to the prebuild (`DM_PSO_PAIRS`). Super Mario Odyssey, Mario Bros. Wonder, Mario Kart 8 Deluxe, 3D World, Animal Crossing, Captain Toad and Tears of the Kingdom have BAHS files too; their tables are made at their next pack.
|
The **prebuild** puts the pulling base of every known shader pair into the archive (`pso_prebuild`), so a cold start's draws have stand-ins. The pairs come from the profile's `shader_pairs.tsv` and `pso_pairs.tsv`. `shader_pairs.tsv` pairs the stages of one BNSH variation (and the other readers' programs). It held none of the pairs from agl's BAHS shader archives (`.sharcb`, the agl / gsys / terrain resource packs), and in Breath of the Wild 109 of the 543 pairs drawn on entering Zora's Domain came from those. `shader_split.py` now reads the binary and program arrays of a BAHS file (`bahs_variations`) and pairs the n-th vertex binary of a program with its n-th fragment binary. Those pairs go to `pso_pairs.tsv`. The shader build reads both tables to pad every vertex output its pixel partners need ([reference](reference/shaders.md#pixel-shader-inputs-the-vertex-shader-does-not-write)); the pairs also control build order and cache dependencies. The profile's `pso_pairs.tsv` is part of its inputs digest. For Breath of the Wild that is 3,012 more pairs, and all 543 pairs drawn are now known. A profile made before the table existed still packs: the NSP script then keeps the table beside the shader package, where the shader build reads it, and hands it to the prebuild (`DM_PSO_PAIRS`). Super Mario Odyssey, Mario Bros. Wonder, Mario Kart 8 Deluxe, 3D World, Animal Crossing, Captain Toad and Tears of the Kingdom have BAHS files too; their tables are made at their next pack.
|
||||||
|
|
||||||
With the prepare step (a window, the iPhone) the prebuild runs right after it when the bases are not there yet, before the game's first frame, behind the same progress screen ("Preparing pipelines"), on every core, flushing every 1,000 bases; a first start waits for it once. Built in the background while playing instead, they took an iPhone 17 Pro Max's GPU from about 13 to 15–18 ms a frame in Breath of the Wild, through the compiler's load on the shared power budget. Without a prepare step it still runs in the background after the prewarm, on a utility queue on half the performance cores (`DM_PSO_PREBUILD=0` turns it off either way). On the Mac it also runs as a tool, `DM_PSO_PREBUILD=tool`: build, write, exit. The NSP script runs the tool in a `--platform mac` build, into `<id>_build/run/pso_archive` (log `pso_prebuild.log`): this is how the Mac gets its archive. It runs once per chain version, depth mode and pair list (`pso_archive/prebuilt.tsv`: the pair lists' size and SHA-256). `DM_PSO_PREBUILD_DEPTH=zero` serves a game that sets the near=0 depth mode (the NSP script sets it from the profile key `depth_mode`); `DM_PSO_PREBUILD_THREADS=<n>` sets the workers. Its files hold bases without a vertex layout only, and are named `pNNNNN.mtl4a`; a base with a vertex layout is never looked up in them, and compaction leaves them alone. Breath of the Wild on an M2 Max: 15,767 bases, 910 MB in 16 files, 187 s with 7 workers for the first 12,755 and 19 s for the 3,012 more. Opening the files took 2.2 s on the first start (0.25 s from the file cache). The host's footprint is about 850 MB larger with them open (2,067 against 1,215 MiB at the title), which is one reason the Metal 4 path stays off on iOS unless a game's profile was tested with it (`pso_mtl4`). On iOS the standard is the archive made on the Mac and shipped in the app (next section); the device's own prebuild is the fallback, for an app without a usable archive.
|
With the prepare step (a window, the iPhone) the prebuild runs right after it when the bases are not there yet, before the game's first frame, behind the same progress screen ("Preparing pipelines"), on every core, flushing every 1,000 bases; a first start waits for it once. Built in the background while playing instead, they took an iPhone 17 Pro Max's GPU from about 13 to 15–18 ms a frame in Breath of the Wild, through the compiler's load on the shared power budget. Without a prepare step it still runs in the background after the prewarm, on a utility queue on half the performance cores (`DM_PSO_PREBUILD=0` turns it off either way). On the Mac it also runs as a tool, `DM_PSO_PREBUILD=tool`: build, write, exit. The NSP script runs the tool in a `--platform mac` build, into `<id>_build/run/pso_archive` (log `pso_prebuild.log`): this is how the Mac gets its archive. It runs once per chain version, depth mode and pair list (`pso_archive/prebuilt.tsv`: the pair lists' size and SHA-256). `DM_PSO_PREBUILD_DEPTH=zero` serves a game that sets the near=0 depth mode (the NSP script sets it from the profile key `depth_mode`); `DM_PSO_PREBUILD_THREADS=<n>` sets the workers. Its files hold bases without a vertex layout only, and are named `pNNNNN.mtl4a`; a base with a vertex layout is never looked up in them, and compaction leaves them alone. Breath of the Wild on an M2 Max: 15,767 bases, 910 MB in 16 files, 187 s with 7 workers for the first 12,755 and 19 s for the 3,012 more. Opening the files took 2.2 s on the first start (0.25 s from the file cache). The host's footprint is about 850 MB larger with them open (2,067 against 1,215 MiB at the title), which is one reason the Metal 4 path stays off on iOS unless a game's profile was tested with it (`pso_mtl4`). On iOS the standard is the archive made on the Mac and shipped in the app (next section); the device's own prebuild is the fallback, for an app without a usable archive.
|
||||||
|
|
||||||
|
|||||||
@@ -332,11 +332,11 @@ Finding, keying and translating shaders (`dm/gfx/tools/shader_split.py`, `dmsh_b
|
|||||||
|
|
||||||
### Pixel shader inputs the vertex shader does not write
|
### Pixel shader inputs the vertex shader does not write
|
||||||
|
|
||||||
- **Symptom:** `Fragment input(s) user(locnN) mismatching vertex shader output type(s) or not written by vertex shader` and the draw is skipped; the host reports `[RED] failed to create render pipeline (vertex … fragment …)`.
|
- **Symptom:** `Fragment input(s) user(locnN) mismatching vertex shader output type(s) or not written by vertex shader` and the draw is skipped; the host reports `[RED] failed to create render pipeline (vertex … fragment …)`. Offline packing fails with `pso_offline: metal-tt failed` for the same reason.
|
||||||
- **Cause:** every pixel shader input must be a vertex shader output. Maxwell attributes are untyped slots: a pixel shader that reads a slot the vertex shader never wrote gets a default, but Metal refuses to build the pipeline. Translating one shader at a time cannot know what the partner reads.
|
- **Cause:** every pixel shader input must be a vertex shader output. Maxwell attributes are untyped slots: a pixel shader that reads a slot the vertex shader never wrote gets a default, but Metal refuses to build the pipeline. Translating one shader at a time cannot know what the partner reads. The offline archive exposed a second omission: prebuild reads both `shader_pairs.tsv` and BAHS program variations in `pso_pairs.tsv`, but padding read only the first table.
|
||||||
- **Fix:** `shader_split.py` writes the pairs of each BNSH variation (`shader_pairs.tsv`) and `dmsh_build.py` gives each vertex shader the outputs its partners read but it doesn't write, as `float4` (0, 0, 0, 1) (`vs_pad`; vertex shaders with partners are finished after the pixel shaders, and the partner list is part of the shader's cache rule, [Shader cache key covering the pairs](#shader-cache-key-covering-the-pairs)). The host caches failed pipelines and reports each once with its vertex / fragment keys; before that, every draw retried the failing compile.
|
- **Fix:** `dmsh_build.py` reads both pair tables and gives each vertex shader the outputs its partners read but it doesn't write, as `float4` (0, 0, 0, 1) (`vs_pad`). A profile without `pso_pairs.tsv` takes the split's copy beside the package, as the archive builder does. The union controls pixel-before-vertex build order and shader cache dependencies ([Shader cache key covering the pairs](#shader-cache-key-covering-the-pairs)); the profile input stamp includes its `pso_pairs.tsv`, and the NSP script's shader reuse key includes the generated fallback. Padding precedes the depth, shadow-input, pulling and geometry-capture variants. Rebuild shaders and the host for the new chain. The host caches failed pipelines and reports each once with its vertex / fragment keys; before that, every draw retried the failing compile.
|
||||||
- **Found on:** [Super Mario Bros. Wonder](../../dm/games/010015100B514000/README.md): 331 of 9,043 pairs failed; the opening course used none of them, later levels did (about 11,500 failed builds in one 11-minute session).
|
- **Found on:** [Super Mario Bros. Wonder](../../dm/games/010015100B514000/README.md): 331 of 9,043 pairs failed; the opening course used none of them, later levels did (about 11,500 failed builds in one 11-minute session). The missing BAHS table was reproduced on [Tears of the Kingdom](../../dm/games/0100F2C0115B6000/README.md#files-and-shaders) ([#283](https://github.com/seedds/nxnative/issues/283)) and [Super Mario Odyssey](../../dm/games/0100000000010000/README.md#files-and-shaders) ([#276](https://github.com/seedds/nxnative/issues/276)); their notes keep the pair counts and validation.
|
||||||
- **Check:** check all pairs offline after a build (recipe in [05](../agents/05-debugging.md)).
|
- **Check:** `dm/gfx/tests/test_dmsh_pairs.py` covers both table locations, union/deduplication, defaults and cache dependencies; Metal compiles the generated sources for Mac and iOS and builds an archive with regular and pulling vertex entries in both depth modes. `test_nsp_mac.py` checks fallback invalidation and progress ordering; `test_nsp_incremental.py` regenerates a missing fallback. Check all pairs offline after a game build (recipe in [05](../agents/05-debugging.md)).
|
||||||
|
|
||||||
### Single-stage programs paired with a cover
|
### Single-stage programs paired with a cover
|
||||||
|
|
||||||
|
|||||||
@@ -1355,7 +1355,8 @@ def split_shaders(py, repo, roms, mains, pkg, env, logs, g, provisional=False, o
|
|||||||
PROG.set(step='building the Mac host for the game\'s Oodle decoder'); env = dict(env, **oodle())
|
PROG.set(step='building the Mac host for the game\'s Oodle decoder'); env = dict(env, **oodle())
|
||||||
inputs += [('files', [env['DM_OODLE_HOST']]), ('stat', Path(env['DM_OODLE_MODULES']))]
|
inputs += [('files', [env['DM_OODLE_HOST']]), ('stat', Path(env['DM_OODLE_MODULES']))]
|
||||||
key = None if ALLOW_DAMAGED else fingerprint(*inputs, ('files', profile_files(g)))
|
key = None if ALLOW_DAMAGED else fingerprint(*inputs, ('files', profile_files(g)))
|
||||||
if key and pkg.is_dir() and (note := reused(kf, key)) is not None:
|
have_pairs = g.table('pso_pairs.tsv').is_file() or pkg.with_name('pso_pairs.tsv').is_file()
|
||||||
|
if key and pkg.is_dir() and have_pairs and (note := reused(kf, key)) is not None:
|
||||||
PROG.end(f'{note}, reused (inputs unchanged)'); return package_key(pkg)
|
PROG.end(f'{note}, reused (inputs unchanged)'); return package_key(pkg)
|
||||||
PROG.set(step='splitting')
|
PROG.set(step='splitting')
|
||||||
tmp = pkg.with_name(pkg.name + '.partial'); rmtree(tmp); (tmp / 'tables').mkdir(parents=True)
|
tmp = pkg.with_name(pkg.name + '.partial'); rmtree(tmp); (tmp / 'tables').mkdir(parents=True)
|
||||||
@@ -1364,8 +1365,8 @@ def split_shaders(py, repo, roms, mains, pkg, env, logs, g, provisional=False, o
|
|||||||
m = re.search(r'shaders: (\d+) \(', (logs / 'shader_split.log').read_text(errors='replace'))
|
m = re.search(r'shaders: (\d+) \(', (logs / 'shader_split.log').read_text(errors='replace'))
|
||||||
if m and m.group(1) == '0': # dmsh_build.py would make no library, and the iOS shader step stop with no word of why
|
if m and m.group(1) == '0': # dmsh_build.py would make no library, and the iOS shader step stop with no word of why
|
||||||
raise BuildError('the shader split found no shaders in main or the RomFS: the game keeps them in a form shader_split.py does not read; locate them (docs/agents) and add a reader')
|
raise BuildError('the shader split found no shaders in main or the RomFS: the game keeps them in a form shader_split.py does not read; locate them (docs/agents) and add a reader')
|
||||||
# pso_pairs.tsv (the pairs of agl BAHS program variations, which only the pipeline prebuild reads): checked like the others where the profile has one; else kept beside the
|
# pso_pairs.tsv (agl BAHS program variations): checked like the others where the profile has one; else kept beside the
|
||||||
# package (pso_pairs_path), so a profile made before the table existed still packs and its prebuild still gets the pairs
|
# package, where dmsh_build.py reads it too, so padding and the pipeline prebuild use the same pairs
|
||||||
if provisional or g.table('pso_pairs.tsv').is_file(): names += ('pso_pairs.tsv',)
|
if provisional or g.table('pso_pairs.tsv').is_file(): names += ('pso_pairs.tsv',)
|
||||||
else: shutil.copyfile(tmp / 'tables' / 'pso_pairs.tsv', pkg.with_name('pso_pairs.tsv'))
|
else: shutil.copyfile(tmp / 'tables' / 'pso_pairs.tsv', pkg.with_name('pso_pairs.tsv'))
|
||||||
if provisional: # a provisional profile takes the tables as they come
|
if provisional: # a provisional profile takes the tables as they come
|
||||||
@@ -1388,12 +1389,15 @@ class ShaderWatch:
|
|||||||
the cache check (a hit restores <key>_b.json into the library directory b-<chain>/: the iOS one, or the Mac's for a Mac-only build; a miss writes <key>.bin into its temporary folder dmsh_* under WORK),
|
the cache check (a hit restores <key>_b.json into the library directory b-<chain>/: the iOS one, or the Mac's for a Mac-only build; a miss writes <key>.bin into its temporary folder dmsh_* under WORK),
|
||||||
the translation of the misses (<key>.spv in the temporary folder), and the compile, which takes the keys in a fixed order (sorted, the vertex shaders with pixel pairs last)
|
the translation of the misses (<key>.spv in the temporary folder), and the compile, which takes the keys in a fixed order (sorted, the vertex shaders with pixel pairs last)
|
||||||
and writes <key>_b.json again for every miss and every pixel shader: the furthest key written so far tells how far it got"""
|
and writes <key>_b.json again for every miss and every pixel shader: the furthest key written so far tells how far it got"""
|
||||||
def __init__(self, g, out, tmp): # out: the library directory b-<chain>/ (the iOS one, or the Mac's for a Mac-only build)
|
def __init__(self, g, out, tmp, pso_pairs=None): # out: the library directory b-<chain>/ (the iOS one, or the Mac's for a Mac-only build)
|
||||||
keys = set()
|
keys = set()
|
||||||
for ln in open(g.table('shader_map.tsv'), errors='replace'): # the keys dmsh_build.py builds: column 4 of the rows from a shader package
|
for ln in g.table('shader_map.tsv').read_text(errors='replace').splitlines(): # the keys dmsh_build.py builds: column 4 of the rows from a shader package
|
||||||
c = ln.rstrip('\n').split('\t')
|
c = ln.rstrip('\n').split('\t')
|
||||||
if len(c) > 4 and c[0].endswith('.bin'): keys.add(c[3])
|
if len(c) > 4 and c[0].endswith('.bin'): keys.add(c[3])
|
||||||
pairs = {ln.split()[0] for ln in open(g.table('shader_pairs.tsv'), errors='replace') if ln.strip() and not ln.startswith('#')}
|
pairs = set()
|
||||||
|
for path in (g.table('shader_pairs.tsv'), pso_pairs):
|
||||||
|
if path and path.is_file():
|
||||||
|
pairs |= {ln.split()[0] for ln in path.read_text(errors='replace').splitlines() if ln.strip() and not ln.startswith('#')}
|
||||||
order = sorted(keys - pairs) + sorted(keys & pairs)
|
order = sorted(keys - pairs) + sorted(keys & pairs)
|
||||||
self.pos, self.n = {k: i for i, k in enumerate(order)}, len(order)
|
self.pos, self.n = {k: i for i, k in enumerate(order)}, len(order)
|
||||||
self.out, self.tmp, self.t0, self.t1, self.w = out, tmp, time.time(), None, None
|
self.out, self.tmp, self.t0, self.t1, self.w = out, tmp, time.time(), None, None
|
||||||
@@ -1443,9 +1447,11 @@ def build_shaders(py, repo, sh, xlate, env, logs, g, pkg_key=None, mac=False):
|
|||||||
chain = chain_of(repo)
|
chain = chain_of(repo)
|
||||||
out = sh / 'cache' if mac else ios
|
out = sh / 'cache' if mac else ios
|
||||||
libs, kf, kmac = out / f'b-{chain}', ios / f'b-{chain}.key', sh / f'mac-b-{chain}.key'
|
libs, kf, kmac = out / f'b-{chain}', ios / f'b-{chain}.key', sh / f'mac-b-{chain}.key'
|
||||||
|
pso_pairs = g.table('pso_pairs.tsv')
|
||||||
|
if not pso_pairs.is_file(): pso_pairs = sh / 'pso_pairs.tsv' # the split's copy is a shader input too (vs_pad)
|
||||||
if pkg_key:
|
if pkg_key:
|
||||||
PROG.set(step='checking the inputs')
|
PROG.set(step='checking the inputs')
|
||||||
key = fingerprint(pkg_key, ('files', chain_files(repo)), ('stat', xlate.parent), ('files', profile_files(g)), *toolchain(), str(xlate),
|
key = fingerprint(pkg_key, ('files', chain_files(repo)), ('stat', xlate.parent), ('files', profile_files(g)), ('files', [pso_pairs]), *toolchain(), str(xlate),
|
||||||
{k: env.get(k) for k in ('XLATE_DEBUG', 'XLATE_REDUCED')}, env=env)
|
{k: env.get(k) for k in ('XLATE_DEBUG', 'XLATE_REDUCED')}, env=env)
|
||||||
note = next((n for k in ((kmac, kf) if mac else (kf,)) if (n := reused(k, key)) is not None), None)
|
note = next((n for k in ((kmac, kf) if mac else (kf,)) if (n := reused(k, key)) is not None), None)
|
||||||
# the libraries are the last build's, so are tables that name their stamp; a missing or stale one is made from that build's Mac outputs, and without them the step runs again
|
# the libraries are the last build's, so are tables that name their stamp; a missing or stale one is made from that build's Mac outputs, and without them the step runs again
|
||||||
@@ -1454,7 +1460,7 @@ def build_shaders(py, repo, sh, xlate, env, logs, g, pkg_key=None, mac=False):
|
|||||||
else: key = None
|
else: key = None
|
||||||
rmtree(kmac)
|
rmtree(kmac)
|
||||||
if not mac: rmtree(kf)
|
if not mac: rmtree(kf)
|
||||||
try: PROG.watch(ShaderWatch(g, libs, Path(env['WORK'])))
|
try: PROG.watch(ShaderWatch(g, libs, Path(env['WORK']), pso_pairs))
|
||||||
except Exception: pass # progress never stops the build (a missing table is dmsh_build.py's to report)
|
except Exception: pass # progress never stops the build (a missing table is dmsh_build.py's to report)
|
||||||
benv = dict(env, XLATE=str(xlate), IOSOUT=str(ios))
|
benv = dict(env, XLATE=str(xlate), IOSOUT=str(ios))
|
||||||
if mac: del benv['IOSOUT'] # dmsh_build.py builds the iOS libraries only with IOSOUT
|
if mac: del benv['IOSOUT'] # dmsh_build.py builds the iOS libraries only with IOSOUT
|
||||||
|
|||||||
Reference in new issue
Block a user