diff --git a/dm/games/0100000000010000/README.md b/dm/games/0100000000010000/README.md index 8accf52..98ec4ec 100644 --- a/dm/games/0100000000010000/README.md +++ b/dm/games/0100000000010000/README.md @@ -41,7 +41,8 @@ No `patches.py`. `op_mode` stays docked: Odyssey only takes its motion moves in ## Files and shaders -- **Resolved in the shader builder (#276): BAHS pipeline interfaces.** The local 1.4.1 libraries have 290 pairs with missing vertex outputs, involving 29 vertex shaders. All are among the split's 12,124 BAHS pairs; the profile's 20,740 other pairs have no missing outputs. The shared fix makes padding read both tables; see [Pixel shader inputs the vertex shader does not write](../../../docs/reference/shaders.md#pixel-shader-inputs-the-vertex-shader-does-not-write). The report's additional `Max Unstruct ID 758 ... exceeds supported limit` compiler error is separate and remains open; fixing missing outputs alone does not establish a successful full archive build. Targeted validation: the affected vertex sources compile for Mac and iOS, and all 290 pairs build into `g13s` and `g16s` Metal archives with both depth entry points (Metal 32023.921). Full Mac rebuild: 27,984 of 27,984 shaders successful, no binding or pulling-library failures, no regressions, and no interface mismatches across all 32,864 pairs. The Mac host also rebuilt with the regenerated uniform tables. No new gameplay or iPhone run. +- **Resolved in the shader builder (#276): BAHS pipeline interfaces.** The local 1.4.1 libraries have 290 pairs with missing vertex outputs, involving 29 vertex shaders. All are among the split's 12,124 BAHS pairs; the profile's 20,740 other pairs have no missing outputs. The shared fix makes padding read both tables; see [Pixel shader inputs the vertex shader does not write](../../../docs/reference/shaders.md#pixel-shader-inputs-the-vertex-shader-does-not-write). The report's `Max Unstruct ID 758 ... exceeds supported limit` lines did not fail the build: see the next item. Targeted validation: the affected vertex sources compile for Mac and iOS, and all 290 pairs build into `g13s` and `g16s` Metal archives with both depth entry points (Metal 32023.921). Full Mac rebuild: 27,984 of 27,984 shaders successful, no binding or pulling-library failures, no regressions, and no interface mismatches across all 32,864 pairs. The Mac host also rebuilt with the regenerated uniform tables. No new gameplay or iPhone run. +- **Resolved in the shader builder (#276): the 25-minute pipeline.** `Max Unstruct ID 758` is a note from Apple's GPU compiler, which then lowers the function a slower way; left to finish, `metal-tt` exits 0 with the pipeline in the archive. The report's exit 1 came from the BAHS link errors above. The function is the float body of `alRenderMaterialUber` (fragment `007d7e411ef14ebd`, BNSH variation 0, paired with vertex `3c3435a62e05975b`): alone on `applegpu_g13s` it took 24.6 minutes and 17.3 MB of code, which is also what held the reporter's startup prebuild at 7,008 of 32,864. With the front end at `-O1` ([Front end at -O1](../../../docs/reference/shaders.md#front-end-at--o1-apples-control-flow-lowering)) it takes 31 s and 1.3 MB; the note remains, at 642. Full Mac app pack with that chain (Metal 32023.921): 27,984 of 27,984 shaders in 32:50, the `g18g` archive in 10:50 with no note, and the reporter's `g13s` archive of 32,864 bases in 15.6 minutes (exit 0, 912 MB). From `save_new1`, frames 4,200 to 6,600 (every 300) against an `-O2` build of the same commit: six identical or within one pixel, frame 5,700 differs by the same 4,191 pixels between two `-O2` runs too, and frames 4,800 and 6,300 by 1,140 and 2,059 grass pixels of at most 3 of 255. GPU time per frame over the same frames (one in 150, three runs each, alternating order): medians 5.05, 4.79 and 4.76 ms at `-O1` (the 5.05 ms run was the first of the series), 4.76, 4.75 and 4.83 ms at `-O2`. No iPhone run. - 27,984 shaders: agl `sharcb` and `bfsha` in `ShaderData`, particles in `EffectData`; groups per RomFS folder in `game.tsv`. - The spin-wait rewrite leaves its modules byte for byte as before the lock and store rules. @@ -56,6 +57,7 @@ No `patches.py`. `op_mode` stays docked: Odyssey only takes its motion moves in General host and tool changes first needed here; each applies to every game: - **BAHS pairs omitted from vertex padding:** [Pixel shader inputs the vertex shader does not write](../../../docs/reference/shaders.md#pixel-shader-inputs-the-vertex-shader-does-not-write). +- **Front end at `-O1`** (`dmsh_build.py` and `ios_metal.py`): [Front end at -O1: Apple's control-flow lowering](../../../docs/reference/shaders.md#front-end-at--o1-apples-control-flow-lowering). - **Resolved: NSZ decompression rejected `--keys`** when packing v0: [NSZ decompression rejects `--keys`](../../../docs/reference/packing.md#nsz-decompression-rejects---keys). diff --git a/dm/gfx/nvn/dmsh_chain.h b/dm/gfx/nvn/dmsh_chain.h index 931c97a..02242cc 100644 --- a/dm/gfx/nvn/dmsh_chain.h +++ b/dm/gfx/nvn/dmsh_chain.h @@ -1,3 +1,3 @@ // Generated file (dm/gfx/tools/dmsh_chain.py --header): the B shader library translation chain version this source tree expects (default options), one for every game; // the host loads only /b-/. Run dmsh_chain.py --header after changing the chain's tools and commit this file with the change -#define DMSH_CHAIN "7fbca6dd477d" +#define DMSH_CHAIN "d374e783beec" diff --git a/dm/gfx/tools/dmsh_build.py b/dm/gfx/tools/dmsh_build.py index c887a21..7e78f4a 100644 --- a/dm/gfx/tools/dmsh_build.py +++ b/dm/gfx/tools/dmsh_build.py @@ -43,7 +43,10 @@ CAS_LOCKS, CAS_LOCKS_GUARD = {}, threading.Lock() PRE_STATS = collections.Counter() # -fpreserve-invariance: vertex positions are marked [[invariant]], but without this flag Metal does not promise that two different shaders compute the same expression to the same value; on Maxwell the same multiply-adds always agree. # Super Mario Odyssey draws a depth pre-pass with alRenderDepthOnly, then the G-buffer with another vertex shader and depth test less-or-equal: a hair off and the test failed, leaving those pixels with old content (Cap Kingdom's ground in 32x32 blocks) -MAC_METAL = ['xcrun', '-sdk', 'macosx', 'metal', '-c', '-fpreserve-invariance'] +# -O1: the front end's -O2 (CorrelatedValuePropagation, then SimplifyCFG) folds the translated shaders' repeated tests of one value into switches whose cases jump past their own join. +# Apple's GPU backend then exceeds its control-flow ID budget ("Max Unstruct ID 758 ... exceeds supported limit", a note, not an error) and falls back to a slower lowering: +# Odyssey's alRenderMaterialUber took 25 minutes and 17.3 MB of GPU code, 31 s and 1.3 MB at -O1. Ordinary shaders come out within a few percent either way (the backend optimizes again) +MAC_METAL = ['xcrun', '-sdk', 'macosx', 'metal', '-c', '-O1', '-fpreserve-invariance'] MAC_RECORD = ' '.join(MAC_METAL) + '; xcrun -sdk macosx metallib' def mac_compile(msl, air, lib): # same interface as ios_metal.compile subprocess.run([*MAC_METAL, msl, '-o', air], check=True) diff --git a/dm/ios/tools/ios_metal.py b/dm/ios/tools/ios_metal.py index 15d40da..f270b18 100644 --- a/dm/ios/tools/ios_metal.py +++ b/dm/ios/tools/ios_metal.py @@ -4,7 +4,8 @@ import subprocess # -fmetal-math-mode=relaxed: keep fast math's reassociation / contraction, but forbid assuming there are no NaN / Inf. The translated shaders reproduce Maxwell's NaN semantics with isnan selects (8337 / 12809 use them), # and the default fast mode lets each GPU's backend compiler treat them as always false; on the A20 Pro the opening course's odd frames had 727k NaN pixels in the main image (a few dozen with the same AIR on the M4 Pro), which became the black band / black-edged glow # -fpreserve-invariance: [[invariant]] position outputs only agree across shaders with this flag (as dmsh_build.py's MAC_METAL: a depth pre-pass followed by a less-or-equal G-buffer) -METAL = ['xcrun', '-sdk', 'iphoneos', 'metal', '-c', '-mios-version-min=18.0', '-fmetal-math-mode=relaxed', '-fpreserve-invariance'] +# -O1: -O2's switch folding leaves a control-flow graph Apple's GPU backend can only lower slowly into very large code (as dmsh_build.py's MAC_METAL) +METAL = ['xcrun', '-sdk', 'iphoneos', 'metal', '-c', '-O1', '-mios-version-min=18.0', '-fmetal-math-mode=relaxed', '-fpreserve-invariance'] METALLIB = ['xcrun', '-sdk', 'iphoneos', 'metallib'] RECORD = ' '.join(METAL) + '; ' + ' '.join(METALLIB) # Written into json chain.metal diff --git a/docs/agents/symptoms.md b/docs/agents/symptoms.md index a6e5c7b..ee29392 100644 --- a/docs/agents/symptoms.md +++ b/docs/agents/symptoms.md @@ -429,6 +429,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - `Fragment input(s) user(locnN) mismatching vertex shader output type(s) or not written by vertex shader` and the draw is skipped; the host reports `[RED] failed to create render pipeline (vertex … fragment …)`. Offline packing fails with `pso_offline: metal-tt failed` for the same reason. → [Pixel shader inputs the vertex shader does not write](../reference/shaders.md#pixel-shader-inputs-the-vertex-shader-does-not-write) - outdoors dim and nothing casts a sun shadow (every shadow-map lookup fails). → [Unwritten components of a written output start as (0, 0, 0, 1)](../reference/shaders.md#unwritten-components-of-a-written-output-start-as-0-0-0-1) - the ground kept stale 32×32 blocks where the G-buffer draw failed the depth test: blocks of a stale image where a depth pre-pass and a later pass with other vertex shaders meet (Odyssey: 32×32 blocks on every fourth frame). → [Position invariance across shaders](../reference/shaders.md#position-invariance-across-shaders) +- making a pipeline archive prints `Max Unstruct ID 758 in function agc.main in shader 13941 exceeds supported limit`, and one pipeline takes many minutes, in `metal-tt` offline or in the host's prebuild on the device (`Preparing pipelines` holding one count for minutes). The line is a note, not an error: `metal-tt` goes on, exits 0 and writes that pipeline into the archive. → [Front end at -O1: Apple's control-flow lowering](../reference/shaders.md#front-end-at--o1-apples-control-flow-lowering) - the half of Mario facing the camera's left was white (GitHub issue 72, on an iPhone). → [min, max and clamp with a NaN operand](../reference/shaders.md#min-max-and-clamp-with-a-nan-operand) - a pipeline red line for a float output meeting an integer attachment (the red line prints the row to add); a BC encoder's pipeline fails. → [Integer fragment outputs declared by hand](../reference/shaders.md#integer-fragment-outputs-declared-by-hand) - `failed to create render pipeline … output of type float4 is not compatible with a MTLPixelFormatR32Uint color attachment` (or `MTLPixelFormatRGBA16Uint`); every draw into the target skipped. → [Integer entry points for integer colour outputs](../reference/shaders.md#integer-entry-points-for-integer-colour-outputs) diff --git a/docs/reference/shaders.md b/docs/reference/shaders.md index 8fa2a3d..61d9308 100644 --- a/docs/reference/shaders.md +++ b/docs/reference/shaders.md @@ -382,6 +382,14 @@ Finding, keying and translating shaders (`dm/gfx/tools/shader_split.py`, `dmsh_b - **Fix:** the iOS libraries are compiled with `-fmetal-math-mode=relaxed` (`dm/ios/tools/ios_metal.py`; the Vulkan and GL libraries in `vk_shaders.py` and `gl_shaders.py` too). Relaxed keeps fast math's reassociation but not that assumption. - **Found on:** listed among the mechanisms first met in [Super Mario Bros. Wonder](../../dm/games/010015100B514000/README.md); the source gives the 8,337 of 12,809 count without naming the game. +### Front end at -O1: Apple's control-flow lowering + +- **Symptom:** making a pipeline archive prints `Max Unstruct ID 758 in function agc.main in shader 13941 exceeds supported limit`, and one pipeline takes many minutes, in `metal-tt` offline or in the host's prebuild on the device (`Preparing pipelines` holding one count for minutes). The line is a note, not an error: `metal-tt` goes on, exits 0 and writes that pipeline into the archive. +- **Cause:** at the front end's default `-O2`, `CorrelatedValuePropagation` and the `SimplifyCFG` after it fold a translated shader's repeated tests of one value into switches whose cases jump past their own join, so nested `if`s become an unstructured graph. Apple's GPU backend lowers unstructured control flow with per-function IDs; past its budget (255 on `applegpu_g13s`, the check in `libapplegpu-nt.dylib` of AIR-TT 32023.921 picks 255 or 65,535 by a capability not yet characterized), it prints the note and lowers the function another way that is far slower and larger. At `-O1` those two passes do not run; the backend still optimizes the AIR itself, so ordinary shaders come out the same. +- **Fix:** `dmsh_build.py` (Mac) and `ios_metal.py` (iOS) compile with `-O1`. +- **Found on:** [Super Mario Odyssey](../../dm/games/0100000000010000/README.md#files-and-shaders), [issue #276](https://github.com/seedds/nxnative/issues/276): the float body of `alRenderMaterialUber` (fragment `007d7e411ef14ebd`, 16,227 `if`s in one function) took 24.6 minutes and 17.3 MB of `g13s` code at `-O2`, 31 s and 1.3 MB at `-O1`. Its `-O1` graph still goes past the budget (642), at that small cost. +- **Check:** random samples of archive bases from seven games (Odyssey, Tears of the Kingdom, Breath of the Wild, Kirby, Luigi's Mansion 3, Wonder, Pikmin 4; 22 to 400 each) compile at `-O1` to vertex and fragment code within a few percent of `-O2` (Odyssey 400 bases: −0.9% / +0.01%; TotK: +0.1% / −1.0%; Kirby: −3.1% / +0.07%), with no `Max Unstruct ID` note at either level, and the front end is faster. Odyssey's full `g13s` archive of 32,864 bases builds in 15.6 minutes (exit 0); its Cap Kingdom frames 4,200 to 6,600 match `-O2` except for scattered differences of at most 3 of 255 in three of nine frames, and their GPU time per frame (`DM_NVN_GPUPROF=every:150`, three runs each on an M-series `g18g`) has medians of 4.76 to 5.05 ms at `-O1` and 4.75 to 4.83 ms at `-O2`, within run-to-run noise. + ### min, max and clamp with a NaN operand - **Symptom:** the half of Mario facing the camera's left was white (GitHub issue 72, on an iPhone).