Check translation chain changes on games their author does not own (#272)

One translation chain serves every game, so a translator or host change was checked
only on its author's games; on another game the first sign was a crash at run time
(c3a0880 broke the image atomics of SHIN MEGAMI TENSEI V, Yoshi's Crafted World and
FINAL FANTASY RESONANCE).

- CI: the tool-tests job installs .NET 10, spirv-cross and spirv-tools and points
  DMSH_TEST_XLATE at the committed translator, so the translator's synthetic-shader
  tests (test_image_atomic_bounds, test_dmsh_geometry) run on every change.
  test_dmsh_geometry imports its sibling module when run from the repository root.
- dm/gfx/tools/shader_regressions.py: lists the shaders that translated in the
  game's previous b-*/ and fail in the current chain's, from dmsh_build.py's build
  reports (not part of the chain). The NSP script runs it after its shader step and
  stops on a regression unless ALLOW_SHADER_REGRESSIONS / --allow-shader-regressions;
  docs/development.md describes it.
- FINAL FANTASY RESONANCE: the c3a0880 regression is resolved since 2e655b7 (image
  atomics with a used result): chain 39e4d9585b1d builds 73,488 of 73,496 shaders
  (the 8 3D image atomics as before), shader_regressions.py finds 0 against the
  earlier build, and a new game from an empty save plays the opening cutscene on
  the Mac with no red line; the two shaders that failed and aborted run their
  atomics on their buffer textures.

Checked: every CI test file as the job runs it (DMSH_TEST_XLATE set), the docs tests.
This commit is contained in:
seedds committed 2026-10-26 23:43:09 +08:00
1 parent 99d0aae38f
commit b117c91317
7 files changed
+200 -1

No files matched your search

+11
View File
@@ -86,9 +86,20 @@ jobs:
cache-dependency-path: .github/workflows/tests.yml # no requirements file: the packages are listed in the step below
- name: Install the Python packages the tools import
run: python -m pip install lz4 capstone fonttools # backports.zstd is for Python before 3.14 only
# The shader translator's tests (test_image_atomic_bounds, test_dmsh_geometry) run the committed translator on synthetic Maxwell shaders, then spirv-val and spirv-cross:
# one translation chain serves every game, so these are what checks a translator change on games its author does not own (GitHub issue #272). Compiling their MSL with
# Metal (DMSH_TEST_METAL=1) and running it on the GPU (DMSH_TEST_GPU=1) stay local.
- uses: actions/setup-dotnet@a98b56852c35b8e3190ac28c8c2271da59106c68 # v6.0.0
with:
dotnet-version: '10.0.x' # dm/gfx/tools/dmsh_xlate_bin targets net10.0
- name: Install spirv-cross and spirv-val
env:
HOMEBREW_NO_AUTO_UPDATE: 1
run: brew install spirv-cross spirv-tools
- name: Run the test files
env:
DM_GAME: ${{ env.GAME }} # test_tbdr_budget runs a tool that needs a profile
DMSH_TEST_XLATE: ${{ github.workspace }}/dm/gfx/tools/dmsh_xlate_bin/dmsh_xlate
run: |
mkdir -p logs
failed=0
+1 -1
View File
@@ -73,4 +73,4 @@ General host changes first needed here; each applies to every game:
- **Known:** once in two runs, the frames that begin the new game's load bind 2D textures where a base pass shader declares 3D ones (red `texture type mismatch` lines, under a black screen).
- 11 shaders do not build (3 geometry, 8 compute with 3D image atomics); none met so far.
- **Open, since the image-atomic guards of `c3a0880` (2026-10-25):** 842 more compute shaders do not translate (`MSL 2.3 image atomics require IGN with a discarded result`; `41b20b5f09269ad8` is dispatched in the opening), and a new game aborts the host at the opening cutscene (`FATAL [RED] image atomics: unsupported or overflowing IMG extent`, compute shader `f3c5ced35cb619d0`, image slot 2, texture ID 2142). Found on 2026-10-26, cause not yet investigated.
- **Resolved: the image-atomic guards of `c3a0880` (2026-10-25) broke the opening.** 842 more compute shaders did not translate (`MSL 2.3 image atomics require IGN with a discarded result`; `41b20b5f09269ad8` is dispatched in the opening), and a new game aborted the host at the opening cutscene (`FATAL [RED] image atomics: unsupported or overflowing IMG extent`, compute shader `f3c5ced35cb619d0`, image slot 2, texture ID 2142). Since `2e655b7` translates image atomics whose result is used, chain `39e4d9585b1d` builds all but the 8 listed above, and a new game from an empty save plays the opening cutscene: both shaders run their atomics directly on their buffer textures (target 10, 8 KB). Checked on the Mac on 2026-10-26.
+1
View File
@@ -189,6 +189,7 @@ class GeometryOutputLayout(unittest.TestCase):
@unittest.skipUnless(os.environ.get('DMSH_TEST_XLATE'), 'set DMSH_TEST_XLATE to the offline translator')
class PassthroughCompiler(unittest.TestCase):
def test_passthrough_emits_positions_and_generic_attributes(self):
sys.path.insert(0, str(Path(__file__).resolve().parent)) # a sibling module: the CI runs each test file from the repository root
from test_dmsh_reflect import REFLECT
root = Path(__file__).resolve().parents[3]
table = (root / 'dm/third_party/ryujinx/src/Ryujinx.Graphics.Shader/Decoders/InstTable.cs').read_text()
+57
View File
@@ -0,0 +1,57 @@
#!/usr/bin/env python3
# shader_regressions.py <library dir> [<chain version>]: the shaders that translated in the game's previous build and fail in this one. The translation chain is one for every game,
# so a translator or host change is checked only on the games its author owns; on another game the first sign was a crash at run time (GitHub issue #272: c3a0880 broke the image
# atomics of SHIN MEGAMI TENSEI V and Yoshi's Crafted World). This compares <library dir>/b-<V>/_b_build.json (default: the current chain, dmsh_chain.py) with the newest other
# b-*/_b_build.json in the same folder, skipping one stamped with another game. It reads only the build reports dmsh_build.py writes, so it is not part of the chain.
# The NSP script runs it after each shader build and stops on a regression unless told to go on (ALLOW_SHADER_REGRESSIONS / --allow-shader-regressions).
# Exit status: 0 = none, or no earlier build to compare with; 1 = regressions, one [RED] line each; 2 = no report for this chain.
import json
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import dmsh_chain
REPORT = '_b_build.json'
def load(path):
with open(path) as f: return json.load(f)
def previous(lib_dir, chain):
"""The newest b-*/ beside b-<chain>/ with a build report, of the same game when both are stamped; None when there is none"""
game = dmsh_chain.read_stamp(os.path.join(lib_dir, 'b-' + chain)).get('game')
found = []
for name in os.listdir(lib_dir):
report = os.path.join(lib_dir, name, REPORT)
if not name.startswith('b-') or name == 'b-' + chain or not os.path.isfile(report): continue
other = dmsh_chain.read_stamp(os.path.join(lib_dir, name)).get('game')
if game and other and other != game: continue
found.append((os.path.getmtime(report), name))
return max(found)[1] if found else None
def compare(lib_dir, chain):
"""(previous b-* name or None, [(key, status, reason)] for the shaders that were ok there and are not now). Keys missing from either build are new or not rebuilt: not counted"""
now = load(os.path.join(lib_dir, 'b-' + chain, REPORT))
prev = previous(lib_dir, chain)
if prev is None: return None, []
was_ok = {r['key'] for r in load(os.path.join(lib_dir, prev, REPORT)) if r['st'] == 'ok'}
return prev, [(r['key'], r['st'], r['err']) for r in now if r['st'] != 'ok' and r['key'] in was_ok]
def main(argv):
if len(argv) not in (2, 3): sys.exit('usage: shader_regressions.py <library dir> [<chain version>]')
lib_dir, chain = argv[1], argv[2] if len(argv) == 3 else dmsh_chain.version()
if not os.path.isfile(os.path.join(lib_dir, 'b-' + chain, REPORT)):
print(f'[RED] no build report {os.path.join(lib_dir, "b-" + chain, REPORT)}: dmsh_build.py has not built chain {chain} there'); return 2
prev, lost = compare(lib_dir, chain)
if prev is None:
print(f'shader regressions: no earlier build in {lib_dir} to compare b-{chain} with'); return 0
print(f'shader regressions against {prev}: {len(lost)}')
for key, st, err in lost: print(f'[RED] {key}: translated in {prev}, {st} in b-{chain}: {err}')
return 1 if lost else 0
if __name__ == '__main__': sys.exit(main(sys.argv))
+104
View File
@@ -0,0 +1,104 @@
"""Shader regressions between two builds of a game's shader libraries (dm/gfx/tools/shader_regressions.py) and the NSP script's stop on them (build_ipa_from_nsp.py
shader_regressions; GitHub issue #272): a shader that translated in the newest other b-*/ of the same game and fails now is listed, new keys and shaders that failed before are not,
another game's stamped library is never the previous build, and the NSP script stops unless ALLOW_SHADER_REGRESSIONS. Build reports only: no game files, translator or Metal needed."""
import importlib.util
import json
import os
from pathlib import Path
import shutil
import subprocess
import sys
import tempfile
import unittest
from unittest import mock
REPO = Path(__file__).resolve().parents[3]
TOOL = REPO / 'dm/gfx/tools/shader_regressions.py'
spec = importlib.util.spec_from_file_location('shader_regressions', TOOL)
sr = importlib.util.module_from_spec(spec); spec.loader.exec_module(sr)
spec = importlib.util.spec_from_file_location('build_ipa_from_nsp', REPO / 'scripts/build_ipa_from_nsp.py')
nsp = importlib.util.module_from_spec(spec); spec.loader.exec_module(nsp)
FAIL = 'NotSupportedException: MSL 2.3 image atomics require IGN with a discarded result'
class Base(unittest.TestCase):
def setUp(self):
self.lib = Path(tempfile.mkdtemp()); self.addCleanup(shutil.rmtree, self.lib)
self.mtime = 1_000_000
def build(self, chain, rows, game=None): # a library folder with dmsh_build.py's report; each later call is newer
d = self.lib / ('b-' + chain); d.mkdir()
(d / sr.REPORT).write_text(json.dumps([{'key': k, 'st': st, 'err': err} for k, st, err in rows]))
if game: (d / 'inputs.tsv').write_text(f'game\t{game}\ninputs\t0123456789ab\n')
self.mtime += 10; os.utime(d / sr.REPORT, (self.mtime, self.mtime))
def tool(self, chain):
return subprocess.run([sys.executable, str(TOOL), str(self.lib), chain], capture_output=True, text=True)
class Compare(Base):
def test_lists_only_shaders_that_were_ok(self):
self.build('old', [('a', 'ok', ''), ('b', 'ok', ''), ('c', 'translation failed', 'x')])
self.build('new', [('a', 'ok', ''), ('b', 'translation failed', FAIL), ('c', 'translation failed', 'x'), ('d', 'metal failed', 'y')])
self.assertEqual(sr.compare(str(self.lib), 'new'), ('b-old', [('b', 'translation failed', FAIL)]))
r = self.tool('new')
self.assertEqual(r.returncode, 1)
self.assertEqual(r.stdout.splitlines(), ['shader regressions against b-old: 1', f'[RED] b: translated in b-old, translation failed in b-new: {FAIL}'])
def test_newest_other_build(self):
self.build('older', [('a', 'ok', '')])
self.build('old', [('a', 'translation failed', 'x')])
self.build('new', [('a', 'translation failed', 'x')])
self.assertEqual(sr.compare(str(self.lib), 'new'), ('b-old', []))
self.assertEqual(self.tool('new').returncode, 0)
def test_other_game_skipped(self): # one chain version serves every game: only the stamp tells the games apart
self.build('mine', [('a', 'ok', '')], game='010069C01AB82000')
self.build('theirs', [('a', 'ok', '')], game='01006000040C2000')
self.build('new', [('a', 'translation failed', FAIL)], game='010069C01AB82000')
self.assertEqual(sr.compare(str(self.lib), 'new')[0], 'b-mine')
def test_nothing_to_compare(self):
self.build('new', [('a', 'translation failed', FAIL)])
(self.lib / 'cas').mkdir(); (self.lib / 'b-empty').mkdir() # dmsh_build.py's compile cache, a folder without a report
r = self.tool('new')
self.assertEqual(r.returncode, 0, r.stdout)
self.assertIn('no earlier build', r.stdout)
def test_no_report(self):
self.assertEqual(self.tool('new').returncode, 2)
class NspScript(Base):
def setUp(self):
super().setUp()
self.logs = self.lib / 'logs'; self.logs.mkdir()
self.said = []
for cm in (mock.patch.object(nsp, 'PROG', mock.Mock()), mock.patch.object(nsp, 'say', self.said.append)):
cm.__enter__(); self.addCleanup(cm.__exit__, None, None, None)
self.build('old', [('a', 'ok', ''), ('b', 'ok', '')])
def check(self):
nsp.shader_regressions(sys.executable, REPO, self.lib, 'new', self.logs)
def test_stops(self):
self.build('new', [('a', 'ok', ''), ('b', 'translation failed', FAIL)])
with mock.patch.object(nsp, 'ALLOW_SHADER_REGRESSIONS', False), self.assertRaises(nsp.BuildError) as e:
self.check()
self.assertIn('1 shaders that translated in b-old fail in b-new', str(e.exception))
self.assertIn('[RED] b:', (self.logs / 'shader_regressions.log').read_text())
def test_allowed(self):
self.build('new', [('a', 'ok', ''), ('b', 'translation failed', FAIL)])
with mock.patch.object(nsp, 'ALLOW_SHADER_REGRESSIONS', True):
self.check()
self.assertTrue(any(s.startswith('[RED] 1 shaders') for s in self.said), self.said)
def test_none(self):
self.build('new', [('a', 'ok', ''), ('b', 'ok', '')])
self.check()
self.assertEqual(self.said, [])
if __name__ == '__main__': unittest.main()
+2
View File
@@ -127,6 +127,7 @@ What the script does:
The Vulkan and GL captures also run on a Mac whose GPU has no Metal 4, such as a virtual Mac (GitHub's `macos-26` runner). Those two layers draw on classic Metal queues of their own, so there the NVN layer starts only its classic part and drops every present, which a headless capture does not need (`nvn_mtl.m` `mtl_device_classic`; `DM_TEST_NO_METAL4=1` does the same on any Mac). The GLSLC capture boots an NVN game, which needs Metal 4.
5. It splits out the shaders and checks them against the profile's tables (a Vulkan profile: it translates the RomFS's SPIR-V and the capture with `vk_shaders.py` instead; an OpenGL one: the GLSL sources and the capture with `gl_shaders.py`).
6. It builds the iOS shader libraries, with the Mac's own Metal compiler and spirv-cross when they differ from those in `dmsh_chain.lock` (`dm/gfx/tools/own_compilers.py`; one line in the output names them). If a registered profile's `dmsh_chain.h` does not hold the version the source tree gives (a chain change left it behind), the script writes it again before any host is built, so it shows as changed in the checkout (`dmsh_build.py` would stop otherwise). Then the script makes the profile's `c1_full.tsv` and `cb_extent.tsv` from this build's Mac outputs when they are missing or name another chain. These two tables are not committed and git ignores them; `build_ipa.sh` stops without a current `c1_full.tsv`. Each line in the output names the file.
It then compares the build with the game's previous one, the newest other `b-*/` in `<id>_build/shader/cache/` (`dm/gfx/tools/shader_regressions.py`, which reads the two `_b_build.json` reports). The translation chain is one for every game, so a translator change is tested only on the games its author owns; shaders that translated before and fail now are listed in `logs/shader_regressions.log` and stop the build ([GitHub issue #272](https://github.com/seedds/nxnative/issues/272)). Report them; `--allow-shader-regressions` (`ALLOW_SHADER_REGRESSIONS = True` in Spyder) packs the app without them.
7. It calls `build_ipa.sh`. Xcode builds the app in the game's own build folder (`<id>_build/ipa/build/ios/pkg/dd`: `build_ipa.sh` sets `SYMROOT` and `OBJROOT`, which a custom build location in Xcode's settings does not override), so a repeat build compiles only what changed and two games built at once never write the same app. Then `dm/ios/tools/pack_ipa.py` packs it with the shader libraries, the add-on content and the RomFS straight from where they are. It reads and deflates files on all cores and writes at about the T7's write speed (300 MB/s): Breath of the Wild's 16 GB IPA takes 57 s, Info-ZIP's copy and zip took 2 min 50 s.
A repeat build skips work whose inputs have not changed. The extraction is kept (`extract/`). The DLC, the rewrite, the shader split and the iOS (or Vulkan) shaders each write a key beside their output: a sha256 of their inputs (the game files, the tools, the profile, Xcode's and spirv-cross's versions; for the DLC, the DLC NSPs' names, sizes and times and nx_extract's sources). When the key matches, the stage line says `reused (inputs unchanged)` and the step does not run. Any change to a tool or the profile runs the step again; deleting the `.key` file forces it. The iOS shaders' inputs are the shader package's content and only the tools `dmsh_build.py` loads (the modules it imports, at any depth): a split that runs again because a reader for another engine changed (a pull that touches `shader_split.py` or `unityfs.py`) writes the same package, and the iOS shaders stay reused.
@@ -293,6 +294,7 @@ GAME=$G W=$W dm/ios/tools/build_ipa.sh [out.ipa]
Notes on these steps:
- `dmsh_build.py` stops when a tool differs from `dm/gfx/tools/dmsh_chain.lock`. With another Xcode or spirv-cross, run `dm/gfx/tools/own_compilers.py` with the same arguments and environment instead: it builds with the local compilers, and only the translator must match.
- After a chain change, `python3 dm/gfx/tools/shader_regressions.py $W/build/cache/metal_shader_cache` lists the shaders that translated in the previous `b-*/` there and fail in the current one (exit status 1), as the NSP script does after its shader step.
- `dmsh_build.py` compiles each shader's Metal source through a cache keyed by that source, the compile flags and the compiler (`dm/gfx/tools/dmsh_mtlcache.py`, in `<cache>/cas/metallib/`). A chain change that leaves a shader's MSL unchanged, such as a translator fix, costs that shader no Metal compile. Before building, the cache takes the libraries of the earlier `b-*/` folders beside its output (and beside `IOSOUT`), and of `DMSH_PREFILL=<Mac root>[=<iOS root>]` (several separated by `:`). It takes a folder only when its recorded flags are the current ones and a fresh compile of a few of its shaders reproduces their libraries byte for byte. The summary line `Metal cache:` shows reused, compiled and filled counts per target.
- `build_ipa.sh` takes the rewritten modules from `$W/build/patched_<package>` and the RomFS from `$W/data/<package>/RomFS`; `PATCHED` and `ROMFS` override them. In unsigned IPA mode, `pack_ipa.py` packs the shader libraries, RomFS and add-on content beside the app Xcode built, as `DmIOS.app/msl/b-<chain version>`, `romfs` and `aoc`. Prepared Xcode projects instead sync these resources into the app before signing.
- It also bundles a standard font (macOS's Arial Unicode, `STD_FONT=` overrides it) and the generated `nintendo_ext.ttf`.
+24
View File
@@ -56,6 +56,8 @@ SHOW_FPS = True # True shows the game's frame rate on th
SURVEY_ONLY = False # a game with no profile in dm/games: only write the survey, no provisional profile and no IPA
PROFILE_OUT = '' # where a provisional profile goes (empty = <NSP folder>/<title ID>_build/profile; the profile is <this>/<title ID>/)
ALLOW_DAMAGED = False # a damaged NSP (files failing the NCA's hash check: an incomplete copy or a broken download) stops the build; True writes those files as read and goes on
ALLOW_SHADER_REGRESSIONS = False # shaders that translated in this game's previous build and fail with the current translation chain stop the build (logs/shader_regressions.log lists them);
# True packs the app without them
PROVISIONAL = False # build from a provisional profile even when dm/games has one for the game (to test the guesses; the profile goes to PROFILE_OUT / <id>_build/profile)
HEAP_CHECK = True # a provisional profile whose game has virtual address memory and sets a heap: boot it on the Mac until its heap request (a few seconds) to size the heap region
HEAP_CHECK_SECS = 30 # the longest heap check (it stops at the first heap request, or HEAP_CHECK_NNMAIN_SECS into nnMain without one)
@@ -1418,11 +1420,30 @@ def build_shaders(py, repo, sh, xlate, env, logs, g, pkg_key=None, mac=False):
note = f'{ok:,} of {total:,} shaders ready{failed}'
PROG.end(f'{ok:,} of {total:,} shaders ready' + (f' ({int(c.group(1)):,} from the cache)' if c else '') + failed)
else: PROG.end()
shader_regressions(py, repo, sh / 'cache', chain, logs) # before the key: a stopped build is checked again on the next run
if key:
for k in (kmac,) if mac else (kf, kmac): k.write_text(f'{key}\n{note}\n')
return out
def shader_regressions(py, repo, lib_dir, chain, logs):
"""dm/gfx/tools/shader_regressions.py on the Mac libraries (dmsh_build.py writes its build report there in an iOS build too): shaders that translated in this game's previous build
and fail with this chain stop the build, unless ALLOW_SHADER_REGRESSIONS. The chain is one for every game, so a change can break a game its author never ran (GitHub issue #272)"""
log = logs / 'shader_regressions.log'
if not (lib_dir / f'b-{chain}' / '_b_build.json').is_file(): return # a full dmsh_build.py run always writes it
r = subprocess.run([py, str(repo / 'dm/gfx/tools/shader_regressions.py'), str(lib_dir), chain], capture_output=True, text=True, errors='replace')
log.write_text(r.stdout + r.stderr)
if r.returncode == 0: return
if r.returncode != 1: raise BuildError(f'shader_regressions.py exit code {r.returncode} (full output: {log})')
lines = r.stdout.splitlines()
prev = re.search(r'against (b-\w+)', r.stdout)
msg = f'{len(lines) - 1} shaders that translated in {prev.group(1) if prev else "the previous build"} fail in b-{chain}: a change to the translation chain broke them'
PROG.say('\n'.join(lines[1:11]) + (f'\n... {len(lines) - 11} more in {log}' if len(lines) > 11 else ''))
if ALLOW_SHADER_REGRESSIONS: say(f'[RED] {msg}; going on (ALLOW_SHADER_REGRESSIONS)'); return
raise BuildError(f'{msg} (list: {log}). Please report it at https://github.com/seedds/nxnative/issues with that list; '
'ALLOW_SHADER_REGRESSIONS = True (--allow-shader-regressions) packs the app without those shaders')
def layer_shaders(py, repo, api, name, roots, sh, key, logs, env, mac):
"""What build_vk_shaders and build_gl_shaders share: dm/gfx/tools/<api>_shaders.py on the roots → sh/<api>/ for the Mac and, unless mac (a Mac-only build), sh/ios/<api>/ for the iPhone.
Returns the folder the host (DM_MSL_DIR) or build_ipa.sh (MSL_IOS) takes the libraries from. key: the inputs' fingerprint (None never reuses). The iOS libraries' key is sh/ios/<api>.key,
@@ -1673,6 +1694,8 @@ def parse_args(argv):
ap.add_argument('--show-fps', action=argparse.BooleanOptionalAction, help='show the game\'s frame rate on the iPhone, frames on screen per second at the top left (default: the SHOW_FPS setting, on)')
ap.add_argument('--survey-only', action='store_true', help='a game with no profile: only the survey, no provisional profile and no IPA')
ap.add_argument('--allow-damaged', action='store_true', help='a damaged NSP (files failing the NCA hash check): write them as read and go on instead of stopping')
ap.add_argument('--allow-shader-regressions', action='store_true', help='shaders that translated in the game\'s previous build and fail with the current translation chain: '
'pack the app without them instead of stopping')
ap.add_argument('--profile-out', help='where the provisional profile goes (default: <NSP folder>/<title ID>_build/profile)')
ap.add_argument('--provisional', action='store_true', help='build from a provisional profile even when dm/games has one for the game (tests the guesses)')
ap.add_argument('--no-heap-check', action='store_true', help='a provisional profile: skip the heap check on the Mac (the alias / heap split stays a guess)')
@@ -1702,6 +1725,7 @@ def main(args=None):
args = args or parse_args([])
PROG.mode = progress_mode(args.progress or PROGRESS)
global ALLOW_DAMAGED; ALLOW_DAMAGED = args.allow_damaged or ALLOW_DAMAGED
global ALLOW_SHADER_REGRESSIONS; ALLOW_SHADER_REGRESSIONS = args.allow_shader_regressions or ALLOW_SHADER_REGRESSIONS
here = Path(globals().get('__file__') or sys.argv[0]).resolve()
repo = find_repo(here)
prepare_xcode = args.prepare_xcode or PREPARE_XCODE