NSZ and XCZ files in the NSP folder (from PR #140): the NSP script converts them itself instead of a second script. scan_nsps lists *.nsz and *.xcz beside *.nsp and *.xci (an NSZ is a PFS0 with the NSP's ticket, so title_of reads it as it is; a compressed file with an NSP of the same name beside it is skipped), and decompress_nsz converts each once with nsz -D, run from the Python running the script (Spyder does not have its bin folder on PATH), into <title ID>_build/nsp/, before the build ID lookup or the survey reads them. nsz writes into an empty temporary folder and the file is moved into nsp/ only when nsz exits 0 and made it, so a partial file left by a failure or an interruption is never taken for a whole one (with -w left out, nsz itself skips an output whose title and version are already there and exits 0); a conversion is reused while its source has the same name, size and modification time. nsz -D prints an NCA hash mismatch but goes on and exits 0 (Decompressor.decompress passes raiseVerificationException=False), so a damaged file is left to nx_extract's own hash check (--allow-damaged). Without the nsz package the script stops with one line: pip install nsz. README: one line in the packing steps; docs/development.md says how it works. AGENTS.md: only build_ipa_from_nsp.py has to run in Spyder with F5; the other scripts in scripts/ run from a terminal (the user). New test dm/tools/tests/test_nsp_nsz.py (nsz stood in for). Checked: the new test, test_nsp_alias, test_nsp_chain, test_nsp_mac, test_nsp_progress and test_xcode_workflow pass; ANIMAL WELL's base game and update compressed with nsz 5.0.0 -C, then the script on a folder holding only the two NSZ files: both converted (byte for byte the original NSPs), the IPA built (exit 0), and a second run reused both conversions. Not checked: an XCZ, the IPA on the iPhone

AGENTS.md

@@ -5,7 +5,7 @@
 - Prefer general optimizations and fixes to game-specific rules. Performance work (frame pacing, power, caches, load times) counts here as much as correctness. First look for a change that holds for every game: Horizon / NVN / Vulkan semantics, or what the binary, its SDK version, its imports or its files show. Use a game-specific rule (a profile key, a rewriter cut, a build-ID gate, a per-game branch in the host) only when it is absolutely necessary, that is, when no general rule can do it.
 - Per-build knowledge is keyed by the NSO build ID: rewriter cuts, observation gates and frame-step stubs in the profile's `patches.py` (a game's own modules) or `dm/tools/sdk_patches.py` (Nintendo's `sdk` modules, which several games ship), which `dm/tools/patch_nso.py` (the general rewriter) loads, the profile's `build` rows (the builds the game was brought up on, and their display versions; the NSP script packs any other version from a derived profile, without those builds' fixes), and host behaviour that uses fixed module addresses checks `module_t.build_id` (from `fixups.tsv`; `dm_main_build()`). When a new game build appears, carry addresses over with `dm/tools/nso_fnmap.py` and check them by disassembly; never let fixed addresses run on a build they were not found in.
 - To analyse or port **another** Switch title, first run `scripts/build_ipa_from_nsp.py` on its NSP folder. With no matching profile, it writes a survey to `<NSP folder>/<title ID>_survey/survey.md`, then a provisional profile guessed from the game files (`dm/games/profile_guess.py`; `<NSP folder>/<title ID>_build/profile/<title ID>/`, found by the tools through `DM_GAMES_DIR`), with a short heap check on the Mac when the game has virtual address memory, and a best-effort IPA that may not boot. Then follow [docs/agents](docs/agents/README.md) (extraction, binary survey, host bring-up failure catalogue, graphics specifics, debugging recipes) and add a profile.
-- Scripts the user runs live in `scripts/` at the repository root (now `build_ipa_from_nsp.py`). Each one runs both in Spyder (F5, configured by the settings block at its top) and from a terminal (command-line options override those settings; non-zero exit on error). Tools that only agents or other scripts call stay under `dm/`.
+- Scripts the user runs live in `scripts/` at the repository root. Each one runs from a terminal (non-zero exit on error). `build_ipa_from_nsp.py` also runs in Spyder (F5, configured by the settings block at its top; command-line options override those settings); the others need not. Tools that only agents or other scripts call stay under `dm/`.
 - Game files, keys and anything extracted from them live in the game's NSP folder, never in this repository and never in the home folder (it is nearly full): the work folder `<NSP folder>/<id>-work` for extraction, runs and analysis, beside the NSP script's `<id>_build/`. Tool venvs and other large scratch data go on that drive too. Do not print key values.
 - Language: everything is in English: `README.md`, docs, code comments, log messages, script output and other files. Red log lines (something unimplemented or wrong) start with `[RED] `. Text that must match a game's own Chinese data is written as escapes (`"\xE8\xB5\xB7"`), never as characters.
 - The shader translation chain has one version for every game, `DMSH_CHAIN` in `dm/gfx/nvn/dmsh_chain.h`, which the host is compiled with and loads only libraries of. It covers the chain's tools: `dm/gfx/tools/dmsh_*` (`dmsh_build.py`, `dmsh_chain.py`, `dmsh_fetch.py`, `dmsh_vin.py`, `dmsh_geometry.py`, `dmsh_facts.py`), `Program.cs`, `ios_metal.py` and `dmsh_chain.lock`. A profile's own inputs (`shader_map.tsv`, `shader_facts.tsv`, `shader_pairs.tsv`, `bf_want.tsv`, its `shader_*` / `nvn_*` keys) are not part of it: they are in each shader's cache key, and their digest stamps the libraries (`b-<V>/inputs.tsv`). The procedure:

README.md

@@ -148,6 +148,8 @@ python3 -m pip install lz4 capstone fonttools "backports.zstd; python_version <
   Super Mario Bros. Wonder [010015100B514800][v327680].nsp
 ```
 
+NSZ and XCZ files (compressed NSPs and XCIs) work too: the script converts them once with [nsz](https://github.com/nicoboss/nsz) (`python3 -m pip install nsz`) and keeps the NSPs in `<title ID>_build/nsp/`, which takes their full size on disk.
+
 Any version works, with or without an update: the script packs the newest version in the folder. DLC is optional too: the DLC in the folder is packed, and without it the game sees none, as on a Switch without it. Each game was tested on one version (an update named in the table above is the one its 60 fps needs). With another version the script builds the game with that version's own shaders and says that the version was not tested; fixes made for the tested version's code, such as the 60 fps ones, don't apply to it.
 
 **3. Run the script:**

dm/tools/tests/test_nsp_nsz.py

@@ -0,0 +1,79 @@
+"""NSP script: NSZ / XCZ files in the NSP folder are listed by their ticket like NSPs (scan_nsps) and converted once with nsz -D into <title ID>_build/nsp/ (decompress_nsz):
+a conversion is moved in only when nsz exits 0 and made the file, and reused while its source is unchanged. nsz itself is stood in for (no game files, keys or nsz needed)."""
+import importlib.util
+import os
+from pathlib import Path
+import shutil
+import struct
+import sys
+import tempfile
+import unittest
+from unittest import mock
+
+REPO = Path(__file__).resolve().parents[3]
+spec = importlib.util.spec_from_file_location('build_ipa_from_nsp', REPO / 'scripts/build_ipa_from_nsp.py')
+nsp = importlib.util.module_from_spec(spec); spec.loader.exec_module(nsp)
+
+APP = 0x010020D01AD24000
+
+
+def pfs0(path, names):   # a PFS0 holding empty files with these names (enough for title_of)
+    strs = b''.join(n.encode() + b'\0' for n in names); ent, o = b'', 0
+    for n in names: ent += struct.pack('<QQII', 0, 0, o, 0); o += len(n) + 1
+    path.write_bytes(b'PFS0' + struct.pack('<III', len(names), len(strs), 0) + ent + strs)
+
+
+class Nsz(unittest.TestCase):
+    def setUp(self):
+        self.tmp = Path(tempfile.mkdtemp()); self.addCleanup(shutil.rmtree, self.tmp)
+        self.base = self.tmp / 'Game [v0].nsz'; pfs0(self.base, [f'{APP:016x}0000000000000011.tik', 'a.ncz'])
+        self.upd = self.tmp / 'Game [v65536].nsz'; pfs0(self.upd, [f'{APP | 0x800:016x}0000000000000011.tik', 'b.ncz'])
+        self.runs, self.fail = [], False
+        for cm in (mock.patch.object(nsp, 'PROG', mock.Mock()), mock.patch.object(nsp, 'say', lambda m: None), mock.patch.object(nsp, 'run', self.fake_nsz),
+                   mock.patch.object(importlib.util, 'find_spec', lambda m, *a: object() if m == 'nsz' else None)):
+            cm.__enter__(); self.addCleanup(cm.__exit__, None, None, None)
+
+    def fake_nsz(self, cmd, log, env=None, cwd=None, on_line=None):   # nsz -D --keys K -o OUT SRC: writes OUT/<stem>.nsp, or a partial one and a non-zero exit
+        src, out = Path(cmd[-1]), Path(cmd[cmd.index('-o') + 1])
+        self.runs.append(src.name)
+        (out / (src.stem + '.nsp')).write_bytes(b'partial' if self.fail else b'nsp of ' + src.name.encode())
+        if self.fail: raise nsp.BuildError('nsz exit code 1')
+
+    def test_scan_and_convert(self):
+        app, fam = nsp.scan_nsps(self.tmp)
+        self.assertEqual((app, fam['base'], fam['updates']), (APP, self.base, [(65536, self.upd)]))
+        fam = nsp.decompress_nsz(self.tmp, app, fam, self.tmp / 'prod.keys', {})
+        out = self.tmp / f'{APP:016X}_build' / 'nsp'
+        self.assertEqual((fam['base'], fam['updates']), (out / 'Game [v0].nsp', [(65536, out / 'Game [v65536].nsp')]))
+        self.assertEqual(fam['base'].read_bytes(), b'nsp of Game [v0].nsz')
+        self.assertFalse((out / 'partial').exists())
+        nsp.decompress_nsz(self.tmp, app, nsp.scan_nsps(self.tmp)[1], self.tmp / 'prod.keys', {})   # unchanged sources: reused
+        self.assertEqual(len(self.runs), 2)
+        os.utime(self.upd, ns=(0, 0))   # a changed source: converted again
+        nsp.decompress_nsz(self.tmp, app, nsp.scan_nsps(self.tmp)[1], self.tmp / 'prod.keys', {})
+        self.assertEqual(self.runs[2:], [self.upd.name])
+
+    def test_failed_conversion_is_not_kept(self):
+        app, fam = nsp.scan_nsps(self.tmp); self.fail = True
+        with self.assertRaises(nsp.BuildError): nsp.decompress_nsz(self.tmp, app, fam, self.tmp / 'prod.keys', {})
+        out = self.tmp / f'{APP:016X}_build' / 'nsp'
+        self.assertEqual(sorted(p.name for p in out.iterdir()), [])   # neither the partial file nor a key
+        self.fail = False
+        nsp.decompress_nsz(self.tmp, app, fam, self.tmp / 'prod.keys', {})
+        self.assertEqual((out / 'Game [v0].nsp').read_bytes(), b'nsp of Game [v0].nsz')
+
+    def test_an_nsp_beside_wins_and_nsps_need_no_nsz(self):
+        pfs0(self.tmp / 'Game [v0].nsp', [f'{APP:016x}0000000000000011.tik']); self.upd.unlink()
+        app, fam = nsp.scan_nsps(self.tmp)
+        self.assertEqual((fam['base'], fam['updates']), (self.tmp / 'Game [v0].nsp', []))
+        with mock.patch.object(importlib.util, 'find_spec', lambda m, *a: None):
+            self.assertIs(nsp.decompress_nsz(self.tmp, app, fam, self.tmp / 'prod.keys', {}), fam)
+
+    def test_missing_nsz(self):
+        app, fam = nsp.scan_nsps(self.tmp)
+        with mock.patch.object(importlib.util, 'find_spec', lambda m, *a: None), self.assertRaisesRegex(nsp.BuildError, 'pip install nsz'):
+            nsp.decompress_nsz(self.tmp, app, fam, self.tmp / 'prod.keys', {})
+
+
+if __name__ == '__main__':
+    unittest.main()

docs/development.md

@@ -79,7 +79,7 @@ When the output goes to a file or a pipe, the script prints a plain line instead
 
 `--progress json` is for a program that shows the build (a GUI): stdout carries one JSON object per line, each with `ev` and the time `t`. The events are `stage` (`k`, `n`, `name`, `step`), `step`, `progress` (`done`, `total`, `unit`, `est`, `note`, `frac`, `eta`, `elapsed`; `null` while unknown), `stage_end` (`ok`, `summary`, `elapsed`) and `say` (`msg`: every message, errors and a failed step's log tail included). A count change goes out at most every 0.5 s, any other change at once. This mode never asks for the NSP folder or the keys; argument errors and an unexpected error's traceback still go to stderr. `dm/tools/tests/test_nsp_progress.py` tests it without game files, keys or Xcode (`python3 -m unittest discover -s dm/tools/tests`).
 
-Everything it writes stays in the NSP folder: intermediate files and logs in `<id>_build/`, the IPA next to the NSPs. The extracted game files stay in `<id>_build/extract/` for the next build; `--no-keep-extracted` (`KEEP_EXTRACTED = False` in Spyder) deletes them at the end. It exits with 0 when done and 1 on an error. The app's bundle ID is `org.nxnative.<base game title ID>` (for example `org.nxnative.0100a6301214e000`), and its version is the game's display version: the update's when an update is packed, else the base game's (Engage 2.0.0); `--app-name` / `--bundle-id` (or `APP_NAME` / `BUNDLE_ID` in Spyder) override the name and ID.
+Everything it writes stays in the NSP folder: intermediate files and logs in `<id>_build/`, the IPA next to the NSPs. NSZ and XCZ files are converted with `nsz -D` (from the `nsz` package in the Python running the script) into `<title ID>_build/nsp/`, through a temporary folder so that only a conversion nsz finished is used, and reused while the source keeps its name, size and modification time; `nsz -D` reports an NCA hash mismatch but goes on, so a damaged file is caught by nx_extract's own hash check (`--allow-damaged`). The extracted game files stay in `<id>_build/extract/` for the next build; `--no-keep-extracted` (`KEEP_EXTRACTED = False` in Spyder) deletes them at the end. It exits with 0 when done and 1 on an error. The app's bundle ID is `org.nxnative.<base game title ID>` (for example `org.nxnative.0100a6301214e000`), and its version is the game's display version: the update's when an update is packed, else the base game's (Engage 2.0.0); `--app-name` / `--bundle-id` (or `APP_NAME` / `BUNDLE_ID` in Spyder) override the name and ID.
 The game language (`--language`, `LANGUAGE`; English by default) is what `nn::oe::GetDesiredLanguage` returns. As on the console, it must be one the game's control data lists (`SupportedLanguageFlag`, `nacp.py --languages`); the packed version's control data counts, since an update may add languages. The same language for another region stands in (`en-GB` for `en-US`, `es` for `es-419`, `fr` for `fr-CA`, `pt` for `pt-BR`, and back). A language the game lacks stops the build with the game's list; English, the default, falls back to the game's first language.
 
 On the device, environment switches go in `Documents/env.txt` (one `KEY=VALUE` per line), and game files are also accepted at `Documents/romfs`.

scripts/build_ipa_from_nsp.py

@@ -15,7 +15,7 @@
 # Uses only three things: this repository, a folder holding one game's NSPs (the base game, with or without an update; DLC NSPs, when present, are packed too and unlock the add-on content already in the game), and prod.keys.
 #   The newest version in the folder is packed. A version the profile registers (with every version in its shader_versions in the folder) is built from the profile as it is; any other
 #   version (the base game alone, another update) from a profile derived from it in <id>_build/profile: the same settings, and the version's own shaders (derive_profile).
-# Every intermediate file is under <id>_build/ in the NSP folder: extract/ (the extracted game files, kept for the next build unless KEEP_EXTRACTED / --no-keep-extracted), aoc/ (each DLC's RomFS by index), tools/ (nx_extract, and the Mac host for a run-time shader capture), patched_*/, shader/ (shader package and caches, kept so the next build is fast), logs/ (each step's full output), tmp/, ipa/. The finished IPA also goes to the NSP folder by default.
+# Every intermediate file is under <id>_build/ in the NSP folder: nsp/ (NSZ / XCZ files converted to NSP / XCI with nsz -D, kept for the next build; pip install nsz), extract/ (the extracted game files, kept for the next build unless KEEP_EXTRACTED / --no-keep-extracted), aoc/ (each DLC's RomFS by index), tools/ (nx_extract, and the Mac host for a run-time shader capture), patched_*/, shader/ (shader package and caches, kept so the next build is fast), logs/ (each step's full output), tmp/, ipa/. The finished IPA also goes to the NSP folder by default.
 # The DLC, the rewrite, the shader split and the iOS (or Vulkan) shaders are reused when all their inputs are as at their last successful run (the game files, the tools, the profile, Xcode and spirv-cross: fingerprint() keys beside their output);
 #   the iOS shaders' inputs are the shader package's content and the chain's own tools, so a split that runs again and writes the same package leaves them reused.
 #   The chain version header dm/gfx/nvn/dmsh_chain.h (one version for every game) is written again when a chain change was committed without it (chain_header: it follows from the source alone),
@@ -25,7 +25,7 @@
 #   are built through dm/gfx/tools/own_compilers.py, which says so in one line and leaves the lock and the chain version as they are.
 # The Python running this file needs lz4, capstone and fonttools (and backports.zstd before Python 3.14); also needed: Xcode, xcodegen, .NET SDK 10, zstd, spirv-cross. A first run takes 2 (Metroid Dread) to 10 minutes (Breath of the Wild: 5 of them the shader split), longer when shader libraries must be compiled; a repeat run with unchanged inputs takes about 20 s (Metroid Dread), mostly packing the IPA. Key values are never printed.
 # The console shows one line per stage ([k/n] stage, with a bar, done/total, percent, elapsed time and ETA where the work can be counted); the tools' own output goes only to logs/ (PROGRESS / --progress). --progress json writes newline-delimited JSON events to stdout.
-import ast, collections, ctypes, getpass, hashlib, json, os, re, shlex, shutil, struct, subprocess, sys, tempfile, threading, time, types, zlib
+import ast, collections, ctypes, getpass, hashlib, importlib.util, json, os, re, shlex, shutil, struct, subprocess, sys, tempfile, threading, time, types, zlib
 from pathlib import Path
 
 # ---- Settings ----
@@ -390,12 +390,18 @@ def version_of(path):
     return int(m.group(1)) if m else None
 
 
+COMPRESSED = {'.nsz': '.nsp', '.xcz': '.xci'}   # nsz's compressed NSP / XCI and what nsz -D makes of them
+
+
 def scan_nsps(nsp_dir):
-    """Group by game: {application title ID: {'base': path, 'updates': [(version, path)], 'dlc': [(title ID, version, path)]}}"""
+    """Group by game: {application title ID: {'base': path, 'updates': [(version, path)], 'dlc': [(title ID, version, path)]}}. An NSZ / XCZ is listed as it is (its ticket names
+    the title as an NSP's does) and converted afterwards (decompress_nsz)"""
     fams, unknown = {}, []
-    for p in sorted([*nsp_dir.glob('*.nsp'), *nsp_dir.glob('*.xci')]):   # a base game may be a cartridge dump (.xci: Monster Hunter Rise); nx_extract reads its secure partition
+    for p in sorted([*nsp_dir.glob('*.nsp'), *nsp_dir.glob('*.xci'), *nsp_dir.glob('*.nsz'), *nsp_dir.glob('*.xcz')]):   # a base game may be a cartridge dump (.xci: Monster Hunter Rise); nx_extract reads its secure partition
         if p.stat().st_size == 0:   # an empty file left by an incomplete download (the Zelda BotW folder holds a 0-byte DLC NSP next to the complete (1).nsp)
             say(f'skipping an empty file: {p.name}'); continue
+        if p.suffix in COMPRESSED and p.with_suffix(COMPRESSED[p.suffix]).exists():
+            say(f'skipping {p.name}: {p.with_suffix(COMPRESSED[p.suffix]).name} is beside it'); continue
         tid = title_of(p)
         if tid is None: unknown.append(p); continue
         low = tid & 0xFFF   # base …000, update …800, DLC = base + 0x1000 + index
@@ -413,6 +419,40 @@ def scan_nsps(nsp_dir):
     return app, fam
 
 
+NSZ_MAIN = "import sys; sys.argv[0] = 'nsz'; from nsz import main; main()"   # the nsz tool of the Python running this file (Spyder does not have its bin folder on PATH)
+
+
+def decompress_nsz(nsp_dir, app, fam, keys, env):
+    """NSZ / XCZ files of fam converted once with nsz -D into <title ID>_build/nsp/; returns fam with the converted files in their place. nsz writes into an empty temporary folder
+    and the file is moved into nsp/ only when nsz exits 0, so a partial one is never taken for a whole one; a conversion is reused while its source has the same name, size and
+    modification time. nsz -D prints an NCA hash mismatch but goes on and exits 0: nx_extract's hash check catches a damaged file (ALLOW_DAMAGED)"""
+    paths = [fam['base']] + [p for _, p in fam['updates']] + [p for _, _, p in fam['dlc']]
+    todo = [p for p in paths if p.suffix in COMPRESSED]
+    if not todo: return fam
+    if importlib.util.find_spec('nsz') is None:
+        raise BuildError(f'NSZ / XCZ files need the nsz tool in the current Python ({sys.executable}): pip install nsz')
+    out = nsp_dir / f'{app:016X}_build' / 'nsp'; logs, tmp = out.parent / 'logs', out / 'partial'
+    PROG.plan(1); PROG.stage('decompress NSZ', done=0, total=len(todo))
+    conv, made = {}, 0
+    for i, p in enumerate(todo):
+        dst = out / (p.stem + COMPRESSED[p.suffix]); keyfile = dst.with_name(dst.name + '.key')
+        st = p.stat(); key = fingerprint(p.name, st.st_size, st.st_mtime_ns)
+        if not (dst.is_file() and reused(keyfile, key) is not None):
+            PROG.set(done=i, step=p.name)
+            rmtree(tmp); tmp.mkdir(parents=True); rmtree(dst); keyfile.unlink(missing_ok=True)
+            log = logs / f'nsz_{p.stem}.log'
+            try:
+                run([sys.executable, '-c', NSZ_MAIN, '-D', '--keys', keys, '-o', tmp, p], log, env=env)
+                if not (tmp / dst.name).is_file(): raise BuildError(f'nsz made no {dst.name} (full output: {log})')
+                (tmp / dst.name).rename(dst)
+            finally: rmtree(tmp)
+            keyfile.write_text(key + '\n'); made += 1
+        conv[p] = dst
+    PROG.end(f'{made} converted, {len(todo) - made} reused ({out})')
+    c = lambda p: conv.get(p, p)
+    return {'base': c(fam['base']), 'updates': [(v, c(p)) for v, p in fam['updates']], 'dlc': [(t, v, c(p)) for t, v, p in fam['dlc']]}
+
+
 def vnum(ver):   # 'v655360' → 655360
     return int(ver[1:])
 
@@ -1561,7 +1601,8 @@ def main(args=None):
     env.setdefault('J', str(os.cpu_count() or 8))   # shader_split.py's and dmsh_build.py's workers (their default is 8)
     py = sys.executable
     app, fam = scan_nsps(nsp_dir)
-    force = args.provisional or PROVISIONAL   # a provisional profile even when dm/games has one (testing the guesses)
+    fam = decompress_nsz(nsp_dir, app, fam, keys, env)
+    force =args.provisional or PROVISIONAL   # a provisional profile even when dm/games has one (testing the guesses)
     g, provisional, shadowed = None if force else gamemod.by_title(app), False, False
     alias = None if force or g else same_build(nsp_dir, app, fam, keys, repo, env)   # a known build under another title ID: packed from a profile derived from that build's
     if alias: