Support masked-predecessor jump tables and external patches (#237)

Recognize offset-table dispatches whose predecessor masks the index while
the dispatch checks a masked copy of the same source. Reject calls and
resolved table paths that bypass masking, undecodable instruction gaps or
targets, and writes that invalidate the table base. Keep the existing
recognizers ahead of this fallback.

Load build-gated patches from DM_GAMES_DIR and include them in rewrite reuse
keys. Preserve repository matching rules for derived profiles, select the
first external profile for provisional matching rules, and deduplicate
patch paths while retaining conflicts between distinct build claims.

Validation on current main: Python tool tests (156, 14 skipped), iOS tests
(6), graphics tests (44, 2 skipped), documentation index and diff checks.
Regression cases fail on the original implementation and pass with these
guards; both supported base-loading forms remain accepted. Real game
modules and gameplay were not revalidated.

Co-authored-by: Amost <6445689+Amost-@users.noreply.github.com>

dm/games/README.md

@@ -10,7 +10,7 @@ Each game the port knows has a folder `dm/games/<id>/`, named after the base gam
 - **`gl_capture.tsv`** (an OpenGL game that puts its GLSL together at run time, Balatro): the files (`<key>.vert` / `<key>.frag`) of the shaders it compiles.
 - **`bf_want.tsv`**.
 - **`README.md`**: the game's notes: what its profile needs and why, what was found on it, how to reach a scene on the Mac, what is still open. Listed in [Games](#games).
-- **`patches.py`** (optional): the rewriter's patches of the game's own builds (`main`), keyed by NSO build ID: `CUTS` (byte cuts, each original asserted), `OBS` (observation gates) and `FRAME_STEP` (frame-step stubs). `dm/tools/patch_nso.py` loads every profile's file and applies those of a module's build, wherever it comes from; a build claimed twice stops the rewrite. Cuts of Nintendo's `sdk` modules, which several games ship, are in `dm/tools/sdk_patches.py`. Opt-in `MATCH_CUTS` can locate verified cuts in unregistered builds of the selected game by function fingerprints; see [Function-matched cuts](../../docs/reference/rewriter.md#function-matched-cuts-for-unregistered-builds) for enrollment and limits. A game without patches has no file, and a derived profile uses its repository profile's rules. `dm/tools/tests/test_patches.py` checks that a file names only its own profile's builds.
+- **`patches.py`** (optional): the rewriter's patches of the game's own builds (`main`), keyed by NSO build ID: `CUTS` (byte cuts, each original asserted), `OBS` (observation gates) and `FRAME_STEP` (frame-step stubs). `dm/tools/patch_nso.py` loads every profile's file and applies those of a module's build, wherever it comes from; a build claimed twice stops the rewrite. A provisional profile (in `DM_GAMES_DIR`) may have one too, which the rewriter and the NSP script's rewrite step take as well. Cuts of Nintendo's `sdk` modules, which several games ship, are in `dm/tools/sdk_patches.py`. Opt-in `MATCH_CUTS` can locate verified cuts in unregistered builds of the selected game by function fingerprints; see [Function-matched cuts](../../docs/reference/rewriter.md#function-matched-cuts-for-unregistered-builds) for enrollment, profile precedence and limits. A game without patches has no file, and a derived profile uses its repository profile's rules. `dm/tools/tests/test_patches.py` checks that a file names only its own profile's builds.
 
 The chain version is not in the profile: one version serves every game (`dm/gfx/nvn/dmsh_chain.h`, written by `dmsh_chain.py --header`). Two more tables live in the profile folder but are not committed (git ignores them). A full `dmsh_build.py` run makes both from its Mac outputs, so each checkout makes its own, and stamps its libraries with the digest of the profile's inputs (`b-<V>/inputs.tsv`, `dmsh_chain.py --inputs`). The host takes a table only when its header names its library's chain version and that digest.
 - **`c1_full.tsv`**: written by `c1_guard.py`. Without it every shader's c1 gets its own 64 KiB.

dm/tools/patch_nso.py

@@ -67,16 +67,26 @@ else:
 #   OBS = {build ID: [(offset, number, original instruction word, reason)]}: observation gates. Numbers match OBS_* in the host's hle.h; the original instruction must be position
 #     independent (no PC-relative / branch / anything involving x18 or tpidr), asserted offline; when the host has no callback attached the gate block goes in and out once and returns
 #   FRAME_STEP = {build ID: function(ctx)}: the frame step stubs of a main ("Frame step in whole vsyncs" below)
-#   MATCH_CUTS = {module name: [nso_match rule, ...]}: fallback for modules without registered CUTS. Only the repository profile selected by DM_GAME participates,
-#     including when DM_GAMES_DIR selects a derived profile. Missing or ambiguous matches are reported and skipped; known builds retain their exact CUTS.
+#   MATCH_CUTS = {module name: [nso_match rule, ...]}: fallback for modules without registered CUTS. Only the profile selected by DM_GAME participates; a derived profile uses
+#     the repository's rules, an external-only provisional profile its own. Missing or ambiguous matches are reported and skipped; known builds retain their exact CUTS.
 PATCH_FILES = [os.path.join(os.path.dirname(os.path.abspath(__file__)), 'sdk_patches.py')] + sorted(glob.glob(os.path.join(os.path.dirname(os.path.abspath(__file__)), '../games/*/patches.py')))
+# Profiles outside the repository (DM_GAMES_DIR, folders separated by os.pathsep, which is ':' as in dm/games/game.py: the NSP script's provisional profile of a game being brought up) may have a
+# patches.py of their own builds too; they are loaded after the repository's, under the same rule (a build ID claimed twice stops the rewrite)
+PATCH_FILES += [p for d in os.environ.get('DM_GAMES_DIR', '').split(os.pathsep) if d for p in sorted(glob.glob(os.path.join(glob.escape(d), '*', 'patches.py')))]
+patch_paths = {}
+for path in PATCH_FILES: patch_paths.setdefault(os.path.realpath(path), path)   # repeated roots and symlinks name the same file, not competing build claims
+PATCH_FILES = list(patch_paths.values())
 def load_patches():
     tables, where = {'CUTS': {}, 'OBS': {}, 'FRAME_STEP': {}, 'MATCH_CUTS': {}}, {}
     for n, path in enumerate(PATCH_FILES):
         spec = importlib.util.spec_from_file_location(f'patches_{n}', path); m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
         for kind, table in tables.items():
             if kind == 'MATCH_CUTS':
-                if getattr(m, kind, {}): table[os.path.basename(os.path.dirname(path))] = m.MATCH_CUTS
+                profile = os.path.basename(os.path.dirname(path))
+                roots = [os.path.join(os.path.dirname(os.path.abspath(__file__)), '../games')] + [d for d in os.environ.get('DM_GAMES_DIR', '').split(os.pathsep) if d]
+                profile_dir = next((os.path.join(root, profile) for root in roots if os.path.isdir(os.path.join(root, profile))), None)
+                if profile_dir and os.path.realpath(os.path.dirname(path)) != os.path.realpath(profile_dir): continue
+                if getattr(m, kind, {}): table[profile] = m.MATCH_CUTS   # repository rules win for derived profiles; otherwise use the first external root
                 continue
             for bid, v in getattr(m, kind, {}).items():
                 assert bid not in table, f'{kind} of build {bid} in both {where[kind, bid]} and {path}'
@@ -670,6 +680,64 @@ def table_cands(insl, rb):
         m = re.fullmatch(rf'{rb}, {rb}, #(0x[0-9a-f]+|\d+)', q.op_str) if q is not None and q.mnemonic == 'add' else None
         if m: cands.add(p.operands[1].imm + int(m.group(1), 0))
     return sorted(cands)
+def jump_masked_pred(insl, j):
+    """The offset-table form whose bounds check is on a masked copy of the index made in the block itself, while the index register was masked in the one block that branches here
+    (FINAL FANTASY CRYSTAL CHRONICLES Remastered Edition 1.0.0 main+0x2d103cc: and x6,x17,#0x7ff; ubfx w18,w17,#0xb,#2; cbz w18,#0x2d103b8 at main+0x2d1024c, then at main+0x2d103b8
+    and w18,w17,#0x7ff; cmp w18,#3; b.hi; ldrsw x18,[x27,x6,lsl #2]; add x18,x18,x27; br x18, with x27 from adrp + add 1,500 instructions back). Its neighbour main+0x2d10590
+    (predecessor main+0x2d102d4: and x6,x17,#0x7ff; ...; cbz w18,#0x2d10548) builds the base and moves registers inside the block: and w18,w17,#0x7ff; cmp w18,#3; b.hi; adrp x24 +
+    add x24,x24,#0x88c; mov x22,x30; ldrsw x18,[x24,x6,lsl #2]; add x18,x18,x24; then 10 register moves before br x18. The two ands give the same value (a mask below 2^32 makes the
+    64-bit and equal to the zero-extended 32-bit one), so the check bounds the index. Accepted only when the block, at most 32 instructions, is and wC,wS,#M; cmp wC,#N; b.hi, then
+    straight-line code (no branch, call or return) holding the ldrsw and the add, with the index unchanged from the b.hi to the load, the base unchanged and x18 untouched from the load
+    to the add and x18 untouched from the add to the br; the block is entered by branches alone (the instruction before it is b / br / ret, and no branch lands inside it after its and, nor any jump table already resolved in the function anywhere in it),
+    and every branch to it follows, within its own straight-line run, "and xI, xS, #M" with the check's source and mask and neither register written in between. The table base is found
+    as jump_offsets' far_base finds it, and every target must be an instruction of the function. Tried only after every other form has failed, so sites that rewrote before come out
+    the same. Returns all targets, or None if not recognized"""
+    def is_branch(p): return p.mnemonic in ('b', 'cbz', 'cbnz', 'tbz', 'tbnz') or p.mnemonic.startswith('b.')
+    def is_cf(p): return is_branch(p) or p.mnemonic in ('br', 'blr', 'bl', 'ret')
+    def writes(p):   # capstone lists a compare's first operand as written; it only sets the flags
+        return set() if p.mnemonic in ('cmp', 'cmn', 'tst') else {num({'lr': 'x30', 'fp': 'x29'}.get(p.reg_name(r), p.reg_name(r))) for r in p.regs_access()[1]}
+    kh = next((q for q in range(j - 1, max(0, j - 32) - 1, -1) if is_cf(insl[q])), None)   # the block's b.hi: the nearest control flow before the br
+    if kh is None or kh < 3 or insl[kh].mnemonic != 'b.hi': return None
+    k, a_, c_ = kh - 2, insl[kh - 2], insl[kh - 1]
+    ma = re.fullmatch(r'(w\d+), (w\d+), #(0x[0-9a-f]+|\d+)', a_.op_str) if a_.mnemonic == 'and' else None
+    mc = re.fullmatch(r'(w\d+), #(0x[0-9a-f]+|\d+)', c_.op_str) if c_.mnemonic == 'cmp' else None
+    if not (ma and mc and ma.group(1) == mc.group(1)): return None
+    if any(insl[q + 1].address != insl[q].address + 4 for q in range(k - 1, j)): return None   # an undecodable word may hide a write or a branch
+    kl = next((q for q in range(kh + 1, j) if insl[q].mnemonic == 'ldrsw' and insl[q].op_str.startswith('x18, ')), None)
+    ml = re.fullmatch(r'x18, \[(x\d+), x(\d+), lsl #2\]', insl[kl].op_str) if kl is not None else None
+    if not ml: return None
+    rb, ix = ml.group(1), int(ml.group(2))
+    src, mask, n = num(ma.group(2)), int(ma.group(3), 0), int(mc.group(2), 0) + 1
+    if ix in (src, num(ma.group(1)), 18) or num(rb) in (ix, 18) or mask >> 32: return None
+    if any(ix in writes(p) for p in insl[kh + 1:kl]): return None   # the index changes between the check and the load
+    ka = next((q for q in range(kl + 1, j) if 18 in writes(insl[q])), None)   # the first write of x18 after the load must be the add
+    if ka is None or insl[ka].mnemonic != 'add' or insl[ka].op_str != f'x18, x18, {rb}': return None
+    if any(num(rb) in writes(p) for p in insl[kl + 1:ka]) or any(18 in writes(p) for p in insl[ka + 1:j]): return None
+    if insl[k - 1].mnemonic not in ('b', 'br', 'ret'): return None   # no fall-through into the block
+    brs = [q for q, p in enumerate(insl) if (is_branch(p) or p.mnemonic == 'bl') and p.operands[-1].type == ARM64_OP_IMM]
+    into = {insl[q].operands[-1].imm for q in brs}
+    if any(p.address in into for p in insl[k + 1:j + 1]): return None   # a branch into the block past its and
+    tabs = {t for tgt in tabjumps.values() for t in tgt}   # a jump table already resolved in this function may land anywhere in the block, with any index value
+    def table_target(a): return a in tabs or any(b <= a < b + 4 * w for b, w in jumps.values())
+    if any(table_target(p.address) for p in insl[k:j + 1]): return None
+    preds = [q for q in brs if insl[q].operands[-1].imm == a_.address]
+    if not preds or any(insl[q].mnemonic == 'bl' for q in preds): return None
+    for q in preds:   # each predecessor must have masked the index the same way, in its own straight-line run
+        for i in range(q - 1, max(0, q - 16) - 1, -1):
+            p = insl[i]
+            m = re.fullmatch(r'([xw]\d+), ([xw]\d+), #(0x[0-9a-f]+|\d+)', p.op_str) if p.mnemonic == 'and' else None
+            if m and num(m.group(1)) == ix:
+                if num(m.group(2)) != src or int(m.group(3), 0) != mask: return None
+                if any(x.address in into or table_target(x.address) for x in insl[i + 1:q + 1]): return None   # a direct branch, call or table must not bypass the mask
+                if any(insl[z + 1].address != insl[z].address + 4 for z in range(i, q)): return None
+                break
+            if is_cf(p) or writes(p) & {ix, src}: return None
+        else: return None
+    tb = copied_base(insl, kl, rb)   # follows adrp + add too, rejecting wN clobbers of xN and calls through volatile bases
+    if tb is None or tb < 0 or tb + 4 * n > len(IMG): return None
+    t = [tb + struct.unpack_from('<i', IMG, tb + 4 * i)[0] for i in range(n)]
+    addresses = {p.address for p in insl}
+    return t if all(x in addresses for x in t) else None
 def tail_call(insl, j):
     """The br x18 is an indirect tail call, not a jump table: the last write of x18 before it is a plain pointer load and the br ends the function. Ace Attorney Investigations Collection
     (Unity 2022.3 IL2CPP) main+0x1d271bc: ldr x18,[x0,#0x18]; ...; ldr x16,[sp]; ...; br x18, a thunk that moves stack arguments and jumps to the method pointer at +0x18"""
@@ -800,6 +868,7 @@ for fa, fb in funcs:
                 if not tgt and tail_call(insl, j): tailcalls.add(ins.address); continue
                 if not tgt and (far := jump_bytes_far(insl, j)): jumps[ins.address] = far; continue
                 if not tgt: tgt = jump_offsets(insl, j, copy_first=True, far_base=True)   # the index copied before the check (check_before_copy) or the base set far back, only where every other form failed
+                if not tgt: tgt = jump_masked_pred(insl, j)   # the check on a masked copy made in the block, the index masked in the block that branches to it
                 assert tgt, f'jump table {ins.address:x}'
                 tabjumps[ins.address] = tgt; continue
             jumps[ins.address] = (base, width)

dm/tools/tests/test_jump_copies.py

@@ -1,4 +1,4 @@
-"""Exercise the rewriter's copied-base recognition on real AArch64 instructions, without game files."""
+"""Exercise the rewriter's copied-base and masked-predecessor jump table recognition on real AArch64 instructions, without game files."""
 import ast
 from pathlib import Path
 import re
@@ -7,16 +7,18 @@ import sys
 import unittest
 
 from capstone import Cs, CS_ARCH_ARM64, CS_MODE_ARM
+from capstone.arm64 import ARM64_OP_IMM
 
 TOOLS = Path(__file__).resolve().parents[1]
 sys.path.insert(0, str(TOOLS))
-from nso_asm import ADRP, ADDI, BLR, NOP
+from nso_asm import ADRP, ADDI, ADDR, B, BLR, BR, NOP
 
 # patch_nso.py is a command-line program whose top level loads and rewrites an image.
 # Compile its actual recognition functions without executing that driver.
 tree = ast.parse((TOOLS / 'patch_nso.py').read_text())
-scope = {'re': re}
-exec(compile(ast.Module(body=[n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name in ('num', 'copied_base')], type_ignores=[]), str(TOOLS / 'patch_nso.py'), 'exec'), scope)
+IMG = bytearray(0x5000); tabjumps = {}; jumps = {}   # the globals the recognizers read: the module image and the jump tables resolved so far in the function
+scope = {'re': re, 'struct': struct, 'ARM64_OP_IMM': ARM64_OP_IMM, 'IMG': IMG, 'tabjumps': tabjumps, 'jumps': jumps}
+exec(compile(ast.Module(body=[n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name in ('num', 'copied_base', 'table_base', 'jump_masked_pred')], type_ignores=[]), str(TOOLS / 'patch_nso.py'), 'exec'), scope)
 
 
 def mov(dst, src, wide=True): return (0xaa0003e0 if wide else 0x2a0003e0) | src << 16 | dst
@@ -53,4 +55,120 @@ class CopiedBase(unittest.TestCase):
         self.assertIsNone(self.follow(self.base() + [mov(7, 0)]))
 
 
+RET = 0xd65f03c0
+AND_X6_X17_7FF = 0x92400000 | 10 << 10 | 17 << 5 | 6      # and x6, x17, #0x7ff
+AND_X6_X17_FF = 0x92400000 | 7 << 10 | 17 << 5 | 6        # and x6, x17, #0xff
+AND_W18_W17_7FF = 0x12000000 | 10 << 10 | 17 << 5 | 18    # and w18, w17, #0x7ff
+UBFX_W18_W17 = 0x53000000 | 11 << 16 | 12 << 10 | 17 << 5 | 18   # ubfx w18, w17, #0xb, #2
+CMP_W18_3 = 0x7100001f | 3 << 10 | 18 << 5                # cmp w18, #3
+CMP_W18_1 = 0x7100001f | 1 << 10 | 18 << 5                # cmp w18, #1
+LDRSW_X18 = 0xb8a07800 | 6 << 16 | 27 << 5 | 18           # ldrsw x18, [x27, x6, lsl #2]
+
+
+def cbz_w18(frm, to): return 0x34000000 | (((to - frm) >> 2) & 0x7ffff) << 5 | 18
+def bhi(frm, to): return 0x54000008 | (((to - frm) >> 2) & 0x7ffff) << 5
+
+
+class MaskedPredecessor(unittest.TestCase):
+    """The block `and w18,w17,#M; cmp w18,#3; b.hi; ldrsw x18,[x27,x6,lsl #2]; add; br x18` whose index x6 was masked from x17 in the block that branches to it.
+    Word indices of the synthetic function: 0-1 base (adrp + add), 2-4 the predecessor (and x6, ubfx, cbz), 5 filler, 6 ret, 7-15 the dispatch block (and at 7, slots at 10, 12 and 14),
+    16 ret, 17-20 the four targets. The table is at 0x4040 in the image."""
+    TABLE = 0x4040
+    TARGETS = [0x44, 0x48, 0x4c, 0x50]
+
+    def setUp(self):
+        for i, t in enumerate(self.TARGETS): struct.pack_into('<i', IMG, self.TABLE + 4 * i, t - self.TABLE)
+        tabjumps.clear(); jumps.clear()
+
+    def words(self, **over):
+        w = [ADRP(27, 0, self.TABLE), ADDI(27, 27, self.TABLE & 0xfff), AND_X6_X17_7FF, UBFX_W18_W17, cbz_w18(16, 28), NOP, RET,
+             AND_W18_W17_7FF, CMP_W18_3, bhi(36, 64), NOP, LDRSW_X18, NOP, ADDR(18, 18, 27), NOP, BR(18), RET, NOP, NOP, NOP, NOP]
+        for k, v in over.items(): w[int(k[1:])] = v
+        return w
+
+    def run_site(self, words, undecodable=False):
+        md = Cs(CS_ARCH_ARM64, CS_MODE_ARM); md.detail = True
+        # The driver skips words Capstone cannot decode and resumes at the next word.
+        ins = [p for i, word in enumerate(words) for p in md.disasm(struct.pack('<I', word), 4 * i)]
+        self.assertEqual(len(ins), len(words) - int(undecodable))
+        return scope['jump_masked_pred'](ins, next(i for i, p in enumerate(ins) if p.mnemonic == 'br' and p.op_str == 'x18'))
+
+    def test_recognized(self):
+        self.assertEqual(self.run_site(self.words()), self.TARGETS)
+
+    def test_base_can_be_hoisted_far_back_or_built_in_dispatch(self):
+        for local in (False, True):
+            with self.subTest(local=local):
+                words = self.words()
+                if local:
+                    words[:2] = [NOP, NOP]
+                    words[10] = ADRP(27, 40, self.TABLE)
+                    words.insert(11, ADDI(27, 27, self.TABLE & 0xfff))
+                    words[16:16] = [mov(22, 30)] * 10   # ten register moves after the add
+                    shift = 44; words[9] = bhi(36, 64 + shift)
+                else:
+                    words[2:2] = [NOP] * 300   # base more than 256 instructions before the load
+                    shift = 1200
+                targets = [t + shift for t in self.TARGETS]
+                for i, t in enumerate(targets): struct.pack_into('<i', IMG, self.TABLE + 4 * i, t - self.TABLE)
+                self.assertEqual(self.run_site(words), targets)
+
+    def test_compare_in_the_block_does_not_count_as_a_write(self):
+        self.assertEqual(self.run_site(self.words(w14=CMP_W18_1)), self.TARGETS)
+
+    def test_predecessor_must_mask_the_same_way(self):
+        self.assertIsNone(self.run_site(self.words(w2=AND_X6_X17_FF)))
+        self.assertIsNone(self.run_site(self.words(w2=NOP)))
+        self.assertIsNone(self.run_site(self.words(w3=0x2a0003f1)))   # mov w17, w0: the source changes between the and and the branch
+
+    def test_block_entered_only_by_branches(self):
+        self.assertIsNone(self.run_site(self.words(w6=NOP)))   # falls through into the and
+        self.assertIsNone(self.run_site(self.words(w5=cbz_w18(20, 44))))   # another branch lands inside the block, past its and
+        tabjumps[0x1000] = {0x20}   # a jump table already resolved in the function lands on the cmp
+        self.assertIsNone(self.run_site(self.words()))
+        tabjumps.clear(); jumps[0x1004] = (0x20, 4)   # a byte or half-word table that reaches the block
+        self.assertIsNone(self.run_site(self.words()))
+
+    def test_registers_unchanged_between_check_and_use(self):
+        self.assertIsNone(self.run_site(self.words(w10=mov(6, 0))))   # the index changes after the check
+        self.assertIsNone(self.run_site(self.words(w12=mov(27, 0))))   # the base changes between the load and the add
+        self.assertIsNone(self.run_site(self.words(w12=mov(18, 0))))   # x18 is rewritten before the add
+        self.assertIsNone(self.run_site(self.words(w14=mov(18, 0))))   # x18 is rewritten after the add
+
+    def test_tables_cannot_bypass_predecessor_mask(self):
+        for target in (12, 16):   # the ubfx or cbz, after the predecessor's and
+            with self.subTest(target=target, table='offset'):
+                tabjumps[0x1000] = {target}
+                self.assertIsNone(self.run_site(self.words()))
+                tabjumps.clear()
+            with self.subTest(target=target, table='byte'):
+                jumps[0x1004] = (target, 1)
+                self.assertIsNone(self.run_site(self.words()))
+                jumps.clear()
+
+    def test_table_can_enter_at_predecessor_mask(self):
+        tabjumps[0x1000] = {8}; jumps[0x1004] = (8, 1)
+        self.assertEqual(self.run_site(self.words()), self.TARGETS)
+
+    def test_calls_cannot_enter_after_a_mask(self):
+        for target in (12, 16, 28, 44):   # predecessor, dispatch head or load
+            with self.subTest(target=target):
+                self.assertIsNone(self.run_site(self.words(w5=B(20, target) | 0x80000000)))
+
+    def test_32_bit_writer_destroys_table_base(self):
+        self.assertIsNone(self.run_site(self.words(w5=mov(27, 0, wide=False))))
+
+    def test_undecodable_words_cannot_be_targets_or_hide_writes(self):
+        for slot in (3, 10, 17):   # predecessor, dispatch, target
+            with self.subTest(slot=slot):
+                self.assertIsNone(self.run_site(self.words(**{f'w{slot}': 0xe7ffdefe}), undecodable=True))
+
+    def test_targets_must_be_instructions_of_the_function(self):
+        struct.pack_into('<i', IMG, self.TABLE + 12, 0x1000)
+        self.assertIsNone(self.run_site(self.words()))
+
+    def test_no_base_no_site(self):
+        self.assertIsNone(self.run_site(self.words(w0=NOP, w1=NOP)))
+
+
 if __name__ == '__main__': unittest.main()

dm/tools/tests/test_patches.py

@@ -1,13 +1,17 @@
 """The rewriter's per-build patch files (dm/tools/sdk_patches.py, dm/games/<id>/patches.py, loaded by dm/tools/patch_nso.py): no build ID is claimed twice, every entry has
 the form patch_nso.py expects, a game's file names only builds of its own profile's non-sdk modules, sdk_patches.py only sdk modules some profile ships; and the NSP script's
 rewrite step keys its reuse on the patch files of the game's builds only. No game files needed (temporary files only in a temporary folder)."""
+import ast
+import glob
 import importlib.util
+import os
 from pathlib import Path
 import re
 import shutil
 import sys
 import tempfile
 import unittest
+from unittest import mock
 
 REPO = Path(__file__).resolve().parents[3]
 sys.path.insert(0, str(REPO / 'dm/tools')); sys.path.insert(0, str(REPO / 'dm/games'))
@@ -24,6 +28,12 @@ def load(path, n):
 MODS = {p: load(p, n) for n, p in enumerate(FILES)}
 HEX = re.compile(r'[0-9a-f]+$')
 
+# Execute the driver's actual file discovery and registry without loading a game image.
+source = REPO / 'dm/tools/patch_nso.py'; tree = ast.parse(source.read_text())
+start = next(i for i, n in enumerate(tree.body) if isinstance(n, ast.Assign) and ast.unparse(n.targets[0]) == 'PATCH_FILES')
+end = next(i for i, n in enumerate(tree.body[start:], start) if isinstance(n, ast.Assign) and ast.unparse(n.targets[0]) == 'BUILD_ID')
+REGISTRY_CODE = compile(ast.Module(body=tree.body[start:end], type_ignores=[]), str(source), 'exec')
+
 
 def builds_of(g):   # {build ID: module name} of a profile's build rows (a profile or its title ID)
     return {bid: m for row in (game.load(g) if isinstance(g, str) else g).builds().values() for m, bid in row.items() if m != 'display'}
@@ -77,6 +87,72 @@ class Registry(unittest.TestCase):
         for bid in m.CUTS: self.assertIn(bid, sdk, f'sdk_patches.py: {bid} is no sdk build of any profile')
 
 
+class ExternalRegistry(unittest.TestCase):
+    def setUp(self):
+        self.tmp = Path(tempfile.mkdtemp()); self.addCleanup(shutil.rmtree, self.tmp)
+        self.tools = self.tmp / 'repo/tools'; self.tools.mkdir(parents=True)
+        (self.tools / 'sdk_patches.py').write_text('')
+        self.gid = '0100000000000000'
+        self.external = self.tmp / 'profiles'
+
+    def patch_file(self, root, text):
+        p = root / self.gid / 'patches.py'; p.parent.mkdir(parents=True, exist_ok=True); p.write_text(text)
+        return p
+
+    def registry(self, roots=None):
+        ns = dict(__file__=str(self.tools / 'patch_nso.py'), os=os, glob=glob, importlib=importlib)
+        with mock.patch.dict(os.environ, DM_GAMES_DIR=os.pathsep.join(map(str, roots or [self.external]))):
+            exec(REGISTRY_CODE, ns)
+        return ns['PATCHES'], ns['PATCH_WHERE']
+
+    def test_external_registered_patches_are_loaded(self):
+        bid = '5e' * 16
+        p = self.patch_file(self.external, f"CUTS = {{'{bid}': []}}\nOBS = {{'{bid}': []}}\nFRAME_STEP = {{'{bid}': lambda ctx: None}}\n")
+        tables, where = self.registry()
+        for kind in ('CUTS', 'OBS', 'FRAME_STEP'):
+            self.assertIn(bid, tables[kind]); self.assertEqual(Path(where[kind, bid]), p)
+
+    def test_derived_profile_keeps_repository_match_rules(self):
+        self.patch_file(self.tools.parent / 'games', "MATCH_CUTS = {'main': ['repository']}\n")
+        self.patch_file(self.external, "MATCH_CUTS = {'main': ['external']}\n")
+        tables, _ = self.registry()
+        self.assertEqual(tables['MATCH_CUTS'][self.gid], {'main': ['repository']})
+
+    def test_external_rules_cannot_enroll_a_repository_profile(self):
+        (self.tools.parent / 'games' / self.gid).mkdir(parents=True)
+        self.patch_file(self.external, "MATCH_CUTS = {'main': ['external']}\n")
+        tables, _ = self.registry()
+        self.assertNotIn(self.gid, tables['MATCH_CUTS'])
+
+    def test_provisional_profile_can_supply_match_rules(self):
+        self.patch_file(self.external, "MATCH_CUTS = {'main': ['external']}\n")
+        tables, _ = self.registry()
+        self.assertEqual(tables['MATCH_CUTS'][self.gid], {'main': ['external']})
+
+    def test_provisional_rules_follow_the_first_profile(self):
+        second = self.tmp / 'second'
+        self.patch_file(second, "MATCH_CUTS = {'main': ['second']}\n")
+        first = self.patch_file(self.external, "MATCH_CUTS = {'main': ['first']}\n")
+        tables, _ = self.registry([self.external, second])
+        self.assertEqual(tables['MATCH_CUTS'][self.gid], {'main': ['first']})
+        first.unlink()   # a selected profile with no rules must not inherit another directory's rules
+        tables, _ = self.registry([self.external, second])
+        self.assertNotIn(self.gid, tables['MATCH_CUTS'])
+
+    def test_repeated_roots_do_not_load_a_file_twice(self):
+        self.patch_file(self.external, "CUTS = {'" + '5e' * 16 + "': []}\n")
+        alias = self.tmp / 'alias'; alias.symlink_to(self.external, target_is_directory=True)
+        tables, _ = self.registry([self.external, self.external, alias])
+        self.assertEqual(len(tables['CUTS']), 1)
+
+    def test_distinct_files_cannot_claim_the_same_build(self):
+        text = "CUTS = {'" + '5e' * 16 + "': []}\n"
+        self.patch_file(self.tools.parent / 'games', text)
+        self.patch_file(self.external, text)
+        with self.assertRaisesRegex(AssertionError, 'in both'):
+            self.registry()
+
+
 class RewriteKey(unittest.TestCase):
     def test_patch_files_of_the_games_builds(self):
         spec = importlib.util.spec_from_file_location('build_ipa_from_nsp', REPO / 'scripts/build_ipa_from_nsp.py')
@@ -93,6 +169,23 @@ class RewriteKey(unittest.TestCase):
         g = game.load(gid)
         self.assertNotIn(REPO / 'dm/games' / gid / 'patches.py', nsp.profile_files(g))   # only the rewrite step reads it
 
+    def test_patch_files_of_a_provisional_profile(self):
+        spec = importlib.util.spec_from_file_location('build_ipa_from_nsp', REPO / 'scripts/build_ipa_from_nsp.py')
+        nsp = importlib.util.module_from_spec(spec); spec.loader.exec_module(nsp)
+        tmp = Path(tempfile.mkdtemp()); self.addCleanup(shutil.rmtree, tmp)
+        bid = '5e' * 16   # a build that no repository profile knows
+        (tmp / 'exe').mkdir(); (tmp / 'exe/main').write_bytes(b'NSO0' + bytes(0x3c) + bytes.fromhex(bid))
+        mine, other = tmp / 'profiles/0100000000000000/patches.py', tmp / 'profiles/0100000000000001/patches.py'
+        for f, text in ((mine, f"CUTS = {{'{bid}': []}}\n"), (other, "CUTS = {'" + '6f' * 16 + "': []}\n")):
+            f.parent.mkdir(parents=True); f.write_text(text)
+        with mock.patch.dict(os.environ, {'DM_GAMES_DIR': str(tmp / 'profiles')}):
+            self.assertEqual(nsp.patch_files(REPO, tmp / 'exe', ['main']), [mine])   # only the file that names the module's build
+            self.assertEqual(nsp.patch_files(REPO, tmp / 'exe', ['main'], '0100000000000001'), [mine, other])   # the selected game's file counts for any build
+        with mock.patch.dict(os.environ, {'DM_GAMES_DIR': ''}):
+            self.assertEqual(nsp.patch_files(REPO, tmp / 'exe', ['main']), [])
+        with mock.patch.dict(os.environ, {'DM_GAMES_DIR': os.pathsep.join([str(tmp / 'profiles')] * 2)}):
+            self.assertEqual(nsp.patch_files(REPO, tmp / 'exe', ['main']), [mine])
+
 
 if __name__ == '__main__':
     unittest.main()

docs/agents/symptoms.md

@@ -12,6 +12,7 @@ One line per symptom in the [subsystem references](../reference/): what you see
 - `jump table <addr>` assertion in `patch_nso.py`, in a Unity / IL2CPP `main`. → [Offset jump tables through x18 read at rewrite time](../reference/rewriter.md#offset-jump-tables-through-x18-read-at-rewrite-time)
 - `jump table <addr>` assertion in `patch_nso.py`, in a `subsdk`. → [Offset jump tables with the bounds check on another register](../reference/rewriter.md#offset-jump-tables-with-the-bounds-check-on-another-register)
 - `jump table <addr>` assertion where the index is a copy made before the check (`mov w0,w18`, then 41 instructions holding two other jump tables on `x0`, then `cmp w18,#4; b.hi; ldrsw x18,[x11,x0,lsl #2]; add x18,x18,x11; br x18`). → [Index copied before the bounds check](../reference/rewriter.md#index-copied-before-the-bounds-check)
+- `jump table <addr>` assertion in a Unity / IL2CPP `main` whose dispatch checks a fresh mask of another register than the one it indexes with. → [Index masked in the block that branches to the dispatch](../reference/rewriter.md#index-masked-in-the-block-that-branches-to-the-dispatch)
 - `jump table <addr>` assertion on an offset table whose base was set far back (`cmp w18,#5; b.hi; ldrsw x18,[x24,x18,lsl #2]; add x18,x18,x24; br x18`). → [Offset table base set far back in the function](../reference/rewriter.md#offset-table-base-set-far-back-in-the-function)
 - `jump table <addr>` assertion on an offset table whose base was set far back and moved through other registers (`ldrb w18,[x14]; sub w18,w18,#1; cmp w18,#6; b.hi; ldrsw x18,[x7,x18,lsl #2]; add x18,x18,x7; br x18`). → [Far table base passed through other registers](../reference/rewriter.md#far-table-base-passed-through-other-registers)
 - `jump table <addr>` assertion, entries relative to an `adr`. → [Offset tables relative to an adr](../reference/rewriter.md#offset-tables-relative-to-an-adr)

docs/reference/rewriter.md

@@ -9,7 +9,7 @@ The NSO rewriter (`dm/tools/patch_nso.py`) and its per-build tables (`dm/tools/s
 - **Symptom:** an updated game loses a needed cut because its NSO build ID changed, although the relevant code still matches a verified version; the rewriter reports `[RED] function patch ... skipped` when that match cannot be established.
 - **Cause:** fixed-address `CUTS` correctly apply only to their registered builds. Copying an old address, or matching a short instruction sequence, does not identify the updated function.
 - **Fix:** a profile can opt individual cuts into `MATCH_CUTS` in its `patches.py`. The shared matcher, `dm/tools/nso_match.py`, selects a unique complete unwind function by length and SHA-256 fingerprint. It retains constants, registers and internal branch destinations; normalizes relocation addresses; and checks the normalized bodies of direct callees or their imported symbol names. Original bytes are checked again before a cut is applied. Missing boundaries, unknown callees, changed code and ambiguous matches produce a red diagnostic and no cut. Overlapping rules stop the rewrite. Every accepted cut is recorded in `<module>.cuts.tsv` with `[function match]` in its reason.
-- **Scope:** the repository profile selected by `DM_GAME`, keyed by module name. A derived profile still uses that repository profile's rules. Other games' rules never participate. A build with registered `CUTS` uses those instead, including an explicitly empty list. SDK patches, observation gates, frame-step stubs and fixed host addresses retain their existing build-ID gates. The rewrite cache includes the selected profile's patch file even for unknown builds, the selected game ID and the matching tool.
+- **Scope:** the profile selected by `DM_GAME`, keyed by module name. A derived profile still uses that repository profile's rules, even when the repository has no matching rules or patch file; an external-only provisional profile can supply its own. External files cannot replace or enroll a repository profile's matching rules. Other games' rules never participate. A build with registered `CUTS` uses those instead, including an explicitly empty list. SDK patches, observation gates, frame-step stubs and fixed host addresses retain their existing build-ID gates. The rewrite cache includes the selected profile's patch file even for unknown builds, the selected game ID and the matching tool. Repeated search roots or symlinks to the same patch file are loaded and counted once; distinct files claiming the same build still stop the rewrite.
 - **Limits:** this is an opt-in code-identity check, not automatic compatibility for every game update. Direct callee bodies are checked one level deep; their callees, indirect-call targets and external data layouts are not proven unchanged. Enroll only a cut whose assumptions remain valid under those changes. Prefer fixing the general host behavior when possible. A replacement must contain no PC-relative instruction; relocated branch replacements need a different mechanism. Failure leaves the game potentially without its workaround, and stays visible in the rewrite log.
 - **Found on:** [Super Mario Bros. Wonder](../../dm/games/010015100B514000/README.md#memory), whose former resource-heap allowance was missing on 1.0.1. The shared matcher remains, but Wonder's allowance and matching rule were later removed after the general command-record fix described in its game note.
 
@@ -46,6 +46,15 @@ To enroll a cut for any game's own module:
 - **Found on:** [DRAGON QUEST XI S: Echoes of an Elusive Age – Definitive Edition](../../dm/games/01006C300E9F0000/README.md) `main+0x2dd8cd0`.
 - **Note:** before fixing one site, a census of all of them saves a run per site: a copy of `patch_nso.py` whose assert prints and continues found this one alone among the 25,508 `x18` sites. That `main` has no unwind entries for its first 70 MB, so the rewriter disassembles them as one function and takes 6 minutes (see [Long functions analysed in pieces](#long-functions-analysed-in-pieces)).
 
+### Index masked in the block that branches to the dispatch
+
+- **Symptom:** `jump table <addr>` assertion in a Unity / IL2CPP `main` whose dispatch checks a fresh mask of another register than the one it indexes with.
+- **Cause:** the dispatch block masks the source again for its bounds check, while the index register was masked from the same source in the block that branches to it: `and x6,x17,#0x7ff; ubfx w18,w17,#0xb,#2; cbz w18,L`, then `L: and w18,w17,#0x7ff; cmp w18,#3; b.hi; ldrsw x18,[x27,x6,lsl #2]; add x18,x18,x27; br x18`, with the base `x27` from an `adrp` + `add` about 1,500 instructions back. A sibling builds its base inside the block and moves registers before the `br` (`adrp x24` + `add`, a `mov`, the load and the add, then 10 register moves). No copy links the two registers, so `jump_offsets` and `check_before_copy` find no check.
+- **Fix:** `jump_masked_pred` accepts the site when the block (at most 32 instructions) is `and wC,wS,#M; cmp wC,#N; b.hi` followed by straight-line code holding the load and the add; it is entered by branches alone (the instruction before it is `b` / `br` / `ret`, no branch lands inside it after its `and`, no jump table already resolved in the function reaches it); every branch into it follows, on a straight-line path, `and xI,xS,#M` with the check's source and mask; and the index, the base and `x18` are unchanged where the form needs them. The base is found as `far_base` finds it and every target must be an instruction of the function. It runs only after every other form has failed.
+- **Guards (resolved):** direct calls and resolved jump tables must not enter between a predecessor's mask and its branch, since that bypasses the index assignment; direct calls into the dispatch are refused too. Undecodable words break either straight-line run, and table targets must be decoded instruction addresses, not just aligned addresses inside the function. The base uses `copied_base`, which follows `adrp` + `add` and register copies while rejecting 32-bit writes that destroy a 64-bit base.
+- **Found on:** FINAL FANTASY CRYSTAL CHRONICLES Remastered Edition 1.0.0 (Unity 2018.4.2f1, SDK 8.2.1) `main+0x2d103cc` and `+0x2d10590`, in one function; the same form in its update 1.0.3.
+- **Check:** it runs only where the rewriter asserted before, so modules that rewrote before come out the same; other games' modules were not on hand to compare.
+
 ### Offset table base set far back in the function
 
 - **Symptom:** `jump table <addr>` assertion on an offset table whose base was set far back (`cmp w18,#5; b.hi; ldrsw x18,[x24,x18,lsl #2]; add x18,x18,x24; br x18`).

scripts/build_ipa_from_nsp.py

@@ -855,10 +855,14 @@ def build_mismatch(d, name, g):   # the modules of an extraction whose build ID
 
 def patch_files(repo, exe_dir, mods, game_id=None):
     """The patch files that name one of the modules' builds (dm/tools/sdk_patches.py, dm/games/*/patches.py): patch_nso.py loads them all but applies only those of a module's
-    build ID. The selected game's file also counts for unknown builds: it may supply MATCH_CUTS. Another game's rules do not change this game's rewrite."""
+    build ID. The selected game's file also counts for unknown builds: it may supply MATCH_CUTS. Another game's rules do not change this game's rewrite. The patches.py of profiles
+    in DM_GAMES_DIR (a provisional profile's) count too, as patch_nso.py loads them"""
     ids = {build_id(exe_dir / m) for m in mods}
     selected = Path(game_id).name if game_id else None
-    return [p for p in [repo / 'dm/tools/sdk_patches.py'] + sorted((repo / 'dm/games').glob('*/patches.py'))
+    extra = [p for d in os.environ.get('DM_GAMES_DIR', '').split(os.pathsep) if d for p in sorted(Path(d).glob('*/patches.py'))]
+    files = {}
+    for p in [repo / 'dm/tools/sdk_patches.py'] + sorted((repo / 'dm/games').glob('*/patches.py')) + extra: files.setdefault(p.resolve(), p)
+    return [p for p in files.values()
             if p.parent.name == selected or any(i in p.read_text(errors='replace') for i in ids)]