Record threading audit and retain verified game workarounds
This commit is contained in:
1 parent
c188131824
commit
02e40ba753
7 files changed
+200
-8
No files matched your search
@@ -19,7 +19,8 @@ Status: brought up on the Mac: initial settings, the opening movie, the prologue
|
||||
- **Resolved: the main thread waited forever on a job's done event at boot while every worker slept, two of them on one event** (4 of 6 Mac runs, `main+0x1426c` waiting on `Job::m_doneEvent`). The engine's job pools wake one worker through the pool's single auto-clear event. A job goes into worker k's own queue (`main+0x154c0`), and if k's state reads asleep the caller signals the event (`main+0x12554`, from the per-worker dispatch loops `main+0x14000` / `main+0x14350` / `main+0x17670`, the single-job submits `main+0x14640` / `main+0x14790` / `main+0x14dc0` and the shutdown `main+0x13df0`). The kernel wakes the waiter that slept first, which is k only while the workers sleep in index order. A worker woken for another's job runs only its own queue (worker loop `main+0x15ce0`) and sleeps again, so with two workers in one pool (threads 0x1006 / 0x1007, later pinned to core 2) the signal is lost: at boot when worker 1 went to sleep before worker 0, and later as a job left in worker 0's queue for 12 s.
|
||||
- The thread dump's `wait #` (the order condition waiters arrived in, which `SignalProcessWideKey` wakes them in at equal priority) showed the order in every hung run.
|
||||
- **Current:** the game's own wait wrapper (`main+0x12500`, `{event, pending}`) signals the event once more when it takes a nonzero pending word after waking, and nothing in `main` set it. The build cut makes the signal set it first, so a worker woken in another's place passes the wake on (in a one-worker pool the extra signal costs one empty loop): 6 of 6 runs boot.
|
||||
- A host change that made `StartThread` return only once the new thread runs (as a free Switch core takes it within microseconds) brought the hangs to 1 in 6 and was dropped.
|
||||
- An experimental host change that made `StartThread` return only once the new thread ran brought the hangs to 1 in 6 and was dropped. That timing change did not establish a kernel guarantee.
|
||||
- **Confirmed, 2026-10-26:** this is a guest queue-wake assumption, not incorrect host condition-wake ordering. The workers are priority 46 versus the main thread's 44 and initially have mask 0x7. The separate [same-core start correction](../../../docs/reference/threads.md#starting-a-higher-priority-thread-on-the-callers-core) therefore does not apply to these starts. With that correction, an unpatched copy hung in 2 of 6 ten-second Mac boots: main at `main+0x1426c`, workers 0x1006 / 0x1007 asleep on the same condition, lock free. The registered cut reached frame 300 in all 6 matching boots. The instruction replay checks one worker and both two-worker arrival orders, targeting each queue: both wrong-worker originals fail and all patched cases pass. See [the replay command](../../../docs/agents/05-debugging.md#threading-cut-regressions). The kernel evidence and distinction between SDK and kernel events are in [the reference entry](../../../docs/reference/threads.md#one-auto-clear-event-shared-by-a-pools-workers).
|
||||
|
||||
## Rendering
|
||||
|
||||
@@ -49,6 +50,7 @@ General host changes first needed here; each applies to every game:
|
||||
- **GLSLC outputs in any RomFS file no other reader takes** (`shader_split.py`): [GLSLC outputs in `main` and in loose files](../../../docs/reference/shaders.md#glslc-outputs-in-main-and-in-loose-files).
|
||||
- **CPU writes into uncached pools uploaded** (`tex_dirty_upload` with `tex_flush_upload`), **integer views of image slots** (`img_fit`), **fragment barriers that end a pass after shader writes** (`swrite_note`, `nvn_refl.m`), **whole clears by channel mask** (`mtl_pf_chmask`) and the **new NVN entry points**: [CPU writes with no flush, caught by page protection (`tex_dirty_upload`)](../../../docs/reference/nvn-objects.md#cpu-writes-with-no-flush-caught-by-page-protection-tex_dirty_upload), [Integer views for images a shader declares uint or int](../../../docs/reference/rendering.md#integer-views-for-images-a-shader-declares-uint-or-int), [Fragment-ordering barriers between draws that write images](../../../docs/reference/rendering.md#fragment-ordering-barriers-between-draws-that-write-images), [A channel mask that covers every channel is a whole clear](../../../docs/reference/rendering.md#a-channel-mask-that-covers-every-channel-is-a-whole-clear), [Plural texture binds, builder-registered samplers, strided texel writes, device item 0x67](../../../docs/reference/nvn-objects.md#plural-texture-binds-builder-registered-samplers-strided-texel-writes-device-item-0x67).
|
||||
- **A job wake lost between two workers on one auto-clear event** (thread dump `wait #`): [One auto-clear event shared by a pool's workers](../../../docs/reference/threads.md#one-auto-clear-event-shared-by-a-pools-workers).
|
||||
- **Priority handoff when starting a same-core child**, found during the audit but not the cause of this game's hang: [Starting a higher-priority thread on the caller's core](../../../docs/reference/threads.md#starting-a-higher-priority-thread-on-the-callers-core).
|
||||
|
||||
## Open
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
CUTS = {
|
||||
# FINAL FANTASY TACTICS - The Ivalice Chronicles 1.4.0 (v393216) main: the job system wakes one worker of a pool through the pool's single auto-clear event. A job goes into worker k's
|
||||
# own queue (main+0x154c0), and if k's state reads asleep the caller signals the event (main+0x12554, from the per-worker dispatch loops main+0x14000 / main+0x14350 / main+0x17670,
|
||||
# the single-job submits main+0x14640 / main+0x14790 / main+0x14dc0 and the shutdown main+0x13df0). The kernel wakes the waiter that slept first, which is k only while the workers sleep in index order; the woken worker runs only its own queue
|
||||
# the single-job submits main+0x14640 / main+0x14790 / main+0x14dc0 and the shutdown main+0x13df0). The SDK event wakes one condition waiter, by priority then FIFO (matching the host and Horizon); it cannot target k's queue. The woken worker runs only its own queue
|
||||
# (worker loop main+0x15ce0) and sleeps again, so with two workers in one pool (threads 0x1006 / 0x1007, later pinned to core 2) the signal went to the wrong one: the main thread waited
|
||||
# forever on a dispatch at boot in 4 of 6 Mac runs, and in a run that got through, a job sat in worker 0's queue for 12 s. The wait wrapper (main+0x12500, {event, pending}) already
|
||||
# signals the event once more when it takes a nonzero pending word after waking, and nothing in main sets it: the signal now sets it first, so a worker woken in another's place
|
||||
|
||||
@@ -33,9 +33,10 @@ Game-specific findings; the general host changes they led to are listed under [F
|
||||
|
||||
## Physics
|
||||
|
||||
- **Resolved: the physics workers crashed when a conversation or cutscene began** (`pc=main+0x10ccf28`, `lr=main+0x13c236c`; on the iPhone and in 2 of 5 Mac runs, in every Mac run under CPU load). The physics step (`main+0x15730`, main thread) rebuilds each world's PhysX completion task in place (a `PxLightCpuTask` at world+0xc8) before every `simulate`. A worker runs it (`main+0x10ccf80`: `fetchResults`, world+0x1d8 done = 1, notify, unlock) and then calls its `release` (`main+0x10cd070`), which calls the task's own destructor (vtable back to the `PxLightCpuTask` base, `mTm = 0`, `main+0x10cced0`). At 30 fps the game steps physics twice per frame (step, wait, step, wait); the unlock hands the world's mutex to the woken main thread, and when the worker is descheduled there the main thread's next step rebuilds the task before the late destructor runs, which then wipes the new one. The next simulation's `NpSceneCompletion::release` reads `mTm = 0`. On the Switch the release follows the unlock within nanoseconds.
|
||||
- **Resolved: the physics workers crashed when a conversation or cutscene began** (`pc=main+0x10ccf28`, `lr=main+0x13c236c`; on the iPhone and in 2 of 5 Mac runs, in every Mac run under CPU load). The physics step (`main+0x15730`, main thread) rebuilds each world's PhysX completion task in place (a `PxLightCpuTask` at world+0xc8) before every `simulate`. A worker runs it (`main+0x10ccf80`: `fetchResults`, world+0x1d8 done = 1, notify, unlock) and then calls its `release` (`main+0x10cd070`), which calls the task's own destructor (vtable back to the `PxLightCpuTask` base, `mTm = 0`, `main+0x10cced0`). At 30 fps the game steps physics twice per frame (step, wait, step, wait); the unlock hands the world's mutex to the woken main thread, and when the worker is descheduled there the main thread's next step rebuilds the task before the late destructor runs, which then wipes the new one. The next simulation's `NpSceneCompletion::release` reads `mTm = 0`.
|
||||
- **Current:** the release becomes a plain `ret`: the destructor only resets the vtable and `mTm` (no resources), the step rebuilds every field, and this release is used by this one vtable (`main+0x1c51308` slot 7) only.
|
||||
- How it was found: 12 busy shell loops beside a scripted Mac run made the crash happen in every run (and the frame rate switch between 60 and 30 as on the phone), and observation gates (`OBS`) on the functions involved gave the order of events ([05, Racy crashes](../../../docs/agents/05-debugging.md#racy-crashes-load-trace-gates)).
|
||||
- **Confirmed, 2026-10-26:** the recorded crash has priority-44 physics workers on cores 1 / 2 (masks 0x2 / 0x4) and a priority-44 main thread with mask 0x7. Their overlap is legal; no same-core priority handoff can protect the task. A controlled replay of the original constructor stores (`main+0x157a4` and `0x157dc`), relocated destructor vtable, release and `removeReference` reproduces the exact null load at `main+0x10ccf28` when the prior release follows reconstruction. Release-before-rebuild passes, as does the late release with the registered cut. This confirms a guest lifetime race; the earlier claim about nanosecond Switch timing was not measured and supplies no synchronization guarantee. The cut remains build-gated and justified. See [the replay command](../../../docs/agents/05-debugging.md#threading-cut-regressions) and [the lifetime contract](../../../docs/reference/threads.md#late-physx-task-destructor-after-the-unlock).
|
||||
|
||||
## Services and SDK 8
|
||||
|
||||
|
||||
@@ -6,8 +6,8 @@ CUTS = {
|
||||
# release (main+0x10cd070), which calls the task's own destructor: vtable back to the PxLightCpuTask base, mTm = 0 (main+0x10cced0). At 30 fps the game steps twice
|
||||
# per frame (step, wait, step, wait); the unlock hands the world's mutex to the woken main thread, and when the worker is descheduled there the main thread's
|
||||
# next step rebuilds the task before the late destructor runs, which then wipes the new task. The next simulation's NpSceneCompletion::release reads its mTm = 0:
|
||||
# crash at main+0x10ccf28 (lr main+0x13c236c; seen on the iPhone and in 2 of 5 Mac runs, in every Mac run under CPU load). On the Switch the release follows the
|
||||
# unlock within nanoseconds. The release becomes a plain ret: the destructor only resets the vtable and mTm (no resources), the step rebuilds every field, and this
|
||||
# crash at main+0x10ccf28 (lr main+0x13c236c; seen on the iPhone and in 2 of 5 Mac runs, in every Mac run under CPU load). The workers can run on other cores:
|
||||
# unlock does not protect their later destructor writes. Actual-instruction replay confirms the guest lifetime race. The release becomes a plain ret: the destructor only resets the vtable and mTm (no resources), the step rebuilds every field, and this
|
||||
# release is used by this one vtable (main+0x1c51308 slot 7) only
|
||||
'909e904af78ac1b8deefe97ab2ccdb51': [
|
||||
(0x10cd070, '080040f9', 'c0035fd6', 'physics completion task release (calls its own destructor): ldr x8,[x0] → ret (no longer destroyed; the step function rebuilds it in place every step)'),
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Replay the two registered threading cuts against the user's original NSOs.
|
||||
|
||||
Usage: python3 dm/tools/check_thread_cuts.py <FFT main NSO> <Link's Awakening main NSO>
|
||||
Requires lz4 and unicorn. No game bytes are stored here or written to disk.
|
||||
The default checks the negative controls and the registered fixes; --original
|
||||
asserts the original programs are safe and deliberately fails on both bugs.
|
||||
|
||||
This is a controlled instruction replay, not a guest scheduler or a gameplay
|
||||
test. FFT's SDK event calls are replaced with a FIFO auto-clear event; Link's
|
||||
Awakening is paused between completion publication and its later release.
|
||||
"""
|
||||
import argparse
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import struct
|
||||
|
||||
from unicorn import Uc, UcError, UC_ARCH_ARM64, UC_MODE_ARM, UC_HOOK_CODE
|
||||
from unicorn.arm64_const import UC_ARM64_REG_X0, UC_ARM64_REG_X1, UC_ARM64_REG_X22, UC_ARM64_REG_SP, UC_ARM64_REG_LR, UC_ARM64_REG_PC
|
||||
|
||||
from nso2elf import load
|
||||
|
||||
BASE, STATE, STACK, STOP = 0x10000000, 0x40000000, 0x50000000, 0x60000000
|
||||
GAMES = Path(__file__).resolve().parents[1] / 'games'
|
||||
|
||||
|
||||
class Replay:
|
||||
def __init__(self, path, game, patched):
|
||||
img, _, bss, bid = load(path)
|
||||
spec = importlib.util.spec_from_file_location('thread_cuts', GAMES / game / 'patches.py')
|
||||
cuts = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(cuts)
|
||||
build = bid[:16].hex()
|
||||
if build not in cuts.CUTS:
|
||||
raise ValueError(f'{path}: unregistered {game} main build {build}')
|
||||
self.vm = Uc(UC_ARCH_ARM64, UC_MODE_ARM)
|
||||
self.vm.mem_map(BASE, (len(img) + bss + 0xfff) & ~0xfff)
|
||||
self.vm.mem_write(BASE, img)
|
||||
for off, before, after, _ in cuts.CUTS[build]:
|
||||
old = bytes.fromhex(before)
|
||||
if img[off:off + len(old)] != old:
|
||||
raise ValueError(f'{path}: original instructions differ at {off:#x}')
|
||||
if patched:
|
||||
self.vm.mem_write(BASE + off, bytes.fromhex(after))
|
||||
# Relocate the original vtables, including the completion destructor.
|
||||
mod = struct.unpack_from('<I', img, 4)[0]
|
||||
dyn = mod + struct.unpack_from('<i', img, mod + 4)[0]
|
||||
tags = {}
|
||||
while True:
|
||||
tag, value = struct.unpack_from('<qQ', img, dyn)
|
||||
dyn += 16
|
||||
if not tag:
|
||||
break
|
||||
tags[tag] = value
|
||||
for off in range(tags[7], tags[7] + tags[8], 24):
|
||||
addr, info, addend = struct.unpack_from('<QQq', img, off)
|
||||
if info & 0xffffffff == 1027: # R_AARCH64_RELATIVE
|
||||
self.put(BASE + addr, BASE + addend)
|
||||
for addr in (STATE, STACK, STOP):
|
||||
self.vm.mem_map(addr, 0x10000)
|
||||
|
||||
def put(self, addr, value):
|
||||
self.vm.mem_write(addr, struct.pack('<Q', value))
|
||||
|
||||
def get(self, addr):
|
||||
return struct.unpack('<Q', self.vm.mem_read(addr, 8))[0]
|
||||
|
||||
def call(self, off, arg):
|
||||
self.vm.reg_write(UC_ARM64_REG_X0, arg)
|
||||
self.vm.reg_write(UC_ARM64_REG_LR, STOP)
|
||||
self.vm.reg_write(UC_ARM64_REG_SP, STACK + 0xfff0)
|
||||
self.vm.emu_start(BASE + off, STOP, count=1000)
|
||||
if self.vm.reg_read(UC_ARM64_REG_PC) != STOP:
|
||||
raise AssertionError(f'instruction budget exhausted at {off:#x}')
|
||||
|
||||
|
||||
def fft(path, patched):
|
||||
r = Replay(path, '010038B015560000', patched)
|
||||
vm = r.vm
|
||||
wrapper, event = STATE, STATE + 0x100
|
||||
r.put(wrapper, event)
|
||||
waiters, ready = [], []
|
||||
signals = 0
|
||||
|
||||
def sdk(uc, pc, size, data):
|
||||
nonlocal signals
|
||||
if uc.reg_read(UC_ARM64_REG_X0) != event:
|
||||
raise AssertionError('event wrapper passed the wrong event pointer')
|
||||
if pc == BASE + 0x651630:
|
||||
signals += 1
|
||||
if waiters:
|
||||
ready.append(waiters.pop(0))
|
||||
uc.reg_write(UC_ARM64_REG_PC, uc.reg_read(UC_ARM64_REG_LR))
|
||||
|
||||
for off in (0x651630, 0x651640):
|
||||
vm.hook_add(UC_HOOK_CODE, sdk, begin=BASE + off, end=BASE + off)
|
||||
results = []
|
||||
for order in ([0], [0, 1], [1, 0]):
|
||||
for target in order:
|
||||
waiters[:] = order
|
||||
ready.clear()
|
||||
signals = 0
|
||||
r.put(wrapper + 8, 0)
|
||||
r.call(0x12554, wrapper)
|
||||
done = False
|
||||
while ready:
|
||||
worker = ready.pop(0)
|
||||
r.call(0x12500, wrapper)
|
||||
if worker == target:
|
||||
done = True
|
||||
else:
|
||||
waiters.append(worker)
|
||||
if signals > 4:
|
||||
raise AssertionError('wake relay did not quiesce')
|
||||
print(f'FFT {"patched" if patched else "original"}: waiters={order}, job={target}, done={done}, signals={signals}')
|
||||
results.append(done)
|
||||
return all(results)
|
||||
|
||||
|
||||
def awakening(path, patched, late):
|
||||
r = Replay(path, '01006BB00C6F0000', patched)
|
||||
vm = r.vm
|
||||
world, manager, manager_vtable = STATE, STATE + 0x1000, STATE + 0x2000
|
||||
task = world + 0xc8
|
||||
r.put(manager, manager_vtable)
|
||||
r.put(manager_vtable + 0xa0, STOP)
|
||||
|
||||
def rebuild():
|
||||
# Actual in-place construction and the subsequent mTm assignment in
|
||||
# the first world's step. The scene's getter supplies our manager.
|
||||
vm.reg_write(UC_ARM64_REG_X22, world)
|
||||
vm.emu_start(BASE + 0x157a4, BASE + 0x157c0, count=20)
|
||||
vm.reg_write(UC_ARM64_REG_X0, manager)
|
||||
vm.emu_start(BASE + 0x157dc, BASE + 0x157e0, count=1)
|
||||
assert r.get(task) == BASE + 0x1c51308 and r.get(task + 0x10) == manager
|
||||
|
||||
rebuild()
|
||||
if late:
|
||||
# The preceding worker has unlocked; main has constructed generation
|
||||
# two, but generation one's release has yet to run on another thread.
|
||||
rebuild()
|
||||
r.call(0x10cd070, task)
|
||||
if not late:
|
||||
rebuild()
|
||||
try:
|
||||
r.call(0x10ccf20, task) # the following simulation removes a reference
|
||||
ok = vm.reg_read(UC_ARM64_REG_X0) == manager and vm.reg_read(UC_ARM64_REG_X1) == task
|
||||
except UcError:
|
||||
if vm.reg_read(UC_ARM64_REG_PC) != BASE + 0x10ccf28 or r.get(task + 0x10) != 0:
|
||||
raise
|
||||
ok = False
|
||||
print(f'Awakening {"patched" if patched else "original"}: late={late}, mTm={r.get(task + 0x10):#x}, safe={ok}')
|
||||
return ok
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument('fft_main', type=Path)
|
||||
ap.add_argument('awakening_main', type=Path)
|
||||
ap.add_argument('--original', action='store_true')
|
||||
args = ap.parse_args()
|
||||
original_fft = fft(args.fft_main, False)
|
||||
# A normal lifetime passes without the cut. Only the late release fails.
|
||||
normal = awakening(args.awakening_main, False, False)
|
||||
original_awakening = awakening(args.awakening_main, False, True)
|
||||
if args.original:
|
||||
return 0 if original_fft and normal and original_awakening else 1
|
||||
fixed_fft = fft(args.fft_main, True)
|
||||
fixed_normal = awakening(args.awakening_main, True, False)
|
||||
fixed_late = awakening(args.awakening_main, True, True)
|
||||
ok = not original_fft and normal and not original_awakening and fixed_fft and fixed_normal and fixed_late
|
||||
print('PASS: both negative controls fail and both registered fixes pass' if ok else 'FAIL: threading cut regression')
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -110,6 +110,16 @@ A crash that needs an unlucky thread interleaving (one run in three, or once in
|
||||
|
||||
To see the order of events, put temporary observation gates (`OBS` in the profile's `patches.py`, only at function entries or right after a `bl`, on position-independent instructions) on the functions involved. Their host callbacks append `(time, thread, event, pointers)` to an in-memory ring, and the crash handler prints the last ~150 entries. Printing each event as it happens is too slow and changes the timing. A function whose first instruction is PC-relative (an `adrp`, typical of short destructors) cannot take a gate: a temporary cut can put `svc #0x8f` at its entry and move its body into the `udf` padding after it (inside the same 4 KB page, an `adrp` keeps its encoding), with `svc_host_call` handling the number. Inside the host callback, the caller's return address is on the guest stack, in the frame the gate pushed (`regs + 800 + 8`), because the gate's own call has replaced `x30`. Remove all of it afterwards.
|
||||
|
||||
## Threading cut regressions
|
||||
|
||||
`dm/tools/check_thread_cuts.py` replays the registered FFT 1.4.0 and Link's Awakening 1.1.0 cuts against original NSOs without storing game bytes in the repository. Use a Python with `lz4` and `unicorn` (on Apple silicon Unicorn's JIT needs execution outside the agent sandbox):
|
||||
|
||||
```bash
|
||||
python3 dm/tools/check_thread_cuts.py "<FFT ExeFS>/main" "<Link's Awakening ExeFS>/main"
|
||||
```
|
||||
|
||||
It verifies build IDs and original instructions, relocates vtables, then checks that the original wrong-worker wake and late destructor fail while the registered cuts pass. `--original` instead asserts the originals are safe and exits nonzero on these failures. This is a controlled instruction replay: FFT's SDK event is a FIFO model, and the physics scene getter supplies a test manager. It does not reproduce the full kernel or gameplay. `make -C dm/host/tests GAME=010038B015560000 preempt` independently checks actual host condition priority/FIFO, mutex reacquisition, start and wake handoffs. Keep the game-specific observations in their notes.
|
||||
|
||||
## Memory growth
|
||||
|
||||
1. The host's memory report line (`memory up to frame …`) every few hundred frames; `footprint -p <pid>` from outside. On a phone, a crash with no log usually means `headroom to the system memory limit` reached 0.
|
||||
|
||||
@@ -74,8 +74,9 @@ Guest threads, their synchronization and how the host schedules them. Each entry
|
||||
### Late PhysX task destructor after the unlock
|
||||
|
||||
- **Symptom:** a PhysX worker crashes in a task's `removeReference` with a null task manager, right as the game switches to 30 fps (`pc=main+0x10ccf28`, `lr=main+0x13c236c`); on the iPhone and in 2 of 5 Mac runs, in every Mac run with 12 busy loops loading the CPU.
|
||||
- **Cause:** a destructor that runs a moment too late. The game rebuilds each physics world's completion task in place before every `simulate` (`main+0x15730`, main thread). A worker runs the task (`fetchResults`, sets the world's done flag, notifies, unlocks) and then calls its `release`, which calls the task's own destructor (vtable back to the `PxLightCpuTask` base, `mTm = 0`). At 30 fps the game steps twice per frame, so the unlock wakes a main thread that steps again at once. On the Switch the `release` follows the unlock within nanoseconds; on Apple devices the worker can be descheduled between them, the main thread rebuilds the task, and the late destructor wipes the new one. The next simulation's `NpSceneCompletion::release` then dereferences `mTm`.
|
||||
- **Cause:** the game publishes completion before task teardown ends, then reuses the same task. Its main thread rebuilds each world's completion task before `simulate`; the worker sets done, notifies and unlocks inside `run`, then calls `release`, whose destructor clears `mTm`. Another core may rebuild it in between. This is a guest lifetime race: an unlock cannot protect subsequent destructor writes. PhysX's [task contract](https://github.com/NVIDIAGameWorks/PhysX-3.4/blob/master/PxShared/include/task/PxTask.h) places recycling in `release`, after `run`, and its `PxLightCpuTask` destructor clears `mTm`. There is no measured Switch timing guarantee that makes the ordering safe. The game's offsets, affinities and replay evidence are in its note.
|
||||
- **Fix:** a per-build cut makes that one `release` a `ret` (its `patches.py`); the destructor frees nothing and the step rewrites every field.
|
||||
- **Check:** `dm/tools/check_thread_cuts.py` replays the original construction, release and `removeReference` instructions with a forced late release: the original faults at the recorded null-manager load; normal release-before-rebuild succeeds; the registered cut preserves the new task. This controlled replay has no host mutex or graphics implementation in its failure path.
|
||||
- **Found on:** [The Legend of Zelda: Link's Awakening](../../dm/games/01006BB00C6F0000/README.md). How it was found is in [05](../agents/05-debugging.md#racy-crashes-load-trace-gates).
|
||||
|
||||
## Yields and spin waits
|
||||
@@ -111,10 +112,11 @@ The yield back-off (`DM_YIELD_BACKOFF`) and the rewriter's spin waits (`patch_ns
|
||||
### One auto-clear event shared by a pool's workers
|
||||
|
||||
- **Symptom:** the main thread waits forever on a job's done event at boot while every worker sleeps, two of them on one event (`main+0x1426c` waiting on `Job::m_doneEvent`); 4 of 6 Mac runs.
|
||||
- **Cause:** the engine's job pools wake one worker through the pool's single auto-clear event. A job goes into worker k's own queue (`main+0x154c0`), and if k's state reads asleep the caller signals the event (`main+0x12554`); the kernel wakes the waiter that slept first, which is k only while the workers sleep in index order. A worker woken for another's job runs only its own queue and sleeps again, so with two workers in one pool the signal is lost: at boot when worker 1 went to sleep before worker 0, and later as a job left in worker 0's queue for 12 s. The thread dump's `wait #` (the order condition waiters arrived in, which `SignalProcessWideKey` wakes them in at equal priority) showed the order in every hung run.
|
||||
- **Cause:** the engine puts a job in worker k's private queue but signals one shared SDK auto-clear event. Its condition wake chooses by priority, then FIFO; it has no target-worker identity. The wrong worker checks only its own queue and sleeps again. This is a guest assumption about waiter order, exposed by host timing. Host `pick` / `wseq` agrees with Atmosphère's [condition-tree comparator](https://github.com/Atmosphere-NX/Atmosphere/blob/master/libraries/libmesosphere/include/mesosphere/kern_k_thread.hpp) and [`Signal`](https://github.com/Atmosphere-NX/Atmosphere/blob/master/libraries/libmesosphere/source/kern_k_condition_variable.cpp), including the historic 0.19.0 implementation. This describes the SDK's condition-based auto-clear event; kernel readable events instead notify all synchronization waiters. Game offsets and observations belong in its note.
|
||||
- **Fix:** the game's own wait wrapper (`main+0x12500`, `{event, pending}`) signals the event once more when it takes a nonzero pending word after waking, and nothing set it: a build cut makes the signal set it first (its `patches.py`), so a worker woken in another's place passes the wake on; 6 of 6 runs boot.
|
||||
- **Found on:** [FINAL FANTASY TACTICS - The Ivalice Chronicles](../../dm/games/010038B015560000/README.md).
|
||||
- **Note:** a host change that made `StartThread` return only once the new thread runs (as a free Switch core takes it within microseconds) brought the hangs to 1 in 6 and was dropped.
|
||||
- **Check:** `t_preempt` checks priority/FIFO selection and mutex reacquisition through real condition SVCs. `dm/tools/check_thread_cuts.py` runs the game's actual signal and wait wrappers with both worker arrival orders: only the original wrong-worker cases strand a job. The game's note records the Mac boot comparison with the corrected host.
|
||||
- **Note:** an unconditional `StartThread` child-entry handshake changed the timing but did not eliminate the hang and is not a kernel guarantee. The separate [same-core start handoff](#starting-a-higher-priority-thread-on-the-callers-core) applies only when priority and affinity require it.
|
||||
|
||||
### The oe notification event and its queue
|
||||
|
||||
|
||||
Reference in new issue
Block a user