diff --git a/README.md b/README.md index 3a7f560..aff40e8 100644 --- a/README.md +++ b/README.md @@ -9,11 +9,11 @@ Same approach as d2r-mobile: the game's own aarch64 code runs natively. A rewrit | Step | State | |---|---| | Rewriter (`dm/tools/patch_nso.py`, `DM_RULESET=native`) | Runs unchanged on Mario `main` / `sdk` | -| Mac host | **Reaches the title screen and renders it correctly at 60 fps** (deferred G-buffer, shadow map, post passes, layout UI, final copy into the window). Memory stays flat (about 900 MB physical footprint) | -| Shaders | All 12,811 shaders the game ships (bfsha archives, particle effects, engine `agl` packages, UI layouts, `main`) are extracted by `shader_split.py` and translated offline; 12,804 compile. 7 `agl` post-effect compute shaders with image atomics make SPIRV-Cross throw | -| Input | Scripted input `DM_KEYS="12:A,14:A"` (seconds:button); `DM_WINDOW=1` for a window with keyboard input. Pressing A on the title screen starts a new game and loads the opening course | -| Current frontier | Loading the opening course crashes: the resource heap of `Model/NpcCommonerA.bfres` (sized from the resource-size table) overflows when its model instances are created, the game gets a null pool and dereferences it (`h_nvnMemoryPoolGetFlags`). Under investigation with `DM_PROBE` | -| Known gaps | Partial / mid-pass clears; sparse virtual-pool mappings; conditional render; TBDR optimizations are off by default (they don't understand bindless textures yet) | +| Mac host | **Plays through the menus (player count, character select) into the opening course**: the course, effects, HUD, menu and dialogue text render in Chinese or English. Title screen at 60 fps, memory flat (about 900 MB physical footprint on the title screen) | +| Shaders | All 12,811 shaders the game ships (bfsha archives, particle effects, engine `agl` packages, UI layouts, `main`) are extracted by `shader_split.py` and translated offline; 12,809 compile. 2 `agl` post-effect compute shaders make SPIRV-Cross throw | +| Input | Scripted input `DM_KEYS="12:A,14:A,20:A,26:A,32:A"` (seconds:button) reaches the opening course; `DM_WINDOW=1` for a window with keyboard input | +| Current frontier | In the opening course the sky is black and no character models (player, NPCs) are visible; controller-button icons in text show as `?` (the host's stand-in system font has no Nintendo-extension glyphs) | +| Known gaps | Partial / mid-pass clears; sparse virtual-pool mappings; conditional render; `DrawElementsIndirect`; TBDR optimizations are off by default (they don't understand bindless textures yet); resource heaps get +16 KB slack because one model's setup came up 0x70 bytes short of the size table (cause unknown) | | Audio | Silent AudioRenderer (`hle_audren.c`); real mixing comes later | | iOS bundle | Not started. Plan: reuse `dm/ios` from d2r-mobile, bundle the RomFS (3.5 GB) inside the app, standard-controller virtual gamepad | @@ -41,15 +41,21 @@ Same approach as d2r-mobile: the game's own aarch64 code runs natively. A rewrit - Texture swizzle (`TextureBuilderSetSwizzle` / `TextureViewSetSwizzle`) is applied through Metal texture views (the font atlas is `(1, 1, 1, R)`). - Non-packaged textures are shared by storage (pool + offset + description): Mario rebuilds its transient render targets on new objects every frame and never finalizes them, which leaked about 13 MB of GPU memory per frame. - TBDR optimizations are off by default (`DM_NVN_TBDR=1` turns them on) until they recognize bindless readers. + - Linear textures in CPU-visible pools are created directly on the pool buffer (zero copy). Mario's glyph cache (2048×8192 R8) is filled by the CPU; as a private GPU texture all menu and dialogue text was blank. + - Host command-memory segments (lent when a command buffer outgrows the memory the game gave it) grow on demand instead of a fixed 64 × 1 MB. + - `RGB10A2SN` vertex format; instanced `DrawArrays`. - Shader chain: - `dm/gfx/tools/shader_split.py` (new) scans RomFS and `main` for BNSH and BAHS (agl `sharcb`) containers and writes the package directory, the shader map and the answer table (compute thread-group size from control +0x748, shared memory from +0x754). - The translator uses GL depth (`-1..1`) and no depth clip control, so vertex shaders remap z' = z/2 + w/2: Mario sets `nvnDeviceSetDepthMode` to near = -w, and without the remap its full-screen copy was clipped away. - - `dmsh_build.py` strips the 0x30 header of compute code areas (magic 0x12345679). + - Compute code areas (magic 0x12345679) start their instructions at +0x100 (0x30 header + zero padding); `dmsh_build.py` strips all of it. Stripping only 0x30 misaligned the translator's 32-byte instruction groups: 12 particle / post-effect compute shaders became endless loops and hung the GPU at the first gameplay scene. - Host: - Guest address space repartitioned: alias region 2.875 GB, heap 2.625 GB. Mario reserves 1.75 GB of `VirtualAddressMemory` (HeapDef) that it maps into the alias region on demand; with 1.25 GB, large layout archives got no address and were skipped. - Threads created with ideal core -2 run on the process default core (0); `GetCurrentProcessorNumber` never returns a negative core. Mario's font engine indexes per-core state with it. - The shared font is served with a BFTTF "already decoded" header so `ScalableFontEngineHelper::Decode` leaves it alone. - Crash reports scan the stack for return addresses and dump the popped frame. + - `DM_PROBE` (`host/probe.c`): register-relative dumps (`@xN+off`), hit filters (`?[xN+off]=val`), a `sead::ExpHeap` walker (`!xN`), thread IDs, `DM_PROBE_ALL`. +- `patch_nso.py` code cuts for Mario `main` (build ID `cd6e42ae…`): resource heaps (`main+0x1e62fc`) get +16 KB. +- `dm/tools/ghidra/Decompile.java`: headless decompile by module offset; `@off` (re)creates functions reached only through vtables and undoes Ghidra's wrong "non-returning" mark on `nn::os` imports. ## Build and run on Mac diff --git a/dm/gfx/nvn/c1_full.tsv b/dm/gfx/nvn/c1_full.tsv index da3e2e9..88d8ee3 100644 --- a/dm/gfx/nvn/c1_full.tsv +++ b/dm/gfx/nvn/c1_full.tsv @@ -1,4 +1,4 @@ -# chain b-7beebd7b9322 2026-10-13T00:36:40+08:00 +# chain b-1b3a96e6a7d3 2026-10-13T08:05:33+08:00 35cbc686dd8461b8 4872bec9a5567f8b 54a618792b2d5f0f diff --git a/dm/gfx/nvn/dmsh_chain.h b/dm/gfx/nvn/dmsh_chain.h index 1e590b8..03dbf63 100644 --- a/dm/gfx/nvn/dmsh_chain.h +++ b/dm/gfx/nvn/dmsh_chain.h @@ -1,2 +1,2 @@ // 生成文件(dm/gfx/tools/dmsh_chain.py --header):本源码树期望的 B 着色器库翻译链版本(缺省选项);宿主只加载 <库目录>/b-/ -#define DMSH_CHAIN "7beebd7b9322" +#define DMSH_CHAIN "1b3a96e6a7d3" diff --git a/dm/gfx/nvn/hle_nvn.c b/dm/gfx/nvn/hle_nvn.c index b85f09b..e36aa18 100644 --- a/dm/gfx/nvn/hle_nvn.c +++ b/dm/gfx/nvn/hle_nvn.c @@ -454,22 +454,30 @@ static void cmd_new_segment(cmd_t *o, uint8_t *mem, size_t n) { // Host-supplied segment when the game's own command memory runs out: when the game's out-of-memory callback cannot allocate from its command memory pool, it calls AddCommandMemory with an empty pool (p = NULL, n = 0) // (opening the advanced stats page on the level 88 sorceress save: many list items in one frame, and the original implementation segfaulted here reading NULL+8). Our records are larger than real-device NVN commands, so the budget the game reserved for real-device usage is not necessarily enough, // so this is not an error state of the game but something we must catch: provide a segment of host memory to keep recording. The segment is replayed in place at submit and reclaimed after being kept for 8 frames (submit is in the same frame as recording or the next one). -#define HOSTSEG_SZ (1u << 20) -#define HOSTSEG_MAX 64 +// 段池按需增长(SMBW 开场关:很多小命令缓冲每帧各溢出一次,旧版固定 64 段 × 1 MB 在 8 帧内用完);总量超过 HOSTSEG_CAP 才中止。 +// The segment pool grows on demand (SMBW opening course: many small command buffers each overflow once per frame, and the old fixed 64 segments × 1 MB ran out within 8 frames); abort only beyond HOSTSEG_CAP in total. +#define HOSTSEG_SZ (256u << 10) +#define HOSTSEG_CAP (1u << 30) uint64_t dm_nvn_frame_count(void); -static struct { uint8_t *mem; uint64_t frame; int busy; } g_hostseg[HOSTSEG_MAX]; +typedef struct { uint8_t *mem; uint64_t frame; int busy; } hostseg_t; +static hostseg_t *g_hostseg; static int g_nhostseg, g_caphostseg; static pthread_mutex_t g_hostseg_mu = PTHREAD_MUTEX_INITIALIZER; static uint8_t *hostseg_take(void) { static unsigned total; uint64_t f = dm_nvn_frame_count(); uint8_t *m = NULL; pthread_mutex_lock(&g_hostseg_mu); - for (int i = 0; i < HOSTSEG_MAX; i++) if (g_hostseg[i].busy && g_hostseg[i].frame + 8 < f) g_hostseg[i].busy = 0; - for (int i = 0; i < HOSTSEG_MAX && !m; i++) if (!g_hostseg[i].busy) { - if (!g_hostseg[i].mem) g_hostseg[i].mem = malloc(HOSTSEG_SZ); - g_hostseg[i].busy = 1; g_hostseg[i].frame = f; m = g_hostseg[i].mem; } - unsigned t = ++total; + for (int i = 0; i < g_nhostseg && !m; i++) { + if (g_hostseg[i].busy && g_hostseg[i].frame + 8 < f) g_hostseg[i].busy = 0; + if (!g_hostseg[i].busy) { g_hostseg[i].busy = 1; g_hostseg[i].frame = f; m = g_hostseg[i].mem; } + } + if (!m && (uint64_t)(g_nhostseg + 1) * HOSTSEG_SZ <= HOSTSEG_CAP) { + if (g_nhostseg == g_caphostseg) { g_caphostseg = g_caphostseg ? g_caphostseg * 2 : 64; g_hostseg = realloc(g_hostseg, sizeof *g_hostseg * (size_t)g_caphostseg); } + m = malloc(HOSTSEG_SZ); + g_hostseg[g_nhostseg++] = (hostseg_t){ m, f, 1 }; + } + unsigned t = ++total; int n = g_nhostseg; pthread_mutex_unlock(&g_hostseg_mu); - if (!m) die("NVN 命令内存:游戏池已空,宿主补段 %d 段也用完(8 帧内)", HOSTSEG_MAX); - if (t == 1 || t % 256 == 0) LOG("[nvn] 游戏的命令内存池已空,宿主补段(累计 %u 次,帧 %llu)", t, (unsigned long long)f); + if (!m) die("NVN 命令内存:游戏池已空,宿主补段已达 %u MB(8 帧内)", HOSTSEG_CAP >> 20); + if (t == 1 || t % 1024 == 0) LOG("[nvn] 游戏的命令内存不够,宿主补段(累计 %u 次,帧 %llu,段池 %d 段 × %u KB)", t, (unsigned long long)f, n, HOSTSEG_SZ >> 10); return m; } NVN(nvnCommandBufferAddCommandMemory, void, NVNcommandBuffer *c, const NVNmemoryPool *p, ptrdiff_t off, size_t n) { @@ -635,7 +643,7 @@ NVN(nvnCommandBufferSetStencilRef, void, NVNcommandBuffer *c, NVNface f, int v) NVN(nvnCommandBufferSetPolygonOffsetClamp, void, NVNcommandBuffer *c, float fa, float un, float cl) { *REC(c, OP_POLY_OFFSET, p_poly_offset) = (p_poly_offset){fa, un, cl}; } -NVN(nvnCommandBufferDrawArrays, void, NVNcommandBuffer *c, NVNdrawPrimitive pr, int first, int n) { *REC(c, OP_DRAW, p_draw) = (p_draw){pr, first, n}; } +NVN(nvnCommandBufferDrawArrays, void, NVNcommandBuffer *c, NVNdrawPrimitive pr, int first, int n) { *REC(c, OP_DRAW, p_draw) = (p_draw){pr, first, n, 1, 0}; } NVN(nvnCommandBufferDrawElementsInstanced, void, NVNcommandBuffer *c, NVNdrawPrimitive pr, NVNindexType it, int n, NVNbufferAddress ib, int bv, int bi, int inst) { *REC(c, OP_DRAW_IDX, p_draw_idx) = (p_draw_idx){pr, it, n, bv, bi, inst, ib}; @@ -1366,8 +1374,7 @@ NVN(nvnCommandBufferBindVertexBuffer, void, NVNcommandBuffer *c, int i, NVNbuffe NVN(nvnCommandBufferDrawElements, void, NVNcommandBuffer *c, NVNdrawPrimitive pr, NVNindexType it, int n, NVNbufferAddress ib) { h_nvnCommandBufferDrawElementsInstanced(c, pr, it, n, ib, 0, 0, 1); } NVN(nvnCommandBufferDrawElementsBaseVertex, void, NVNcommandBuffer *c, NVNdrawPrimitive pr, NVNindexType it, int n, NVNbufferAddress ib, int bv) { h_nvnCommandBufferDrawElementsInstanced(c, pr, it, n, ib, bv, 0, 1); } NVN(nvnCommandBufferDrawArraysInstanced, void, NVNcommandBuffer *c, NVNdrawPrimitive pr, int first, int n, int bi, int inst) { - if (inst != 1 || bi) ONCE_LOG("【红】DrawArraysInstanced 实例数 %d / 基实例 %d:宿主按 1 个实例画", inst, bi); - h_nvnCommandBufferDrawArrays(c, pr, first, n); + *REC(c, OP_DRAW, p_draw) = (p_draw){pr, first, n, inst, bi}; } NVN(nvnCommandBufferDrawElementsIndirect, void, NVNcommandBuffer *c, NVNdrawPrimitive pr, NVNindexType it, NVNbufferAddress ib, NVNbufferAddress ind) { (void)c; (void)pr; (void)it; (void)ib; (void)ind; ONCE_LOG("【红】DrawElementsIndirect:宿主未实现(跳过)"); diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index faf5460..e8201d2 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -763,7 +763,7 @@ void mtl_drain(void) { present_drop_for_suspend(); mtl_present_flush(); mtl_init // ================= 渲染回放 ================= // ================= Render replay ================= #include "../../host/front.h" -static int g_red_once[64]; +static int g_red_once[80]; #define RED_ONCE(k, ...) do { if (!g_red_once[k]++) MLOG("【红】" __VA_ARGS__); } while (0) // ---------- 贴图 → MTLTexture ---------- @@ -1194,6 +1194,26 @@ static id pkg_upload_(const tex_t *t, uint64_t *stamp) { up_flush(); return mt; } +// 线性贴图(LINEAR)放在 CPU 可见池里:游戏用 CPU 直接往池内存写(SMBW 的字形缓存:2048×8192 R8,IMAGE|LINEAR,池 CPU_UNCACHED|GPU_CACHED, +// 字体引擎在 CPU 上把字形画进去)。GPU 私有贴图看不到这些写入 → 菜单 / 对话框文字全空。统一内存下直接在池缓冲上建线性贴图(零拷贝),GPU 执行时读到当前内容,同 NVN 语义。 +// Linear textures (LINEAR) in a CPU-visible pool: the game writes the pool memory directly from the CPU (SMBW's glyph cache: 2048×8192 R8, IMAGE|LINEAR, pool CPU_UNCACHED|GPU_CACHED, +// the font engine rasterizes glyphs into it on the CPU). A GPU-private texture never sees those writes → all menu / dialog text is blank. On unified memory, create a linear texture directly on the pool buffer (zero copy): the GPU reads the current content at execution time, same as NVN. +static id tex_linear(const tex_t *t) { + if (!(t->d.flags & NVN_TEXTURE_FLAGS_LINEAR) || !t->mem || !t->pool || (t->pool->flags & NVN_MEMORY_POOL_FLAGS_CPU_NO_ACCESS)) return nil; + if (t->d.target != NVN_TEXTURE_TARGET_2D || t->d.levels > 1 || t->d.samples > 1) return nil; + uint32_t pf = 0; uint8_t swz[4]; int bw, bh, bpb = nvn_fmt_bpb(t->d.format, &bw, &bh); + if (!nvn_fmt_to_mtl(t->d.format, &pf, swz) || !pf || !bpb || bw != 1 || bh != 1) return nil; + size_t row = t->d.stride ? t->d.stride : (size_t)t->d.width * bpb; + void *mb; uint64_t mo; + if (!mtl_addr((uint64_t)t->mem, (uint64_t)row * (t->d.height ? t->d.height : 1), &mb, &mo)) { RED_ONCE(64, "线性贴图的存储不在内存池里(%ux%u 格式 %#x)", t->d.width, t->d.height, t->d.format); return nil; } + id b = (__bridge id)mb; + NSUInteger al = [g_dev minimumLinearTextureAlignmentForPixelFormat:(MTLPixelFormat)pf]; + if (mo % al || row % al) { RED_ONCE(65, "线性贴图偏移 %#llx / 行距 %zu 不满足 %lu 字节对齐,按私有贴图建(CPU 写入不可见)", (unsigned long long)mo, row, (unsigned long)al); return nil; } + MTLTextureDescriptor *d = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:(MTLPixelFormat)pf width:t->d.width ? t->d.width : 1 height:t->d.height ? t->d.height : 1 mipmapped:NO]; + d.resourceOptions = b.resourceOptions; + d.usage = MTLTextureUsageShaderRead | (t->d.flags & NVN_TEXTURE_FLAGS_IMAGE ? MTLTextureUsageShaderWrite : 0) | (t->d.flags & NVN_TEXTURE_FLAGS_LINEAR_RENDER_TARGET ? MTLTextureUsageRenderTarget : 0); + return [b newTextureWithDescriptor:d offset:mo bytesPerRow:row]; +} static id tex_mtl_(const tex_t *ct); static id tex_mtl(const tex_t *ct) { AP_B; id r = tex_mtl_(ct); AP_E(4); return r; } static id tex_mtl_(const tex_t *ct) { @@ -1246,11 +1266,15 @@ static id tex_mtl_(const tex_t *ct) { alias_key ak; memset(&ak, 0, sizeof ak); ak.pool = (uint64_t)(uintptr_t)t->pool; ak.off = t->off; ak.d = t->d; pthread_mutex_lock(&g_smu); __unsafe_unretained id shared = hm_get_id(&g_alias, &ak); pthread_mutex_unlock(&g_smu); if (shared) { serial_put(t->serial, shared); return shared; } - MTLTextureDescriptor *d = tex_desc(t, 0); - if (!d) return nil; - id mt = [g_dev newTextureWithDescriptor:d]; - if (!mt) { RED_ONCE(2, "建 MTLTexture 失败 %ux%u 格式 %#x 目标 %d", t->d.width, t->d.height, t->d.format, t->d.target); return nil; } + id mt = tex_linear(t); + if (!mt) { + MTLTextureDescriptor *d = tex_desc(t, 0); + if (!d) return nil; + mt = [g_dev newTextureWithDescriptor:d]; + if (!mt) { RED_ONCE(2, "建 MTLTexture 失败 %ux%u 格式 %#x 目标 %d", t->d.width, t->d.height, t->d.format, t->d.target); return nil; } + } if (g_allocprof > 0) { static int n; if (n++ < 30 || n % 500 == 0) MLOG("新建贴图(%s)#%d ser=%llu %ux%ux%u 格式 %#x 目标 %d 级 %u 存储 %p", __func__, n, (unsigned long long)t->serial, t->d.width, t->d.height, t->d.depth, t->d.format, t->d.target, t->d.levels, (void *)t->mem); } + if (getenv("DM_NVN_TEXCPU") && t->pool && !(t->pool->flags & NVN_MEMORY_POOL_FLAGS_CPU_NO_ACCESS)) MLOG("CPU 可见池上的非预排布贴图 ser=%llu %ux%ux%u 格式 %#x 目标 %d 级 %u 贴图标志 %#x 池标志 %#x 存储 %p", (unsigned long long)t->serial, t->d.width, t->d.height, t->d.depth, t->d.format, t->d.target, t->d.levels, t->d.flags, t->pool->flags, (void *)t->mem); res_add(mt); pthread_mutex_lock(&g_smu); if (!g_alias_set) g_alias_set = CFSetCreateMutable(NULL, 0, NULL); @@ -2374,7 +2398,7 @@ static MTLVertexFormat vfmt(int f) { case 0x25: return MTLVertexFormatUChar4Normalized; case 0x26: return MTLVertexFormatChar4Normalized; case 0x27: return MTLVertexFormatUChar4; case 0x28: return MTLVertexFormatChar4; case 0x29: return MTLVertexFormatHalf4; case 0x2A: return MTLVertexFormatUShort4Normalized; case 0x2B: return MTLVertexFormatShort4Normalized; case 0x2C: return MTLVertexFormatUShort4; case 0x2D: return MTLVertexFormatShort4; case 0x2E: return MTLVertexFormatFloat4; case 0x2F: return MTLVertexFormatUInt4; case 0x30: return MTLVertexFormatInt4; - case 0x3D: return MTLVertexFormatUInt1010102Normalized; case 0x3F: return MTLVertexFormatFloatRG11B10; case 0x9A: return MTLVertexFormatUChar4Normalized_BGRA; + case 0x3D: return MTLVertexFormatUInt1010102Normalized; case 0x69: return MTLVertexFormatInt1010102Normalized; case 0x3F: return MTLVertexFormatFloatRG11B10; case 0x9A: return MTLVertexFormatUChar4Normalized_BGRA; } return MTLVertexFormatInvalid; } @@ -4509,7 +4533,7 @@ frz_plain:; if (passdump()) { // 通道首次绘制的特征(判读入是否可省:全屏、无混合、写满通道、不依赖深度);DM_NVN_PASSDUMP_ALL 时每次绘制都记 | Characteristics of a render pass's first draw (to judge whether the load can be omitted: full screen, no blending, writes all channels, does not depend on depth); with DM_NVN_PASSDUMP_ALL every draw is recorded static uint64_t last_pass = ~0ull; static int all = -1; if (all < 0) all = getenv("DM_NVN_PASSDUMP_ALL") != NULL; if (all) PD("d %llu bv=%d bi=%d first=%d vs=%s fs=%s vc=%d inst=%d blend=%#x b0=%#llx dt=%d dw=%d df=%d cull=%d st=%d sf=%d,%d sop=%d,%d,%d/%d,%d,%d smask=%#x,%#x/%#x,%#x sref=%#x,%#x ib=%#llx it=%d cmask=%#x", g_stat_pass, di ? di->base_vertex : 0, di ? di->base_instance : 0, dw ? dw->first : 0, pr->sh[0] ? pr->sh[0]->key : "-", pr->sh[1] ? pr->sh[1]->key : "-", - dw ? dw->count : di->count, dw ? 1 : di->instances, S.color.blend_en, (unsigned long long)*(uint64_t *)&S.blend[0], (int)S.ds.dt, (int)S.ds.dw, (int)S.ds.df, (int)S.poly.cull, + dw ? dw->count : di->count, dw ? dw->instances : di->instances, S.color.blend_en, (unsigned long long)*(uint64_t *)&S.blend[0], (int)S.ds.dt, (int)S.ds.dw, (int)S.ds.df, (int)S.poly.cull, (int)S.ds.st, (int)S.ds.sf_f, (int)S.ds.sf_b, (int)S.ds.fail_f, (int)S.ds.zfail_f, (int)S.ds.zpass_f, (int)S.ds.fail_b, (int)S.ds.zfail_b, (int)S.ds.zpass_b, S.st_vmask[0], S.st_vmask[1], S.st_wmask[0], S.st_wmask[1], S.st_ref[0], S.st_ref[1], di ? (unsigned long long)di->ibuf : 0ull, di ? di->itype : 0, S.chmask.mask); @@ -4532,7 +4556,7 @@ frz_plain:; if (last_pass != g_stat_pass) { last_pass = g_stat_pass; PD("D %llu prim=%d vc=%d inst=%d blend=%#x wm=%#x dt=%d dw=%d df=%d st=%d sc=%lu,%lu,%lu,%lu tgt=%lux%lu vp=%.0f,%.0f,%.0f,%.0f", g_stat_pass, prim, - dw ? dw->count : di->count, dw ? 1 : di->instances, S.color.blend_en, S.chmask.mask, (int)S.ds.dt, (int)S.ds.dw, (int)S.ds.df, (int)S.ds.st, + dw ? dw->count : di->count, dw ? dw->instances : di->instances, S.color.blend_en, S.chmask.mask, (int)S.ds.dt, (int)S.ds.dw, (int)S.ds.df, (int)S.ds.st, (unsigned long)sc.x, (unsigned long)sc.y, (unsigned long)sc.width, (unsigned long)sc.height, (unsigned long)tw, (unsigned long)th, S.has_vp ? S.vp[0][0] : 0, S.has_vp ? S.vp[0][1] : 0, S.has_vp ? S.vp[0][2] : (float)tw, S.has_vp ? S.vp[0][3] : (float)th); } @@ -4551,9 +4575,10 @@ frz_plain:; for (uint32_t q = 0; q < n; q++) { uint32_t b = q * 4; uint32_t v[6] = {b, b + 1, b + 2, b, b + 2, b + 3}; memcpy(ix + q * 6, v, 24); } g_quad_ib = [g_dev newBufferWithBytes:ix length:n * 24 options:MTLResourceStorageModeShared]; g_quad_gpu = g_quad_ib.gpuAddress; res_add(g_quad_ib); free(ix); } - if (dw->count >= 4) enc_r_draw_indexed(en, mp, (dw->count / 4) * 6, MTLIndexTypeUInt32, g_quad_gpu, g_quad_ib.length, 1, dw->first, 0); + if (dw->count >= 4) enc_r_draw_indexed(en, mp, (dw->count / 4) * 6, MTLIndexTypeUInt32, g_quad_gpu, g_quad_ib.length, dw->instances > 0 ? dw->instances : 1, dw->first, dw->base_instance); } else if (dw) { - if (dw->count) enc_r_draw(en, mp, dw->first, dw->count); + if (dw->count && (dw->instances > 1 || dw->base_instance)) enc_r_draw_instanced(en, mp, dw->first, dw->count, dw->instances > 0 ? dw->instances : 1, dw->base_instance); + else if (dw->count) enc_r_draw(en, mp, dw->first, dw->count); } else { if (prim == 7) { RED_ONCE(20, "带索引的四边形未实现"); return; } if (di->itype == 0) { // 8 位索引:Metal 只有 16 / 32 位。回放时从客体内存读出、展开成 16 位放进本帧竞技场(SMBW 的全屏通道与最后拷进窗口那一下都用 8 位索引,不展开就整屏黑) diff --git a/dm/gfx/nvn/nvn_obj.h b/dm/gfx/nvn/nvn_obj.h index 65e1e89..8810ff9 100644 --- a/dm/gfx/nvn/nvn_obj.h +++ b/dm/gfx/nvn/nvn_obj.h @@ -124,7 +124,7 @@ typedef struct { int32_t stage, index; uint64_t addr, size; } p_bind_buf; // UB typedef struct { int32_t stage, first, n; uint64_t h[32]; } p_bind_handles; // 分离贴图 / 采样器句柄 | Separate texture / sampler handles typedef struct { int32_t face, value_mask, mask, ref, which; } p_stencil; // which:0 值掩码 1 写掩码 2 参考值 | which: 0 value mask 1 write mask 2 reference value typedef struct { float factor, units, clamp; } p_poly_offset; -typedef struct { int32_t prim, first, count; } p_draw; +typedef struct { int32_t prim, first, count, instances, base_instance; } p_draw; // instances 0 按 1 | instances 0 counts as 1 typedef struct { int32_t prim, itype, count, base_vertex, base_instance, instances; uint64_t ibuf; } p_draw_idx; typedef struct { int32_t x, y, z; uint64_t caller; } p_dispatch; // caller:游戏调用点返回地址(定位派发者) | caller: return address of the game call site (to locate the dispatcher) typedef struct { const dpool_t *pool; } p_set_pool; diff --git a/dm/gfx/tools/dmsh_build.py b/dm/gfx/tools/dmsh_build.py index 1e5169f..29ee12e 100644 --- a/dm/gfx/tools/dmsh_build.py +++ b/dm/gfx/tools/dmsh_build.py @@ -253,7 +253,14 @@ def job_for(key, name, rec, w): ci = rec.find(bytes.fromhex('34127698')); code = rec[0x38:ci]; stage = stage_of.get(key, name[:2]) c1o, c1t = struct.unpack_from('<2I', rec, ci + 0x700) # 控制段 +0x700:c1 相对指令区起点、+0x704:指令区总长(c1 = 两者之间;全库长度 0 / 256 / 512 / 768) | control segment +0x700: c1 relative to the start of the instruction area, +0x704: total length of the instruction area (c1 = what lies between the two; lengths across the whole library are 0 / 256 / 512 / 768) c1 = rec[0x38 + c1o:0x38 + c1t] - if stage != 'cs' or code[:4] == bytes.fromhex('79563412'): code = code[0x30:] # 去 0x30 头:图形阶段都有;计算阶段本作有(魔数 0x12345679),D2R 的包记录没有 | strip the 0x30 header: every graphics stage has one; compute has one in this game (magic 0x12345679), D2R's package records do not + if stage != 'cs': code = code[0x30:] # 去 0x30 头(其后 0x50 SPH 由翻译器自己读) | strip the 0x30 header (the 0x50 SPH after it is read by the translator itself) + elif code[:4] == bytes.fromhex('79563412'): + # 计算阶段本作有头(魔数 0x12345679,D2R 的包记录没有):头 0x30 + 零填充,指令从代码区 +0x100 起(全库 71 个实测;BRA 自跳与控制字都按这个起点 32 字节对齐)。 + # 只去 0x30 时翻译器从 +0x30 起解码:每 32 字节的控制字位置错开 0x10,间接跳转 BRX 的跳转表(相对程序起点)全部错位 → 粒子 / 后处理计算着色器死循环、GPU 卡死。 + # Compute in this game has a header (magic 0x12345679; D2R's package records have none): 0x30 header + zero padding, instructions start at code area +0x100 (measured on all 71; the BRA-to-self and the control words are 32-byte aligned to this start). + # Stripping only 0x30 made the translator decode from +0x30: the control word every 32 bytes is off by 0x10 and the jump tables of indirect BRX (relative to the program start) are all misplaced → particle / post-effect compute shaders loop forever and hang the GPU. + if any(code[0x30:0x100]): sys.exit(f'【红】{key}:计算代码区 +0x30..+0x100 不全为零,指令起点不是 +0x100') + code = code[0x100:] cp = os.path.join(w, key + '.bin'); open(cp, 'wb').write(code) j = {'key': key, 'stage': stage, 'code': cp, 'tex': {}, 'img': {}, 'attr': {}, 'fout': {}, 'c1': c1.hex(), 'debug': os.environ.get('XLATE_DEBUG', '1') != '0', 'reduced': os.environ.get('XLATE_REDUCED', '1') == '1'} # 翻译选项(见 dmsh_xlate/Program.cs);放宽精度缺省开(GPU 浮点末位误差不进游戏状态,mini 整帧 −26%) # c1:翻译器读间接跳转表用 | translation options (see dmsh_xlate/Program.cs); relaxed precision is on by default (GPU float last-bit error does not enter game state, whole frame −26% on the second Mac) # c1: used by the translator to read indirect jump tables diff --git a/dm/host/probe.c b/dm/host/probe.c index 8d44bc3..f654480 100644 --- a/dm/host/probe.c +++ b/dm/host/probe.c @@ -19,7 +19,28 @@ void crash_set_trap_hook(int (*fn)(ucontext_t *uc)); // The trampoline borrows x16: place probes only at function entries or at points where x16 is dead; the original instruction must not be PC-relative (adr/adrp/b/bl/ldr literal/cbz...). #define MAX_PROBE 16 typedef void (*probe_fn)(ucontext_t *uc); -static struct { uint64_t at; uint32_t *slot; unsigned hits; char name[40]; long dump; probe_fn fn; } g_probe[MAX_PROBE]; // dump:命中时打印 [x0+dump] 16 字节(-1 = 不打);fn:专用处理(替代缺省打印) | dump: on hit print 16 bytes at [x0+dump] (-1 = do not print); fn: dedicated handler (replaces the default print) +static struct { uint64_t at; uint32_t *slot; unsigned hits; char name[40]; long dump; probe_fn fn; int freg; long foff; uint64_t fval; int hreg; } g_probe[MAX_PROBE]; + +// !xN:把 xN 当 sead::ExpHeap,打印空闲块与已用块(SMBW 布局:空闲表 +0xb0、节点偏移 +0xc4;已用表 +0xc8、节点偏移 +0xdc;块大小在块 +0x18) +// !xN: treat xN as a sead::ExpHeap and print its free and used blocks (SMBW layout: free list +0xb0, node offset +0xc4; used list +0xc8, node offset +0xdc; block size at block +0x18) +static int rd64(uint64_t a, uint64_t *v) { vm_size_t g = 0; return !vm_read_overwrite(mach_task_self(), a, 8, (vm_address_t)v, &g) && g == 8; } +static void probe_heapwalk(uint64_t h) { + uint64_t start = 0, size = 0; rd64(h + 0x28, &start); rd64(h + 0x30, &size); + fprintf(stderr, "[dm] ExpHeap %#llx 起 %#llx 大小 %#llx\n", (unsigned long long)h, (unsigned long long)start, (unsigned long long)size); + static const struct { int list, off; const char *nm; } L[2] = { { 0xb0, 0xc4, "空闲/free" }, { 0xc8, 0xdc, "已用/used" } }; + for (int k = 0; k < 2; k++) { + uint64_t offw = 0, node = 0, sum = 0; int n = 0; rd64(h + L[k].off, &offw); int32_t off = (int32_t)offw; + rd64(h + L[k].list + 8, &node); + fprintf(stderr, "[dm] %s:", L[k].nm); + while (node && node != h + L[k].list && n < 2000) { + uint64_t blk = node - off, bsz = 0, w2 = 0; rd64(blk + 0x18, &bsz); rd64(blk + 0x10, &w2); + fprintf(stderr, "%s%llx+%llx", n % 8 ? " " : "\n[dm] ", (unsigned long long)(blk - start), (unsigned long long)bsz); + sum += bsz; n++; + if (!rd64(node + 8, &node)) break; + } + fprintf(stderr, "\n[dm] %s 共 %d 块 %#llx 字节\n", L[k].nm, n, (unsigned long long)sum); + } +} // dump:命中时打印 [x0+dump] 16 字节(-1 = 不打);fn:专用处理(替代缺省打印) | dump: on hit print 16 bytes at [x0+dump] (-1 = do not print); fn: dedicated handler (replaces the default print) static int g_nprobe; static void probe_str(uint64_t p) { char buf[80]; vm_size_t got = 0; @@ -55,23 +76,31 @@ static int probe_hit(ucontext_t *uc) { // 由 probe_install 注册给 crash.c for (int i = 0; i < g_nprobe; i++) if (g_probe[i].at == pc) { probe_dump_once(); if (g_probe[i].fn) { g_probe[i].hits++; g_probe[i].fn(uc); uc->uc_mcontext->__ss.__pc = (uint64_t)g_probe[i].slot; return 1; } - { extern uint64_t g_t0_ns; uint64_t now_ns(void); fprintf(stderr, "[dm] PROBE %s #%u [%.4fs]", g_probe[i].name, ++g_probe[i].hits, (now_ns() - g_t0_ns) / 1e9); } + if (g_probe[i].freg >= 0) { // ?xN=值 / ?[xN+偏移]=值:不符就静默放行 | ?xN=value / ?[xN+offset]=value: pass through silently when it does not match + uint64_t v = uc->uc_mcontext->__ss.__x[g_probe[i].freg]; vm_size_t g = 0; + if (g_probe[i].foff >= 0 && vm_read_overwrite(mach_task_self(), v + g_probe[i].foff, 8, (vm_address_t)&v, &g)) v = ~g_probe[i].fval; + if (v != g_probe[i].fval) { uc->uc_mcontext->__ss.__pc = (uint64_t)g_probe[i].slot; return 1; } + } + { extern uint64_t g_t0_ns; uint64_t now_ns(void); uint32_t thr_cur_handle(void); fprintf(stderr, "[dm] PROBE %s #%u [%.4fs] T=%x", g_probe[i].name, ++g_probe[i].hits, (now_ns() - g_t0_ns) / 1e9, thr_cur_handle()); } for (int r = 0; r < 8; r++) { uint64_t v = uc->uc_mcontext->__ss.__x[r]; fprintf(stderr, " x%d=%#llx", r, v); probe_str(v); } for (int r = 19; r < 24; r++) fprintf(stderr, " x%d=%#llx", r, uc->uc_mcontext->__ss.__x[r]); + if (getenv("DM_PROBE_ALL")) for (int r = 9; r < 19; r++) fprintf(stderr, " x%d=%#llx", r, uc->uc_mcontext->__ss.__x[r]); // DM_PROBE_ALL=1:再打 x9–x18 | also print x9–x18 fprintf(stderr, " lr=%#llx", uc->uc_mcontext->__ss.__lr); { uint64_t x8 = uc->uc_mcontext->__ss.__x[8]; Dl_info di; fprintf(stderr, " x8=%#llx", x8); // x8:间接调用目标常在这里(blr x8 前下探针) | x8: indirect call targets are often here (probe placed before blr x8) if (dladdr((void *)x8, &di) && di.dli_sname) fprintf(stderr, "(%s+%#lx)", di.dli_sname, (unsigned long)(x8 - (uint64_t)di.dli_saddr)); } if (g_probe[i].dump >= 0) { uint8_t b[16]; vm_size_t got = 0; - uint64_t at = uc->uc_mcontext->__ss.__x[0] + (g_probe[i].dump & 0xffffff); + int dr = (g_probe[i].dump >> 25) & 31; // @xN+偏移:以 xN 为基址 | @xN+offset: based on xN + uint64_t at = uc->uc_mcontext->__ss.__x[dr] + (g_probe[i].dump & 0xffffff); if (g_probe[i].dump & 0x1000000) { uint64_t p = 0; vm_size_t g = 0; if (!vm_read_overwrite(mach_task_self(), at, 8, (vm_address_t)&p, &g)) at = p; else at = 0; } // @*偏移:先解引用再转储 | @*offset: dereference first, then dump if (at && !vm_read_overwrite(mach_task_self(), at, 16, (vm_address_t)b, &got) && got == 16) { - fprintf(stderr, " [%s%#lx]=", (g_probe[i].dump & 0x1000000) ? "*x0+" : "x0+", g_probe[i].dump & 0xffffff); for (int k = 0; k < 16; k++) fprintf(stderr, "%02x", b[k]); } + fprintf(stderr, " [%sx%d+%#lx]=", (g_probe[i].dump & 0x1000000) ? "*" : "", dr, g_probe[i].dump & 0xffffff); for (int k = 0; k < 16; k++) fprintf(stderr, "%02x", b[k]); } } fputc('\n', stderr); { static long xd = -2; if (xd == -2) { const char *e = getenv("DM_PROBE_X2"); xd = e ? strtol(e, 0, 16) : -1; } // DM_PROBE_X2=字节数:另打 x2 指向的内存(看对象字段) | DM_PROBE_X2=bytes: also print the memory x2 points to (to read object fields) if (xd > 0) { uint64_t at = uc->uc_mcontext->__ss.__x[2]; uint8_t b[512]; vm_size_t got = 0; long n = xd > 512 ? 512 : xd; if (!vm_read_overwrite(mach_task_self(), at, n, (vm_address_t)b, &got)) { fprintf(stderr, "[dm] [x2]:"); for (long k = 0; k < (long)got; k += 8) { uint64_t v; memcpy(&v, b + k, 8); fprintf(stderr, "%s%llx", k % 32 ? " " : "\n[dm] ", (unsigned long long)v); } fputc('\n', stderr); } } } + if (g_probe[i].hreg >= 0) probe_heapwalk(uc->uc_mcontext->__ss.__x[g_probe[i].hreg]); dm_backtrace(uc->uc_mcontext->__ss.__fp); uc->uc_mcontext->__ss.__pc = (uint64_t)g_probe[i].slot; return 1; @@ -93,7 +122,7 @@ static void probe_add(module_t *m, uint64_t off, long dump, probe_fn fn) { *w = 0xD43E0000; // brk #0xf000 mprotect((void *)pg, 0x4000, PROT_READ | PROT_EXEC); sys_icache_invalidate(w, 4); - g_probe[g_nprobe].at = at; g_probe[g_nprobe].slot = sl; g_probe[g_nprobe].dump = dump; g_probe[g_nprobe].fn = fn; + g_probe[g_nprobe].at = at; g_probe[g_nprobe].slot = sl; g_probe[g_nprobe].dump = dump; g_probe[g_nprobe].fn = fn; g_probe[g_nprobe].freg = -1; g_probe[g_nprobe].hreg = -1; snprintf(g_probe[g_nprobe].name, sizeof g_probe[0].name, "%s+%#llx", m->name, off); LOG("探针 %s(原指令 %08x)", g_probe[g_nprobe].name, sl[0]); g_nprobe++; @@ -114,8 +143,24 @@ void probe_install(void) { if (!m) die("DM_PROBE 无此模块 %s", t); char *at_ = strchr(c + 1, '@'); long dump = -1; - if (at_) dump = at_[1] == '*' ? (0x1000000 | strtol(at_ + 2, 0, 16)) : strtol(at_ + 1, 0, 16); // @偏移 / @*偏移 | @offset / @*offset + if (at_) { // @偏移 / @*偏移 / @xN+偏移 / @*xN+偏移 | @offset / @*offset / @xN+offset / @*xN+offset + const char *q = at_ + 1; long deref = 0, reg = 0; + if (*q == '*') { deref = 0x1000000; q++; } + if (*q == 'x') { char *e2; reg = strtol(q + 1, &e2, 10) & 31; q = *e2 == '+' ? e2 + 1 : e2; } + dump = deref | (reg << 25) | (strtol(q, 0, 16) & 0xffffff); + } + char *qm = strchr(c + 1, '?'), *bang = strchr(c + 1, '!'); probe_add(m, strtoull(c + 1, 0, 16), dump, NULL); + if (bang && bang[1] == 'x') g_probe[g_nprobe - 1].hreg = (int)strtol(bang + 2, 0, 10) & 31; + if (qm) { // 过滤:?x0=706c400000 或 ?[x0+38]=781548(十六进制) | Filter: ?x0=706c400000 or ?[x0+38]=781548 (hex) + const char *q = qm + 1; int br = *q == '['; + if (br) q++; + if (*q != 'x') die("DM_PROBE 过滤格式:?xN=值 或 ?[xN+偏移]=值"); + char *e2; g_probe[g_nprobe - 1].freg = (int)strtol(q + 1, &e2, 10) & 31; g_probe[g_nprobe - 1].foff = -1; + if (br) { g_probe[g_nprobe - 1].foff = *e2 == '+' ? strtol(e2 + 1, &e2, 16) : 0; if (*e2 == ']') e2++; } + if (*e2 != '=') die("DM_PROBE 过滤缺 ="); + g_probe[g_nprobe - 1].fval = strtoull(e2 + 1, 0, 16); + } } } if (g_slots && !g_slots_sealed) { mprotect(g_slots, 0x4000, PROT_READ | PROT_EXEC); sys_icache_invalidate(g_slots, 0x4000); g_slots_sealed = 1; } diff --git a/dm/tools/ghidra/Decompile.java b/dm/tools/ghidra/Decompile.java new file mode 100644 index 0000000..27e120f --- /dev/null +++ b/dm/tools/ghidra/Decompile.java @@ -0,0 +1,65 @@ +// 按地址反编译:-postScript Decompile.java <模块偏移,十六进制,逗号分隔;前缀 @ = 在该地址(重)建函数> [映像基址,缺省 0x100000] +// Decompile functions by address: -postScript Decompile.java [image base, default 0x100000] +import ghidra.app.script.GhidraScript; +import ghidra.app.decompiler.*; +import ghidra.program.model.listing.Function; +import ghidra.program.model.address.Address; + +public class Decompile extends GhidraScript { + // 曾被当成不返回的 nn::os 导入调用之后的代码没反汇编:清掉调用返回覆盖并继续反汇编 + // Calls to nn::os imports once treated as non-returning left their fall-through undisassembled; clear the call-return override and disassemble past them + void fixFlow(Address start) { + for (int round = 0; round < 64; round++) { + disassemble(start); + boolean changed = false; + Address end = start.add(0x20000); + ghidra.program.model.listing.InstructionIterator it = currentProgram.getListing().getInstructions(start, true); + while (it.hasNext()) { + ghidra.program.model.listing.Instruction ins = it.next(); + if (ins.getAddress().compareTo(end) > 0) break; + if (!ins.getFlowType().isCall()) continue; + if (System.getenv("DECOMP_DEBUG") != null) println("call " + ins.getAddress() + " " + ins + " flow=" + ins.getFlowType() + " ftovr=" + ins.isFallThroughOverridden() + " ft=" + ins.getFallThrough() + " ovr=" + ins.getFlowOverride()); + Address[] fl = ins.getFlows(); if (fl.length == 0) continue; + Function t = getFunctionAt(fl[0]); if (t == null) continue; + Function r = t.isThunk() ? t.getThunkedFunction(true) : t; + if (!r.getName(true).contains("nn::os::")) continue; + if (ins.getFlowOverride() != ghidra.program.model.listing.FlowOverride.NONE) { ins.setFlowOverride(ghidra.program.model.listing.FlowOverride.NONE); changed = true; } + if (ins.isFallThroughOverridden()) { ins.clearFallThroughOverride(); changed = true; } + Address nx = ins.getMaxAddress().next(); + if (getInstructionAt(nx) == null) { disassemble(nx); changed = true; } + } + if (!changed) break; + } + } + @Override public void run() throws Exception { + String[] args = getScriptArgs(); + long base = args.length > 1 ? Long.decode(args[1]) : 0x100000L; + // Ghidra 把部分 nn::os 导入(LockMutex)标成不返回,函数在第一次加锁后就被截断;撤掉这个标记 + // Ghidra marks some nn::os imports (LockMutex) as non-returning, which cuts functions short; undo that + java.util.List all = new java.util.ArrayList<>(); + currentProgram.getFunctionManager().getFunctions(true).forEach(all::add); + currentProgram.getFunctionManager().getExternalFunctions().forEach(all::add); + for (Function g : all) { + String n = g.getName(true); + if (g.hasNoReturn() && (g.isThunk() || g.isExternal()) && !n.matches("(?i).*(abort|terminate|throw|exit|panic|unwind).*")) g.setNoReturn(false); + } + DecompInterface di = new DecompInterface(); + di.openProgram(currentProgram); + for (String a : args[0].split(",")) { + // "@偏移":没有函数包含它时在此建函数(只经虚表到达的代码) + // "@off" = create a function starting at off when none contains it (code reached only through vtables) + boolean mk = a.trim().startsWith("@"); + Address addr = toAddr(base + Long.decode(a.trim().replace("@", ""))); + Function f = getFunctionContaining(addr); + // "@偏移" 正是已有函数的入口:重建它(函数体可能在误标不返回的调用处被截断) + // "@off" on an existing function: rebuild it, since its body may have been cut at a call wrongly marked non-returning + if (mk && f != null && f.getEntryPoint().equals(addr)) { removeFunction(f); f = null; } + if (f == null && mk) { fixFlow(addr); f = createFunction(addr, null); } + if (f == null) { println("NO FUNCTION at " + a); continue; } + DecompileResults r = di.decompileFunction(f, 120, monitor); + println("===== " + a + " in " + f.getName() + " @" + f.getEntryPoint() + " ====="); + println(r.decompileCompleted() ? r.getDecompiledFunction().getC() : ("decompile failed: " + r.getErrorMessage())); + } + di.dispose(); + } +} diff --git a/dm/tools/patch_nso.py b/dm/tools/patch_nso.py index 4c01167..027c1dc 100644 --- a/dm/tools/patch_nso.py +++ b/dm/tools/patch_nso.py @@ -49,7 +49,20 @@ PATCH_OFF = (len(IMG) + 0xFFFF) & ~0xFFFF # ---- Registered cuts: {NSO build ID: [(offset, original bytes, new bytes, reason)]} ---- # Networking cuts (excluded scope: Battle.net / online play): taken from a community offline fix (LFS Mod Offline Fix for v2228224, which differs from the official 94012 main in only these 7 words). # Measured: the system service layer (nifm offline / DNS black hole / account service) cannot get around it: the game itself sends Battle.net requests directly and allows offline play according to the "last online" time. -CUTS = {'db0507f36b84200ac05f75d76ed5026f': [ +CUTS = { + # Super Mario Bros. Wonder v0 main + 'cd6e42aee7934f4d8393cd43893ae25e': [ + # 资源堆留余量:资源单元的 ExpHeap 按资源尺寸表值 + 0x1008 建(main+0x1e603c → ExpHeap::create main+0x4aac38),表值是任天堂实测的峰值,几乎不留余量 + # (PlayerYoshiAnm 用到 99%)。宿主上开场关 Course900 的 Model/NpcCommonerA.bfres 差约 0x70 字节:模型资源初始化最后一块显存(MatBlock, + # 0x9000 字节、4 KB 对齐,main+0x4290c0)放下后,它的 0x598 字节描述块放不下,显存分配失败 → 缓冲区建在空内存池上 → main+0x106f5c8 读空崩溃。 + # 与真机差在哪里未查明(本块之前的每一项分配尺寸都由游戏代码与数据决定);先给每个资源堆 +16 KB。 + # Resource heap slack: each resource unit's ExpHeap is created with the resource size table value + 0x1008 (main+0x1e603c → ExpHeap::create main+0x4aac38); the table holds Nintendo's measured peaks with almost no margin + # (PlayerYoshiAnm uses 99%). On the host, Model/NpcCommonerA.bfres in the opening course Course900 is about 0x70 bytes short: after the model resource's last GPU chunk (MatBlock, + # 0x9000 bytes, 4 KB aligned, main+0x4290c0) is placed, its 0x598-byte descriptor does not fit, the GPU allocation fails → a buffer is built on a null memory pool → null read crash at main+0x106f5c8. + # Where this differs from hardware is not known (every allocation size before this block is determined by game code and data); for now every resource heap gets +16 KB. + (0x1e62fc, 'e00317aa', 'e0124091', '资源堆尺寸:mov x0,x23 → add x0,x23,#0x4000(ExpHeap::create 的尺寸参数 +16 KB)'), + ], + 'db0507f36b84200ac05f75d76ed5026f': [ # (放进所有产物共用的基础改写):游戏显存分配器预算 260 MB → 512 MB # (Placed in the base rewrite shared by all build outputs): game video memory allocator budget 260 MB → 512 MB (0x8524c4, '0808a2d2', '0800a4d2', '显存分配器初始化配置:mov x8,#0x10400000(260 MB)→ mov x8,#0x20000000(512 MB),随后 movk #0xa0,lsl48 与 str [sp,#0x238] 不变;' diff --git a/docs/agents/02-survey.md b/docs/agents/02-survey.md index 0f8fe0d..4bf6d3d 100644 --- a/docs/agents/02-survey.md +++ b/docs/agents/02-survey.md @@ -46,7 +46,7 @@ Do this before touching the host. Every item answers "what will break, and how m ```bash JAVA_HOME= $GHIDRA/support/analyzeHeadless ~/-work/ghidra/proj <name> -process <module>.elf \ - -noanalysis -readOnly -scriptPath ~/<title>-work/ghidra/scripts -postScript Decompile.java 0x107dbc0,0x3707b0 + -noanalysis -readOnly -scriptPath <repo>/dm/tools/ghidra -postScript Decompile.java 0x107dbc0,0x3707b0 ``` `analyzeHeadless` needs `JAVA_HOME` set even when Java is on PATH. Code only reachable through vtables may have no function defined at an address; use `objdump -d --start-address=… --stop-address=… <module>.elf` instead. diff --git a/docs/agents/03-bringup.md b/docs/agents/03-bringup.md index 3a9fe9b..b62cb20 100644 --- a/docs/agents/03-bringup.md +++ b/docs/agents/03-bringup.md @@ -55,7 +55,16 @@ Each entry: symptom → root cause → fix (where the fix lives). Check these fi - **File-not-found noise** → games probe many optional paths (debug parameters, other-language voice files); with `DM_FS=1` these show as `fs … 不存在`. Only a missing file followed by a failure matters. - **A buffer is created with a null pool right after a scene change** → NVN *virtual* memory pools (flag 0x200) have no storage until `nvnMemoryPoolMapVirtual` maps physical pool pages in; if the host returns failure, the game treats the pool as unusable. Mario maps its single 128 MB virtual pool onto one physical pool in one request, so the host aliases it (`hle_nvn.c`). True sparse mappings are still unimplemented. -- **A null-pool buffer even with virtual pools working** → the game's own heap ran out. Trace it with probes ([05](05-debugging.md)): the gsys GPU allocator at `main+0x4290c0` and the heap's `tryAlloc` (vtable slot +0x30) show which heap, its size (resource heaps are sized from the resource-size table + 0x1008) and every allocation into it. Open in Mario: `Model/NpcCommonerA.bfres`'s heap overflows when the opening course creates its instances. +- **A null-pool buffer even with virtual pools working** → the game's own heap ran out. Trace it with probes ([05](05-debugging.md)): the gsys GPU allocator at `main+0x4290c0` and the heap's `tryAlloc` (vtable slot +0x30) show which heap, its size and every allocation into it; the `!xN` heap walk shows the free and used blocks at the moment of failure. + - Mario: each resource gets its own `sead::ExpHeap` sized *resource-size-table value + 0x1008* (`main+0x1e603c` → `ExpHeap::create` `main+0x4aac38`). The table values are Nintendo's measured peaks with almost no margin (Yoshi's animation file uses 99 % of its budget). `Model/NpcCommonerA.bfres` in the opening course came up about 0x70 bytes short: its last GPU chunk (0x9000 bytes, 4 KB aligned) fit, but the chunk's 0x598-byte descriptor did not. Every allocation size before it is fixed by game code and data, so the difference from hardware is still unknown. + - Fix: a registered code cut gives every resource heap +16 KB (`CUTS` in `dm/tools/patch_nso.py`, keyed by the module build ID, original bytes verified). Per-title code patches go there, with the reason. + - Read the probe log carefully: `tryAlloc` probes show *requests*, not successes, so a running total that already exceeds the heap means an earlier request failed. + +- **`FATAL NVN 命令内存:游戏池已空…`** → our command records are larger than real NVN commands, so command buffers the game sizes for hardware overflow; when the game's out-of-memory callback declines to add memory, the host lends a segment. Many small command buffers each overflowing once per frame exhausted a fixed pool of 64 × 1 MB. The pool now grows on demand (256 KB segments, reused after 8 frames, abort only beyond 1 GB; `hostseg_take` in `hle_nvn.c`). +- **Frames stop, no crash, the main thread sits in `nvnSyncWait(sync, -1)`, and the watchdog's wait-state line shows the main fence one short (`已提交 N / 已完成 N-1`)** → a GPU command buffer never completes. Check GPU load from outside: `ioreg -r -d 1 -c IOAccelerator | grep 'Device Utilization'` at 100 % means a shader is looping forever. Find the shaders first used in the last frames (`DM_NVN_PASSDUMP`, compare the `K` keys) and look at their `.metal` for an empty `do {} while (true);` with no `return` (see [04](04-graphics.md) section 1: the compute program start). + +- **Menus and dialogue boxes draw but contain no text; images and pictogram digits are fine** → the game rasterizes glyphs on the CPU into a *linear* texture in a CPU-visible pool (Mario: 2048×8192 R8, flags IMAGE|LINEAR). A private Metal texture never sees those writes. Linear textures in CPU-visible pools are now created on the pool buffer (`tex_linear` in `nvn_mtl.m`). `DM_NVN_TEXCPU=1` lists non-packaged textures in CPU-visible pools. +- **Controller-button icons in text show as `?`** → those glyphs come from the system shared font's Nintendo-extension set; the host serves a desktop font for every shared-font type. Open. ### NVN object sizes (memory corruption) diff --git a/docs/agents/04-graphics.md b/docs/agents/04-graphics.md index be3b327..d819521 100644 --- a/docs/agents/04-graphics.md +++ b/docs/agents/04-graphics.md @@ -10,7 +10,8 @@ Do not stop at the obvious archive. In Mario the main `bfsha` held 4,888 of the - **BNSH** (`BNSH\0\0\0\0` + `grsc` block at +0x60): inside `bfsha`, particle effects (`esetb`, VFXB), UI layouts (`blarc`), and `main` itself. Variation table at +0x80 (count at +0x7c); binary program at variation +0x10; six stage pointers from program +0x08 (vs, hs, ds, gs, ps, cs); stage code descriptor: control ptr +0x08, code ptr +0x10, code size +0x18, control size +0x1c. All pointers relative to the BNSH start. - **BAHS** (agl `sharcb`, engine utility shaders): code region at header +0x1c (size +0x18). Binary entries in the table area are 16 bytes `{code offset, 0, control size (0x880/0x88c), signed offset from this entry to the control section}`. Control sections are shared between programs, so pair by entry, never by searching for code near a control. -- **Code area**: 0x30-byte header (magic `0x12345678` graphics, `0x12345679` compute) + 0x50 SPH (graphics only) + instructions. **Control section**: magic `0x98761234`; +0x700 = c1 offset in the code area, +0x704 = code size, +0x748 = compute thread-group size (u32 ×3), +0x754 = compute shared memory bytes. +- **Code area**: graphics = 0x30-byte header (magic `0x12345678`) + 0x50 SPH + instructions (at +0x80); compute = 0x30-byte header (magic `0x12345679`) + **zero padding, instructions at +0x100** (all 71 of Mario's). **Control section**: magic `0x98761234`; +0x700 = c1 offset in the code area, +0x704 = code size, +0x748 = compute thread-group size (u32 ×3), +0x754 = compute shared memory bytes. +- **The program start must be exact.** Ryujinx's decoder treats every 32-byte boundary *relative to the start* as a scheduling word. Mario's compute code was first fed from +0x30: the groups were misaligned by 0x10, instructions were dropped or misread, and 12 compute shaders (particles, post effects) translated into an endless loop that hung the GPU at the first gameplay scene. Check a new title: the first non-zero word after the header must be a control word, and the trailing `EXIT` (`e30000000007000f`) / `BRA`-to-self (`e2400fffff87000f`) must sit in instruction slots of 32-byte groups counted from your chosen start. `dmsh_build.py` now refuses compute code whose padding is not zero. Validate a new title's coverage at run time: `DM_NVN_SHDUMP=<dir>` makes the host dump the control section and code of every program it can't find in the map; search those bytes in the game files to find the container that was missed. @@ -50,7 +51,10 @@ Consequence: any optimization that infers "which textures a draw reads" from bou - **Inline uniform updates** (`CommandBufferUpdateUniformBuffer`) are replayed as in-order GPU copies; reading the uniform buffer from the CPU at encode time gives the frame's *last* value, not the draw's. - **Compute code areas have a header** in Mario (`0x12345679`), not in D2R; `dmsh_build.py` strips it. - **Virtual memory pools** (flag 0x200): aliased onto the physical pool named by `MapVirtual` when all requests share one base (Mario's case); otherwise unimplemented. -- **Image atomics**: 7 agl compute shaders produce SPIR-V that SPIRV-Cross can't parse (`nullptr` exception). Open issue. +- **SPIRV-Cross failures**: 2 agl post-effect compute shaders still produce SPIR-V that SPIRV-Cross can't parse (`nullptr` exception). Before the compute start-offset fix it was 7: misdecoded code produced the bad SPIR-V. +- **CPU-written linear textures**: a texture with `NVN_TEXTURE_FLAGS_LINEAR` in a CPU-visible pool is created on the pool's `MTLBuffer` (`tex_linear`), so CPU writes (Mario's glyph cache) are visible to the GPU. Block-linear CPU-written textures still need the packaged-texture re-upload path. +- **Vertex formats**: Mario uses `RGB10A2SN` (0x69) → `MTLVertexFormatInt1010102Normalized`. Unmapped formats fall back to a 4-component read and print `顶点属性格式 … 未映射` once. +- **Instanced array draws** (`DrawArraysInstanced`) carry instance count and base instance in the `OP_DRAW` record. ## 7. Diagnostic switches (host) diff --git a/docs/agents/05-debugging.md b/docs/agents/05-debugging.md index 780f383..b8390dc 100644 --- a/docs/agents/05-debugging.md +++ b/docs/agents/05-debugging.md @@ -21,17 +21,29 @@ x0=… x28=… # decompile the functions containing these module offsets JAVA_HOME=/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home \ /opt/homebrew/opt/ghidra/libexec/support/analyzeHeadless ~/<title>-work/ghidra/proj <proj> -process <module>.elf \ - -noanalysis -readOnly -scriptPath ~/<title>-work/ghidra/scripts -postScript Decompile.java 0x373e04,0x2ea2ac + -noanalysis -readOnly -scriptPath <repo>/dm/tools/ghidra -postScript Decompile.java 0x373e04,0x2ea2ac # raw disassembly (works where Ghidra defined no function) objdump -d --start-address=0x2ea1c0 --stop-address=0x2ea2b0 ~/<title>-work/elf/<module>.elf ``` +Prefix an offset with `@` (`... Decompile.java @0x49740`) to (re)create the function at exactly that address. Code reached only through vtables has no function, and Ghidra's analysis marked some `nn::os` imports (`LockMutex`) as non-returning, which cut many functions short right after the first lock. The script clears that flag and the call-return overrides on such calls before decompiling. + Map GOT slots to NVN names with the survey table (`census/*_nvn_slots.tsv`): a call through `[x8, #0x3e8]` after `adrp x8, 0x35ad000` is GOT entry 0x35ad3e8. ## Probes without a debugger (`DM_PROBE`) -`DM_PROBE=main:4290c0,main:9712c` (module:offset, comma-separated; `host/probe.c`, Mac only) replaces the instruction with `brk`, and on every hit prints x0–x7 (and strings they point to), x19–x23, lr, x8 and a backtrace, then runs the original instruction from a trampoline. Put probes at function entries or where x16 is dead; the probed instruction must not be PC-relative. `DM_PROBE_X2=<bytes>` also dumps the memory x2 points to (object fields). Output from several threads interleaves; parse it with a regex rather than by line. Probes slow the game, which can shift heap addresses between runs. +`DM_PROBE=main:4290c0,main:9712c` (module:offset, comma-separated; `host/probe.c`, Mac only) replaces the instruction with `brk`, and on every hit prints the guest thread (`T=`), x0–x7 (and strings they point to), x19–x23, lr, x8 and a backtrace, then runs the original instruction from a trampoline. Put probes at function entries or where x16 is dead; the probed instruction must not be PC-relative (no `adrp`, `adr`, `b`, `bl`, literal `ldr`: probe the next instruction instead). Output from several threads interleaves; parse it with a regex rather than by line. Probes slow the game, which can shift heap addresses between runs. + +Per-probe suffixes, in this order: + +| Suffix | Effect | +|---|---| +| `@off`, `@*off`, `@xN+off`, `@*xN+off` | Print 16 bytes at x0+off (or xN+off); `*` dereferences that address first | +| `?xN=val`, `?[xN+off]=val` | Only report hits where the register (or the 64-bit word at xN+off) equals `val` (hex); others pass silently. E.g. `main:4743c0?[x0+30]=781548` = only allocations into the heap whose size field is 0x781548 | +| `!xN` | Treat xN as a `sead::ExpHeap` and print its free and used blocks (offset+size) | + +`DM_PROBE_ALL=1` adds x9–x18; `DM_PROBE_X2=<bytes>` dumps the memory x2 points to. Use it when lldb is unavailable: after macOS revokes the debugger authorization (`DevToolsSecurity -status` says developer mode is disabled), `lldb` launches hang waiting for a GUI approval prompt. `sudo DevToolsSecurity -enable` (run by the user) restores it.