Port Mario + Rabbids Sparks of Hope

docs/agents/symptoms.md

@@ -56,6 +56,7 @@ One line per symptom in the [subsystem references](../reference/): what you see
 - nothing aborts at startup as with Unity 2022.3, but the provisional profile gave a 1 GB block and a 256 MB alias region while Unity maps 1.5 GB in the first room. → [Unity 6 takes its memory with no StandardAllocator and no heap](../reference/memory.md#unity-6-takes-its-memory-with-no-standardallocator-and-no-heap)
 - `nn::diag` abort from `StandardAllocator::Initialize` long after `nninitStartup`, right after the first Vulkan query pool (`sdk+0x12dad0` ← `main+0x62c52c`). → [The movie player's own StandardAllocator](../reference/memory.md#the-movie-players-own-standardallocator)
 - `StandardAllocator::Initialize` aborts in `nninitStartup` (Pokémon Legends: Z-A: `main+0x1b2e690`); `nn::diag` abort at `sdk+0x11ff34` (`StandardAllocator::Initialize`) in `nninitStartup`, SDK 15.3.2, called from `main+0x31370c` (Paper Mario: The Thousand-Year Door). → [StandardAllocators over 12 to 62 GB of address space](../reference/memory.md#standardallocators-over-12-to-62-gb-of-address-space)
+- a write to `0x20000ffff` in static initialization (`main+0x164990`), after `AllocateAddressRegion` returns `0x1803`; reducing the request to 3 GB still returns `0x1803` with the SDK's original large-region alignment. With only the outer ranges reduced, a later free crashes at `main+0x1668dc` after reading compressed file data as a slab-owner pointer. → [Separate virtual ranges in Snowdrop's page allocator](../reference/memory.md#separate-virtual-ranges-in-snowdrops-page-allocator)
 - `nn::diag` abort at `sdk+0x12dad0` in `nninitStartup`, called from `main+0x756310`. → [.NET NativeAOT malloc and the GC region range](../reference/memory.md#net-nativeaot-malloc-and-the-gc-region-range)
 - the game aborts in its static initialisers, before `nnMain`, when its own 62 GB address region fails (`main+0x37c9bd0`, `AllocateAddressRegion(&p, 0xf80000000)` at `main+0x37c9a5c`). → [A 62 GB page allocator behind `nu::VirtualAllocHook`](../reference/memory.md#a-62-gb-page-allocator-behind-nuvirtualallochook)
 - the game calls `exit(1)` in `nninitStartup` (Octopath Traveler 0: `main+0x12e74`); `nn::diag::Abort` from `main+0x352fbec` (Unreal's fatal error) after the first commit, `AllocateMemoryPages(0, 2 MB)` while the RHI starts (`main+0x353ff8c`, result 0xe03) (Pikmin 4); a crash in `NumberLineAllocator::Compaction` at boot (DRAGON QUEST XI S, region option 4224). → [Unreal's engine address region and its options](../reference/memory.md#unreals-engine-address-region-and-its-options)
@@ -142,6 +143,7 @@ One line per symptom in the [subsystem references](../reference/): what you see
 - SIGSEGV in `android::Surface::getQueuedFrameNumber` under `nn::vi::GetLatestFrameNumber`. → [Layer frame numbers](../reference/services.md#layer-frame-numbers)
 - SIGSEGV in `vi::GetDefaultDisplayPanelIdentifier` (`main+0xa9a4`, before the window). → [Default display panel identifier](../reference/services.md#default-display-panel-identifier)
 - the first boot crashed in `oe::DisableAlbumImageTakenNotification`: a null read at `sdk+0x338070`, the first call in `nnMain` (`main+0x6b188c`); other `oe` calls that reach am, found in the imports before they were met. → [oe calls answered without an applet proxy](../reference/services.md#oe-calls-answered-without-an-applet-proxy)
+- a null read at `sdk+0x36ec60` in `oe::EnableAlbumImageTakenNotification`, called from `nnMain` (`main+0x286268`). → [oe calls answered without an applet proxy](../reference/services.md#oe-calls-answered-without-an-applet-proxy)
 - abort in `nn::sm::Initialize` under `nn::oe::FinishStartupLogo`, right after the first frames. → [The startup logo](../reference/services.md#the-startup-logo)
 - null read in `oe::InitializeCopyrightFrameBuffer`, right after the window setup. → [Copyright frame buffer overlay](../reference/services.md#copyright-frame-buffer-overlay)
 - SDK 20 calls that reach a service, each one boot; SDK 20's `OpenAudioRenderer` aborts on an ARUID of 0. → [Applet resource user ID, launch parameters and VR mode](../reference/services.md#applet-resource-user-id-launch-parameters-and-vr-mode)
@@ -302,6 +304,7 @@ One line per symptom in the [subsystem references](../reference/): what you see
 - an abort in device setup when `nvnTexturePoolGetSize` / `nvnSamplerPoolGetSize` returned 0. → [Pool size getters, texture address and command memory size](../reference/nvn-objects.md#pool-size-getters-texture-address-and-command-memory-size)
 - red stub lines for NVN calls. → [NVN calls that were red stubs](../reference/nvn-objects.md#nvn-calls-that-were-red-stubs)
 - `nvnWindowGetPresentInterval`, `nvnCommandBufferSetCopyImageStride` missing. → [SDK 8 window and copy-stride entry points](../reference/nvn-objects.md#sdk-8-window-and-copy-stride-entry-points)
+- `[RED] texture bound to an image slot lacks the shader-write usage`, and compute image writes go to a dummy texture. → [Image bindings supply shader-write intent](../reference/nvn-objects.md#image-bindings-supply-shader-write-intent)
 
 ## Rendering
 
@@ -400,6 +403,7 @@ One line per symptom in the [subsystem references](../reference/): what you see
 - the split found no shader in the game (0 shaders). → [GLSLC outputs holding several stages](../reference/shaders.md#glslc-outputs-holding-several-stages)
 - an NVN game's split finds none of its shipped shaders, and the ones it compiles at run time were not captured: the game's renderer is in NROs that call `glslcCompile`, and its whole RomFS is one zip. → [GLSLC outputs in a zip archive, and those NROs compile at run time](../reference/shaders.md#glslc-outputs-in-a-zip-archive-and-those-nros-compile-at-run-time)
 - 24 shaders before the control-length fix, 1,355 after. → [SPKG packages in KTGL LINKDATA](../reference/shaders.md#spkg-packages-in-ktgl-linkdata)
+- the split finds no shaders in a RomFS made of `.sdfdata`, `.sdftoc` and `.sdfver` files. → [Shader records in Snowdrop SDF data slices](../reference/shaders.md#shader-records-in-snowdrop-sdf-data-slices)
 - `[RED] ... PKZ version 0x10000: only 0x20000 is known` stops the shader split. → [PKZ version 1 archives](../reference/shaders.md#pkz-version-1-archives)
 - the split finds zero shaders and refuses the shader build, although the CPK's DAT archives contain the objects passed to NVN at boot. → [Size-delimited PlatinumGames shader objects](../reference/shaders.md#size-delimited-platinumgames-shader-objects)
 - the split finds no shaders, although `main` contains NVN control sections; it reports that no data record or code-address pair identifies their code areas. → [Needle shader containers in main and GFD archives](../reference/shaders.md#needle-shader-containers-in-main-and-gfd-archives)

docs/development.md

@@ -145,7 +145,7 @@ What the script does:
    A Vulkan game without SPIR-V files in its RomFS (Super Mario Galaxy) gets the same kind of run: the game boots in the Mac host with `DM_VK_CAPTURE` pointing at `<id>_build/extract/<version>/vk_capture` until every SPIR-V key in the profile's `vk_capture.tsv` is there (Galaxy: 3,089, about 7 s after launch), and gives up after 120 s.
    So does an OpenGL game with no GLSL source in its RomFS or `main` (Balatro: LÖVE puts its shaders together at run time, in Lua): `DM_GL_CAPTURE` at `<id>_build/extract/<version>/gl_capture` until every file in the profile's `gl_capture.tsv` is there (Balatro: 35, made before its first frame). Without the profile's tables (a provisional profile) a capture stops once nothing has been added for 20 s.
    The Vulkan and GL captures also run on a Mac whose GPU has no Metal 4, such as a virtual Mac (GitHub's `macos-26` runner). Those two layers draw on classic Metal queues of their own, so there the NVN layer starts only its classic part and drops every present, which a headless capture does not need (`nvn_mtl.m` `mtl_device_classic`; `DM_TEST_NO_METAL4=1` does the same on any Mac). The GLSLC capture boots an NVN game, which needs Metal 4.
-5. It splits out the shaders and checks them against the profile's tables (a Vulkan profile: it translates the RomFS's SPIR-V and the capture with `vk_shaders.py` instead; an OpenGL one: the GLSL sources and the capture with `gl_shaders.py`).
+5. It splits out the shaders from the game and every extracted DLC RomFS and checks them against the profile's tables (a Vulkan profile: it translates the RomFS's SPIR-V and the capture with `vk_shaders.py` instead; an OpenGL one: the GLSL sources and the capture with `gl_shaders.py`). Optional DLC that changes only the NVN shader union gets new tables in an external profile after the non-DLC shaders and facts have matched; registered module builds and fixes stay in use.
 6. It builds the iOS shader libraries, with the Mac's own Metal compiler and spirv-cross when they differ from those in `dmsh_chain.lock` (`dm/gfx/tools/own_compilers.py`; one line in the output names them). If a registered profile's `dmsh_chain.h` does not hold the version the source tree gives (a chain change left it behind), the script writes it again before any host is built, so it shows as changed in the checkout (`dmsh_build.py` would stop otherwise). Then the script makes the profile's `c1_full.tsv` and `cb_extent.tsv` from this build's Mac outputs when they are missing or name another chain. These two tables are not committed and git ignores them; `build_ipa.sh` stops without a current `c1_full.tsv`. Each line in the output names the file.
    It then compares the build with the game's previous one, the newest other `b-*/` in `<id>_build/shader/cache/` (`dm/gfx/tools/shader_regressions.py`, which reads the two `_b_build.json` reports). The translation chain is one for every game, so a translator change is tested only on the games its author owns; shaders that translated before and fail now are listed in `logs/shader_regressions.log` and stop the build ([GitHub issue #272](https://github.com/seedds/nxnative/issues/272)). Report them; `--allow-shader-regressions` (`ALLOW_SHADER_REGRESSIONS = True` in Spyder) packs the app without them.
 7. It calls `build_ipa.sh`. Xcode builds the app in the game's own build folder (`<id>_build/ipa/build/ios/pkg/dd`: `build_ipa.sh` sets `SYMROOT` and `OBJROOT`, which a custom build location in Xcode's settings does not override), so a repeat build compiles only what changed and two games built at once never write the same app. Then `dm/ios/tools/pack_ipa.py` packs it with the shader libraries, the add-on content and the RomFS straight from where they are. It reads and deflates files on all cores and writes at about the T7's write speed (300 MB/s): Breath of the Wild's 16 GB IPA takes 57 s, Info-ZIP's copy and zip took 2 min 50 s. A repeat pack reuses the previous IPA through `<ipa>.idx`, which records each packed file's size, times and inode: the unchanged leading entries stay in place in an APFS clone of the old IPA, and after the first changed entry, an unchanged file is copied compressed from it (both only when the source folder is on APFS; elsewhere every file is deflated again). `build_ipa.sh` passes the RomFS, the add-on content, the pipeline archive and the shader libraries before the app, so after a host fix only the app is written: Breath of the Wild repacks in 12-23 s instead of 105-163 s.

docs/reference/memory.md

@@ -156,6 +156,13 @@ The guest address space, heaps, pools, virtual address memory and the allocators
 - **Found on:** [Pokémon Legends: Z-A](../../dm/games/0100F43008C44000/README.md) (2.0.2, SDK 19.3.11; one boot); [Paper Mario: The Thousand-Year Door](../../dm/games/0100ECD018EBE000/README.md): the allocator maps 1.5 GB at boot and 2.0 GB by the arrival in Rogueport. The `sdk` build is [Celeste](../../dm/games/01002B30028F6000/README.md)'s. Absolum's malloc StandardAllocator (40 GB, cut to 2.5 GB) is in [.NET NativeAOT malloc and the GC region range](#net-nativeaot-malloc-and-the-gc-region-range).
 - **Note:** on Pokémon Legends: Z-A, `QueryMemoryInfo` reporting more than 4 GB would take the game's big-memory path (heaps of allocatable − 4.27 GB instead of − 2.76 GB).
 
+### Separate virtual ranges in Snowdrop's page allocator
+
+- **Symptom:** a write to `0x20000ffff` in static initialization (`main+0x164990`), after `AllocateAddressRegion` returns `0x1803`; reducing the request to 3 GB still returns `0x1803` with the SDK's original large-region alignment. With only the outer ranges reduced, a later free crashes at `main+0x1668dc` after reading compressed file data as a slab-owner pointer.
+- **Cause:** the engine reserves three adjacent virtual ranges before committing pages in them. Its enclosing reservation exceeds the iPhone's available guest address space, and the engine does not check the failed reserve before deriving the page addresses. A smaller reservation also needs the SDK's node area and large-region alignment adapted: a separate metadata reservation comes first, and the 1 GB alignment wastes the remaining contiguous space. The large-allocation child has an independent maximum page count; reducing only the outer range lets it allocate across the boundary used by `free` to distinguish allocation classes.
+- **Fix:** the build-gated game cuts reduce the range sizes, their combined reservation and the large allocator's maximum page count together. The fixed metadata reserve remains large enough, and page allocation, commits and frees retain their behavior. The SDK build gets the existing [node-area cut](#the-sdks-address-region-node-area) and [32 MB alignment](#1-gb-alignment-of-regions-of-1-gb-and-more). Per-range occupancy and the original growth fallbacks remain **open** until sustained gameplay validates the smaller ranges; see the game note for the layout, cut sites and current validation.
+- **Found on:** [Mario + Rabbids Sparks of Hope](../../dm/games/0100317013770000/README.md) (1.6.2225577, SDK 13.3.5).
+
 ### .NET NativeAOT malloc and the GC region range
 
 - **Symptom:** `nn::diag` abort at `sdk+0x12dad0` in `nninitStartup`, called from `main+0x756310`.

docs/reference/nvn-objects.md

@@ -424,6 +424,13 @@ Games read this count as a position in their own command memory and size their h
 - `nvnTexturePoolRegisterImage` writes the same table as `RegisterTexture` (image handles are pool indices).
 - **Found on:** [Hyrule Warriors: Age of Calamity](../../dm/games/01002B00111A2000/README.md).
 
+### Image bindings supply shader-write intent
+
+- **Symptom:** `[RED] texture bound to an image slot lacks the shader-write usage`, and compute image writes go to a dummy texture.
+- **Cause:** Metal usage came only from the texture builder's `IMAGE` flag. A render texture may instead have only `COMPRESSIBLE`, then its descriptor be used in a shader image slot. Snowdrop writes these descriptors with `RegisterTexture`, not `RegisterImage`. Its depth-conversion compute shader `cf5ba107e29b51f4` gathers from a depth texture and unconditionally writes four pixels per invocation to a 256×256 R16F texture whose flags are `0x8`. Its Metal texture had read / render-target usage (`0x5`), so the image binding was rejected.
+- **Fix:** before a draw or dispatch uses a writable image, `image_prepare` resolves it with image intent and ensures its backing allows shader writes. `RegisterImage` also keeps this intent in its host snapshot; neither path changes the guest texture. A texture already rendered into is upgraded through `tex_upgrade`: end its render pass first, copy every mip / slice, replace all shared serials and invalidate cached sampled views. Load-only image slots accept read-only storage without upgrading, and ordinary render targets retain their original usage. `t_image_usage.m` exercises both registration entry points, a late registration while a render pass is open, the clear and other mip surviving the upgrade, and a GPU image write read through a sampled alias.
+- **Found on:** [Mario + Rabbids Sparks of Hope](../../dm/games/0100317013770000/README.md) (SDK 13.3.5, Snowdrop).
+
 ### Plural texture binds, builder-registered samplers, strided texel writes, device item 0x67
 
 - `nvnCommandBufferBindTextures` (the plural of `BindTexture`), `nvnSamplerPoolRegisterSamplerBuilder` (a sampler registered from its builder), `nvnTextureWriteTexelsStrided` (`WriteTexels` with row and image strides) and device item 0x67, which the game reads before every program it loads and uses as the alignment of its copies of the shader (answered 256, not measured).

docs/reference/services.md

@@ -356,7 +356,9 @@ Most service calls can be found before they crash, before the first boot or the
 ### oe calls answered without an applet proxy
 
 - **Symptom:** the first boot crashed in `oe::DisableAlbumImageTakenNotification`: a null read at `sdk+0x338070`, the first call in `nnMain` (`main+0x6b188c`); other `oe` calls that reach am, found in the imports before they were met.
+- **Symptom:** a null read at `sdk+0x36ec60` in `oe::EnableAlbumImageTakenNotification`, called from `nnMain` (`main+0x286268`).
 - **Cause:** the SDK asks am through the applet proxy the host never made.
+- **Fix:** `EnableAlbumImageTakenNotification` uses the same host implementation as the disable call: no system-album notifications are produced. Found on [Mario + Rabbids Sparks of Hope](../../dm/games/0100317013770000/README.md) (SDK 13.3.5); the next boot reaches the NVN frame loop.
 - **Fix:** hooked (`hle_oe.c`): `oe::DisableAlbumImageTakenNotification`; `oe::IsInStoreDemoModeEnabled` (no); `oe::GetDefaultDisplayResolution` and its change event; `oe::SendServerMaintenanceOverlayNotification`; `oe::SetUserInactivityDetectionTimeExtendedUnsafe`; `oe::GetPerformanceConfiguration`; `oe::SetAlbumImageOrientation`; `oe::Begin` / `EndAutoSleepDisabledSection` and `IsAutoSleepDisabled` (counted); `oe::ExecuteProgram` (logged red: the host runs one program); `oe::GetUserInactivityDetectionTimeExtensionMode` (0, as `IsUserInactivityDetectionTimeExtended` reports no). The ones with a symptom of their own follow: [The startup logo](#the-startup-logo), [Copyright frame buffer overlay](#copyright-frame-buffer-overlay), [Applet resource user ID, launch parameters and VR mode](#applet-resource-user-id-launch-parameters-and-vr-mode), [CPU overclock calls without an apm session](#cpu-overclock-calls-without-an-apm-session), [Focus handling mode decides whether focus changes are notified](#focus-handling-mode-decides-whether-focus-changes-are-notified).
 - **Found on:** [Monster Hunter Rise](../../dm/games/0100B04011742000/README.md) (SDK 11: `GetDefaultDisplayResolution`, `SendServerMaintenanceOverlayNotification`); [13 Sentinels: Aegis Rim](../../dm/games/01003FC01670C000/README.md) (SDK 11: `SetUserInactivityDetectionTimeExtendedUnsafe`); [Super Mario 3D World + Bowser's Fury](../../dm/games/010028600EBDA000/README.md) (SDK 10: `GetPerformanceConfiguration`, `SetAlbumImageOrientation`); [DRAGON QUEST XI S](../../dm/games/01006C300E9F0000/README.md) (SDK 7.3.2: the auto-sleep calls); [Octopath Traveler 0](../../dm/games/01005270232F2000/README.md) (`oe::ExecuteProgram`); [The Great Ace Attorney Chronicles](../../dm/games/010036E00FB20000/README.md) (SDK 10.4.1, Capcom's MT Framework: `DisableAlbumImageTakenNotification`, then `IsInStoreDemoModeEnabled` from the import list); [Donkey Kong Country: Tropical Freeze](../../dm/games/0100C1F0051B6000/README.md) (SDK 4.4.0: `GetUserInactivityDetectionTimeExtensionMode`, a null read at `sdk+0x15b73c` when the first level's cinematic camera starts, `CScreenDimmingUtil::DisableScreenDimming`). After all of its fixes The Great Ace Attorney Chronicles runs its frame loop at 30 fps; the black screen it showed was its textures, never uploaded (CPU-written textures, the profile's `tex_flush_upload` and `tex_dirty_upload`: [Mapped-range flushes taken as uploads (`tex_flush_upload`)](nvn-objects.md#mapped-range-flushes-taken-as-uploads-tex_flush_upload), [CPU writes with no flush, caught by page protection (`tex_dirty_upload`)](nvn-objects.md#cpu-writes-with-no-flush-caught-by-page-protection-tex_dirty_upload)).
 

docs/reference/shaders.md

@@ -116,6 +116,14 @@ Finding, keying and translating shaders (`dm/gfx/tools/shader_split.py`, `dmsh_b
 - **Reader:** `kpka_jobs` maps the PAKs, takes the newest copy of each path, peeks at 4 decompressed bytes and inflates only SDF entries (`sdf_shaders`).
 - **Found on:** [Monster Hunter Rise](../../dm/games/0100B04011742000/README.md): 63,111 shaders (30,233 vertex, 21,475 pixel, 11,271 compute, 53 geometry, 33 hull, 46 domain) and 102,345 pairs, in 5 s with 12 jobs.
 
+### Shader records in Snowdrop SDF data slices
+
+- **Symptom:** the split finds no shaders in a RomFS made of `.sdfdata`, `.sdftoc` and `.sdfver` files.
+- **Cause:** Snowdrop stores the shaders in individually compressed records inside its `BERG` data slices. These SDF archives are a different format from RE Engine's shader definitions above. A shader record has a 20-byte header `{u32 0xabcd00c6, flags, 0, raw length, stored length}`, then one zstd frame; reflection data precedes the NVN sections inside it.
+- **Fix:** `shader_split.py` recognizes `BERG` by magic and reads the length-delimited compressed records (`snowdrop_shaders`). Every control section is followed immediately by its code, with lengths from control +0x20 and +0x704; they are byte-packed, so a control need not be aligned. A single record can hold more than one shader. Each section must pass the control / code / stage checks. Programs hold one stage, and the vertex shaders take the [single-stage cover](#single-stage-programs-paired-with-a-cover). Scanning for the record header avoids mistaking a coincidental zstd magic inside compressed data for another asset.
+- **DLC:** additional shader records ship in the DLC's data slices. The NSP script includes each extracted DLC RomFS as an extra shader root, without an ExeFS module, and fingerprints its index and extraction timestamps for split reuse. Vulkan and OpenGL builds include the DLC roots too. When installing or removing optional DLC changes an NVN shader union, the script uses an external profile with new shader tables only after every non-DLC shader row and fact matches the registered tables; package indices may change. The registered module builds, settings and build-gated cuts are preserved. A cached proof of this comparison permits repeat builds to reuse the split.
+- **Found on:** [Mario + Rabbids Sparks of Hope](../../dm/games/0100317013770000/README.md) (Snowdrop, SDF version 22).
+
 ### SPK shader package in an MT Framework ARC
 
 - **How it is stored:** the engine's one shader package, `sc\NX_64\root` (`SPK\0`, type hash `0x02358e1a`) in `nativeNXx64/sa/NX_64/root.arc` ([01](../agents/01-extract.md)). After the package's tables every shader is a code area followed directly by its control section, back to back to the end of the file: 230 vertex shaders, then 230 pixel shaders.