Octopath Traveler 0 boots on the Mac: Unreal pak and Oodle shader decode, the host's allocator for Unreal

Octopath Traveler 0 1.0.5 (Unreal Engine 4.26 / 4.27 mobile renderer, SDK 19.3.13) plays from boot through character
creation into the story on the Mac (15 minutes with the iPhone memory layout, no crash); not yet run on an iPhone.
- Shaders: the 10,324 are in the two shader code archives of the game's pak, each one Oodle-compressed. uepak.py reads
  the version 11 pak (AES-256 index; the key is the window of main that decrypts it, never printed or written; AES
  through CommonCrypto) and the archives; the Mac host decodes them with the game's own OodleLZ_Decompress
  (DM_OODLE_JOB, hle_oodle.c, build-gated) and shader_split.py reads the decoded LSLG records (profile key
  shader_ue_decode; the NSP script's decode_ue_shaders stage). Bindings measured: Engage's separate texture layout.
- Memory: Unreal's nninitStartup reserves a 62 GB address region and its FMallocBinned3 37 GB of it. A cut skips the
  allocator's constructor and the host puts its own FMalloc vtable on a malloc zone in the object (hle_uemalloc.c);
  the region is capped at 2 GB with the game's own -SwitchVirtualAddressSpaceSize (new profile key game_args, also
  for the iOS app) and SDK 19's node area is cut to 62 MB as Engage's, for a 3.75 GB guest block.
- Host: game arguments are set before nninitStartup, as on hardware; an undefined weak import nothing defines binds
  to 0, as rtld binds it (Tears of the Kingdom boots the same with the old and new hosts: its weak glslc* imports
  now resolve to 0); nvnCommandBufferFenceSync (OP_FENCE_SYNC), the texture / sampler pool getters,
  vi::GetLatestFrameNumber / ListFrameHistory; DM_ALIAS_WINDOWS for sizing fixed-window allocators.
- Rewriter: a byte jump table whose add sits 17 instructions before its br (jump_bytes_far, tried last).
- build_ipa.sh quotes FONT_PY (a python path with spaces failed the font step).
Open: 61 vertex shaders pick a texture slot at run time (dynamic handle), 16 compute shaders with 3D image atomics
do not build. The profile (dm/games/01005270232F2000) holds the shader tables, translation chain b-ba76ba138e78.
This commit is contained in:
seedds committed 2026-10-20 23:01:20 +08:00
1 parent 9c23f51370
commit 476b1135b4
33 files changed
+16500 -15

No files matched your search

+1
View File
@@ -0,0 +1 @@
# generated
1 # generated
+125
View File
@@ -0,0 +1,125 @@
# chain b-ba76ba138e78 2026-10-20T21:58:38+08:00
006a737523e70720
00da789b4ca18c03
00e5f2d3b5851fab
01c68522dd339947
044e7e0b19ae6481
06ab3ebbdc2177e1
0a0551f70d9c192b
0bce8ae9934a6dce
0cf17304495aa7dd
143cf7401bbd4376
14c00b3713e2b615
15e60bb36751538b
175c5afb589f33c9
1a350dd0d607e2d4
1af892cf934fc8f0
1dcf7963bfc4e093
1e1245a19b392ba5
1f4530824a808831
1f6e17337a7574f8
20aede653fcc6bb9
2392fb90cd9fa28e
23e3a0be0542bd20
24284846733abdd0
248e2932b77e88e0
2507b2967a9d618d
26ae65e8a3433f74
2bc7b889dd8b9612
30793382e220fb22
32f303aac4a64b69
379e597ecf193953
388d6d1b0c817b87
38e609e83109d7df
3ccd68e767373832
3de42b90591721c2
3e490cb64462f248
3e57d65cf687cc52
3e6f5903c4fffd28
3f31425fc0cabbac
40375f3250772ee1
416d73ab1cf6392e
41b77498b5d60c6f
41f6dd8d3fcd4da0
431ede4b30597c18
4367afcd3bafd093
462b254bb40d9e3b
47faedc352e6220f
49621957acd4158d
49f6023fcbdb4bee
4a0b160cc8334154
4a997acb7180e4d9
4b830b4629930a42
4cd0d18194fe108a
4d8caa770f68daa4
52dd36356caea987
5ae4eb5623e9bedf
5dff948bcbdf975a
62f6575e9a9e12fe
63837bfe0156f621
645d53144063bfa0
64b17e88d4df40fe
64ee9d1948139094
6d5bc03adb4eeea4
7114294dcbbb2db2
73795cc711cb9093
7404200a26d068ea
74dc46ac2fe84144
779b5730c19ca30f
79836c60f9f29512
7a4a8129afd6939c
7c996a2c2575f47c
7d1dd3eedb270069
7eafc9f557909f8f
88fb79889ffd0e92
89c0fb23f677430f
8a8404d2c48ee58d
8d460f0497612c5f
8dea4a0ae4a727ca
90f03461cf9d693d
918ef76ef4a4719f
919ca4c67680905f
932c7c6704948e2c
95d9181ada99f966
96ed34f374a36f68
98b614d3a587d38a
99499447aead65de
9b9b831b86f2a96a
9c50583d3aaf9872
9d2df1495afd147a
9da2eb9bcdddc6d0
a073bc1a294b70bb
a2cbfdc6e6e89f11
a40192569d73c527
a942b69e884f50b8
ad95c1bb802230e6
ae93bf6abcdc10b9
b0c5f8e8d4b85ae5
b296dece365133fd
b741908ff586e384
bbcad6d40d65bef9
bbf9ef3e84e5cfe1
becc79d370730635
c86cc595faca8820
d7f0bf1032d3932b
d889176067f2a41c
da63f2ef2867ba30
dd94a4edad966037
e20edc269d403a38
e2a6675d216d9d7f
e3c75303af0ad5d8
e47a29735b72d56c
e50f790acecdcd73
e52fc0f2d1f7a28a
e77f6aa16838f769
ebf8917bedf51f31
eede63ffdb991a1f
ef4f43bc08fee992
ef7a8a78fd5c217c
efd220d4adbf694d
f0e6f0d6be98d147
f1322c464eefc910
f91e3a24828308fd
f997b5ea5f9c6edf
fa253bc6db371323
fe4d4b7fd6469b9f
1 # chain b-ba76ba138e78 2026-10-20T21:58:38+08:00
2 006a737523e70720
3 00da789b4ca18c03
4 00e5f2d3b5851fab
5 01c68522dd339947
6 044e7e0b19ae6481
7 06ab3ebbdc2177e1
8 0a0551f70d9c192b
9 0bce8ae9934a6dce
10 0cf17304495aa7dd
11 143cf7401bbd4376
12 14c00b3713e2b615
13 15e60bb36751538b
14 175c5afb589f33c9
15 1a350dd0d607e2d4
16 1af892cf934fc8f0
17 1dcf7963bfc4e093
18 1e1245a19b392ba5
19 1f4530824a808831
20 1f6e17337a7574f8
21 20aede653fcc6bb9
22 2392fb90cd9fa28e
23 23e3a0be0542bd20
24 24284846733abdd0
25 248e2932b77e88e0
26 2507b2967a9d618d
27 26ae65e8a3433f74
28 2bc7b889dd8b9612
29 30793382e220fb22
30 32f303aac4a64b69
31 379e597ecf193953
32 388d6d1b0c817b87
33 38e609e83109d7df
34 3ccd68e767373832
35 3de42b90591721c2
36 3e490cb64462f248
37 3e57d65cf687cc52
38 3e6f5903c4fffd28
39 3f31425fc0cabbac
40 40375f3250772ee1
41 416d73ab1cf6392e
42 41b77498b5d60c6f
43 41f6dd8d3fcd4da0
44 431ede4b30597c18
45 4367afcd3bafd093
46 462b254bb40d9e3b
47 47faedc352e6220f
48 49621957acd4158d
49 49f6023fcbdb4bee
50 4a0b160cc8334154
51 4a997acb7180e4d9
52 4b830b4629930a42
53 4cd0d18194fe108a
54 4d8caa770f68daa4
55 52dd36356caea987
56 5ae4eb5623e9bedf
57 5dff948bcbdf975a
58 62f6575e9a9e12fe
59 63837bfe0156f621
60 645d53144063bfa0
61 64b17e88d4df40fe
62 64ee9d1948139094
63 6d5bc03adb4eeea4
64 7114294dcbbb2db2
65 73795cc711cb9093
66 7404200a26d068ea
67 74dc46ac2fe84144
68 779b5730c19ca30f
69 79836c60f9f29512
70 7a4a8129afd6939c
71 7c996a2c2575f47c
72 7d1dd3eedb270069
73 7eafc9f557909f8f
74 88fb79889ffd0e92
75 89c0fb23f677430f
76 8a8404d2c48ee58d
77 8d460f0497612c5f
78 8dea4a0ae4a727ca
79 90f03461cf9d693d
80 918ef76ef4a4719f
81 919ca4c67680905f
82 932c7c6704948e2c
83 95d9181ada99f966
84 96ed34f374a36f68
85 98b614d3a587d38a
86 99499447aead65de
87 9b9b831b86f2a96a
88 9c50583d3aaf9872
89 9d2df1495afd147a
90 9da2eb9bcdddc6d0
91 a073bc1a294b70bb
92 a2cbfdc6e6e89f11
93 a40192569d73c527
94 a942b69e884f50b8
95 ad95c1bb802230e6
96 ae93bf6abcdc10b9
97 b0c5f8e8d4b85ae5
98 b296dece365133fd
99 b741908ff586e384
100 bbcad6d40d65bef9
101 bbf9ef3e84e5cfe1
102 becc79d370730635
103 c86cc595faca8820
104 d7f0bf1032d3932b
105 d889176067f2a41c
106 da63f2ef2867ba30
107 dd94a4edad966037
108 e20edc269d403a38
109 e2a6675d216d9d7f
110 e3c75303af0ad5d8
111 e47a29735b72d56c
112 e50f790acecdcd73
113 e52fc0f2d1f7a28a
114 e77f6aa16838f769
115 ebf8917bedf51f31
116 eede63ffdb991a1f
117 ef4f43bc08fee992
118 ef7a8a78fd5c217c
119 efd220d4adbf694d
120 f0e6f0d6be98d147
121 f1322c464eefc910
122 f91e3a24828308fd
123 f997b5ea5f9c6edf
124 fa253bc6db371323
125 fe4d4b7fd6469b9f
+2
View File
@@ -0,0 +1,2 @@
// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 01005270232F2000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/
#define DMSH_CHAIN "ba76ba138e78"
+54
View File
@@ -0,0 +1,54 @@
# OCTOPATH TRAVELER 0: Square Enix, Unreal Engine 4.26 / 4.27 (mobile renderer, SF_GLSL_SWITCH_MOBILE shaders), SDK 19.3.13, NVN, CRIWARE sound and movies; update 1.0.5.
# Started from the NSP script's provisional profile (dm/games/profile_guess.py) and brought up on the Mac (docs/agents)
# ---- Identity ----
title_id 01005270232F2000
program_id 01005270232F2000
# ---- App ----
name OCTOPATH TRAVELER 0
ipa OctopathTraveler0.ipa
app_category public.app-category.games
pad_layout simple
# ---- Versions ----
build v327680 1.0.5 main=9fe60a212cff9b9a3656ab63d4f882e5 subsdk0=ca7f78a09c547c54beac3a33ed6bf364 sdk=7bba185a35ee9b4d0af66a4df910a823
package v327680
shader_versions v327680
# ---- Files ----
# subsdk0 is NintendoSDK_movie (main imports from it)
modules main subsdk0 sdk
romfs_marker Octopath_Traveler0/Content/Paks/pakchunk0-Switch.pak
zip_store .awb:.pak:.ttf:.txt:.usm:.metallib
# ---- Graphics ----
graphics nvn
# ---- Memory ----
# Unreal Engine: nninitStartup (main+0x12d74) needs virtual address memory and reserves one address region for the engine (-SwitchVirtualAddressSpaceSize= in MB, game_args below; 62 GB by default).
# Its allocator, an FMallocBinned3 (37 GB of address space), is replaced by the host's (patch_nso.py cut, hle_uemalloc.c), and the SDK's node area is cut to 62 MB, so the region holds what the engine maps
# itself (709 MB at character creation, spread over 1 GB). Alias region = node area + up to 1 GB of alignment (regions of 1 GB and more are 1 GB aligned) + the 2 GB region; 256 MB stacks, 128 MB heap
game_args -SwitchVirtualAddressSpaceSize=2048
mem_reserve 0xF0000000
mem_alias 0xC8000000
mem_stack 0x10000000
app_pool 0xCD500000
# ---- NVN driver constant segments ----
# Measured over all translated shaders: separate textures at c2 0x168 + 8 x slot and separate samplers at 0x568 + 8 x slot (Ryujinx handles fp_t_cb196610_1056805A ...: words 0x5A / 0x15A), as Engage
nvn_tex_base 0x168
nvn_tex_slots 128
nvn_samp_base 0x568
nvn_img_base 0x120
nvn_img_slots 8
nvn_ssbo_bases vs=0x110 ps=0x510 cs=0x310
# ---- Shader split ----
# The decoded shader code archives: the engine's global shaders and the game's material shaders
shader_groups ShaderArchive-Global-=global ShaderArchive-=game
shader_note OCTOPATH TRAVELER 0 shaders
# The shaders are in the shader code archives of the pak (ShaderArchive-*.ushaderbytecode), each one Oodle-compressed; the NSP script decodes them on the Mac with the game's own Oodle (uepak.py, DM_OODLE_JOB)
shader_ue_decode on
# Unreal combines vertex and pixel shaders per draw from its pipeline state: every vertex shader is made to link with any pixel shader
shader_separable on
1 # OCTOPATH TRAVELER 0: Square Enix, Unreal Engine 4.26 / 4.27 (mobile renderer, SF_GLSL_SWITCH_MOBILE shaders), SDK 19.3.13, NVN, CRIWARE sound and movies; update 1.0.5.
2 # Started from the NSP script's provisional profile (dm/games/profile_guess.py) and brought up on the Mac (docs/agents)
3 # ---- Identity ----
4 title_id 01005270232F2000
5 program_id 01005270232F2000
6 # ---- App ----
7 name OCTOPATH TRAVELER 0
8 ipa OctopathTraveler0.ipa
9 app_category public.app-category.games
10 pad_layout simple
11 # ---- Versions ----
12 build v327680 1.0.5 main=9fe60a212cff9b9a3656ab63d4f882e5 subsdk0=ca7f78a09c547c54beac3a33ed6bf364 sdk=7bba185a35ee9b4d0af66a4df910a823
13 package v327680
14 shader_versions v327680
15 # ---- Files ----
16 # subsdk0 is NintendoSDK_movie (main imports from it)
17 modules main subsdk0 sdk
18 romfs_marker Octopath_Traveler0/Content/Paks/pakchunk0-Switch.pak
19 zip_store .awb:.pak:.ttf:.txt:.usm:.metallib
20 # ---- Graphics ----
21 graphics nvn
22 # ---- Memory ----
23 # Unreal Engine: nninitStartup (main+0x12d74) needs virtual address memory and reserves one address region for the engine (-SwitchVirtualAddressSpaceSize= in MB, game_args below; 62 GB by default).
24 # Its allocator, an FMallocBinned3 (37 GB of address space), is replaced by the host's (patch_nso.py cut, hle_uemalloc.c), and the SDK's node area is cut to 62 MB, so the region holds what the engine maps
25 # itself (709 MB at character creation, spread over 1 GB). Alias region = node area + up to 1 GB of alignment (regions of 1 GB and more are 1 GB aligned) + the 2 GB region; 256 MB stacks, 128 MB heap
26 game_args -SwitchVirtualAddressSpaceSize=2048
27 mem_reserve 0xF0000000
28 mem_alias 0xC8000000
29 mem_stack 0x10000000
30 app_pool 0xCD500000
31 # ---- NVN driver constant segments ----
32 # Measured over all translated shaders: separate textures at c2 0x168 + 8 x slot and separate samplers at 0x568 + 8 x slot (Ryujinx handles fp_t_cb196610_1056805A ...: words 0x5A / 0x15A), as Engage
33 nvn_tex_base 0x168
34 nvn_tex_slots 128
35 nvn_samp_base 0x568
36 nvn_img_base 0x120
37 nvn_img_slots 8
38 nvn_ssbo_bases vs=0x110 ps=0x510 cs=0x310
39 # ---- Shader split ----
40 # The decoded shader code archives: the engine's global shaders and the game's material shaders
41 shader_groups ShaderArchive-Global-=global ShaderArchive-=game
42 shader_note OCTOPATH TRAVELER 0 shaders
43 # The shaders are in the shader code archives of the pak (ShaderArchive-*.ushaderbytecode), each one Oodle-compressed; the NSP script decodes them on the Mac with the game's own Oodle (uepak.py, DM_OODLE_JOB)
44 shader_ue_decode on
45 # Unreal combines vertex and pixel shaders per draw from its pipeline state: every vertex shader is made to link with any pixel shader
46 shader_separable on
@@ -0,0 +1,2 @@
# Prewarm seed list (shipped in the app, next to shader_map.tsv). Rows: S<TAB>shader key<TAB>stage / R<TAB>render pipeline key (pso_skey hex) / C<TAB>compute shader key.
# Not recorded yet for this game (DM_PSO_RECORD): an empty list = no prewarm. Row order = first-use order; do not sort.
1 # Prewarm seed list (shipped in the app, next to shader_map.tsv). Rows: S<TAB>shader key<TAB>stage / R<TAB>render pipeline key (pso_skey hex) / C<TAB>compute shader key.
2 # Not recorded yet for this game (DM_PSO_RECORD): an empty list = no prewarm. Row order = first-use order; do not sort.
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,2 @@
# kind: evidence control-section key conflict rulings: empty
dm_ctl_sha16 chosen_pc class candidates records
1 # kind: evidence control-section key conflict rulings: empty
2 dm_ctl_sha16 chosen_pc class candidates records
File diff suppressed because it is too large. Load diff
+2
View File
@@ -40,6 +40,7 @@ None of them has a default game: without one they stop and list the profiles. To
| `graphics` | `nvn` (the default) or `vulkan`: the game renders with Vulkan and the host implements its `vk*` imports on Metal (`dm/gfx/vk/hle_vk.m`, Hades). Its shaders are the game's SPIR-V files, translated by `dm/gfx/tools/vk_shaders.py <RomFS> <shader dir> [--ios <dir>]` into `<dir>/vk/`; `build_ipa.sh` then ships `$MSL_IOS/vk` instead of the NVN chain libraries, and the NVN shader tables stay empty |
| `mem_reserve`, `mem_alias`, `mem_stack`, `app_pool` | Guest address-space partition and application pool (`hle_svc.c`) |
| `mem_used` | Memory `GetInfo(UsedMemorySize)` reports beside the heap (default `0x4000000`, 64 MB). `SetHeapSize` checks a request against the real size of the loaded modules, so a game that sizes its heap as total − used and has more than 64 MB of modules needs at least their size here. Hyrule Warriors: Age of Calamity: 66.6 MB of modules, so the default made `SetHeapSize(pool − 64 MB)` fail and `nnMain` return at once; it sets 72 MB |
| `game_args` | Arguments the game gets when the command line gives none (the iOS app, the NSP script's Mac runs), split at spaces; `dmhost <dir> -- …` overrides them. The SDK hands them to the game before `nninitStartup`, as on hardware (`main.c`). Octopath Traveler 0 caps the address region Unreal's `nninitStartup` reserves (62 GB by default) with `-SwitchVirtualAddressSpaceSize=2048` (MB) |
| `present_interval` | `game` (the default): frames are paced by the present interval the game sets, as on the Switch (a frame presented with interval 2 stays on screen for two display ticks; a game that keeps asking for 2 for a second gets the 30 fps target, and on iOS the display link then runs at 30 Hz; Mario 1.2.1 switches to 2 for single frames after a slow one; Engage asks for 2 on the Switch, and a rewriter cut on its 2.0.0 `main` makes it ask for 1, see `0100A6301214E000/game.tsv`). `display`: one frame per display tick whatever the game asks, which is only right for games whose logic does not count frames (D2R's runs on its own 25 Hz clock). `DM_PRESENT_INTERVAL=game` / `display` overrides it, also from `Documents/env.txt` |
| `fps_target` | Frame rate cap of the rewriter's frame step, for builds that have one (`patch_nso.py` `FRAME_STEP`: Tears of the Kingdom 1.4.3, Breath of the Wild 1.9.0), or the engine's own frame limit the host writes into game memory (`hle_fpslock.c`: Monster Hunter Rise 16.0.2, which also needs `present_interval display`). `0` (the default): the game's own step. `30` / `60`: the host decides each frame's step in whole vsyncs and keeps the frame on screen that long (`hle_nvn.c` `fstep_tick`, `nvn_fstep.h`): 1 at a 60 cap, 2 when 60 does not hold or at a 30 cap (and while a prerendered movie plays), plus one vsync while the game is behind real time after a late frame. Presents follow that hold instead of the interval the game sets. `DM_FPS_TARGET` overrides it, also from `Documents/env.txt`; the log line `frame step: game time … of real time` checks the game's speed |
| `frame_step` | `ticks` (the default): the frame step above. `legacy`: each game's method before it, for the iPhone A/B test: Tears of the Kingdom steps by the measured frame time capped at `fps_target`, Breath of the Wild runs its own interval controller with the 60 fps cuts. `DM_FRAME_STEP=ticks` / `legacy` overrides it, also from `Documents/env.txt` |
@@ -63,6 +64,7 @@ None of them has a default game: without one they stop and list the profiles. To
| `shader_separable` | `on` / `off` (the default): the game builds single-stage programs and combines them per draw, so the real vertex / pixel pairs exist only at run time. `shader_split.py` then pairs every vertex shader with a cover of pixel shaders that together read every input any pixel shader reads, and `dmsh_build.py`'s `vs_pad` makes every vertex shader link with any pixel shader (13 Sentinels) |
| `shader_vk_capture` | `on` / `off` (the default), with `graphics vulkan`: the game generates its SPIR-V shaders at run time (Super Mario Galaxy: Hagi's GX shaders, 3,089 modules created from its pipeline database while it shows "Initializing system..."), so no game file holds them. They are captured on the Mac with `DM_VK_CAPTURE=<dir>` (`dm/gfx/vk/hle_vk.m`; the NSP script does it itself, booting the game in the Mac host until every key the profile's `vk_capture.tsv` lists is there), and `vk_shaders.py` translates the capture folder with the RomFS |
| `shader_glslc_capture` | `on` / `off` (the default): the game compiles some shaders at run time with its own GLSLC (Fire Emblem: Three Houses: 9 built-in KTGL shaders at boot), so no game file holds them. They are captured on the Mac with `DM_GLSLC_CAPTURE=<dir>` (`dm/host/hle_glslc.c`; the NSP script does this itself, booting the game in the Mac host until every capture file the profile's `shader_map.tsv` names is there), and `shader_split.py` reads `<version dir>/glslc_capture/*.glslc` (beside that version's `RomFS`) or `$DM_GLSLC_CAPTURE`; without the folder those shaders get no translation and the tables differ from the profile's |
| `shader_ue_decode` | `on` / `off` (the default): an Unreal Engine game keeps its shaders in the shader code archives of its paks (`*.ushaderbytecode`), each shader Oodle-compressed, which only the game's own Oodle decodes. The NSP script (`decode_ue_shaders`) takes the archives out of the paks with `dm/gfx/tools/uepak.py` (the pak index is AES-encrypted with a key it finds in `main`), boots the game in the Mac host with `DM_OODLE_JOB` (`dm/host/hle_oodle.c`: the game's `OodleLZ_Decompress` decodes every shader right before `nnMain`, then the host exits) and writes the decoded archives to `<version dir>/ue_shaders`, where `shader_split.py` reads them (`DM_UE_SHADERS` overrides). Octopath Traveler 0: 10,596 shaders in 2 s. Its vertex and pixel shaders combine per draw, so the profile also sets `shader_separable` |
The translator (`dmsh_xlate/Program.cs`) still has Mario's texture / sampler bases and the depth convention built in. It uses the bases only to look up per-slot texture / image types in `shader_facts.tsv`; `dmsh_build.py` decodes the bindings from the profile. So a profile with other bases (Engage: `0x168` / `0x568`) builds as long as its facts table has no per-slot texture / image entries; otherwise `dmsh_build.py` stops, and the translator must take the bases from the profile first.
+1
View File
@@ -138,6 +138,7 @@ def header(g):
f'#define DM_GAME_MEM_STACK {g.int("mem_stack"):#x}ull',
f'#define DM_GAME_APP_POOL {g.int("app_pool"):#x}ull',
f'#define DM_GAME_MEM_USED {int(g.get("mem_used", "0x4000000"), 0):#x}ull', # used memory reported beside the heap (hle_svc.c GetInfo UsedMemorySize)
f'#define DM_GAME_ARGS {q(g.get("game_args", ""))}', # game arguments when the command line gives none (main.c; Octopath Traveler 0: Unreal's -SwitchVirtualAddressSpaceSize=)
f'#define DM_GAME_NVN_SSBO_BASE(stage) ({sb})',
f'#define DM_GAME_NVN_TEX_BASE {g.int("nvn_tex_base"):#x}',
f'#define DM_GAME_NVN_TEX_SLOTS {g.int("nvn_tex_slots")}',
+9 -1
View File
@@ -166,6 +166,14 @@ def guess(F, hooks=None):
G['device_api'] = ('53.313', f'SDK {F["sdk"]}: the game wants NVN API 53')
if 'Unity IL2CPP' in F.get('main_markers', ()):
G['nvn_tex_base'] = ('0x168', 'Unity: separate texture bindings (Engage)'); G['nvn_tex_slots'] = ('128', '')
if F.get('main_markers', set()) & {'Unreal', 'Unreal 4'} and R['exts'].get('.pak') and 'nvnBootstrapLoader' in imp:
G['shader_ue_decode'] = ('on', 'Unreal Engine with paks: the shaders are Oodle-compressed in the paks\' shader code archives (Octopath Traveler 0), decoded by booting the game on the Mac')
G['shader_separable'] = ('on', 'Unreal combines vertex and pixel shaders per draw from its pipeline state')
G['shader_groups'] = ('ShaderArchive-Global-=global ShaderArchive-=game', 'the decoded archives: the engine\'s global shaders and the game\'s material shaders')
G['nvn_tex_base'] = ('0x168', 'Unreal: separate texture bindings (Octopath Traveler 0, measured over all its shaders)'); G['nvn_tex_slots'] = ('128', '')
G['mem_reserve'] = ('0x1028000000', 'Unreal: nninitStartup reserves a 62 GB address region and its FMallocBinned3 takes 37 GB of it, so a 64 GB alias region for the bring-up on the Mac. '
'An iPhone needs the allocator cut for the build and a capped region (game_args), as Octopath Traveler 0\'s profile (docs/agents/03)')
G['mem_alias'] = ('0x1008000000', '')
G.update(preset)
return G
@@ -221,7 +229,7 @@ def heap_signals(log, g):
# ---- Checking the rules against the hand-made profiles ----
DEFAULTS = {'graphics': 'nvn', 'shader_vk_capture': 'off', 'shader_main_glslc': 'off', 'shader_separable': 'off', 'shader_glslc_capture': 'off', 'shared_font': 'bfttf',
DEFAULTS = {'graphics': 'nvn', 'shader_vk_capture': 'off', 'shader_main_glslc': 'off', 'shader_separable': 'off', 'shader_glslc_capture': 'off', 'shader_ue_decode': 'off', 'shared_font': 'bfttf',
'device_api': '55.15', 'op_mode': 'console', 'pad_layout': 'simple', 'nvn_tex_base': '0x20', 'nvn_tex_slots': '32'}
NO_SIGNAL = {'pad_layout', 'op_mode'} # no static signal outside the engine presets; the player switches the pad layout in game, DM_OPMODE in env.txt sets the mode
+10
View File
@@ -31,6 +31,16 @@ HLE_FORCE("_ZN2nn2vi12SetLayerCropEPNS0_5LayerEiiii", "layer crop; the host reco
HLE_FORCE("_ZN2nn2vi19SetLayerScalingModeEPNS0_5LayerENS0_11ScalingModeE", "layer scaling mode; the host present does its own scaling", Result, h_vi_SetScaling, void *layer, int mode) {
(void)layer; LOG("vi::SetLayerScalingMode %d (ignored)", mode); return RES_OK;
}
// Unreal Engine paces its frames by the layer's frame number (Octopath Traveler 0, main+0x10140e8); the SDK's versions read the Android surface of the vi service the host never made
// (SIGSEGV in android::Surface::getQueuedFrameNumber). The host's frame number is the count of frames the game has presented
uint64_t dm_nvn_frame_count(void);
HLE_FORCE("_ZN2nn2vi20GetLatestFrameNumberEPmPKNS0_5LayerE", "the layer's latest frame number: the frames presented so far", Result, h_vi_FrameNo, uint64_t *out, const void *layer) {
(void)layer; *out = dm_nvn_frame_count(); return RES_OK;
}
HLE_FORCE("_ZN2nn2vi16ListFrameHistoryEPNS0_9FrameInfoEiPKNS0_5LayerE", "the layer's frame history: none kept, 0 entries", int, h_vi_FrameHist, void *out, int n, const void *layer) {
(void)out; (void)layer; static int once; if (!once++) LOG("vi::ListFrameHistory (%d entries asked): the host keeps no frame history, 0 returned", n);
return 0;
}
HLE_FORCE("_ZN2nn2vi12DestroyLayerEPNS0_5LayerE", "destroy layer; host static object", void, h_vi_DestroyLayer, void *layer) { (void)layer; }
HLE_FORCE("_ZN2nn2vi12CloseDisplayEPNS0_7DisplayE", "close display; host static object", void, h_vi_CloseDisp, void *disp) { (void)disp; }
HLE_FORCE("_ZN2nn2vi8FinalizeEv", "display service shutdown; the host has no vi service", void, h_vi_Fini, void) {}
+12
View File
@@ -236,6 +236,13 @@ NVN(nvnSamplerPoolInitialize, NVNboolean, NVNsamplerPool *sp, const NVNmemoryPoo
}
NVN(nvnSamplerPoolFinalize, void, NVNsamplerPool *sp) { (void)sp; }
NVN(nvnSamplerPoolRegisterSampler, void, const NVNsamplerPool *sp, int id, const NVNsampler *s) { mtl_reg_samp(dpool_base(g_spb, g_nspb, sp) + id, &((const samp_t *)s)->p); }
// Unreal Engine reads a descriptor pool's size back at device setup (Octopath Traveler 0)
NVN(nvnTexturePoolGetMemoryPool, const NVNmemoryPool *, const NVNtexturePool *tp) { return (const NVNmemoryPool *)((const dpool_t *)tp)->pool; }
NVN(nvnTexturePoolGetMemoryOffset, ptrdiff_t, const NVNtexturePool *tp) { return (ptrdiff_t)((const dpool_t *)tp)->off; }
NVN(nvnTexturePoolGetSize, int, const NVNtexturePool *tp) { return ((const dpool_t *)tp)->n; }
NVN(nvnSamplerPoolGetMemoryPool, const NVNmemoryPool *, const NVNsamplerPool *sp) { return (const NVNmemoryPool *)((const dpool_t *)sp)->pool; }
NVN(nvnSamplerPoolGetMemoryOffset, ptrdiff_t, const NVNsamplerPool *sp) { return (ptrdiff_t)((const dpool_t *)sp)->off; }
NVN(nvnSamplerPoolGetSize, int, const NVNsamplerPool *sp) { return ((const dpool_t *)sp)->n; }
// ---------- Textures ----------
#define TB(b) ((tex_builder_t *)(b))
@@ -701,6 +708,11 @@ NVN(nvnCommandBufferSignalEvent, void, NVNcommandBuffer *c, const NVNevent *e, N
const ev_t *ev = (const ev_t *)e; volatile uint32_t *val = ev->val; uint32_t mm = m;
memcpy(p + 8, &val, 8); memcpy(p + 16, &mm, 4); memcpy(p + 20, &v, 4);
}
// A fence recorded in a command buffer: the replay fences the sync with the submission that reaches this point (OP_FENCE_SYNC). Unreal Engine's NVN RHI fences its command buffers this way (Octopath Traveler 0)
NVN(nvnCommandBufferFenceSync, void, NVNcommandBuffer *c, NVNsync *s, NVNsyncCondition cond, int flags) {
(void)cond; (void)flags;
uint8_t *p = cmd_alloc(CB(c), 16); put_hdr(p, OP_FENCE_SYNC, 16); memcpy(p + 8, &s, 8);
}
NVN(nvnCommandBufferResetCounter, void, NVNcommandBuffer *c, NVNcounterType t) {
uint8_t *p = cmd_alloc(CB(c), 16); put_hdr(p, OP_RESET_COUNTER, 16); uint64_t tt = t; memcpy(p + 8, &tt, 8);
}
+2
View File
@@ -6733,6 +6733,8 @@ void mtl_exec(const uint8_t *p) {
g_rc_skip = known && (rc->mode ? x == y : x != y);
PD("E %llu conditional render %s addr=%#llx values %llu / %llu%s%s", g_stat_pass, rc->mode ? "draw if not equal" : "draw if equal", (unsigned long long)rc->addr, (unsigned long long)x, (unsigned long long)y, known ? "" : " (unknown, drawn)", g_rc_skip ? " → not drawn" : ""); }
break; }
case OP_FENCE_SYNC: { sync_t *s; memcpy(&s, a, 8); s->fence = g_fence + 1; s->fenced = 1; // the fence value this submission (segment) commits with
PD("Y fence_sync cmd=%p sync=%p fence=%llu", p, (void *)s, (unsigned long long)s->fence); break; }
case OP_RESET_COUNTER: { uint64_t t; memcpy(&t, a, 8); g_cnt_reset[t & 31]++; PD("Q reset type=%llu cmd=%p", (unsigned long long)t, p);
if (t == NVN_COUNTER_TYPE_SAMPLES_PASSED) {
#ifdef NVN_RT_RANGE_TEST
+1 -1
View File
@@ -100,7 +100,7 @@ enum {
OP_BIND_PROGRAM, OP_BIND_STATE, OP_BIND_VATTR, OP_BIND_VSTREAM, OP_BIND_VBUF,
OP_BIND_UBO, OP_BIND_SSBO, OP_BIND_TEX, OP_BIND_SAMP,
OP_STENCIL, OP_POLY_OFFSET, OP_DRAW, OP_DRAW_IDX, OP_DISPATCH, OP_BARRIER,
OP_SET_TEX_POOL, OP_SET_SAMP_POOL, OP_COPY_B2T, OP_COPY_STRIDE, OP_BLEND_COLOR, OP_COPY_T2B, OP_BIND_IMG, OP_COPY_T2T, OP_RESET_COUNTER, OP_CLEAR_TEX, OP_CLEAR_BUF, OP_DEPTH_CLAMP, OP_RENDER_COND, OP_COUNT_
OP_SET_TEX_POOL, OP_SET_SAMP_POOL, OP_COPY_B2T, OP_COPY_STRIDE, OP_BLEND_COLOR, OP_COPY_T2B, OP_BIND_IMG, OP_COPY_T2T, OP_RESET_COUNTER, OP_CLEAR_TEX, OP_CLEAR_BUF, OP_DEPTH_CLAMP, OP_RENDER_COND, OP_FENCE_SYNC, OP_COUNT_
};
// Payload (immediately follows op_hdr)
typedef struct { int32_t n; const tex_t *color[8]; view_t cview[8]; uint8_t has_cview[8]; const tex_t *depth; view_t dview; uint8_t has_dview; } p_set_rt;
+26 -1
View File
@@ -36,6 +36,7 @@ GROUPS, NOTE = GAME.pairs('shader_groups'), GAME.get('shader_note')
GLSLC = GAME.get('shader_main_glslc', 'off') == 'on' # GLSLC compiler outputs in main (glslc_shaders)
SEPARABLE = GAME.get('shader_separable', 'off') == 'on' # single-stage programs combined per stage at draw time (separable_pairs)
CAPTURE = GAME.get('shader_glslc_capture', 'off') == 'on' # GLSLC outputs the game compiles at run time, captured on the Mac (DM_GLSLC_CAPTURE, hle_glslc.c)
UE = GAME.get('shader_ue_decode', 'off') == 'on' # Unreal Engine shader code archives, decoded on the Mac with the game's own Oodle (uepak.py, DM_OODLE_JOB, hle_oodle.c)
CTL_MAGIC, GFX_MAGIC, CS_MAGIC = bytes.fromhex('34127698'), bytes.fromhex('78563412'), bytes.fromhex('79563412')
STAGES = ['vs', 'hs', 'ds', 'gs', 'ps', 'cs']
@@ -433,6 +434,24 @@ def separable_pairs(keys):
k = min(ins, key=lambda k: (-len(ins[k] & left), k)); cover.append(k); left -= ins[k]
return {(v, p) for v in keys if keys[v][1] == 'vs' for p in cover}, cover
UE_TAG = {'SV': 'vs', 'SH': 'hs', 'SD': 'ds', 'SG': 'gs', 'SP': 'ps', 'SC': 'cs'}
def ue_shaders(d, label):
"""An Unreal Engine shader code archive with every shader decoded (Octopath Traveler 0: the two .ushaderbytecode files of its pak, whose shaders are Oodle-compressed; the NSP script decodes them
on the Mac with the game's own Oodle, uepak.py): each shader is an LSLG record holding one stage's code area and control section (uepak.lslg); the stage is the record's tag (SV vertex,
SP pixel, SC compute), checked against the archive entry's frequency. The engine combines vertex and pixel shaders per draw, so the profile pairs them with shader_separable"""
import uepak
a, out = uepak.archive(d), []
for i, (off, size, raw, fr) in enumerate(a['shaders']):
if size != raw: red(f'{label}: shader {i} is still compressed (an archive the NSP script has not decoded)')
rec = uepak.lslg(d[a['code'] + off:a['code'] + off + size])
if not rec: red(f'{label}: shader {i} is not an LSLG record')
tag, ctl, code = rec; st = UE_TAG.get(tag)
if not st or uepak.FREQ.get(fr) != st: red(f'{label}: shader {i} has tag {tag} but frequency {fr}')
why = check(label, st, ctl, code)
if why: red(f'{label}: shader {i} ({st}): {why}')
out.append((f'{label}#UE{i}', st, ctl, code))
return out
def var_pairs(found):
"""(vertex key, pixel key) within the same BNSH variation"""
var = collections.defaultdict(dict)
@@ -508,6 +527,11 @@ def main():
names = sorted(f for f in os.listdir(cap) if f.endswith('.glslc')) if os.path.isdir(cap) else []
if not names: print(f'[RED] the profile has shader_glslc_capture on, but there are no captured GLSLC outputs in {cap}: the shaders the game compiles at run time will have no translation', flush=True)
for nm in names: found += glslc_shaders(open(os.path.join(cap, nm), 'rb').read(), f'/glslc_capture/{nm}')
if UE: # Unreal Engine shader code archives decoded on the Mac (uepak.py): <version dir>/ue_shaders, or DM_UE_SHADERS
ued = os.environ.get('DM_UE_SHADERS') or os.path.join(os.path.dirname(os.path.abspath(romfs)), 'ue_shaders')
names = sorted(f for f in os.listdir(ued) if f.endswith('.ushaderbytecode')) if os.path.isdir(ued) else []
if not names: red(f'the profile has shader_ue_decode on, but there are no decoded shader code archives in {ued} (the NSP script writes them)')
for nm in names: found += ue_shaders(open(os.path.join(ued, nm), 'rb').read(), f'/ue_shaders/{nm}')
results.append(('main', {skey(c, k): (w, s, c, k) for w, s, c, k in found}, var_pairs(found), {}))
zdicts = {k: v for r in romfs_v for k, v in zs_dicts(r).items()} # Tears of the Kingdom's zstd dictionaries (zs_dicts)
keys = {}
@@ -559,7 +583,8 @@ def main():
with open(os.path.join(pkg, name), 'wb') as f: f.write(b''.join(rr))
with open(mapout, 'w') as f:
kinds = ' / '.join(k for k, hit in (('BNSH / BAHS', any('#BNSH@' in r[4] or '#BAHS@' in r[4] for r in rows)), ('Unity Shader', any('#var' in r[4] and '#BNSH@' not in r[4] and '#MSHD#' not in r[4] and '#SDF#' not in r[4] for r in rows)),
('MSHD', any('#MSHD#' in r[4] for r in rows)), ('GLSLC', any('#GLSLC@' in r[4] for r in rows)), ('RE Engine SDF', any('#SDF#' in r[4] for r in rows))) if hit)
('MSHD', any('#MSHD#' in r[4] for r in rows)), ('GLSLC', any('#GLSLC@' in r[4] for r in rows)), ('RE Engine SDF', any('#SDF#' in r[4] for r in rows)),
('Unreal Engine shader code archives', any('#UE' in r[4] for r in rows))) if hit)
f.write(f'# kind: evidence {NOTE} ({kinds} in RomFS + main) → control section + code area keys (dm/gfx/tools/shader_split.py)\n')
f.write('# column 5 records where each key first appears; this game has no PC version to pair with\n')
f.write('package\tindex\tstage\tdm_ctl_sha16\tsource\n')
+214
View File
@@ -0,0 +1,214 @@
#!/usr/bin/env python3
# Unreal Engine 4 game files for the shader tools: pak files (version 11, UE 4.26 / 4.27), shader code archives (.ushaderbytecode version 2) and the job files the Mac host decodes Oodle with
# (DM_OODLE_JOB, dm/host/hle_oodle.c). Octopath Traveler 0 keeps everything in one pak, Octopath_Traveler0/Content/Paks/pakchunk0-Switch.pak (1.5 GB, 159,757 files): its index is AES-256 encrypted,
# its files are stored or Oodle-compressed in blocks. Its two shader code archives are stored, but every shader inside is Oodle-compressed (Mermaid, block header 8C 0A), which no decoder in this
# repository can undo, so the game's own OodleLZ_Decompress decodes them in the Mac host (profile key shader_ue_decode, the NSP script's decode_ue_shaders).
# The AES key is in main (Unreal embeds it): the 32-byte window of main's read-only or data segment that decrypts the index start to its mount point ("../../../"; Octopath Traveler 0: one window of
# 116,287 tried). It is only kept in memory, never printed or written.
# Pak footer (version 11, the last 0xdd bytes): encryption key GUID (16), encrypted index flag (u8), magic 0x5A6F12E1, version (u32), index offset (u64), index size (u64), index SHA-1 (20),
# 5 compression method names (32 bytes each).
# Index: mount point (FString), entry count (i32), path hash seed (u64), path hash index (i32 flag, then u64 offset, u64 size, SHA-1), full directory index (the same, encrypted like the index),
# encoded entries (i32 size + bytes), the count of unencoded entries (0 here). Directory index: i32 count, then per directory its name (FString) and its files (i32 count, then name + i32 offset into
# the encoded entries). FString: i32 length with the NUL, UTF-8; a negative length is UTF-16.
# Encoded entry: u32 bits (0-5 compression block size >> 11, 0x3f = a u32 follows; 6-21 block count; 22 encrypted; 23-28 compression method, 1-based; 29 stored size fits 32 bits; 30 raw size fits
# 32 bits; 31 offset fits 32 bits), then offset, raw size, stored size (compressed entries only) and, for several blocks or an encrypted one, each block's u32 stored size. The data starts after
# the FPakEntry the pak repeats in front of it (53 bytes, + 4 + 16 per block when compressed); blocks follow each other, 16-aligned when encrypted.
# Shader code archive (version 2): shader map hashes (20 bytes each), shader hashes (20 each), shader map entries {u32 first index, u32 shader count, u32 first preload, u32 preload count}, shader
# entries {u64 offset, u32 size, u32 raw size, u8 frequency} (17 bytes, packed), preload entries (16 each), shader indices (u32): each array is a u32 count and its elements; the code follows,
# offsets relative to its start. A shader whose size differs from its raw size is compressed.
# Usage: uepak.py list <pak> <main NSO> <out tsv> every file in the pak: path, raw size, stored size, compression, encrypted, blocks
# uepak.py job <RomFS> <main NSO> <work dir> the shader code archives of every pak, and an Oodle job of their compressed shaders
# uepak.py finish <work dir> <out dir> after the host has run the job: the archives with every shader decoded
import ctypes, json, os, struct, sys
PAK_MAGIC, ARCHIVE_VERSION = 0x5A6F12E1, 2
JOB_MAGIC, OUT_MAGIC = 0x4a4f4d44, 0x524f4d44 # "DMOJ", "DMOR" (hle_oodle.c)
FREQ = {0: 'vs', 1: 'hs', 2: 'ds', 3: 'ps', 4: 'gs', 5: 'cs'} # EShaderFrequency (4.26)
_cc = None
def aes_decrypt(key, data):
"""AES-256 ECB decryption through CommonCrypto (every Mac has it; no Python package needed)"""
global _cc
if _cc is None:
_cc = ctypes.CDLL('/usr/lib/libSystem.B.dylib').CCCrypt
_cc.restype = ctypes.c_int32
_cc.argtypes = [ctypes.c_uint32, ctypes.c_uint32, ctypes.c_uint32, ctypes.c_char_p, ctypes.c_size_t, ctypes.c_void_p, ctypes.c_char_p, ctypes.c_size_t, ctypes.c_void_p, ctypes.c_size_t, ctypes.POINTER(ctypes.c_size_t)]
out = ctypes.create_string_buffer(len(data)); moved = ctypes.c_size_t()
if _cc(1, 0, 2, key, len(key), None, data, len(data), out, len(data), ctypes.byref(moved)): raise ValueError('CCCrypt failed') # decrypt, AES, ECB
return out.raw[:moved.value]
def nso_image(path):
"""A decompressed NSO image and its (read-only start, data end)"""
import lz4.block
d = open(path, 'rb').read()
flags = struct.unpack_from('<I', d, 0xc)[0]; segs = [struct.unpack_from('<III', d, 0x10 + i * 0x10) for i in range(3)]; csz = struct.unpack_from('<III', d, 0x60)
img = bytearray(segs[2][1] + segs[2][2])
for i, (fo, mo, sz) in enumerate(segs):
raw = d[fo:fo + csz[i]]
img[mo:mo + sz] = lz4.block.decompress(raw, uncompressed_size=sz) if flags >> i & 1 else raw
return bytes(img), segs[1][1], segs[2][1] + segs[2][2]
class R:
def __init__(self, b, o=0): self.b, self.o = b, o
def take(self, fmt):
v = struct.unpack_from(fmt, self.b, self.o); self.o += struct.calcsize(fmt); return v if len(v) > 1 else v[0]
def raw(self, n): v = self.b[self.o:self.o + n]; self.o += n; return v
def fstr(self):
n = self.take('<i')
if n < 0: return self.raw(-2 * n).decode('utf-16-le')[:-1]
return self.raw(n)[:-1].decode('utf-8', 'replace') if n else ''
def footer(f):
f.seek(-0xdd, 2); ft = f.read(0xdd)
k = ft.find(struct.pack('<I', PAK_MAGIC))
if k < 1: raise ValueError('no pak footer')
ver, ioff, isz = struct.unpack_from('<IQQ', ft, k + 4)
if ver != 11: raise ValueError(f'pak version {ver}, only 11 is read')
methods = [ft[k + 44 + 32 * i:k + 76 + 32 * i].rstrip(b'\0').decode('latin1') for i in range(5)]
return ft[k - 1], ioff, isz, methods
def find_key(img, lo, hi, pak):
"""The AES key among main's 32-byte windows (read-only and data segments, 4-aligned, at least 20 distinct bytes): the one that decrypts the index start to a short mount point"""
with open(pak, 'rb') as f:
_, ioff, _, _ = footer(f); f.seek(ioff); blk = f.read(16)
for a in range(lo, hi - 32, 4):
w = img[a:a + 32]
if len(set(w)) < 20: continue
d = aes_decrypt(w, blk)
if 0 < struct.unpack_from('<i', d)[0] < 512 and d[4:7] == b'../': return w
return None
class Pak:
def __init__(self, path, key=None):
self.path, self.key = path, key; self.f = open(path, 'rb')
enc, ioff, isz, self.methods = footer(self.f)
if enc and not key: raise ValueError(f'{path}: the index is encrypted and no key was given')
r = R(self.read_at(ioff, isz, enc))
self.mount = r.fstr(); self.count = r.take('<i'); r.take('<Q')
if r.take('<i'): r.take('<QQ'); r.raw(20) # path hash index (not needed with the full directory index)
if not r.take('<i'): raise ValueError(f'{path}: no full directory index')
doff, dsz = r.take('<QQ'); r.raw(20)
self.enc = r.raw(r.take('<i'))
if r.take('<i'): raise ValueError(f'{path}: unencoded entries are not read')
d = R(self.read_at(doff, dsz, enc)); self.files = {}
for _ in range(d.take('<i')):
dn = d.fstr()
for _ in range(d.take('<i')): fn = d.fstr(); self.files[(dn + fn).lstrip('/')] = d.take('<i')
def read_at(self, off, n, enc):
self.f.seek(off); b = self.f.read((n + 15) & ~15 if enc else n)
return aes_decrypt(self.key, b)[:n] if enc else b
def entry(self, path):
r = R(self.enc, self.files[path]); v = r.take('<I')
bsz = (v & 0x3f) << 11
if v & 0x3f == 0x3f: bsz = r.take('<I')
nblk, enc, meth = v >> 6 & 0xffff, v >> 22 & 1, v >> 23 & 0x3f
off = r.take('<I' if v >> 31 & 1 else '<Q'); raw = r.take('<I' if v >> 30 & 1 else '<Q')
size = r.take('<I' if v >> 29 & 1 else '<Q') if meth else raw
cur, blocks = 53 + (4 + 16 * nblk if meth else 0), []
if nblk == 1 and not enc: blocks = [(cur, size)]
else:
for _ in range(nblk): s = r.take('<I'); blocks.append((cur, s)); cur += (s + 15) & ~15 if enc else s
return dict(off=off, raw=raw, size=size, method=self.methods[meth - 1] if meth else '', encrypted=enc, block_size=bsz, blocks=blocks)
def read(self, path):
"""A stored file's bytes (compressed files need the Oodle job)"""
e = self.entry(path)
if e['method']: raise ValueError(f'{path} is {e["method"]}-compressed in the pak')
return self.read_at(e['off'] + 53, e['size'], e['encrypted'])
def archive(b):
"""A shader code archive's tables: {'maps': [(first index, count)], 'shaders': [(offset, size, raw size, frequency)], 'indices': [...], 'code': start of the code}"""
r = R(b)
if r.take('<I') != ARCHIVE_VERSION: raise ValueError('not a version 2 shader code archive')
nmap = r.take('<I'); r.raw(20 * nmap); nsh = r.take('<I'); r.raw(20 * nsh)
maps = [r.take('<IIII')[:2] for _ in range(r.take('<I'))]
shaders = [r.take('<QIIB') for _ in range(r.take('<I'))]
r.raw(16 * r.take('<I'))
n = r.take('<I'); indices = list(struct.unpack_from(f'<{n}I', b, r.o)); r.o += 4 * n
return dict(maps=maps, shaders=shaders, indices=indices, code=r.o)
def decoded_archive(b, raws):
"""The archive with every shader replaced by its decoded bytes (raws, in entry order): the same tables, each entry's size = its raw size, the code laid out again
(the preload entries keep their old file ranges; only the shader tools read these copies)"""
a = archive(b); r = R(b); r.take('<I')
for elem in (20, 20, 16): r.raw(elem * r.take('<I'))
head, ents_at = b[:r.o + 4], r.o + 4; n = len(a['shaders']); tail_at = ents_at + 17 * n
ents, off = bytearray(), 0
for (_, _, raw, fr), d in zip(a['shaders'], raws):
if len(d) != raw: raise ValueError(f'decoded shader of {len(d)} bytes, the archive says {raw}')
ents += struct.pack('<QIIB', off, raw, raw, fr); off += raw
return head + bytes(ents) + b[tail_at:a['code']] + b''.join(raws)
def paks(romfs):
"""Every pak under the RomFS"""
return sorted(os.path.join(dp, f) for dp, _, fs in os.walk(romfs) for f in fs if f.endswith('.pak'))
def job(romfs, main_nso, work):
"""The shader code archives of every pak under the RomFS into <work>/archives, and one Oodle job (<work>/oodle.job) of every compressed shader in them; <work>/job.json lists them"""
os.makedirs(os.path.join(work, 'archives'), exist_ok=True)
img, lo, hi = nso_image(main_nso); key = None; listing, recs = [], []
for pak in paks(romfs):
with open(pak, 'rb') as f: enc = footer(f)[0]
if enc and key is None:
key = find_key(img, lo, hi, pak)
if key is None: raise ValueError(f'{pak}: the index is encrypted and no window of main decrypts it')
p = Pak(pak, key)
for path in sorted(x for x in p.files if x.endswith('.ushaderbytecode')):
b = p.read(path); a = archive(b); name = os.path.basename(path)
open(os.path.join(work, 'archives', name), 'wb').write(b)
comp = [i for i, (_, size, raw, _) in enumerate(a['shaders']) if size != raw]
for i in comp:
off, size, raw, _ = a['shaders'][i]; recs.append((name, i, b[a['code'] + off:a['code'] + off + size], raw))
listing.append(dict(name=name, pak=os.path.relpath(pak, romfs), path=path, shaders=len(a['shaders']), compressed=len(comp)))
with open(os.path.join(work, 'oodle.job'), 'wb') as f:
f.write(struct.pack('<II', JOB_MAGIC, len(recs)))
for _, _, c, raw in recs: f.write(struct.pack('<II', len(c), raw)); f.write(c)
json.dump(dict(archives=listing, records=[(n, i) for n, i, _, _ in recs]), open(os.path.join(work, 'job.json'), 'w'), indent=1)
return listing, len(recs)
def finish(work, out):
"""The decoded archives into out, from <work>/oodle.job.out (the host's results, in job order)"""
meta = json.load(open(os.path.join(work, 'job.json')))
b = open(os.path.join(work, 'oodle.job.out'), 'rb').read()
magic, n = struct.unpack_from('<II', b)
if magic != OUT_MAGIC or n != len(meta['records']): raise ValueError('the job output does not match the job')
o, dec = 8, {}
for name, i in meta['records']:
l = struct.unpack_from('<I', b, o)[0]; o += 4
if not l: raise ValueError(f'{name}: shader {i} was not decoded')
dec[(name, i)] = b[o:o + l]; o += l
os.makedirs(out, exist_ok=True)
for x in meta['archives']:
src = open(os.path.join(work, 'archives', x['name']), 'rb').read(); a = archive(src)
raws = [dec.get((x['name'], i)) or src[a['code'] + off:a['code'] + off + size] for i, (off, size, _, _) in enumerate(a['shaders'])]
tmp = os.path.join(out, x['name'] + '.partial'); open(tmp, 'wb').write(decoded_archive(src, raws)); os.replace(tmp, os.path.join(out, x['name']))
return [x['name'] for x in meta['archives']]
def lslg(s):
"""A decoded Switch shader (LSLG): 'LSLG', a stage tag (SV vertex, SP pixel, SC compute, ...), a header with the shader's source name, then u32 code length, the code area,
u32 control length and the control section (0x880 bytes), and a short tail (24 or 29 bytes). → (tag, control section, code area), or None if the layout differs"""
if s[:4] != b'LSLG': return None
k = s.find(b'\x34\x12\x76\x98')
if k < 8: return None
nctl, ncode = struct.unpack_from('<I', s, k - 4)[0], struct.unpack_from('<I', s, k + 0x704)[0]
c = k - 4 - ncode
if c < 4 or struct.unpack_from('<I', s, c - 4)[0] != ncode or k + nctl > len(s): return None
return s[4:6].decode('latin1'), s[k:k + nctl], s[c:c + ncode]
if __name__ == '__main__':
cmd = sys.argv[1] if len(sys.argv) > 1 else ''
if cmd == 'list':
img, lo, hi = nso_image(sys.argv[3])
with open(sys.argv[2], 'rb') as f: enc = footer(f)[0]
p = Pak(sys.argv[2], find_key(img, lo, hi, sys.argv[2]) if enc else None)
with open(sys.argv[4], 'w') as f:
f.write('path\traw\tstored\tmethod\tencrypted\tblocks\n')
for path in sorted(p.files):
e = p.entry(path); f.write(f"{path}\t{e['raw']}\t{e['size']}\t{e['method'] or 'none'}\t{e['encrypted']}\t{len(e['blocks'])}\n")
print(f'pak version 11, mount {p.mount}, {len(p.files)} files, compression {[m for m in p.methods if m]}')
elif cmd == 'job':
listing, n = job(*sys.argv[2:5])
for x in listing: print(f"{x['name']}: {x['shaders']} shaders, {x['compressed']} compressed")
print(f'{n} records in {os.path.join(sys.argv[4], "oodle.job")}')
elif cmd == 'finish':
print('decoded:', ', '.join(finish(*sys.argv[2:4])))
else: sys.exit(__doc__ or 'usage: uepak.py list | job | finish (see the header)')
+2 -2
View File
@@ -20,9 +20,9 @@ typedef struct { uint8_t *d; size_t n; } dm_blob; // One save file of the orig
enum { DT_INIT = 12, DT_RELA = 7, DT_RELASZ = 8, DT_JMPREL = 23, DT_PLTRELSZ = 2, DT_SYMTAB = 6, DT_STRTAB = 5,
DT_INIT_ARRAY = 25, DT_INIT_ARRAYSZ = 27, DT_RELRSZ = 35, DT_RELR = 36 };
enum { R_AARCH64_ABS64 = 257, R_AARCH64_GLOB_DAT = 1025, R_AARCH64_JUMP_SLOT = 1026, R_AARCH64_RELATIVE = 1027 };
#define ELF64_ST_BIND(i) ((i) >> 4) // symbol binding: 0 = STB_LOCAL
#define ELF64_ST_BIND(i) ((i) >> 4) // symbol binding: 0 = STB_LOCAL, 2 = STB_WEAK
#define ELF64_ST_VISIBILITY(o) ((o) & 3) // visibility: 0 = STV_DEFAULT
enum { STB_LOCAL = 0, STV_DEFAULT = 0 };
enum { STB_LOCAL = 0, STB_WEAK = 2, STV_DEFAULT = 0 };
// Run-time state of one NSO module
typedef struct {
+7
View File
@@ -44,6 +44,13 @@ uint64_t resolve_import(module_t *m, const char *name) {
return (uint64_t)hlecount_wrap("STUB", name, (void *)&unimpl_stubs[k]);
}
// The host implements this import (the swhle table), without allocating a stub for it
int hle_implements(const char *name) {
unsigned long secsz; const hle_ent *sec = (const hle_ent *)getsectiondata(&_mh_execute_header, "__DATA", "swhle", &secsz);
for (unsigned i = 0; sec && i < secsz / sizeof(hle_ent); i++) if (!strcmp(sec[i].name, name)) return 1;
return 0;
}
void stubs_report(void) { LOG("stubs=%u", g_nstub); }
// ---------- Host gate dispatch ----------
+50
View File
@@ -0,0 +1,50 @@
// The game's own Oodle decoder, for the offline shader tools. Unreal Engine games compress each shader in their shader code archives with Oodle (Octopath Traveler 0: every entry of the
// two .ushaderbytecode files in pakchunk0-Switch.pak, block header 8C 0A = Mermaid). Shaders are translated offline only and Oodle's decoder is not open source, so the NSP script decodes
// them once on the Mac with the copy statically linked into main: DM_OODLE_JOB=<job file> runs OodleLZ_Decompress on every record right before nnMain, writes <job file>.out and exits.
// Job file: "DMOJ", u32 count, then per record u32 compressed length, u32 raw length and the compressed bytes. Output: "DMOR", u32 count, then per record u32 decoded length
// (0 = failed) and the decoded bytes. Fixed addresses: each row runs only on the main build it was found in (AGENTS.md)
#include "hle.h"
#include <unistd.h>
static const struct { const char *build, *what; uint64_t off; } k_oodle[] = {
// Octopath Traveler 0 1.0.5: OodleLZ_Decompress, found by the function name its profiler hook passes (the string at main+0x5696117, referenced at main+0x4f77e64)
// and its argument checks (compBuf x0, rawBuf x2, rawLen x3, decBufBase x7)
{ "9fe60a212cff9b9a3656ab63d4f882e5", "Octopath Traveler 0 1.0.5", 0x4f77e10 },
};
// OodleLZ_Decompress(compBuf, compBufSize, rawBuf, rawLen, fuzzSafe, checkCRC, verbosity, decBufBase, decBufSize, fpCallback, callbackUserData, decoderMemory, decoderMemorySize, threadPhase).
// Arguments 9-14 go on the stack; all are 64-bit here so the Apple and AAPCS64 stack layouts agree (Apple packs smaller stack arguments)
typedef int64_t (*oodle_fn)(const void *, int64_t, void *, int64_t, int64_t, int64_t, int64_t, void *, int64_t, void *, void *, void *, int64_t, int64_t);
static uint32_t rd32(FILE *f) { uint32_t v = 0; if (fread(&v, 4, 1, f) != 1) die("DM_OODLE_JOB: the job file ends early"); return v; }
void dm_oodle_job_maybe(void) {
const char *job = getenv("DM_OODLE_JOB");
if (!job || !*job) return;
oodle_fn fn = NULL; const char *what = NULL;
for (int i = 0; i < g_nmod; i++) if (!strcmp(g_mod[i].name, "main"))
for (int k = 0; k < (int)(sizeof k_oodle / sizeof k_oodle[0]); k++)
if (!strcmp(g_mod[i].build_id, k_oodle[k].build)) { fn = (oodle_fn)(void *)(g_mod[i].base + k_oodle[k].off); what = k_oodle[k].what; }
if (!fn) die("DM_OODLE_JOB: no OodleLZ_Decompress address for main build %s", dm_main_build());
FILE *in = fopen(job, "rb"); if (!in) die("DM_OODLE_JOB: cannot open %s", job);
if (rd32(in) != 0x4a4f4d44) die("DM_OODLE_JOB: %s is not a job file", job);
uint32_t n = rd32(in);
char outp[1024]; snprintf(outp, sizeof outp, "%s.out", job);
FILE *out = fopen(outp, "wb"); if (!out) die("DM_OODLE_JOB: cannot write %s", outp);
uint32_t hdr[2] = {0x524f4d44, n}; fwrite(hdr, 4, 2, out);
size_t scratch_n = 16u << 20; void *scratch = malloc(scratch_n); // the decoder's work memory, so Oodle never calls its allocator
uint32_t bad = 0; uint64_t raw_total = 0;
for (uint32_t i = 0; i < n; i++) {
uint32_t cl = rd32(in), rl = rd32(in);
uint8_t *c = malloc(cl ? cl : 1), *r = malloc(rl ? rl : 1);
if (fread(c, 1, cl, in) != cl) die("DM_OODLE_JOB: record %u ends early", i);
int64_t got = fn(c, cl, r, rl, 1, 0, 0, NULL, 0, NULL, NULL, scratch, (int64_t)scratch_n, 3); // fuzz safe, no CRC check, silent, unthreaded
uint32_t ol = got == (int64_t)rl ? rl : 0;
if (!ol && bad++ < 8) LOG("[RED] DM_OODLE_JOB: record %u (%u → %u bytes) failed (decoder returned %lld)", i, cl, rl, (long long)got);
fwrite(&ol, 4, 1, out); if (ol) fwrite(r, 1, ol, out);
raw_total += ol; free(c); free(r);
}
fclose(in); fclose(out); free(scratch);
LOG("DM_OODLE_JOB: %u records decoded with %s's OodleLZ_Decompress (%u failed, %.1f MB out) → %s", n - bad, what, bad, raw_total / 1048576.0, outp);
fflush(stderr); _exit(bad ? 1 : 0);
}
+21
View File
@@ -26,6 +26,7 @@ static uint8_t *g_stack; static size_t g_stack_cap;
// The address lies in the guest address block (every guest thread stack is here: the game's own heap or the stack region); the big-stack entry (hle_bigstack.c) uses it to decide whether to switch stacks
int dm_guest_as_contains(uintptr_t a) { uintptr_t b = (uintptr_t)g_as; return g_as && a >= b && a - b < RESERVE; }
static void *alias_windows_thread(void *arg);
static void mem_init(void) {
if (g_as) return;
g_as = mmap(NULL, RESERVE, PROT_NONE, MAP_PRIVATE | MAP_ANON, -1, 0);
@@ -41,6 +42,8 @@ static void mem_init(void) {
g_heap = g_stack + g_stack_cap; g_heap_cap = RESERVE - (size_t)(g_heap - g_as);
if (!(g_alias_bits = calloc((g_alias_cap >> 18) + 1, sizeof *g_alias_bits))) die("alias region page bitmap calloc");
LOG("guest AS %p +%#llx heap %p alias %p stack %p", g_as, (unsigned long long)RESERVE, g_heap, g_alias, g_stack);
const char *aw = getenv("DM_ALIAS_WINDOWS");
if (aw && atoi(aw) > 0) { pthread_t th; if (!pthread_create(&th, NULL, alias_windows_thread, (void *)(uintptr_t)atoi(aw))) pthread_detach(th); }
}
// Application memory pool = the 3285 MB the Switch (4 GB model) gives applications. At startup (main+0x1b66160) the game sets the whole total reported by QueryMemoryInfo as the heap, retrying with 2 MB less on failure,
@@ -93,6 +96,24 @@ static int64_t alias_pages(uint64_t off, uint64_t n, int map) {
return map ? ch : -ch;
}
// Diagnostic DM_ALIAS_WINDOWS=<window MB>: every 30 s, the mapped memory of each window of that size in the alias region (windows aligned to the size in absolute addresses, counted from the page
// bitmap; windows with nothing mapped are left out). For sizing an allocator that reserves fixed, aligned windows and commits pages from each window's start: Unreal's FMallocBinned3 takes
// 74 pools of 512 MB (Octopath Traveler 0, main+0x9f748)
static void *alias_windows_thread(void *arg) {
uint64_t win = (uint64_t)(uintptr_t)arg << 20, a0 = (uint64_t)(uintptr_t)g_alias, a1 = a0 + g_alias_cap;
for (;;) {
sleep(30);
char line[8192]; size_t o = 0; uint64_t tot = 0;
for (uint64_t w = a0 & ~(win - 1); w < a1 && o < sizeof line - 64; w += win) {
uint64_t n = 0, p0 = (w > a0 ? w - a0 : 0) >> 12, p1 = ((w + win < a1 ? w + win : a1) - a0) >> 12;
for (uint64_t p = p0; p < p1; p++) n += atomic_load(&g_alias_bits[p >> 6]) >> (p & 63) & 1;
if (n) { o += snprintf(line + o, sizeof line - o, " +%#llx:%.1f", (unsigned long long)(w - (a0 & ~(win - 1))), n * 4096 / 1048576.0); tot += n; }
}
LOG("[mem] alias windows of %llu MB (window offset:MB mapped), %.1f MB in all:%s", (unsigned long long)(win >> 20), tot * 4096 / 1048576.0, line);
}
return NULL;
}
int svc_thread(int num, uint64_t *regs); // hle_thread.c: thread and synchronization kind
uint64_t svc_dispatch(int num, uint64_t *regs, uint64_t orig_addr) {
+75
View File
@@ -0,0 +1,75 @@
// Unreal Engine's allocator on a host malloc zone. Octopath Traveler 0's engine allocator is an FMallocBinned3 constructed in place in nninitStartup; its constructor reserves 37 GB of
// address space (74 pools of 512 MB), which an iPhone cannot give (about 5 GB without the extended virtual addressing entitlement), and its pools are too uneven to shrink (patch_nso.py
// CUTS: the constructor call is cut). The host puts this vtable in the object instead: every FMemory allocation reaches it through the object (FPlatformMemory::BaseAllocator returns it,
// and the thread-local storage allocator calls its Malloc / Free slots), and the guest's own trivial functions stay in the slots that do nothing.
// The FMalloc slots (UE 4.26): 0-1 destructors, 2-5 FExec, 6 Malloc, 7 TryMalloc, 8 Realloc, 9 TryRealloc, 10 Free, 11 QuantizeSize, 12 GetAllocationSize, 13 Trim, 14 SetupTLSCachesOnCurrentThread,
// 15 ClearAndDisableTLSCachesOnCurrentThread, 16 InitializeStatsMetadata, 17 UpdateStats, 18 GetAllocatorStats, 19 DumpAllocatorStats, 20 IsInternallyThreadSafe, 21 ValidateHeap,
// 22 GetDescriptiveName, 23-25 OnMallocInitialized / OnPreFork / OnPostFork (the thread-local callbacks main+0x12e80 / main+0x12ec0 call +0x30 and +0x50).
// Fixed addresses: each row runs only on the main build it was found in (AGENTS.md)
#include "hle.h"
#include <malloc/malloc.h>
#include <stdatomic.h>
static const struct { const char *build, *what; uint64_t obj, vtable, cut; } k_ue[] = {
// Octopath Traveler 0 1.0.5: the FMallocBinned3 object (main+0x68022c8, bss), its vtable (main+0x5b06f38, 26 slots), the cut constructor call (main+0x12e40, a nop after the rewrite)
{ "9fe60a212cff9b9a3656ab63d4f882e5", "Octopath Traveler 0 1.0.5", 0x68022c8, 0x5b06f38, 0x12e40 },
};
enum { NSLOT = 26 };
static malloc_zone_t *g_zone;
static _Atomic int64_t g_used, g_peak; // bytes in use (usable sizes) and their high-water mark, for the [mem] line (one per 256 MB of new peak)
static void used(int64_t d) {
int64_t after = atomic_fetch_add(&g_used, d) + d, peak = atomic_load(&g_peak);
while (after > peak && !atomic_compare_exchange_weak(&g_peak, &peak, after)) {}
if (after > peak && after >> 28 != peak >> 28) LOG("[mem] Unreal heap (host malloc zone) up to %.2f GB in use", after / 1073741824.0);
}
static size_t align_of(uint32_t a) { return a > 16 ? a : 16; } // 0 = DEFAULT_ALIGNMENT, 16 bytes as Binned3 gives
static void *alloc(size_t n, uint32_t a) {
void *p = align_of(a) == 16 ? malloc_zone_malloc(g_zone, n ? n : 1) : malloc_zone_memalign(g_zone, align_of(a), n ? n : 1);
if (p) used((int64_t)malloc_size(p));
return p;
}
static void release(void *p) { if (p) { used(-(int64_t)malloc_size(p)); malloc_zone_free(g_zone, p); } }
static void *realloc_aligned(void *p, size_t n, uint32_t a) {
if (!p) return alloc(n, a);
if (!n) { release(p); return NULL; }
size_t old = malloc_size(p);
if (align_of(a) == 16) { void *q = malloc_zone_realloc(g_zone, p, n); if (q) used((int64_t)malloc_size(q) - (int64_t)old); return q; }
if (!((uintptr_t)p & (align_of(a) - 1)) && old >= n) return p;
void *q = alloc(n, a); if (!q) return NULL;
memcpy(q, p, old < n ? old : n); release(p); return q;
}
static void *m_Malloc(void *self, size_t n, uint32_t a) {
(void)self; void *p = alloc(n, a);
if (!p) die("Unreal heap: out of memory (%zu bytes, alignment %u, %.2f GB in use)", n, a, atomic_load(&g_used) / 1073741824.0);
return p;
}
static void *m_TryMalloc(void *self, size_t n, uint32_t a) { (void)self; return alloc(n, a); }
static void *m_Realloc(void *self, void *p, size_t n, uint32_t a) {
(void)self; void *q = realloc_aligned(p, n, a);
if (!q && n) die("Unreal heap: out of memory in Realloc (%zu bytes, alignment %u, %.2f GB in use)", n, a, atomic_load(&g_used) / 1073741824.0);
return q;
}
static void *m_TryRealloc(void *self, void *p, size_t n, uint32_t a) { (void)self; return realloc_aligned(p, n, a); }
static void m_Free(void *self, void *p) { (void)self; release(p); }
static size_t m_QuantizeSize(void *self, size_t n, uint32_t a) { (void)self; (void)a; return malloc_good_size(n); }
static bool m_GetAllocationSize(void *self, void *p, size_t *out) { (void)self; if (!p) return false; *out = malloc_size(p); return true; }
static void m_Nothing(void *self) { (void)self; } // Trim (its bool argument is ignored), the thread-local cache setup and teardown, DumpAllocatorStats
static void *g_vt[NSLOT];
// Called once the modules are loaded and relocated, before any guest code runs
void dm_ue_malloc_install(void) {
for (int i = 0; i < g_nmod; i++) if (!strcmp(g_mod[i].name, "main"))
for (int k = 0; k < (int)(sizeof k_ue / sizeof k_ue[0]); k++) if (!strcmp(g_mod[i].build_id, k_ue[k].build)) {
uint8_t *b = g_mod[i].base;
if (*(uint32_t *)(b + k_ue[k].cut) != 0xd503201f) { LOG("[RED] %s: main is rewritten without the allocator cut (patch_nso.py): Unreal's own FMallocBinned3 runs", k_ue[k].what); return; }
memcpy(g_vt, b + k_ue[k].vtable, sizeof g_vt); // the relocated vtable: the trivial slots stay the guest's
g_vt[6] = (void *)m_Malloc; g_vt[7] = (void *)m_TryMalloc; g_vt[8] = (void *)m_Realloc; g_vt[9] = (void *)m_TryRealloc; g_vt[10] = (void *)m_Free;
g_vt[11] = (void *)m_QuantizeSize; g_vt[12] = (void *)m_GetAllocationSize;
g_vt[13] = g_vt[14] = g_vt[15] = g_vt[19] = (void *)m_Nothing;
g_zone = malloc_create_zone(0, 0); malloc_set_zone_name(g_zone, "guest Unreal heap");
*(void **)(b + k_ue[k].obj) = g_vt;
LOG("%s: Unreal's allocator (FMallocBinned3 at main+%#llx) runs on a host malloc zone", k_ue[k].what, (unsigned long long)k_ue[k].obj);
}
}
+8 -2
View File
@@ -17,6 +17,7 @@ pthread_key_t g_keyres[2], g_keyaspect, g_keyaspectui; // g_keyaspect: aspect
enum { KEY_K18, KEY_KTLS, KEY_KTLR, KEY_KGATE, KEY_KRES0, KEY_KRES1, KEY_KASPECT, KEY_KASPECTUI, KEY_N };
uint64_t resolve_import(module_t *m, const char *name);
int hle_implements(const char *name);
void gate_register(uint64_t ret_addr, const char *kind, int arg, uint64_t orig);
static pthread_key_t key_value(unsigned i) {
@@ -252,7 +253,7 @@ static void relocate(module_t *m) {
case DT_RELR: relr = d->d_un.d_ptr; break;
case DT_RELRSZ: relrsz = d->d_un.d_val; break;
}
int cnt[3] = {0}, nint = 0;
int cnt[3] = {0}, nint = 0, nweak0 = 0;
unsigned nrelr = relr ? relr_apply(m, relr, relrsz, (uint64_t)m->base) : 0;
const Elf64_Rela *tabs[2] = {(const Elf64_Rela *)(m->base + rela), (const Elf64_Rela *)(m->base + jmprel)};
uint64_t sizes[2] = {relasz, pltsz};
@@ -273,7 +274,12 @@ static void relocate(module_t *m) {
if (!v) v = self;
if (v != self && nint++ < 8) LOG("%s: %s bound to an earlier module's definition (%s)", m->name, m->strtab + s->st_name, module_of((void *)v) ? module_of((void *)v)->name : "?");
}
if (!v) v = resolve_sym(m, m->strtab + s->st_name);
// An undefined weak symbol that no module defines and the host does not implement is 0, as Horizon's rtld binds it: the code tests its address before calling it.
// Octopath Traveler 0's main imports the thread_local init function _ZTHN13LowLevelTasks5FTask10ActiveTaskE weakly; a named stub made it look defined and the game called it
const char *nm = m->strtab + s->st_name;
int weak0 = !v && !s->st_shndx && ELF64_ST_BIND(s->st_info) == STB_WEAK && !guest_sym(nm) && !hle_implements(nm);
if (weak0 && nweak0++ < 8) LOG("%s: undefined weak %s bound to 0", m->name, nm);
if (!v && !weak0) v = resolve_sym(m, nm);
if (type == R_AARCH64_JUMP_SLOT) { *where = v; cnt[2]++; }
else { *where = v + r->r_addend; cnt[1]++; } // ABS64 / GLOB_DAT
}
+11 -2
View File
@@ -25,15 +25,18 @@ static void guest_main(void) {
LOG(">>> nnosInitialize"); ((fn2)req("nnosInitialize"))(hmain, 0);
((fn0)req("_init_libc1"))();
LOG(">>> nninitInitializeSdkModule"); ((fn0)req("nninitInitializeSdkModule"))();
// The game arguments are in place before nninitStartup, as on hardware (nn::os takes them from the process argument area in nnosInitialize): Unreal Engine's nninitStartup
// (Octopath Traveler 0, main+0x12afc) reads -SwitchVirtualAddressSpaceSize= and its other memory options there
((fnp)req("_ZN2nn2os11SetHostArgcEi"))((void *)(uint64_t)g_argc);
((fnp)req("_ZN2nn2os11SetHostArgvEPPc"))(g_argv);
LOG(">>> nninitStartup"); ((fn0)req("nninitStartup"))();
((fn0)req("_init_libc2"))();
dm_prof_mark("② SDK init (nnosInitialize → nninitStartup)");
run_module_inits(); // rtld callback B: each module's DT_INIT (global constructors)
dm_prof_mark("③ module init_arrays");
((fnp)req("_ZN2nn2os11SetHostArgcEi"))((void *)(uint64_t)g_argc);
((fnp)req("_ZN2nn2os11SetHostArgvEPPc"))(g_argv);
{ void dm_guest_spawn_selftest(void); dm_guest_spawn_selftest(); } // DM_GUEST_SPAWN_TEST self-test of host-created guest threads (hle_guestthread.c)
{ void dm_padmode_boot(void); dm_padmode_boot(); } // Touch overlay present: repoint the layout prefixes to the touch version (dm_padmode.c); must happen before the game builds its panels
{ void dm_oodle_job_maybe(void); dm_oodle_job_maybe(); } // DM_OODLE_JOB: decode a job file with the game's own Oodle and exit (hle_oodle.c, for the offline shader tools)
LOG(">>> nnMain"); ((fn0)req("nnMain"))();
LOG("nnMain returned");
}
@@ -89,6 +92,11 @@ static void host_preload(int argc, char **argv) {
int i = 2;
for (; i < argc && strcmp(argv[i], "--") && n < MAX_MOD; i++) names[n++] = argv[i];
if (i < argc && !strcmp(argv[i], "--")) for (i++; i < argc && g_argc < 63; i++) g_argv[g_argc++] = argv[i];
else if (*DM_GAME_ARGS) { // no "--" on the command line (the iOS app, the NSP script's Mac runs): the game profile's game_args, split at spaces
char *s = strdup(DM_GAME_ARGS);
for (char *t = strtok(s, " "); t && g_argc < 63; t = strtok(NULL, " ")) g_argv[g_argc++] = t;
LOG("game arguments from the game profile: %s", DM_GAME_ARGS);
}
if (!n) { static const char *def[] = DM_GAME_MODULES; for (int k = 0; k < DM_GAME_NMOD && n < MAX_MOD; k++) names[n++] = def[k]; } // default module list from the game profile's modules (Mario: main sdk)
load_modules(argv[1], names, n);
uint64_t ta = now_ns();
@@ -104,6 +112,7 @@ static void host_preload(int argc, char **argv) {
if (g_flag.bootprof) LOG("① breakdown (after loading): probes / tick gates / keys %.1f ms, prefetch start (reading the list) %.1f ms, stub report %.1f ms", (tb - ta) / 1e6, (tc - tb) / 1e6, (now_ns() - tc) / 1e6);
dm_prof_mark("① host loading (read NSO / relocate / forced takeover)");
{ void loader_hash_log(void); loader_hash_log(); } // DM_LOADHASH=1: address-independent hash of the load result (loader.c)
{ void dm_ue_malloc_install(void); dm_ue_malloc_install(); } // Unreal's allocator on a host malloc zone, for the builds whose constructor call is cut (hle_uemalloc.c)
}
static struct { int argc; char **argv; pthread_t th; _Atomic int started; } g_pre;
static void *preload_thread(void *a) { (void)a; host_preload(g_pre.argc, g_pre.argv); return NULL; }
+30
View File
@@ -235,6 +235,21 @@ CUTS = {
# SleepThread and a poll), then steps the game by the time that really passed (+0x24, seconds). fps (+0x30) is 30, from engine data (main+0x166daa4)
(0x166db80, '14159f1a', '94078052', 'KIDS frame task wait: csinc w20,w8,wzr,ne (fps, at least 1) → mov w20,#60 (wait 1/60 s; the step stays the measured time)', 'host-only'),
],
# Octopath Traveler 0 1.0.5 (v327680) main: Unreal's nninitStartup (main+0x12d74) constructs the engine allocator, an FMallocBinned3, in place at main+0x68022c8 (main+0x12e40 → constructor
# main+0x9f480). Its constructor reserves 37 GB of address space (main+0x9f748: 74 pools of 512 MB, pool index = offset >> 29), which only the Mac can give: an iPhone has about 5 GB without the
# extended virtual addressing entitlement, and the pools are too uneven to shrink (at character creation the largest holds 68 MB, the 74 together 732 MB). The constructor is not called;
# the host puts its own allocator's vtable in the object instead (dm/host/hle_uemalloc.c: Malloc, Realloc, Free and the size queries on a host malloc zone), and every allocation goes through it
# (FMemory reaches the object through FPlatformMemory::BaseAllocator, main+0x132b4, and the thread-local storage allocator nninitStartup registers, main+0x12e80 / main+0x12ec0, through its vtable)
'9fe60a212cff9b9a3656ab63d4f882e5': [
(0x12e40, '90310294', '1f2003d5', 'nninitStartup: bl 0x9f480 (FMallocBinned3 constructor, 37 GB of address space) → nop; the host allocator is in the object already (hle_uemalloc.c)', 'host-only'),
],
# Octopath Traveler 0 1.0.5 (v327680) sdk (SDK 19.3.13): the AddressRegionManager node area 0x3fe00000 → 0x3e00000 (62 MB), as Engage's SDK 12 (above). The same constructor (sdk+0x1423ec): +0x1423fc
# sets the node area limit, +0x142468 registers it as an occupied region. Unreal's region (-SwitchVirtualAddressSpaceSize, the profile's game_args) then starts 62 MB into the alias region, and the guest
# block stays small enough for an iPhone next to the Unreal heap (hle_uemalloc.c), which lives outside it
'7bba185a35ee9b4d0af66a4df910a823': [
(0x1423fc, '08fca752', '087ca052', 'AddressRegionManager constructor: node area limit mov w8,#0x3fe00000 → mov w8,#0x3e00000', 'host-only'),
(0x142468, '09fca752', '097ca052', 'AddressRegionManager constructor: length of the node area registered as an occupied region mov w9,#0x3fe00000 → mov w9,#0x3e00000', 'host-only'),
],
}
# ---- Registered observation gates: {NSO build ID: [(offset, number, original instruction word, reason)]}. Numbers match OBS_* in the host's hle.h;
# the original instruction must be position independent (no PC-relative / branch / anything involving x18 or tpidr), asserted offline; when the host has no callback attached the gate block goes in and out once and returns ----
@@ -816,6 +831,20 @@ def tail_call(insl, j):
if 'x18' not in [p.reg_name(r) for r in p.regs_access()[1]]: continue
return p.mnemonic == 'ldr' and bool(re.fullmatch(r'x18, \[(x\d+|sp)(, #(0x[0-9a-f]+|\d+))?\]', p.op_str))
return False
def jump_bytes_far(insl, j):
"""The byte / halfword table form with its add further back (Octopath Traveler 0 main+0x37f9bc8: adr x18,#L; ldrb w0,[x17,x15]; add x18,x18,x0,lsl #2, then 17 register moves before br x18).
Up to 32 instructions back, and x18 must not change between the add and the br. Tried only where the 16-instruction search and the offset forms find nothing. Returns (base, width) or None"""
ctx = insl[max(0, j - 32):j]
for k in range(len(ctx) - 1, -1, -1):
p = ctx[k]
if p.mnemonic == 'add' and p.op_str.startswith('x18, x18,') and 'lsl #2' in p.op_str: break
if any(p.reg_name(r) == 'x18' for r in p.regs_access()[1]): return None
else: return None
rn = p.reg_name(p.operands[2].reg); idx = {'lr': 30, 'fp': 29}.get(rn, num(rn))
base = next((q.operands[1].imm for q in reversed(ctx[:k]) if q.mnemonic == 'adr' and q.op_str.startswith('x18,')), None)
ld = next((q for q in reversed(ctx[:k]) if q.mnemonic in ('ldrb', 'ldrsb', 'ldrh', 'ldrsh') and num(q.reg_name(q.operands[0].reg)) == idx), None)
if base is None or ld is None: return None
return base, 256 if ld.mnemonic in ('ldrb', 'ldrsb') else 65536
def disasm_fn(fa, fb):
"""A function's instructions. Capstone stops at the first word it cannot decode, so go on after it: Monster Hunter Rise main+0x6b34540 is a trap word (0xe7ffdefe)
in the middle of a function, and the br x18 jump table after it (main+0x6b346d0) was never seen"""
@@ -849,6 +878,7 @@ for fa, fb in funcs:
if not (base and width):
tgt = jump_offsets(insl, j) or jump_adr_offsets(insl, j)
if not tgt and tail_call(insl, j): tailcalls.add(ins.address); continue
if not tgt and (far := jump_bytes_far(insl, j)): jumps[ins.address] = far; continue
assert tgt, f'jump table {ins.address:x}'
tabjumps[ins.address] = tgt; continue
jumps[ins.address] = (base, width)
+1
View File
@@ -57,4 +57,5 @@ The work folder `$W` sits in the game's NSP folder, beside the NSP script's `<ti
- `*.sarc` / `*.pack` / `*.blarc` / `*.bfarc` are SARC archives; `*.byml` / `*.bgyml` are BYML (version 7 in Mario). A minimal reader for both is short enough to write inline when needed (see `shader_split.py` for SARC). Unity titles: `dm/gfx/tools/unityfs.py` reads UnityFS bundles, serialized files and objects by type tree (`bundle_files`, `serialized_objects`).
- Capcom's RE Engine (Monster Hunter Rise) keeps everything in PAK archives: `re_chunk_000.pak` plus `re_chunk_000.pak.patch_001.pak` … `patch_016` (19.6 GB in all). A PAK is `KPKA`, u8 version 4, u8 0, u16 features (0: a plain entry table), u32 count, u32 fingerprint, then one 0x30-byte entry per file: u32 murmur hashes of the lower- and upper-case UTF-16 path, u64 offset, u64 stored length, u64 length, u64 attributes (bits 0–3: 0 stored, 1 raw deflate, 2 zstd), u64 checksum. File names are only hashes. The game reads all of them as one view: a path in a later patch replaces the same hash in earlier PAKs (`kpka_jobs` in `shader_split.py` takes only the newest copy). The first bytes of each file name its format (`MESH`, `TEX\0`, `SDF\0` shader definitions, `BKHD` / `AKPK` Wwise, `USR\0`, `PFB\0`, …).
- Koei Tecmo's KTGL (Hyrule Warriors: Age of Calamity) keeps everything in RDB archives: `asset/<name>.rdb` is the index (`_DRK0000` header, then one `IDRK0000` record per file), and `asset/<name>.rdb.bin*` plus `asset/data/0x<KTID>.file` hold the data as a run of `IDRK0000` entries: u64 entry length, u64 stored length, u64 length, u32 0, u32 file KTID, u32 type KTID, u32 flags, a few parameter words, then the stored data at the end of the entry. Entries follow each other directly, some after padding to 64 KB. Flag 0x100000 = zlib in chunks: u32 stored length + a zlib stream each, up to 16 KB inflated per chunk, a zero length ends. The first bytes of the inflated data name the format (`GT1G` textures, `_M1G` models, `_S2G` shaders, `KTSR` / `TSRS` / `ASRS` sound). The DLC folders add `.rdb.bin3_0` files of the same layout.
- Unreal Engine 4 (Octopath Traveler 0, UE 4.26 / 4.27) keeps everything in pak files: `Octopath_Traveler0/Content/Paks/pakchunk0-Switch.pak` (1.5 GB, 159,757 files) next to loose CRIWARE `.awb` sound banks and one `.usm` movie. A version 11 pak has its footer in the last 0xdd bytes (encryption key GUID, encrypted-index flag, magic `0x5A6F12E1`, version, index offset and size, SHA-1, five compression method names) and an AES-256 (ECB) encrypted index: mount point, entry count, path hash seed, path hash index, full directory index (a second encrypted block: directories, then file names with offsets into the encoded entries) and the encoded entries (one bit-packed u32 per file: block size, block count, encrypted, compression method, then offset and sizes). The key is in `main`: the one 32-byte window of its read-only or data segment that decrypts the index start to the mount point `../../../` (Octopath Traveler 0: `main+0x57c7f70`, found among 116,287 candidates in under a second). Never print it. Most files are Oodle-compressed in 64 KB blocks; the two shader code archives are stored. `dm/gfx/tools/uepak.py` reads all of this (AES through macOS CommonCrypto, no Python package) and lists a pak (`uepak.py list <pak> <main NSO> <out tsv>`).
- The older KTGL of Fire Emblem: Three Houses uses a LINKDATA archive instead: `DATA0.bin` is an index of 32-byte entries `{u64 offset, u64 length, u64 stored length, u64 compressed}` into `DATA1.bin` (6.9 GB), and `patch1`–`patch4` hold loose overrides listed by `patchN/INFO0.bin` (0x120-byte entries ending in a `rom:/patchN/…` path). Its compressed entries and `*.gz` files are another chunked zlib: u32 chunk size (0x10000), u32 chunk count, u32 length, the stored length of each chunk, then the chunks at 0x80 alignment, each a u32 length + one zlib stream, or the plain bytes when the stored length equals the chunk's length (`kt_gz` in `shader_split.py`).
+5
View File
@@ -38,6 +38,7 @@ Each entry: symptom → root cause → fix (where the fix lives). Check these fi
- **`branch into window <addr>` assertion** (The Legend of Zelda: Tears of the Kingdom 1.4.3 `main`, 8 sites) → the compiler moved an atomic update's retry path out of line: the failed `stxr` branches to a block `ldxr; cmp; b.eq <the stxr>` that re-enters the window at its store. That block cannot jump into the window's patch copy (the `x18` stand-in would be loaded between its exclusive load and the store), so it gets a copy of its own: the retry block and the window's tail from the store on, under one prologue (`retry_block` / `site_retry_window`). A window also never spans a second exclusive load: before this rule, a retry block's `ldxr` opened a window that ran on into the next, unrelated `ldxr … stxr` loop (`main+0x5a4674`), and branches to that loop's head landed mid-window.
- **`jump table <addr>` assertion, entries relative to an `adr`** (Tears of the Kingdom `main+0x393bb0` and others) → `cmp wI, #N; b.hi; …; adr x18, #L; ldrsw xE, [xT, xI, lsl #2]; add x18, x18, xE; br x18`: target = L + entry, not table + entry (`jump_adr_offsets`, tried only where the table-relative form is not found). The bounds check can be on a register copied into the index (`mov w16, w15`), and the table base can be built far from the dispatch; then every `adrp` + `add` pair in the function that builds that register is tried, and a table is accepted only if exactly one makes every target an instruction of the function.
- **`jump table <addr>` assertion on a byte table indexed by x30** (Hyrule Warriors: Age of Calamity `main+0x138216c`: `adr x18, #L; ldrb w30, [x14, x12]; add x18, x18, x30, lsl #2; br x18`) → capstone names x30 `lr` (and x29 `fp`), so the index register looked like no register and the `ldrb` never matched. The byte / halfword table check maps those names back to 30 / 29.
- **`jump table <addr>` assertion on a byte table whose `add` is far from the `br`** (Octopath Traveler 0 `main+0x37f9bc8`: `adr x18,#L; ldrb w0,[x17,x15]; add x18,x18,x0,lsl #2`, then 17 register moves before `br x18`) → the byte / halfword search looked 16 instructions back. `jump_bytes_far` looks 32 back, requires that nothing writes x18 between the `add` and the `br`, and runs only when every other form has failed, so sites that rewrote before come out the same. One of the 12 `br x18` sites in this `main`; a census of all of them first (each shape within 16 / 32 / 48 instructions) saves a 12-minute rewrite per site.
- **Checking a rewriter change**: rewrite every extracted module of every title with the old and the new `patch_nso.py` and compare all outputs; only `build_src.sha256` (the rewriter's own hash) may differ for titles the change is not meant for.
- **SIGILL / trap on a system register read** → newer SDK reads a register macOS traps (`CTR_EL0` in SDK 16's `FlushDataCache`) → rewrite to a constant.
- **`B()` range assertion in `patch_nso.py` (`site_plain`), only for one module** (Metroid Dread `main`) → its bss is 1.8 GB, the engine's static heap, so a patch area appended after the image starts 1.9 GB from the code and no `B` (±128 MB) reaches it. When an appended patch area would start beyond 112 MB, the rewriter puts it 32 MB *before* the module base (negative `PATCH_OFF`). `patched.bin` then holds only text + ro + data (`FILE_IMAGE_LEN`) followed by the patch bytes, and the loader maps the bss and the key page as zero pages (`IMAGE_SIZE`). The Mac loader, the iOS embedding (`gen_embed.py`: no bss in the executable) and `QueryMemory` handle both layouts; Mario's and Engage's output is byte-identical to before.
@@ -51,6 +52,7 @@ Each entry: symptom → root cause → fix (where the fix lives). Check these fi
- **`StandardAllocator::Initialize` aborts in `nninitStartup`** → the NPDM declares a system resource (offset 0x14 of `META`), so `IsVirtualAddressMemoryEnabled` is true and the allocator reserves all free memory (about 3.2 GB for Engage) as one region in the alias region, after the ~1 GB node area → the profile's `mem_alias` / `mem_reserve` must hold it (Engage: 5.5 / 8 GB on the Mac).
- **C++ exception aborts in `nn::ro::detail::GetExceptionInfo`** → the SDK's unwinder finds a module from `svcQueryMemory`: it walks back over Code+RX regions to the module start, forward over Code then CodeData, and reads `MOD0` for `.eh_frame_hdr`. The host must report each loaded module's real layout and report other host memory as free (`hle_svc.c`). IL2CPP throws a C++ exception for every managed exception, so Unity games need this; Mario never threw.
- **First `AllocateAddressRegion` fails with 0x1803** → SDK 16's AddressRegionManager takes 0x3fe00000 at the start of the alias region as a node area → the alias region must be larger than 1 GB.
- **The game calls `exit(1)` in `nninitStartup`** (Octopath Traveler 0, Unreal Engine: `main+0x12e74`) → Unreal's `nninitStartup` needs virtual address memory and reserves one address region for the engine with `AllocateAddressRegion`, 62 GB by default (`-SwitchVirtualAddressSpaceSize=` in MB, or in bytes above 4095; the other options it reads are `-SwitchMaxMemorySize=`, `-NVNCommandBufferGranularity=`, `-NVNControlBufferSize=`, `-SwitchMainThreadStackSize=`, `-NVNTempMemoryPageSize=`, `-ResX=` / `-ResY=` and `-SwitchDisableVirtualAllocator`, which also exits). It reads them from the game arguments, which the host used to set only after `nninitStartup`; they are set before it now, as on hardware, and the profile key `game_args` supplies them on iOS. Its allocator, an FMallocBinned3, then reserves 37 GB of that region (74 pools of 512 MB, pool index = offset >> 29; the null read in the out-of-memory report at `main+0x1005824` is the symptom when the region is smaller). The pools are uneven (at character creation the largest holds 68 MB, the 74 together 732 MB; measured with `DM_ALIAS_WINDOWS=512`), so no smaller window fits a phone's ~5 GB. The constructor call is cut (`patch_nso.py`) and the host puts its own allocator in the object (`hle_uemalloc.c`: the FMalloc vtable on a host malloc zone, the trivial slots kept). With it the region holds only what the engine maps itself (709 MB at character creation) and is capped at 2 GB; SDK 19's AddressRegionManager node area is cut to 62 MB as Engage's (`sdk+0x1423fc` / `+0x142468`), so the guest block is 3.75 GB.
- **A null-based write minutes into play, at the pool size minus 0x20 (Engage: `addr=0x1fffe0` at `main+0x9a4664`)** → Unity's TLSF heap added a pool, and its low-level virtual allocator (vtable at `main+0x61ffeb0`; reserve `main+0x9aa0f4` → `nn::os::AllocateAddressRegion`) got no address region and returned 0. The alias region ran out of *address space*, not memory: the AddressRegionManager aligns each region by its size (64 KB under 2 MB, 2 MB under 4 MB, 4 MB under 32 MB, 32 MB under 1 GB, else 1 GB) and Unity asks for its block plus alignment slack, so a 2.25 MB block takes a 4 MB slot and a 16.25 MB block a 20 MB one. At the crash Engage held 4.17 of the 4.375 GB left after the node area (StandardAllocator 3.14 GB, Unity about 1.05 GB, the rest fragments). On hardware the alias region is 64 GB. → shrink the node area with a rewriter cut for that SDK build (`patch_nso.py`, Engage: the two `mov w8/w9, #0x3fe00000` in the constructor → 62 MB). To measure, probe the SDK's `VammManager::AllocateAddressRegion` at its success path (`ReleaseWriteLock` return: x21 = size, x22 = address) and `FreeAddressRegion` after its prologue (x1 = address), and sum the live regions by caller.
- **Sizing the reservation for a title that uses virtual address memory (Engage)** → such a title may never call `SetHeapSize`: everything comes from `MapPhysicalMemory` in the alias region. The `[mem]` log lines report the alias high-water mark (one line per 256 MB), the total actually mapped (page bitmap) and every heap growth. What fills the alias region first is address space, not memory: StandardAllocator reserves the whole free pool (pool − 64 MB, 1 GB aligned) but Engage commits only 0.58 GB of it, and Unity's own regions commit 0.97 GB. So Engage's profile reports a 2.5 GB pool (`app_pool`) and uses a 4.375 GB alias region in a 5 GB block (reserved address space peaks at 3.47 GB of 4.31 GB). iOS without the extended-virtual-addressing entitlement allows a single 6 GB reservation, but Metal needs address space too; 5 GB is the total proven on a phone. To measure commits, probe `VammManager::AllocateMemoryPages` / `FreeMemoryPages` (x1 address, x2 size) and split them by region.
- **`StandardAllocator::Initialize` aborts in `nninitStartup` on SDK 17, then the alias region fills in minutes** (Ace Attorney Investigations Collection, Unity 2022.3, SDK 17.5.3) → the same node area as Engage's SDK: `CentralHeap::Init(nullptr, pool − used)` got no 1 GB-aligned region for 3.14 GB after the 1 GB node area. The constructor is at `sdk+0x139458` on that build and takes the same cut. With Engage's 2.5 GB pool the game then ran, but probes on `VammManager::AllocateAddressRegion`'s success path (`sdk+0x1395a8`: x21 size, x22 address) and `FreeAddressRegion` (`sdk+0x139df0`) showed StandardAllocator's 2.44 GB reservation beside Unity's pools (1.06 GB live, from `main+0x40f1d4`) ending 4.37 GB into the 4.375 GB alias region after 140 s, while the `MapPhysicalMemory` log showed StandardAllocator committing 0.19 GB. The profile reports a 1 GB pool, a little less than the 1.02 GB the game maps at its peak (the host does not enforce the pool, and only `nninitStartup` reads it); ten minutes then peak at +2.85 GB of address space, against +3.91 GB with a 1.5 GB pool.
@@ -116,6 +118,9 @@ Each entry: symptom → root cause → fix (where the fix lives). Check these fi
- **DLC the player owns is not unlocked** → the host reports add-on content from `DM_AOC_DIR`, one numbered folder per DLC index (title ID − base − 0x1000) holding that DLC's RomFS. `build_ipa_from_nsp.py` extracts the DLC NSPs in the folder with `nx_extract --aoc`, `build_ipa.sh` copies them to `<App>/aoc/`, and the app sets `DM_AOC_DIR`. `aoc::CountAddOnContent` / `ListAddOnContent` report those indices (`hle_misc.c`), and `fs::MountAddOnContent` mounts the folder read-only (`hle_fs.c`). Engage's two DLC packages each hold one empty marker file (`e0`, `h0`), and the content is already in update 2.0.0. `DLCManager.RefreshHasList` marks a content owned when its mount succeeded (probe `main+0x29f3c88`). D2R keeps its fixed `{1}` without `DM_AOC_DIR`; `DM_AOC_OWNED=0` reports none.
- **Abort in `nn::sm::Initialize` under `nn::pctl::IsFreeCommunicationAvailable` (Engage, entering its first battle map)** → the SDK connects to the parental-control service through `sm`. The host answers it like the other free-communication calls: not available (`hle_misc.c`).
- **Abort inside `nn::sm::Initialize`** → an SDK path reached a system service the host doesn't replace → hook the public SDK entry point named in the backtrace (`HLE_FORCE` in the relevant `hle_*.c`). The log shows it as `SVC 0x1f` (ConnectToNamedPort, port name `sm:`) and `SVC 0x21` answered by placeholders, then `nn::diag` abort.
- **A weakly imported function the game calls although nothing defines it** (Octopath Traveler 0: the `thread_local` init function `_ZTHN13LowLevelTasks5FTask10ActiveTaskE`, an `UNIMPL` line, then an abort) → the loader bound every undefined import to a named stub, so the address test the compiler emits for a weak symbol saw a function. An undefined weak symbol that no module defines and the host does not implement is now 0, as Horizon's rtld binds it (`loader.c`, logged as `undefined weak … bound to 0`).
- **SIGSEGV in `android::Surface::getQueuedFrameNumber`** under `nn::vi::GetLatestFrameNumber` (Octopath Traveler 0: Unreal paces frames by the layer's frame number, `main+0x10140e8`) → the SDK reads the vi service's surface, which the host never made. `hle_vi.c` answers with the frames presented so far; `ListFrameHistory` returns no entries.
- **NVN calls new with Unreal** (Octopath Traveler 0): `nvnTexturePoolGetSize` / `nvnSamplerPoolGetSize` and their memory pool / offset getters (an abort in device setup when they returned 0), and `nvnCommandBufferFenceSync`, which Unreal's RHI records in its command buffers (`OP_FENCE_SYNC`: the replay fences the sync with the submission that reaches it).
- **A game update aborts at boot where the base game ran** → the update imports SDK functions the base game didn't. Diff the undefined dynamic symbols of the two `main` modules before the first run. Mario 1.2.1 added 28, among them `nn::nfp` (amiibo, initialized at boot), `pl::IsRunningOnOunce` (is this a Switch 2), `ec::ShowShopApplicationInformationForUpgrade`, `account::PopOpenUsers` and `aoc::CountAddOnContent`. Read the call sites before choosing answers: Mario only reads the count from `PopOpenUsers` and falls back to `TryOpenPreselectedUser` on 0, and only counts add-on content when running on a Switch 2. The host answers as a Switch 1 with no amiibo and no add-on content (`hle_sdk16.c`, `hle_account.c`, `hle_misc.c`).
- **SDK 20 calls that reach a service** (Breath of the Wild 1.9.0, SDK 20.5.22): `applet::GetAppletResourceUserId` (SDK 20's `OpenAudioRenderer` aborts on an ARUID of 0, which the SDK's own getter returns without am; the host reports 0x80), `pl::RequestSharedFontLoad`, the VR-mode calls (`pl::IsVrMode` / `BeginVrMode` / `EndVrMode`), `pctl::CheckStereoVisionPermission` (bool, permitted), `oe::TryPopLaunchParameter` (none), `time::StandardSteadyClock::GetCurrentTimePoint` (host monotonic seconds, fixed source ID; `GetSpanBetween` runs in `sdk`), the console six-axis sensor (`hid::*SevenSixAxisSensor*`: zero states) and more `nfp` calls (`AttachActivateEvent` / `AttachDeactivateEvent` crashed in the SDK at `sdk+0x3f2a78`; also two-argument `Mount`, `Unmount`, `GetCommonInfo`, `GetRegisterInfo`, `GetState`, `Finalize`). Two new imports need nothing: `nvnPrivate0` is a flag in `sdk`'s bss that enables an inlined command-buffer fast path when the NVN driver sets it (it stays 0), and `nn::ro::ProtectRelro` (called from `main`'s DSO init) runs natively on the host's no-op `SetMemoryPermission` / `SetMemoryAttribute`.
- **More SDK 20 calls that reach a service** (Tears of the Kingdom 1.4.3, the same `sdk` build as Breath of the Wild 1.9.0; each aborted in `sm::Initialize` or read a null applet proxy, one per boot): `oe::GetPreviousProgramIndex` (−1: launched from the home menu); the hardware Opus decoder's C interface (`GetHardwareOpusDecoderWorkBufferSize`, `InitializeHardwareOpusDecoder`, `DecodeOpusInterleavedWithHardware`, …), forwarded to the `sdk`'s software `OpusDecoder` C interface with a host-held decoder state per game object; cache storage (`OpenCacheStorageList` / `ReadCacheStorageList` report none, `CreateCacheStorage` / `MountCacheStorage` use a host folder beside the temporary storage); `fs::IsSaveDataExisting(ApplicationId, Uid)` for another title (it asks for Breath of the Wild's save: none) and `MountSaveDataReadOnly`; `time::ClockSnapshot::CreateWithStandardSystemClock` (0xD0 bytes, filled on the host from the steady clock, the user clock and the calendar; the network fields stay 0). The import list itself says where to look: diff the undefined symbols against the closest title the host already runs, drop what the host hooks, and the rest are the candidates. To find which `sdk` function aborted, map the backtrace offsets to the nearest exported symbol.
+1
View File
@@ -16,6 +16,7 @@ Do not stop at the obvious archive. In Mario the main `bfsha` held 4,888 of the
- **G2S in RDB archives** (Hyrule Warriors: Age of Calamity, Koei Tecmo's KTGL): each material's shaders are a G2S container (`_S2G3000`, type KTID `0x7bcd279f`; 23,108 entries in `asset/system.rdb.bin*`, 2.3 GB inflated) of GLSLC outputs, as in 13 Sentinels' `main`, inside zlib-chunked IDRK entries ([01](01-extract.md)); one `_DOK0000` container (type `0x20a6a0bb`, the engine's system resources) holds 151 more bare GLSLC outputs, 21 of them drawn in the first battle. `shader_split.py` maps the RDB files and inflates only entries that start with those two magics (`idrk_shaders`), with `shader_main_glslc on` for the 117 outputs in `main`. Many outputs carry 0x850-byte control sections and the compute ones have no code header, so `glslc_shaders` accepts controls from 0x760 bytes and takes the stage of headerless code from the thread group size at control +0x748. 40,726 shaders (37,043 pixel, 3,440 vertex, 241 compute, 2 geometry), Mario's texture handle layout (`0x20 + 8 × slot`); pairs come from `shader_separable on`. The older KTGL of Three Houses packs the same G2S containers differently (next item).
- **SPKG in LINKDATA** (Fire Emblem: Three Houses, the older KTGL): shader packages (`SPKG`: `patch*/nx/shader/*.spkg` and 132 entries of `DATA1.bin`) of G2S containers as above, one `_SPK4100` package without G2S (the post effects, `DATA1.bin` entry 549), and map and model archives carrying packages of their own. `kt_shaders` inflates every LINKDATA entry and loose `*.gz` / `*.spkg` file ([01](01-extract.md)), takes every GLSLC output that passes the triple check, and scans its SARCs for BNSH; `DATA1.bin` is split into 64 scan jobs by entry range (`linkdata_jobs`). Its GLSLC (output version 0x1d.2) writes the 0x848 / 0x850-byte controls and headerless compute code described above, which `glslc_shaders` used to drop: 24 shaders before, 1,355 after. A few outputs of version 0x1d.0 keep something else at control +0x20 (odd values such as 0x136b), so a control length must be a multiple of 4. Nine more shaders exist only at run time (Seen in Fire Emblem: Three Houses, below).
- **SDF in RE Engine PAKs** (Monster Hunter Rise): shader definitions (`SDF\0`, 2,685 files in the PAK view, [01](01-extract.md)): u16 version 1, u16 entry count, u64, then 0x108-byte entries from +0x10. Entry +0x10 holds six u64 offsets (vertex, hull, domain, geometry, pixel, compute; 0 = none), each to `{u32 control length, u32 code length}` followed by the control section (0x880 bytes) and the code area; compute code has no header. An entry's stages are one program, so its vertex / pixel stages are a pair. Empty definitions are 16 bytes of version 0. `kpka_jobs` maps the PAKs, takes the newest copy of each path, peeks at 4 decompressed bytes and inflates only SDF entries (`sdf_shaders`): 63,111 shaders (30,233 vertex, 21,475 pixel, 11,271 compute, 53 geometry, 33 hull, 46 domain) and 102,345 pairs, in 5 s with 12 jobs.
- **Unreal Engine shader code archives** (Octopath Traveler 0: `ShaderArchive-Global-SF_GLSL_SWITCH_MOBILE-SP_GLSL_SWITCH_MOBILE.ushaderbytecode`, 1,219 shaders, and `ShaderArchive-Octopath_Traveler0_Chunk0-…`, 9,377, both stored in the pak, [01](01-extract.md)): version 2 archives of shader map hashes, shader hashes, shader map entries, shader entries `{u64 offset, u32 size, u32 raw size, u8 frequency}` (17 bytes packed; frequency 0 vertex, 3 pixel, 5 compute), preload entries and shader indices, then the code. Every shader is Oodle-compressed (Mermaid, block header `8C 0A`); there is no Oodle decoder in this repository, so the game's own `OodleLZ_Decompress` (statically linked into `main`; Octopath Traveler 0: `main+0x4f77e10`, found from the function name its profiler hook passes) decodes them in the Mac host: profile key `shader_ue_decode`, `DM_OODLE_JOB` (`dm/host/hle_oodle.c`), the NSP script's `decode_ue_shaders`. A decoded shader is an `LSLG` record: `LSLG`, a stage tag (`SV`, `SP`, `SC`), a header with the source name (`/Engine/Private/…usf:MainPS`), then `u32` code length, the code area, `u32` control length (0x880), the control section and a 24- or 29-byte tail, all at unaligned offsets (`uepak.lslg`). The host keys came out exactly: 37 of 37 shaders a first run reported missing were in the two archives. 10,324 unique shaders (1,457 vertex, 6,346 pixel, 2,521 compute); vertex and pixel shaders combine per draw, so `shader_separable on`. Bindings: separate textures and samplers, Engage's layout (`0x168` / `0x568`); 61 vertex shaders read a texture buffer through a fully dynamic handle (`vp_tb_cb0_40000000`, not supported), and 16 compute shaders do atomics on a 3D `r32ui` image, which SPIRV-Cross's MSL 2.3 image-atomic emulation indexes with an `int3` (`array subscript is not an integer`; not built).
- **BAHS** (agl `sharcb`, engine utility shaders): code region at header +0x1c (size +0x18). Binary entries in the table area are 16 bytes `{code offset, 0, control size (0x880/0x88c; 0x848 in BAHS version 9, Super Mario 3D World), signed offset from this entry to the control section}`. Control sections are shared between programs, so pair by entry, never by searching for code near a control.
- **Shorter control sections (SDK 10, Super Mario 3D World + Bowser's Fury)**: the control section stores its own length at +0x20. Mario's, Engage's, Dread's and 13 Sentinels' are all at least 0x880 bytes; 3D World has 2,888 of 0x850 bytes (its ShaderData bfsha and six layouts) and 0x848 in BAHS version 9, each followed directly by other data. The key therefore hashes the first `min(0x880, control +0x20)` bytes (`ctl_len` in `shader_split.py`, `keys_hash` in `hle_nvn.c`); the other games' tables came out byte-identical.
- **Compute code without a header (SDK 10)**: 3D World's particle compute shaders start their instructions at code +0 (like D2R), not after a `0x12345679` header. `shader_split.py` accepts both; `dmsh_build.py` strips only a header that is there.
+2
View File
@@ -96,6 +96,8 @@ To see the order of events, put temporary observation gates (`OBS` in `patch_nso
4. GPU allocated climbing while the residency set, `shared textures` and the upload counts stay flat: retired objects kept alive by an autorelease pool that never drains. Any objc call on the guest submit thread outside `g_sub_pool` leaks what it autoreleases (Metroid Dread; Breath of the Wild, GitHub issue #9). Build the host with the suspect wrapped in `@autoreleasepool` and compare.
5. `serial objects` climbing by the same amount every frame with GPU memory flat: entries the serial table never deletes. A census of `g_by_serial` by class and size in `mtl_mem_stat` shows which (BotW: finalized textures replay resolved again; they now go to the small table counted as `finalized textures still referenced`).
6. Address space rather than memory (an iPhone has about 5 GB of it without the extended virtual addressing entitlement): `DM_ALIAS_WINDOWS=<MB>` logs every 30 s how much of each window of that size in the alias region is mapped (windows aligned to their size in absolute addresses). For an allocator that reserves fixed windows and commits each from its start, the windows are its pools: Unreal's FMallocBinned3 in Octopath Traveler 0 showed 74 pools of 512 MB holding 732 MB together, the largest 68 MB, so no smaller uniform window fitted ([03](03-bringup.md)). Unreal's own heap on the host malloc zone (`hle_uemalloc.c`) logs its peak as `[mem] Unreal heap … in use`.
Instruments' Metal Resource Events didn't record this process from the command line (`xctrace record --launch`, with `--env` for the variables); the in-host probes were faster.
## Save failures
+1 -1
View File
@@ -2,7 +2,7 @@
These pages are for an agent asked to analyse a Switch NSP and port it with the native-execution approach of this repository and [d2r-mobile](https://github.com/jhaizhou-ops/d2r-mobile): the game's own aarch64 code runs natively on Apple hardware, a rewriter patches only the instructions Apple's kernel cannot run, the host replaces Horizon system services, and NVN is reimplemented on Metal.
Everything here was learned on Diablo II: Resurrected (SDK ~12, NVN 53, the original target of d2r-mobile) and Super Mario Bros. Wonder (SDK 16.2.0, NVN 55, this repository; its update 1.2.1 uses SDK 16.2.12), then extended by Fire Emblem Engage (Unity, SDK 12.3.7), Metroid Dread (MercurySteam's engine, SDK 11.4.3) 13 Sentinels: Aegis Rim (Vanillaware's engine, SDK 11.4.3, the same `sdk` and movie module as Dread) Super Mario 3D World + Bowser's Fury (Nintendo EPD's sead / agl engine, SDK 10.4.0, NVN API 53) The Legend of Zelda: Breath of the Wild (EPD's sead / ksys engine, SDK 20.5.22, `DT_RELR` relocations, update 1.9.0), The Legend of Zelda: Link's Awakening (Grezzo's engine on NintendoWare, SDK 8.2.0, lower-left window origin) Hyrule Warriors: Age of Calamity (Koei Tecmo's KTGL, SDK 9.6.0, RDB archives, shaders as GLSLC outputs) Super Mario Galaxy (a Wii game on NERD's Hagi layer: recompiled PowerPC code, SDK 18.3.5, Vulkan with SPIR-V generated at run time, a 2.9 GB bss) and Ace Attorney Investigations Collection (Unity 2022.3 IL2CPP, SDK 17.5.3: the C# file API through the SDK's C library). Where titles differed, each is given; the next title will differ again, so treat every number below as "measure it, don't assume it".
Everything here was learned on Diablo II: Resurrected (SDK ~12, NVN 53, the original target of d2r-mobile) and Super Mario Bros. Wonder (SDK 16.2.0, NVN 55, this repository; its update 1.2.1 uses SDK 16.2.12), then extended by Fire Emblem Engage (Unity, SDK 12.3.7), Metroid Dread (MercurySteam's engine, SDK 11.4.3) 13 Sentinels: Aegis Rim (Vanillaware's engine, SDK 11.4.3, the same `sdk` and movie module as Dread) Super Mario 3D World + Bowser's Fury (Nintendo EPD's sead / agl engine, SDK 10.4.0, NVN API 53) The Legend of Zelda: Breath of the Wild (EPD's sead / ksys engine, SDK 20.5.22, `DT_RELR` relocations, update 1.9.0), The Legend of Zelda: Link's Awakening (Grezzo's engine on NintendoWare, SDK 8.2.0, lower-left window origin) Hyrule Warriors: Age of Calamity (Koei Tecmo's KTGL, SDK 9.6.0, RDB archives, shaders as GLSLC outputs) Super Mario Galaxy (a Wii game on NERD's Hagi layer: recompiled PowerPC code, SDK 18.3.5, Vulkan with SPIR-V generated at run time, a 2.9 GB bss), Ace Attorney Investigations Collection (Unity 2022.3 IL2CPP, SDK 17.5.3: the C# file API through the SDK's C library), and the bring-up of Octopath Traveler 0 (Unreal Engine 4, SDK 19.3.13: an AES-encrypted pak, Oodle-compressed shaders that the game's own code decodes on the Mac, its allocator replaced by the host's). Where titles differed, each is given; the next title will differ again, so treat every number below as "measure it, don't assume it".
| Page | Covers |
|---|---|
+9
View File
@@ -288,6 +288,15 @@ The game's own work and its frame rate stay as they are; the host avoids work of
- **Frame rate:** 60 fps. The game sets present interval 1; on the iPhone the frame time is 16.7 ms at the median.
- **Mac input:** `DM_KEYS=f2400:A*20/300` goes from the title screen (about frame 2,500) through a new game into the opening story scene (about frame 4,900).
**Octopath Traveler 0 (`01005270232F2000`)**
- **Engine:** Unreal Engine 4.26 / 4.27 on its mobile renderer, SDK 19.3.13, update 1.0.5 (v327680); the base game is an XCI. Everything is in one pak, `pakchunk0-Switch.pak` (AES-encrypted index; the key is found in `main` and never printed), with CRIWARE sound banks and a USM movie beside it.
- **Shaders:** 10,324, in the pak's two shader code archives, each one Oodle-compressed. The NSP script decodes them on the Mac with the game's own `OodleLZ_Decompress` (`shader_ue_decode`, `dm/gfx/tools/uepak.py`, `DM_OODLE_JOB`), about 10 s for all of them. 16 compute shaders with 3D image atomics do not build, and 61 vertex shaders read a texture buffer through a fully dynamic handle the host does not support (7 of them failed to load in the first 15 minutes).
- **Memory:** Unreal reserves one address region (62 GB on the Switch) and its allocator, FMallocBinned3, 37 GB of it. A cut skips the allocator's constructor and the host runs Unreal's heap on a malloc zone (`hle_uemalloc.c`); the region is capped at 2 GB (`game_args`) and the SDK's node area cut to 62 MB, so the guest block is 3.75 GB. 15 minutes into the story on the Mac: region mapped up to 1.26 GB of the alias region, Unreal heap 0.75 GB, 2.8 GB resident, 1.84 GB of GPU allocations.
- **Frame rate:** 30 fps, as on the Switch (the game asks for present interval 2).
- **Keyboard:** the character's name and favourite dish come from the system keyboard (`DM_KBD` on the Mac, the input box on the iPhone).
- **Mac input:** a fresh save reaches the story with `DM_KBD=Lyra DM_KEYS="20:Plus,24:A*27/4,150:Plus,154:A*62/3,340:Right*12/3,341.5:A*12/3,380:Plus*5/3,396:A*20/3,470:A,474:Up,476:A,480:A*140/3"` (initial settings, the prologue, character creation with three belongings, "Begin Lyra's journey?").
- **Status:** brought up on the Mac through character creation into the story; not yet run on an iPhone.
## Tools (`dm/tools`)
| Tool | What it does |
+50 -5
View File
@@ -489,12 +489,12 @@ def write_profile(pdir, app, f):
'# ---- Memory ----', *key('mem_reserve'), *key('mem_alias'), *key('mem_stack'), *key('app_pool'), '',
'# ---- NVN driver constant segments (Mario\'s values, the translator\'s built-in layout, unless the engine says otherwise) ----',
*key('nvn_tex_base', '0x20'), *key('nvn_tex_slots', '32'), f'nvn_samp_base{T}0x568', f'nvn_img_base{T}0x120', f'nvn_img_slots{T}8', f'nvn_ssbo_bases{T}vs=0x110 ps=0x510 cs=0x310', '',
'# ---- Shader split ----', f'shader_groups{T}{" ".join(groups) or "/=romfs"}', f'shader_note{T}{name} shaders (provisional profile)',
*key('shader_main_glslc'), *key('shader_separable'), *key('shader_glslc_capture')]
'# ---- Shader split ----', *key('shader_groups', " ".join(groups) or "/=romfs"), f'shader_note{T}{name} shaders (provisional profile)',
*key('shader_main_glslc'), *key('shader_separable'), *key('shader_glslc_capture'), *key('shader_ue_decode')]
(pdir / 'game.tsv').write_text('\n'.join(L) + '\n', encoding='utf-8')
for n, text in EMPTY_TABLES.items(): (pdir / n).write_text(text)
say(f'wrote a provisional game profile: {pdir / "game.tsv"}')
for k in ('graphics', 'modules', 'shader_vk_capture', 'shader_main_glslc', 'shader_glslc_capture', 'mem_reserve', 'op_mode'):
for k in ('graphics', 'modules', 'shader_vk_capture', 'shader_main_glslc', 'shader_glslc_capture', 'shader_ue_decode', 'mem_reserve', 'op_mode'):
if k in G and G[k][1]: say(f' {k} {G[k][0]}: {G[k][1]}')
@@ -744,6 +744,48 @@ def capture_glslc(repo, dirs, pkg_ver, patched, work, env, logs, g):
PROG.end(f'{len(got)} shaders in {hms(time.time() - t0)}')
UE_DECODE_SECS = 300 # the longest Oodle decode run: Octopath Traveler 0's 10,596 shaders take about 10 s, the boot up to nnMain included
def ue_stamp(d): # what an extraction's decoded shader archives were made from: its paks (paths and sizes)
return ''.join(f'{p.relative_to(d / "RomFS")}\t{p.stat().st_size}\n' for p in sorted((d / 'RomFS').rglob('*.pak')))
def decode_ue_shaders(repo, dirs, pkg_ver, patched, work, env, logs, g):
"""Profiles with shader_ue_decode on (Octopath Traveler 0): Unreal Engine keeps its shaders in the shader code archives of its paks, each shader Oodle-compressed, and only the game's own Oodle
decodes them. dm/gfx/tools/uepak.py takes the archives out of the paks (the pak index is AES-encrypted with a key from main) and writes one Oodle job; the game's code decodes it in the Mac host
(DM_OODLE_JOB, dm/host/hle_oodle.c: right before nnMain, then the host exits), and the decoded archives go to <version dir>/ue_shaders, where shader_split.py reads them.
The job runs the packed version's code (Oodle's format does not change between builds). A folder whose stamp matches its paks is reused"""
sys.path.insert(0, str(repo / 'dm/gfx/tools')); import uepak
todo = {v: d for v, d in dirs.items() if not ((d / 'ue_shaders/stamp').exists() and (d / 'ue_shaders/stamp').read_text() == ue_stamp(d))}
PROG.stage('UE shader decode')
if not todo:
say(f'reusing the decoded shader archives: {", ".join(str(d / "ue_shaders") for d in dirs.values())}')
PROG.end('reused'); return
PROG.set(step='building the Mac host')
host = build_mac_host(repo, work / 'tools/dmhost', env, logs, g)
t0, total = time.time(), 0
for v, d in todo.items():
jd, rd, log = work / f'tmp/ue_job_{v}', work / 'tmp/ue_run', logs / f'ue_decode_{v}.log'
rmtree(jd); rmtree(rd)
for s in ('save', 'temp'): (rd / s).mkdir(parents=True)
PROG.set(step=f'{v}: shader archives out of the paks')
listing, n = uepak.job(str(d / 'RomFS'), str(d / 'ExeFS/main'), str(jd))
if not listing: raise BuildError(f'the profile has shader_ue_decode on, but no pak under {d / "RomFS"} holds a shader code archive')
say(f'{v}: ' + '; '.join(f'{x["name"]} {x["shaders"]:,} shaders' for x in listing) + f'; decoding {n:,} with the game\'s Oodle in the Mac host')
PROG.set(step=f'{v}: decoding {n:,} shaders in the Mac host')
henv = dict(env, DM_ROMFS=str(dirs[pkg_ver] / 'RomFS'), DM_SAVE=str(rd / 'save'), DM_TEMP=str(rd / 'temp'), DM_MSL_DIR=str(rd / 'msl'), DM_OODLE_JOB=str(jd / 'oodle.job'))
rc = boot_mac(host, patched, henv, rd, log, UE_DECODE_SECS)
if rc != 0 or not (jd / 'oodle.job.out').exists():
PROG.end(ok=False)
PROG.say(f'---- last lines of {log} ----\n' + ''.join(collections.deque(open(log, errors='replace'), maxlen=20)).rstrip('\n'))
raise BuildError(f'the Oodle decode run stopped with exit {rc} (host output: {log})')
out = d / 'ue_shaders.partial'; rmtree(out)
uepak.finish(str(jd), str(out)); (out / 'stamp').write_text(ue_stamp(d))
swap_in(out, d / 'ue_shaders'); rmtree(jd); rmtree(rd); total += n
PROG.end(f'{total:,} shaders in {hms(time.time() - t0)}')
VK_CAPTURE_SECS = 120 # the longest Vulkan capture run: Super Mario Galaxy creates its 3,089 modules within 6 s of its first one (about 15 s after boot on an M4 Pro)
@@ -878,7 +920,8 @@ def split_shaders(py, repo, roms, mains, pkg, env, logs, g, provisional=False):
PROG.stage('shader split', step='checking the inputs') # then a spinner: shader_split.py writes the package and the tables only at the end
kf = pkg.with_name(pkg.name + '.key')
key = None if provisional or ALLOW_DAMAGED else fingerprint(('files', tool_files(repo)), ('files', profile_files(g)),
*[x for r, m in zip(roms, mains) for x in (('files', [m]), ('tree', r), ('files', sorted((r.parent / 'glslc_capture').glob('*.glslc'))))])
*[x for r, m in zip(roms, mains) for x in (('files', [m]), ('tree', r), ('files', sorted((r.parent / 'glslc_capture').glob('*.glslc'))),
('files', sorted((r.parent / 'ue_shaders').glob('*.ushaderbytecode'))))])
if key and pkg.is_dir() and (note := reused(kf, key)) is not None:
PROG.end(f'{note}, reused (inputs unchanged)'); return key
PROG.set(step='splitting')
@@ -1151,8 +1194,9 @@ def main(args=None):
if free_gb(work) < gb and not all((ex / v).exists() for v in versions): raise BuildError(f'less than {gb:.0f} GB free on the volume of {work}')
glslc = gfx == 'nvn' and g.get('shader_glslc_capture', 'off') == 'on' # shaders the game compiles at run time, captured in the Mac host (capture_glslc)
vkcap = gfx == 'vulkan' and g.get('shader_vk_capture', 'off') == 'on' # SPIR-V shaders the game generates at run time, captured in the Mac host (capture_vk)
uedec = gfx == 'nvn' and g.get('shader_ue_decode', 'off') == 'on' # Unreal Engine shaders, Oodle-compressed in the paks' shader code archives, decoded in the Mac host (decode_ue_shaders)
heapck = provisional and HEAP_CHECK and not args.no_heap_check and not heap_checked(g) # the heap check, when the game needs one (once per provisional profile: it stamps the profile)
PROG.plan(1 + len(versions) + bool(fam['dlc']) + glslc + vkcap + (5 if gfx == 'nvn' else 4)) # nx_extract, each version's extraction, the DLC, rewrite, the GLSLC / Vulkan capture, shader split + iOS shaders (or the Vulkan shaders), app build, IPA pack
PROG.plan(1 + len(versions) + bool(fam['dlc']) + glslc + vkcap + uedec + (5 if gfx == 'nvn' else 4)) # nx_extract, each version's extraction, the DLC, rewrite, the GLSLC / Vulkan capture or the UE shader decode, shader split + iOS shaders (or the Vulkan shaders), app build, IPA pack
PROG.stage('build nx_extract')
nx = build_nx_extract(repo, work, logs, env)
PROG.end()
@@ -1180,6 +1224,7 @@ def main(args=None):
if heapck and pg.heap_check_needed(game / 'ExeFS'):
PROG.add(1); g = heap_check(repo, game, patched, work, env, logs, g)
if glslc: capture_glslc(repo, {v: dirs[v] for v in need}, pkg_ver, patched, work, env, logs, g)
if uedec: decode_ue_shaders(repo, {v: dirs[v] for v in need}, pkg_ver, patched, work, env, logs, g)
sh = work / 'shader'
if gfx == 'vulkan': # the packed version's merged RomFS holds the SPIR-V files the game loads; a capture holds those it generates
cap = capture_vk(repo, game, patched, work, env, logs, g) if vkcap else None