Digimon Story Cyber Sleuth: Complete Edition plays on the Mac: profile 010014E00DB56000 (update 1.0.3, SDK 6.4.0), an OpenGL layer on Metal (dm/gfx/gl), its GLSL translated offline (gl_shaders.py), MVGL archives (mvgl.py)
This commit is contained in:
seedds committed
1 parent
1eaaab88ca
commit
298fa96f0b
27 files changed
+2022
-30
No files matched your search
@@ -13,7 +13,7 @@ This project is built on other people's work:
|
||||
- **[Atmosphère](https://github.com/Atmosphere-NX/Atmosphere)**: its kernel, mesosphère, is the reference for Horizon's synchronization semantics (address arbiter, condition variables).
|
||||
- **yuzu**: Maxwell instruction encodings for the shader reference tool, and a cross-check for kernel behaviour.
|
||||
- **[LibHac](https://github.com/Thealexbarney/LibHac)**: reads NSPs and layers updates over the base game in `dm/tools/nx_extract`. **hactool** did the first extractions.
|
||||
- **[SPIRV-Cross](https://github.com/KhronosGroup/SPIRV-Cross)**: SPIR-V to Metal Shading Language.
|
||||
- **[SPIRV-Cross](https://github.com/KhronosGroup/SPIRV-Cross)**: SPIR-V to Metal Shading Language. **[glslang](https://github.com/KhronosGroup/glslang)** compiles the GLSL of OpenGL games to SPIR-V for it.
|
||||
- **deko3d** (zlib): its texture layout size algorithm is ported in `dm/gfx/nvn/nvn_tex.c`.
|
||||
- **libvpx** (VP9 movies), **stb_vorbis** (movie sound), the **Vulkan headers** and **NVIDIA FLIP** ship in `dm/third_party` with their licenses ([list](dm/third_party/README.md)).
|
||||
- **Capstone**, **lz4**, **fontTools** and **XcodeGen** are used by the build.
|
||||
@@ -53,7 +53,7 @@ Needs iOS / iPadOS 26 or later on an iPhone 15 Pro or newer (A17 Pro chip or lat
|
||||
|
||||
```
|
||||
git clone https://github.com/seedds/nxnative.git && cd nxnative
|
||||
brew install xcodegen zstd spirv-cross
|
||||
brew install xcodegen zstd spirv-cross glslang
|
||||
brew install --cask dotnet-sdk
|
||||
python3 -m pip install lz4 capstone fonttools "backports.zstd; python_version < '3.14'"
|
||||
```
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
# generated
|
||||
|
@@ -0,0 +1,2 @@
|
||||
// Generated file (dm/gfx/tools/dmsh_chain.py --header, game 010014E00DB56000): the B shader library translation chain version this source tree expects (default options); the host loads only <library dir>/b-<V>/
|
||||
#define DMSH_CHAIN "9c194b63501c"
|
||||
@@ -0,0 +1,45 @@
|
||||
# Game profile of Digimon Story Cyber Sleuth: Complete Edition (Switch, update 1.0.3 = v196608). Plays on the iPhone; on the Mac from the logos through the EDEN opening and character creation into EDEN, under control
|
||||
# Format: see dm/games/README.md
|
||||
# Media.Vision's MVGL engine (the PC version's), SDK 6.4.0, on NVIDIA's desktop OpenGL 4.5 core driver through EGL (subsdk0, not loaded): main gets its GL functions from a glad loader
|
||||
# (eglGetProcAddress) and calls 79 of them, which the host's GL layer implements on Metal (dm/gfx/gl/hle_gl.m). Its NVN device serves only a helper beside the movie
|
||||
# player (a memory pool and four textures). Every game file is in nine MVGL archives (DSDB*.nx64.mvgl: MDB1, doboz-compressed entries, dm/gfx/tools/mvgl.py; four of them are CRI AFS2
|
||||
# sound banks under the same extension) but the movies, loose CRI USM files (H.264 through the SDK's movie library, subsdk1, into NV12 buffers the game uploads into GL textures)
|
||||
# The shaders are GLSL ES 3.20 sources the game compiles at run time: 1,682 .shad files in the archives (615 distinct) and 37 built-in strings in main; dm/gfx/tools/gl_shaders.py
|
||||
# translates them offline (glslang → SPIR-V → spirv-cross), and the host finds each by the hash of its text
|
||||
|
||||
# ---- Identity ----
|
||||
title_id 010014E00DB56000
|
||||
program_id 010014E00DB56000
|
||||
|
||||
# ---- App ----
|
||||
# Game name: only a label for profile listings and the Mac window title; the iOS app name is the game name in the base NSP, and the bundle ID comes from the base game title ID (build_ipa.sh / NSP script)
|
||||
name Digimon Story Cyber Sleuth: Complete Edition
|
||||
ipa DigimonStoryCyberSleuthCompleteEdition.ipa
|
||||
app_category public.app-category.role-playing-games
|
||||
|
||||
# ---- Versions ----
|
||||
# only the merged 1.0.3 copy is extracted
|
||||
build v196608 1.0.3 main=bedae701dd8565cdc494ca8dc3c840e8 subsdk1=31ed558beb51f9e273a569dab4f7db82 sdk=e29be8ad6778527aa248310d0c71a8a5
|
||||
package v196608
|
||||
shader_versions v196608
|
||||
|
||||
# ---- Files ----
|
||||
# load order as on Horizon: main, subsdk*, sdk (subsdk1 = NintendoSDK_movie). subsdk0 is NVIDIA's OpenGL driver, which programs the GPU through NvRm: never loaded, the GL layer answers main's EGL imports
|
||||
modules main subsdk1 sdk
|
||||
romfs_marker DSDB.nx64.mvgl
|
||||
zip_store .mvgl:.otf:.ttf:.usm:.metallib
|
||||
|
||||
# ---- Memory ----
|
||||
# The NPDM declares no system resource, so the game has no virtual address memory: nninitStartup gives its malloc a heap of the pool − used memory (3.13 GB of the 3.25 GB heap region),
|
||||
# which it maps only as it touches it
|
||||
mem_reserve 0x100000000
|
||||
mem_alias 0x10000000
|
||||
mem_stack 0x10000000
|
||||
app_pool 0xCD500000
|
||||
sys_resource 0x0
|
||||
# the rewritten modules are larger than the default 64 MB of used memory (Age of Calamity: 66.6 MB → 72 MB)
|
||||
mem_used 0x4a00000
|
||||
|
||||
# ---- Shader split ----
|
||||
# no NVN shaders: the NVN tables stay empty (gl_shaders.py writes the GL library and its table, not the profile)
|
||||
shader_note Digimon Story Cyber Sleuth: Complete Edition has no NVN shaders (its GLSL shaders go through the GL layer)
|
||||
|
@@ -0,0 +1 @@
|
||||
# Prewarm seed list (shipped in the app, next to shader_map.tsv). Rows: S<TAB>shader key<TAB>stage / R<TAB>render pipeline key (pso_skey hex) / C<TAB>compute shader key.
|
||||
|
@@ -0,0 +1 @@
|
||||
# Answer table for translation queries (generated by dm/gfx/tools/shader_split.py, do not edit): key <TAB> fact JSON. This game has no PC version to pair with: only the thread group / shared memory of compute shaders (taken from the control section); other queries get the translator defaults
|
||||
|
@@ -0,0 +1,2 @@
|
||||
# kind: evidence Digimon Story Cyber Sleuth: Complete Edition has no NVN shaders: the table stays empty (its GLSL shaders go through the GL layer)
|
||||
package index stage dm_ctl_sha16 source
|
||||
|
@@ -0,0 +1,2 @@
|
||||
# kind: evidence control-section key conflict rulings: empty
|
||||
dm_ctl_sha16 chosen_pc class candidates records
|
||||
|
@@ -0,0 +1 @@
|
||||
# kind: evidence vertex / pixel shader pairs within BNSH variations (generated by dm/gfx/tools/shader_split.py, do not edit): vertex key <TAB> pixel key
|
||||
|
+15
-12
@@ -70,7 +70,7 @@ def npdm(path): # (system resource size, program ID) from main.npdm; a non-zer
|
||||
def facts_of(exefs, romfs):
|
||||
"""What the profile rules (and the survey report) need from a game's code modules and RomFS"""
|
||||
exefs = Path(exefs)
|
||||
F = {'mods': {}, 'markers': {}, 'sdkver': set(), 'comps': set(), 'unity': set(), 'glslc': 0, 'glsl_src': 0, 'sysres': 0, 'program': None}
|
||||
F = {'mods': {}, 'markers': {}, 'sdkver': set(), 'comps': set(), 'unity': set(), 'glslc': 0, 'glsl_src': 0, 'gl_names': 0, 'sysres': 0, 'program': None}
|
||||
for p in sorted(exefs.iterdir()):
|
||||
with open(p, 'rb') as f:
|
||||
if f.read(4) != b'NSO0': continue
|
||||
@@ -86,7 +86,7 @@ def facts_of(exefs, romfs):
|
||||
if pat in img: F['markers'].setdefault(name, []).append(p.name)
|
||||
if p.name == 'main':
|
||||
F['unity'] = {v.decode() for v in re.findall(rb'20[12][0-9]\.[0-9]+\.[0-9]+f[0-9]', img)}
|
||||
F['glslc'], F['glsl_src'] = img.count(GLSLC_MAGIC), img.count(b'#version ')
|
||||
F['glslc'], F['glsl_src'], F['gl_names'] = img.count(GLSLC_MAGIC), img.count(b'#version '), img.count(b'glShaderSource')
|
||||
F['main_markers'] = {name for pat, name in MARKERS if pat in img}
|
||||
m['kind'] = {'main': 'the game', 'sdk': 'nnSdk', 'rtld': 'rtld'}.get(p.name) or kind(m)
|
||||
F['sysres'], F['program'] = npdm(exefs / 'main.npdm')
|
||||
@@ -167,28 +167,31 @@ GPU_DRIVERS = {'NintendoSDK_OpenGL', 'NintendoSDK_Vulkan'}
|
||||
|
||||
|
||||
# ---- Shader sources: decided from the game files at every build, not kept in the profile (scripts/build_ipa_from_nsp.py; shader_split.py reads what those steps write) ----
|
||||
def shader_sources(imports, glsl_src, glslc_mods, spirv, unreal_paks, trinity_packs=0, legacy_paks=0):
|
||||
"""{name: (value, reason)}: graphics (vulkan when main imports Vulkan functions and not nvnBootstrapLoader, else nvn), vk_capture (a Vulkan game without SPIR-V files generates its shaders at
|
||||
def shader_sources(imports, glsl_src, glslc_mods, spirv, unreal_paks, trinity_packs=0, legacy_paks=0, gl_names=0):
|
||||
"""{name: (value, reason)}: graphics (vulkan when main imports Vulkan functions and not nvnBootstrapLoader; gl when it imports eglCreateWindowSurface and names glShaderSource, so it compiles
|
||||
GLSL through OpenGL (gl_names: how often main has that name; Absolum imports the same EGL functions for SDL3 but renders with NVN and names no GL function); else nvn), vk_capture (a Vulkan game without SPIR-V files generates its shaders at
|
||||
run time), glslc_capture (main holds GLSL sources and calls glslcCompile in a module the game loads: it compiles some shaders at run time), ue_decode (Unreal paks keep the shaders Oodle-compressed
|
||||
in their shader code archives, which only the game's own Oodle decodes), oodle_packs (Trinity packs, Oodle-compressed file by file, which the shader split decodes with the game's own decoder in
|
||||
the Mac host's tool mode), ue_legacy (legacy Unreal paks, version 4: the shaders are inline in the cooked packages, which the shader split reads through the game's own decoder the same way).
|
||||
unreal_paks counts the version 11 paks, legacy_paks the older ones. Checked against every profile: Hades and Super Mario Galaxy Vulkan, Galaxy alone with no SPIR-V file, Three Houses alone compiling
|
||||
shaders (13 Sentinels holds 106 GLSL sources but does not import glslcCompile; Captain Toad: Treasure Tracker imports it with agl's GLSL sources, but no module defines it), Octopath Traveler 0
|
||||
alone with version 11 Unreal paks, DRAGON QUEST XI S alone with legacy ones, Pokémon Legends: Z-A alone with Trinity packs"""
|
||||
alone with version 11 Unreal paks, DRAGON QUEST XI S alone with legacy ones, Pokémon Legends: Z-A alone with Trinity packs, Digimon Story Cyber Sleuth alone with OpenGL (of the 22 games extracted)"""
|
||||
vk = sum(1 for s in imports if re.match(r'vk[A-Z]', s)); vulkan = bool(vk) and 'nvnBootstrapLoader' not in imports
|
||||
return {'graphics': ('vulkan', f'main imports {vk} Vulkan functions and not nvnBootstrapLoader: the host\'s Vulkan layer (dm/gfx/vk/hle_vk.m)') if vulkan else ('nvn', 'main imports nvnBootstrapLoader'),
|
||||
gl = not vulkan and 'eglCreateWindowSurface' in imports and bool(gl_names)
|
||||
return {'graphics': ('vulkan', f'main imports {vk} Vulkan functions and not nvnBootstrapLoader: the host\'s Vulkan layer (dm/gfx/vk/hle_vk.m)') if vulkan else
|
||||
('gl', 'main imports EGL and names glShaderSource: it renders with OpenGL and compiles GLSL, the host\'s GL layer (dm/gfx/gl/hle_gl.m)') if gl else ('nvn', 'main imports nvnBootstrapLoader'),
|
||||
'vk_capture': (vulkan and not spirv, 'no SPIR-V file in the RomFS: the game generates its shaders at run time (Super Mario Galaxy), captured by booting it on the Mac'),
|
||||
'glslc_capture': (not vulkan and bool(glsl_src) and 'glslcCompile' in imports and bool(glslc_mods),
|
||||
'glslc_capture': (not vulkan and not gl and bool(glsl_src) and 'glslcCompile' in imports and bool(glslc_mods),
|
||||
f'main holds {glsl_src} GLSL sources and calls glslcCompile in {", ".join(glslc_mods) or "no module it loads"}: shaders compiled at run time (Fire Emblem: Three Houses), captured by booting it on the Mac'),
|
||||
'ue_decode': (not vulkan and bool(unreal_paks), f'{unreal_paks} Unreal pak(s): the shaders are Oodle-compressed in their shader code archives (Octopath Traveler 0), decoded by booting the game on the Mac'),
|
||||
'oodle_packs': (not vulkan and bool(trinity_packs), f'{trinity_packs} Trinity pack file(s) (ONEPACK, arc/data.trpfs: Pokémon Legends: Z-A), Oodle-compressed: the shader split decodes them with the game\'s own decoder in the Mac host (dm/host/hle_oodle.c DM_OODLE_SERVE)'),
|
||||
'ue_legacy': (not vulkan and bool(legacy_paks), f'{legacy_paks} legacy Unreal pak(s) (version 4, UE 4.18: DRAGON QUEST XI S): the shaders are inline in the cooked packages, compressed in the pak; the shader split decodes them with the game\'s own decoder in the Mac host (dm/host/hle_oodle.c DM_OODLE_SERVE)')}
|
||||
'ue_decode': (not vulkan and not gl and bool(unreal_paks), f'{unreal_paks} Unreal pak(s): the shaders are Oodle-compressed in their shader code archives (Octopath Traveler 0), decoded by booting the game on the Mac'),
|
||||
'oodle_packs': (not vulkan and not gl and bool(trinity_packs), f'{trinity_packs} Trinity pack file(s) (ONEPACK, arc/data.trpfs: Pokémon Legends: Z-A), Oodle-compressed: the shader split decodes them with the game\'s own decoder in the Mac host (dm/host/hle_oodle.c DM_OODLE_SERVE)'),
|
||||
'ue_legacy': (not vulkan and not gl and bool(legacy_paks), f'{legacy_paks} legacy Unreal pak(s) (version 4, UE 4.18: DRAGON QUEST XI S): the shaders are inline in the cooked packages, compressed in the pak; the shader split decodes them with the game\'s own decoder in the Mac host (dm/host/hle_oodle.c DM_OODLE_SERVE)')}
|
||||
|
||||
|
||||
def build_sources(exefs, romfs, modules):
|
||||
"""shader_sources from an extraction (its ExeFS and RomFS folders) and the modules the profile loads"""
|
||||
exefs = Path(exefs); main = nso(exefs / 'main')[0]
|
||||
imports = {s for s, defined, _ in nso_syms.dynsyms(main) if not defined}; glsl_src = main.count(b'#version ')
|
||||
imports = {s for s, defined, _ in nso_syms.dynsyms(main) if not defined}; glsl_src = main.count(b'#version '); gl_names = main.count(b'glShaderSource')
|
||||
glslc_mods = [m for m in modules if m != 'main' and glsl_src and 'glslcCompile' in imports and (exefs / m).exists()
|
||||
and any(s == 'glslcCompile' and defined for s, defined, _ in nso_syms.dynsyms(nso(exefs / m)[0]))]
|
||||
spirv = 0
|
||||
@@ -200,7 +203,7 @@ def build_sources(exefs, romfs, modules):
|
||||
for n in names:
|
||||
if n.endswith('.trpfs'):
|
||||
with open(os.path.join(d, n), 'rb') as f: trinity += f.read(8) == b'ONEPACK\0'
|
||||
return shader_sources(imports, glsl_src, glslc_mods, spirv, len(uepak.unreal_paks(str(romfs), legacy=False)), trinity, len(uepak.unreal_paks(str(romfs), legacy=True)))
|
||||
return shader_sources(imports, glsl_src, glslc_mods, spirv, len(uepak.unreal_paks(str(romfs), legacy=False)), trinity, len(uepak.unreal_paks(str(romfs), legacy=True)), gl_names)
|
||||
|
||||
|
||||
def va_heap(sysres, imports): # virtual address memory (an NPDM system resource) and a heap: the alias / heap split is a starting point the heap check corrects (Wonder, Tears of the Kingdom; Engage's pool is its own measurement)
|
||||
|
||||
@@ -78,6 +78,8 @@ Binding conventions followed during translation (the NVN driver's layout, the sa
|
||||
| `nvn/include/nvn.h` | NVN interface declarations (third party; for the license see the file header) |
|
||||
| `nvn/names.txt` | The NVN API name list. The per-game tables described above and `dmsh_chain.h` live in the game profile `dm/games/<id>/` (named after the base game's title ID; tools pick it with `DM_GAME`, make with `GAME`; there is no default game) |
|
||||
| `tex/` | Texture format mapping, deswizzling, BC decompression, see [tex/README.md](tex/README.md) |
|
||||
| `vk/hle_vk.m` | Vulkan on Metal, for games that render with Vulkan (Hades, Super Mario Galaxy); their SPIR-V shaders are translated by `tools/vk_shaders.py` |
|
||||
| `gl/hle_gl.m` | OpenGL through EGL on Metal, for games that render with NVIDIA's GL driver (Digimon Story Cyber Sleuth: Complete Edition); their GLSL sources are translated by `tools/gl_shaders.py` (glslang, then SPIRV-Cross) |
|
||||
| `metal/` | A standalone minimal Metal offscreen example, not compiled into the host, see [metal/README.md](metal/README.md) |
|
||||
| `tools/` | Offline tools, see below |
|
||||
| `tests/` | Unit tests |
|
||||
|
||||
+1512
File diff suppressed because it is too large.
Load diff
+7
-2
@@ -14,9 +14,14 @@ HLE_FORCE("_ZN2nn2vi11CreateLayerEPPNS0_5LayerEPNS0_7DisplayE", "create layer; t
|
||||
(void)disp; *out = &g_layer; return RES_OK;
|
||||
}
|
||||
// The overload with the layer's size (SDL3's Switch video driver, Absolum: 1920x1080 docked); the SDK's version calls through the vi service object the host never made (SIGSEGV at +0xe4).
|
||||
// The window's textures decide what is presented, so the size is only logged
|
||||
// An NVN or Vulkan window's textures decide what is presented; an EGL window surface takes the layer's size (dm/gfx/gl/hle_gl.m: Digimon Story Cyber Sleuth sizes its layer
|
||||
// itself and draws into the surface at that size)
|
||||
static _Atomic uint32_t g_layer_wh; // w | h << 16 of the last sized CreateLayer, 0 = none
|
||||
void dm_vi_layer_size(int *w, int *h) { uint32_t v = g_layer_wh; *w = (int)(v & 0xffff); *h = (int)(v >> 16); }
|
||||
HLE_FORCE("_ZN2nn2vi11CreateLayerEPPNS0_5LayerEPNS0_7DisplayEii", "create layer of a given size; the host returns a static object", Result, h_vi_CreateLayerWH, void **out, void *disp, int w, int h) {
|
||||
(void)disp; LOG("vi::CreateLayer %dx%d", w, h); *out = &g_layer; return RES_OK;
|
||||
(void)disp; LOG("vi::CreateLayer %dx%d", w, h); *out = &g_layer;
|
||||
if (w > 0 && h > 0 && w < 0x10000 && h < 0x10000) g_layer_wh = (uint32_t)w | (uint32_t)h << 16;
|
||||
return RES_OK;
|
||||
}
|
||||
HLE_FORCE("_ZN2nn2vi15GetNativeWindowEPPvPNS0_5LayerE", "native window handle, passed to the NVN window builder; the host returns a static object", Result, h_vi_GetNative, void **out, void *layer) {
|
||||
(void)layer; *out = &g_native; return RES_OK;
|
||||
|
||||
@@ -1093,13 +1093,14 @@ static void gfx_leave(void) { dm_host_work--; pthread_mutex_lock(&g_susp_mu); g_
|
||||
void dm_gfx_enter(void) { gfx_enter(); } // the same gate for the Vulkan layer's submits and presents (dm/gfx/vk)
|
||||
void dm_gfx_leave(void) { gfx_leave(); }
|
||||
__attribute__((weak)) void dm_vk_drain(void) {} // dm/gfx/vk/hle_vk.m: wait for the Vulkan layer's queue (the graphics unit tests do not link it)
|
||||
__attribute__((weak)) void dm_gl_drain(void) {} // dm/gfx/gl/hle_gl.m: the same for the GL layer
|
||||
void dm_gfx_suspend(int on) {
|
||||
pthread_mutex_lock(&g_susp_mu);
|
||||
g_susp = on != 0;
|
||||
if (on) while (g_inflight_api) pthread_cond_wait(&g_susp_cv, &g_susp_mu);
|
||||
else pthread_cond_broadcast(&g_susp_cv);
|
||||
pthread_mutex_unlock(&g_susp_mu);
|
||||
if (on) { dm_vk_drain(); mtl_drain(); LOG("[nvn] graphics suspended: in-flight GPU work drained"); } else LOG("[nvn] graphics resumed"); // the Vulkan queue first: presents on the NVN queue wait for its frames
|
||||
if (on) { dm_vk_drain(); dm_gl_drain(); mtl_drain(); LOG("[nvn] graphics suspended: in-flight GPU work drained"); } else LOG("[nvn] graphics resumed"); // the Vulkan and GL queues first: presents on the NVN queue wait for their frames
|
||||
}
|
||||
// One submission at a time: the Metal replay keeps one command buffer and one upload batch for the submission in progress (nvn_mtl.m g_cb). Fire Emblem: Three Houses initializes two queues and
|
||||
// submits on both from two threads; the second submission arrived while the first was replaying and the host stopped with "submit re-entered" at the ninth frame. NVN orders work on different queues only
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
#!/usr/bin/env python3
|
||||
# OpenGL games' GLSL shaders → one Metal library + a table for the host's GL layer (dm/gfx/gl/hle_gl.m). Digimon Story Cyber Sleuth: Complete Edition renders with
|
||||
# NVIDIA's desktop GL through EGL and compiles GLSL ES 3.20 source at run time (glShaderSource / glCompileShader); the host finds each shader again by the hash of its source.
|
||||
# Sources, all searched in every input path:
|
||||
# · MVGL archives (*.mvgl, dm/gfx/tools/mvgl.py): every .shad file, named <material>_vp / <material>_fp (1,682 files, 615 distinct sources in Digimon);
|
||||
# · an NSO (a file that starts with NSO0, the game's main): every NUL-terminated string that starts with '#version' (its built-in post effects, movie and shadow shaders);
|
||||
# · capture folders (the Mac host's DM_GL_CAPTURE): <key>.vert / <key>.frag, what the game really compiled, for sources it puts together at run time.
|
||||
# Per shader:
|
||||
# · key = sha256(the source text up to its first NUL)[:16] (hex), as the host hashes the strings glShaderSource gets; the Metal function is gl_<key>_v or gl_<key>_f;
|
||||
# · the stage: a capture's extension, an archive file's _vp / _fp; a string in main says nothing of its stage, so it is translated as each stage that compiles (one of
|
||||
# Digimon's built-in fragment shaders names gl_Position in a #define);
|
||||
# · GLSL → SPIR-V with glslang (-V -R: relaxed Vulkan rules, which gather the loose uniforms into one block, gl_DefaultUniformBlock; --aml / --amb number the inputs,
|
||||
# outputs and bindings), SPIR-V → MSL with spirv-cross (--msl-decoration-binding: Metal slots = those bindings; vertex shaders --fixup-clipspace: GL's z from −w to w → 0 to w);
|
||||
# · varyings are matched by name, as GL links them: --aml numbers each stage's outputs in its own declaration order, so [[user(locnN)]] becomes [[user(<variable name>)]];
|
||||
# · the host renders in GL's orientation (row 0 at the bottom, the NVN layer's lower-left origin), so nothing is flipped here.
|
||||
# Output: <out>/gl/gl_shaders.metallib and <out>/gl/gl_shaders.tsv; --ios <dir> also writes <dir>/gl/ with an iPhone library (same table)
|
||||
# Table rows: key <TAB> stage (vert / frag) <TAB> fields, space-separated (a key can have a row per stage):
|
||||
# a:<name>=<slot>:<type> a vertex input, [[attribute(slot)]]
|
||||
# u:<name>=<offset>:<type>:<count>:<stride> a member of the default uniform block (count 0 = not an array)
|
||||
# b:<slot>:<size> the default uniform block: its buffer slot and size in bytes
|
||||
# t:<name>=<slot>:<type> a combined sampler: texture and sampler slot
|
||||
# o:<name>=<location> a fragment output, [[color(location)]]
|
||||
# <TAB> where it came from
|
||||
# Usage: gl_shaders.py <input path>... <out dir> [--ios <dir>] (input paths: the RomFS, the main NSO, capture folders)
|
||||
import concurrent.futures as cf, hashlib, json, os, re, subprocess, sys, tempfile
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import mvgl
|
||||
|
||||
VEC_TYPES = {'float', 'vec2', 'vec3', 'vec4', 'int', 'ivec2', 'ivec3', 'ivec4', 'uint', 'uvec2', 'uvec3', 'uvec4', 'bool', 'mat2', 'mat3', 'mat4'}
|
||||
|
||||
def red(msg): sys.exit('[RED] ' + msg)
|
||||
|
||||
def text_of(b):
|
||||
b = bytes(b); z = b.find(b'\0')
|
||||
return b[:z] if z >= 0 else b
|
||||
|
||||
def nso_strings(path):
|
||||
"""'#version' strings of an NSO's read-only segment (lz4 segments, as dm/tools/nso.py decompresses them)"""
|
||||
import lz4.block, struct
|
||||
d = open(path, 'rb').read()
|
||||
flags = struct.unpack_from('<I', d, 0xc)[0]; fo, mo, sz = struct.unpack_from('<III', d, 0x20); csz = struct.unpack_from('<I', d, 0x64)[0]
|
||||
img = d[fo:fo + csz]
|
||||
if flags >> 1 & 1: img = lz4.block.decompress(img, uncompressed_size=sz)
|
||||
out, i = [], 0
|
||||
while True:
|
||||
i = img.find(b'#version', i)
|
||||
if i < 0: break
|
||||
if i == 0 or img[i - 1] == 0:
|
||||
out.append(text_of(img[i:i + 0x10000]))
|
||||
i += 8
|
||||
return out
|
||||
|
||||
def sources(roots):
|
||||
"""(source bytes, stage or None, where) for every shader in the input paths"""
|
||||
for r in roots:
|
||||
if os.path.isfile(r):
|
||||
with open(r, 'rb') as f: head = f.read(4)
|
||||
if head == b'NSO0':
|
||||
for k, s in enumerate(nso_strings(r)): yield s, None, f'{os.path.basename(r)}#{k}'
|
||||
continue
|
||||
for d, _, fs in os.walk(r):
|
||||
for fn in sorted(fs):
|
||||
p = os.path.join(d, fn)
|
||||
if fn.endswith('.mvgl'):
|
||||
try: a = mvgl.Archive(p)
|
||||
except ValueError: continue # the sound archives are CRI AFS2 under the same extension
|
||||
for nm in sorted(a.files):
|
||||
if not nm.endswith('.shad'): continue
|
||||
st = 'vert' if nm.endswith('_vp.shad') else 'frag' if nm.endswith('_fp.shad') else None
|
||||
yield text_of(a.read(nm)), st, f'/{os.path.relpath(p, r)}:{nm}'
|
||||
elif fn.endswith(('.vert', '.frag')):
|
||||
yield text_of(open(p, 'rb').read()), fn.rsplit('.', 1)[1], '/' + os.path.relpath(p, r)
|
||||
|
||||
# NVIDIA's GLSL compiler accepts more than the GLSL ES 3.20 specification, and glslang does not; these fix-ups run on the preprocessed text (glslang -E, so macros are expanded):
|
||||
# · the GLSL ES 1.0 texture functions (89 of Digimon's fragment shaders call textureCube; with -V glslang reads textureCube as Vulkan's texture type, so a macro cannot rename it);
|
||||
# · a redeclaration of gl_Position as a plain output (the built-in sprite shader of main declares `out float4 vPosition;` with vPosition defined as gl_Position);
|
||||
# · textureOffset with an offset that is not a constant (the built-in FXAA shader): Metal's sample offsets are constants too, so it becomes texture at P + offset / level 0 size
|
||||
OLD_TEX = [(r'\btexture2DLod\s*\(', 'textureLod('), (r'\btextureCubeLod\s*\(', 'textureLod('), (r'\btexture2DProj\s*\(', 'textureProj('),
|
||||
(r'\btexture2D\s*\(', 'texture('), (r'\btextureCube\s*\(', 'texture(')]
|
||||
|
||||
def call_args(t, i):
|
||||
"""the arguments of the call whose '(' is at t[i], and the index after its ')'"""
|
||||
depth, args, start = 0, [], i + 1
|
||||
for k in range(i, len(t)):
|
||||
c = t[k]
|
||||
if c == '(': depth += 1
|
||||
elif c == ')':
|
||||
depth -= 1
|
||||
if depth == 0: args.append(t[start:k]); return args, k + 1
|
||||
elif c == ',' and depth == 1: args.append(t[start:k]); start = k + 1
|
||||
raise ValueError('unbalanced parentheses')
|
||||
|
||||
def fixups(t):
|
||||
for a, b in OLD_TEX: t = re.sub(a, b, t)
|
||||
t = re.sub(r'^[ \t]*out\s+(?:highp\s+|mediump\s+|lowp\s+)?vec4\s+gl_Position\s*;', '', t, flags=re.M)
|
||||
out, i = [], 0
|
||||
for m in re.finditer(r'\btextureOffset\s*\(', t):
|
||||
if m.start() < i: continue
|
||||
args, end = call_args(t, m.end() - 1)
|
||||
if len(args) == 3 and re.search(r'[A-Za-z_]\w*', re.sub(r'\b[iu]?vec[234]\b', '', args[2])): # an offset with a variable in it
|
||||
s_, p_, o_ = (x.strip() for x in args)
|
||||
out.append(t[i:m.start()] + f'texture({s_}, ({p_}) + vec2({o_}) / vec2(textureSize({s_}, 0)))'); i = end
|
||||
return ''.join(out) + t[i:]
|
||||
|
||||
def glsl_type(t, refl):
|
||||
return refl['types'][t]['name'] if t in refl.get('types', {}) else t
|
||||
|
||||
def translate(src, stage, key, tmp):
|
||||
"""GLSL → MSL; returns (msl path, table fields) or an error string"""
|
||||
g = os.path.join(tmp, f'{key}.{stage}'); spv = g + '.spv'; msl = os.path.join(tmp, f'{key}_{stage[0]}.metal')
|
||||
open(g + '.in', 'wb').write(src)
|
||||
r = subprocess.run(['glslangValidator', '-E', '-S', stage, g + '.in'], capture_output=True, text=True)
|
||||
if r.returncode: return 'glslang -E: ' + ' | '.join(l for l in (r.stdout + r.stderr).splitlines() if 'ERROR' in l and 'compilation' not in l)[:400]
|
||||
pre = r.stdout[r.stdout.find('#version'):] # warnings come first
|
||||
open(g, 'w').write(fixups(pre))
|
||||
r = subprocess.run(['glslangValidator', '-V', '-R', '--aml', '--amb', '-S', stage, g, '-o', spv], capture_output=True, text=True)
|
||||
if r.returncode: return 'glslang: ' + ' | '.join(l for l in (r.stdout + r.stderr).splitlines() if 'ERROR' in l and 'compilation' not in l)[:400]
|
||||
refl = json.loads(subprocess.run(['spirv-cross', spv, '--reflect'], capture_output=True, text=True, check=True).stdout)
|
||||
cmd = ['spirv-cross', spv, '--msl', '--msl-version', '20300', '--msl-decoration-binding', '--rename-entry-point', 'main', f'gl_{key}_{stage[0]}', stage, '--output', msl]
|
||||
if stage == 'vert': cmd.append('--fixup-clipspace')
|
||||
r = subprocess.run(cmd, capture_output=True, text=True)
|
||||
if r.returncode: return 'spirv-cross: ' + r.stderr.strip()[:300]
|
||||
m = open(msl).read()
|
||||
# Varyings by name: the out struct of a vertex shader and the in struct of a fragment shader (stage_in members of a vertex shader are [[attribute(n)]])
|
||||
m = re.sub(r'(\b(\w+)\s+\[\[)user\(locn\d+\)', lambda x: f'{x.group(1)}user({x.group(2)})', m)
|
||||
open(msl, 'w').write(m)
|
||||
f = []
|
||||
if stage == 'vert':
|
||||
for v in refl.get('inputs', []): f.append(f'a:{v["name"]}={v["location"]}:{v["type"]}')
|
||||
else:
|
||||
for v in refl.get('outputs', []): f.append(f'o:{v["name"]}={v["location"]}')
|
||||
for u in refl.get('ubos', []):
|
||||
t = refl['types'][u['type']]
|
||||
if t['name'] != 'gl_DefaultUniformBlock': return f'uniform block {t["name"]} is not the default block'
|
||||
f.append(f'b:{u["binding"]}:{u["block_size"]}')
|
||||
for mb in t['members']:
|
||||
arr = mb.get('array', [])
|
||||
if len(arr) > 1: return f'uniform {mb["name"]}: arrays of arrays'
|
||||
ty = glsl_type(mb['type'], refl)
|
||||
if ty not in VEC_TYPES: return f'uniform {mb["name"]}: type {ty} (structs are not implemented)'
|
||||
f.append(f'u:{mb["name"]}={mb["offset"]}:{ty}:{arr[0] if arr else 0}:{mb.get("array_stride", 0)}')
|
||||
for t in refl.get('textures', []):
|
||||
if t.get('array'): return f'sampler array {t["name"]}'
|
||||
f.append(f't:{t["name"]}={t["binding"]}:{t["type"]}')
|
||||
for k in ('separate_images', 'separate_samplers', 'images', 'ssbos', 'push_constants'):
|
||||
if refl.get(k): return f'{k} are not implemented'
|
||||
if any(int(x) > 15 for x in re.findall(r'\[\[(?:sampler|buffer)\((\d+)\)\]\]', m)): return 'a buffer or sampler slot above 15'
|
||||
return msl, f
|
||||
|
||||
def one(item, tmp):
|
||||
"""[(key, stage, fields, msl, where)], or [(key, None, errors, None, where)] when no stage translates"""
|
||||
src, stage, where = item
|
||||
key = hashlib.sha256(src).hexdigest()[:16]
|
||||
out, errs = [], []
|
||||
for st in [stage] if stage else ['vert', 'frag']:
|
||||
r = translate(src, st, key, tmp)
|
||||
if isinstance(r, tuple): out.append((key, st, r[1], r[0], where))
|
||||
else: errs.append(f'{st}: {r}')
|
||||
return out or [(key, None, errs, None, where)]
|
||||
|
||||
def metallib(msls, out, sdk, extra):
|
||||
os.makedirs(out, exist_ok=True)
|
||||
def comp(m):
|
||||
a = m[:-6] + f'.{sdk}.air'
|
||||
r = subprocess.run(['xcrun', '-sdk', sdk, 'metal', '-c', '-fpreserve-invariance', *extra, m, '-o', a], capture_output=True, text=True)
|
||||
if r.returncode: red(f'{m}: metal compile failed: {r.stderr.strip()[:400]}')
|
||||
return a
|
||||
with cf.ThreadPoolExecutor(os.cpu_count()) as ex: airs = list(ex.map(comp, msls))
|
||||
lib = os.path.join(out, 'gl_shaders.metallib')
|
||||
subprocess.run(['xcrun', '-sdk', sdk, 'metallib', *airs, '-o', lib], check=True)
|
||||
return lib
|
||||
|
||||
def main():
|
||||
a = sys.argv[1:]
|
||||
ios = None
|
||||
if '--ios' in a: i = a.index('--ios'); ios = a[i + 1]; del a[i:i + 2]
|
||||
if len(a) < 2: sys.exit('usage: gl_shaders.py <input path>... <out dir> [--ios <dir>]')
|
||||
roots, out = a[:-1], a[-1]
|
||||
items, seen_src = [], set()
|
||||
for s, st, where in sources(roots):
|
||||
if (s, st) in seen_src: continue
|
||||
seen_src.add((s, st)); items.append((s, st, where))
|
||||
if not items: red(f'no GLSL shaders in {" ".join(roots)}')
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
with cf.ThreadPoolExecutor(os.cpu_count()) as ex: rows = [r for rs in ex.map(lambda it: one(it, tmp), items) for r in rs]
|
||||
seen, failed = {}, []
|
||||
for key, stage, f, msl, where in rows:
|
||||
if not stage: failed.append((key, where, f)); continue
|
||||
if (key, stage) not in seen: seen[(key, stage)] = (stage, f, msl, where)
|
||||
for key, where, errs in failed:
|
||||
if not any(k == key for k, _ in seen): print(f'[RED] {where} ({key}) does not translate: {" / ".join(errs)}')
|
||||
tsv = ['# GL shaders (dm/gfx/tools/gl_shaders.py): key = sha256(GLSL source)[:16], Metal function gl_<key>_v / gl_<key>_f; a: vertex inputs, u: default uniform block members '
|
||||
'(offset:type:count:stride), b: the block (slot:size), t: combined samplers (slot:type), o: fragment outputs',
|
||||
'# ' + ' '.join(subprocess.run(['glslangValidator', '--version'], capture_output=True, text=True).stdout.split()[:3]) + '; spirv-cross '
|
||||
+ ' '.join((lambda r: r.stdout + r.stderr)(subprocess.run(['spirv-cross', '--revision'], capture_output=True, text=True)).split())]
|
||||
tsv += [f'{k[0]}\t{v[0]}\t{" ".join(v[1])}\t{v[3]}' for k, v in sorted(seen.items())]
|
||||
dirs = [(os.path.join(out, 'gl'), 'macosx', [])]
|
||||
if ios: dirs.append((os.path.join(ios, 'gl'), 'iphoneos', ['-mios-version-min=18.0']))
|
||||
msls = [v[2] for v in seen.values()]
|
||||
for d, sdk, extra in dirs:
|
||||
lib = metallib(msls, d, sdk, extra)
|
||||
open(os.path.join(d, 'gl_shaders.tsv'), 'w').write('\n'.join(tsv) + '\n')
|
||||
print(f'{sdk}: {len(seen)} shaders ({sum(1 for v in seen.values() if v[0] == "vert")} vertex) → {lib}' + (f'; {len(failed)} failed' if failed else ''))
|
||||
|
||||
if __name__ == '__main__': main()
|
||||
@@ -0,0 +1,77 @@
|
||||
#!/usr/bin/env python3
|
||||
# Media.Vision's MVGL archives (Digimon Story Cyber Sleuth: Complete Edition, RomFS DSDB*.nx64.mvgl): every game file is in one of nine of them.
|
||||
# Layout, little-endian, no encryption on the Switch:
|
||||
# +0x00 'MDB1', u16 tree entries, u16 name entries, u32 data entries, u32 data start, u32 archive size
|
||||
# +0x14 tree entries, 8 bytes each: u16 compare bit, u16 data index (0xffff for the root and inner nodes), u16 left, u16 right (a Patricia tree over the names)
|
||||
# then name entries, 0x40 bytes each, by tree index: char[4] extension (space-padded), char[0x3c] name ('\'-separated path without extension)
|
||||
# then data entries, 12 bytes each: u32 offset (from data start), u32 length, u32 stored length
|
||||
# A file whose stored length is below its length is doboz-compressed (Attila T. Afra's doboz, version 0): one attributes byte (bits 0-2 version, 3-5 size width - 1,
|
||||
# bit 7 stored), the length and the stored length in that width, then the stream: a 32-bit control word per 32 items (bit 0 first, 0 = literal byte, 1 = match),
|
||||
# refilled when only its sentinel bit is left; a match is 1 to 4 bytes by its low 3 bits (offset and length - 3, see MATCH). The last 8 bytes are always literals.
|
||||
# Usage: mvgl.py list <archive> [ext] | mvgl.py get <archive> <name>.<ext> <out file>
|
||||
import struct, sys
|
||||
|
||||
MATCH = [(0xff, 2, 0, 0, 1), (0xffff, 2, 0, 0, 2), (0xffff, 6, 15, 2, 2), (0xffffff, 8, 31, 3, 3),
|
||||
(0xff, 2, 0, 0, 1), (0xffff, 2, 0, 0, 2), (0xffff, 6, 15, 2, 2), (0xffffffff, 11, 255, 3, 4)] # mask, offset shift, length mask, length shift, size
|
||||
TAIL = 8
|
||||
|
||||
|
||||
def doboz(b):
|
||||
a = b[0]
|
||||
if a & 7: raise ValueError('doboz version %d' % (a & 7))
|
||||
w = ((a >> 3) & 7) + 1
|
||||
size, csize = int.from_bytes(b[1:1 + w], 'little'), int.from_bytes(b[1 + w:1 + 2 * w], 'little')
|
||||
i = 1 + 2 * w
|
||||
if a & 0x80: return bytes(b[i:i + size])
|
||||
out = bytearray(); src = b + b'\0' * 8; ctl = 1
|
||||
while len(out) < size:
|
||||
if ctl == 1: ctl = int.from_bytes(src[i:i + 4], 'little'); i += 4
|
||||
if not ctl & 1:
|
||||
if len(out) < size - TAIL: out.append(src[i]); i += 1
|
||||
else: # the tail: every remaining byte is a literal, whatever the control bits say (doboz's decoder does the same)
|
||||
while len(out) < size:
|
||||
if ctl == 1: ctl = int.from_bytes(src[i:i + 4], 'little'); i += 4
|
||||
out.append(src[i]); i += 1; ctl >>= 1
|
||||
break
|
||||
else:
|
||||
word = int.from_bytes(src[i:i + 4], 'little'); m = MATCH[word & 7]
|
||||
off = (word & m[0]) >> m[1]; n = ((word >> m[3]) & m[2]) + 3; i += m[4]
|
||||
s = len(out) - off
|
||||
if off >= n: out += out[s:s + n]
|
||||
else:
|
||||
for k in range(n): out.append(out[s + k])
|
||||
ctl >>= 1
|
||||
if i - (1 + 2 * w) > csize + 4: raise ValueError('doboz stream overrun')
|
||||
return bytes(out)
|
||||
|
||||
|
||||
class Archive:
|
||||
def __init__(self, path):
|
||||
self.f = open(path, 'rb')
|
||||
magic, nt, nn, nd, self.start, self.size = struct.unpack('<4sHHIII', self.f.read(0x14))
|
||||
if magic != b'MDB1': raise ValueError('%s: not an MVGL archive' % path)
|
||||
tree, names, data = self.f.read(nt * 8), self.f.read(nn * 0x40), self.f.read(nd * 12)
|
||||
self.files = {} # 'name.ext' → (offset, length, stored length)
|
||||
for k in range(nt):
|
||||
idx = struct.unpack_from('<H', tree, k * 8 + 2)[0]
|
||||
if idx == 0xffff: continue
|
||||
ext = names[k * 0x40:k * 0x40 + 4].rstrip(b'\0 ').decode('latin1')
|
||||
name = names[k * 0x40 + 4:(k + 1) * 0x40].split(b'\0')[0].decode('latin1')
|
||||
off, n, cn = struct.unpack_from('<III', data, idx * 12)
|
||||
self.files[name + '.' + ext] = (self.start + off, n, cn)
|
||||
|
||||
def read(self, name):
|
||||
off, n, cn = self.files[name]
|
||||
self.f.seek(off); b = self.f.read(cn)
|
||||
return doboz(b) if cn < n else b
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
if len(sys.argv) >= 3 and sys.argv[1] == 'list':
|
||||
a = Archive(sys.argv[2])
|
||||
for nm, (off, n, cn) in sorted(a.files.items()):
|
||||
if len(sys.argv) < 4 or nm.endswith('.' + sys.argv[3]): print('%s\t%d\t%d' % (nm, n, cn))
|
||||
elif len(sys.argv) == 5 and sys.argv[1] == 'get':
|
||||
open(sys.argv[4], 'wb').write(Archive(sys.argv[2]).read(sys.argv[3]))
|
||||
else:
|
||||
sys.exit(__doc__ or 'usage: mvgl.py list <archive> [ext] | get <archive> <name.ext> <out>')
|
||||
+2
-2
@@ -18,8 +18,8 @@ CFLAGS=-O1 -g -Wall -Wno-unused-function
|
||||
# Service-layer takeover hle_*.c files are included by wildcard: adding a file does not touch this line (avoids merge conflicts)
|
||||
SRC=main.c loader.c hle.c crash.c probe.c thread.c memstat.m $(sort $(wildcard hle_*.c)) front_mac.m entry.S
|
||||
# NVN null device + nn::vi + the Vulkan layer (dm/gfx/)
|
||||
# Sources under dm/gfx, dm/gfx/nvn, dm/gfx/tex and dm/gfx/vk are included by wildcard (tests/ and metal/ do not go into the host): adding a file does not touch this line
|
||||
GFX=$(sort $(wildcard ../gfx/*.c ../gfx/nvn/*.c ../gfx/nvn/*.m ../gfx/nvn/*.S ../gfx/tex/*.c ../gfx/tex/*.m ../gfx/vk/*.m))
|
||||
# Sources under dm/gfx, dm/gfx/nvn, dm/gfx/tex, dm/gfx/vk and dm/gfx/gl are included by wildcard (tests/ and metal/ do not go into the host): adding a file does not touch this line
|
||||
GFX=$(sort $(wildcard ../gfx/*.c ../gfx/nvn/*.c ../gfx/nvn/*.m ../gfx/nvn/*.S ../gfx/tex/*.c ../gfx/tex/*.m ../gfx/vk/*.m ../gfx/gl/*.m))
|
||||
GFX_H=$(sort $(wildcard ../gfx/nvn/*.h ../gfx/nvn/include/*.h ../gfx/tex/*.h))
|
||||
# Third party: libvpx ships only the VP9 decoder's sources (movie decoding, hle_movie.c), with its configuration headers in config/ (arm64, shared by Mac and iOS). Built separately into a static library, file by file, so host changes do not recompile it
|
||||
VPX=../third_party/libvpx
|
||||
|
||||
+1
-1
@@ -43,7 +43,7 @@ typedef struct { const char *name; void *fn; const char *why; } hle_force_ent;
|
||||
extern const char cname##_bs[]; \
|
||||
__asm__(".text\n.p2align 2\n.globl _" #cname "_bs\n.private_extern _" #cname "_bs\n_" #cname "_bs:\n" \
|
||||
" adrp x16, _" #cname "@PAGE\n add x16, x16, _" #cname "@PAGEOFF\n b _dm_bigstack_enter\n"); \
|
||||
__attribute__((used, section("__DATA,swforce"))) static const hle_force_ent hle_force_##cname = {mangled, (const void *)cname##_bs, why}; \
|
||||
__attribute__((used, section("__DATA,swforce"))) static const hle_force_ent hle_force_##cname = {mangled, (void *)(uintptr_t)cname##_bs, why}; \
|
||||
static ret cname(__VA_ARGS__)
|
||||
|
||||
// Operation mode (0 handheld / 1 docked): owned by hle_oe.c, and the hid controller style follows it; the front end can switch it at runtime
|
||||
|
||||
+4
-2
@@ -28,9 +28,11 @@ static void aoc_load(void) {
|
||||
char list[256] = ""; for (int i = 0; i < g_naoc; i++) snprintf(list + strlen(list), sizeof list - strlen(list), "%s%d", i ? "," : "", g_aoc[i]);
|
||||
LOG("aoc: reporting %d add-on contents {%s} (%s)", g_naoc, list, why);
|
||||
}
|
||||
HLE_FORCE("_ZN2nn3aoc17CountAddOnContentEv", "add-on content count: length of the owned index list (DM_AOC_DIR)", int, h_aoc_Count, void) { aoc_load(); return g_naoc; }
|
||||
// On the big host stack: the first query lists DM_AOC_DIR and logs, a few KB, and Digimon Story Cyber Sleuth asks from a thread with little stack left (SIGBUS in __vfprintf
|
||||
// under aoc_load, a store right below a page of the guest thread's stack)
|
||||
HLE_FORCE_BIG("_ZN2nn3aoc17CountAddOnContentEv", "add-on content count: length of the owned index list (DM_AOC_DIR)", int, h_aoc_Count, void) { aoc_load(); return g_naoc; }
|
||||
// ListAddOnContent(out, start index, max count): writes out from the owned index list starting at the start index, returns the number written
|
||||
HLE_FORCE("_ZN2nn3aoc16ListAddOnContentEPiii", "add-on content list: the owned index list", int, h_aoc_List, int *out, int off, int n) {
|
||||
HLE_FORCE_BIG("_ZN2nn3aoc16ListAddOnContentEPiii", "add-on content list: the owned index list", int, h_aoc_List, int *out, int off, int n) {
|
||||
int w = 0; aoc_load();
|
||||
for (int i = off < 0 ? 0 : off; i < g_naoc && w < n; i++) out[w++] = g_aoc[i];
|
||||
return w;
|
||||
|
||||
@@ -7,6 +7,8 @@ static int unexpected, checks, fails;
|
||||
void *guest_sym(const char *name) { (void)name; unexpected++; return NULL; }
|
||||
module_t *module_of(const void *p) { (void)p; unexpected++; return NULL; }
|
||||
void hle_bind_system_event(void *g, int autoclear) { (void)g; (void)autoclear; unexpected++; }
|
||||
// The big-stack take-overs' stubs (HLE_FORCE_BIG) jump here with the implementation in x16; off a guest stack the real entry calls it straight
|
||||
__asm__(".text\n.p2align 2\n.globl _dm_bigstack_enter\n_dm_bigstack_enter:\n br x16\n");
|
||||
extern const struct mach_header_64 _mh_execute_header;
|
||||
#define CHECK(c, msg) do { checks++; if (!(c)) { fails++; fprintf(stderr, "FAIL: %s\n", msg); } } while (0)
|
||||
static void *hook(const char *name) {
|
||||
|
||||
@@ -32,10 +32,13 @@ export DM_LANG=${DM_LANG:-en-US}
|
||||
for f in $MODS; do [ -f "$PATCHED/$f.patched.bin" ] && [ -f "$PATCHED/$f.fixups.tsv" ] || die "no rewritten module $PATCHED/$f (run dm/tools/patch_nso.py first)"; done
|
||||
[ -e "$ROMFS/$MARKER" ] || die "no RomFS at $ROMFS (missing $MARKER)"
|
||||
V=$(sed -n 's/^#define DMSH_CHAIN "\(.*\)"/\1/p' "$GD/dmsh_chain.h")
|
||||
# A Vulkan game (Hades): the shaders are the game's SPIR-V, translated by dm/gfx/tools/vk_shaders.py into $MSL_IOS/vk; there are no NVN chain libraries. GFXAPI=nvn / vulkan says which
|
||||
# (the NSP script passes what main imports); without it, the shader libraries in $MSL_IOS tell: the chain's b-$V, else vk
|
||||
if [ -z "${GFXAPI:-}" ]; then if ls "$MSL_IOS/b-$V" 2> /dev/null | grep -q '_b.metallib$'; then GFXAPI=nvn; elif [ -f "$MSL_IOS/vk/vk_shaders.metallib" ]; then GFXAPI=vulkan; else GFXAPI=nvn; fi; fi
|
||||
# A Vulkan game (Hades): the shaders are the game's SPIR-V, translated by dm/gfx/tools/vk_shaders.py into $MSL_IOS/vk; an OpenGL game (Digimon Story Cyber Sleuth): its GLSL, translated by
|
||||
# dm/gfx/tools/gl_shaders.py into $MSL_IOS/gl; neither has NVN chain libraries. GFXAPI=nvn / vulkan / gl says which (the NSP script passes what main imports); without it, the shader
|
||||
# libraries in $MSL_IOS tell: the chain's b-$V, else vk, else gl
|
||||
if [ -z "${GFXAPI:-}" ]; then if ls "$MSL_IOS/b-$V" 2> /dev/null | grep -q '_b.metallib$'; then GFXAPI=nvn; elif [ -f "$MSL_IOS/vk/vk_shaders.metallib" ]; then GFXAPI=vulkan;
|
||||
elif [ -f "$MSL_IOS/gl/gl_shaders.metallib" ]; then GFXAPI=gl; else GFXAPI=nvn; fi; fi
|
||||
if [ "$GFXAPI" = vulkan ]; then [ -f "$MSL_IOS/vk/vk_shaders.metallib" ] || die "no iOS Vulkan shader library $MSL_IOS/vk (dm/gfx/tools/vk_shaders.py \"$ROMFS\" <Mac dir> --ios \"$MSL_IOS\")"
|
||||
elif [ "$GFXAPI" = gl ]; then [ -f "$MSL_IOS/gl/gl_shaders.metallib" ] || die "no iOS GL shader library $MSL_IOS/gl (dm/gfx/tools/gl_shaders.py \"$ROMFS\" <main> <Mac dir> --ios \"$MSL_IOS\")"
|
||||
else [ -n "$V" ] && ls "$MSL_IOS/b-$V" 2> /dev/null | grep -q '_b.metallib$' || die "no iOS shader libraries $MSL_IOS/b-$V (IOSOUT=$MSL_IOS dmsh_build.py ...)"; fi
|
||||
|
||||
t "app: $DM_DISPLAY_NAME $DM_APP_VERSION ($DM_BUNDLE_ID), language $DM_LANG$([ $DM_SHOW_FPS = YES ] && echo ', frame-rate counter on')"
|
||||
@@ -45,13 +48,13 @@ python3 tools/gen_build.py ../host Gen > Gen/dm.xcconfig || die "gen_build"
|
||||
F=$(python3 tools/gen_embed.py "$PATCHED" Gen/embed.S $MODS) || die "gen_embed"
|
||||
echo "OTHER_LDFLAGS = \$(inherited) \$(DM_FRAMEWORKS) $F" >> Gen/dm.xcconfig
|
||||
cp "$PATCHED"/*.fixups.tsv "$PATCHED"/*.gates.tsv "$PATCHED"/*.cuts.tsv "$PATCHED"/*.obs.tsv Gen/dm/ || die "copy rewriter tables"
|
||||
if [ "$GFXAPI" = vulkan ]; then cp "$GD/shader_map.tsv" "$GD/pso_manifest.tsv" Gen/dm/ || die "copy shader tables" # empty NVN tables: the NVN layer is never used
|
||||
if [ "$GFXAPI" = vulkan ] || [ "$GFXAPI" = gl ]; then cp "$GD/shader_map.tsv" "$GD/pso_manifest.tsv" Gen/dm/ || die "copy shader tables" # empty NVN tables: the NVN layer draws nothing
|
||||
else cp "$GD/shader_map.tsv" "$GD/pso_manifest.tsv" "$GD/c1_full.tsv" Gen/dm/ || die "copy shader tables"
|
||||
if [ -f "$GD/cb_extent.tsv" ]; then cp "$GD/cb_extent.tsv" Gen/dm/ || die "copy cb_extent.tsv"; fi; fi # optional: without it the host copies whole uniform bindings
|
||||
# Diagnostic build: DM_ENV_FILE=file → dm/env.txt in the bundle (main.m sets one environment variable per line at boot)
|
||||
if [ -n "${DM_ENV_FILE:-}" ]; then cp "$DM_ENV_FILE" Gen/dm/env.txt || die "copy DM_ENV_FILE"; t "env: $(grep -v '^#' "$DM_ENV_FILE" | tr '\n' ' ')"; fi
|
||||
# The shader libraries go into the archive as DmIOS.app/msl/<folder> (main.m sets DM_MSL_DIR to it)
|
||||
if [ "$GFXAPI" = vulkan ]; then MSLDIR=vk; else MSLDIR=b-$V; fi
|
||||
if [ "$GFXAPI" = vulkan ]; then MSLDIR=vk; elif [ "$GFXAPI" = gl ]; then MSLDIR=gl; else MSLDIR=b-$V; fi
|
||||
# Shared-font stand-ins: standard = macOS's Arial Unicode (absent on iOS), Nintendo extension = button icons generated by make_ext_font.py (needs fontTools; FONT_PY names a python that has it)
|
||||
mkdir -p Gen/dm/font && cp "${STD_FONT:-/System/Library/Fonts/Supplemental/Arial Unicode.ttf}" Gen/dm/font/standard.ttf || die "copy standard font (STD_FONT=a TTF)"
|
||||
"${FONT_PY:-python3}" "$DM/tools/make_ext_font.py" Gen/dm/font/nintendo_ext.ttf > /dev/null || die "make_ext_font.py (pip install fonttools, or FONT_PY=python with fontTools)"
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env python3
|
||||
# Which loaded entry points does a module's code really call? For loaders that fetch functions by name and keep each pointer in a global (glad's GL loader over eglGetProcAddress,
|
||||
# NVN's C proc loader over nvnDeviceGetProcAddress): the loader is a run of "name → x0 or x1 (adrp + add), call the get-proc function, adrp + ldr of a GOT slot, str x0 through it".
|
||||
# Every GOT slot stored that way is named after its string; the code outside the loader then reaches a function by loading its slot (adrp + ldr of the same GOT address), so the
|
||||
# slots loaded elsewhere are the functions the game calls. Digimon Story Cyber Sleuth: Complete Edition: glad asks for 2,956 GL names, the code loads 79 of them; its NVN loader
|
||||
# fetches 512, the code loads 30 (docs/agents/02-survey.md).
|
||||
# Usage: loader_census.py <NSO or decompressed image> [--sites name,name...] (an NSO is decompressed in memory; a .bin is taken as is, text from 0 to the end of the first segment it
|
||||
# cannot tell, so give an NSO when you can)
|
||||
import struct, sys, collections
|
||||
|
||||
def image(path):
|
||||
d = open(path, 'rb').read()
|
||||
if d[:4] != b'NSO0': return d, len(d)
|
||||
import lz4.block
|
||||
flags = struct.unpack_from('<I', d, 0xc)[0]; segs = [struct.unpack_from('<III', d, 0x10 + i * 0x10) for i in range(3)]; csz = struct.unpack_from('<III', d, 0x60)
|
||||
img = bytearray(segs[2][1] + segs[2][2])
|
||||
for i, (fo, mo, sz) in enumerate(segs):
|
||||
raw = d[fo:fo + csz[i]]
|
||||
img[mo:mo + sz] = lz4.block.decompress(raw, uncompressed_size=sz) if flags >> i & 1 else raw
|
||||
return bytes(img), segs[0][2]
|
||||
|
||||
def adrp(w, pc):
|
||||
if (w & 0x9f000000) != 0x90000000: return None
|
||||
imm = ((((w >> 5) & 0x7ffff) << 2) | ((w >> 29) & 3)) << 12
|
||||
if imm & (1 << 32): imm -= 1 << 33
|
||||
return w & 31, (pc & ~0xfff) + imm
|
||||
def addimm(w): return (w & 31, (w >> 5) & 31, (w >> 10) & 0xfff) if (w & 0xffc00000) == 0x91000000 else None
|
||||
def ldr64(w): return (w & 31, (w >> 5) & 31, ((w >> 10) & 0xfff) * 8) if (w & 0xffc00000) == 0xf9400000 else None
|
||||
def is_call(w): return (w & 0xfc000000) == 0x94000000 or (w & 0xfffffc1f) == 0xd63f0000 # bl, blr
|
||||
def is_str_x0(w, base): return (w & 0xffc003ff) == 0xf9000000 | (base << 5) and ((w >> 10) & 0xfff) == 0 # str x0, [base]
|
||||
|
||||
def main():
|
||||
a = sys.argv[1:]
|
||||
want = []
|
||||
if '--sites' in a: i = a.index('--sites'); want = a[i + 1].split(','); del a[i:i + 2]
|
||||
if len(a) != 1: sys.exit('usage: loader_census.py <NSO or image> [--sites name,...]')
|
||||
img, text = image(a[0])
|
||||
n = text // 4; words = struct.unpack_from(f'<{n}I', img, 0)
|
||||
def cstr(p):
|
||||
e = img.find(b'\0', p, p + 256)
|
||||
try: return img[p:e].decode('ascii') if e > p else None
|
||||
except UnicodeDecodeError: return None
|
||||
slot, loader = {}, set()
|
||||
for i in range(4, n - 3):
|
||||
if not is_call(words[i]): continue
|
||||
name = None
|
||||
for k in range(i - 4, i - 1): # the name: adrp + add into x0 or x1 just before the call
|
||||
r = adrp(words[k], k * 4); s = addimm(words[k + 1])
|
||||
if r and s and r[0] in (0, 1) and s[1] == r[0] and s[0] == r[0]: name = cstr(r[1] + s[2])
|
||||
r = adrp(words[i + 1], (i + 1) * 4); l = ldr64(words[i + 2])
|
||||
if not (name and r and l and l[1] == r[0] and is_str_x0(words[i + 3], l[0])): continue
|
||||
slot[r[1] + l[2]] = name
|
||||
loader.update(range((i - 4) * 4, (i + 4) * 4, 4))
|
||||
refs, sites = collections.Counter(), collections.defaultdict(list)
|
||||
for i in range(n - 1):
|
||||
r = adrp(words[i], i * 4)
|
||||
if not r or i * 4 in loader: continue
|
||||
for j in range(i + 1, min(i + 6, n)):
|
||||
l = ldr64(words[j])
|
||||
if l and l[1] == r[0]:
|
||||
s = r[1] + l[2]
|
||||
if s in slot: refs[slot[s]] += 1; sites[slot[s]].append(i * 4)
|
||||
break
|
||||
fams = collections.Counter(nm[:2] if nm.startswith('gl') else nm[:3] for nm in slot.values())
|
||||
print(f'{len(slot)} slots stored by a loader ({", ".join(f"{k}* {v}" for k, v in fams.most_common(6))}); {len(refs)} loaded by other code:')
|
||||
for nm in sorted(refs): print(f' {refs[nm]:4d} {nm}')
|
||||
for nm in want: print(f'{nm}: ' + ' '.join(hex(x) for x in sites.get(nm, [])))
|
||||
|
||||
if __name__ == '__main__': main()
|
||||
@@ -48,6 +48,7 @@ The tables below are what to check by hand afterwards.
|
||||
|
||||
## NVN surface
|
||||
|
||||
- First check that the game draws with NVN at all. Importing `nvnBootstrapLoader` is not enough: Digimon Story Cyber Sleuth: Complete Edition imports it and EGL, makes an NVN device for a helper beside its movie player, and renders everything with NVIDIA's desktop OpenGL (`eglBindAPI(EGL_OPENGL_API)`, a glad loader asking `eglGetProcAddress` for 2,956 names). Read the graphics setup that calls `nvnBootstrapLoader` / `eglGetDisplay` / `vkCreateInstance`, and count which loaded entry points the code really reaches: for a glad or NVN C loader that stores each pointer in a global, the GOT slots of those globals that code outside the loader loads (Digimon: 79 GL functions, 30 NVN ones, all in that helper). `profile_guess.py` decides `graphics gl` when `main` imports `eglCreateWindowSurface` and names `glShaderSource`; the host's GL layer is `dm/gfx/gl/hle_gl.m` ([04](04-graphics.md)).
|
||||
- Find how the game obtains NVN entry points. Some games write the slots with `adrp`-to-slot code; Mario's loader (`main+0x1070910`) calls `GetProcAddress` per name and stores through the GOT (`dm/tools/nvn_slots_got.py <main.bin> <loader> <text end> <out tsv>`). The output lists every fetched function and every call site.
|
||||
- Mario fetched 534 functions and called 236 of them; 79 were new to the host (CommandBuffer 31, Texture 15, Buffer 10, Sampler 10). Names the host doesn't know at all go into `dm/gfx/nvn/names.txt` → `gen_protos.py`.
|
||||
|
||||
|
||||
@@ -179,6 +179,9 @@ Each entry: symptom → root cause → fix (where the fix lives). Check these fi
|
||||
- **The first loading screen never ends; `DM_FS=1` shows `save:/system` "not found" and no `fs mount save`** → in SDK 6 `account::OpenPreselectedUser` returns a Result, and the game skips its save mount when it is not 0 (`main+0x53b7e0`). The host's hook returned void and left the handle pointer in w0. It returns 0 now; SDK 7 and later callers ignore it.
|
||||
- **No sound; red `audren: command 1 in the command list is malformed (magic 0 size 0)`** → SDK 6's DSP command header is +0x04 size (s16), +0x06 command ID (s16), +0x10 enabled, where SDK 8–20 have +0x04 enabled, +0x05 ID, +0x06 size (`CommandListProcessor::Process`, `sdk+0xaaa00`); its DeviceSink is ID 0x10. `capture` in `hle_audren.c` takes the layout whose walk finds the 0xcafebabe magic on every command of the first full list.
|
||||
- **Shaders missing from the map that no game file holds** → KTGL compiles 9 built-in GLSL shaders at boot with the game's own GLSLC (`glslcCompile` from `main+0x9c89f8`; the sources are strings in `main`). `DM_GLSLC_CAPTURE=<dir>` saves every compile's output (`hle_glslc.c`; the compile object's first word points at {status, output}), and `shader_split.py` reads `<version dir>/glslc_capture` (the NSP script makes it for any game whose `main` holds GLSL sources and imports `glslcCompile`). The outputs are identical on every run.
|
||||
- **SDK 6.4.0 (Digimon Story Cyber Sleuth: Complete Edition, Media.Vision's MVGL engine on OpenGL), one boot each**:
|
||||
- **`UNIMPL eglGetDisplay`, then `Break` right after `nv::InitializeGraphics` and `vi::OpenDefaultDisplay`** (`main+0x5ed9d0`) → the game renders with NVIDIA's desktop OpenGL through EGL; its NVN device (made just before) serves only a helper beside the movie player. The host's GL layer (`dm/gfx/gl/hle_gl.m`) answers the EGL imports and `eglGetProcAddress` ([04](04-graphics.md)).
|
||||
- **SIGBUS in `__vfprintf` under `aoc_load`, from `h_aoc_Count`** (a store to `0x…67ff0`, right below a page of the guest thread's stack) → the first add-on content query lists `DM_AOC_DIR` and logs, a few KB of stack, and the game asks from a thread with little left. `aoc::CountAddOnContent` and `ListAddOnContent` run on the big host stack (`HLE_FORCE_BIG`, `hle_misc.c`).
|
||||
- **SDK 8 (Link's Awakening, 8.2.0)**:
|
||||
- **Abort in `os::WaitSystemEvent` under `oe::PopNotificationMessage`** → the blocking pop waits on the system event `oe::Initialize` would have created, and the host's `Initialize` is a no-op. The host now implements it on its notification queue (a condition variable, then the `TryPop` path).
|
||||
- **SIGSEGV in `vi::SetLayerCrop`** → the SDK's version dereferences vi service objects the host never made; `SetLayerCrop`, `DestroyLayer`, `CloseDisplay` and `vi::Finalize` are host no-ops.
|
||||
|
||||
@@ -383,3 +383,18 @@ Atlus's GFD engine on SDK 8.2.1: 3,128 shaders (1,244 vertex, 1,872 pixel, 12 co
|
||||
- **Shader type 0 in every vertex shader's SPH** (word 0 = `0x0061`; every other game's vertex shaders have `0x0461`) → `shader_split.py` found no stage for them and stopped at the first captured output (`no GLSLC output`). Ryujinx's `ShaderHeader` reads a graphics header's type 0 as a vertex shader; `sph_stage` does too. A census of the other games' packages and of every `main` on hand found no other type-0 vertex shader.
|
||||
- **Fixed: every post pass and the window drew one texel's colour (a black screen with sound).** That shader type is Maxwell's first vertex slot (VERTEX_CULL_BEFORE_FETCH), and this GLSLC (GPU code version 1.15) writes every vertex shader as two programs in one code area: part A at code +0 (the SPH above) loads the position attribute and writes only the position; part B, at control +0x700 with a 0x30-byte header of zeros and an SPH of type 1, reads the other attributes and writes the other outputs. Control +0x748 lists both as {SPH offset, length from the SPH, ?}. The hardware runs A, then B. Translated alone, A left every varying at (0, 0, 0, 1), so the full-screen passes sampled their sources at (0, 0) and the 2D draws their atlases' first texel. `dmsh_build.py` (`vertex_ab`) merges the two as Ryujinx's `TranslatorContext.Translate(VertexA)` does: A's instructions with each unconditional `EXIT` turned into a `BRA` to B's first instruction, then B's, under B's SPH with A's flags and input / output maps ORed in and the larger local memory sizes. All 1,244: A writes no generic output, A ends in an `EXIT` predicated on CC FCSM_TR (which Ryujinx never takes; left as it is), an unconditional `EXIT` and a `BRA` to itself, and neither part has an absolute or table jump, so moving B keeps every branch. c1 is what follows part B (zero padding in all of them). A type-0 shader with no part B listed is translated as it is.
|
||||
- **Fixed: the cutscene subtitle band was a hard-edged dark box with a magenta stroke through it.** The band is a 32x512 BC3 gradient the game copies from a staging buffer (`nvnCommandBufferCopyBufferToTexture`, recorded at the start of a frame, `main+0x305e64`), and the GPU copy read the next texture's blocks: the source was right when recorded and when encoded, but the game's resource thread wrote the next texture into the same staging block 64 ms later, before that first submission had even been committed, because the host spent 300 ms building pipelines inside it. On the Switch the GPU runs a submission's copies as soon as the submit returns. A segment is now committed at the first pass boundary after any buffer → texture copy (`g_cut_copy`, `sub_cut` in `nvn_mtl.m`), so the copies run while the host encodes the rest; the `residency set` line counts these cuts (`N after buffer → texture copies`, a few per 300 frames in this game, only while it loads). Found with `DM_NVN_HILITE` on the band's pixel shader, `DM_NVN_TEXRAW` of the texture (decoded: opaque black and magenta blocks, not a fade) and `DM_PROBE_WATCH` on the staging address beside a log of the copy's submission completing.
|
||||
|
||||
### Seen in Digimon Story Cyber Sleuth: Complete Edition (OpenGL)
|
||||
|
||||
Media.Vision's MVGL engine on SDK 6.4.0 renders with **NVIDIA's desktop OpenGL 4.5 core profile through EGL**, not NVN. The host implements GL itself on Metal (`dm/gfx/gl/hle_gl.m`), as it does Vulkan for Hades; NVIDIA's GL driver (`subsdk0`) is not loaded.
|
||||
|
||||
- **Telling the API apart**: `main` imports `nvnBootstrapLoader` and 13 EGL functions. Its graphics setup (`main+0x5ed8f0`) makes an NVN device first, which only a helper beside the movie player uses (`main+0x528000`–`+0x52c400`: a memory pool and four textures next to the movie's GL textures). Then `eglGetDisplay`, `eglBindAPI(EGL_OPENGL_API)`, a 4.5 core context, a window surface on `vi::CreateLayer(w, h)` and a glad loader (`main+0x52c460`). Absolum imports the same EGL functions (SDL3) but renders with NVN and names no GL function. `profile_guess.py` decides `graphics gl` when `main` imports `eglCreateWindowSurface` and names `glShaderSource`; of the 22 games extracted here only Digimon does.
|
||||
- **Which GL functions**: glad asks `eglGetProcAddress` for 2,956 names and stores each in a global its code reaches through the GOT. A census of the GOT slots the code loads outside the loader gives the 79 functions the game calls: buffers, vertex arrays, textures (2D and cube, S3TC), framebuffers and renderbuffers, programs with plain `glUniform*`, `glDrawElements` / `glDrawArrays`, and three compatibility calls (`glTexEnvi`, `glColorTable`, `glPixelTransferi`). No uniform buffers, instancing or buffer mapping. Every other name gets a null pointer.
|
||||
- **Shaders are GLSL ES 3.20 source**, compiled at run time with `glShaderSource` / `glCompileShader`: 1,682 `.shad` files in the MVGL archives (615 distinct) and 37 strings in `main` (post effects, FXAA, shadows, the movie's YUV). `gl_shaders.py` translates them offline: glslang (`-V -R`: relaxed Vulkan rules gather the loose uniforms into one `gl_DefaultUniformBlock`, whose member offsets the table lists) → SPIR-V → SPIRV-Cross. The host finds each shader by the hash of its text and its stage. Varyings are matched by name, as GL links them: `[[user(locnN)]]` becomes `[[user(<name>)]]`, so a vertex and a fragment shader translated apart still meet. `--fixup-clipspace` maps GL's clip z to Metal's.
|
||||
- **What NVIDIA's compiler accepts and glslang does not** (fix-ups on the preprocessed text): the GLSL ES 1.0 functions `textureCube` / `texture2D` in ES 3.20 code (89 fragment shaders; with `-V` glslang reads `textureCube` as Vulkan's texture type, so a macro cannot rename it), `out vec4 gl_Position` redeclared through a macro (the built-in sprite shader), and `textureOffset` with an offset that is not a constant (FXAA), which becomes a plain sample at P + offset ÷ level 0 size. A string in `main` says nothing of its stage, so it is translated as each stage that compiles (one fragment shader defines `vPosition` as `gl_Position`).
|
||||
- **Orientation**: the layer renders as GL does, bottom row first, through the NVN layer's lower-left origin (`mtl_set_origin_lower_left`): flipped viewports, reversed winding, a flipped present. Scissor rectangles and `gl_FragCoord` need no change, and render targets read back with GL's texture coordinates.
|
||||
- **State**: uniforms are CPU copies per stage, set as bytes at a draw after they change; pipelines are cached per program by vertex layout, attachment formats, blend and colour mask; disabled vertex arrays read GL's default (0, 0, 0, 1) through a constant-step buffer. The game's attribute index is the Metal one: `glGetAttribLocation` returns the shader's own attribute numbers.
|
||||
- **Textures** are made on first use from the levels given so far (the loader defines them level by level). An upload into a texture a pending command buffer may still read is a blit in command order; any other is a CPU write. Metal reads a channel a format lacks as 0 (alpha 1) as GL does, so only luminance / alpha formats get a swizzle: Metal refuses a swizzle on a texture that can be a render target (`Texture swizzling is not compatable with MTLTextureUsageRenderTarget`, the first story scene).
|
||||
- **Buffers**: `glBufferData` on a buffer the GPU may still read gets a new `MTLBuffer`, as GL drivers orphan it. **Clears**: a whole-framebuffer clear becomes the next pass's load action; a scissored or masked one is a quad.
|
||||
- **Present**: three window textures through the NVN layer's present path (`mtl_ext_tex`, as the Vulkan swapchain), paced by the swap interval in display-link ticks. The game sets the interval to 60 ÷ its frame rate before each swap: 1 at the title and in the DigiLab, 2 from the story's opening on.
|
||||
- **Movies**: the USM movies' H.264 is decoded through the SDK's movie library (the host's decoder) into NV12 buffers (3,110,400 bytes a 1920x1080 frame); the game uploads the planes into a `GL_RED` and a `GL_RG` texture (two `glTexSubImage2D` a frame) and converts YUV in its own shader (in `main`, `SamplerY` / `SamplerUV`).
|
||||
+13
-1
@@ -8,7 +8,7 @@ The README covers packing and playing. This page is for working on nxnative itse
|
||||
|---|---|
|
||||
| Rewriter (`dm/tools/patch_nso.py`) | Runs unchanged on every module of every supported game. Per-build cuts are keyed by the module's build ID |
|
||||
| Host (`dm/host`) | Stands in for the Horizon kernel and the system services the games call: threads and synchronization, memory, file system with transactional saves, HID, audio, movies (VP9 through libvpx, Vorbis sound through stb_vorbis), software keyboard, focus and lifecycle |
|
||||
| Graphics (`dm/gfx`) | NVN on Metal. Shaders are extracted by `shader_split.py` from Nintendo's containers and from Unity's `Shader` objects, then translated offline. Mario: 16,266 of 16,268 shaders compile. Engage: 9,268 of 9,269 |
|
||||
| Graphics (`dm/gfx`) | NVN on Metal. Shaders are extracted by `shader_split.py` from Nintendo's containers and from Unity's `Shader` objects, then translated offline. Mario: 16,266 of 16,268 shaders compile. Engage: 9,268 of 9,269. Games that render with Vulkan (Hades) or OpenGL (Digimon Story Cyber Sleuth) go through layers of their own on Metal (`dm/gfx/vk`, `dm/gfx/gl`) |
|
||||
| Audio | Games that carry the SDK's audio renderer (Mario) run it in-process; games that use AudioOut directly (Engage: Wwise) feed it straight. Output goes through Core Audio (default on iOS, `DM_AUDIO=1` on the Mac) |
|
||||
| Input | iOS: the touch gamepad and physical controllers; shaking the phone shakes the controller (its six-axis sensor, `hle_hid.c`; the accelerometer in `DmGamepad.m`, threshold `DM_SHAKE_G`, 1.5 G by default). Mac: scripted input (`DM_KEYS=<seconds>:<button>,…`, `Shake` for a shake, with `DM_KBD=<text>` answering the system keyboard) or `DM_WINDOW=1` for a window with keyboard input (Space shakes) and physical controllers (buttons by position, as on the iPhone). The window goes fullscreen from its green button or ⌃⌘F; `DM_FULLSCREEN=1` starts it that way. The left mouse button is the touchscreen, on the 16:9 picture |
|
||||
| iOS app | One unsigned IPA for iPhone and iPad. The rewritten code is embedded in the executable; the shader libraries, the RomFS and the game icon are inside the app |
|
||||
@@ -509,6 +509,17 @@ The game's own work and its frame rate stay as they are; the host avoids work of
|
||||
- **Input:** + pauses a cutscene (B resumes, Y skips); the left stick moves Vincent, A grabs blocks.
|
||||
- **Mac input:** from a fresh save, `DM_KEYS="5:A*25/2,60:Plus*14/4,61:Y*14/4,62:A*14/4"` skips the intro scenes into the first Nightmare stage (about frame 3,300); `120:LUp~0.4*40/1.5` then climbs.
|
||||
|
||||
**Digimon Story Cyber Sleuth: Complete Edition (`010014E00DB56000`)**
|
||||
- **Status:** brought up on the Mac: the logos, the title, the privacy policy, the DigiLab story choice, the EDEN chat opening (movies and real-time scenes), character creation and the login, into EDEN (the fly-in, then its entrance plaza in 3D with Taiga under control), with sound, at 60 fps in the menus and 30 from the story on (the game's own rates); no red line. About 0.95 GB resident (0.3 GB of Metal allocations in EDEN), flat over half an hour. Playable on an iPhone.
|
||||
- **Engine:** Media.Vision's MVGL engine (the PC version's) on SDK 6.4.0, update 1.0.3 (v196608). It renders with **OpenGL**: NVIDIA's desktop GL 4.5 core profile through EGL, from `subsdk0`, which is not loaded. `main` gets every GL function from a glad loader (`eglGetProcAddress`, 2,956 names) and calls 79 of them; the host implements those on Metal (`dm/gfx/gl/hle_gl.m`). Its NVN device serves only a helper beside the movie player. CRI's middleware plays the sound (one 6-channel AudioOut) and the USM movies (H.264 through the SDK's movie library, `subsdk1`, into NV12 buffers the game uploads into GL textures).
|
||||
- **Files:** everything but the movies is in nine MVGL archives (`DSDB*.nx64.mvgl`: `MDB1`, a name tree, doboz-compressed entries; `dm/gfx/tools/mvgl.py` lists and reads them); four of them are CRI AFS2 sound banks under the same extension.
|
||||
- **GL layer:** one context; GL calls are encoded into Metal as they come, one command buffer per frame; GL's bottom-up orientation through the NVN layer's lower-left origin; textures made on first use from the levels given; buffers orphaned on `glBufferData`; the window surface (the size of the game's `vi::CreateLayer`, 1920x1080) presented through the NVN layer's present path, paced by the swap interval ([04](agents/04-graphics.md)).
|
||||
- **Shaders:** GLSL ES 3.20 source compiled at run time: 1,682 `.shad` files in the archives (615 distinct) and 37 strings in `main`. `dm/gfx/tools/gl_shaders.py <RomFS> <main> <dir> [--ios <dir>]` translates them with glslang and SPIRV-Cross into `<dir>/gl/` (676 shaders: built-in strings are translated as each stage that compiles); the host finds each by the hash of its text. `DM_GL_CAPTURE=<dir>` writes every source the game compiles, which `gl_shaders.py` reads like a RomFS; every one captured on the route so far is in the archives or `main`.
|
||||
- **Host changes:** the add-on content queries run on the big host stack (`aoc_load` logs from a game thread with little stack left) ([03](agents/03-bringup.md)); `vi::CreateLayer(w, h)` keeps the layer's size for the window surface.
|
||||
- **Memory:** no system resource; the game sets a 3.13 GB heap (pool − used) in the 4 GB layout. The rewritten modules need 74 MB of used memory (`mem_used`).
|
||||
- **Packing:** for a GL game the NSP script runs `gl_shaders.py` (Mac and iOS libraries in `<id>_build/shader/gl` and `shader/ios/gl`) in place of the NVN shader split and chain build; it needs glslang (`brew install glslang`).
|
||||
- **Mac input:** a fresh save reaches EDEN with `DM_KBD=Taiga DM_KEYS="f2000:A*26/200,f7300:Right,f7400:A,f7600:A,f7800:Right,f7850:A,f8100:A*39/250,f9000:Plus*15/600,f18000:Down*3/1200,f18100:A*3/1200,f21700:Left*300/250,f21800:A*300/250"` (the title, the privacy policy, the DigiLab: Cyber Sleuth, no options, A through the opening chat; the login form comes at frame 18,500 to 19,700). The name is a field of that form: A opens the keyboard, and the game's name screen stays up about 500 frames after the text arrives, where another A opens it again, so Down and A come 1,200 frames apart. The login prompt defaults to No, hence Left before each A from there; EDEN at about frame 22,500.
|
||||
|
||||
## Tools (`dm/tools`)
|
||||
|
||||
| Tool | What it does |
|
||||
@@ -520,3 +531,4 @@ The game's own work and its frame rate stay as they are; the host avoids work of
|
||||
| `nso_code.py` | Function ranges from `.eh_frame` and the data blocks in a text segment, which the rewriter and the census leave out |
|
||||
| `nso_syms.py` | Compares the SDK import surface of two games, or of one game (`-` for the reference) with the host hooks |
|
||||
| `nvn_slots_got.py` | Builds the NVN fetch table and call sites (GOT-indirect loaders) |
|
||||
| `loader_census.py` | Which of the functions a name loader fetches (glad's GL loader over `eglGetProcAddress`, NVN's C proc loader) the code really calls: the GOT slots the loader stores, loaded by other code |
|
||||
@@ -497,11 +497,12 @@ def survey(nsp_dir, app, fam, keys, repo, py, env):
|
||||
G = pg.guess(F) # the provisional profile's keys (write_profile writes the same)
|
||||
keep = G['modules'][0].split() # how the build will get the shaders (decided again from the extraction at build time; the survey has only a RomFS manifest, so Unreal paks are guessed from main's markers)
|
||||
S = pg.shader_sources(mods['main']['imports'], F['glsl_src'], [m for m in keep if 'glslcCompile' in mods[m]['exports']], rom['mags']['SPIR-V'],
|
||||
rom['exts'].get('.pak', 0) if F.get('main_markers', set()) & {'Unreal', 'Unreal 4'} else 0, rom['mags']['Trinity pack'])
|
||||
rom['exts'].get('.pak', 0) if F.get('main_markers', set()) & {'Unreal', 'Unreal 4'} else 0, rom['mags']['Trinity pack'], gl_names=F.get('gl_names', 0))
|
||||
vulkan = S['graphics'][0] == 'vulkan'
|
||||
notes = []
|
||||
if not vulkan and 'nvnBootstrapLoader' not in mods['main']['imports']: notes.append('main imports neither nvnBootstrapLoader nor Vulkan functions: the host implements only NVN and Vulkan')
|
||||
if vulkan: notes.append(f'{S["graphics"][1]}; the NVN shader steps are skipped' + (', and its shaders are captured by booting it on the Mac (no SPIR-V file in the RomFS)' if S['vk_capture'][0] else ''))
|
||||
if S['graphics'][0] == 'gl': notes.append(f'{S["graphics"][1]}; the NVN shader steps are skipped: gl_shaders.py translates its GLSL sources (MVGL archives, main\'s strings)')
|
||||
notes += [S[k][1] for k in ('glslc_capture',) if S[k][0]]
|
||||
if S['ue_decode'][0]: # the manifest has no pak footers, so both kinds are named; the build tells them apart by the footer's version (uepak.py)
|
||||
notes.append(f'{rom["exts"].get(".pak", 0)} Unreal pak(s): version 11 paks keep the shaders Oodle-compressed in shader code archives (Octopath Traveler 0), decoded by booting the game on the Mac; '
|
||||
@@ -1191,6 +1192,27 @@ def build_vk_shaders(py, repo, roots, sh, env, logs, g, reuse=True):
|
||||
return ios
|
||||
|
||||
|
||||
def build_gl_shaders(py, repo, roots, sh, env, logs, g, reuse=True):
|
||||
"""An OpenGL game's GLSL sources (roots: the RomFS, main, a run-time capture folder) → one Metal library each for the Mac (sh/gl/) and the iPhone (sh/ios/gl/, what build_ipa.sh takes from MSL_IOS),
|
||||
replacing the NVN split and chain build, as build_vk_shaders does for a Vulkan game; glslang (glslangValidator) compiles the GLSL to SPIR-V for spirv-cross"""
|
||||
ios = sh / 'ios'; kf = ios / 'gl.key'
|
||||
PROG.stage('GL shaders', step='checking the inputs')
|
||||
if not shutil.which('glslangValidator', path=env['PATH']): raise BuildError('glslangValidator not found (brew install glslang): it compiles an OpenGL game\'s GLSL shaders')
|
||||
key = fingerprint(*[('tree', r) if Path(r).is_dir() else ('files', [r]) for r in roots], ('files', tool_files(repo)), ('files', profile_files(g)), *toolchain(),
|
||||
('cmd', ['glslangValidator', '--version']), env=env) if reuse else None
|
||||
if key and (ios / 'gl/gl_shaders.metallib').is_file() and (note := reused(kf, key)) is not None:
|
||||
PROG.end(f'{note}, reused (inputs unchanged)'); return ios
|
||||
PROG.set(step='translating')
|
||||
rmtree(kf); rmtree(sh / 'gl'); rmtree(ios / 'gl')
|
||||
run([py, repo / 'dm/gfx/tools/gl_shaders.py', *roots, sh, '--ios', ios], logs / 'gl_shaders.log', env=dict(env, TMPDIR=env['WORK']))
|
||||
if not (ios / 'gl/gl_shaders.metallib').is_file(): raise BuildError(f'no iOS GL shader library in {ios / "gl"}')
|
||||
m = re.search(r'iphoneos: (\d+)', (logs / 'gl_shaders.log').read_text(errors='replace'))
|
||||
note = f'{int(m.group(1)):,} shaders for the Mac and iOS' if m else ''
|
||||
if key: kf.write_text(f'{key}\n{note}\n')
|
||||
PROG.end(note)
|
||||
return ios
|
||||
|
||||
|
||||
class IpaWatch:
|
||||
"""Progress of build_ipa.sh, from its step lines ("[time] generate Gen/ …", "xcodebuild …", "pack IPA …", then pack_ipa.py's "pack: N entries, X bytes") and the files it writes:
|
||||
the CompileC lines in xcodebuild's build.log (one per source file: Gen/src, Gen/main/main.c, Gen/embed.S and the app's Sources), then the bytes of the archive pack_ipa.py is
|
||||
@@ -1373,11 +1395,12 @@ def main(args=None):
|
||||
game = dirs[pkg_ver]
|
||||
# how the shaders are got follows from the packed version's files (profile_guess.py shader_sources): the graphics API, and the shaders made only at run time
|
||||
src = pg.build_sources(game / 'ExeFS', game / 'RomFS', g.words('modules'))
|
||||
gfx = src['graphics'][0] # nvn, or vulkan (Hades: the host's Vulkan layer, SPIR-V shaders)
|
||||
gfx = src['graphics'][0] # nvn, vulkan (Hades: the host's Vulkan layer, SPIR-V shaders) or gl (Digimon Story Cyber Sleuth: the host's GL layer, GLSL sources)
|
||||
glslc = gfx == 'nvn' and src['glslc_capture'][0] # shaders the game compiles at run time, captured in the Mac host (capture_glslc)
|
||||
vkcap = gfx == 'vulkan' and src['vk_capture'][0] # SPIR-V shaders the game generates at run time, captured in the Mac host (capture_vk)
|
||||
uedec = gfx == 'nvn' and src['ue_decode'][0] # Unreal Engine shaders, Oodle-compressed in the paks' shader code archives, decoded in the Mac host (decode_ue_shaders)
|
||||
if gfx == 'vulkan': say(f'Vulkan game ({src["graphics"][1]}): vk_shaders.py translates the shaders instead of the NVN shader chain')
|
||||
if gfx == 'gl': say(f'OpenGL game ({src["graphics"][1]}): gl_shaders.py translates the shaders instead of the NVN shader chain')
|
||||
for on, k in ((glslc, 'glslc_capture'), (vkcap, 'vk_capture'), (uedec, 'ue_decode'), (gfx == 'nvn' and src['oodle_packs'][0], 'oodle_packs'), (gfx == 'nvn' and src['ue_legacy'][0], 'ue_legacy')):
|
||||
if on: say(src[k][1])
|
||||
PROG.add(glslc + vkcap + uedec + (gfx == 'nvn'))
|
||||
@@ -1409,6 +1432,8 @@ def main(args=None):
|
||||
if gfx == 'vulkan': # the packed version's merged RomFS holds the SPIR-V files the game loads; a capture holds those it generates
|
||||
cap = capture_vk(repo, game, patched, work, env, logs, g) if vkcap else None
|
||||
ios = build_vk_shaders(py, repo, [game / 'RomFS'] + ([cap] if cap else []), sh, env, logs, g, reuse=not (provisional or derived or ALLOW_DAMAGED))
|
||||
elif gfx == 'gl': # the GLSL sources in the packed version's archives and in its main (the game compiles them at run time; the host finds each by the hash of its text)
|
||||
ios = build_gl_shaders(py, repo, [game / 'RomFS', game / 'ExeFS/main'], sh, env, logs, g, reuse=not (provisional or derived or ALLOW_DAMAGED))
|
||||
else:
|
||||
# Trinity packs and legacy Unreal paks: the split decodes them with the packed version's own decoder (Oodle, LZ5), in the Mac host built for the profile
|
||||
oodle = (lambda: {'DM_OODLE_HOST': str(build_mac_host(repo, work / 'tools/dmhost', env, logs, g)), 'DM_OODLE_MODULES': str(patched)}) if src['oodle_packs'][0] or src['ue_legacy'][0] else None
|
||||
|
||||
Reference in new issue
Block a user