# Upstream provenance This directory was `vendor/MetasequoiaImeEngine`, a submodule of . It is now a component of this repository, vendored in-tree at `engine/` alongside `server/`, `ui/`, `windows/` and `installer/`. | | | | --- | --- | | Upstream repository | `metasequoiaime/MSIME-Engine` | | Imported commit | `c810d201f549b337ae0c4a65a9d694103f1c1754` | | Engine version at import | 0.9.0 (`version.txt`) | That commit is the one `product-lock.json` pinned before the import, so this is a relocation and not an engine upgrade: every source file here was verified against that commit's blob after the move. The engine's own nested submodules were expanded in place at the commits it pinned: | Path | Upstream | Commit | | --- | --- | --- | | `googlepinyinime-rev/` | `metasequoiaime/Google-PinyinIME-Rev` | `12db5237adfcecb79b8ac602d80f3576639ea219` | | `utfcpp/` | `nemtrif/utfcpp` | `2d8e20b22dcb3e9b3c4f52103182ebda949c6089` | | `ngram/kenlm/` | `kpu/kenlm` | `4cb443e60b7bf2c0ddf3c745378f76cb59e254e5` | | `ngram/octagram/darts.h` | `s-yata/darts-clone` | `87b71afd6cf784953e3c08f24c64203397f3b724` (tag `v0.32h`) | | `voice/third_party/miniaudio/` | `mackron/miniaudio` | `9634bedb5b5a2ca38c1ee7108a9358a4e233f14d` | `ngram/octagram/darts.h` is the only file of that copy in the tree — upstream ships it as `include/darts.h` next to `COPYING.md`, both of which are renamed on import to sit beside the `octagram_gram` reader that is ours. It is a header-only double-array trie: `octagram_gram` uses `set_array` to adopt the mmap of a `.gram` file, then `traverse` and `commonPrefixSearch`. None of that is registered with darts-clone upstream, so it is tracked by tag rather than by branch. **This copy is patched, not verbatim.** Replaying an upgrade means re-applying all of these against the new tag: - `DoubleArrayImpl::open()` drops upstream's 256-unit header check — the `size < 256 || (size & 0xFF)` guard, the `units[0]` validation, and the two-stage read that copies those units into `buf` before reading the remainder — and reads the array in a single `fread`. - `DoubleArrayImpl::save()` drops its `offset` parameter and the `fseek` that used it, becoming `save(const char *, const char *, std::size_t) const`. - `commonLongestPrefixSearch` is deleted: both overloads declared on the class and the out-of-line template definition. Only `commonPrefixSearch` and `traverse` are called by `octagram_gram`. - `DoubleArrayBuilder`'s `BLOCK_SIZE` / `NUM_EXTRA_BLOCKS` / `NUM_EXTRAS` change from an unnamed `enum` to `static const std::size_t`. - A `label()` comment typo is fixed (`associted` → `associated`). The `open()` and `save()` rewrites follow from the same decision as the rest of this reader: loading goes through Win32 mmap, so the file-opening and file-writing entry points are dead here. Verify the current state against the tag before assuming any of them still applies. `ngram/kenlm/` did not arrive with the engine import; it was added later, at the commit libime pins for its own `src/libime/core/kenlm` submodule, so the query code and the shipped `sc.lm` come from the same generation of kenlm. Only the query subset was copied: `lm/*.{hh,cc}`, `util/` and `util/double-conversion/`. `lm/builder`, `lm/filter`, `lm/interpolate` and `lm/wrappers` are not here, and no file that is here includes them. The sources are unmodified except for the one local patch below; the two Windows adaptations it needs — a wide-character `util::_open` and the `_HAS_AUTO_PTR_ETC` definition — live outside the subtree, in `ngram/kenlm_file_io.h` and `ngram/CMakeLists.txt`. **Local patch:** `util/string_stream.hh` `StringStream::AdvanceTo` — `&*out_.end()` and `&*out_.begin()` dereference string iterators, which the MSVC checked iterators reject (a "Debug Error!" assert under any `_DEBUG` build, on every formatted kenlm message). `out_.data() + out_.size()` computes the same past-the-end pointer without touching iterators; release behaviour is unchanged because the asserts were compiled out there anyway. Two of those files, `googlepinyinime-rev/src/{include/userdict.h,share/userdict.cpp}`, are committed upstream with CRLF and arrive here with LF. That is this repository's `.gitattributes` (`* text=auto eol=lf`) doing what it does to every other file; the content is unchanged. `googlepinyinime-rev/` is no longer tracked against its upstream and is edited here directly, in its original formatting and under its original licence. Local changes so far: a per-syllable Hanzi constraint for the shuangpin mid-sentence helpcode — `MatrixSearch::set_char_constraints`, `MatrixSearch::filter_lpis_by_constraints` (called from `add_char_qwerty` before `extend_mtrx_nd`), `im_set_char_constraints` and the `CharConstraintFn` typedef in `dictdef.h`; and in the multi-syllable branch of `MatrixSearch::get_lpis`, items equal to the full sentence candidate are dropped before deduplication, so a better-scored duplicate of a dropped item no longer writes `lma_buf[-1]` (crashed the Server on shuangpin `zjle`, i.e. `zan'l`). ## Why this file and not `product-lock.json` The engine used to be pinned by a gitlink, and `product-lock.json` recorded the commit so CI could check that the checked-out submodule matched the reviewed one. There is no gitlink any more, so there is nothing left for CI to compare against: the first Windows-specific fix to `engine/` makes "this tree is upstream commit X" false, and nothing would notice. Recording the provenance in prose says what is true — where the code came from — without claiming an equality that no longer holds. Treat `engine/` as first-party code from here on. Changes to it are reviewed, formatted and tested like any other directory of this repository (`scripts/format.sh` covers it), not merged from upstream. Upstream is history, not a remote to track. ## What was removed during the Windows specialization Everything below was deleted from the imported tree. No engine source file was otherwise modified. `voice/CMakeLists.txt` lost the option blocks belonging to the deleted sources; the rest of the changes are documentation (`README*.md`, `AGENTS.md`, `NOTICE.md`, the `contracts/` and `docs/` prose) rewritten to describe a directory rather than a repository. **Non-Windows platform code** - `voice/third_party/whisper.cpp/` and the whisper provider (worker, header, test) — roughly 36 MB of local-inference ASR. This product's voice input uses cloud ASR; nothing here built or shipped it. - `voice/examples/macos/` — a standalone macOS host application. Portable `if(APPLE)` / `elseif(WIN32)` / `else()` guards inside the surviving sources were left alone. Stripping them would be a large, untested rewrite of working code for no build-output gain. **The dictionary build pipeline** - `dictionary/` in full — about 228 MB of raw corpora (`cn`, `en`, `source`, `makecikudb`) and the scripts that turn them into a dictionary release. No CMake target in this repository consumes them, and the workflow that built them lived in the engine's own CI. The built dictionaries are still fetched at build time from their release, as `product-lock.json` records. - `build_assets.py` and `build_profile.py`, which import `dictionary.build_profile`. `contracts/assets/` was kept: `assets.h` is a generated header included by roughly twenty engine translation units, and it is unrelated to the corpora despite the name. **Upstream governance and tooling** - `.github/` (the engine's own CI), `.vscode/`, `.gitmodules`, `.mailmap`, `.git-blame-ignore-revs`, `CHANGELOG.md`, `release-please-config.json`, `.release-please-manifest.json` — all of these describe a standalone repository with its own release train, which this directory no longer is. - `scripts/format.sh` — this repository formats the engine from `scripts/format.sh` at the root, so a second entry point with its own exclusion list would only be a way for the two to disagree. The exclusions it carried were folded into the root script. - Unused portions of the vendored `miniaudio`, `googlepinyinime-rev` and `utfcpp` copies. What remains of those three is still third-party and still under its own licence; see `NOTICE.md`.