msime-windows/engine/UPSTREAM.md

# Upstream provenance

This directory was `vendor/MetasequoiaImeEngine`, a submodule of
<https://github.com/metasequoiaime/MSIME-Engine>. It is now a component of this repository, vendored
in-tree at `engine/` alongside `server/`, `ui/`, `windows/` and `installer/`.

| | |
| --- | --- |
| Upstream repository | `metasequoiaime/MSIME-Engine` |
| Imported commit | `c810d201f549b337ae0c4a65a9d694103f1c1754` |
| Engine version at import | 0.9.0 (`version.txt`) |

That commit is the one `product-lock.json` pinned before the import, so this is a relocation and not
an engine upgrade: every source file here was verified against that commit's blob after the move.

The engine's own nested submodules were expanded in place at the commits it pinned:

| Path | Upstream | Commit |
| --- | --- | --- |
| `googlepinyinime-rev/` | `metasequoiaime/Google-PinyinIME-Rev` | `12db5237adfcecb79b8ac602d80f3576639ea219` |
| `utfcpp/` | `nemtrif/utfcpp` | `2d8e20b22dcb3e9b3c4f52103182ebda949c6089` |
| `ngram/kenlm/` | `kpu/kenlm` | `4cb443e60b7bf2c0ddf3c745378f76cb59e254e5` |
| `ngram/octagram/darts.h` | `s-yata/darts-clone` | `87b71afd6cf784953e3c08f24c64203397f3b724` (tag `v0.32h`) |
| `voice/third_party/miniaudio/` | `mackron/miniaudio` | `9634bedb5b5a2ca38c1ee7108a9358a4e233f14d` |

`ngram/octagram/darts.h` is the only file of that copy in the tree — upstream ships it as
`include/darts.h` next to `COPYING.md`, both of which are renamed on import to sit beside the
`octagram_gram` reader that is ours. It is a header-only double-array trie: `octagram_gram` uses
`set_array` to adopt the mmap of a `.gram` file, then `traverse` and `commonPrefixSearch`. None of
that is registered with darts-clone upstream, so it is tracked by tag rather than by branch.

**This copy is patched, not verbatim.** Replaying an upgrade means re-applying all of these against
the new tag:

- `DoubleArrayImpl::open()` drops upstream's 256-unit header check — the `size < 256 || (size & 0xFF)`
  guard, the `units[0]` validation, and the two-stage read that copies those units into `buf` before
  reading the remainder — and reads the array in a single `fread`.
- `DoubleArrayImpl::save()` drops its `offset` parameter and the `fseek` that used it, becoming
  `save(const char *, const char *, std::size_t) const`.
- `commonLongestPrefixSearch` is deleted: both overloads declared on the class and the out-of-line
  template definition. Only `commonPrefixSearch` and `traverse` are called by `octagram_gram`.
- `DoubleArrayBuilder`'s `BLOCK_SIZE` / `NUM_EXTRA_BLOCKS` / `NUM_EXTRAS` change from an unnamed `enum`
  to `static const std::size_t`.
- A `label()` comment typo is fixed (`associted` → `associated`).

The `open()` and `save()` rewrites follow from the same decision as the rest of this reader: loading
goes through Win32 mmap, so the file-opening and file-writing entry points are dead here. Verify the
current state against the tag before assuming any of them still applies.

`ngram/kenlm/` did not arrive with the engine import; it was added later, at the commit libime pins
for its own `src/libime/core/kenlm` submodule, so the query code and the shipped `sc.lm` come from
the same generation of kenlm. Only the query subset was copied: `lm/*.{hh,cc}`, `util/` and
`util/double-conversion/`. `lm/builder`, `lm/filter`, `lm/interpolate` and `lm/wrappers` are not
here, and no file that is here includes them. The sources are unmodified except for the one local
patch below; the two Windows adaptations it needs — a wide-character `util::_open` and the
`_HAS_AUTO_PTR_ETC` definition — live outside the subtree, in `ngram/kenlm_file_io.h` and
`ngram/CMakeLists.txt`.

**Local patch:** `util/string_stream.hh` `StringStream::AdvanceTo` — `&*out_.end()` and
`&*out_.begin()` dereference string iterators, which the MSVC checked iterators reject (a
"Debug Error!" assert under any `_DEBUG` build, on every formatted kenlm message). `out_.data() +
out_.size()` computes the same past-the-end pointer without touching iterators; release behaviour
is unchanged because the asserts were compiled out there anyway.

Two of those files, `googlepinyinime-rev/src/{include/userdict.h,share/userdict.cpp}`, are committed
upstream with CRLF and arrive here with LF. That is this repository's `.gitattributes` (`* text=auto
eol=lf`) doing what it does to every other file; the content is unchanged.

`googlepinyinime-rev/` is no longer tracked against its upstream and is edited here directly, in its
original formatting and under its original licence. Local changes so far: a per-syllable Hanzi
constraint for the shuangpin mid-sentence helpcode — `MatrixSearch::set_char_constraints`,
`MatrixSearch::filter_lpis_by_constraints` (called from `add_char_qwerty` before
`extend_mtrx_nd`), `im_set_char_constraints` and the `CharConstraintFn` typedef in `dictdef.h`;
and in the multi-syllable branch of `MatrixSearch::get_lpis`, items equal to the full sentence
candidate are dropped before deduplication, so a better-scored duplicate of a dropped item no longer
writes `lma_buf[-1]` (crashed the Server on shuangpin `zjle`, i.e. `zan'l`).

## Why this file and not `product-lock.json`

The engine used to be pinned by a gitlink, and `product-lock.json` recorded the commit so CI could
check that the checked-out submodule matched the reviewed one. There is no gitlink any more, so
there is nothing left for CI to compare against: the first Windows-specific fix to `engine/` makes
"this tree is upstream commit X" false, and nothing would notice. Recording the provenance in prose
says what is true — where the code came from — without claiming an equality that no longer holds.

Treat `engine/` as first-party code from here on. Changes to it are reviewed, formatted and tested
like any other directory of this repository (`scripts/format.sh` covers it), not merged from
upstream. Upstream is history, not a remote to track.

## What was removed during the Windows specialization

Everything below was deleted from the imported tree. No engine source file was otherwise modified.
`voice/CMakeLists.txt` lost the option blocks belonging to the deleted sources; the rest of the
changes are documentation (`README*.md`, `AGENTS.md`, `NOTICE.md`, the `contracts/` and `docs/`
prose) rewritten to describe a directory rather than a repository.

**Non-Windows platform code**

- `voice/third_party/whisper.cpp/` and the whisper provider (worker, header, test) — roughly 36 MB of
  local-inference ASR. This product's voice input uses cloud ASR; nothing here built or shipped it.
- `voice/examples/macos/` — a standalone macOS host application.

Portable `if(APPLE)` / `elseif(WIN32)` / `else()` guards inside the surviving sources were left
alone. Stripping them would be a large, untested rewrite of working code for no build-output gain.

**The dictionary build pipeline**

- `dictionary/` in full — about 228 MB of raw corpora (`cn`, `en`, `source`, `makecikudb`) and the
  scripts that turn them into a dictionary release. No CMake target in this repository consumes
  them, and the workflow that built them lived in the engine's own CI. The built dictionaries are
  still fetched at build time from their release, as `product-lock.json` records.
- `build_assets.py` and `build_profile.py`, which import `dictionary.build_profile`.

`contracts/assets/` was kept: `assets.h` is a generated header included by roughly twenty engine
translation units, and it is unrelated to the corpora despite the name.

**Upstream governance and tooling**

- `.github/` (the engine's own CI), `.vscode/`, `.gitmodules`, `.mailmap`,
  `.git-blame-ignore-revs`, `CHANGELOG.md`, `release-please-config.json`,
  `.release-please-manifest.json` — all of these describe a standalone repository with its own
  release train, which this directory no longer is.
- `scripts/format.sh` — this repository formats the engine from `scripts/format.sh` at the root, so
  a second entry point with its own exclusion list would only be a way for the two to disagree. The
  exclusions it carried were folded into the root script.
- Unused portions of the vendored `miniaudio`, `googlepinyinime-rev` and `utfcpp` copies. What
  remains of those three is still third-party and still under its own licence; see `NOTICE.md`.