fanlusky
80b1fc42a1
词格原先的整句打分是一份没校准过的启发式 unigram，质量很差。这里内嵌 libime 在用的 kenlm 查询子集，改由它的三元模型 sc.lm 来打路径分。 - engine/ngram 只编 kenlm 的 query / trie-build 路径。不定义 WITH_THREADS， parallel_read.cc 因此退化成串行实现，整个子集零 Boost 依赖。 KENLM_MAX_ORDER=3 必须与建模阶数一致，改它等于读不懂已发布的模型文件。 - ngram::LanguageModel 是唯一 include kenlm 头文件的地方，对外只给出一个纯 std 的头，_HAS_AUTO_PTR_ETC 和 kenlm 的 include 目录都保持 PRIVATE。 - shared_language_model 按资源路径共享一份 mmap。kenlm 的查询是只读的且把 上下文放在调用方的 State 里，所以并发会话不需要额外加锁。 - 模型缺失时 valid() 为假，词格退回原来的启发式打分：整句候选只是变差，不会 停止工作。测试覆盖了缺模型降级与按路径共享。 - 出货的 ARPA 不带 <s>/</s>，所以解码从 null_state() 起算而非 begin_state()。 kenlm 是上游副本，不重排版、不进 clang-format，其 LGPL-2.1+ 与 BSD 例外清单 已补进 THIRD_PARTY_NOTICES.txt。googlepinyinime 那条 Fallback 整句路径不变。
5.1 KiB
Upstream provenance
This directory was vendor/MetasequoiaImeEngine, a submodule of
https://github.com/metasequoiaime/MSIME-Engine. It is now a component of this repository, vendored
in-tree at engine/ alongside server/, ui/, windows/ and installer/.
| Upstream repository | metasequoiaime/MSIME-Engine |
| Imported commit | c810d201f549b337ae0c4a65a9d694103f1c1754 |
| Engine version at import | 0.9.0 (version.txt) |
That commit is the one product-lock.json pinned before the import, so this is a relocation and not
an engine upgrade: every source file here was verified against that commit's blob after the move.
The engine's own nested submodules were expanded in place at the commits it pinned:
| Path | Upstream | Commit |
|---|---|---|
googlepinyinime-rev/ |
metasequoiaime/Google-PinyinIME-Rev |
12db5237adfcecb79b8ac602d80f3576639ea219 |
utfcpp/ |
nemtrif/utfcpp |
2d8e20b22dcb3e9b3c4f52103182ebda949c6089 |
ngram/kenlm/ |
kpu/kenlm |
4cb443e60b7bf2c0ddf3c745378f76cb59e254e5 |
voice/third_party/miniaudio/ |
mackron/miniaudio |
9634bedb5b5a2ca38c1ee7108a9358a4e233f14d |
ngram/kenlm/ did not arrive with the engine import; it was added later, at the commit libime pins
for its own src/libime/core/kenlm submodule, so the query code and the shipped sc.lm come from
the same generation of kenlm. Only the query subset was copied: lm/*.{hh,cc}, util/ and
util/double-conversion/. lm/builder, lm/filter, lm/interpolate and lm/wrappers are not
here, and no file that is here includes them. The sources are unmodified; the two Windows
adaptations it needs — a wide-character util::_open and the _HAS_AUTO_PTR_ETC definition — live
outside the subtree, in ngram/kenlm_file_io.h and ngram/CMakeLists.txt.
Two of those files, googlepinyinime-rev/src/{include/userdict.h,share/userdict.cpp}, are committed
upstream with CRLF and arrive here with LF. That is this repository's .gitattributes (* text=auto eol=lf) doing what it does to every other file; the content is unchanged.
Why this file and not product-lock.json
The engine used to be pinned by a gitlink, and product-lock.json recorded the commit so CI could
check that the checked-out submodule matched the reviewed one. There is no gitlink any more, so
there is nothing left for CI to compare against: the first Windows-specific fix to engine/ makes
"this tree is upstream commit X" false, and nothing would notice. Recording the provenance in prose
says what is true — where the code came from — without claiming an equality that no longer holds.
Treat engine/ as first-party code from here on. Changes to it are reviewed, formatted and tested
like any other directory of this repository (scripts/format.sh covers it), not merged from
upstream. Upstream is history, not a remote to track.
What was removed during the Windows specialization
Everything below was deleted from the imported tree. No engine source file was otherwise modified.
voice/CMakeLists.txt lost the option blocks belonging to the deleted sources; the rest of the
changes are documentation (README*.md, AGENTS.md, NOTICE.md, the contracts/ and docs/
prose) rewritten to describe a directory rather than a repository.
Non-Windows platform code
voice/third_party/whisper.cpp/and the whisper provider (worker, header, test) — roughly 36 MB of local-inference ASR. This product's voice input uses cloud ASR; nothing here built or shipped it.voice/examples/macos/— a standalone macOS host application.
Portable if(APPLE) / elseif(WIN32) / else() guards inside the surviving sources were left
alone. Stripping them would be a large, untested rewrite of working code for no build-output gain.
The dictionary build pipeline
dictionary/in full — about 228 MB of raw corpora (cn,en,source,makecikudb) and the scripts that turn them into a dictionary release. No CMake target in this repository consumes them, and the workflow that built them lived in the engine's own CI. The built dictionaries are still fetched at build time from their release, asproduct-lock.jsonrecords.build_assets.pyandbuild_profile.py, which importdictionary.build_profile.
contracts/assets/ was kept: assets.h is a generated header included by roughly twenty engine
translation units, and it is unrelated to the corpora despite the name.
Upstream governance and tooling
.github/(the engine's own CI),.vscode/,.gitmodules,.mailmap,.git-blame-ignore-revs,CHANGELOG.md,release-please-config.json,.release-please-manifest.json— all of these describe a standalone repository with its own release train, which this directory no longer is.scripts/format.sh— this repository formats the engine fromscripts/format.shat the root, so a second entry point with its own exclusion list would only be a way for the two to disagree. The exclusions it carried were folded into the root script.- Unused portions of the vendored
miniaudio,googlepinyinime-revandutfcppcopies. What remains of those three is still third-party and still under its own licence; seeNOTICE.md.