fanlusky
c064cc641a
问题:merge_alternative_segmentations 把首选读音与所有备选读音的候选
混在一起、纯按词典词频排序(注释自陈"跨切分词频不可直接比较",实现
却照排)。于是输入 gau 时,高频的 改/该(gai，邻键，权重13)盖过了
低频的 挂(gua，换位，权重10),尽管 gua 的编辑代价更低、更贴近"交换
字母顺序"的用户意图。
改动:
- AutocorrectCut 增加 edge_count/weight,report k-best 的编辑代价;
same_cost_as() 判定同代价。
- resolve_series_query 按代价把备选拆成两层:与首选同代价的进入词频
竞争(alternative_corrected_cuts,保留 uanli→{quan,cuan,...} 那类
等权消歧),严格更贵的落入 costlier_corrected_cuts。
- query_exact 将更贵读音排在整个首选代价层之后追加(仍查询、去重、
标注 corrected_from),词频不再能把更贵的纠错顶到便宜的前面。
权衡(重要):离线 eval(seed 42,300 样本)显示这是"编辑代价优先"
对"词频优先"的取舍——mixed 模型 R@1/R@3 各档下降约 8~17 个点,
deletion 模型小幅下降 1~3 点。原因是注错样本的原词分布跟随词频,
词频优先在整体召回上更强;本改动以召回换取"最便宜纠错优先"的确定性
排序,是产品取向的选择,经维护者确认后合入。
验证:imetest 全绿(新增 gau 代价分层用例、元音尾守卫用例);
scripts/format.sh --check 通过。
Input engine (engine/)
The C++ conversion engine behind Metasequoia IME for Windows: input sessions, candidate lookup, helpcodes, the shared contracts and the voice module.
This directory used to be a submodule of metasequoiaime/MSIME-Engine. It is now a component of this repository, a sibling of server/, ui/, windows/ and installer/. Its upstream commit, and exactly what the Windows specialization removed, are recorded in UPSTREAM.md. Treat it as first-party code: reviewed, formatted and tested like anything else here, no longer merged from upstream.
| Directory | Purpose |
|---|---|
core/, quanpin/, shuangpin/, english/, japanese/ |
Input sessions and candidate lookup |
contracts/ |
The authoritative IPC wire format, dictionary format, punctuation policy and WebView message schema |
helpcode/ |
Helpcode (形码) tables and generators |
local_modes/, providers/, user_dictionary/ |
Local modes, online candidate sources, the user dictionary |
voice/ |
Recording, WAV encoding, recognition and text cleanup; builds independently |
googlepinyinime-rev/, utfcpp/, voice/third_party/ |
Vendored third-party code, kept in its upstream formatting and licence |
Integration is through <metasequoia/session.h>. See runtime architecture.
Build
The product build does not build this directory on its own: server/CMakeLists.txt pulls it in with add_subdirectory, so CI compiles and exercises the engine through the server and the TSF frontend. After changing something here, run the server's build and tests.
Building it standalone still works and is much faster while working on the engine. Dependencies are CMake 3.25+, a C++17 compiler, Boost, fmt, spdlog and SQLite3.
vcpkg install --triplet x64-windows-static
cmake -S . -B build -DCMAKE_TOOLCHAIN_FILE="$env:VCPKG_INSTALLATION_ROOT/scripts/buildsystems/vcpkg.cmake" -DVCPKG_TARGET_TRIPLET=x64-windows-static
cmake --build build --config Release --parallel
ctest --test-dir build -C Release --output-on-failure --timeout 20
ctest runs every engine target registered in the root CMakeLists.txt: the segmentation, IPC and punctuation contracts, the platform API, data paths, runtime isolation, dictionary state and the per-mode session tests. tests/ additionally holds a separate Windows-only project that is not part of that build — add new tests to the registered targets so they actually run.
Formatting is driven from the repository root by scripts/format.sh, using the .clang-format in this directory (voice/ has its own with a wider column limit). The vendored third-party trees and the generated headers are excluded.
Contract rules
The IPC wire format, opcodes, voice framing and WebView message definitions live in contracts/ and are the single source of truth. Consumers include the same headers; do not restate them on either side. Dictionary naming is defined by contracts/dictionary/format.json, shared with the repository root's scripts/dictionary_product.py.
Data and licensing
The dictionaries are neither stored nor built here. They arrive as a release pinned by tag and digest in product-lock.json and are downloaded at build time; the pipeline that produced them left with the specialization. Per-source licensing is tracked in NOTICE.md — read it before redistributing the built databases.
Recognition goes through a cloud provider; local Whisper inference was removed. The only committed model is voice/assets/models/silero_vad.onnx, loaded solely by the standalone Windows host that is off by default.
Licence
GPL-3.0. Vendored googlepinyinime-rev, utfcpp and voice/third_party/miniaudio keep their upstream licences.