fanlusky
80b1fc42a1
词格原先的整句打分是一份没校准过的启发式 unigram，质量很差。这里内嵌 libime 在用的 kenlm 查询子集，改由它的三元模型 sc.lm 来打路径分。 - engine/ngram 只编 kenlm 的 query / trie-build 路径。不定义 WITH_THREADS， parallel_read.cc 因此退化成串行实现，整个子集零 Boost 依赖。 KENLM_MAX_ORDER=3 必须与建模阶数一致，改它等于读不懂已发布的模型文件。 - ngram::LanguageModel 是唯一 include kenlm 头文件的地方，对外只给出一个纯 std 的头，_HAS_AUTO_PTR_ETC 和 kenlm 的 include 目录都保持 PRIVATE。 - shared_language_model 按资源路径共享一份 mmap。kenlm 的查询是只读的且把 上下文放在调用方的 State 里，所以并发会话不需要额外加锁。 - 模型缺失时 valid() 为假，词格退回原来的启发式打分：整句候选只是变差，不会 停止工作。测试覆盖了缺模型降级与按路径共享。 - 出货的 ARPA 不带 <s>/</s>，所以解码从 null_state() 起算而非 begin_state()。 kenlm 是上游副本，不重排版、不进 clang-format，其 LGPL-2.1+ 与 BSD 例外清单 已补进 THIRD_PARTY_NOTICES.txt。googlepinyinime 那条 Fallback 整句路径不变。
192 lines
12 KiB
Markdown
192 lines
12 KiB
Markdown
# Runtime architecture and platform API
|
||
|
||
Engine owns portable input behavior. Platform hosts own native keys, UI, permissions,
|
||
network transport and text insertion. The supported entry point for new platform code is
|
||
`<metasequoia/session.h>`; link `MetasequoiaIme::Engine`. It exposes options, editing intents,
|
||
value snapshots and asynchronous candidate requests, without exposing SQLite or providers.
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
Host[Platform host] --> Session[Session public facade]
|
||
Session --> Input[InputSession composition policy]
|
||
Input --> Punctuation[PunctuationPolicy]
|
||
Input --> Queries[CandidateQueries: local and mixed candidates]
|
||
Input --> Online[OnlineRequestGuard]
|
||
Input --> Ime[ImeSession]
|
||
Ime --> Scheme[Input scheme -> QueryRequest]
|
||
Ime --> Providers[Candidate providers]
|
||
Providers --> Dictionaries[User working dictionaries]
|
||
Providers --> Decoder[PinyinDecoder serialized service]
|
||
Providers --> Japanese[Immutable Japanese model by resource path]
|
||
```
|
||
|
||
`InputSession` remains the compatibility/advanced interface for existing hosts. Its raw
|
||
sequence setters and engine-key methods are not part of the new `Session` API. Existing
|
||
frontends can migrate independently after the producer commit merges; this change does not
|
||
move their locks to an unmerged producer. Do not expand the compatibility interface for
|
||
new platform features when they can use editing intents and snapshots.
|
||
|
||
Desktop hosts may call `finish(highlighted_index)` to commit the highlighted first candidate
|
||
and all remaining segments through Engine's existing composition policy. `finish()` retains
|
||
the leading-candidate behavior. Snapshots include `dedicated_english` even with empty preedit,
|
||
so hosts need not maintain a second mode flag. `set_helpcode_enabled` applies to both pinyin
|
||
schemes; a host with separate persisted preferences applies the selected flag on scheme change.
|
||
|
||
For the Microsoft shuangpin profile, `character(';')` routes the ing final through the
|
||
decoder's existing position validation. Other profiles leave this character unhandled.
|
||
|
||
`MoveLeft`, `MoveRight`, `MoveHome`, `MoveEnd` and `DeleteForward` are composition
|
||
commands. `character` inserts at the session-owned caret; `Backspace` erases before it.
|
||
Snapshots append `editing_text` (the ASCII source input, preserving case) and
|
||
`caret_position` (its byte offset). These are distinct from rendered preedit, notably
|
||
for Japanese romaji conversion. Hosts render the snapshot instead of rewriting raw input.
|
||
Movement alone preserves candidate/request identity; mutations invalidate online requests
|
||
even when subsequent edits restore identical spelling. Selection places the caret at the
|
||
end of any unconsumed suffix; cancel and scheme changes reset it. Local-mode markers
|
||
remain protected from middle editing; backspace on a marker-only composition still cancels.
|
||
Empty-composition movement is unhandled so normal host navigation can proceed.
|
||
|
||
`pin(index)` promotes a currently visible dictionary candidate without selecting it or changing
|
||
preedit. It persists through the session's user journal and working dictionaries, then refreshes
|
||
the snapshot. Explicit pinning works even when automatic learning is disabled. Pinyin (including
|
||
shuangpin canonical keys), Wubi and English dictionary candidates use their existing ranking
|
||
policy; generated, expressive, online and Japanese candidates are unhandled. English pinning
|
||
changes dictionary order within the existing mixed/temporary-mode layout, so it does not move
|
||
an English completion ahead of the temporary mode's raw-text entry. Invalid indices are unhandled;
|
||
a failed write returns a handled result with a diagnostic and no commit. Hosts serialize this
|
||
index action against the snapshot they display, just as they do for `select(index)`.
|
||
|
||
`remove(index)` deletes a visible dictionary phrase without committing text or changing preedit.
|
||
It uses the exact canonical pinyin key (or Wubi/English key), writes a deletion journal entry,
|
||
and refreshes candidates. Automatic learning may be disabled. Single-character non-English
|
||
candidates and non-dictionary sources are protected and return unhandled, as do invalid indices.
|
||
A database or journal failure returns a diagnostic with no commit. The working-dictionary delete
|
||
and journal entry share an attached SQLite transaction, so a statement/commit failure rolls back
|
||
both; this does not promise multi-file power-loss atomicity when SQLite uses WAL. Existing
|
||
quanpin/shuangpin and English deletion callers share this persistence helper. Hosts serialize
|
||
candidate-index actions against their displayed snapshot.
|
||
|
||
## Resources and user state
|
||
|
||
`fix_position(index, slot)` and `clear_position(index)` manage slots 1–5 for dictionary
|
||
candidates in the current ranking context. They reuse the existing fixed-position table
|
||
in the session's user journal, not a new resource format. Successful actions refresh the
|
||
snapshot without committing or changing preedit; persistence failure returns a diagnostic.
|
||
Unsupported sources and invalid indices/slots are unhandled. Fixed positions are applied
|
||
during candidate refresh, so snapshot reads perform no extra database work. The public
|
||
facade enables this ordering; compatibility InputSession callers retain their host-side
|
||
ordering until migration. Pinyin, Wubi, super-jianpin and English contexts use the captured
|
||
user layout, including temporary and mixed English candidates.
|
||
|
||
The host first verifies the downloaded bundle using `contracts/assets/product.py` and its
|
||
pinned ZIP digest, extracts it into an immutable resource directory, and supplies that
|
||
content ID to `prepare_runtime_paths`. All directory paths must be absolute. For this
|
||
generation-preparation API, resources, user data and cache must be disjoint directory trees:
|
||
none may equal or contain another, including through resolved symlinks. Invalid roots are
|
||
rejected before creating directories or copying databases. The legacy layout remains available
|
||
through `RuntimePaths::legacy()` for consumers that have not yet migrated.
|
||
|
||
```cpp
|
||
#include <metasequoia/session.h>
|
||
|
||
// Quiesce sessions and user-data writers while preparing/switching generations.
|
||
auto paths = metasequoia::prepare_runtime_paths(
|
||
resource_directory, user_directory, cache_directory, verified_bundle_digest);
|
||
metasequoia::SessionOptions options;
|
||
options.paths = paths;
|
||
options.helpcode_schema = "xiaohe";
|
||
metasequoia::Session session(options);
|
||
|
||
auto result = session.character('n');
|
||
// The host inserts result.commit when present, then renders this value snapshot.
|
||
auto view = session.snapshot();
|
||
```
|
||
|
||
The directory ownership is explicit:
|
||
|
||
| Directory | Contents and lifecycle |
|
||
| --- | --- |
|
||
| `resources` | Verified immutable DB inputs, models, helpcodes, translation sidecar, notices |
|
||
| `user_data` | Durable `msime_user.db`, decoder user dictionary, and working DB generations |
|
||
| `user_data/dictionaries/<content ID>` | Writable `msime.db`/`english.db` copies used by learning |
|
||
| `cache` | Disposable host cache; deleting it must not delete user dictionaries |
|
||
|
||
Preparation copies both desktop DBs using SQLite backup (including committed WAL data),
|
||
replays the existing journal inside the existing replay transaction, then publishes the new
|
||
generation with one directory rename. Failed preparation removes only its staging directory
|
||
and does not switch the host or replace an old generation. Returning to an existing
|
||
prepared generation replays the current journal so newer learning survives a rollback.
|
||
The host switches only after preparation succeeds, and retains generations for its rollback
|
||
policy. An existing `.incoming` directory is reported rather than silently deleted; after
|
||
confirming no preparation is active, the host may remove that interrupted staging directory.
|
||
|
||
Preparation currently targets the full desktop dictionary profile. Other product layouts
|
||
may supply explicit `RuntimePaths`; they must provide durable writable DBs and preserve
|
||
journal replay themselves. `RuntimePaths::legacy()` captures the old environment/default
|
||
layout once for compatibility. Existing sessions are never redirected by later environment
|
||
changes. Directories and models must not be replaced in place while a session uses them.
|
||
|
||
Translation overlays are loaded from `resources` even when `english.db` is a writable user
|
||
copy. File names and helpcode scheme mappings come from generated `contracts/assets/assets.h`.
|
||
|
||
## Isolation and lifetime
|
||
|
||
A host serializes calls on each session. Different sessions may run concurrently and can
|
||
use different paths, profiles and helpcode schemes. Profiles are owned values, helpcode
|
||
maps are immutable per-session values, and the Japanese model cache is keyed by resource
|
||
path with weak ownership. Unused or failed models are not held forever by a global future.
|
||
`set_helpcode_schema()` affects only that session. Legacy static `select_helpcode_schema()`
|
||
changes the default for subsequently created sessions; it no longer changes live sessions.
|
||
|
||
Google Pinyin internally uses global search, spelling, language-model and lemma-cache state.
|
||
`PinyinDecoder` is the sole caller of its C API. One mutex covers model/user selection,
|
||
model-specific cache reset, search reset, search and copying the resulting text. Construction
|
||
of another dictionary no longer destroys a live decoder. Changing model/user paths closes
|
||
and reloads the service; the last decoder client closes its resources. Calls never retain
|
||
upstream candidate pointers or incremental search state. This makes independent sessions
|
||
safe while serializing their fallback decode work. Windows decoder file IO uses namespace
|
||
adapters for UTF-8 paths without modifying the vendored decoder or the process locale.
|
||
|
||
The dictionary word lattice is the other whole-sentence source, and it is scored by a kenlm
|
||
trigram rather than by Google Pinyin. `ngram::LanguageModel` (engine/ngram) wraps the vendored
|
||
kenlm query subset; `ngram::shared_language_model` hands out one mmapped instance per resource
|
||
path, so the quanpin and shuangpin dictionaries in every session share a single 34 MB mapping.
|
||
kenlm queries are const and carry their context in the caller's `ngram::State`, so concurrent
|
||
sessions need no lock around it. A missing or unreadable `sc.lm` yields a `valid() == false`
|
||
model and the lattice falls back to its previous uncalibrated unigram heuristic: sentences get
|
||
worse, nothing stops working. The shipped ARPA has no `<s>`/`</s>`, so decoding starts from
|
||
`null_state()`, not `begin_state()`.
|
||
|
||
User journal operations own their SQLite connections, so separate calls cannot accidentally
|
||
join another session's transaction through a process-wide connection. SQLite locking still
|
||
coordinates writes to the same user's databases. Resource upgrades must quiesce writers.
|
||
|
||
Online requests contain both a session ID and a composition generation. Hosts copy the
|
||
entire request unchanged through their asynchronous transport. A response from another
|
||
session is rejected even if the two sessions contain identical input.
|
||
|
||
## Verification
|
||
|
||
The root CTest targets include public-header compilation, two independent resource/user
|
||
layouts, per-session helpcodes and punctuation, local query paths, concurrent sessions,
|
||
Japanese model path isolation, real Google Pinyin model interleaving/concurrency,
|
||
per-path sharing and missing-model degradation of the kenlm trigram, Unicode
|
||
resource paths, user phrase replay, return to a previous generation and failed replay
|
||
preserving old working data. Existing candidate, learning, temporary-mode and online tests
|
||
continue to exercise the compatibility layer and extracted components.
|
||
|
||
```bash
|
||
python3 contracts/assets/generate.py --check
|
||
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
|
||
cmake --build build --parallel
|
||
ctest --test-dir build --output-on-failure --timeout 20
|
||
```
|
||
|
||
Those commands build the engine on its own, which is useful while working on it. The product build
|
||
does not run them: `server/CMakeLists.txt` adds this directory as a subdirectory, so CI exercises
|
||
the engine through the server and the TSF frontend instead.
|
||
|
||
The dictionary producer used to run a further consumer test against its complete produced archive.
|
||
That pipeline is not part of this repository; the dictionaries arrive as a release, pinned by
|
||
`product-lock.json`. See [../UPSTREAM.md](../UPSTREAM.md).
|