fanlusky
80b1fc42a1
词格原先的整句打分是一份没校准过的启发式 unigram，质量很差。这里内嵌 libime 在用的 kenlm 查询子集，改由它的三元模型 sc.lm 来打路径分。 - engine/ngram 只编 kenlm 的 query / trie-build 路径。不定义 WITH_THREADS， parallel_read.cc 因此退化成串行实现，整个子集零 Boost 依赖。 KENLM_MAX_ORDER=3 必须与建模阶数一致，改它等于读不懂已发布的模型文件。 - ngram::LanguageModel 是唯一 include kenlm 头文件的地方，对外只给出一个纯 std 的头，_HAS_AUTO_PTR_ETC 和 kenlm 的 include 目录都保持 PRIVATE。 - shared_language_model 按资源路径共享一份 mmap。kenlm 的查询是只读的且把 上下文放在调用方的 State 里，所以并发会话不需要额外加锁。 - 模型缺失时 valid() 为假，词格退回原来的启发式打分：整句候选只是变差，不会 停止工作。测试覆盖了缺模型降级与按路径共享。 - 出货的 ARPA 不带 <s>/</s>，所以解码从 null_state() 起算而非 begin_state()。 kenlm 是上游副本，不重排版、不进 clang-format，其 LGPL-2.1+ 与 BSD 例外清单 已补进 THIRD_PARTY_NOTICES.txt。googlepinyinime 那条 Fallback 整句路径不变。
48 lines
2.8 KiB
Markdown
48 lines
2.8 KiB
Markdown
# Runtime assets contract
|
|
|
|
`assets.json` is the shared inventory for the desktop runtime bundle. Each entry declares
|
|
its stable ID, installation path, role, source and profiles. `user` entries name writable
|
|
user files and are deliberately excluded from shipping profiles. Dictionary database
|
|
format/schema compatibility remains owned by `contracts/dictionary/`.
|
|
|
|
- `generate.py` produces `assets.h` for C++ resource lookup and helpcode scheme selection.
|
|
This is the live part: roughly twenty engine translation units include that header.
|
|
- `product.py` reads the inventory for packaging and verifies the exact ZIP member set,
|
|
individual sizes/digests, contract version, dictionary product and source-commit agreement.
|
|
|
|
The builder that produced the `engine-assets-desktop.zip` bundle described below is **not** in
|
|
this repository; it left with the dictionary pipeline (see [UPSTREAM.md](../../UPSTREAM.md)). The
|
|
inventory and its verifier are kept because the header is generated from them and because the
|
|
bundle format is still what the release side is checked against.
|
|
|
|
Run `python3 contracts/assets/generate.py --check` after editing the inventory. CI checks the
|
|
generated header.
|
|
|
|
A consumer of the bundle uses the shared verifier before extraction:
|
|
|
|
```python
|
|
from pathlib import Path
|
|
from contracts.assets.product import verify
|
|
|
|
manifest = verify(Path("engine-assets-desktop.zip"), expected_sha256=product_lock_digest)
|
|
```
|
|
|
|
The expected ZIP digest comes from the platform product lock, not the downloaded manifest.
|
|
Verification is read-only and does not extract into application directories. The returned
|
|
manifest records the producer commit and Google Pinyin source commit. The bundle carries
|
|
source/permission notices without changing the underlying data permissions.
|
|
|
|
Contract version 2 adds `sc.lm`, the kenlm trigram that scores word-lattice sentences, and its
|
|
notice. Because `verify()` compares the manifest's `assets_contract_version` against this
|
|
inventory, a bundle built against version 1 no longer verifies: the release side has to produce
|
|
one that carries `sc.lm` before a host can accept it. The model itself is not committed — the
|
|
`language-model/` source path names the directory where `scripts/build-language-model.ps1`
|
|
converts it from the ARPA corpus pinned in `language-model/lock.json`, which is how the Windows
|
|
installer obtains it. A bundle producer can use the same script, or its own equivalent, as long
|
|
as the bytes match that lock.
|
|
|
|
`mutable-copy` files are immutable release inputs but writable runtime dictionaries after
|
|
installation. `resource` files remain in the resource directory; `user` files remain under
|
|
the user's durable data directory. `engine-assets-manifest.json` is versioned separately
|
|
from the dictionary database format. Older standalone `dict-*` assets remain supported.
|