fanlusky
80b1fc42a1
词格原先的整句打分是一份没校准过的启发式 unigram，质量很差。这里内嵌 libime 在用的 kenlm 查询子集，改由它的三元模型 sc.lm 来打路径分。 - engine/ngram 只编 kenlm 的 query / trie-build 路径。不定义 WITH_THREADS， parallel_read.cc 因此退化成串行实现，整个子集零 Boost 依赖。 KENLM_MAX_ORDER=3 必须与建模阶数一致，改它等于读不懂已发布的模型文件。 - ngram::LanguageModel 是唯一 include kenlm 头文件的地方，对外只给出一个纯 std 的头，_HAS_AUTO_PTR_ETC 和 kenlm 的 include 目录都保持 PRIVATE。 - shared_language_model 按资源路径共享一份 mmap。kenlm 的查询是只读的且把 上下文放在调用方的 State 里，所以并发会话不需要额外加锁。 - 模型缺失时 valid() 为假，词格退回原来的启发式打分：整句候选只是变差，不会 停止工作。测试覆盖了缺模型降级与按路径共享。 - 出货的 ARPA 不带 <s>/</s>，所以解码从 null_state() 起算而非 begin_state()。 kenlm 是上游副本，不重排版、不进 clang-format，其 LGPL-2.1+ 与 BSD 例外清单 已补进 THIRD_PARTY_NOTICES.txt。googlepinyinime 那条 Fallback 整句路径不变。
2.8 KiB
Runtime assets contract
assets.json is the shared inventory for the desktop runtime bundle. Each entry declares
its stable ID, installation path, role, source and profiles. user entries name writable
user files and are deliberately excluded from shipping profiles. Dictionary database
format/schema compatibility remains owned by contracts/dictionary/.
generate.pyproducesassets.hfor C++ resource lookup and helpcode scheme selection. This is the live part: roughly twenty engine translation units include that header.product.pyreads the inventory for packaging and verifies the exact ZIP member set, individual sizes/digests, contract version, dictionary product and source-commit agreement.
The builder that produced the engine-assets-desktop.zip bundle described below is not in
this repository; it left with the dictionary pipeline (see UPSTREAM.md). The
inventory and its verifier are kept because the header is generated from them and because the
bundle format is still what the release side is checked against.
Run python3 contracts/assets/generate.py --check after editing the inventory. CI checks the
generated header.
A consumer of the bundle uses the shared verifier before extraction:
from pathlib import Path
from contracts.assets.product import verify
manifest = verify(Path("engine-assets-desktop.zip"), expected_sha256=product_lock_digest)
The expected ZIP digest comes from the platform product lock, not the downloaded manifest. Verification is read-only and does not extract into application directories. The returned manifest records the producer commit and Google Pinyin source commit. The bundle carries source/permission notices without changing the underlying data permissions.
Contract version 2 adds sc.lm, the kenlm trigram that scores word-lattice sentences, and its
notice. Because verify() compares the manifest's assets_contract_version against this
inventory, a bundle built against version 1 no longer verifies: the release side has to produce
one that carries sc.lm before a host can accept it. The model itself is not committed — the
language-model/ source path names the directory where scripts/build-language-model.ps1
converts it from the ARPA corpus pinned in language-model/lock.json, which is how the Windows
installer obtains it. A bundle producer can use the same script, or its own equivalent, as long
as the bytes match that lock.
mutable-copy files are immutable release inputs but writable runtime dictionaries after
installation. resource files remain in the resource directory; user files remain under
the user's durable data directory. engine-assets-manifest.json is versioned separately
from the dictionary database format. Older standalone dict-* assets remain supported.