fanlusky
e7f7addd4f
- 新增 engine/neural：移植 chinese-ime-lm 的字符级 Transformer 推理，在词格解出的最多 12 条 n-best 里重排挑选整句，desktop / keyboard 两档模型各出一条首选，后台 worker 异步打分 - 词格支持 n-best 输出与重排（quanpin/lattice_rerank），全拼、双拼共用 - 新增 SentenceAssociationOptions，经 QueryRequest 传给各 provider；四个来源默认全关 - 资源契约登记两个 safetensors 与 NOTICE；模型二进制不进仓库，由 scripts/fetch-neural-model.ps1 按 neural-model/lock.json 下载并校验 SHA256 - 新增 neural_sentence 测试
912 B
912 B
chinese-ime-lm sentence models
The two neural sentence models shipped here are from the chinese-ime-lm project
(https://github.com/metasequoiaime/chinese-ime-lm), release model-v1:
sentence-model.safetensors— "keyboard" preset (vocab 8192, 6 layers, n_embd 192, context 64).sentence-model-desktop.safetensors— "desktop" preset (vocab 12288, 8 layers, n_embd 448, context 128).
Both are character-level Transformer language models used to decode pinyin into whole sentences.
License
The model weights are licensed under Apache-2.0 (recorded in each file's
__metadata__["license"]). The C++ inference code in engine/neural/ is a port of the project's
Apache-2.0 reference implementation (reference/).
Attribution
Trained on the Chinese portion of C4 (ODC-BY, https://huggingface.co/datasets/allenai/c4) and LCCC (MIT, https://github.com/thu-coai/CDial-GPT).