fanlusky
e7f7addd4f
- 新增 engine/neural：移植 chinese-ime-lm 的字符级 Transformer 推理，在词格解出的最多 12 条 n-best 里重排挑选整句，desktop / keyboard 两档模型各出一条首选，后台 worker 异步打分 - 词格支持 n-best 输出与重排（quanpin/lattice_rerank），全拼、双拼共用 - 新增 SentenceAssociationOptions，经 QueryRequest 传给各 provider；四个来源默认全关 - 资源契约登记两个 safetensors 与 NOTICE；模型二进制不进仓库，由 scripts/fetch-neural-model.ps1 按 neural-model/lock.json 下载并校验 SHA256 - 新增 neural_sentence 测试
21 lines
912 B
Markdown
21 lines
912 B
Markdown
# chinese-ime-lm sentence models
|
|
|
|
The two neural sentence models shipped here are from the **chinese-ime-lm** project
|
|
(<https://github.com/metasequoiaime/chinese-ime-lm>), release `model-v1`:
|
|
|
|
- `sentence-model.safetensors` — "keyboard" preset (vocab 8192, 6 layers, n_embd 192, context 64).
|
|
- `sentence-model-desktop.safetensors` — "desktop" preset (vocab 12288, 8 layers, n_embd 448, context 128).
|
|
|
|
Both are character-level Transformer language models used to decode pinyin into whole sentences.
|
|
|
|
## License
|
|
|
|
The model **weights** are licensed under **Apache-2.0** (recorded in each file's
|
|
`__metadata__["license"]`). The C++ inference code in `engine/neural/` is a port of the project's
|
|
Apache-2.0 reference implementation (`reference/`).
|
|
|
|
## Attribution
|
|
|
|
Trained on the Chinese portion of C4 (ODC-BY, <https://huggingface.co/datasets/allenai/c4>) and
|
|
LCCC (MIT, <https://github.com/thu-coai/CDial-GPT>).
|