Commit Graph
61 Commits
Author SHA1 Message Date
Jia Suiandfanlusky 60c2b26698 fix(engine,server): 全拼纠错收窄「多字」纠错并让开关真正关闭纠错 (#889)
* fix(engine,server): 全拼多字纠错收窄为仅 QWERTY 邻键插入

多字(插入)纠错只保留「误插 QWERTY 邻键」:生成器 insertion_variants 不再生成重复相邻键(双击),生成式空间不再生成任意字母插入(远键插入)。autocorrect_table.h 随生成器重新生成。

* fix(engine): 全拼纠错开关关闭时真正不再改写切分

纠错开关关掉后真正不纠错:把手误拼写别名层从常开改为跟掩码走(ü 归一 jv->ju / nue->nve 仍常开);两个早于掩码可知的切分入口 QuanpinScheme::build_request 与 QuanpinDictionary::get_or_compute_segments 显式带上掩码,切分缓存键随之隔离。

* fix(engine,server): 纠错开关关闭时辅助码、写回与排序上下文也按字面切分

关掉全拼纠错后,只有方案的 build_request 和查询自身切分不再跑手误别名层,
其余几条路径仍用默认(带别名)的切分,和查询落在不同的键上:

- 辅助码路径自己切底串作为显式切分交给词典,haoyonga 仍被切成 hao'yong'ga;
- 调频、造词、删词、建 SQL、校验去掉分隔符后重切,开关关掉时选中 hao'yong'a
  的词会去更新 hao'yong'ga,调频静默失败;
- 排序 / 固定位置的上下文 key(引擎与 server 两处)仍按别名切,laing 与 liang
  的固定位置互相串。

改动:
- cut_pinyin_by_mode 的 include_correction_aliases 改为必填,每个调用点显式决定;
  双拼词典、设置页词库校验和离线工具传 true,保持原行为。
- 新增 quanpin::correction_aliases_enabled,方案、词典、辅助码与排序 key 共用
  (只看换位、邻键两位),修正只带删除 / 插入位的掩码在方案与词典两层判断不一致。
- 词典按最近一次 query 的掩码记录别名开关,写回路径统一走 cut_for_write。
- 会话改为在换方案、改开关时 bind_quanpin_scheme 一次,不再每次 refresh 下发。
- 方案只存 bool;jv -> ju 放回纠错表,去掉合并表。
- 在 CI 实际执行的 test_input_session 中补回归用例(写回、辅助码、固定位置、
  方案掩码),每条都确认回退修复后会失败。

---------

Co-authored-by: fanlusky <1730976608@qq.com>
2026-10-12 20:41:23 +08:00
Jia Sui 3c2f3ae374 feat(engine): 全拼纠错补齐漏字/多字,k-best 切分按词频消解歧义
此前纠错只有交换(sahng -> shang)与 QWERTY 邻键(shabg -> shang)两类,
生成器在离线阶段就把「一个错串对应多个音节」的变体整个丢掉,运行时也只
取代价最低的一条切分。三类输入因此没有出路:漏字的(shng)、多敲一键的
(shangg、sjhang)、漏声母连基础切分都切不出来的(uanli 想打 quan'li)。

本次按专利 CN 101133411 B 的思路重排整条链路:错串可能对应的读音全部
保留到运行时,先由 k-best 切分带出平行读音,再让词典词频决定谁排前面。

- 纠错类型从两类扩到四类:新增漏字(shng -> shang)与多字(shangg 双打、
  sjhang 邻键 -> shang)。两个新位跟随既有 autocorrect_transposition /
  autocorrect_neighbor 开关联动(任一打开即生效),定义保持独立,留待
  将来的独立配置键。查询与 preedit 重建共用同一映射,两侧纠正空间一致。
- 生成器从两表扩到四表并保留歧义变体(ahan -> {shan, zhan});同一条
  (错串, 正确) 对被两条规则产出时在两张表各留一份,保证每个类型开关
  单独有效。正确音节收敛进 kCorrectSyllables 单表、行内改存 16-bit
  下标,一万七千多行不再各自重复音节字符串,--check 拦截过期表头。
- 切分层新增 autocorrect_cut_kbest:按位置自左向右做 top-k 假设传播,
  排序键是(纠错边数,边权之和,生成序)——最少纠错优先的契约不变,
  权重只在同边数内打破平局,引擎绝不用一条额外纠错换更低权重。边权取
  专利 [0052] 允许的估常量(交换 10 / 漏字 11 / 多字 12 / 邻键 13),
  标定路径是挖掘用户日志。漏字/多字边不等长,切分段改按 raw 长度推进
  位置,preedit 分隔跟着原始字母的区间走。
- 词典层固定 k=9,为离线评测定值:漏声母键 uan 有 10-15 个等权漏字
  目标,k=3 按表序截断够不到 quan'li 的正确读音;k=9 提升 R@1/R@3
  且 p95 不变。主切分仍是查询键,其余读音交给
  merge_alternative_segmentations 与主读法在词频下竞争;备选读音在
  缓存命中路径同样发布,corrected_from 标记对全部纠错读音生效;纠错
  读音排在音形别名读音之前。结尾单插入(shangg = shang + g)天然是
  简拼尾形状,仍被简拼形守卫拦下——多字纠错的可达面在中间/开头插入
  与多音节输入。
- 纠错门放宽:!segments.empty() 从门里去掉。BFS 工作在原始字母上,
  基础切分为空的丢声母输入恰是最需要纠错的一批,jian -> ian 现在能
  走到纠错候选并按词频合并到首位。简拼形守卫保留:合法音节加至多一个
  尾字母(zheg)是用户意图,任何纠错表都不得改写。
- preedit 补两道防线:zheg 在显式三位掩码下仍走简拼守卫,不被漏字表
  重排为 zheng;只有 BFS 切分的读音与 scheme 别名层完全一致时才按
  BFS 位置重画分隔符,否则 sahng 会被切成 sa'hng 而候选实际来自
  别名读音 shang。

新增独立评测工具 eval_quanpin_autocorrect:从真实 msime.db 采样高频
词条,按交换/邻键/漏字/多字模型注错,测 R@1/R@3、读音还原率与温态
p50/p95,固定种子可复现。它是评测工具不是测试,永远不进 CI,作为
纠错阈值与门逻辑改动的零回归标尺;注错规则与生成器逐键一致,改任一
侧须同一提交同步另一侧。

表格测试改为按 (错串, 正确) 对断言唯一并加覆盖门(合计 >= 4700 条),
新增歧义保留、漏字/多字变体、k-best 排序与权重定序、preedit 守卫等
用例;engine/tests 的 README 与 AGENTS.md 同步写清 imetest 与评测
工具两个产物的边界。
2026-09-13 11:04:34 +08:00
fanlusky 88a086a75b perf(build): Release 编译跳过重复 configure,只构建打包用的目标
本地打包每次都要等很久,其中编译这一段有两处纯粹的重复劳动。

Visual Studio 生成器把 ZERO_CHECK 接进了每个目标,CMakeLists.txt 变化时
构建会自己重新生成,所以已有 CMake 缓存时前置的 cmake --preset 不改变任何
结果,只是每个构建目录白等约 10 秒,三个目录就是 30 秒。改成缓存存在就跳过,
需要强制重跑(改了 preset、换了工具链)时传 -Reconfigure。

构建目标也从 ALL_BUILD 收窄到 MetasequoiaImeServer / MetasequoiaImeTsf。
原先会一并构建 MetasequoiaImeServerTests、test_webview_contract、
test_windows_ipc_contract,而 Prepare-PackageFiles.ps1 随后又把它们从包里
删掉。server 目标的依赖闭包已经覆盖 Watchdog、三个面板、Settings、
DictionaryReplay 和 OpenCC 词典,产物一个不少。要连测试一起构建(比如为了
跑 ctest)传 -Target ALL_BUILD,server/AGENTS.md 里的 ctest 段落已注明。

顺带给 cmake --build 加上 --parallel,真正有代码改动时也能并行。

无改动时的增量编译:TSF 18.0s -> 3.7s,Server 21.9s -> 10.9s。
CI 不受影响,它直接调 cmake,不经过这些脚本。
2026-09-13 03:56:40 +08:00
fanlusky a7779d2b05 refactor(engine): 将 engine 从 submodule 内嵌为仓内一等组件
`vendor/MetasequoiaImeEngine` submodule 改为 `engine/` 目录,导入的是
product-lock.json 原本钉住的提交 c810d201,逐文件核对过 blob,因此这是一次
搬迁而不是引擎升级。引擎自己的嵌套 submodule(googlepinyinime-rev、utfcpp、
miniaudio)按各自 gitlink 一并展开,来源与提交见 engine/UPSTREAM.md。

Windows 专门化裁掉了不属于本产品的部分:whisper.cpp 与本地推理 provider、
macOS 示例宿主、整个 dictionary/ 建库流水线、上游 CI 与 release-please 配置。
词库仍按 product-lock.json 在构建时取发布产物。

引擎不再进 product-lock.json:没有 gitlink 就没有可比对的东西,锁里写一个上游
commit 等于宣称这棵树仍等于上游,而第一个 Windows 专门化改动落地时这句话就假了。
verify-contracts / verify-checkout 随之移除,verify-published 保留。

CI 与工具链:引擎源码并入根 scripts/format.sh(删掉引擎自己的那份,避免两套排除
列表分叉);引擎 voice.yml 里的 silero_vad.onnx 摘要校验移到本仓 lock job;
engine-bump-triage 工作流与其测试删除;gitsubmodule 的 dependabot 频率由 daily
改为 monthly。

README、README.en、AGENTS.md、CONTRIBUTING、docs/product-release.md 及各子目录
文档同步说明新的位置与边界。
2026-09-12 03:40:48 +08:00
Jia Sui 2fcd57c88d style(server): 全拼纠错相关代码过 scripts/format.sh
- 生成脚本 Count 行模板改为单行,与 clang-format 输出一致,
  生成器 --check 与 format.sh --check 从此不再互斥
- test_quanpin_scheme 格式化
2026-09-10 17:32:17 +08:00
Jia Sui f8f13e7083 feat(server): 全拼纠错分类开关配置接线与候选标记
- 配置拆为 quanpin.autocorrect_transposition/neighbor 平铺键(默认关),
  旧键 autocorrect 废弃不读取不迁移;写入时段缺失自动追加,
  出厂模板无 [quanpin] 段时设置页保存不再静默失败
- engine_input_session 用 Engine 命名常量合成位掩码注入
- settings_app 补 configSnapshot quanpin 段与 ApplyConfigUpdate 路由
- BuildCurrentCandidatePage 按 corrected_from 给 view.text 追加纠错标记,
  msimeui 与 WebView2 两渲染路径共同源单点注入,提交/词频路径不受污染
- 生成脚本 add_variant 键长下限 2→3(2 字母空间归简拼),重建表 747+3244
- 测试:AC9 配置往返与旧键忽略、开关矩阵适配新签名、守卫用例
2026-09-10 17:31:51 +08:00
Evan Hu 0baaf9a1d0 fix(server): 将本地测试入口接回合仓后的主工程 2026-09-06 09:40:36 +09:00
Evan Hu 98a5e590c2 test(build): verify manifest arguments and stop on embedding failure 2026-09-06 09:02:13 +09:00
Evan Hu b2d736cd79 fix(build): finish local tooling migration to the monorepo 2026-09-06 09:02:13 +09:00
Evan Hu 373f8d1473 refactor: collapse the duplicated submodules into one vendor/ directory
Windows, Server and UiHtml each carried their own gitlink to MSIME-Engine, all three pinned to the same commit, and Server carried a gitlink to MSIME-UI that is now the ui/ component. Git only reads the root .gitmodules, so the nested ones had to go regardless.

The engine and the two third-party dependencies now live once at vendor/MetasequoiaImeEngine, vendor/opencc and vendor/cpp-pinyin. Server reaches them through MSIME_VENDOR_DIR and builds ui/ as an out-of-tree subdirectory; the Windows tip's relative contract includes gained the extra level; the ui-html contract sync and the Server env/table generators follow the engine to its new location.
2026-09-06 02:27:52 +09:00
Evan 261035a33c chore: consume the engine as a library and stop prepare_env clobbering the build (#14)
* chore: consume the engine as a library and stop prepare_env clobbering the build

The engine submodule was pinned at a197746, eight commits behind the engine's
main branch and older than the commit that first gave the engine a CMakeLists
of its own. That is why this repository listed the engine's translation units
by hand: at the pinned commit there was no target to link against. There is
one now, and Apple and Linux already consume it, so do the same here.

- advance the engine submodule from a197746 to 8cc306c
- add_subdirectory the engine and link MetasequoiaIme::Engine from the server,
  the dictionary replay tool and the settings executable, dropping the two
  hand-maintained source lists
- point the engine and msimeui submodule URLs at MSIME-Engine and MSIME-UI,
  and update the README, which still used the pre-rename repository names

prepare_env.py, which the README tells every new contributor to run first,
overwrote four tracked build files from templates under scripts/config_files
that had fallen years behind:

- CMakeLists.txt was replaced by a 147-line copy of what is now a 400-line
  build, dropping opencc, cpp-pinyin, msimeui, settings and the panels
- tests/CMakeLists.txt was replaced by an unrelated generic WinCppTemplate
- CMakePresets.json lost the vcpkg-release preset that llaunch-release.ps1
  needs

Running the documented first build step therefore broke the build. The script
now only writes the two files that are genuinely generated, and rewrites
CMakePresets.json in place instead of copying a stale template over it.

The .clangd generation addressed template lines by index and the indices had
drifted: the Boost path was written over the WebView2 include and the Sciter
path over a comment, leaving clangd unable to resolve the WebView2 headers.
The template now carries named placeholders, and the script fails loudly if
any are left unsubstituted.

Also remove src/sciter. It is a third candidate window backend that no
CMakeLists has referenced for some time; the shipped backends are d2d and
webview2, and those are the only two ui_backend accepts.

* docs: keep the historical checkout directory names in the build steps

The clone URLs had to change, since the old ones only resolved through
GitHub's rename redirect, but cloning without an explicit target would also
have renamed the working directories. MSIME-Windows's AGENTS.md records that
the historical directory names are deliberate: msime-installer's
Prepare-PackageFiles.ps1 expects siblings called MetasequoiaImeServer and
MetasequoiaImeDict, and the release workflow checks the current repositories
out into exactly those paths. Clone into them explicitly instead.

* chore: re-pin the engine at the WIN32_LEAN_AND_MEAN fix

Moves the submodule from 8cc306c to f91cd20. That carries metasequoiaime/MSIME-Engine#14,
which stops the engine exporting WIN32_LEAN_AND_MEAN as a PUBLIC compile definition and so
unblocks MetasequoiaImeSettings, whose WebView2.h include failed with 101 instances of C2146
against the trimmed Windows.h the definition forced on every consumer.

The range also picks up the two dependabot merges the engine took in the meantime: #12 bumps
utfcpp and #13 bumps the pinned action SHAs.

* build(tests): link the engine target instead of listing its sources

The test executable hand-listed the same 41 engine translation units the root
CMakeLists.txt did, and compiled them without NOMINMAX. That went unnoticed
while the pinned engine predated core/key_event.h; now that the engine headers
pull in Windows.h, the min and max macros collide with std::max and the build
fails with C2589 in quanpin_query.cpp.

Linking MetasequoiaIme::Engine fixes it at the source: NOMINMAX is one of the
definitions the engine exports PUBLIC precisely because it changes the declared
API surface.

* docs(agents): correct the tests wiring and describe the new prepare_env

Two sections went stale or were inaccurate.

The tests section said the root CMakeLists.txt does not reference tests/ at
all. It does: include(CTest) at :8 defaults BUILD_TESTING to ON and :411
add_subdirectory(tests) picks it up, so the test target is configured and
compiled on every CI run. This PR proved it from the other side, by breaking
the test target's build with a NOMINMAX regression that CI caught. The real
gap is narrower and still worth the warning: ci.yml has no ctest step, so the
executable is built and never run.

The generated-files section described prepare_env.py copying templates over
CMakeLists.txt and CMakePresets.json by fixed line index. This PR removes that
behaviour, so the section now describes what the script actually does.
2026-09-05 19:02:43 +09:00
Evan 999c70368e fix(scripts): read vcpkg and boost roots from the environment 2026-09-05 16:53:52 +09:00
Jia Sui bfd42d314e feat(quanpin): wire up typing autocorrect config and tests
- config.toml [quanpin] autocorrect = true (default on)
- ime_config: GetConfiguredQuanpinAutocorrectEnabled /
  SetConfiguredQuanpinAutocorrectEnabled with template-preserving write
- engine_input_session injects the flag into the engine session
- tests: autocorrect table integrity, autocorrect_cut units, dictionary-level
  corrected-candidate-first behavior, legacy behavior when disabled, and
  cache isolation for plain spellings
- bump MetasequoiaImeEngine to 2422fd1
2026-08-31 16:01:25 +08:00
fanlusky b159d8c9ab Update local path. Optimize audio interruption. 2026-08-23 23:37:00 +08:00
fanlusky 943b6fbd4b Add J mode, the shorthand pinyin mode. 2026-08-19 08:05:44 +08:00
fanlusky 7c7df2e4c6 Add inline preedit in-time ars result support for voice input. 2026-08-17 00:55:45 +08:00
fanlusky 8565b0531b Add kaomoji to mixed candidates (after emoji) behind general.kaomoji_mixed_input
kaomoji_ime: async worker mirroring emoji_ime; KaomojiQuery::QueryPrefix
gains a db_path override. NormalizeMixedCandidateOrder places kaomoji right
after emoji, shifting with cloud/AI. event_listener wires
UpdateKaomojiInput/ApplyKaomojiCandidates/EnqueueKaomojiCandidates and the
PrepareCandidateList trigger; config general.kaomoji_mixed_input is plumbed
through settings_app/webview2; main.cpp starts/stops KaomojiIme; ordering test
added.
2026-08-17 00:08:24 +08:00
fanlusky 7a35c8c20e Add M-mode (Shift+M) for quick kaomoji input
Shift+M enters a kaomoji quick-input mode backed by the kaomoji table in
others.db, supporting full pinyin, abbreviated pinyin, English words, and
shuangpin expansion. Candidates commit directly via the new
CandidateSource::Kaomoji; a [utility] kaomoji_mode toggle is wired through
settings. Hardens candidate payload rendering (HTML / backtick / comma
escaping) so kaomoji, which often contain < > &  and commas, display correctly.
2026-08-16 23:16:11 +08:00
fanlusky d15319bd66 Add emoji to mixed candidates (slot 3, shifts with cloud/AI) behind general.emoji_mixed_input. 2026-08-16 19:29:52 +08:00
fanlusky 08ee504609 Add E-mode (Shift+E) for quick emoji input
Shift+E enters an emoji quick-input mode backed by the emoji_pinyin index in
others.db, supporting full pinyin (xiaolian), abbreviated pinyin (xl), English
words (laugh), and shuangpin codes (Xiaohe xnlm expands to xiaolian).

- src/emoji/emoji_query.{h,cpp}: prefix lookup over emoji_pinyin with
  shuangpin->quanpin expansion driven by the active scheme
- event_listener: Shift+E trigger, composition helpers, candidate branch, and
  direct commit through the new CandidateSource::Emoji
- config: [utility] emoji_mode toggle wired through settings_app / webview2
- tests: emoji_query coverage for full/abbrev pinyin, English, Xiaohe shuangpin
- engine submodule: CandidateSource::Emoji (core/word_item.h)
2026-08-16 18:54:54 +08:00
fanlusky 33a86c11b6 Add date-time quick input (T-mode) feature, strengthen installation to not overwrite user's config.toml, and add a cold-start protection for FTB that "displays within the page ready / up to 6s grace period, then shows/hides according to actual status" (still rebuilds webview2 on upgrade). 2026-08-08 12:37:24 +08:00
fanlusky 5b162ddb93 compile manifest to release bin file 2026-08-01 02:38:10 +08:00
fanlusky de5b764a72 fix(ftb): reconcile toolbar visibility from IPC active-client state
WM_IMEACTIVATE could be dropped or skipped when activation was unchanged,
leaving the floating toolbar hidden until the user switched IME away and
back. Derive g_is_ime_active from the live active client on every visibility
apply, and re-apply once the FTB WebView finishes its first navigation.
Also restore Watchdog startup with a short profile-ready retry window.
2026-07-31 00:43:55 +08:00
fanlusky ac8d5dbcf2 chore: sign script for release version 2026-07-30 21:25:14 +08:00
fanlusky e417eaee64 Refactor to prepare for adding the Ziranma scheme. 2026-07-14 23:39:34 +08:00
fanlusky df6002c03e Add preliminary support for Chinese-English mixed input. 2026-07-14 19:00:09 +08:00
fanlusky e4fe985896 Thoroughly solve the settings window flickering and focus instability issues. Separate it out as a standalone exe. 2026-07-14 00:09:06 +08:00
fanlusky e6c8500521 tmp 2026-07-03 22:18:51 +08:00
fanlusky 3dbe168da1 Start to support new-version engine 2026-06-19 23:01:50 +08:00
fanlusky 744e117f8f Refactor 2026-06-19 22:27:12 +08:00
fanlusky d38a6bfb0a Phase 1 dependency consolidation. 2026-06-19 21:27:43 +08:00
fanlusky 3e57162cf0 Some simple refactoring 2026-06-19 21:04:01 +08:00
fanlusky f9275788c4 add compile release scripts 2026-04-02 18:25:57 +08:00
fanlumaster 5e6ee87626 [feat]: add cloud api 2026-03-26 01:44:07 +08:00
fanlumaster 991c037182 chore 2026-03-05 14:45:10 +08:00
fanlumaster 4be4a96992 make window truely top-most with sign 2026-03-04 01:43:53 +08:00
fanlumaster 1e916a2226 adjust vcpkg config 2026-02-28 18:19:11 +08:00
fanlumaster 704cae5162 [dep]: use vcpkg webview2 lib to replace origianal nuget lib 2026-02-28 15:05:38 +08:00
fanlumaster 009228c534 [dev]: Update webview2 version 2026-02-04 11:10:57 +08:00
fanlumaster 28d11c0d9f [config]: Read toml config file from appdata location 2026-01-30 12:27:23 +08:00
fanlumaster eb5014777c [config]: Update to VS2026 2025-12-29 21:54:51 +08:00
fanlumaster bd5ba1f771 [ui]: settings window 2025-11-03 13:12:51 +08:00
fanlumaster 1441fbc387 [refactor]: tidy some codes 2025-10-31 16:07:26 +08:00
fanlumaster 2f41e73e37 [feat]: Add delete word feature 2025-10-29 15:58:51 +08:00
fanlumaster c9e4ec2685 squashed commits for create word feat 2025-10-25 12:48:22 +08:00
fanlumaster 92e031e234 [ui]: Revert to webview2 2025-09-15 14:25:01 +08:00
fanlumaster 6656e5cca1 [chore]: Docs and Env 2025-06-25 16:20:36 +08:00
fanlumaster 1e5824bae1 [dependency]: Update MetasequoiaImeEngine 2025-06-20 00:09:45 +08:00
fanlumaster 5b19b102b3 [Render]: Replace sciter with direct2d and directwrite, now only support basic layout(hard code) 2025-06-16 17:04:05 +08:00
fanlumaster e908a7bf1b [dev]: Measure ui size in advance 2025-06-15 17:33:41 +08:00