fanlusky
80b1fc42a1
词格原先的整句打分是一份没校准过的启发式 unigram，质量很差。这里内嵌 libime 在用的 kenlm 查询子集，改由它的三元模型 sc.lm 来打路径分。 - engine/ngram 只编 kenlm 的 query / trie-build 路径。不定义 WITH_THREADS， parallel_read.cc 因此退化成串行实现，整个子集零 Boost 依赖。 KENLM_MAX_ORDER=3 必须与建模阶数一致，改它等于读不懂已发布的模型文件。 - ngram::LanguageModel 是唯一 include kenlm 头文件的地方，对外只给出一个纯 std 的头，_HAS_AUTO_PTR_ETC 和 kenlm 的 include 目录都保持 PRIVATE。 - shared_language_model 按资源路径共享一份 mmap。kenlm 的查询是只读的且把 上下文放在调用方的 State 里，所以并发会话不需要额外加锁。 - 模型缺失时 valid() 为假，词格退回原来的启发式打分：整句候选只是变差，不会 停止工作。测试覆盖了缺模型降级与按路径共享。 - 出货的 ARPA 不带 <s>/</s>，所以解码从 null_state() 起算而非 begin_state()。 kenlm 是上游副本，不重排版、不进 clang-format，其 LGPL-2.1+ 与 BSD 例外清单 已补进 THIRD_PARTY_NOTICES.txt。googlepinyinime 那条 Fallback 整句路径不变。
72 lines
3.1 KiB
Bash
Executable File
72 lines
3.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
# Formatting has to be reproducible across contributor machines and CI, so the
|
|
# formatter version is pinned here and installed from PyPI instead of being
|
|
# taken from whatever the host toolchain happens to ship. Apple, Debian and
|
|
# LLVM upstream all disagree about clang-format defaults between releases.
|
|
#
|
|
# Ported from MSIME-Linux, which has run this in CI since before the
|
|
# consolidation. Each component keeps its own .clang-format; --style=file picks
|
|
# up the nearest one, so server/, windows/, ui/, log/ and engine/ stay
|
|
# independent -- engine/voice/.clang-format widens the column limit for that
|
|
# subtree alone, and the nearest-file rule is what keeps it that way.
|
|
clang_format_version=18.1.8
|
|
|
|
project_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
|
|
venv_root=${METASEQUOIA_CLANG_FORMAT_VENV:-"${TMPDIR:-/tmp}/metasequoia-clang-format-$clang_format_version"}
|
|
clang_format="$venv_root/bin/clang-format"
|
|
|
|
if [[ ! -x "$clang_format" ]]; then
|
|
python3 -m venv "$venv_root"
|
|
"$venv_root/bin/pip" install --quiet "clang-format==$clang_format_version"
|
|
fi
|
|
|
|
mode=${1:---write}
|
|
case "$mode" in
|
|
--check) clang_format_arguments=(--dry-run --Werror) ;;
|
|
--write) clang_format_arguments=(-i) ;;
|
|
*)
|
|
echo "Usage: ${BASH_SOURCE[0]} [--check|--write]" >&2
|
|
exit 2
|
|
;;
|
|
esac
|
|
|
|
cd "$project_root"
|
|
# --others --exclude-standard so a newly written file that has not been staged
|
|
# yet is still formatted. Without it the script reports a clean tree while
|
|
# skipping exactly the file being worked on.
|
|
#
|
|
# vendor/ and experiments/ are excluded: the first is third-party, the second is
|
|
# a scratch project that is not part of the shipped product.
|
|
#
|
|
# engine/ is covered here rather than by a script of its own. It used to be a
|
|
# separate repository with its own formatting gate; now that it is a directory
|
|
# of this one, a Windows-specific fix to it has to be held to the same standard
|
|
# as the code that calls it, and one entry point is what makes that automatic.
|
|
# The exclusions below are the ones that gate carried:
|
|
#
|
|
# - googlepinyinime-rev keeps its AOSP formatting, and ngram/kenlm and utfcpp
|
|
# are upstream copies, so none of them is ours to reformat; same for
|
|
# */third_party/ (miniaudio).
|
|
# - The four generated headers are emitted from JSON by
|
|
# contracts/{assets,dictionary,punctuation,webview}/generate.py, and CI
|
|
# re-runs each generator with --check. Reformatting the checked-in copy
|
|
# would only make it disagree with what the generator produces.
|
|
git ls-files --cached --others --exclude-standard \
|
|
'server/*.cpp' 'server/*.h' \
|
|
'windows/*.cpp' 'windows/*.h' \
|
|
'ui/*.cpp' 'ui/*.h' \
|
|
'log/*.cpp' 'log/*.h' \
|
|
'engine/*.cpp' 'engine/*.h' 'engine/*.hpp' \
|
|
| grep -v '/vendor/' \
|
|
| grep -vE '^engine/(googlepinyinime-rev|ngram/kenlm|utfcpp)/' \
|
|
| grep -v '/third_party/' \
|
|
| grep -vxF \
|
|
-e engine/contracts/assets/assets.h \
|
|
-e engine/contracts/dictionary/format.h \
|
|
-e engine/contracts/punctuation/policy.h \
|
|
-e engine/contracts/webview/schema.h \
|
|
| sort -u \
|
|
| xargs "$clang_format" "${clang_format_arguments[@]}" --style=file
|