Files
fanlusky 80b1fc42a1 feat(engine): 词格整句改用 kenlm 三元模型打分
词格原先的整句打分是一份没校准过的启发式 unigram,质量很差。这里内嵌 libime
在用的 kenlm 查询子集,改由它的三元模型 sc.lm 来打路径分。

- engine/ngram 只编 kenlm 的 query / trie-build 路径。不定义 WITH_THREADS,
  parallel_read.cc 因此退化成串行实现,整个子集零 Boost 依赖。
  KENLM_MAX_ORDER=3 必须与建模阶数一致,改它等于读不懂已发布的模型文件。
- ngram::LanguageModel 是唯一 include kenlm 头文件的地方,对外只给出一个纯
  std 的头,_HAS_AUTO_PTR_ETC 和 kenlm 的 include 目录都保持 PRIVATE。
- shared_language_model 按资源路径共享一份 mmap。kenlm 的查询是只读的且把
  上下文放在调用方的 State 里,所以并发会话不需要额外加锁。
- 模型缺失时 valid() 为假,词格退回原来的启发式打分:整句候选只是变差,不会
  停止工作。测试覆盖了缺模型降级与按路径共享。
- 出货的 ARPA 不带 <s>/</s>,所以解码从 null_state() 起算而非 begin_state()。

kenlm 是上游副本,不重排版、不进 clang-format,其 LGPL-2.1+ 与 BSD 例外清单
已补进 THIRD_PARTY_NOTICES.txt。googlepinyinime 那条 Fallback 整句路径不变。
2026-09-19 10:08:39 +08:00

207 lines
5.4 KiB
C++

#ifndef UTIL_FIXED_ARRAY_H
#define UTIL_FIXED_ARRAY_H
#include "scoped.hh"
#include <cstddef>
#include <cassert>
#include <cstdlib>
namespace util {
/**
* Defines an array with fixed maximum size.
*
* Ever want an array of things but they don't have a default constructor or
* are non-copyable? FixedArray allows constructing one at a time.
*/
template <class T> class FixedArray {
public:
/** Initialize with a given size bound but do not construct the objects. */
explicit FixedArray(std::size_t limit) {
Init(limit);
}
/**
* Constructs an instance, but does not initialize it.
*
* Any objects constructed in this manner must be subsequently @ref FixedArray::Init() "initialized" prior to use.
*
* @see FixedArray::Init()
*/
FixedArray()
: newed_end_(NULL)
#ifndef NDEBUG
, allocated_end_(NULL)
#endif
{}
/**
* Initialize with a given size bound but do not construct the objects.
*
* This method is responsible for allocating memory.
* Objects stored in this array will be constructed in a location within this allocated memory.
*/
void Init(std::size_t count) {
assert(!block_.get());
block_.reset(malloc(sizeof(T) * count));
if (!block_.get()) throw std::bad_alloc();
newed_end_ = begin();
#ifndef NDEBUG
allocated_end_ = begin() + count;
#endif
}
/**
* Constructs a copy of the provided array.
*
* @param from Array whose elements should be copied into this newly-constructed data structure.
*/
FixedArray(const FixedArray &from) {
std::size_t size = from.newed_end_ - static_cast<const T*>(from.block_.get());
Init(size);
for (std::size_t i = 0; i < size; ++i) {
push_back(from[i]);
}
}
/**
* Frees the memory held by this object.
*/
~FixedArray() { clear(); }
#if __cplusplus >= 201103L
FixedArray(FixedArray &&from)
: block_(std::move(from.block_)),
newed_end_(from.newed_end_)
# ifndef NDEBUG
, allocated_end_(from.allocated_end_)
# endif // NDEBUG
{
from.newed_end_ = NULL;
# ifndef NDEBUG
from.allocated_end_ = NULL;
# endif // NDEBUG
}
#endif // C++11
/** Gets a pointer to the first object currently stored in this data structure. */
T *begin() { return static_cast<T*>(block_.get()); }
/** Gets a const pointer to the last object currently stored in this data structure. */
const T *begin() const { return static_cast<const T*>(block_.get()); }
/** Gets a pointer to the last object currently stored in this data structure. */
T *end() { return newed_end_; }
/** Gets a const pointer to the last object currently stored in this data structure. */
const T *end() const { return newed_end_; }
/** Gets a reference to the last object currently stored in this data structure. */
T &back() { return *(end() - 1); }
/** Gets a const reference to the last object currently stored in this data structure. */
const T &back() const { return *(end() - 1); }
/** Gets the number of objects currently stored in this data structure. */
std::size_t size() const { return end() - begin(); }
/** Returns true if there are no objects currently stored in this data structure. */
bool empty() const { return begin() == end(); }
/**
* Gets a reference to the object with index i currently stored in this data structure.
*
* @param i Index of the object to reference
*/
T &operator[](std::size_t i) {
assert(i < size());
return begin()[i];
}
/**
* Gets a const reference to the object with index i currently stored in this data structure.
*
* @param i Index of the object to reference
*/
const T &operator[](std::size_t i) const {
assert(i < size());
return begin()[i];
}
/**
* Constructs a new object using the provided parameter,
* and stores it in this data structure.
*
* The memory backing the constructed object is managed by this data structure.
* I miss C++11 variadic templates.
*/
#if __cplusplus >= 201103L
template <typename... Construct> T *emplace_back(Construct&&... construct) {
T *ret = end();
new (end()) T(construct...);
Constructed();
return ret;
}
template <typename... Construct> T *push_back(Construct&&... construct) {
T *ret = end();
new (end()) T(construct...);
Constructed();
return ret;
}
#else
void push_back() {
new (end()) T();
Constructed();
}
template <class C> void push_back(const C &c) {
new (end()) T(c);
Constructed();
}
template <class C> void push_back(C &c) {
new (end()) T(c);
Constructed();
}
template <class C, class D> void push_back(const C &c, const D &d) {
new (end()) T(c, d);
Constructed();
}
#endif
void pop_back() {
back().~T();
--newed_end_;
}
/**
* Removes all elements from this array.
*/
void clear() {
while (newed_end_ != begin())
pop_back();
}
protected:
// Always call Constructed after successful completion of new.
void Constructed() {
++newed_end_;
#ifndef NDEBUG
assert(newed_end_ <= allocated_end_);
#endif
}
private:
util::scoped_malloc block_;
T *newed_end_;
#ifndef NDEBUG
T *allocated_end_;
#endif
};
} // namespace util
#endif // UTIL_FIXED_ARRAY_H