fanlusky
e7f7addd4f
- 新增 engine/neural：移植 chinese-ime-lm 的字符级 Transformer 推理，在词格解出的最多 12 条 n-best 里重排挑选整句，desktop / keyboard 两档模型各出一条首选，后台 worker 异步打分 - 词格支持 n-best 输出与重排（quanpin/lattice_rerank），全拼、双拼共用 - 新增 SentenceAssociationOptions，经 QueryRequest 传给各 provider；四个来源默认全关 - 资源契约登记两个 safetensors 与 NOTICE；模型二进制不进仓库，由 scripts/fetch-neural-model.ps1 按 neural-model/lock.json 下载并校验 SHA256 - 新增 neural_sentence 测试
54 lines
1.9 KiB
PowerShell
54 lines
1.9 KiB
PowerShell
<#
|
||
.SYNOPSIS
|
||
下载 chinese-ime-lm 的神经整句模型（safetensors）到 neural-model/，并按 lock.json 校验 sha256。
|
||
|
||
.DESCRIPTION
|
||
与 build-language-model.ps1 对 sc.lm 的处理同理：模型二进制不进仓库（见 .gitignore），
|
||
只有 lock.json 和 NOTICE.md 进仓库。打包（installer/Prepare-PackageFiles.ps1）前跑一次
|
||
本脚本把两个模型取到位。已存在且校验通过的文件跳过重复下载（除非 -Force）。
|
||
|
||
.PARAMETER Force
|
||
忽略已存在文件，强制重新下载。
|
||
#>
|
||
[CmdletBinding()]
|
||
param(
|
||
[switch]$Force
|
||
)
|
||
|
||
$ErrorActionPreference = 'Stop'
|
||
Set-StrictMode -Version Latest
|
||
|
||
$here = Split-Path -Parent $PSScriptRoot
|
||
$modelDir = Join-Path $here 'neural-model'
|
||
$lockPath = Join-Path $modelDir 'lock.json'
|
||
if (-not (Test-Path -LiteralPath $lockPath)) {
|
||
throw "找不到 lock.json：$lockPath"
|
||
}
|
||
$lock = Get-Content -LiteralPath $lockPath -Raw | ConvertFrom-Json
|
||
|
||
foreach ($model in $lock.models) {
|
||
$dest = Join-Path $modelDir $model.file
|
||
$needsDownload = $true
|
||
if ((Test-Path -LiteralPath $dest) -and -not $Force) {
|
||
$hash = (Get-FileHash -LiteralPath $dest -Algorithm SHA256).Hash.ToLowerInvariant()
|
||
if ($hash -eq $model.sha256.ToLowerInvariant()) {
|
||
Write-Host "已存在且校验通过，跳过：$($model.file)"
|
||
$needsDownload = $false
|
||
}
|
||
else {
|
||
Write-Host "校验不符，重新下载：$($model.file)"
|
||
}
|
||
}
|
||
if ($needsDownload) {
|
||
Write-Host "下载 $($model.file) ..."
|
||
Invoke-WebRequest -Uri $model.url -OutFile $dest
|
||
$hash = (Get-FileHash -LiteralPath $dest -Algorithm SHA256).Hash.ToLowerInvariant()
|
||
if ($hash -ne $model.sha256.ToLowerInvariant()) {
|
||
throw "$($model.file) sha256 校验失败：期望 $($model.sha256)，实际 $hash"
|
||
}
|
||
Write-Host " 校验通过：$($model.file)"
|
||
}
|
||
}
|
||
|
||
Write-Host '神经整句模型准备完毕。'
|