feat(asr): 支持讯飞实时语音转写(RTASR)提供商(#842) (#894)

- 新增 asr/xfyun.rs RTASR 流式客户端:signa = Base64(HmacSHA1(MD5(appid+ts), apiKey))
  鉴权,1280B/40ms 分块发送,{"end": true} 收尾,seg_id 去重拼接,断连 partial 兜底,
  鉴权/限流/超时错误分类可读。
- 全链路接入:dictation 主流程、QA 面板、重新转录、取消、设置页凭据预检与
  "验证"探活;凭据 xfyun.app_id / xfyun.api_key 存系统凭据库,不落日志。
- 设置页新增「讯飞实时语音转写」选项(AppID + API Key),5 种语言 i18n。
- 文档:docs/xfyun-asr.md + README/README.zh/USAGE 同步。

已知限制:标准版 RTASR 无请求级热词参数(在讯飞控制台配置),语种默认中文普通话。
This commit is contained in:
Chris233 authored and GitHub committed 2026-08-03 21:37:10 +08:00
1 parent a3d014b687
commit 5be28b1dbd
23 files changed
+1269 -16

No files matched your search

+8 -8
View File
@@ -159,7 +159,7 @@ The closest alternatives are subscription SaaS products: a monthly fee, no optio
OpenLess targets the same end-user experience, but:
- **Fully open source, local-first.** The code lives in this repository, and all of your data stays on your machine.
- **Bring your own cloud credentials.** Volcengine streaming ASR with Ark / DeepSeek-compatible chat completions. No vendor lock-in.
- **Bring your own cloud credentials.** A dozen-plus ASR and polish providers — Volcengine, iFlytek, Alibaba Cloud Bailian, StepFun, OpenAI-compatible endpoints, and more — or any endpoint you bring. No vendor lock-in.
- **Tuned for AI prompts.** The structured mode reshapes loose speech into a prompt complete with context, constraints, and requests — ready to paste into ChatGPT, Claude, or Cursor.
- **It will not answer for you.** The model only cleans up your text. If you say "what features does this app still need?", it returns that as a clean question — it does not hand you a feature list. For that, ask the AI itself.
@@ -198,7 +198,7 @@ OpenLess does one thing: it **turns speech into usable written text — AI promp
| [Lazy](https://heylazy.com) | Closed-source notes / capture tool | Not a notes container — inserts straight into any input field |
| [Superwhisper](https://superwhisper.com) | Closed-source macOS, subscription | Open source; cloud ASR today, local ASR on the roadmap |
## Status (v1.3.6)
## Status
Every item below is one more layer sedimented into a default — a capability you authorize once and then never manage again. This is the infrastructure you stand on after launch:
@@ -206,9 +206,9 @@ Every item below is one more layer sedimented into a default — a capability yo
- 🎨 **Style Pack Marketplace** — browse, install, and like community **style packs** from the in-app Marketplace, and publish your own (custom system prompt per pack, switchable by hotkey). Backed by a moderated marketplace backend; uploads are reviewed before they go public.
- ⚡ **Streaming insertion** — polished text is written to the cursor character by character to reduce perceived latency, with an automatic one-shot-paste fallback. Toggle in Settings → Recording.
- **Toggle and push-to-talk** recording modes, plus a **MediaPlayPause trigger** so wired-earbud inline controls can start and stop recording. `Esc` cancels at any phase, including polish and insertion.
- **Cloud ASR**: Volcengine streaming ASR, OpenAI Whisper-compatible batch ASR, Apple Speech (macOS).
- **Local ASR**: bundled Qwen3-ASR (0.6B / 1.7B) via vendored `Open-Less/qwen-asr`; Windows Foundry Local Whisper variants.
- **Polish providers**: Ark / DeepSeek / OpenAI / Doubao / Anthropic-compatible chat completions, plus any OpenAI-compatible endpoint you bring.
- **Cloud ASR**: Volcengine streaming ASR (bigasr), iFlytek realtime ASR (RTASR), Alibaba Cloud Bailian (classic realtime / Qwen3 realtime / Fun-ASR-Flash file transcription), StepFun StepAudio (batch + realtime), Zhipu GLM-ASR, Xiaomi MiMo ASR, ElevenLabs Scribe, OpenAI-compatible batch transcription (OpenAI Whisper / Groq / SiliconFlow SenseVoice / OpenRouter), and Apple Speech (macOS).
- **Local ASR**: bundled Qwen3-ASR (0.6B / 1.7B) via vendored `Open-Less/qwen-asr` (macOS); Windows Foundry Local Whisper and sherpa-onnx (experimental) variants.
- **Polish providers**: Ark (Volcengine), DeepSeek, OpenAI, Google Gemini, Codex OAuth, SiliconFlow, Atlas Cloud, Xiaomi MiMo, CometAPI, OpenRouter, Alibaba Cloud Coding Plan, CodingPlanX, MiniMax, and StepFun — plus any OpenAI-compatible endpoint you bring.
- **Four output modes**: raw, light polish, structured (**AI-prompt mode**), and formal. Plus a **translation hotkey** that converts speech directly into the configured target language ([#43](../../issues/43)).
- **Selection-ask QA panel** — a separate hotkey opens a floating panel that runs voice Q&A against the highlighted text in any app ([#118](../../issues/118)).
- **Main window**: Overview / History / Vocab / Style / Marketplace / Settings. Persistent tray icon, plus a mini status capsule that floats on screen and follows the display you are typing on (multi-monitor).
@@ -337,6 +337,7 @@ New credential writes do not persist plaintext secrets. The repository contains
You will need:
- **Volcengine streaming ASR**: APP ID, Access Token, Resource ID.
- **iFlytek realtime ASR (RTASR)**: AppID, API Key. See [docs/xfyun-asr.md](docs/xfyun-asr.md).
- **Ark polish**: API Key, Model ID, Endpoint. The Ark default endpoint is `https://ark.cn-beijing.volces.com/api/v3/chat/completions`.
## Prompt-handling principles
@@ -363,7 +364,7 @@ Long-term reference rewrites are stored as `raw → polished → rule` triples a
The dictionary handles your proper nouns, product names, names of people, and new words. Today it supports:
- Manually adding the correct spelling, a category, and notes. You do not need to maintain misspellings or context hints.
- Enabled entries are sent as Volcengine ASR `context.hotwords` so they are recognized correctly during transcription.
- Enabled entries are sent to the ASR provider that supports hotwords (Volcengine `context.hotwords`, StepFun `hotwords`, Whisper-compatible `prompt`, Bailian vocabulary ID) so they are recognized correctly during transcription. iFlytek realtime ASR has no request-level hotword parameter — configure personalized hotwords in the iFlytek console instead.
- Entries are also injected into the polish prompt: the model decides per sentence whether to substitute. If "Cloud" clearly refers to the AI product `Claude` in context, it is corrected; if it genuinely means cloud computing, it is left as is.
- The app auto-learns candidate corrections such as `Claude`, `ChatGPT`, and `OpenLess` from your history and offers them later.
@@ -379,7 +380,7 @@ The active implementation is Tauri 2 (`openless-all/app/`). Releases are split i
types.rs Pure value types: DictationSession, PolishMode, HotkeyBinding, errors
hotkey.rs Global hotkey (CGEventTap on macOS, WH_KEYBOARD_LL on Windows, rdev on Linux)
recorder.rs Mic → 16 kHz mono Int16 PCM, RMS callback
asr/ Volcengine streaming ASR (WebSocket) + Whisper HTTP
asr/ Streaming ASR clients (Volcengine / Bailian / Qwen3 / StepFun / iFlytek over WebSocket) + Whisper-compatible batch HTTP
polish.rs OpenAI-compatible chat completions (Ark / DeepSeek / etc.)
insertion.rs AX focused-element → clipboard + Cmd+V → copy-only fallback
persistence.rs History / preferences / vocab JSON + platform credential vault
@@ -398,7 +399,6 @@ See [CLAUDE.md](CLAUDE.md) for invariants and module-wiring rules.
Planned but not yet shipped:
- Dictation translation mode: hold a separate hotkey, speak in your language, insert in the target language ([#43](../../issues/43)).
- Cross-session style memory: polish learns the user's tone over time ([#46](../../issues/46)).
- Snippets (no UI or trigger logic yet).
- History enhancements: copy button, search, re-polish, re-insert.
+7 -7
View File
@@ -198,7 +198,7 @@ OpenLess 只做一件事:**把语音变成可用的书面文字(尤其是 AI 提
| [Lazy](https://heylazy.com) | 闭源的笔记 / 速记工具 | 不是笔记容器——直接插入到任意输入框 |
| [Superwhisper](https://superwhisper.com) | 闭源 macOS,订阅制 | 开源;目前云端 ASR,本地 ASR 在路线图中 |
## 当前状态(v1.3.6)
## 当前状态
下面每一项,都是一层已经沉降为默认、你授权一次之后就不必再操心的能力——这就是开屏之后你所站立的基础设施:
@@ -206,9 +206,9 @@ OpenLess 只做一件事:**把语音变成可用的书面文字(尤其是 AI 提
- 🎨 **风格包市场**——在应用内的 Marketplace 浏览、安装、点赞社区**风格包**,并发布自己的(每个包一套自定义系统提示词,可用快捷键切换)。由经过审核的市场后端支撑;上传内容公开前会经过审核。
- ⚡ **流式插入**——润色后的文本逐字符写入光标以降低感知延迟,并带有自动的一次性粘贴回退。可在 设置 → 录音 中切换。
- **切换式与按住说话(push-to-talk)** 两种录音模式,外加 **MediaPlayPause 触发**,让有线耳机的线控也能开始 / 停止录音。`Esc` 可在任意阶段取消,包括润色与插入。
- **云端 ASR**:Volcengine 流式 ASR、OpenAI Whisper 兼容的批量 ASR、Apple Speech(macOS)。
- **本地 ASR**:通过 vendored 的 `Open-Less/qwen-asr` 内置 Qwen3-ASR(0.6B / 1.7B);Windows 上的 Foundry Local Whisper 变体。
- **润色提供方**:Ark / DeepSeek / OpenAI / Doubao / Anthropic 兼容的 chat completions,以及你自带的任意 OpenAI 兼容端点。
- **云端 ASR**:Volcengine 流式 ASR(bigasr)、讯飞实时语音转写(RTASR)、阿里云百炼(经典实时 / Qwen3 实时 / Fun-ASR-Flash 录音文件)、阶跃星辰 StepAudio(批式 + 实时)、智谱 GLM-ASR、小米 MiMo ASR、ElevenLabs Scribe、OpenAI 兼容批量转写(OpenAI Whisper / Groq / 硅基流动 SenseVoice / OpenRouter),以及 Apple Speech(macOS)。
- **本地 ASR**:通过 vendored 的 `Open-Less/qwen-asr` 内置 Qwen3-ASR(0.6B / 1.7B)(macOS);Windows 上的 Foundry Local Whisper 与 sherpa-onnx(实验性)变体。
- **润色提供方**:Ark(火山方舟)、DeepSeek、OpenAI、Google Gemini、Codex OAuth、硅基流动、Atlas Cloud、小米 MiMo、CometAPI、OpenRouter、阿里云 Coding Plan、CodingPlanX、MiniMax、StepFun,以及你自带的任意 OpenAI 兼容端点。
- **四种输出模式**:原文、轻度润色、结构化(**AI 提示词模式**)、正式。另有一个**翻译快捷键**,将语音直接转换为所配置的目标语言([#43](../../issues/43))。
- **选区问答面板**——一个独立快捷键打开浮动面板,针对任意应用中被高亮选中的文本进行语音问答([#118](../../issues/118))。
- **主窗口**:概览 / 历史 / 词典 / 风格 / 市场 / 设置。常驻托盘图标,以及一个浮于屏幕、并跟随你正在输入的显示器的迷你状态胶囊(多显示器)。
@@ -218,7 +218,7 @@ OpenLess 只做一件事:**把语音变成可用的书面文字(尤其是 AI 提
- **Beta 频道(可选加入)**——设置 → 关于 → 加入 Beta 频道,可下载最新预发布版本进行手动安装。Beta 版本绝不会自动推送给 Stable 用户(见[贡献流程](#贡献流程))。
- **分发渠道**——从 [Releases](../../releases) 直接下载 DMG/EXE、Homebrew Cask(`brew install --cask openless`)、Windows 安装包。
- **单实例锁**——防止两个 OpenLess 进程争抢同一个快捷键边沿。
- 词典条目作为 Volcengine ASR 的 `context.hotwords` 注入,并在润色时作为语义提示;命中次数按会话累计。
- 词典条目注入到支持热词的 ASR 提供方(Volcengine 的 `context.hotwords`、StepFun 的 `hotwords`、Whisper 兼容的 `prompt`、百炼的 vocabulary_id),并在润色时作为语义提示;命中次数按会话累计。讯飞实时语音转写标准版没有请求级热词参数,需在讯飞控制台配置个性化热词。
- 平台原生全局快捷键:macOS 上为 CGEventTap,Windows 上为低级键盘钩子(`WH_KEYBOARD_LL`)。
## 下载与安装(终端用户)
@@ -337,6 +337,7 @@ Windows: %APPDATA%\OpenLess\credentials.json
你需要准备:
- **Volcengine 流式 ASR**:APP ID、Access Token、Resource ID。
- **讯飞实时语音转写(RTASR)**:AppID、API Key。见 [docs/xfyun-asr.md](docs/xfyun-asr.md)。
- **Ark 润色**:API Key、Model ID、Endpoint。Ark 默认端点为 `https://ark.cn-beijing.volces.com/api/v3/chat/completions`。
## 文本处理原则
@@ -379,7 +380,7 @@ OpenLess 的润色模型只重塑文本。它不回答问题、不执行任务
types.rs Pure value types: DictationSession, PolishMode, HotkeyBinding, errors
hotkey.rs Global hotkey (CGEventTap on macOS, WH_KEYBOARD_LL on Windows, rdev on Linux)
recorder.rs Mic → 16 kHz mono Int16 PCM, RMS callback
asr/ Volcengine streaming ASR (WebSocket) + Whisper HTTP
asr/ 流式 ASR 客户端(Volcengine / Bailian / Qwen3 / StepFun / iFlytek 走 WebSocket)+ Whisper 兼容批量 HTTP
polish.rs OpenAI-compatible chat completions (Ark / DeepSeek / etc.)
insertion.rs AX focused-element → clipboard + Cmd+V → copy-only fallback
persistence.rs History / preferences / vocab JSON + platform credential vault
@@ -398,7 +399,6 @@ commands.rs Tauri IPC surface
已规划但尚未发布:
- 听写翻译模式:按住一个独立快捷键,用你的语言说,插入为目标语言([#43](../../issues/43))。
- 跨会话风格记忆:润色随时间学习用户的语气([#46](../../issues/46))。
- 片段(Snippets,尚无 UI 或触发逻辑)。
- 历史增强:复制按钮、搜索、重新润色、重新插入。
+2
View File
@@ -35,6 +35,8 @@
不会配火山 ASR 的话,先看这篇图文引导:
[OpenLess 火山 ASR 配置](docs/volcengine-setup.md)
用讯飞实时语音转写的话,看 [OpenLess 讯飞 ASR 配置](docs/xfyun-asr.md)(AppID + API Key,设置 → 语音提供商 → 讯飞实时语音转写)。
打开 OpenLess → **设置**,填入以下字段:
| 字段 | 说明 |
+39
View File
@@ -0,0 +1,39 @@
# OpenLess 讯飞 ASR 配置
OpenLess 支持讯飞开放平台的**实时语音转写(RTASR,标准版)**,用于把录音实时转写成文字。
## 所需服务
- 服务:讯飞开放平台 **实时语音转写(标准版)**
- 接口文档:<https://www.xfyun.cn/doc/asr/rtasr/API.html>
> 说明:录音文件转写(IFASR)、实时语音转写大模型(202 方言 / 37 语种免切)目前
> **未接入**。若你有这些需求,欢迎在 issue #842 留言。
## 凭证获取
1. 登录讯飞开放平台:<https://www.xfyun.cn/>
2. 控制台 → 我的应用 → 创建应用,添加 **实时语音转写** 服务(需先在服务页领取免费额度或购买)。
3. 在服务管理页查看并复制 **AppID** 与 **API Key**。
4. 打开 OpenLess `Settings → Providers → ASR`,选择「讯飞实时语音转写」,填入 AppID 与 API Key。
## 音频约束
- 采样率 16 kHz、位深 16 bit、单声道 PCM(与 OpenLess 录音输出完全一致,无需转码)。
- 支持中文普通话、中英混合与英文;方言/小语种需在讯飞控制台开通后由 `lang` 参数指定,
首期 OpenLess 固定使用中文普通话(`lang=cn`)。
- IP 白名单:若在讯飞控制台开启了 IP 白名单,请把运行 OpenLess 机器的公网 IP 加入,
否则握手会返回 `10105 illegal access`。
## 已知限制
- 标准版 RTASR **没有请求参数级热词**:OpenLess 的词典热词不会传给讯飞,个性化热词
请在讯飞控制台上传。
- 实时出字由服务端驱动:OpenLess 的胶囊 UI 在收尾阶段一次性展示最终文本(与其它
流式 provider 行为一致)。
## 排障
- `10105` / `10110`:AppID / API Key 错误、IP 白名单未放行,或账号未开通实时语音转写服务。
- `10800`:超过授权连接数(并发路数已满)。
- `37005`:超过 15 秒未发送音频(弱网时偶发,可重试)。
+12
View File
@@ -3144,6 +3144,16 @@ version = "0.7.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0e7465ac9959cc2b1404e8e2367b43684a6d13790fe23056cc8c6c5a6b7bcb94"
[[package]]
name = "md-5"
version = "0.10.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf"
dependencies = [
"cfg-if",
"digest",
]
[[package]]
name = "memchr"
version = "2.8.2"
@@ -3941,6 +3951,7 @@ dependencies = [
"libc",
"local-ip-address",
"log",
"md-5",
"minisign-verify",
"ndk-context",
"objc2 0.5.2",
@@ -3954,6 +3965,7 @@ dependencies = [
"rustls",
"serde",
"serde_json",
"sha1",
"sha2",
"sherpa-onnx",
"simplelog",
+4
View File
@@ -29,6 +29,10 @@ serde_json = "1"
# OpenRouter ASR 把音频以标准 base64(带 padding)放进 JSON body(issue #582)。
base64 = "0.22"
sha2 = "0.10"
# 讯飞 RTASR/IFASR 签名:signa = Base64(HmacSHA1(MD5(appid + ts), apiKey))。
md-5 = "0.10"
# HmacSHA1 的摘要实现(signa 签名用);已在依赖树(传递依赖),提升为直接依赖零成本。
sha1 = "0.10"
# issue #609 F-03:history.json 完整性校验(HMAC-SHA256),密钥存系统凭据库。
hmac = "0.12"
# issue #609 H-02:HMAC 比较走常量时间,避免计时侧信道。已在依赖树(sha2/hmac 传递),
@@ -17,6 +17,7 @@ pub mod stepfun_realtime;
pub mod volcengine;
pub mod wav;
pub mod whisper;
pub mod xfyun;
pub use bailian::{BailianCredentials, BailianRealtimeASR};
pub use dashscope_multimodal::DashScopeMultimodalASR;
@@ -26,6 +27,7 @@ pub use qwen_realtime::{Qwen3RealtimeASR, Qwen3RealtimeCredentials};
pub use stepfun_realtime::{StepfunRealtimeASR, StepfunRealtimeCredentials};
pub use volcengine::{VolcengineCredentials, VolcengineStreamingASR};
pub use whisper::WhisperBatchASR;
pub use xfyun::{XfyunCredentials, XfyunStreamingASR};
/// Sink for raw 16 kHz / 16-bit / mono PCM bytes coming off the recorder.
///
+863
View File
@@ -0,0 +1,863 @@
//! iFlytek(讯飞开放平台)实时语音转写(RTASR)流式客户端。
//!
//! 官方文档:https://www.xfyun.cn/doc/asr/rtasr/API.html
//!
//! 协议要点(标准版):
//! - 端点:`wss://rtasr.xfyun.cn/v1/ws`,鉴权走查询参数
//! `appid` + `ts` + `signa`,其中 `signa = Base64(HmacSHA1(MD5(appid + ts), apiKey))`;
//! - 音频:16 kHz / 16-bit / 单声道 PCM,与 OpenLess recorder 输出完全一致;
//! - 建议每 40ms 发送 1280 字节,发送过快可能触发引擎报错;
//! - 上传结束:发送二进制消息 `{"end": true}`;
//! - 结果:服务端以 text message 返回 `{"action":"result","data":"<json 字符串>"}`,
//! `data.cn.st.type` 为 `0`(最终结果)/ `1`(中间结果),全部结果发完后服务端断开连接。
//!
//! 已知限制:标准版 RTASR 没有请求参数级热词(个性化热词只能在讯飞控制台上传);
//! 方言/小语种需在控制台开通后通过 `lang` 参数指定,首期固定中文普通话(`cn`)。
use std::collections::BTreeMap;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
use base64::Engine;
use futures_util::{SinkExt, StreamExt};
use hmac::{Hmac, Mac};
use md5::{Digest as Md5Digest, Md5};
use parking_lot::Mutex as ParkingMutex;
use serde_json::Value;
use sha1::Sha1;
use tokio::net::TcpStream;
use tokio::runtime::Handle;
use tokio::sync::{mpsc, oneshot, Mutex as AsyncMutex, Notify};
use tokio_tungstenite::tungstenite::client::IntoClientRequest;
use tokio_tungstenite::tungstenite::Message;
use tokio_tungstenite::{connect_async, MaybeTlsStream, WebSocketStream};
use super::{AudioConsumer, RawTranscript};
pub const PROVIDER_ID: &str = "iflytek";
pub const DEFAULT_ENDPOINT: &str = "wss://rtasr.xfyun.cn/v1/ws";
/// RTASR 文档建议:每 40ms 发送 1280 字节(16k/16-bit/mono = 32000 B/s)。
pub const TARGET_AUDIO_CHUNK_BYTES: usize = 1_280;
/// 16 kHz · 16-bit · mono = 32 000 bytes/sec → 32 bytes/ms。
const BYTES_PER_MS: u64 = 32;
const FINAL_RESULT_TIMEOUT: Duration = Duration::from_secs(12);
/// WebSocket 建连(TCP + TLS + HTTP upgrade)上限,避免弱网下握手挂死热键线程。
const CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
/// 握手阶段等待 `action=started` 的上限。鉴权失败(10105/10110)应在此窗口内快速失败,
/// 而不是把错误拖到收尾阶段才暴露。
const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(5);
/// 默认语种:中文普通话。方言/小语种需在讯飞控制台开通后传对应 `lang` 参数。
const DEFAULT_LANG: &str = "cn";
type WsStream = WebSocketStream<MaybeTlsStream<TcpStream>>;
type WsSink = futures_util::stream::SplitSink<WsStream, Message>;
type SharedWriter = Arc<AsyncMutex<Option<WsSink>>>;
#[derive(Clone, Debug)]
pub struct XfyunCredentials {
/// 讯飞开放平台应用 ID。
pub app_id: String,
/// 实时语音转写服务对应的 APIKey(接口密钥)。
pub api_key: String,
}
impl XfyunCredentials {
pub fn auth_ok(&self) -> bool {
!self.app_id.trim().is_empty() && !self.api_key.trim().is_empty()
}
}
#[derive(Debug, Clone, thiserror::Error)]
pub enum XfyunASRError {
#[error("credentials missing")]
CredentialsMissing,
#[error("连接失败: {0}")]
ConnectionFailed(String),
/// 握手阶段服务端返回 10105 / 10110:AppID / APIKey 错误、IP 白名单未配置、
/// 或账号未开通实时语音转写服务。
#[error("凭据被拒或未开通服务({0})")]
AuthRejected(String),
/// 10800:超过授权连接数 / 并发受限。
#[error("并发受限({0})")]
RateLimited(String),
#[error("识别失败: {0}")]
TaskFailed(String),
#[error("no final result")]
NoFinalResult,
#[error("final result timed out")]
FinalResultTimeout,
}
#[derive(Default)]
struct SyncState {
pending_audio: Vec<u8>,
bytes_sent: u64,
started: bool,
finished: bool,
runtime: Option<Handle>,
start: Option<Instant>,
final_tx: Option<oneshot::Sender<Result<RawTranscript, XfyunASRError>>>,
/// seg_id → 最终(type=0)分段文本。同一 seg_id 的后到结果覆盖前一个。
final_segments: BTreeMap<i64, String>,
/// seg_id → 最近一次中间(type=1)分段文本,服务端在 final 前断连时兜底用。
partial_segments: BTreeMap<i64, String>,
last_result_text: String,
}
pub struct XfyunStreamingASR {
credentials: XfyunCredentials,
state: ParkingMutex<SyncState>,
writer: SharedWriter,
final_rx: ParkingMutex<Option<oneshot::Receiver<Result<RawTranscript, XfyunASRError>>>>,
/// 握手结果通道:receive loop 收到 `action=started` 后发 Ok,收到 error 发 Err。
/// `open_session` 等待它以在鉴权失败时快速失败。
handshake_tx: ParkingMutex<Option<oneshot::Sender<Result<(), XfyunASRError>>>>,
/// 音频发送队列:consume_pcm_chunk 入队,唯一 worker 串行 send,保证时序。
audio_tx: ParkingMutex<Option<mpsc::UnboundedSender<Vec<u8>>>>,
/// 队列里 + worker 在飞的 audio 帧总数。send_last_frame 必须等它归零再发
/// `{"end": true}`,否则末帧先到、尾部音频被服务端当「end 之后的数据」丢弃。
pending_sends: Arc<AtomicUsize>,
send_done: Arc<Notify>,
}
impl XfyunStreamingASR {
pub fn new(credentials: XfyunCredentials) -> Self {
Self {
credentials,
state: ParkingMutex::new(SyncState::default()),
writer: Arc::new(AsyncMutex::new(None)),
final_rx: ParkingMutex::new(None),
handshake_tx: ParkingMutex::new(None),
audio_tx: ParkingMutex::new(None),
pending_sends: Arc::new(AtomicUsize::new(0)),
send_done: Arc::new(Notify::new()),
}
}
/// 构建带鉴权参数的 WebSocket 地址:
/// `wss://rtasr.xfyun.cn/v1/ws?appid=..&ts=..&signa=..&lang=cn`。
pub fn connect_url(&self) -> String {
connect_url(&self.credentials)
}
pub async fn open_session(self: &Arc<Self>) -> Result<(), XfyunASRError> {
if !self.credentials.auth_ok() {
return Err(XfyunASRError::CredentialsMissing);
}
let request = self
.connect_url()
.into_client_request()
.map_err(|e| XfyunASRError::ConnectionFailed(e.to_string()))?;
let (ws, _resp) = tokio::time::timeout(CONNECT_TIMEOUT, connect_async(request))
.await
.map_err(|_| {
XfyunASRError::ConnectionFailed(format!(
"连接超时({} ms)",
CONNECT_TIMEOUT.as_millis()
))
})?
.map_err(|e| XfyunASRError::ConnectionFailed(e.to_string()))?;
let (write, read) = ws.split();
*self.writer.lock().await = Some(write);
let (final_tx, final_rx) = oneshot::channel();
let (audio_tx, mut audio_rx) = mpsc::unbounded_channel::<Vec<u8>>();
let (handshake_tx, handshake_rx) = oneshot::channel();
{
let mut st = self.state.lock();
*st = SyncState::default();
st.runtime = Some(Handle::current());
st.start = Some(Instant::now());
st.final_tx = Some(final_tx);
}
*self.final_rx.lock() = Some(final_rx);
*self.audio_tx.lock() = Some(audio_tx);
*self.handshake_tx.lock() = Some(handshake_tx);
self.pending_sends.store(0, Ordering::SeqCst);
// 音频 worker:FIFO recv + 串行 send_binary,保证 chunk 顺序。
let writer_for_worker = Arc::clone(&self.writer);
let pending_for_worker = Arc::clone(&self.pending_sends);
let notify_for_worker = Arc::clone(&self.send_done);
tokio::spawn(async move {
while let Some(chunk) = audio_rx.recv().await {
if let Err(e) = send_binary(&writer_for_worker, chunk).await {
log::error!("[xfyun-asr] audio frame send failed: {e}");
}
if pending_for_worker.fetch_sub(1, Ordering::SeqCst) == 1 {
notify_for_worker.notify_waiters();
}
}
});
// receive loop:处理 started / result / error,以及服务端断开。
let weak_self = Arc::downgrade(self);
tokio::spawn(async move {
let mut read = read;
while let Some(msg) = read.next().await {
let Some(this) = weak_self.upgrade() else {
break;
};
match msg {
Ok(Message::Text(text)) => {
if !this.handle_text_message(&text) {
break;
}
}
Ok(Message::Close(_)) => {
// 服务端在全部结果发完后主动断开;也覆盖 37005(15s 无音频)
// 等中断场景 —— finish_on_close 会按已有内容兜底。
this.finish_on_close();
break;
}
Ok(_) => { /* ignore binary/ping/pong */ }
Err(e) => {
log::error!("[xfyun-asr] receive loop error: {e}");
this.finish_with_partial_or_error(XfyunASRError::ConnectionFailed(
e.to_string(),
));
break;
}
}
}
});
// 等待握手结果:鉴权错误 / 连接被拒在这里快速失败,不等用户说完话。
match tokio::time::timeout(HANDSHAKE_TIMEOUT, handshake_rx).await {
Ok(Ok(Ok(()))) => Ok(()),
Ok(Ok(Err(e))) => {
self.cancel();
Err(e)
}
Ok(Err(_)) => {
self.cancel();
Err(XfyunASRError::ConnectionFailed(
"握手通道提前关闭".to_string(),
))
}
Err(_) => {
self.cancel();
Err(XfyunASRError::ConnectionFailed(format!(
"握手超时({} ms)",
HANDSHAKE_TIMEOUT.as_millis()
)))
}
}
}
pub async fn send_last_frame(&self) -> Result<(), XfyunASRError> {
// 等所有在途音频帧发完,再发 `{"end": true}`(上限 800ms 防极端网络下永远等)。
let drain_deadline = Instant::now() + Duration::from_millis(800);
while self.pending_sends.load(Ordering::SeqCst) > 0 {
let remaining = drain_deadline.saturating_duration_since(Instant::now());
if remaining.is_zero() {
log::warn!(
"[xfyun-asr] send_last_frame: pending {} 帧未发送完,超时强制继续",
self.pending_sends.load(Ordering::SeqCst)
);
break;
}
let _ = tokio::time::timeout(remaining, self.send_done.notified()).await;
}
// 冲刷尾部不足一块的残余音频。
let leftover = {
let mut st = self.state.lock();
if st.pending_audio.is_empty() {
None
} else {
Some(std::mem::take(&mut st.pending_audio))
}
};
if let Some(buf) = leftover {
let len = buf.len() as u64;
self.state.lock().bytes_sent += len;
let Some(tx) = self.audio_tx.lock().as_ref().cloned() else {
return Err(XfyunASRError::ConnectionFailed(
"websocket not open".into(),
));
};
self.pending_sends.fetch_add(1, Ordering::SeqCst);
if tx.send(buf).is_err() {
self.pending_sends.fetch_sub(1, Ordering::SeqCst);
}
}
// 等尾部音频也发完,再发结束标识(内容与文档一致,必须走 binary message)。
let drain_deadline = Instant::now() + Duration::from_millis(800);
while self.pending_sends.load(Ordering::SeqCst) > 0 {
let remaining = drain_deadline.saturating_duration_since(Instant::now());
if remaining.is_zero() {
log::warn!(
"[xfyun-asr] send_last_frame: tail {} 帧未发送完,超时强制结束",
self.pending_sends.load(Ordering::SeqCst)
);
break;
}
let _ = tokio::time::timeout(remaining, self.send_done.notified()).await;
}
send_binary(&self.writer, br#"{"end": true}"#.to_vec()).await
}
pub async fn await_final_result(&self) -> Result<RawTranscript, XfyunASRError> {
self.await_final_result_with_timeout(FINAL_RESULT_TIMEOUT)
.await
}
pub async fn await_final_result_with_timeout(
&self,
timeout: Duration,
) -> Result<RawTranscript, XfyunASRError> {
let rx = self.final_rx.lock().take();
let Some(rx) = rx else {
return Err(XfyunASRError::NoFinalResult);
};
match tokio::time::timeout(timeout, rx).await {
Ok(Ok(result)) => result,
Ok(Err(_)) => Err(XfyunASRError::NoFinalResult),
Err(_) => {
log::error!(
"[xfyun-asr] final result timed out after {} ms",
timeout.as_millis()
);
self.cancel();
Err(XfyunASRError::FinalResultTimeout)
}
}
}
pub fn cancel(&self) {
let runtime = {
let mut st = self.state.lock();
st.pending_audio.clear();
st.runtime.clone()
};
// 释放握手通道:open_session 若仍在等 started,会立刻收到 Err 返回。
*self.handshake_tx.lock() = None;
// 关闭音频队列 → worker 的 recv() 返回 None → 退出,不再 hold writer。
*self.audio_tx.lock() = None;
if let Some(runtime) = runtime {
let writer = Arc::clone(&self.writer);
runtime.spawn(async move {
let mut guard = writer.lock().await;
if let Some(mut ws) = guard.take() {
let _ = ws.close().await;
}
});
}
self.signal_error(XfyunASRError::NoFinalResult);
}
// ---- internals ----
fn handle_text_message(&self, text: &str) -> bool {
let value: Value = match serde_json::from_str(text) {
Ok(v) => v,
Err(e) => {
log::warn!("[xfyun-asr] invalid json event: {e}");
return true;
}
};
let action = value
.get("action")
.and_then(Value::as_str)
.unwrap_or_default();
match action {
"started" => {
self.mark_started();
true
}
"result" => {
self.record_result(&value);
true
}
"error" => {
let code = value
.get("code")
.and_then(Value::as_str)
.unwrap_or("")
.to_string();
let desc = value
.get("desc")
.and_then(Value::as_str)
.unwrap_or("")
.to_string();
log::error!("[xfyun-asr] server error code={code} desc={desc}");
let error = classify_server_error(&code, &desc);
// 握手阶段就报错:把错误直接交给 open_session 的 handshake 等待方。
let handed = self.handshake_tx.lock().take();
if let Some(tx) = handed {
let _ = tx.send(Err(error.clone()));
}
self.finish_error(error);
false
}
_ => true,
}
}
fn mark_started(&self) {
self.state.lock().started = true;
let tx = self.handshake_tx.lock().take();
if let Some(tx) = tx {
let _ = tx.send(Ok(()));
}
}
fn record_result(&self, value: &Value) {
let Some(data_str) = value.get("data").and_then(Value::as_str) else {
return;
};
let Ok(data) = serde_json::from_str::<Value>(data_str) else {
log::warn!("[xfyun-asr] result data is not valid JSON");
return;
};
let Some(seg_id) = data.get("seg_id").and_then(Value::as_i64) else {
return;
};
let Some(st) = data.get("cn").and_then(|c| c.get("st")) else {
return;
};
// `type` 字段文档为字符串("0" 最终 / "1" 中间)。防御性兼容数字形态
// (0/1):若服务端以数字返回而只认字符串,多句最终结果会被整体降级成
// 中间结果,收尾 fallback 只剩最后一句 —— 前句丢失。
let is_final = st
.get("type")
.and_then(|v| {
v.as_str()
.map(|s| s == "0")
.or_else(|| v.as_i64().map(|n| n == 0))
})
.unwrap_or(false);
let text = extract_words(st);
let trimmed = text.trim();
if trimmed.is_empty() {
return;
}
let mut state = self.state.lock();
state.last_result_text = trimmed.to_string();
if is_final {
// 最终结果:以 seg_id 去重覆盖,收尾按 seg_id 顺序拼接。
state
.final_segments
.insert(seg_id, trimmed.to_string());
state.partial_segments.remove(&seg_id);
} else {
state
.partial_segments
.insert(seg_id, trimmed.to_string());
}
}
/// 服务端断开连接:正常路径(全部结果已发完)或异常中断。
/// 握手尚未完成就被关闭时,优先把错误交给 open_session 的等待方。
fn finish_on_close(&self) {
let handshake = self.handshake_tx.lock().take();
if let Some(tx) = handshake {
let _ = tx.send(Err(XfyunASRError::ConnectionFailed(
"连接在握手完成前被关闭".to_string(),
)));
return;
}
self.finish_with_partial_or_error(XfyunASRError::NoFinalResult);
}
/// 有已识别内容(最终或中间结果)就兜底返回,否则报错 —— 与 Volcengine / Bailian
/// 的「服务端在 final 前断连不丢已识别文字」策略保持一致。
fn finish_with_partial_or_error(&self, error: XfyunASRError) {
let has_partial = {
let st = self.state.lock();
!st.last_result_text.trim().is_empty() || !st.partial_segments.is_empty()
};
if has_partial {
self.finish_success();
} else {
self.finish_error(error);
}
}
fn finish_success(&self) {
let (tx, text, duration_ms) = {
let mut st = self.state.lock();
if st.finished {
return;
}
st.finished = true;
st.pending_audio.clear();
let text = if st.final_segments.is_empty() {
st.last_result_text.clone()
} else {
let segments: Vec<String> = st.final_segments.values().cloned().collect();
super::mimo::join_transcript_chunks(&segments)
};
let duration_ms = if st.bytes_sent > 0 {
st.bytes_sent / BYTES_PER_MS
} else {
st.start
.map(|start| start.elapsed().as_millis() as u64)
.unwrap_or_default()
};
(st.final_tx.take(), text, duration_ms)
};
*self.audio_tx.lock() = None;
if let Some(tx) = tx {
let _ = tx.send(Ok(RawTranscript { text, duration_ms }));
}
self.close_writer();
}
fn signal_error(&self, error: XfyunASRError) {
let tx = {
let mut st = self.state.lock();
if st.finished {
return;
}
st.finished = true;
st.final_tx.take()
};
*self.audio_tx.lock() = None;
if let Some(tx) = tx {
let _ = tx.send(Err(error));
}
}
fn finish_error(&self, error: XfyunASRError) {
// 握手尚未完成就出错(如 receive loop 网络中断、服务端 error 消息)时,
// 必须把错误交给 open_session 的 handshake 等待方 —— 否则它只能空等
// HANDSHAKE_TIMEOUT(5s)才返回。已过握手的会话这里 take 到 None,幂等跳过。
let handshake = self.handshake_tx.lock().take();
if let Some(tx) = handshake {
let _ = tx.send(Err(error.clone()));
}
self.signal_error(error);
self.close_writer();
}
fn close_writer(&self) {
let writer = Arc::clone(&self.writer);
if let Some(handle) = self.state.lock().runtime.clone() {
handle.spawn(async move {
let mut guard = writer.lock().await;
if let Some(mut ws) = guard.take() {
let _ = ws.close().await;
}
});
}
}
}
impl AudioConsumer for XfyunStreamingASR {
fn consume_pcm_chunk(&self, pcm: &[u8]) {
let chunks: Vec<Vec<u8>> = {
let mut st = self.state.lock();
if !st.started || st.finished {
return;
}
st.pending_audio.extend_from_slice(pcm);
let mut out = Vec::new();
while st.pending_audio.len() >= TARGET_AUDIO_CHUNK_BYTES {
let chunk: Vec<u8> = st
.pending_audio
.drain(..TARGET_AUDIO_CHUNK_BYTES)
.collect();
st.bytes_sent += chunk.len() as u64;
out.push(chunk);
}
out
};
if chunks.is_empty() {
return;
}
let Some(tx) = self.audio_tx.lock().as_ref().cloned() else {
return;
};
for chunk in chunks {
// pending_sends 必须先 +1 再入队:否则 worker 可能先 recv + 发送 + 减 1,
// 把 usize 计数器 underflow。
self.pending_sends.fetch_add(1, Ordering::SeqCst);
if tx.send(chunk).is_err() {
if self.pending_sends.fetch_sub(1, Ordering::SeqCst) == 1 {
self.send_done.notify_waiters();
}
log::warn!("[xfyun-asr] audio queue closed; dropping subsequent frames");
return;
}
}
}
}
async fn send_binary(writer: &SharedWriter, data: Vec<u8>) -> Result<(), XfyunASRError> {
let mut guard = writer.lock().await;
let Some(ws) = guard.as_mut() else {
return Err(XfyunASRError::ConnectionFailed(
"websocket not open".to_string(),
));
};
ws.send(Message::Binary(data))
.await
.map_err(|e| XfyunASRError::ConnectionFailed(e.to_string()))
}
fn connect_url(credentials: &XfyunCredentials) -> String {
let ts = chrono::Utc::now().timestamp().to_string();
let signa = compute_signa(&credentials.app_id, &credentials.api_key, &ts);
let mut url = url::Url::parse(DEFAULT_ENDPOINT).expect("static endpoint parses");
url.query_pairs_mut()
.append_pair("appid", credentials.app_id.trim())
.append_pair("ts", &ts)
.append_pair("signa", &signa)
.append_pair("lang", DEFAULT_LANG);
url.to_string()
}
/// `signa = Base64(HmacSHA1(MD5(appid + ts), apiKey))`,与讯飞开放平台文档公式一致。
pub fn compute_signa(app_id: &str, api_key: &str, ts: &str) -> String {
let base = format!("{app_id}{ts}");
let md5_hex = md5_hex(base.as_bytes());
let mut mac = Hmac::<Sha1>::new_from_slice(api_key.trim().as_bytes())
.expect("HMAC accepts any key length");
mac.update(md5_hex.as_bytes());
base64::engine::general_purpose::STANDARD.encode(mac.finalize().into_bytes())
}
fn md5_hex(input: &[u8]) -> String {
let digest = Md5::digest(input);
digest
.iter()
.map(|byte| format!("{byte:02x}"))
.collect::<String>()
}
/// 从 `cn.st` 节点提取全部词:`rt[].ws[].cw[]` 取第一个候选的 `w` 依次拼接。
fn extract_words(st: &Value) -> String {
let mut out = String::new();
let Some(rt) = st.get("rt").and_then(Value::as_array) else {
return out;
};
for sentence in rt {
let Some(ws) = sentence.get("ws").and_then(Value::as_array) else {
continue;
};
for word in ws {
let Some(cw) = word.get("cw").and_then(Value::as_array) else {
continue;
};
for candidate in cw {
if let Some(w) = candidate.get("w").and_then(Value::as_str) {
out.push_str(w);
break;
}
}
}
}
out
}
/// 把讯飞错误码归类为对用户可读的类别:鉴权/授权问题与并发限制单独分类,
/// 其余归通用识别失败(避免 capsule 文案笼统指向「网络失败」)。
fn classify_server_error(code: &str, desc: &str) -> XfyunASRError {
match code {
"10105" | "10110" => XfyunASRError::AuthRejected(format!("{code} {desc}")),
"10800" => XfyunASRError::RateLimited(format!("{code} {desc}")),
_ => XfyunASRError::TaskFailed(format!("{code} {desc}")),
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn signa_matches_official_documentation_example() {
// 官方文档示例:appid=595f23df,ts=1512041814,apiKey=d9f4aa7ea6d94faca62cd88a28fd5234
// → signa = IrrzsJeOFk1NGfJHW6SkHUoN9CU=
let signa = compute_signa(
"595f23df",
"d9f4aa7ea6d94faca62cd88a28fd5234",
"1512041814",
);
assert_eq!(signa, "IrrzsJeOFk1NGfJHW6SkHUoN9CU=");
}
#[test]
fn md5_hex_matches_known_vector() {
// MD5("595f23df1512041814") = 0829d4012497c14a30e7e72aeebe565e(文档示例)
assert_eq!(
md5_hex(b"595f23df1512041814"),
"0829d4012497c14a30e7e72aeebe565e"
);
}
#[test]
fn connect_url_contains_encoded_auth_params() {
let creds = XfyunCredentials {
app_id: "595f23df".into(),
api_key: "d9f4aa7ea6d94faca62cd88a28fd5234".into(),
};
let url = connect_url(&creds);
assert!(url.starts_with("wss://rtasr.xfyun.cn/v1/ws?"));
assert!(url.contains("appid=595f23df"));
assert!(url.contains("ts="));
assert!(url.contains("signa="));
assert!(url.contains("lang=cn"));
}
#[test]
fn extract_words_joins_candidates_in_order() {
let data = serde_json::json!({
"cn": {
"st": {
"bg": "820",
"ed": "0",
"rt": [
{"ws": [
{"cw": [{"w": "啊", "wp": "n"}], "wb": 0, "we": 0},
{"cw": [{"w": "喂", "wp": "n"}], "wb": 0, "we": 0},
{"cw": [{"w": "!", "wp": "p"}], "wb": 0, "we": 0},
{"cw": [{"w": "你好", "wp": "n"}], "wb": 0, "we": 0}
]}
],
"type": "1"
}
},
"seg_id": 5
});
let text = extract_words(&data["cn"]["st"]);
assert_eq!(text, "啊喂!你好");
}
#[test]
fn record_result_stores_final_and_partial_separately() {
let asr = XfyunStreamingASR::new(XfyunCredentials {
app_id: "app".into(),
api_key: "key".into(),
});
asr.record_result(&serde_json::json!({
"action": "result",
"data": "{\"cn\":{\"st\":{\"rt\":[{\"ws\":[{\"cw\":[{\"w\":\"中间\"}]}]}],\"type\":\"1\"}},\"seg_id\":1}"
}));
{
let st = asr.state.lock();
assert!(st.final_segments.is_empty());
assert_eq!(st.partial_segments.get(&1).unwrap(), "中间");
}
asr.record_result(&serde_json::json!({
"action": "result",
"data": "{\"cn\":{\"st\":{\"rt\":[{\"ws\":[{\"cw\":[{\"w\":\"最终\"}]}]}],\"type\":\"0\"}},\"seg_id\":1}"
}));
{
let st = asr.state.lock();
assert_eq!(st.final_segments.get(&1).unwrap(), "最终");
assert!(st.partial_segments.is_empty(), "final 应清除同 seg 的 partial");
}
}
#[test]
fn record_result_accepts_numeric_type_for_final() {
// 防御性:服务端若以数字 0/1 返回 type(而非文档字符串 "0"/"1"),
// 最终结果仍要落 final_segments,不能整体降级成中间结果丢句。
let asr = XfyunStreamingASR::new(XfyunCredentials {
app_id: "app".into(),
api_key: "key".into(),
});
asr.record_result(&serde_json::json!({
"action": "result",
"data": "{\"cn\":{\"st\":{\"rt\":[{\"ws\":[{\"cw\":[{\"w\":\"第一句\"}]}]}],\"type\":0}},\"seg_id\":1}"
}));
asr.record_result(&serde_json::json!({
"action": "result",
"data": "{\"cn\":{\"st\":{\"rt\":[{\"ws\":[{\"cw\":[{\"w\":\"第二句\"}]}]}],\"type\":0}},\"seg_id\":2}"
}));
let st = asr.state.lock();
assert_eq!(st.final_segments.len(), 2);
assert_eq!(st.final_segments.get(&1).unwrap(), "第一句");
assert_eq!(st.final_segments.get(&2).unwrap(), "第二句");
assert!(st.partial_segments.is_empty());
}
#[test]
fn duplicate_final_segment_overwrites_not_duplicates() {
let asr = XfyunStreamingASR::new(XfyunCredentials {
app_id: "app".into(),
api_key: "key".into(),
});
let event = |text: &str| {
serde_json::json!({
"action": "result",
"data": format!(
"{{\"cn\":{{\"st\":{{\"rt\":[{{\"ws\":[{{\"cw\":[{{\"w\":\"{text}\"}}]}}]}}],\"type\":\"0\"}}}},\"seg_id\":2}}"
)
})
};
asr.record_result(&event("第一版"));
asr.record_result(&event("第二版"));
let st = asr.state.lock();
assert_eq!(st.final_segments.len(), 1);
assert_eq!(st.final_segments.get(&2).unwrap(), "第二版");
}
#[test]
fn auth_and_license_errors_classify_as_auth_rejected() {
assert!(matches!(
classify_server_error("10105", "illegal access|illegal client_ip"),
XfyunASRError::AuthRejected(_)
));
assert!(matches!(
classify_server_error("10110", "no license"),
XfyunASRError::AuthRejected(_)
));
}
#[test]
fn concurrency_limit_classifies_as_rate_limited() {
assert!(matches!(
classify_server_error("10800", "over max connect limit"),
XfyunASRError::RateLimited(_)
));
}
#[test]
fn generic_errors_stay_task_failed() {
assert!(matches!(
classify_server_error("37005", "no audio data"),
XfyunASRError::TaskFailed(_)
));
}
#[test]
fn credentials_require_both_fields_trimmed() {
let ok = XfyunCredentials {
app_id: "app".into(),
api_key: "key".into(),
};
assert!(ok.auth_ok());
assert!(!XfyunCredentials { app_id: "".into(), api_key: "key".into() }.auth_ok());
assert!(!XfyunCredentials { app_id: "app".into(), api_key: " ".into() }.auth_ok());
}
#[test]
fn finish_error_notifies_pending_handshake_waiter() {
// 握手前出错(如网络中断)必须唤醒 open_session 的 handshake 等待方,
// 否则其空等 HANDSHAKE_TIMEOUT。已过握手的会话 take 到 None,幂等。
let asr = XfyunStreamingASR::new(XfyunCredentials {
app_id: "app".into(),
api_key: "key".into(),
});
let (tx, mut rx) = oneshot::channel();
*asr.handshake_tx.lock() = Some(tx);
asr.finish_error(XfyunASRError::ConnectionFailed("boom".into()));
match rx.try_recv() {
Ok(Err(XfyunASRError::ConnectionFailed(_))) => {}
other => panic!("handshake 等待方应收到错误,实际: {other:?}"),
}
// 已 take:重复调用不 panic、不重复通知。
asr.finish_error(XfyunASRError::ConnectionFailed("again".into()));
assert!(asr.handshake_tx.lock().is_none());
}
}
@@ -76,6 +76,9 @@ pub(crate) fn asr_configured_for_provider(provider: &str, snap: &CredentialsSnap
configured(&snap.asr_endpoint) && configured(&snap.asr_model)
}
AsrConfiguredFields::VolcAppKey => volcengine_configured(snap),
AsrConfiguredFields::XfyunAppKey => {
configured(&snap.xfyun_app_id) && configured(&snap.xfyun_api_key)
}
}
}
@@ -213,6 +216,8 @@ pub async fn set_credential(
| CredentialAccount::AsrEndpoint
| CredentialAccount::AsrModel
| CredentialAccount::AsrVocabularyId
| CredentialAccount::XfyunAppId
| CredentialAccount::XfyunApiKey
) {
return Err("provider-scoped credential must be an ASR account".to_string());
}
@@ -355,6 +360,8 @@ fn parse_account(s: &str) -> Result<CredentialAccount, String> {
"asr.endpoint" => Ok(CredentialAccount::AsrEndpoint),
"asr.model" => Ok(CredentialAccount::AsrModel),
"asr.vocabulary_id" => Ok(CredentialAccount::AsrVocabularyId),
"xfyun.app_id" => Ok(CredentialAccount::XfyunAppId),
"xfyun.api_key" => Ok(CredentialAccount::XfyunApiKey),
_ => Err(format!("unknown account: {s}")),
}
}
@@ -280,6 +280,9 @@ async fn validate_asr_provider() -> Result<(), String> {
if active_asr == crate::asr::elevenlabs::PROVIDER_ID {
return validate_elevenlabs_asr_provider().await;
}
if active_asr == crate::asr::xfyun::PROVIDER_ID {
return validate_xfyun_asr_provider().await;
}
// StepFun 一入口双协议:`*-stream` 模型走实时 WS 验证,其余走批式
// /audio/transcriptions(与 build 侧 resolve_effective_asr_provider 同判据)。
if active_asr == "stepfun" || active_asr == crate::asr::stepfun_realtime::PROVIDER_ID {
@@ -301,6 +304,38 @@ async fn validate_asr_provider() -> Result<(), String> {
validate_asr_transcription(&config, model.trim()).await
}
/// 讯飞 RTASR 验证:真连 + 500ms 静音 + 收尾。鉴权错误(10105 / 10110)在握手阶段
/// 即返回;纯静音会话服务端可能直接关闭且不返回任何 result(等价于「没说话」),
/// 这类 `NoFinalResult` 不算验证失败 —— 握手成功已经证明 AppID/APIKey 有效。
async fn validate_xfyun_asr_provider() -> Result<(), String> {
let app_id = CredentialsVault::get(CredentialAccount::XfyunAppId)
.map_err(|e| e.to_string())?
.unwrap_or_default();
if app_id.trim().is_empty() {
return Err("讯飞 AppID 为空".to_string());
}
let api_key = CredentialsVault::get(CredentialAccount::XfyunApiKey)
.map_err(|e| e.to_string())?
.unwrap_or_default();
if api_key.trim().is_empty() {
return Err("讯飞 API Key 为空".to_string());
}
let asr = std::sync::Arc::new(crate::asr::XfyunStreamingASR::new(
crate::asr::XfyunCredentials { app_id, api_key },
));
asr.open_session().await.map_err(|e| e.to_string())?;
crate::asr::AudioConsumer::consume_pcm_chunk(
&*asr,
&vec![0u8; crate::asr::xfyun::TARGET_AUDIO_CHUNK_BYTES * 5],
);
asr.send_last_frame().await.map_err(|e| e.to_string())?;
match asr.await_final_result().await {
Ok(_) => Ok(()),
Err(crate::asr::xfyun::XfyunASRError::NoFinalResult) => Ok(()),
Err(e) => Err(e.to_string()),
}
}
/// StepFun 实时 WS 验证:真连 + session.update + 500ms 静音 + 收尾。
/// 协议无 finish 事件,收尾走静音帧 + 宽限期(纯静音会话以空文本成功返回,
/// 见 stepfun_realtime 模块注释),全程 ~2s。
@@ -194,6 +194,8 @@ enum ActiveAsr {
Qwen3Realtime(Arc<Qwen3RealtimeASR>),
/// 阶跃星辰 StepAudio 实时(OpenAI Realtime 风格 WS,收尾靠静音帧驱动 VAD)。
StepfunRealtime(Arc<crate::asr::StepfunRealtimeASR>),
/// 讯飞开放平台实时语音转写(RTASR)流式。
Xfyun(Arc<crate::asr::XfyunStreamingASR>),
#[cfg(target_os = "windows")]
FoundryLocalWhisper(Arc<FoundryLocalWhisperAsr>),
/// Windows sherpa-onnx 本地 ASR(offline batch + 实验 online streaming)。
@@ -237,6 +239,7 @@ pub(crate) enum ActiveAsrProviderKind {
ElevenLabs,
WhisperCompatible,
Volcengine,
Xfyun,
}
/// 「能否开始一次会话」所需的凭据形态(对应 `ensure_asr_credentials` 预检门)。
@@ -246,6 +249,8 @@ pub(crate) enum AsrPreflightCredential {
AsrApiKey,
/// 需要火山引擎 App Key + Access Key。
VolcAppKey,
/// 需要讯飞 AppID + APIKey。
XfyunAppKey,
}
/// 概览页「已配置 / 未配置」状态所需的字段(对应 `asr_configured_for_provider`)。
@@ -260,6 +265,8 @@ pub(crate) enum AsrConfiguredFields {
EndpointModelOnly,
/// 火山引擎三件套。
VolcAppKey,
/// 讯飞 AppID + APIKey。
XfyunAppKey,
}
impl ActiveAsrProviderKind {
@@ -273,6 +280,7 @@ impl ActiveAsrProviderKind {
| ActiveAsrProviderKind::ElevenLabs
| ActiveAsrProviderKind::WhisperCompatible => AsrPreflightCredential::AsrApiKey,
ActiveAsrProviderKind::Volcengine => AsrPreflightCredential::VolcAppKey,
ActiveAsrProviderKind::Xfyun => AsrPreflightCredential::XfyunAppKey,
}
}
@@ -292,6 +300,7 @@ impl ActiveAsrProviderKind {
ActiveAsrProviderKind::WhisperCompatible
| ActiveAsrProviderKind::StepfunRealtime => AsrConfiguredFields::EndpointModelOnly,
ActiveAsrProviderKind::Volcengine => AsrConfiguredFields::VolcAppKey,
ActiveAsrProviderKind::Xfyun => AsrConfiguredFields::XfyunAppKey,
}
}
}
@@ -311,6 +320,8 @@ pub(crate) fn active_asr_provider_kind(id: &str) -> ActiveAsrProviderKind {
ActiveAsrProviderKind::ElevenLabs
} else if is_whisper_compatible_provider(id) {
ActiveAsrProviderKind::WhisperCompatible
} else if is_xfyun_provider(id) {
ActiveAsrProviderKind::Xfyun
} else {
ActiveAsrProviderKind::Volcengine
}
@@ -2046,6 +2057,13 @@ impl Coordinator {
.map_err(|_| "重新转录超时".to_string())?
.map_err(|e| e.to_string())?
}
ActiveAsr::Xfyun(asr) => {
asr.send_last_frame().await.map_err(|e| e.to_string())?;
tokio::time::timeout(timeout, asr.await_final_result())
.await
.map_err(|_| "重新转录超时".to_string())?
.map_err(|e| e.to_string())?
}
ActiveAsr::Whisper(w) => tokio::time::timeout(timeout, w.transcribe())
.await
.map_err(|_| "重新转录超时".to_string())?
@@ -2662,6 +2680,18 @@ fn read_volc_credentials() -> VolcengineCredentials {
}
}
fn read_xfyun_credentials() -> crate::asr::XfyunCredentials {
let app_id = CredentialsVault::get(CredentialAccount::XfyunAppId)
.ok()
.flatten()
.unwrap_or_default();
let api_key = CredentialsVault::get(CredentialAccount::XfyunApiKey)
.ok()
.flatten()
.unwrap_or_default();
crate::asr::XfyunCredentials { app_id, api_key }
}
fn enabled_hotwords(inner: &Arc<Inner>) -> Vec<DictionaryHotword> {
inner
.vocab
@@ -3185,6 +3215,7 @@ mod tests {
assert_eq!(ElevenLabs.preflight_credential(), AsrApiKey);
assert_eq!(WhisperCompatible.preflight_credential(), AsrApiKey);
assert_eq!(Volcengine.preflight_credential(), VolcAppKey);
assert_eq!(Xfyun.preflight_credential(), XfyunAppKey);
}
#[test]
@@ -3326,6 +3357,7 @@ mod tests {
assert_eq!(ElevenLabs.configured_fields(), ApiKeyOnly);
assert_eq!(WhisperCompatible.configured_fields(), EndpointModelOnly);
assert_eq!(Volcengine.configured_fields(), VolcAppKey);
assert_eq!(Xfyun.configured_fields(), XfyunAppKey);
}
#[cfg(target_os = "windows")]
@@ -136,6 +136,14 @@ pub(super) fn ensure_asr_credentials() -> Result<(), String> {
}
}
}
AsrPreflightCredential::XfyunAppKey => {
let creds = read_xfyun_credentials();
if creds.auth_ok() {
Ok(())
} else {
Err("请先在设置中填写讯飞 AppID 和 API Key".to_string())
}
}
}
}
@@ -415,6 +423,10 @@ pub(super) fn is_elevenlabs_provider(id: &str) -> bool {
id == crate::asr::elevenlabs::PROVIDER_ID
}
pub(super) fn is_xfyun_provider(id: &str) -> bool {
id == crate::asr::xfyun::PROVIDER_ID
}
pub(super) fn apply_chinese_script_preference(text: &str, pref: ChineseScriptPreference) -> String {
if text.is_empty() {
return String::new();
@@ -453,6 +465,10 @@ pub(super) enum QaAsrStart {
asr: Arc<crate::asr::StepfunRealtimeASR>,
bridge: Arc<DeferredAsrBridge>,
},
Xfyun {
asr: Arc<crate::asr::XfyunStreamingASR>,
bridge: Arc<DeferredAsrBridge>,
},
Ready {
active: ActiveAsr,
consumer: Arc<dyn crate::recorder::AudioConsumer>,
@@ -466,6 +482,7 @@ impl QaAsrStart {
QaAsrStart::Bailian { asr, .. } => ActiveAsr::Bailian(Arc::clone(asr)),
QaAsrStart::Qwen3Realtime { asr, .. } => ActiveAsr::Qwen3Realtime(Arc::clone(asr)),
QaAsrStart::StepfunRealtime { asr, .. } => ActiveAsr::StepfunRealtime(Arc::clone(asr)),
QaAsrStart::Xfyun { asr, .. } => ActiveAsr::Xfyun(Arc::clone(asr)),
QaAsrStart::Ready { active, .. } => active.clone(),
}
}
@@ -476,6 +493,7 @@ impl QaAsrStart {
QaAsrStart::Bailian { bridge, .. } => Arc::clone(bridge) as _,
QaAsrStart::Qwen3Realtime { bridge, .. } => Arc::clone(bridge) as _,
QaAsrStart::StepfunRealtime { bridge, .. } => Arc::clone(bridge) as _,
QaAsrStart::Xfyun { bridge, .. } => Arc::clone(bridge) as _,
QaAsrStart::Ready { consumer, .. } => Arc::clone(consumer),
}
}
@@ -516,6 +534,15 @@ impl QaAsrStart {
);
Ok(())
}
QaAsrStart::Xfyun { asr, bridge } => {
asr.open_session().await.map_err(|e| e.to_string())?;
let target: Arc<dyn crate::asr::AudioConsumer> = Arc::clone(asr) as _;
let flushed = bridge.attach(target);
log::info!(
"[coord] QA iFlytek ASR connected; flushed {flushed} deferred audio bytes"
);
Ok(())
}
QaAsrStart::Ready { .. } => Ok(()),
}
}
@@ -710,5 +737,16 @@ pub(super) async fn build_qa_asr_start(
label,
))
}
ActiveAsrProviderKind::Xfyun => {
let creds = read_xfyun_credentials();
let label = AsrCallLabel::new(effective_asr.clone(), None);
Ok((
QaAsrStart::Xfyun {
asr: Arc::new(crate::asr::XfyunStreamingASR::new(creds)),
bridge: Arc::new(DeferredAsrBridge::new()),
},
label,
))
}
}
}
@@ -1861,7 +1861,8 @@ pub(super) async fn begin_session_as(
| ActiveAsrProviderKind::DashScopeMultimodal
| ActiveAsrProviderKind::ElevenLabs
| ActiveAsrProviderKind::WhisperCompatible
| ActiveAsrProviderKind::Volcengine => {}
| ActiveAsrProviderKind::Volcengine
| ActiveAsrProviderKind::Xfyun => {}
}
if is_bailian_provider(&effective_asr) {
@@ -2173,6 +2174,82 @@ pub(super) async fn begin_session_as(
let consumer: Arc<dyn crate::recorder::AudioConsumer> = whisper;
start_recorder_and_enter_listening(inner, current_session_id, &active_asr, consumer)
.await?;
} else if is_xfyun_provider(&effective_asr) {
// 讯飞 RTASR 实时流式:与 Bailian / 火山同构(open_session → 录音 → end → final)。
let creds = read_xfyun_credentials();
let asr_call_label = AsrCallLabel::new(effective_asr.clone(), None);
let asr = Arc::new(crate::asr::XfyunStreamingASR::new(creds));
let bridge = Arc::new(DeferredAsrBridge::new());
let consumer: Arc<dyn crate::recorder::AudioConsumer> = bridge.clone();
store_asr_for_session(
inner,
current_session_id,
ActiveAsr::Xfyun(Arc::clone(&asr)),
asr_call_label,
);
start_recorder_for_starting(inner, current_session_id, &active_asr, consumer).await?;
if let Err(e) = asr.open_session().await {
log::error!("[coord] open iFlytek ASR session failed: {e}");
match startup_race_status_for_starting(inner, current_session_id) {
StartupRaceStatus::StaleContinuation => {
log::info!(
"[coord] stale iFlytek ASR open_session error from session {current_session_id} — ignoring"
);
asr.cancel();
discard_startup_resources_for_session(inner, current_session_id);
restore_prepared_windows_ime_session(inner, current_session_id);
return Ok(());
}
StartupRaceStatus::CancelRaced => {
asr.cancel();
discard_startup_resources_for_session(inner, current_session_id);
restore_prepared_windows_ime_session(inner, current_session_id);
set_phase_idle_if_session_matches(inner, current_session_id);
return Ok(());
}
StartupRaceStatus::ActiveStarting => {
asr.cancel();
}
}
discard_startup_resources_for_session(inner, current_session_id);
emit_capsule(
inner,
CapsuleState::Error,
0.0,
0,
Some(format!("ASR 连接失败: {e}")),
None,
);
restore_prepared_windows_ime_session(inner, current_session_id);
set_phase_idle_if_session_matches(inner, current_session_id);
schedule_capsule_idle(inner, CAPSULE_AUTO_HIDE_DELAY_MS);
return Err(e.to_string());
}
match startup_race_status_for_starting(inner, current_session_id) {
StartupRaceStatus::ActiveStarting => {}
StartupRaceStatus::CancelRaced => {
log::info!("[coord] cancel raced during iFlytek ASR open_session — aborting begin");
asr.cancel();
discard_startup_resources_for_session(inner, current_session_id);
restore_prepared_windows_ime_session(inner, current_session_id);
set_phase_idle_if_session_matches(inner, current_session_id);
return Ok(());
}
StartupRaceStatus::StaleContinuation => {
log::info!(
"[coord] stale iFlytek ASR open_session continuation from session {current_session_id} — ignoring"
);
asr.cancel();
discard_startup_resources_for_session(inner, current_session_id);
restore_prepared_windows_ime_session(inner, current_session_id);
return Ok(());
}
}
let target: Arc<dyn crate::asr::AudioConsumer> = asr;
let flushed_bytes = bridge.attach(target);
log::info!("[coord] iFlytek ASR connected; flushed {flushed_bytes} deferred audio bytes");
finish_starting_session(inner, current_session_id).await;
} else {
let hotwords = enabled_hotwords(inner);
let creds = read_volc_credentials();
@@ -3048,6 +3125,34 @@ pub(super) async fn end_session(inner: &Arc<Inner>) -> Result<(), String> {
}
}
}
ActiveAsr::Xfyun(asr) => {
debug_assert!(uses_global_timeout);
if let Err(e) = asr.send_last_frame().await {
log::error!("[coord] iFlytek ASR send last frame failed: {e}");
}
let timeout_duration =
std::time::Duration::from_secs(COORDINATOR_GLOBAL_TIMEOUT_SECS);
match tokio::time::timeout(timeout_duration, asr.await_final_result()).await {
Ok(Ok(r)) => Ok(r),
Ok(Err(e)) => {
log::error!("[coord] iFlytek ASR await final failed: {e}");
// 关闭 WebSocket 连接,避免流式 ASR 资源泄漏
asr.cancel();
Err(TranscribeFail::new(format!("识别失败: {e}"), e.to_string()))
}
Err(_) => {
log::error!(
"[coord] iFlytek ASR 全局超时 {} 秒",
COORDINATOR_GLOBAL_TIMEOUT_SECS
);
asr.cancel();
Err(TranscribeFail::new(
"识别超时".to_string(),
"xfyun global timeout".to_string(),
))
}
}
}
#[cfg(target_os = "windows")]
ActiveAsr::FoundryLocalWhisper(local) => {
debug_assert!(!uses_global_timeout);
@@ -459,6 +459,21 @@ pub(super) async fn transcribe_overlay_dictation_asr(
}
}
}
ActiveAsr::Xfyun(asr) => {
debug_assert!(uses_global_timeout);
if let Err(error) = asr.send_last_frame().await {
log::error!("[coord] overlay QA: iFlytek ASR send last frame failed: {error}");
}
let timeout_duration = std::time::Duration::from_secs(COORDINATOR_GLOBAL_TIMEOUT_SECS);
match tokio::time::timeout(timeout_duration, asr.await_final_result()).await {
Ok(Ok(raw)) => Ok(raw),
Ok(Err(error)) => Err(error.to_string()),
Err(_) => {
asr.cancel();
Err("xfyun global timeout".to_string())
}
}
}
ActiveAsr::Whisper(whisper) => {
debug_assert!(uses_global_timeout);
let timeout_duration = std::time::Duration::from_secs(COORDINATOR_GLOBAL_TIMEOUT_SECS);
@@ -1085,6 +1100,30 @@ pub(super) async fn end_qa_session(inner: &Arc<Inner>) -> Result<(), String> {
}
}
}
ActiveAsr::Xfyun(asr) => {
debug_assert!(uses_global_timeout);
if let Err(e) = asr.send_last_frame().await {
log::error!("[coord] QA: iFlytek ASR send last frame failed: {e}");
}
let timeout_duration = std::time::Duration::from_secs(COORDINATOR_GLOBAL_TIMEOUT_SECS);
match tokio::time::timeout(timeout_duration, asr.await_final_result()).await {
Ok(Ok(r)) => r,
Ok(Err(e)) => {
log::error!("[coord] QA: iFlytek ASR await final failed: {e}");
finish_qa_with_error_if_current(inner, session_id, format!("识别失败: {e}"));
return Err(e.to_string());
}
Err(_) => {
log::error!(
"[coord] QA: iFlytek ASR 全局超时 {} 秒",
COORDINATOR_GLOBAL_TIMEOUT_SECS
);
asr.cancel();
finish_qa_with_error_if_current(inner, session_id, "识别超时".to_string());
return Err("xfyun global timeout".to_string());
}
}
}
ActiveAsr::Qwen3Realtime(asr) => {
debug_assert!(uses_global_timeout);
if let Err(e) = asr.send_last_frame().await {
@@ -91,6 +91,7 @@ pub(super) fn cancel_active_asr(asr: ActiveAsr) {
ActiveAsr::Bailian(b) => b.cancel(),
ActiveAsr::Qwen3Realtime(q) => q.cancel(),
ActiveAsr::StepfunRealtime(s) => s.cancel(),
ActiveAsr::Xfyun(x) => x.cancel(),
#[cfg(target_os = "windows")]
ActiveAsr::FoundryLocalWhisper(local) => local.cancel(),
#[cfg(target_os = "windows")]
@@ -220,6 +220,12 @@ struct CredsAsrEntry {
volcengineApiKey: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
vocabularyId: Option<String>,
/// 讯飞开放平台应用 ID(RTASR/IFASR 鉴权用)。
#[serde(skip_serializing_if = "Option::is_none")]
xfyunAppId: Option<String>,
/// 讯飞实时语音转写 APIKey(接口密钥)。
#[serde(skip_serializing_if = "Option::is_none")]
xfyunApiKey: Option<String>,
}
impl CredsAsrEntry {
@@ -233,6 +239,8 @@ impl CredsAsrEntry {
&& self.authMode.as_deref().unwrap_or("").is_empty()
&& self.volcengineApiKey.as_deref().unwrap_or("").is_empty()
&& self.vocabularyId.as_deref().unwrap_or("").is_empty()
&& self.xfyunAppId.as_deref().unwrap_or("").is_empty()
&& self.xfyunApiKey.as_deref().unwrap_or("").is_empty()
}
}
@@ -1126,6 +1134,8 @@ fn lookup_account(root: &CredsRoot, account: CredentialAccount) -> Option<String
CredentialAccount::AsrEndpoint => asr.and_then(|e| pick(&e.baseURL)),
CredentialAccount::AsrModel => asr.and_then(|e| pick(&e.model)),
CredentialAccount::AsrVocabularyId => asr.and_then(|e| pick(&e.vocabularyId)),
CredentialAccount::XfyunAppId => asr.and_then(|e| pick(&e.xfyunAppId)),
CredentialAccount::XfyunApiKey => asr.and_then(|e| pick(&e.xfyunApiKey)),
}
}
@@ -1182,6 +1192,14 @@ fn write_account(root: &mut CredsRoot, account: CredentialAccount, value: Option
let entry = root.providers.asr.entry(asr_id).or_default();
entry.vocabularyId = normalized;
}
CredentialAccount::XfyunAppId => {
let entry = root.providers.asr.entry(asr_id).or_default();
entry.xfyunAppId = normalized;
}
CredentialAccount::XfyunApiKey => {
let entry = root.providers.asr.entry(asr_id).or_default();
entry.xfyunApiKey = normalized;
}
}
}
@@ -1204,6 +1222,10 @@ pub enum CredentialAccount {
AsrModel,
/// Active ASR provider's optional hotword vocabulary ID.
AsrVocabularyId,
/// 讯飞开放平台应用 ID。
XfyunAppId,
/// 讯飞实时语音转写 APIKey。
XfyunApiKey,
}
impl CredentialAccount {
@@ -1224,6 +1246,8 @@ impl CredentialAccount {
CredentialAccount::AsrEndpoint => "asr.endpoint",
CredentialAccount::AsrModel => "asr.model",
CredentialAccount::AsrVocabularyId => "asr.vocabulary_id",
CredentialAccount::XfyunAppId => "xfyun.app_id",
CredentialAccount::XfyunApiKey => "xfyun.api_key",
}
}
@@ -1241,6 +1265,8 @@ impl CredentialAccount {
CredentialAccount::AsrEndpoint,
CredentialAccount::AsrModel,
CredentialAccount::AsrVocabularyId,
CredentialAccount::XfyunAppId,
CredentialAccount::XfyunApiKey,
]
}
}
@@ -1256,6 +1282,8 @@ pub struct CredentialsSnapshot {
pub asr_api_key: Option<String>,
pub asr_endpoint: Option<String>,
pub asr_model: Option<String>,
pub xfyun_app_id: Option<String>,
pub xfyun_api_key: Option<String>,
pub ark_api_key: Option<String>,
pub ark_model_id: Option<String>,
pub ark_endpoint: Option<String>,
@@ -1476,6 +1504,8 @@ impl CredentialsVault {
asr_api_key: lookup_account(&root, CredentialAccount::AsrApiKey),
asr_endpoint: lookup_account(&root, CredentialAccount::AsrEndpoint),
asr_model: lookup_account(&root, CredentialAccount::AsrModel),
xfyun_app_id: lookup_account(&root, CredentialAccount::XfyunAppId),
xfyun_api_key: lookup_account(&root, CredentialAccount::XfyunApiKey),
ark_api_key: lookup_account(&root, CredentialAccount::ArkApiKey),
ark_model_id: lookup_account(&root, CredentialAccount::ArkModelId),
ark_endpoint: lookup_account(&root, CredentialAccount::ArkEndpoint),
+4
View File
@@ -859,6 +859,7 @@ export const en: typeof zhCN = {
asrWhisper: 'OpenAI Whisper (compatible)',
asrOpenrouter: 'OpenRouter Whisper',
asrXiaomiMimo: 'Xiaomi MiMo ASR',
asrIflytek: 'iFlytek Realtime ASR',
asrElevenLabs: 'ElevenLabs Scribe',
asrSherpaOnnxLocal: 'Local sherpa-onnx (experimental)',
asrFoundryLocalWhisper: 'Local Whisper (Foundry Local)',
@@ -875,6 +876,9 @@ export const en: typeof zhCN = {
volcengineAuthModeApiKey: 'Ark API Key',
volcengineMappingNote: 'Secret Key is not required right now. Resource ID defaults to volc.seedasr.sauc.duration.',
volcengineApiKeyNote: 'Authenticate with Volcengine Ark speech model API Key, no APP ID needed. Resource ID defaults to volc.seedasr.sauc.duration.',
xfyunAppIdLabel: 'AppID',
xfyunApiKeyLabel: 'API Key',
xfyunNote: 'Get AppID and API Key from the iFlytek Open Platform "Realtime ASR" service page. Audio is 16 kHz / 16-bit / mono PCM; the standard API has no hotword parameter (configure personalized hotwords in the iFlytek console), and the language defaults to Mandarin Chinese.',
localAsrActiveNotice: 'Local ASR ({{name}}) is currently active. Switch or disable it from the Advanced tab.',
localAsrTakeoverHint: 'Once "{{name}}" is enabled, the ASR provider will be taken over.',
asrProviderTakenOver: 'A local engine is active. Pick another provider in the dropdown above to switch (the local engine stops automatically); manage local models under Advanced → Local models.',
+4
View File
@@ -861,6 +861,7 @@ export const ja: typeof zhCN = {
asrWhisper: 'OpenAI Whisper(互換)',
asrOpenrouter: 'OpenRouter Whisper',
asrXiaomiMimo: 'Xiaomi MiMo ASR',
asrIflytek: 'iFlytek リアルタイム音声認識',
asrElevenLabs: 'ElevenLabs Scribe',
asrSherpaOnnxLocal: 'ローカル sherpa-onnx(実験的)',
asrFoundryLocalWhisper: 'ローカル Whisper(Foundry Local)',
@@ -877,6 +878,9 @@ export const ja: typeof zhCN = {
volcengineAuthModeApiKey: 'Ark API Key',
volcengineMappingNote: 'Secret Key は現在不要。Resource ID のデフォルトは volc.seedasr.sauc.duration。',
volcengineApiKeyNote: 'Volcengine Ark 音声モデルの API Key で認証、APP ID は不要です。Resource ID のデフォルトは volc.seedasr.sauc.duration。',
xfyunAppIdLabel: 'AppID',
xfyunApiKeyLabel: 'API Key',
xfyunNote: 'iFlytek オープンプラットフォームの「リアルタイム音声認識」サービスページで AppID と API Key を取得します。音声は 16kHz / 16bit / モノラル PCM。標準版 API にホットワード引数はありません(iFlytek コンソールで個別ホットワードを設定)。言語はデフォルトで中国語(普通話)です。',
localAsrActiveNotice: '現在「{{name}}」を使用中。「詳細設定」タブから切り替えまたは無効化できます。',
localAsrTakeoverHint: '「{{name}}」を有効化すると ASR プロバイダーが引き継がれます。',
asrProviderTakenOver: 'ASR プロバイダーは引き継ぎ済み',
+4
View File
@@ -861,6 +861,7 @@ export const ko: typeof zhCN = {
asrWhisper: 'OpenAI Whisper(호환)',
asrOpenrouter: 'OpenRouter Whisper',
asrXiaomiMimo: 'Xiaomi MiMo ASR',
asrIflytek: 'iFlytek 실시간 음성 인식',
asrElevenLabs: 'ElevenLabs Scribe',
asrSherpaOnnxLocal: '로컬 sherpa-onnx(실험적)',
asrFoundryLocalWhisper: '로컬 Whisper(Foundry Local)',
@@ -877,6 +878,9 @@ export const ko: typeof zhCN = {
volcengineAuthModeApiKey: 'Ark API Key',
volcengineMappingNote: 'Secret Key 는 현재 입력 불필요. Resource ID 기본값은 volc.seedasr.sauc.duration.',
volcengineApiKeyNote: 'Volcengine Ark 음성 모델 API 키로 인증하며 APP ID는 필요하지 않습니다. Resource ID 기본값은 volc.seedasr.sauc.duration입니다.',
xfyunAppIdLabel: 'AppID',
xfyunApiKeyLabel: 'API Key',
xfyunNote: 'iFlytek 오픈 플랫폼 "실시간 음성 인식" 서비스 페이지에서 AppID와 API Key를 가져옵니다. 오디오는 16kHz/16bit/모노 PCM입니다. 표준 API에는 핫워드 매개변수가 없으며(iFlytek 콘솔에서 개별 핫워드 설정), 언어는 기본적으로 중국어(보통화)입니다.',
localAsrActiveNotice: '현재 "{{name}}" 사용 중. "고급" 탭에서 전환 또는 비활성화할 수 있습니다.',
localAsrTakeoverHint: '"{{name}}" 활성화 시 ASR 프로바이더가 인수됩니다.',
asrProviderTakenOver: 'ASR 프로바이더 인수 완료',
+4
View File
@@ -857,6 +857,7 @@ export const zhCN = {
asrWhisper: 'OpenAI Whisper(兼容)',
asrOpenrouter: 'OpenRouter Whisper',
asrXiaomiMimo: '小米 MiMo ASR',
asrIflytek: '讯飞实时语音转写',
asrElevenLabs: 'ElevenLabs Scribe',
asrSherpaOnnxLocal: '本地 sherpa-onnx(实验性)',
asrFoundryLocalWhisper: '本地 Whisper(Foundry Local)',
@@ -873,6 +874,9 @@ export const zhCN = {
volcengineAuthModeApiKey: '方舟 API Key',
volcengineMappingNote: 'Secret Key 当前无需填写。Resource ID 默认使用 volc.seedasr.sauc.duration。',
volcengineApiKeyNote: '使用火山方舟语音模型的 API Key 鉴权,无需 APP ID。Resource ID 默认使用 volc.seedasr.sauc.duration。',
xfyunAppIdLabel: 'AppID',
xfyunApiKeyLabel: 'API Key',
xfyunNote: '在讯飞开放平台「实时语音转写」服务页获取 AppID 与 API Key。音频为 16kHz/16bit/单声道 PCM;标准版接口暂不支持热词参数(可在讯飞控制台配置个性化热词),语种默认中文普通话。',
localAsrActiveNotice: '当前已启用「{{name}}」,可在「高级」中切换或禁用。',
localAsrTakeoverHint: '启动「{{name}}」后,ASR 提供商将被接管。',
asrProviderTakenOver: '当前用的是本地引擎,在上方下拉直接选其它供应商即可切换(本地引擎会自动停用);本地模型在「高级 → 本地模型」里管理。',
+4
View File
@@ -859,6 +859,7 @@ export const zhTW: typeof zhCN = {
asrWhisper: 'OpenAI Whisper(兼容)',
asrOpenrouter: 'OpenRouter Whisper',
asrXiaomiMimo: '小米 MiMo ASR',
asrIflytek: '訊飛即時語音轉寫',
asrElevenLabs: 'ElevenLabs Scribe',
asrSherpaOnnxLocal: '本地 sherpa-onnx(實驗性)',
asrFoundryLocalWhisper: '本地 Whisper(Foundry Local)',
@@ -875,6 +876,9 @@ export const zhTW: typeof zhCN = {
volcengineAuthModeApiKey: '方舟 API Key',
volcengineMappingNote: 'Secret Key 當前無需填寫。Resource ID 默認使用 volc.seedasr.sauc.duration。',
volcengineApiKeyNote: '使用火山方舟語音模型的 API Key 鑑權,無需 APP ID。Resource ID 預設使用 volc.seedasr.sauc.duration。',
xfyunAppIdLabel: 'AppID',
xfyunApiKeyLabel: 'API Key',
xfyunNote: '在訊飛開放平台「即時語音轉寫」服務頁取得 AppID 與 API Key。音訊為 16kHz/16bit/單聲道 PCM;標準版介面暫不支援熱詞參數(可在訊飛控制台設定個人化熱詞),語種預設中文普通話。',
localAsrActiveNotice: '當前已啓用「{{name}}」,可在「高級」中切換或停用。',
localAsrTakeoverHint: '啓動「{{name}}」後,ASR 提供商將被接管。',
asrProviderTakenOver: 'ASR 提供商已被接管',
@@ -588,6 +588,27 @@ export function ProvidersSection({ kind = 'all' }: ProvidersSectionProps = {}) {
: t('settings.providers.volcengineMappingNote')}
</div>
</>
) : committedAsrProvider === 'iflytek' ? (
<>
<CredentialField
key={`${committedAsrProvider}:app_id`}
label={t('settings.providers.xfyunAppIdLabel')}
account="xfyun.app_id"
provider={committedAsrProvider}
mono
/>
<CredentialField
key={`${committedAsrProvider}:api_key`}
label={t('settings.providers.xfyunApiKeyLabel')}
account="xfyun.api_key"
provider={committedAsrProvider}
mono
mask
/>
<div style={{ marginTop: 2, fontSize: 11.5, color: 'var(--ol-ink-4)', lineHeight: 1.6 }}>
{t('settings.providers.xfyunNote')}
</div>
</>
) : committedAsrProvider === 'local-qwen3' || committedAsrProvider === 'foundry-local-whisper' || committedAsrProvider === 'sherpa-onnx-local' || committedAsrProvider === 'apple-speech' ? (
// 用户已经在用本地 ASR——dropdown 行的 asrProviderTakenOver 已经把
// "在高级中切换或禁用"讲清楚了,body 不再重复。
@@ -238,6 +238,9 @@ export const ASR_PRESETS = [
// 小米 MiMo ASR 按官方文档走 /chat/completions + input_audio,不是
// Whisper /audio/transcriptions;后端由 asr/mimo.rs 专用 client 处理。
{ id: 'xiaomi-mimo-asr', nameKey: 'asrXiaomiMimo', baseUrl: 'https://api.xiaomimimo.com/v1', model: 'mimo-v2.5-asr' },
// 讯飞实时语音转写(RTASR)流式:无 baseUrl/model 配置,凭据是 AppID + APIKey
// 两字段(asr/xfyun.rs);音频 16k/16bit/mono,与 recorder 输出一致。
{ id: 'iflytek', nameKey: 'asrIflytek', baseUrl: '', model: '' },
{ id: 'foundry-local-whisper', nameKey: 'asrFoundryLocalWhisper', baseUrl: '', model: '' },
// 本地引擎(Foundry / sherpa-onnx / Qwen3):无 baseUrl/model 配置,
// 模型在「高级 → 本地模型」里下载与切换。