feat(asr): 接入阶跃星辰 StepAudio 2.5 批式转写,词典走一等 hotwords (#835)

* feat(asr): 接入阶跃星辰 StepAudio 2.5 批式转写,词典走一等 hotwords

StepFun 的 /v1/audio/transcriptions 是标准 OpenAI multipart(实测 2026-07),
归入 Whisper 兼容路由即可,无需新协议客户端:

- 前端新增 stepfun preset(https://api.stepfun.com/v1 + stepaudio-2.5-asr),
  i18n 五语言补齐;
- StepFun 静默忽略 prompt(实测返回 200 但不参与偏置),词汇偏置走专门的
  hotwords 参数(JSON 数组字符串)——WhisperBatchASR 新增 with_hotwords,
  wiring 按 whisper_uses_hotwords 在 prompt / hotwords 间二选一路由,
  用户词典在 StepFun 下才真正生效;
- response_format 只认 json/text → verbose_json 幻听过滤保持关闭;
- 修「验证」按钮对静音敏感厂商的假阴性:探针是 250ms 纯静音,StepFun 返回
  400 "no speech found",但走到内容拒收说明鉴权(错 key=401)与模型名
  (错模型=404)均已通过——判为验证成功而非凭据错误。

实测矩阵(真实 key 逐项打点):真实语音+json/缺省/带 prompt/带 hotwords 均
200 且转写准确(5.5s 音频约 0.95s 往返);verbose_json 400;静音 400
no speech;错 key 401;错模型 404。

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(asr): StepFun 流式实时转写——同一入口按模型名自动切协议

照百炼「一个入口、按模型二次路由」的先例:stepfun preset 下模型填
`stepaudio-2.5-asr-stream`(或任意 `*-stream`)即走新的实时 WS 客户端
(asr/stepfun_realtime.rs),其余模型维持批式 Whisper 兼容路径。凭据与
「已配置」判定仍看真实 active `stepfun`,一把 key、同一个 https base
(wss URL 由 client 自动派生)。

协议与 qwen_realtime 同为 OpenAI Realtime 风格,但四处关键差异
(2026-07-16 真实接口逐项实测确认):

- 模型在 session.update 里传(session.audio.input.transcription.model),
  session 配置是 audio.input.{format,transcription,turn_detection} 嵌套形状;
- delta 的 text/stash 是「已确定前缀 + 未定尾巴」的拼接关系(Qwen 是互斥取一);
- 没有服务端结束事件:session.finish 回 transcript.response.error,server_vad
  下 commit 被静默忽略。收尾靠客户端状态机:松手时冲刷尾音频 + 补送 700ms
  静音帧逼 VAD 关段,finishing && open_segments==0 即完成;纯静音会话由
  1.2s 宽限期兜底空文本成功(验证按钮复用该路径,全程 ~2s);
- 实时协议词汇偏置走 transcription.prompt(批式恰相反只认 hotwords),
  词典按 effective 协议自动选通道。

真机端到端(真实 key、模拟 recorder 时序):session 建立 186ms,
松手→最终结果 251ms(批式 ~950ms),5.5s 中文语音转写一字不差,
多句段 VAD 断句由 join_segments 复用 qwen 拼接规则。

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(asr): wait for StepFun tail frame before finishing

* fix(asr): guard StepFun completion before tail ack

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Chris233 <h-chris233@outlook.com>
This commit is contained in:
authored and GitHub committed 2026-07-18 15:43:24 +08:00
1 parent 07f394f5a2
commit 17fe67c59b
16 files changed
+1596 -13

No files matched your search

@@ -13,6 +13,7 @@ pub mod local;
pub mod mimo;
pub mod pcm;
pub mod qwen_realtime;
pub mod stepfun_realtime;
pub mod volcengine;
pub mod wav;
pub mod whisper;
@@ -22,6 +23,7 @@ pub use dashscope_multimodal::DashScopeMultimodalASR;
pub use elevenlabs::ElevenLabsBatchASR;
pub use mimo::MimoBatchASR;
pub use qwen_realtime::{Qwen3RealtimeASR, Qwen3RealtimeCredentials};
pub use stepfun_realtime::{StepfunRealtimeASR, StepfunRealtimeCredentials};
pub use volcengine::{VolcengineCredentials, VolcengineStreamingASR};
pub use whisper::WhisperBatchASR;
@@ -576,7 +576,8 @@ fn drain_audio_chunks(buffer: &mut Vec<u8>) -> Vec<Vec<u8>> {
}
/// VAD 句段拼接:CJK 之间直接相连;拉丁词之间补空格,避免英文句段黏连。
fn join_segments(segments: &[String]) -> String {
/// `stepfun_realtime` 的多句段收尾复用同一套拼接逻辑,故 `pub(crate)`。
pub(crate) fn join_segments(segments: &[String]) -> String {
let mut joined = String::new();
for seg in segments.iter().map(|s| s.trim()) {
if seg.is_empty() {
File diff suppressed because it is too large. Load diff
@@ -52,6 +52,9 @@ pub struct WhisperBatchASR {
verbose_json: bool,
/// 请求体编码方式。默认 `Multipart`,OpenRouter 走 `OpenRouterJson`。
request_format: AsrRequestFormat,
/// 一等 `hotwords` 参数(JSON 数组字符串)。StepFun 等厂商不认 `prompt`
/// (静默忽略),但提供专门的热词字段——用它词典才真正生效。空 = 不发。
hotwords: Vec<String>,
buffer: Mutex<Vec<u8>>,
}
@@ -72,6 +75,7 @@ impl WhisperBatchASR {
max_chunk_duration_ms,
verbose_json,
request_format: AsrRequestFormat::Multipart,
hotwords: Vec::new(),
buffer: Mutex::new(Vec::new()),
}
}
@@ -83,6 +87,13 @@ impl WhisperBatchASR {
self
}
/// 设置一等热词列表(默认空 = 不发)。仅 Multipart 编码下生效;与 `prompt`
/// 二选一由 wiring 决定(见 coordinator 的 `whisper_uses_hotwords`)。
pub fn with_hotwords(mut self, hotwords: Vec<String>) -> Self {
self.hotwords = hotwords;
self
}
/// Stop collecting audio, encode the buffer as WAV, and POST to the
/// Whisper transcriptions endpoint.
///
@@ -172,6 +183,20 @@ impl WhisperBatchASR {
}
}
// 一等热词(StepFun 形状:可解析的 JSON 数组字符串,如
// `["热词1","热词2"]`)。空白词条过滤后再编码,全空则不发该字段。
let hotwords: Vec<&str> = self
.hotwords
.iter()
.map(|w| w.trim())
.filter(|w| !w.is_empty())
.collect();
if !hotwords.is_empty() {
if let Ok(encoded) = serde_json::to_string(&hotwords) {
form = form.text("hotwords", encoded);
}
}
client
.post(&url)
.header("Authorization", format!("Bearer {}", self.api_key))
@@ -833,6 +858,63 @@ mod tests {
server.join().unwrap();
}
#[tokio::test]
async fn hotwords_sent_as_json_array_field_without_prompt() {
// StepFun 形状:词典走一等 `hotwords`(JSON 数组字符串)而非 `prompt`;
// 空白词条过滤后编码。
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
listener.set_nonblocking(true).unwrap();
let addr = listener.local_addr().unwrap();
let server = thread::spawn(move || {
let deadline = Instant::now() + Duration::from_secs(5);
let mut stream = loop {
match listener.accept() {
Ok((stream, _)) => break stream,
Err(err) if err.kind() == std::io::ErrorKind::WouldBlock => {
assert!(
Instant::now() < deadline,
"timed out waiting for ASR test request"
);
thread::sleep(Duration::from_millis(10));
}
Err(err) => panic!("accept ASR test request failed: {err}"),
}
};
stream.set_nonblocking(false).unwrap();
stream
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let request = read_http_request(&mut stream);
let request_text = String::from_utf8_lossy(&request);
assert!(request_text.starts_with("POST /audio/transcriptions HTTP/1.1"));
assert!(request_text.contains(r#"name="hotwords""#));
assert!(request_text.contains(r#"["阶跃星辰","OpenLess"]"#));
assert!(!request_text.contains(r#"name="prompt""#));
write_json_response(&mut stream, r#"{"text":"hotwords ok"}"#);
});
let base_url = format!("http://{}", addr);
let asr = WhisperBatchASR::new(
"key".to_string(),
base_url,
"stepaudio-2.5-asr".to_string(),
None,
None,
false,
)
.with_hotwords(vec![
"阶跃星辰".to_string(),
" ".to_string(),
"OpenLess".to_string(),
]);
let pcm = vec![0u8; 32_000 * 2];
asr.consume_pcm_chunk(&pcm);
let transcript = asr.transcribe().await.unwrap();
assert_eq!(transcript.text, "hotwords ok");
server.join().unwrap();
}
fn start_whisper_test_server(texts: Vec<&'static str>) -> (String, thread::JoinHandle<()>) {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
listener.set_nonblocking(true).unwrap();
@@ -270,6 +270,18 @@ async fn validate_asr_provider() -> Result<(), String> {
if active_asr == crate::asr::elevenlabs::PROVIDER_ID {
return validate_elevenlabs_asr_provider().await;
}
// StepFun 一入口双协议:`*-stream` 模型走实时 WS 验证,其余走批式
// /audio/transcriptions(与 build 侧 resolve_effective_asr_provider 同判据)。
if active_asr == "stepfun" || active_asr == crate::asr::stepfun_realtime::PROVIDER_ID {
let model = CredentialsVault::get(CredentialAccount::AsrModel)
.map_err(|e| e.to_string())?
.unwrap_or_default();
if active_asr == crate::asr::stepfun_realtime::PROVIDER_ID
|| crate::coordinator::stepfun_model_is_stream(&model)
{
return validate_stepfun_realtime_asr_provider().await;
}
}
let config = read_openai_provider_config("asr")?;
let model = CredentialsVault::get(CredentialAccount::AsrModel)
@@ -279,6 +291,43 @@ async fn validate_asr_provider() -> Result<(), String> {
validate_asr_transcription(&config, model.trim()).await
}
/// StepFun 实时 WS 验证:真连 + session.update + 500ms 静音 + 收尾。
/// 协议无 finish 事件,收尾走静音帧 + 宽限期(纯静音会话以空文本成功返回,
/// 见 stepfun_realtime 模块注释),全程 ~2s。
async fn validate_stepfun_realtime_asr_provider() -> Result<(), String> {
let api_key = CredentialsVault::get(CredentialAccount::AsrApiKey)
.map_err(|e| e.to_string())?
.unwrap_or_default();
if api_key.trim().is_empty() {
return Err("API Key 为空".to_string());
}
let endpoint = CredentialsVault::get(CredentialAccount::AsrEndpoint)
.map_err(|e| e.to_string())?
.unwrap_or_default();
let model = CredentialsVault::get(CredentialAccount::AsrModel)
.map_err(|e| e.to_string())?
.filter(|s| !s.trim().is_empty())
.unwrap_or_else(|| crate::asr::stepfun_realtime::DEFAULT_MODEL.to_string());
let asr = std::sync::Arc::new(crate::asr::StepfunRealtimeASR::new(
crate::asr::StepfunRealtimeCredentials {
api_key,
endpoint,
model,
prompt: None,
},
));
asr.open_session().await.map_err(|e| e.to_string())?;
crate::asr::AudioConsumer::consume_pcm_chunk(
&*asr,
&vec![0u8; crate::asr::stepfun_realtime::TARGET_AUDIO_CHUNK_BYTES * 5],
);
asr.send_last_frame().await.map_err(|e| e.to_string())?;
asr.await_final_result()
.await
.map(|_| ())
.map_err(|e| e.to_string())
}
async fn validate_mimo_asr_provider() -> Result<(), String> {
let config = read_openai_provider_config("asr")?;
let model = CredentialsVault::get(CredentialAccount::AsrModel)
@@ -588,6 +637,16 @@ async fn validate_asr_transcription(config: &ProviderConfig, model: &str) -> Res
};
let status = response.status();
if !status.is_success() {
// 探针音频是纯静音,有的厂商(StepFun)对无语音内容直接 400
// "no speech found"。走到这一步说明鉴权(错 key 是 401)和模型名
// (错模型是 404 model_invalid)都已通过、转写管线是通的——这类
// 内容拒收判为验证成功,避免对静音敏感的厂商恒报假阴性。
if status.as_u16() == 400 {
let body = response.text().await.unwrap_or_default();
if asr_error_is_no_speech_rejection(&body) {
return Ok(());
}
}
return Err(format!("providerHttpStatus:{}", status.as_u16()));
}
if let Some(len) = response.content_length() {
@@ -612,6 +671,13 @@ async fn validate_asr_transcription(config: &ProviderConfig, model: &str) -> Res
Ok(())
}
/// 400 应答体是否是「音频里没有语音」类内容拒收(而非参数错误)。
/// 只匹配语义明确的措辞,宁可漏判(用户看到 400 后实测仍可用)也不误判
/// 真正的参数错误为成功。
fn asr_error_is_no_speech_rejection(body: &str) -> bool {
body.to_ascii_lowercase().contains("no speech")
}
pub(crate) fn asr_transcriptions_url(base_url: &str) -> Result<String, String> {
let parsed = reqwest::Url::parse(base_url.trim()).map_err(|_| "endpointInvalid".to_string())?;
@@ -836,11 +902,26 @@ mod tests {
// LLM 路径的 SSRF 校验。read_openai_provider_config 依赖凭据库无法纯单测,这里直接对它调用
// 的校验器锁定 ASR 形态 endpoint 的拒绝/放行契约。
use super::{
fetch_provider_models, models_url, provider_llm_error_message, provider_log_context,
provider_request_error_message, sanitized_provider_destination, ProviderConfig,
asr_error_is_no_speech_rejection, fetch_provider_models, models_url,
provider_llm_error_message, provider_log_context, provider_request_error_message,
sanitized_provider_destination, ProviderConfig,
};
use crate::endpoint_security::validate_http_endpoint;
#[test]
fn silence_probe_content_rejection_is_not_a_credential_error() {
// StepFun 对静音探针的实测应答(2026-07):鉴权/模型都通过,只是探针
// 音频没有语音内容——不能报成凭据错误。
assert!(asr_error_is_no_speech_rejection(
r#"{"error":{"message":"no speech found","type":"request_params_invalid"}}"#
));
// 真正的参数错误不能被误判成功。
assert!(!asr_error_is_no_speech_rejection(
r#"{"error":{"message":"Request param: response_format is invalid","type":"input_invalid"}}"#
));
assert!(!asr_error_is_no_speech_rejection(""));
}
#[test]
fn provider_destination_redacts_userinfo_query_and_fragment() {
let raw = "https://alice:password@example.com:8443/v1/models?api_key=query-secret#private-fragment";
+109 -1
View File
@@ -187,6 +187,8 @@ enum ActiveAsr {
Bailian(Arc<BailianRealtimeASR>),
/// 百炼 Qwen3-ASR-Flash 实时(OpenAI Realtime 风格 WS 协议)。
Qwen3Realtime(Arc<Qwen3RealtimeASR>),
/// 阶跃星辰 StepAudio 实时(OpenAI Realtime 风格 WS,收尾靠静音帧驱动 VAD)。
StepfunRealtime(Arc<crate::asr::StepfunRealtimeASR>),
#[cfg(target_os = "windows")]
FoundryLocalWhisper(Arc<FoundryLocalWhisperAsr>),
/// Windows sherpa-onnx 本地 ASR(offline batch + 实验 online streaming)。
@@ -224,6 +226,7 @@ fn asr_transcribe_uses_global_timeout(asr: &ActiveAsr) -> bool {
pub(crate) enum ActiveAsrProviderKind {
Bailian,
Qwen3Realtime,
StepfunRealtime,
Mimo,
DashScopeMultimodal,
ElevenLabs,
@@ -259,6 +262,7 @@ impl ActiveAsrProviderKind {
match self {
ActiveAsrProviderKind::Bailian
| ActiveAsrProviderKind::Qwen3Realtime
| ActiveAsrProviderKind::StepfunRealtime
| ActiveAsrProviderKind::Mimo
| ActiveAsrProviderKind::DashScopeMultimodal
| ActiveAsrProviderKind::ElevenLabs
@@ -277,7 +281,11 @@ impl ActiveAsrProviderKind {
ActiveAsrProviderKind::Mimo | ActiveAsrProviderKind::DashScopeMultimodal => {
AsrConfiguredFields::ApiKeyEndpointModel
}
ActiveAsrProviderKind::WhisperCompatible => AsrConfiguredFields::EndpointModelOnly,
// StepfunRealtime 只经 `stepfun` 的模型路由可达(隐藏 effective id),
// 「已配置」判定看真实 active `stepfun` → WhisperCompatible;此处形态
// 与之对齐,保证直接停在该 id 上也语义一致。
ActiveAsrProviderKind::WhisperCompatible
| ActiveAsrProviderKind::StepfunRealtime => AsrConfiguredFields::EndpointModelOnly,
ActiveAsrProviderKind::Volcengine => AsrConfiguredFields::VolcAppKey,
}
}
@@ -288,6 +296,8 @@ pub(crate) fn active_asr_provider_kind(id: &str) -> ActiveAsrProviderKind {
ActiveAsrProviderKind::Bailian
} else if is_qwen3_realtime_provider(id) {
ActiveAsrProviderKind::Qwen3Realtime
} else if is_stepfun_realtime_provider(id) {
ActiveAsrProviderKind::StepfunRealtime
} else if is_mimo_provider(id) {
ActiveAsrProviderKind::Mimo
} else if is_dashscope_multimodal_provider(id) {
@@ -318,6 +328,12 @@ pub(crate) fn resolve_effective_asr_provider(
if is_dashscope_multimodal_provider(active_asr) {
validate_dashscope_multimodal_model(model)?;
}
// StepFun 同款「一个入口按模型切协议」:`*-stream` 模型走实时 WS
// 客户端,其余走批式 Whisper 兼容路径。凭据 / 「已配置」判定仍看
// 真实 active `stepfun`。
if active_asr == "stepfun" && stepfun_model_is_stream(model) {
return Ok(crate::asr::stepfun_realtime::PROVIDER_ID.to_string());
}
return Ok(active_asr.to_string());
}
@@ -347,6 +363,12 @@ fn is_classic_bailian_realtime_model(model: &str) -> bool {
|| model.starts_with("sensevoice-realtime")
}
/// StepFun 的流式模型命名恒以 `-stream` 结尾(stepaudio-2.5-asr-stream /
/// step-asr-1.1-stream),其余(含空 = 默认批式模型)走批式。
pub(crate) fn stepfun_model_is_stream(model: &str) -> bool {
model.trim().ends_with("-stream")
}
pub(crate) fn validate_dashscope_multimodal_model(model: &str) -> Result<(), String> {
let model = model.trim();
if model.is_empty() || model == crate::asr::dashscope_multimodal::DEFAULT_MODEL {
@@ -1881,6 +1903,13 @@ impl Coordinator {
.map_err(|_| "重新转录超时".to_string())?
.map_err(|e| e.to_string())?
}
ActiveAsr::StepfunRealtime(asr) => {
asr.send_last_frame().await.map_err(|e| e.to_string())?;
tokio::time::timeout(timeout, asr.await_final_result())
.await
.map_err(|_| "重新转录超时".to_string())?
.map_err(|e| e.to_string())?
}
ActiveAsr::Whisper(w) => tokio::time::timeout(timeout, w.transcribe())
.await
.map_err(|_| "重新转录超时".to_string())?
@@ -2423,6 +2452,33 @@ fn read_qwen3_realtime_credentials() -> Qwen3RealtimeCredentials {
}
}
/// StepFun 实时凭据与批式共用同一组槽位(一把 key、同一个 https base、
/// 模型名区分协议);wss URL 由 client 的 `connect_url()` 从 base 派生。
/// `prompt` 由调用方按用户词典填充(实时协议接受 prompt、批式只认 hotwords)。
fn read_stepfun_realtime_credentials(
prompt: Option<String>,
) -> crate::asr::StepfunRealtimeCredentials {
let api_key = CredentialsVault::get(CredentialAccount::AsrApiKey)
.ok()
.flatten()
.unwrap_or_default();
let endpoint = CredentialsVault::get(CredentialAccount::AsrEndpoint)
.ok()
.flatten()
.unwrap_or_default();
let model = CredentialsVault::get(CredentialAccount::AsrModel)
.ok()
.flatten()
.filter(|s| !s.trim().is_empty())
.unwrap_or_else(|| crate::asr::stepfun_realtime::DEFAULT_MODEL.to_string());
crate::asr::StepfunRealtimeCredentials {
api_key,
endpoint,
model,
prompt,
}
}
fn read_volc_credentials() -> VolcengineCredentials {
let app_id = CredentialsVault::get(CredentialAccount::VolcengineAppKey)
.ok()
@@ -2859,6 +2915,58 @@ mod tests {
assert_eq!(batch_asr_chunk_limit_ms("openrouter"), Some(30_000));
}
#[test]
fn stepfun_is_whisper_compatible_with_hotwords_vocab() {
use crate::asr::whisper::AsrRequestFormat;
// StepFun /audio/transcriptions 是标准 multipart(实测 2026-07),
// response_format 只认 json/text → verbose_json 关闭;100MB 上限
// (约 54 分钟 16k WAV)→ 无需切分。
assert!(is_whisper_compatible_provider("stepfun"));
assert_eq!(
active_asr_provider_kind("stepfun"),
ActiveAsrProviderKind::WhisperCompatible
);
assert_eq!(
whisper_request_format("stepfun"),
AsrRequestFormat::Multipart
);
assert!(!whisper_supports_verbose_json("stepfun"));
assert_eq!(batch_asr_chunk_limit_ms("stepfun"), None);
// 一入口双协议:`*-stream` 模型路由到实时 WS 客户端,其余留在批式。
assert_eq!(
resolve_effective_asr_provider("stepfun", "stepaudio-2.5-asr").unwrap(),
"stepfun"
);
assert_eq!(
resolve_effective_asr_provider("stepfun", "").unwrap(),
"stepfun"
);
assert_eq!(
resolve_effective_asr_provider("stepfun", "stepaudio-2.5-asr-stream").unwrap(),
crate::asr::stepfun_realtime::PROVIDER_ID
);
assert_eq!(
resolve_effective_asr_provider("stepfun", "step-asr-1.1-stream").unwrap(),
crate::asr::stepfun_realtime::PROVIDER_ID
);
assert_eq!(
active_asr_provider_kind(crate::asr::stepfun_realtime::PROVIDER_ID),
ActiveAsrProviderKind::StepfunRealtime
);
// 词典路由:StepFun 批式忽略 prompt,走一等 hotwords;其余厂商维持 prompt。
assert!(whisper_uses_hotwords("stepfun"));
assert!(!whisper_uses_hotwords("whisper"));
let phrases = vec!["阶跃星辰".to_string()];
let (prompt, hotwords) = whisper_vocab_for_provider("stepfun", phrases.clone());
assert_eq!(prompt, None);
assert_eq!(hotwords, phrases);
let (prompt, hotwords) = whisper_vocab_for_provider("groq", phrases);
assert_eq!(prompt.as_deref(), Some("阶跃星辰."));
assert!(hotwords.is_empty());
}
#[test]
fn qa_asr_provider_kind_tracks_active_provider() {
assert_eq!(
@@ -320,8 +320,8 @@ pub(super) fn build_apple_speech(
Arc::new(crate::asr::local::AppleSpeechAsr::new(locale))
}
/// `whisper` 是 OpenAI 原生;`siliconflow` / `zhipu` / `groq` 都暴露
/// OpenAI 兼容的 `/audio/transcriptions`,统一走 `WhisperBatchASR`。
/// `whisper` 是 OpenAI 原生;`siliconflow` / `zhipu` / `groq` / `stepfun`
/// 都暴露 OpenAI 兼容的 `/audio/transcriptions`,统一走 `WhisperBatchASR`。
/// 新增 OpenAI 兼容 ASR 时只需在这里加一项。
///
/// 注:DashScope 的 Qwen3-ASR-Flash 不在此列——它用 MultiModalConversation
@@ -330,10 +330,35 @@ pub(super) fn build_apple_speech(
pub(super) fn is_whisper_compatible_provider(id: &str) -> bool {
matches!(
id,
"whisper" | "siliconflow" | "zhipu" | "groq" | "openrouter"
"whisper" | "siliconflow" | "zhipu" | "groq" | "openrouter" | "stepfun"
)
}
/// 用户词典该走 `prompt` 还是一等 `hotwords` 参数。
///
/// StepFun 的 `/audio/transcriptions` **静默忽略** `prompt`(实测 2026-07:带
/// prompt 返回 200 但不参与偏置),词汇偏置走专门的 `hotwords` 字段(可解析的
/// JSON 数组字符串)。其余兼容厂商维持 Whisper 惯例的 `prompt`。
pub(super) fn whisper_uses_hotwords(provider_id: &str) -> bool {
provider_id == "stepfun"
}
/// 词典启用词条 → (prompt, hotwords) 二选一路由,QA 与听写两处构造点共用。
/// hotwords 厂商不再拼 prompt(免得白占请求体),prompt 厂商 hotwords 恒空。
pub(super) fn whisper_vocab_for_provider(
provider_id: &str,
phrases: Vec<String>,
) -> (Option<String>, Vec<String>) {
if whisper_uses_hotwords(provider_id) {
(None, phrases)
} else {
(
crate::asr::whisper::build_prompt_from_phrases(&phrases),
Vec::new(),
)
}
}
/// 该 provider 的请求体编码方式。OpenRouter 的 `/audio/transcriptions` 是
/// `application/json` + base64 音频(issue #582),其余兼容厂商沿用 multipart。
pub(super) fn whisper_request_format(provider_id: &str) -> crate::asr::whisper::AsrRequestFormat {
@@ -364,6 +389,10 @@ pub(super) fn is_qwen3_realtime_provider(id: &str) -> bool {
id == crate::asr::qwen_realtime::PROVIDER_ID
}
pub(super) fn is_stepfun_realtime_provider(id: &str) -> bool {
id == crate::asr::stepfun_realtime::PROVIDER_ID
}
pub(super) fn is_mimo_provider(id: &str) -> bool {
id == crate::asr::mimo::PROVIDER_ID
}
@@ -410,6 +439,10 @@ pub(super) enum QaAsrStart {
asr: Arc<Qwen3RealtimeASR>,
bridge: Arc<DeferredAsrBridge>,
},
StepfunRealtime {
asr: Arc<crate::asr::StepfunRealtimeASR>,
bridge: Arc<DeferredAsrBridge>,
},
Ready {
active: ActiveAsr,
consumer: Arc<dyn crate::recorder::AudioConsumer>,
@@ -422,6 +455,7 @@ impl QaAsrStart {
QaAsrStart::Volcengine { asr, .. } => ActiveAsr::Volcengine(Arc::clone(asr)),
QaAsrStart::Bailian { asr, .. } => ActiveAsr::Bailian(Arc::clone(asr)),
QaAsrStart::Qwen3Realtime { asr, .. } => ActiveAsr::Qwen3Realtime(Arc::clone(asr)),
QaAsrStart::StepfunRealtime { asr, .. } => ActiveAsr::StepfunRealtime(Arc::clone(asr)),
QaAsrStart::Ready { active, .. } => active.clone(),
}
}
@@ -431,6 +465,7 @@ impl QaAsrStart {
QaAsrStart::Volcengine { bridge, .. } => Arc::clone(bridge) as _,
QaAsrStart::Bailian { bridge, .. } => Arc::clone(bridge) as _,
QaAsrStart::Qwen3Realtime { bridge, .. } => Arc::clone(bridge) as _,
QaAsrStart::StepfunRealtime { bridge, .. } => Arc::clone(bridge) as _,
QaAsrStart::Ready { consumer, .. } => Arc::clone(consumer),
}
}
@@ -462,6 +497,15 @@ impl QaAsrStart {
);
Ok(())
}
QaAsrStart::StepfunRealtime { asr, bridge } => {
asr.open_session().await.map_err(|e| e.to_string())?;
let target: Arc<dyn crate::asr::AudioConsumer> = Arc::clone(asr) as _;
let flushed = bridge.attach(target);
log::info!(
"[coord] QA StepFun realtime ASR connected; flushed {flushed} deferred audio bytes"
);
Ok(())
}
QaAsrStart::Ready { .. } => Ok(()),
}
}
@@ -585,6 +629,18 @@ pub(super) async fn build_qa_asr_start(
label,
))
}
ActiveAsrProviderKind::StepfunRealtime => {
let prompt = crate::asr::whisper::build_prompt_from_phrases(&enabled_phrases(inner));
let creds = read_stepfun_realtime_credentials(prompt);
let label = AsrCallLabel::new(effective_asr.clone(), Some(creds.model.clone()));
Ok((
QaAsrStart::StepfunRealtime {
asr: Arc::new(crate::asr::StepfunRealtimeASR::new(creds)),
bridge: Arc::new(DeferredAsrBridge::new()),
},
label,
))
}
ActiveAsrProviderKind::Mimo => {
let (api_key, base_url, model) = read_mimo_credentials();
let label = AsrCallLabel::new(effective_asr.clone(), Some(model.clone()));
@@ -612,8 +668,8 @@ pub(super) async fn build_qa_asr_start(
ActiveAsrProviderKind::WhisperCompatible => {
let (api_key, base_url, model) = read_whisper_credentials();
let label = AsrCallLabel::new(effective_asr.clone(), Some(model.clone()));
let whisper_prompt =
crate::asr::whisper::build_prompt_from_phrases(&enabled_phrases(inner));
let (whisper_prompt, hotwords) =
whisper_vocab_for_provider(active_asr, enabled_phrases(inner));
let whisper = Arc::new(
WhisperBatchASR::new(
api_key,
@@ -623,7 +679,8 @@ pub(super) async fn build_qa_asr_start(
batch_asr_chunk_limit_ms(active_asr),
whisper_supports_verbose_json(active_asr),
)
.with_request_format(whisper_request_format(active_asr)),
.with_request_format(whisper_request_format(active_asr))
.with_hotwords(hotwords),
);
let active = ActiveAsr::Whisper(Arc::clone(&whisper));
let consumer: Arc<dyn crate::recorder::AudioConsumer> = whisper;
@@ -1672,6 +1672,7 @@ pub(super) async fn begin_session_as(
match active_asr_provider_kind(&effective_asr) {
ActiveAsrProviderKind::Bailian
| ActiveAsrProviderKind::Qwen3Realtime
| ActiveAsrProviderKind::StepfunRealtime
| ActiveAsrProviderKind::Mimo
| ActiveAsrProviderKind::DashScopeMultimodal
| ActiveAsrProviderKind::ElevenLabs
@@ -1834,6 +1835,88 @@ pub(super) async fn begin_session_as(
"[coord] Qwen3 realtime ASR connected; flushed {flushed_bytes} deferred audio bytes"
);
finish_starting_session(inner, current_session_id).await;
} else if is_stepfun_realtime_provider(&effective_asr) {
// 与 Qwen3 realtime 分支同构:流式 WS 会话 + DeferredAsrBridge 缓冲开链前音频。
// 实时协议的词汇偏置走 transcription.prompt(批式 stepfun 则相反走 hotwords)。
let prompt = crate::asr::whisper::build_prompt_from_phrases(&enabled_phrases(inner));
let creds = read_stepfun_realtime_credentials(prompt);
let asr_call_label = AsrCallLabel::new(effective_asr.clone(), Some(creds.model.clone()));
let asr = Arc::new(crate::asr::StepfunRealtimeASR::new(creds));
let bridge = Arc::new(DeferredAsrBridge::new());
let consumer: Arc<dyn crate::recorder::AudioConsumer> = bridge.clone();
store_asr_for_session(
inner,
current_session_id,
ActiveAsr::StepfunRealtime(Arc::clone(&asr)),
asr_call_label,
);
start_recorder_for_starting(inner, current_session_id, &active_asr, consumer).await?;
if let Err(e) = asr.open_session().await {
log::error!("[coord] open StepFun realtime ASR session failed: {e}");
match startup_race_status_for_starting(inner, current_session_id) {
StartupRaceStatus::StaleContinuation => {
log::info!(
"[coord] stale StepFun realtime ASR open_session error from session {current_session_id} — ignoring"
);
asr.cancel();
discard_startup_resources_for_session(inner, current_session_id);
restore_prepared_windows_ime_session(inner, current_session_id);
return Ok(());
}
StartupRaceStatus::CancelRaced => {
asr.cancel();
discard_startup_resources_for_session(inner, current_session_id);
restore_prepared_windows_ime_session(inner, current_session_id);
set_phase_idle_if_session_matches(inner, current_session_id);
return Ok(());
}
StartupRaceStatus::ActiveStarting => {
asr.cancel();
}
}
discard_startup_resources_for_session(inner, current_session_id);
emit_capsule(
inner,
CapsuleState::Error,
0.0,
0,
Some(format!("ASR 连接失败: {e}")),
None,
);
restore_prepared_windows_ime_session(inner, current_session_id);
set_phase_idle_if_session_matches(inner, current_session_id);
schedule_capsule_idle(inner, CAPSULE_AUTO_HIDE_DELAY_MS);
return Err(e.to_string());
}
match startup_race_status_for_starting(inner, current_session_id) {
StartupRaceStatus::ActiveStarting => {}
StartupRaceStatus::CancelRaced => {
log::info!(
"[coord] cancel raced during StepFun realtime ASR open_session — aborting begin"
);
asr.cancel();
discard_startup_resources_for_session(inner, current_session_id);
restore_prepared_windows_ime_session(inner, current_session_id);
set_phase_idle_if_session_matches(inner, current_session_id);
return Ok(());
}
StartupRaceStatus::StaleContinuation => {
log::info!(
"[coord] stale StepFun realtime ASR open_session continuation from session {current_session_id} — ignoring"
);
asr.cancel();
discard_startup_resources_for_session(inner, current_session_id);
restore_prepared_windows_ime_session(inner, current_session_id);
return Ok(());
}
}
let target: Arc<dyn crate::asr::AudioConsumer> = asr;
let flushed_bytes = bridge.attach(target);
log::info!(
"[coord] StepFun realtime ASR connected; flushed {flushed_bytes} deferred audio bytes"
);
finish_starting_session(inner, current_session_id).await;
} else if is_mimo_provider(&effective_asr) {
let (api_key, base_url, model) = read_mimo_credentials();
let asr_call_label = AsrCallLabel::new(effective_asr.clone(), Some(model.clone()));
@@ -1882,8 +1965,8 @@ pub(super) async fn begin_session_as(
// hotword を受け取っており、UI 説明文も「ASR ホットワードと後処理
// モデルのコンテキスト両方に渡される」と明示しているので、Whisper
// 互換プロバイダにも揃えるのが筋。
let whisper_prompt =
crate::asr::whisper::build_prompt_from_phrases(&enabled_phrases(inner));
let (whisper_prompt, hotwords) =
whisper_vocab_for_provider(&active_asr, enabled_phrases(inner));
let asr_call_label = AsrCallLabel::new(effective_asr.clone(), Some(model.clone()));
let whisper = Arc::new(
WhisperBatchASR::new(
@@ -1894,7 +1977,8 @@ pub(super) async fn begin_session_as(
batch_asr_chunk_limit_ms(&active_asr),
whisper_supports_verbose_json(&active_asr),
)
.with_request_format(whisper_request_format(&active_asr)),
.with_request_format(whisper_request_format(&active_asr))
.with_hotwords(hotwords),
);
store_asr_for_session(
inner,
@@ -2743,6 +2827,34 @@ pub(super) async fn end_session(inner: &Arc<Inner>) -> Result<(), String> {
}
}
}
ActiveAsr::StepfunRealtime(asr) => {
debug_assert!(uses_global_timeout);
if let Err(e) = asr.send_last_frame().await {
log::error!("[coord] StepFun realtime send last frame failed: {e}");
}
let timeout_duration =
std::time::Duration::from_secs(COORDINATOR_GLOBAL_TIMEOUT_SECS);
match tokio::time::timeout(timeout_duration, asr.await_final_result()).await {
Ok(Ok(r)) => Ok(r),
Ok(Err(e)) => {
log::error!("[coord] StepFun realtime await final failed: {e}");
// 关闭 WebSocket 连接,避免流式 ASR 资源泄漏
asr.cancel();
Err(TranscribeFail::new(format!("识别失败: {e}"), e.to_string()))
}
Err(_) => {
log::error!(
"[coord] StepFun realtime 全局超时 {} 秒",
COORDINATOR_GLOBAL_TIMEOUT_SECS
);
asr.cancel();
Err(TranscribeFail::new(
"识别超时".to_string(),
"stepfun realtime global timeout".to_string(),
))
}
}
}
#[cfg(target_os = "windows")]
ActiveAsr::FoundryLocalWhisper(local) => {
debug_assert!(!uses_global_timeout);
@@ -444,6 +444,21 @@ pub(super) async fn transcribe_overlay_dictation_asr(
}
}
}
ActiveAsr::StepfunRealtime(asr) => {
debug_assert!(uses_global_timeout);
if let Err(error) = asr.send_last_frame().await {
log::error!("[coord] overlay QA: StepFun realtime send last frame failed: {error}");
}
let timeout_duration = std::time::Duration::from_secs(COORDINATOR_GLOBAL_TIMEOUT_SECS);
match tokio::time::timeout(timeout_duration, asr.await_final_result()).await {
Ok(Ok(raw)) => Ok(raw),
Ok(Err(error)) => Err(error.to_string()),
Err(_) => {
asr.cancel();
Err("stepfun realtime global timeout".to_string())
}
}
}
ActiveAsr::Whisper(whisper) => {
debug_assert!(uses_global_timeout);
let timeout_duration = std::time::Duration::from_secs(COORDINATOR_GLOBAL_TIMEOUT_SECS);
@@ -1039,6 +1054,30 @@ pub(super) async fn end_qa_session(inner: &Arc<Inner>) -> Result<(), String> {
}
}
}
ActiveAsr::StepfunRealtime(asr) => {
debug_assert!(uses_global_timeout);
if let Err(e) = asr.send_last_frame().await {
log::error!("[coord] QA: StepFun realtime send last frame failed: {e}");
}
let timeout_duration = std::time::Duration::from_secs(COORDINATOR_GLOBAL_TIMEOUT_SECS);
match tokio::time::timeout(timeout_duration, asr.await_final_result()).await {
Ok(Ok(r)) => r,
Ok(Err(e)) => {
log::error!("[coord] QA: StepFun realtime await final failed: {e}");
finish_qa_with_error_if_current(inner, session_id, format!("识别失败: {e}"));
return Err(e.to_string());
}
Err(_) => {
log::error!(
"[coord] QA: StepFun realtime 全局超时 {} 秒",
COORDINATOR_GLOBAL_TIMEOUT_SECS
);
asr.cancel();
finish_qa_with_error_if_current(inner, session_id, "识别超时".to_string());
return Err("stepfun realtime global timeout".to_string());
}
}
}
ActiveAsr::Qwen3Realtime(asr) => {
debug_assert!(uses_global_timeout);
if let Err(e) = asr.send_last_frame().await {
@@ -90,6 +90,7 @@ pub(super) fn cancel_active_asr(asr: ActiveAsr) {
ActiveAsr::ElevenLabs(e) => e.cancel(),
ActiveAsr::Bailian(b) => b.cancel(),
ActiveAsr::Qwen3Realtime(q) => q.cancel(),
ActiveAsr::StepfunRealtime(s) => s.cancel(),
#[cfg(target_os = "windows")]
ActiveAsr::FoundryLocalWhisper(local) => local.cancel(),
#[cfg(target_os = "windows")]
+1
View File
@@ -788,6 +788,7 @@ export const en: typeof zhCN = {
asrBailianQwen3: 'Bailian Qwen3 Realtime ASR',
asrBailianFunAsrFlash: 'Bailian Fun-ASR-Flash (recorded file)',
asrSiliconflow: 'SiliconFlow SenseVoice',
asrStepfun: 'StepFun StepAudio ASR',
asrZhipu: 'Zhipu GLM-ASR',
asrGroq: 'Groq Whisper-large-v3',
asrWhisper: 'OpenAI Whisper (compatible)',
+1
View File
@@ -790,6 +790,7 @@ export const ja: typeof zhCN = {
asrBailianQwen3: 'Bailian Qwen3 リアルタイム ASR',
asrBailianFunAsrFlash: 'Bailian Fun-ASR-Flash(録音ファイル)',
asrSiliconflow: 'SiliconFlow SenseVoice',
asrStepfun: 'StepFun StepAudio ASR',
asrZhipu: 'Zhipu GLM-ASR',
asrGroq: 'Groq Whisper-large-v3',
asrWhisper: 'OpenAI Whisper(互換)',
+1
View File
@@ -790,6 +790,7 @@ export const ko: typeof zhCN = {
asrBailianQwen3: 'Bailian Qwen3 실시간 ASR',
asrBailianFunAsrFlash: 'Bailian Fun-ASR-Flash (녹음 파일)',
asrSiliconflow: 'SiliconFlow SenseVoice',
asrStepfun: 'StepFun StepAudio ASR',
asrZhipu: 'Zhipu GLM-ASR',
asrGroq: 'Groq Whisper-large-v3',
asrWhisper: 'OpenAI Whisper(호환)',
+1
View File
@@ -786,6 +786,7 @@ export const zhCN = {
asrBailianQwen3: '阿里云百炼 Qwen3 实时 ASR',
asrBailianFunAsrFlash: '阿里云百炼 Fun-ASR-Flash(录音文件)',
asrSiliconflow: '硅基流动 SenseVoice',
asrStepfun: '阶跃星辰 StepAudio',
asrZhipu: '智谱 GLM-ASR',
asrGroq: 'Groq Whisper-large-v3',
asrWhisper: 'OpenAI Whisper(兼容)',
+1
View File
@@ -788,6 +788,7 @@ export const zhTW: typeof zhCN = {
asrBailianQwen3: '阿里雲百煉 Qwen3 即時 ASR',
asrBailianFunAsrFlash: '阿里雲百煉 Fun-ASR-Flash(錄音檔)',
asrSiliconflow: '硅基流動 SenseVoice',
asrStepfun: '階躍星辰 StepAudio',
asrZhipu: '智譜 GLM-ASR',
asrGroq: 'Groq Whisper-large-v3',
asrWhisper: 'OpenAI Whisper(兼容)',
@@ -223,6 +223,12 @@ export const ASR_PRESETS = [
// 由 asr/dashscope_multimodal.rs 专用批量 client 处理。API key 与百炼同一把。
{ id: 'bailian-fun-asr-flash', nameKey: 'asrBailianFunAsrFlash', baseUrl: 'https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation', model: 'fun-asr-flash-2026-06-15' },
{ id: 'siliconflow', nameKey: 'asrSiliconflow', baseUrl: 'https://api.siliconflow.cn/v1', model: 'FunAudioLLM/SenseVoiceSmall' },
// 阶跃星辰 StepAudio:一个入口双协议(照百炼先例,后端按模型名二次路由)。
// 默认批式模型走 Whisper 兼容 multipart,词典经一等 hotwords 参数生效;
// 模型改成 stepaudio-2.5-asr-stream 则走实时 WS(asr/stepfun_realtime.rs),
// 词典改经 transcription.prompt 生效。endpoint 一律填 https base,
// wss URL 由后端自动派生。
{ id: 'stepfun', nameKey: 'asrStepfun', baseUrl: 'https://api.stepfun.com/v1', model: 'stepaudio-2.5-asr' },
{ id: 'zhipu', nameKey: 'asrZhipu', baseUrl: 'https://open.bigmodel.cn/api/paas/v4', model: 'glm-asr-2512' },
{ id: 'groq', nameKey: 'asrGroq', baseUrl: 'https://api.groq.com/openai/v1', model: 'whisper-large-v3-turbo' },
{ id: 'whisper', nameKey: 'asrWhisper', baseUrl: 'https://api.openai.com/v1', model: 'whisper-1' },