From 2384351ad86ab4ef575fd5cbbdacaf29f898f1f5 Mon Sep 17 00:00:00 2001 From: baiqing Date: Mon, 11 May 2026 23:37:42 +0800 Subject: [PATCH] docs: refresh CLAUDE.md + localize empty-ASR capsule message MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CLAUDE.md: - 模块表加上 coordinator 拆分（dictation/qa/resources）、coordinator_state、qa_hotkey 与三套并行 hotkey 监听（hotkey / combo_hotkey / qa_hotkey + global_hotkey_runtime） - 新增 Q&A pipeline 章节，记录 panel 路由、capsule sweep、selection 3-tier fallback、 selection 4000 char 截断与多轮 messages 行为 - 新增 Insertion paths 章节，区分跨平台默认路径与 Windows IME-IPC 路径 - 顶部 adjacent docs 段落补 AGENTS.md / README.zh.md / USAGE.md 指引 dictation.rs: - 空转写场景 capsule 文案 "ASR returned empty transcript" → "没有识别到语音"， 与界面其他位置一致。line 907 的内部 Err 字符串保留英文（属于上层日志/调用方 消费的内部 error code，不直接对用户显示） --- CLAUDE.md | 85 +++++++++++++++---- .../src-tauri/src/coordinator/dictation.rs | 2 +- 2 files changed, 69 insertions(+), 18 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ddad74c9..71ddd9b1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,6 +10,10 @@ The active codebase lives at `openless-all/app/` and is **Tauri 2 + Rust backend UI must match `openless-all/design_handoff_openless/*.jsx` pixel-for-pixel; the JSX is reference-only, never imported. +Adjacent docs: +- `AGENTS.md` is the parallel of this file for **Codex** sessions; the research-before-coding rules at the bottom of this file delegate to it. +- `README.md` / `README.zh.md` (root) are user-facing install + feature guides; `USAGE.md` covers runtime usage. Update them when shipping user-visible features, not for internal refactors. + ## Build, Run, Test ### Tauri (current — start here) @@ -53,27 +57,48 @@ There is no test runner wired in for the frontend. `src/lib/providerSetup.test.t ## Architecture -`coordinator::Coordinator` is the **single owner of session state**. Hotkey edges drive a small phase enum (`Idle → Starting → Listening → Processing`); recorder, ASR, polish, insertion, and history are wired here and nowhere else. Library/module code never calls across modules — they each depend only on shared types. +`coordinator::Coordinator` is the **single owner of all session state** — both the dictation phase machine (`Idle → Starting → Listening → Processing → Inserting → Done`) **and** the parallel QA phase machine (`Idle → Recording → Processing`). Hotkey edges drive both. Recorder, ASR, polish, insertion, selection capture, and history are wired here and nowhere else. Leaf modules never call across each other — they each depend only on `types.rs`. + +The coordinator was split into a module: `coordinator.rs` is the public entry; `coordinator/{dictation,qa,resources}.rs` carry per-pipeline logic; `coordinator_state.rs` is the pure (no Tauri / audio / clipboard) state-transition layer that makes phase decisions unit-testable. ``` -Rust (openless-all/app/src-tauri/src) Purpose -────────────────────────────────────── ──────────────────────────────── -types.rs Pure value types: DictationSession, PolishMode, HotkeyBinding, errors -hotkey.rs Global hotkey monitor (modifier-key edges) -recorder.rs Mic → 16 kHz mono Int16 PCM, RMS callback -asr/{mod,frame,volcengine,whisper}.rs ASR providers: Volcengine streaming WebSocket + Whisper HTTP -polish.rs OpenAI-compatible chat completions (Ark / DeepSeek / etc.) -insertion.rs AX focused-element write → clipboard + Cmd+V → copy-only fallback -persistence.rs History/preferences/vocab JSON + platform credential vault -coordinator.rs + commands.rs + lib.rs State machine, IPC surface, tray icon, window plumbing -permissions.rs TCC checks (Accessibility / Microphone) +Rust (openless-all/app/src-tauri/src) Purpose +────────────────────────────────────────── ──────────────────────────────── +types.rs Pure value types: sessions, PolishMode, HotkeyBinding, errors, QaChatMessage +coordinator.rs Public entry; owns Inner, hotkey wiring, capsule emits +coordinator/{dictation,qa,resources}.rs Dictation pipeline / QA pipeline / shared helpers (begin/end/cancel) +coordinator_state.rs Pure state transitions — Tauri-free, unit-testable +commands.rs + lib.rs + main.rs IPC surface (`invoke_handler!`), tray icon, window plumbing, entry +permissions.rs TCC checks (Accessibility / Microphone / AppleEvents) + +— Hotkeys (three parallel monitors) — +hotkey.rs Modifier-only hotkey via native CGEventTap (macOS) / rdev (Win/Linux) +combo_hotkey.rs Custom-combo dictation hotkey (when user picks combo over modifier-only) +qa_hotkey.rs QA toggle hotkey (default Cmd/Ctrl+Shift+;) via `global-hotkey` crate +global_hotkey_runtime.rs Shared `global-hotkey` Carbon/Win event runtime (combo + QA share it) +shortcut_binding.rs Shared parse/validate of user-configurable bindings + +— Audio / ASR / LLM — +recorder.rs Mic → 16 kHz mono Int16 PCM, RMS callback +audio_mute.rs System-output mute guard while recording (RAII) +asr/{mod,frame,volcengine,whisper}.rs + asr/local/* ASR providers: Volcengine streaming WS, Whisper HTTP, Bailian, local Foundry +polish.rs OpenAI-compatible chat completions (Ark / DeepSeek / Codex OAuth reuse) +llm_gemini.rs Native Google Gemini client — NOT OpenAI-compatible (separate auth, thinkingConfig, role:model) +correction.rs User-defined correction rules (separate from vocab dictionary) + +— Insertion (two paths) — +insertion.rs AX focused-element write → clipboard + paste shortcut → copy-only fallback +windows_ime_{ipc,profile,protocol,session}.rs Windows IME-side text injection over IPC (parallel insertion path; activates OpenLess TSF profile and submits text via named pipe) +selection.rs Cross-platform selection capture for QA: macOS AX → Cmd/Ctrl+C simulate-copy → Linux PRIMARY (best-effort) + +persistence.rs history.json / preferences.json / dictionary.json + platform credential vault Frontend (openless-all/app/src) -src/components/Capsule.tsx Capsule view + state enum -src/ (React) Main window UI: Overview / History / Vocab / Style / Settings -src/i18n/ react-i18next init + zh-CN / en resources -src/pages/_atoms.tsx Recoil atoms — global frontend state -src/state/HotkeySettingsContext.tsx HotkeySettings React context (capability + binding from backend) +src/components/Capsule.tsx Capsule view + state enum +src/ (React) Main window UI: Overview / History / Vocab / Style / Settings +src/i18n/ react-i18next init + zh-CN / en resources (zh-CN is source of truth) +src/pages/_atoms.tsx Recoil atoms — global frontend state +src/state/HotkeySettingsContext.tsx HotkeySettings React context (capability + binding from backend) ``` ### Dictation pipeline @@ -89,6 +114,32 @@ Invariants: - **`BufferingAudioConsumer`** queues PCM until the WebSocket is ready, then drains. Recorder always pushes to it; ASR is attached after `openSession` resolves. - **Hotkey is toggle-only**, not press-and-hold. The monitor yields one edge per modifier-key keydown; the coordinator interprets odd/even. +### Q&A pipeline (selection-based ask-the-LLM) + +Parallel state machine, lives in `coordinator/qa.rs` + `qa_hotkey.rs` + `selection.rs`. Default trigger: `Cmd+Shift+;` (macOS) / `Ctrl+Shift+;` (Win/Linux). + +``` +QA hotkey edge → toggle panel: open → capture front_app, clear messages, show QA window + close → cancel session, hide window, sweep capsule +Option/dictation edge → routed by panel_visible flag (see below): + while panel_visible & dictation Idle → handle_qa_option_edge: + QaPhase::Idle → begin_qa_session: capture_selection() → Recorder.start → ASR.openSession + QaPhase::Recording → end_qa_session: Recorder.stop → ASR final → LLM (with selection as context) → emit qa:state + QaPhase::Processing→ ignored (LLM in flight) + otherwise handle_pressed (normal dictation) +``` + +Invariants & gotchas: +- **Hotkey routing.** When the QA panel is visible, the dictation hotkey edge routes to QA — *unless* a dictation session is already mid-flight (`Starting/Listening/Processing/Inserting`), in which case the edge stays with dictation. Otherwise QA's `begin_qa_session` would race for the same mic device (cpal rejects the second `build_input_stream` on macOS/Win, PipeWire opens two streams on Linux — neither is recoverable from the QA panel UI). See audit 3.3.1 in `coordinator/dictation.rs`. +- **Capsule sweep on panel open.** Open emits a fresh `CapsuleState::Idle` *only if* dictation is Idle. If dictation is Recording/Polishing/Inserting/Done, the sweep is suppressed so the user's in-flight feedback isn't wiped. See audit 3.3.4. +- **Selection capture is a 3-tier fallback** (`selection.rs`): (1) macOS AX `kAXSelectedTextAttribute` direct read, no clipboard touched; (2) macOS/Windows simulate Cmd/Ctrl+C → snapshot + restore original clipboard, 80 ms read window; (3) Linux PRIMARY via `wl-paste` / `xclip` / `xsel`, best-effort. Returns `None` when the user genuinely selected nothing. +- **Selection truncation.** Hard cap 4000 chars; over → keep first 2000 + `[…truncated…]` + last 2000. Don't raise this without checking LLM context budgeting — Gemini and Ark have different limits. +- **Multi-turn memory.** `QaSessionState.messages` accumulates `user→assistant` pairs across turns within a single panel session; closing the panel clears them. + +### Insertion paths + +`insertion.rs` is the cross-platform default. On Windows there is a **second insertion path** in `windows_ime_{ipc,profile,protocol,session}.rs` that activates a TSF profile (CLSID + GUID baked in `windows_ime_profile.rs`) and submits text over a named-pipe IPC. The coordinator picks one based on user preference / fallback status; both routes return the same `InsertStatus` (`Inserted` / `CopiedFallback`). When changing insertion behavior, decide which path you're touching — they don't share code. + ### Permissions, credentials, on-disk state - **Bundle ID `com.openless.app`** is hard-coded in `openless-all/app/src-tauri/tauri.conf.json` and `CredentialsVault.serviceName`. Changing it breaks system credential vault lookups *and* every existing TCC grant. diff --git a/openless-all/app/src-tauri/src/coordinator/dictation.rs b/openless-all/app/src-tauri/src/coordinator/dictation.rs index 0a92b475..2675d1b6 100644 --- a/openless-all/app/src-tauri/src/coordinator/dictation.rs +++ b/openless-all/app/src-tauri/src/coordinator/dictation.rs @@ -898,7 +898,7 @@ pub(super) async fn end_session(inner: &Arc) -> Result<(), String> { CapsuleState::Error, 0.0, elapsed, - Some("ASR returned empty transcript".to_string()), + Some("没有识别到语音".to_string()), None, ); restore_prepared_windows_ime_session(inner, current_session_id);