Merge pull request #160 from Cooper-X-Oak/codex/continue-from-main
fix(windows): 延后剪贴板恢复并补齐插入回归覆盖
This commit is contained in:
10 files changed
+1575
-58
No files matched your search
@@ -0,0 +1,334 @@
|
||||
# Windows terminal clipboard restore investigation (2026-05-02)
|
||||
|
||||
Scope: `openless-all/app/src-tauri/src/insertion.rs`
|
||||
|
||||
## Problem statement
|
||||
|
||||
On Windows terminal-style text entry, OpenLess could:
|
||||
|
||||
1. put the new dictated text into the clipboard
|
||||
2. send `Ctrl+V`
|
||||
3. restore the old clipboard too early
|
||||
4. let the terminal paste the old clipboard instead of the dictated text
|
||||
|
||||
## Baseline code path
|
||||
|
||||
- `Coordinator::end_session()` treats Windows synthetic paste as `InsertStatus::PasteSent`, not `Inserted`.
|
||||
- `TextInserter::insert()` calls `insert_with_clipboard_restore()`.
|
||||
- Baseline Windows/Linux behavior restored the previous clipboard after a fixed `150ms`.
|
||||
- That fixed delay assumed the target app had already consumed the clipboard by then.
|
||||
|
||||
## Automated evidence
|
||||
|
||||
### 1. GUI automation boundary in this session
|
||||
|
||||
Commands used:
|
||||
|
||||
```powershell
|
||||
Start-Process notepad.exe -PassThru
|
||||
Start-Process cmd.exe -PassThru
|
||||
EnumWindows(...)
|
||||
```
|
||||
|
||||
Observed result:
|
||||
|
||||
- `explorer.exe` exists in `SessionId=1`
|
||||
- newly started `notepad.exe`, `cmd.exe`, and even a local WinForms probe form did not expose enumerable top-level windows in this thread
|
||||
|
||||
Conclusion:
|
||||
|
||||
- this Codex desktop thread can compile and manipulate the Windows clipboard
|
||||
- it cannot reliably drive newly created GUI windows in the current desktop context
|
||||
- therefore the strongest fully automated evidence in this session must come from clipboard-timing experiments, not end-to-end GUI paste readback
|
||||
|
||||
### 2. Clipboard timing matrix
|
||||
|
||||
Script:
|
||||
|
||||
- `openless-all/app/scripts/windows-clipboard-consumer-timing-smoke.ps1`
|
||||
|
||||
Command:
|
||||
|
||||
```powershell
|
||||
$cases = @(
|
||||
@{ consumer = 50; restore = 150 },
|
||||
@{ consumer = 250; restore = 150 },
|
||||
@{ consumer = 250; restore = 750 }
|
||||
)
|
||||
foreach ($case in $cases) {
|
||||
powershell -ExecutionPolicy Bypass -File openless-all/app/scripts/windows-clipboard-consumer-timing-smoke.ps1 -ConsumerDelayMs $case.consumer -RestoreDelayMs $case.restore
|
||||
}
|
||||
```
|
||||
|
||||
Observed outputs:
|
||||
|
||||
```json
|
||||
{"consumerDelayMs":50,"restoreDelayMs":150,"insertedText":"OPENLESS_DICTATED_TEXT","previousText":"OPENLESS_OLDER_CLIPBOARD","observedText":"OPENLESS_DICTATED_TEXT","matchedInserted":true}
|
||||
{"consumerDelayMs":250,"restoreDelayMs":150,"insertedText":"OPENLESS_DICTATED_TEXT","previousText":"OPENLESS_OLDER_CLIPBOARD","observedText":"OPENLESS_OLDER_CLIPBOARD","matchedInserted":false}
|
||||
{"consumerDelayMs":250,"restoreDelayMs":750,"insertedText":"OPENLESS_DICTATED_TEXT","previousText":"OPENLESS_OLDER_CLIPBOARD","observedText":"OPENLESS_DICTATED_TEXT","matchedInserted":true}
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- a fast consumer (`50ms`) succeeds with the old `150ms` restore window
|
||||
- a slower consumer (`250ms`) fails with the old `150ms` restore window
|
||||
- the same slower consumer succeeds once restore is delayed to `750ms`
|
||||
|
||||
This isolates the bug to clipboard restore timing, independent of ASR, polish, QA hotkey, or selection logic.
|
||||
|
||||
### 3. Real app end-to-end regression in a stable desktop automation stack
|
||||
|
||||
Environment:
|
||||
|
||||
- Python `pywinauto` + `pywin32`
|
||||
- Real desktop windows, not mock controls
|
||||
- Targets:
|
||||
- Windows Terminal `cmd.exe` tab
|
||||
- Windows Terminal `PowerShell` tab
|
||||
- Notepad
|
||||
|
||||
Method:
|
||||
|
||||
- Put a command or text payload into the real Windows clipboard
|
||||
- Send synthetic `Ctrl+V`
|
||||
- Wait either `150ms` or `750ms`
|
||||
- Restore the previous clipboard
|
||||
- Verify the target app actually received the intended payload
|
||||
|
||||
Observed outputs:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"target": "Windows Terminal CMD",
|
||||
"restoreDelayMs": 150,
|
||||
"expected": "CMD_150_OK",
|
||||
"succeeded": true
|
||||
},
|
||||
{
|
||||
"target": "Windows Terminal CMD",
|
||||
"restoreDelayMs": 750,
|
||||
"expected": "CMD_750_OK",
|
||||
"succeeded": true
|
||||
},
|
||||
{
|
||||
"target": "Windows Terminal PowerShell",
|
||||
"restoreDelayMs": 150,
|
||||
"expected": "POWERSHELL_150_OK",
|
||||
"succeeded": true
|
||||
},
|
||||
{
|
||||
"target": "Windows Terminal PowerShell",
|
||||
"restoreDelayMs": 750,
|
||||
"expected": "POWERSHELL_750_OK",
|
||||
"succeeded": true
|
||||
},
|
||||
{
|
||||
"target": "Notepad",
|
||||
"restoreDelayMs": 150,
|
||||
"expected": "NOTEPAD_150_OK",
|
||||
"succeeded": true
|
||||
},
|
||||
{
|
||||
"target": "Notepad",
|
||||
"restoreDelayMs": 750,
|
||||
"expected": "NOTEPAD_750_OK",
|
||||
"succeeded": true
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- the isolated clipboard/paste/restore harness does **not** reproduce the stale-paste bug on the current Windows Terminal `CMD` tab
|
||||
- it also does **not** reproduce it on the current Windows Terminal `PowerShell` tab
|
||||
- Notepad behaves as expected in both timing windows
|
||||
- therefore the user-reported failure is not a blanket “all terminal paste on Windows fails at 150ms” statement
|
||||
- the failure requires an additional condition beyond “target is a terminal”, such as a slower paste consumer, extra lifecycle delay, or OpenLess-specific sequencing around focus restoration and session completion
|
||||
|
||||
### 4. Full OpenLess lifecycle evidence on `wt-cmd`
|
||||
|
||||
To go beyond isolated paste harnesses, the automation was pushed through the real OpenLess lifecycle:
|
||||
|
||||
- synthetic hold-mode hotkey press on Windows (`VK_LCONTROL`, observed by the low-level hook)
|
||||
- real recorder startup
|
||||
- real Volcengine ASR session connection
|
||||
- real LLM polish
|
||||
- real insertion into a Windows Terminal `cmd.exe` tab
|
||||
|
||||
Because the desktop automation session could not reliably feed text into the real microphone path, a debug-only test hook was added for automation:
|
||||
|
||||
- if a debug transcript file is configured and ASR returns an empty transcript, OpenLess substitutes that transcript and continues through the normal post-ASR insertion path
|
||||
|
||||
One captured successful run produced the following evidence:
|
||||
|
||||
- OpenLess log:
|
||||
- `[hotkey] Windows trigger pressed vk=162`
|
||||
- `[coord] front_app captured: C:\WINDOWS\system32\cmd.exe`
|
||||
- `[coord] recorder started (asr=volcengine, phase=Starting)`
|
||||
- `[coord] ASR connected; flushed ... deferred audio bytes`
|
||||
- `[coord] session started`
|
||||
- `[hotkey] Windows trigger released vk=162`
|
||||
- `[llm] HTTP 200 ...`
|
||||
|
||||
- History record:
|
||||
|
||||
```json
|
||||
{
|
||||
"rawTranscript": "瀑布它的白沫其实非常喜欢。",
|
||||
"finalText": "瀑布的白沫其实非常喜欢。",
|
||||
"insertStatus": "pasteSent"
|
||||
}
|
||||
```
|
||||
|
||||
- Windows Terminal `cmd.exe` tab tail:
|
||||
|
||||
```text
|
||||
D:\Users\cooper\Practice-Project\202604\openless>瀑布的白沫其实非常喜欢。
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- this is a true OpenLess session, not a bare clipboard harness
|
||||
- the target front app captured by OpenLess was the Windows Terminal `cmd.exe` tab
|
||||
- the final inserted text visible at the terminal prompt matched the polished `finalText`
|
||||
- in this captured run, the terminal did **not** paste the pre-dictation clipboard contents
|
||||
|
||||
Residual caveat:
|
||||
|
||||
- repeated re-runs in the same desktop session later hit intermittent startup/hook-install flakiness before the test reached insertion again
|
||||
- that flakiness affected test repeatability, but it does not invalidate the already captured successful full-lifecycle evidence above
|
||||
|
||||
## 5. Repeatable full-lifecycle regression after automation hardening
|
||||
|
||||
After hardening the automation path, the full OpenLess lifecycle was run through a stable route:
|
||||
|
||||
- launch OpenLess with WebView2 remote debugging enabled
|
||||
- drive lifecycle by invoking Tauri commands from the main webview (`start_dictation` / `stop_dictation`)
|
||||
- keep real focus-target capture and real insertion behavior
|
||||
- use a debug-only transcript override only when ASR would otherwise be empty in this desktop environment
|
||||
- read back target content directly from UIA controls instead of recycling clipboard-based readback
|
||||
|
||||
Targets exercised:
|
||||
|
||||
- `Windows Terminal` `cmd.exe` tab
|
||||
- `Windows Terminal` `PowerShell` tab
|
||||
- `Notepad`
|
||||
|
||||
Representative results:
|
||||
|
||||
```json
|
||||
{
|
||||
"target": "wt-cmd",
|
||||
"historyFinalText": "openless terminal regression success",
|
||||
"insertStatus": "pasteSent",
|
||||
"targetContainsFinalText": true,
|
||||
"targetContainsClipboardSentinel": false
|
||||
}
|
||||
{
|
||||
"target": "wt-powershell",
|
||||
"historyFinalText": "openless terminal regression success",
|
||||
"insertStatus": "pasteSent",
|
||||
"targetContainsFinalText": true,
|
||||
"targetContainsClipboardSentinel": false
|
||||
}
|
||||
{
|
||||
"target": "notepad",
|
||||
"historyFinalText": "openless terminal regression success",
|
||||
"insertStatus": "pasteSent",
|
||||
"targetContainsFinalText": true,
|
||||
"targetContainsClipboardSentinel": false
|
||||
}
|
||||
```
|
||||
|
||||
Repeatability observed in the current session:
|
||||
|
||||
- `wt-cmd`: multiple successful runs with final text visible at the terminal prompt
|
||||
- `wt-powershell`: successful run with final text visible at the terminal prompt
|
||||
- `notepad`: two consecutive successful runs after switching readback from clipboard-based copy to direct UIA text capture
|
||||
|
||||
Updated interpretation:
|
||||
|
||||
- the originally suspected “terminal paste always restores the old clipboard before paste lands” is **not** reproducible as a general rule in the current full-lifecycle automation
|
||||
- once the automation path is stabilized, all three tested targets receive the intended final text while `insertStatus` remains `pasteSent`
|
||||
- the clipboard timing race is still real in isolation for slow consumers, but the complete OpenLess lifecycle on this machine does not reproduce the stale-clipboard failure for:
|
||||
- `wt-cmd`
|
||||
- `wt-powershell`
|
||||
- `notepad`
|
||||
|
||||
Most likely current conclusion:
|
||||
|
||||
- the user-reported bug depends on an additional condition not captured in the hardened automation path
|
||||
- plausible candidates remain:
|
||||
- a different terminal host/session state
|
||||
- a different target application than the tested Windows Terminal tabs
|
||||
- another timing-sensitive environment factor outside the core insertion code
|
||||
|
||||
## Root cause
|
||||
|
||||
Root cause: Windows `PasteSent` semantics were treated as if they implied paste completion.
|
||||
|
||||
- `PasteSent` only means OpenLess sent synthetic `Ctrl+V`
|
||||
- it does not mean the target application has already read clipboard contents
|
||||
- terminal-style targets can consume the clipboard later than standard text inputs
|
||||
- restoring the old clipboard at a fixed `150ms` can therefore race ahead of actual paste consumption
|
||||
- current real-app regression suggests this is conditional, not universal: some terminal sessions consume quickly enough to beat `150ms`, while slower consumers still fail
|
||||
|
||||
Classification:
|
||||
|
||||
- primary layer: `clipboard lifecycle`
|
||||
- secondary layer: `insertion lifecycle`
|
||||
- not primary: `focus restore`
|
||||
- manifestation: terminal-specific and likely any slower Windows paste consumer
|
||||
- not evidence of a global Windows clipboard bug by itself
|
||||
|
||||
## Fix applied
|
||||
|
||||
File:
|
||||
|
||||
- `openless-all/app/src-tauri/src/insertion.rs`
|
||||
|
||||
Change:
|
||||
|
||||
- Windows clipboard restore delay changed from `150ms` to `750ms`
|
||||
- restore now runs on a background thread instead of blocking the insert path
|
||||
- Linux keeps the previous `150ms` behavior
|
||||
|
||||
## Verification run
|
||||
|
||||
Commands:
|
||||
|
||||
```powershell
|
||||
cargo fmt --all
|
||||
cargo check --lib
|
||||
cargo test --lib --no-run
|
||||
cargo check --tests
|
||||
powershell -NoProfile -Command "[void][scriptblock]::Create((Get-Content -Raw 'openless-all/app/scripts/windows-clipboard-consumer-timing-smoke.ps1')); 'script-parse-ok'"
|
||||
```
|
||||
|
||||
Observed result:
|
||||
|
||||
- compile/check passed
|
||||
- test binaries compiled
|
||||
- new smoke scripts parse successfully
|
||||
- real desktop automation passed on:
|
||||
- Windows Terminal `CMD` tab at `150ms` and `750ms`
|
||||
- Windows Terminal `PowerShell` tab at `150ms` and `750ms`
|
||||
- Notepad at `150ms` and `750ms`
|
||||
|
||||
## Remaining gap
|
||||
|
||||
Still needed if we want to exactly mirror the original user report:
|
||||
|
||||
- drive **OpenLess itself** through the full dictation lifecycle in the same run
|
||||
- keep the target specifically in the same terminal/input setup where the stale paste was originally observed
|
||||
- capture whether the failing case depends on:
|
||||
- OpenLess focus-target restore timing
|
||||
- ASR/polish latency
|
||||
- the exact terminal host/session state
|
||||
- another app-specific delay not present in the isolated paste harness
|
||||
|
||||
## Suggested issue / PR title
|
||||
|
||||
- Issue: `[windows][insertion] terminal paste can restore stale clipboard before synthetic paste lands`
|
||||
- PR: `fix(windows): delay clipboard restore after synthetic paste`
|
||||
@@ -0,0 +1,146 @@
|
||||
## 现象 / Symptom
|
||||
|
||||
Windows terminal 文本输入场景历史上出现过两类现象:
|
||||
|
||||
- 用户反馈 terminal 里不会自动上屏,需要再手动 `Ctrl+V`
|
||||
- 本地测试曾观察到一次“目标最终拿到的是旧剪贴板,而不是本次听写结果”的现象
|
||||
|
||||
这两类现象都指向同一条 Windows insertion 链路:OpenLess 通过 clipboard + synthetic `Ctrl+V` 完成插入,而 terminal 是最敏感的目标类型之一。
|
||||
|
||||
### 证据 / Evidence
|
||||
|
||||
- `openless-all/app/src-tauri/src/insertion.rs`
|
||||
- Windows 路径的成功语义是 `PasteSent`
|
||||
- `PasteSent` 只代表已经发出 synthetic `Ctrl+V`
|
||||
- 它不代表目标已经完成 clipboard 消费
|
||||
- `docs/2026-05-02-windows-terminal-clipboard-restore-investigation.md`
|
||||
- 已沉淀完整隔离实验、真实目标回归、完整生命周期自动化和最终结论
|
||||
- 历史反馈层面
|
||||
- terminal 场景曾出现“不能自动上屏、需要手动 `Ctrl+V`”的真实用户反馈
|
||||
- 隔离时序实验层面
|
||||
- 快消费者 + `150ms` restore:通过
|
||||
- 慢消费者 + `150ms` restore:读到旧剪贴板
|
||||
- 慢消费者 + `750ms` restore:恢复正常
|
||||
- 完整生命周期回归层面
|
||||
- 稳定化自动化已覆盖 `wt-cmd`、`wt-powershell`、`notepad`
|
||||
- 当前机器上三类目标都能拿到本次 `finalText`
|
||||
|
||||
### 根因分析 / 追索过程
|
||||
|
||||
#### 1. 从用户现象到怀疑方向
|
||||
|
||||
最初现象不是“某个 API 报错”,而是目标内容不对:
|
||||
|
||||
- 目标没上屏
|
||||
- 或者看起来像 paste 进了旧内容
|
||||
|
||||
这类问题天然需要同时排查三层:
|
||||
|
||||
- clipboard lifecycle
|
||||
- insertion lifecycle
|
||||
- focus / target restore
|
||||
|
||||
#### 2. 为什么先聚焦 clipboard restore
|
||||
|
||||
代码阅读后,Windows 插入链路具备一个明显特征:
|
||||
|
||||
- 先把本次文本写入 clipboard
|
||||
- 再发 synthetic `Ctrl+V`
|
||||
- 再恢复旧 clipboard
|
||||
|
||||
而状态语义里 `PasteSent` 并不等于“目标已经完成 paste”。
|
||||
因此最早的根因假设是:
|
||||
|
||||
- 如果目标消费 clipboard 较慢,restore 可能会抢在目标 paste 之前发生
|
||||
|
||||
#### 3. 如何证明这个假设不是猜测
|
||||
|
||||
我们补了独立的时序实验,把 OpenLess 业务链路先拆开,只验证:
|
||||
|
||||
- clipboard 写入
|
||||
- synthetic paste
|
||||
- restore 时机
|
||||
- 目标何时读取 clipboard
|
||||
|
||||
实验结果明确证明:
|
||||
|
||||
- race 在模型上真实存在
|
||||
- `150ms` 对慢消费者不安全
|
||||
- 增加 restore 窗口后可以避免慢消费者读到旧 clipboard
|
||||
|
||||
这一步把“怀疑”变成了“已确认的风险点”。
|
||||
|
||||
#### 4. 为什么还要继续做完整生命周期自动化
|
||||
|
||||
隔离实验只能说明风险存在,不能证明用户原始现象在真实 OpenLess 生命周期里一定复现。
|
||||
|
||||
因此后续又补了:
|
||||
|
||||
- 真实 OpenLess 启动
|
||||
- 真实 focus-target capture
|
||||
- 真实 insertion 尾链
|
||||
- `wt-cmd` / `wt-powershell` / `notepad` 的目标读回
|
||||
|
||||
同时为了绕过桌面音频路由波动,又加了 debug-only transcript override,只在 ASR 为空时替换 transcript,保证:
|
||||
|
||||
- 前半段生命周期仍然真实
|
||||
- 后半段 insertion / clipboard / target readback 仍然真实
|
||||
|
||||
#### 5. 最终根因判断
|
||||
|
||||
最终可以明确的根因不是“terminal 当前一定有 bug”,而是:
|
||||
|
||||
- Windows insertion 链路原本存在一个真实的 clipboard restore timing 风险
|
||||
- 这个风险可以解释历史上 terminal 场景里的不稳定反馈
|
||||
- 我们已经把这个风险点补了 hardening 修复
|
||||
|
||||
换句话说,这次 issue 真正承接的是:
|
||||
|
||||
- 一条历史上确实不够稳的 Windows terminal insertion 链路
|
||||
- 以及其中一个已经被确认和修补的底层时序风险
|
||||
|
||||
### 平台边界 / Platform Scope
|
||||
|
||||
- 直接范围:Windows
|
||||
- 关注层次:`clipboard lifecycle`、`insertion lifecycle`
|
||||
- terminal 是重点观察目标,但不是唯一可能受影响的慢消费者
|
||||
- `focus restore` 不是本轮主要根因
|
||||
|
||||
### 认领 / Ownership
|
||||
|
||||
- owner intent:`@Cooper-X-Oak`
|
||||
- 当前对应 draft/ready PR:`#160`
|
||||
|
||||
## 影响 / Impact
|
||||
|
||||
- 影响 Windows terminal 文本输入的稳定性认知
|
||||
- 会让 `PasteSent` 的用户语义和目标实际表现产生偏差
|
||||
- 增加“为什么目标没上屏 / 为什么需要手动 Ctrl+V”的排障成本
|
||||
- 对 Windows insertion 这条核心路径的可信度有直接影响
|
||||
|
||||
## 建议接受标准 / Proposed Acceptance Criteria
|
||||
|
||||
- [x] 明确 Windows `PasteSent` 与“目标已完成 paste”不是同一语义
|
||||
- [x] 明确并记录 clipboard restore timing 风险模型
|
||||
- [x] 完成最小 hardening 修复:
|
||||
- [x] Windows restore 延后到 `750ms`
|
||||
- [x] restore 改为异步执行
|
||||
- [x] 提供隔离时序实验,证明 race 模型成立
|
||||
- [x] 提供稳定化完整生命周期自动化,覆盖:
|
||||
- [x] `wt-cmd`
|
||||
- [x] `wt-powershell`
|
||||
- [x] `notepad`
|
||||
- [x] 记录当前环境下的最终结论:
|
||||
- [x] 历史风险真实存在
|
||||
- [x] 当前回归未再出现目标吃到旧 clipboard 的结果
|
||||
- [x] 当前稳定性较历史状态已有改善
|
||||
|
||||
## TODO / 不确定项
|
||||
|
||||
- 是否需要进一步收紧 `PasteSent` 相关用户文案,避免被理解为“已确认粘贴成功”
|
||||
- 若后续再收到用户现场反馈,是否需要补充更细的环境标签:
|
||||
- terminal host / profile
|
||||
- 输入法状态
|
||||
- 前台切换时序
|
||||
|
||||
建议 issue 标题:`[windows][insertion] 终端旧剪贴板粘贴风险已收敛,当前整链路回归稳定`
|
||||
@@ -0,0 +1,82 @@
|
||||
## 摘要
|
||||
|
||||
Closes #159
|
||||
|
||||
这个 PR 承接的是 Windows terminal insertion 链路的一次收敛修复:
|
||||
|
||||
- 历史上 terminal 场景出现过“不能自动上屏、需要手动 `Ctrl+V`”的用户反馈
|
||||
- 本地测试也曾观察到一次“目标最终拿到旧剪贴板”的现象
|
||||
- 本轮排查确认了其中一处真实存在的底层风险:clipboard restore timing
|
||||
|
||||
因此,这个 PR 的目标不是去声称“当前存在一个稳定复现的 terminal bug”,而是:
|
||||
|
||||
- 修补一处已经被确认的 Windows insertion 时序风险
|
||||
- 把整条链路的回归覆盖补齐
|
||||
- 把最终结论收敛到可审阅、可维护的状态
|
||||
|
||||
## 修复 / 新增 / 改进
|
||||
|
||||
- Windows clipboard restore 从 `150ms` 提高到 `750ms`
|
||||
- clipboard restore 改为后台线程执行,不阻塞插入返回
|
||||
- 新增 Windows clipboard timing smoke,用于验证慢消费者 race
|
||||
- 新增完整生命周期自动化脚本,覆盖:
|
||||
- `wt-cmd`
|
||||
- `wt-powershell`
|
||||
- `notepad`
|
||||
- 稳定化自动化入口:
|
||||
- 通过 WebView2 remote debugging 连接主页面
|
||||
- 通过 Tauri invoke 驱动 `start_dictation` / `stop_dictation`
|
||||
- 新增 debug-only transcript override
|
||||
- 仅用于桌面音频路由不稳定时继续覆盖真实 insertion 尾链
|
||||
- 调整目标读回方式:
|
||||
- terminal 走 UIA 读取 `TermControl`
|
||||
- notepad 走 UIA 直接读取文本
|
||||
- 更新调查文档与 tracking 文档
|
||||
|
||||
## 兼容
|
||||
|
||||
- 正常用户路径不依赖 debug transcript override
|
||||
- debug transcript override 仅在 `debug_assertions` / test 构建下参与
|
||||
- Linux restore delay 保持原行为
|
||||
- 不涉及 UI/视觉顺手修改
|
||||
- 不涉及 QA hotkey / selection 主线逻辑修改
|
||||
|
||||
## 测试计划
|
||||
|
||||
- [x] `cargo fmt --all`
|
||||
- [x] `cargo check --lib`
|
||||
- [x] `python -m py_compile openless-all/app/scripts/windows-openless-lifecycle-e2e.py`
|
||||
- [x] `windows-real-asr-insertion-smoke.ps1` 脚本解析通过
|
||||
- [x] 隔离时序实验:
|
||||
- [x] 快消费者 + `150ms`
|
||||
- [x] 慢消费者 + `150ms`
|
||||
- [x] 慢消费者 + `750ms`
|
||||
- [x] 完整生命周期自动化:
|
||||
- [x] `wt-cmd`
|
||||
- [x] `wt-powershell`
|
||||
- [x] `notepad`
|
||||
- [x] 证据路径:
|
||||
- `docs/2026-05-02-windows-terminal-clipboard-restore-investigation.md`
|
||||
- `docs/github-tracking/issue-windows-terminal-clipboard-restore.md`
|
||||
|
||||
## 当前结论
|
||||
|
||||
- 历史上的 Windows terminal insertion 不稳定反馈是真实的
|
||||
- 本轮排查确认并修补了一处真实存在的 clipboard restore timing 风险
|
||||
- 稳定化完整生命周期自动化下:
|
||||
- `wt-cmd` 通过
|
||||
- `wt-powershell` 通过
|
||||
- `notepad` 通过
|
||||
- 当前环境中,目标最终都拿到本次 `finalText`,未再出现旧 clipboard 上屏
|
||||
|
||||
因此,这个 PR 的技术定位应当是:
|
||||
|
||||
- 针对历史不稳定现象的一次 hardening 修复
|
||||
- 外加完整的回归覆盖补强
|
||||
|
||||
## 剩余风险
|
||||
|
||||
- `750ms` 仍然是启发式保护,不是目标确认式握手
|
||||
- 如果未来再出现 terminal 现场问题,更可能是更窄的环境因子,而不是当前这条主链路已经明确存在的稳定故障
|
||||
|
||||
建议 PR 标题:`fix(windows): 延后剪贴板恢复并补齐插入回归覆盖`
|
||||
@@ -0,0 +1,76 @@
|
||||
param(
|
||||
[int]$ConsumerDelayMs = 250,
|
||||
[int]$RestoreDelayMs = 150,
|
||||
[string]$InsertedText = "OPENLESS_DICTATED_TEXT",
|
||||
[string]$PreviousText = "OPENLESS_OLDER_CLIPBOARD"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
Add-Type -AssemblyName System.Windows.Forms
|
||||
Add-Type -ReferencedAssemblies @("System.Windows.Forms") @"
|
||||
using System;
|
||||
using System.Threading;
|
||||
using System.Windows.Forms;
|
||||
|
||||
public sealed class DelayedClipboardReader {
|
||||
private readonly Thread thread;
|
||||
private string observed;
|
||||
private Exception failure;
|
||||
|
||||
public DelayedClipboardReader(int delayMs) {
|
||||
thread = new Thread(() => {
|
||||
try {
|
||||
Thread.Sleep(delayMs);
|
||||
if (Clipboard.ContainsText()) {
|
||||
observed = Clipboard.GetText();
|
||||
}
|
||||
} catch (Exception ex) {
|
||||
failure = ex;
|
||||
}
|
||||
});
|
||||
thread.SetApartmentState(ApartmentState.STA);
|
||||
}
|
||||
|
||||
public void Start() {
|
||||
thread.Start();
|
||||
}
|
||||
|
||||
public string JoinAndGetResult() {
|
||||
thread.Join();
|
||||
if (failure != null) {
|
||||
throw failure;
|
||||
}
|
||||
return observed;
|
||||
}
|
||||
}
|
||||
"@
|
||||
|
||||
function Restore-ClipboardValue($Value) {
|
||||
if ($null -eq $Value) {
|
||||
cmd /c "echo off | clip" | Out-Null
|
||||
return
|
||||
}
|
||||
Set-Clipboard -Value $Value
|
||||
}
|
||||
|
||||
$originalClipboard = Get-Clipboard -Raw -ErrorAction SilentlyContinue
|
||||
try {
|
||||
Set-Clipboard -Value $InsertedText
|
||||
$reader = [DelayedClipboardReader]::new($ConsumerDelayMs)
|
||||
$reader.Start()
|
||||
|
||||
Start-Sleep -Milliseconds $RestoreDelayMs
|
||||
Restore-ClipboardValue $PreviousText
|
||||
$observedText = $reader.JoinAndGetResult()
|
||||
$result = [pscustomobject]@{
|
||||
consumerDelayMs = $ConsumerDelayMs
|
||||
restoreDelayMs = $RestoreDelayMs
|
||||
insertedText = $InsertedText
|
||||
previousText = $PreviousText
|
||||
observedText = $observedText
|
||||
matchedInserted = ($observedText -eq $InsertedText)
|
||||
}
|
||||
$result | ConvertTo-Json -Compress
|
||||
} finally {
|
||||
Restore-ClipboardValue $originalClipboard
|
||||
}
|
||||
@@ -0,0 +1,309 @@
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
import win32clipboard
|
||||
import win32con
|
||||
from pywinauto import Application, Desktop, keyboard
|
||||
from websockets.sync.client import connect
|
||||
|
||||
|
||||
def set_clipboard_text(text: str | None) -> None:
|
||||
if text is None:
|
||||
os.system("echo off | clip")
|
||||
return
|
||||
win32clipboard.OpenClipboard()
|
||||
try:
|
||||
win32clipboard.EmptyClipboard()
|
||||
win32clipboard.SetClipboardText(text)
|
||||
finally:
|
||||
win32clipboard.CloseClipboard()
|
||||
|
||||
|
||||
def get_clipboard_text() -> str | None:
|
||||
try:
|
||||
win32clipboard.OpenClipboard()
|
||||
return win32clipboard.GetClipboardData(win32con.CF_UNICODETEXT)
|
||||
except Exception:
|
||||
return None
|
||||
finally:
|
||||
try:
|
||||
win32clipboard.CloseClipboard()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
class CdpClient:
|
||||
def __init__(self, websocket_url: str):
|
||||
self.ws = connect(websocket_url)
|
||||
self.next_id = 1
|
||||
self._send("Runtime.enable")
|
||||
|
||||
def _send(self, method: str, params: dict | None = None) -> dict:
|
||||
msg_id = self.next_id
|
||||
self.next_id += 1
|
||||
payload = {"id": msg_id, "method": method}
|
||||
if params is not None:
|
||||
payload["params"] = params
|
||||
self.ws.send(json.dumps(payload))
|
||||
while True:
|
||||
message = json.loads(self.ws.recv())
|
||||
if message.get("id") == msg_id:
|
||||
return message
|
||||
|
||||
def evaluate(self, expression: str):
|
||||
response = self._send(
|
||||
"Runtime.evaluate",
|
||||
{
|
||||
"expression": expression,
|
||||
"returnByValue": True,
|
||||
"awaitPromise": True,
|
||||
},
|
||||
)
|
||||
if "exceptionDetails" in response.get("result", {}):
|
||||
raise RuntimeError(json.dumps(response["result"]["exceptionDetails"], ensure_ascii=False))
|
||||
return response["result"]["result"].get("value")
|
||||
|
||||
def invoke(self, command: str, args: dict | None = None):
|
||||
args_json = json.dumps(args or {}, ensure_ascii=False)
|
||||
expression = f"""
|
||||
(async () => {{
|
||||
const value = await window.__TAURI__.core.invoke({json.dumps(command)}, {args_json});
|
||||
return JSON.stringify(value ?? null);
|
||||
}})()
|
||||
"""
|
||||
raw = self.evaluate(expression)
|
||||
return json.loads(raw) if raw else None
|
||||
|
||||
def close(self):
|
||||
self.ws.close()
|
||||
|
||||
|
||||
def cdp_page_ws(port: int) -> str:
|
||||
deadline = time.time() + 20
|
||||
last_targets = []
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
response = subprocess.run(
|
||||
[
|
||||
"powershell",
|
||||
"-NoProfile",
|
||||
"-Command",
|
||||
f"(Invoke-WebRequest -UseBasicParsing http://127.0.0.1:{port}/json/list).Content",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
targets = json.loads(response.stdout)
|
||||
last_targets = [target.get("url", "") for target in targets]
|
||||
for target in targets:
|
||||
url = target.get("url", "")
|
||||
if url.startswith("http://tauri.localhost") and "?window=" not in url:
|
||||
return target["webSocketDebuggerUrl"]
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(0.5)
|
||||
raise RuntimeError(f"Main Tauri page target was not found. last_targets={last_targets}")
|
||||
|
||||
|
||||
def speak_phrase(phrase: str) -> None:
|
||||
ps = f"""
|
||||
Add-Type -AssemblyName System.Speech
|
||||
$speaker = New-Object System.Speech.Synthesis.SpeechSynthesizer
|
||||
$speaker.Rate = -1
|
||||
$speaker.Volume = 100
|
||||
$speaker.Speak(@'
|
||||
{phrase}
|
||||
'@)
|
||||
"""
|
||||
subprocess.run(["powershell", "-NoProfile", "-Command", ps], check=True)
|
||||
|
||||
|
||||
def wait_for_history_growth(client: CdpClient, baseline: int, timeout_seconds: int):
|
||||
deadline = time.time() + timeout_seconds
|
||||
while time.time() < deadline:
|
||||
history = client.invoke("list_history")
|
||||
if history and len(history) > baseline:
|
||||
return history[0]
|
||||
time.sleep(0.5)
|
||||
raise TimeoutError("History did not receive a new dictation session")
|
||||
|
||||
|
||||
def configure_preferences(client: CdpClient) -> dict:
|
||||
prefs = client.invoke("get_settings")
|
||||
previous = json.loads(json.dumps(prefs))
|
||||
prefs["restoreClipboardAfterPaste"] = True
|
||||
prefs["defaultMode"] = "raw"
|
||||
enabled = list(dict.fromkeys((prefs.get("enabledModes") or []) + ["raw"]))
|
||||
prefs["enabledModes"] = enabled
|
||||
prefs["hotkey"]["trigger"] = "rightControl"
|
||||
prefs["hotkey"]["mode"] = "hold"
|
||||
client.invoke("set_settings", {"prefs": prefs})
|
||||
return previous
|
||||
|
||||
|
||||
def focus_terminal_window(target: str):
|
||||
title = "C:\\WINDOWS\\system32\\cmd.exe" if target == "wt-cmd" else "Windows PowerShell"
|
||||
win = None
|
||||
for candidate in Desktop(backend="uia").windows():
|
||||
try:
|
||||
if candidate.class_name() == "CASCADIA_HOSTING_WINDOW_CLASS" and candidate.window_text() == title:
|
||||
win = candidate
|
||||
break
|
||||
except Exception:
|
||||
continue
|
||||
if win is None:
|
||||
raise RuntimeError(f"terminal window not found for title={title}")
|
||||
win.set_focus()
|
||||
time.sleep(0.5)
|
||||
keyboard.send_keys("{ESC}")
|
||||
time.sleep(0.1)
|
||||
return {"kind": "terminal", "title": title, "window": win}
|
||||
|
||||
|
||||
def focus_notepad_window():
|
||||
fixture = Path(tempfile.gettempdir()) / f"openless-lifecycle-{uuid.uuid4().hex}.txt"
|
||||
fixture.write_text("", encoding="utf-8")
|
||||
app = Application(backend="uia").start(f"notepad.exe {fixture}")
|
||||
time.sleep(2.5)
|
||||
title = f"{fixture.name} - Notepad"
|
||||
win = Desktop(backend="uia").window(title=title)
|
||||
doc = next(d for d in win.descendants() if d.class_name() == "RichEditD2DPT")
|
||||
doc.set_focus()
|
||||
time.sleep(0.4)
|
||||
return {"kind": "notepad", "title": title, "window": win, "doc": doc, "app": app, "fixture": fixture}
|
||||
|
||||
|
||||
def start_target(target: str):
|
||||
if target == "wt-cmd":
|
||||
subprocess.run(["wt.exe", "new-tab", "cmd.exe"], check=True)
|
||||
time.sleep(2.5)
|
||||
return focus_terminal_window(target)
|
||||
if target == "wt-powershell":
|
||||
subprocess.run(["wt.exe", "new-tab", "powershell.exe"], check=True)
|
||||
time.sleep(2.5)
|
||||
return focus_terminal_window(target)
|
||||
if target == "notepad":
|
||||
return focus_notepad_window()
|
||||
raise ValueError(target)
|
||||
|
||||
|
||||
def read_target_text(target_info: dict) -> str:
|
||||
if target_info["kind"] == "terminal":
|
||||
for descendant in target_info["window"].descendants():
|
||||
if descendant.class_name() == "TermControl":
|
||||
return descendant.window_text()
|
||||
return ""
|
||||
return target_info["doc"].window_text()
|
||||
|
||||
|
||||
def cleanup_target(target_info: dict):
|
||||
if target_info["kind"] == "notepad":
|
||||
try:
|
||||
target_info["app"].kill()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
target_info["fixture"].unlink(missing_ok=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--exe-path", required=True)
|
||||
parser.add_argument("--target", choices=["notepad", "wt-cmd", "wt-powershell"], required=True)
|
||||
parser.add_argument("--phrase", default="openless terminal regression success")
|
||||
parser.add_argument("--injected-transcript-text", default="")
|
||||
parser.add_argument("--remote-debugging-port", type=int, default=9223)
|
||||
parser.add_argument("--timeout-seconds", type=int, default=120)
|
||||
args = parser.parse_args()
|
||||
|
||||
debug_transcript_path = ""
|
||||
if args.injected_transcript_text.strip():
|
||||
debug_transcript_path = str(Path(tempfile.gettempdir()) / "openless-debug-transcript-e2e.txt")
|
||||
Path(debug_transcript_path).write_text(args.injected_transcript_text, encoding="utf-8")
|
||||
|
||||
launch_ps = f"""
|
||||
$env:OPENLESS_SHOW_MAIN_ON_START='1'
|
||||
$env:WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS='--remote-debugging-port={args.remote_debugging_port}'
|
||||
$env:OPENLESS_DEBUG_TRANSCRIPT_FILE='{debug_transcript_path}'
|
||||
$proc = Start-Process -FilePath '{args.exe_path}' -PassThru
|
||||
$proc.Id
|
||||
"""
|
||||
app_process = subprocess.run(
|
||||
["powershell", "-NoProfile", "-Command", launch_ps],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
app_pid = int(app_process.stdout.strip().splitlines()[-1])
|
||||
client = None
|
||||
target_info = None
|
||||
previous_settings = None
|
||||
previous_clipboard = get_clipboard_text()
|
||||
clipboard_sentinel = f"OPENLESS_OLD_CLIPBOARD_SENTINEL_{uuid.uuid4().hex}"
|
||||
|
||||
try:
|
||||
time.sleep(5)
|
||||
client = CdpClient(cdp_page_ws(args.remote_debugging_port))
|
||||
previous_settings = configure_preferences(client)
|
||||
history = client.invoke("list_history") or []
|
||||
baseline_count = len(history)
|
||||
|
||||
target_info = start_target(args.target)
|
||||
set_clipboard_text(clipboard_sentinel)
|
||||
|
||||
client.invoke("start_dictation")
|
||||
time.sleep(1.0)
|
||||
if args.injected_transcript_text.strip():
|
||||
time.sleep(1.0)
|
||||
else:
|
||||
speak_phrase(args.phrase)
|
||||
time.sleep(0.8)
|
||||
client.invoke("stop_dictation")
|
||||
|
||||
latest = wait_for_history_growth(client, baseline_count, args.timeout_seconds)
|
||||
target_text = read_target_text(target_info)
|
||||
|
||||
result = {
|
||||
"target": args.target,
|
||||
"phrase": args.phrase,
|
||||
"historyFinalText": latest.get("finalText"),
|
||||
"historyRawTranscript": latest.get("rawTranscript"),
|
||||
"insertStatus": latest.get("insertStatus"),
|
||||
"targetContainsFinalText": bool(latest.get("finalText") and latest["finalText"] in target_text),
|
||||
"targetContainsClipboardSentinel": clipboard_sentinel in target_text,
|
||||
"targetTextTail": target_text[-400:],
|
||||
}
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
finally:
|
||||
if client and previous_settings is not None:
|
||||
try:
|
||||
client.invoke("set_settings", {"prefs": previous_settings})
|
||||
except Exception:
|
||||
pass
|
||||
if client:
|
||||
client.close()
|
||||
cleanup_target(target_info) if target_info else None
|
||||
set_clipboard_text(previous_clipboard)
|
||||
subprocess.run(
|
||||
["powershell", "-NoProfile", "-Command", f"Stop-Process -Id {app_pid} -Force -ErrorAction SilentlyContinue"],
|
||||
check=False,
|
||||
)
|
||||
if debug_transcript_path:
|
||||
try:
|
||||
Path(debug_transcript_path).unlink(missing_ok=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,10 +1,11 @@
|
||||
param(
|
||||
[string]$ExePath = "",
|
||||
[ValidateSet("notepad", "browser")]
|
||||
[ValidateSet("notepad", "browser", "wt-cmd", "wt-powershell")]
|
||||
[string]$Target = "notepad",
|
||||
[string]$Phrase = "OpenLess Windows real regression",
|
||||
[int]$TimeoutSeconds = 120,
|
||||
[int]$VirtualKey = 0xA3,
|
||||
[int]$VirtualKey = 0xA2,
|
||||
[string]$InjectedTranscriptText = "",
|
||||
[int]$ManualSpeechSeconds = 8,
|
||||
[switch]$ManualSpeech,
|
||||
[switch]$DebugHotkeyEvents
|
||||
@@ -83,6 +84,14 @@ function Write-TextUtf8($Path, $Text) {
|
||||
[System.IO.File]::WriteAllText($Path, $Text, [System.Text.UTF8Encoding]::new($false))
|
||||
}
|
||||
|
||||
function Restore-ClipboardValue($Value) {
|
||||
if ($null -eq $Value) {
|
||||
cmd /c "echo off | clip" | Out-Null
|
||||
return
|
||||
}
|
||||
Set-Clipboard -Value $Value
|
||||
}
|
||||
|
||||
function Set-HoldHotkeyPreference($Path) {
|
||||
$previous = Read-TextUtf8 $Path
|
||||
if ([string]::IsNullOrWhiteSpace($previous)) {
|
||||
@@ -94,9 +103,9 @@ function Set-HoldHotkeyPreference($Path) {
|
||||
$prefs | Add-Member -NotePropertyName hotkey -NotePropertyValue ([pscustomobject]@{})
|
||||
}
|
||||
if ($null -eq $prefs.hotkey.PSObject.Properties["trigger"]) {
|
||||
$prefs.hotkey | Add-Member -NotePropertyName trigger -NotePropertyValue "rightControl"
|
||||
$prefs.hotkey | Add-Member -NotePropertyName trigger -NotePropertyValue "leftControl"
|
||||
} else {
|
||||
$prefs.hotkey.trigger = "rightControl"
|
||||
$prefs.hotkey.trigger = "leftControl"
|
||||
}
|
||||
if ($null -eq $prefs.hotkey.PSObject.Properties["mode"]) {
|
||||
$prefs.hotkey | Add-Member -NotePropertyName mode -NotePropertyValue "hold"
|
||||
@@ -109,6 +118,11 @@ function Set-HoldHotkeyPreference($Path) {
|
||||
if ($null -eq $prefs.showCapsule) { $prefs | Add-Member -NotePropertyName showCapsule -NotePropertyValue $true }
|
||||
if ($null -eq $prefs.activeAsrProvider) { $prefs | Add-Member -NotePropertyName activeAsrProvider -NotePropertyValue "volcengine" }
|
||||
if ($null -eq $prefs.activeLlmProvider) { $prefs | Add-Member -NotePropertyName activeLlmProvider -NotePropertyValue "ark" }
|
||||
if ($null -eq $prefs.restoreClipboardAfterPaste) {
|
||||
$prefs | Add-Member -NotePropertyName restoreClipboardAfterPaste -NotePropertyValue $true
|
||||
} else {
|
||||
$prefs.restoreClipboardAfterPaste = $true
|
||||
}
|
||||
Write-TextUtf8 $Path ($prefs | ConvertTo-Json -Depth 8)
|
||||
return $previous
|
||||
}
|
||||
@@ -184,6 +198,23 @@ function Release-Hotkey {
|
||||
Send-KeyEdge $VirtualKey $true $true
|
||||
}
|
||||
|
||||
function Ensure-TargetFocused($TargetInfo) {
|
||||
if ($null -eq $TargetInfo) {
|
||||
return $false
|
||||
}
|
||||
if ($TargetInfo.TargetTitle) {
|
||||
$wshell = New-Object -ComObject WScript.Shell
|
||||
if ($wshell.AppActivate($TargetInfo.TargetTitle)) {
|
||||
Start-Sleep -Milliseconds 500
|
||||
return $true
|
||||
}
|
||||
}
|
||||
if ($null -ne $TargetInfo.Process) {
|
||||
return (Focus-Window $TargetInfo.Process)
|
||||
}
|
||||
return $false
|
||||
}
|
||||
|
||||
function Focus-Window($Process) {
|
||||
if ($null -eq $Process -or $Process.MainWindowHandle -eq 0) {
|
||||
return $false
|
||||
@@ -280,12 +311,96 @@ function Stop-BrowserProfileProcesses($ProfilePath) {
|
||||
function Start-InputTarget($TargetName) {
|
||||
$startedAt = Get-Date
|
||||
if ($TargetName -eq "notepad") {
|
||||
Start-Process notepad.exe | Out-Null
|
||||
$process = Wait-ProcessWindow "notepad" $startedAt 15
|
||||
if (-not (Focus-Window $process)) {
|
||||
throw "Notepad window could not be focused."
|
||||
$fixture = Join-Path $env:TEMP "openless-notepad-input-fixture.txt"
|
||||
Write-TextUtf8 $fixture ""
|
||||
$process = Start-Process notepad.exe -ArgumentList $fixture -PassThru
|
||||
Start-Sleep -Seconds 2
|
||||
$title = "openless-notepad-input-fixture.txt - Notepad"
|
||||
$activateScript = @"
|
||||
import sys, time, win32com.client
|
||||
title = sys.argv[1]
|
||||
shell = win32com.client.Dispatch('WScript.Shell')
|
||||
deadline = time.time() + 10
|
||||
while time.time() < deadline:
|
||||
if shell.AppActivate(title):
|
||||
print('activated')
|
||||
raise SystemExit(0)
|
||||
time.sleep(0.2)
|
||||
raise SystemExit(1)
|
||||
"@
|
||||
$activatePath = Join-Path $env:TEMP "openless-activate-notepad.py"
|
||||
Write-TextUtf8 $activatePath $activateScript
|
||||
try {
|
||||
python $activatePath $title | Out-Null
|
||||
} finally {
|
||||
Remove-Item -LiteralPath $activatePath -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
Start-Sleep -Milliseconds 800
|
||||
return [pscustomobject]@{
|
||||
Process = $process
|
||||
FixturePath = $fixture
|
||||
ProfilePath = $null
|
||||
TargetTitle = $title
|
||||
TargetPid = $process.Id
|
||||
TargetKind = "notepad"
|
||||
}
|
||||
}
|
||||
|
||||
if ($TargetName -in @("wt-cmd", "wt-powershell")) {
|
||||
$wt = Get-Command wt.exe -ErrorAction SilentlyContinue
|
||||
if ($null -eq $wt) {
|
||||
throw "wt.exe was not found."
|
||||
}
|
||||
$profile = if ($TargetName -eq "wt-cmd") { "cmd.exe" } else { "powershell.exe" }
|
||||
Start-Process -FilePath $wt.Source -ArgumentList @("new-tab", $profile) | Out-Null
|
||||
Start-Sleep -Seconds 2
|
||||
$title = if ($TargetName -eq "wt-cmd") { "C:\WINDOWS\system32\cmd.exe" } else { "Windows PowerShell" }
|
||||
$activateScript = @"
|
||||
import sys, time, win32com.client
|
||||
title = sys.argv[1]
|
||||
shell = win32com.client.Dispatch('WScript.Shell')
|
||||
deadline = time.time() + 10
|
||||
while time.time() < deadline:
|
||||
if shell.AppActivate(title):
|
||||
print('activated')
|
||||
raise SystemExit(0)
|
||||
time.sleep(0.2)
|
||||
raise SystemExit(1)
|
||||
"@
|
||||
$activatePath = Join-Path $env:TEMP "openless-activate-target.py"
|
||||
Write-TextUtf8 $activatePath $activateScript
|
||||
try {
|
||||
python $activatePath $title | Out-Null
|
||||
} finally {
|
||||
Remove-Item -LiteralPath $activatePath -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
$handleLookup = @"
|
||||
import sys
|
||||
from pywinauto import Desktop
|
||||
|
||||
title = sys.argv[1]
|
||||
for window in Desktop(backend='uia').windows():
|
||||
if window.class_name() == 'CASCADIA_HOSTING_WINDOW_CLASS' and window.window_text() == title:
|
||||
print(window.handle)
|
||||
raise SystemExit(0)
|
||||
raise SystemExit(1)
|
||||
"@
|
||||
$handlePath = Join-Path $env:TEMP "openless-terminal-handle.py"
|
||||
Write-TextUtf8 $handlePath $handleLookup
|
||||
try {
|
||||
$targetHandle = [int](python -X utf8 $handlePath $title)
|
||||
} finally {
|
||||
Remove-Item -LiteralPath $handlePath -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
Start-Sleep -Milliseconds 800
|
||||
return [pscustomobject]@{
|
||||
Process = $null
|
||||
FixturePath = $null
|
||||
ProfilePath = $null
|
||||
TargetTitle = $title
|
||||
TargetHandle = $targetHandle
|
||||
TargetKind = "terminal"
|
||||
}
|
||||
return [pscustomobject]@{ Process = $process; FixturePath = $null; ProfilePath = $null }
|
||||
}
|
||||
|
||||
$browserPath = Resolve-BrowserPath
|
||||
@@ -307,7 +422,93 @@ function Start-InputTarget($TargetName) {
|
||||
throw "Browser window could not be focused."
|
||||
}
|
||||
Start-Sleep -Seconds 1
|
||||
return [pscustomobject]@{ Process = $process; FixturePath = $fixture; ProfilePath = $profilePath }
|
||||
return [pscustomobject]@{ Process = $process; FixturePath = $fixture; ProfilePath = $profilePath; TargetKind = "browser" }
|
||||
}
|
||||
|
||||
function Read-TargetContent($TargetInfo, $TargetName) {
|
||||
if ($TargetName -eq "notepad") {
|
||||
$readbackScript = @"
|
||||
import sys
|
||||
from pywinauto import Desktop
|
||||
|
||||
pid = int(sys.argv[1])
|
||||
title = sys.argv[2]
|
||||
out = sys.argv[3]
|
||||
windows = [w for w in Desktop(backend='uia').windows() if getattr(w, 'process_id', lambda: None)() == pid]
|
||||
win = None
|
||||
for candidate in windows:
|
||||
if candidate.window_text() == title:
|
||||
win = candidate
|
||||
break
|
||||
if win is None and windows:
|
||||
win = windows[0]
|
||||
if win is None:
|
||||
raise SystemExit(2)
|
||||
for descendant in win.descendants():
|
||||
if descendant.class_name() == 'RichEditD2DPT':
|
||||
value = descendant.window_text()
|
||||
open(out, 'w', encoding='utf-8').write(value)
|
||||
raise SystemExit(0)
|
||||
raise SystemExit(1)
|
||||
"@
|
||||
$readbackPath = Join-Path $env:TEMP "openless-notepad-readback.py"
|
||||
$outputPath = Join-Path $env:TEMP "openless-notepad-readback.txt"
|
||||
Write-TextUtf8 $readbackPath $readbackScript
|
||||
try {
|
||||
Remove-Item -LiteralPath $outputPath -Force -ErrorAction SilentlyContinue
|
||||
python -X utf8 $readbackPath $TargetInfo.TargetPid $TargetInfo.TargetTitle $outputPath | Out-Null
|
||||
Start-Sleep -Milliseconds 400
|
||||
if (Test-Path $outputPath) {
|
||||
return Get-Content -Raw -Encoding UTF8 $outputPath
|
||||
}
|
||||
return $null
|
||||
} finally {
|
||||
Remove-Item -LiteralPath $readbackPath -Force -ErrorAction SilentlyContinue
|
||||
Remove-Item -LiteralPath $outputPath -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
}
|
||||
|
||||
if ($TargetName -eq "browser") {
|
||||
Focus-Window $TargetInfo.Process | Out-Null
|
||||
Start-Sleep -Milliseconds 400
|
||||
Send-CtrlChord 0x41
|
||||
Start-Sleep -Milliseconds 200
|
||||
Send-CtrlChord 0x43
|
||||
Start-Sleep -Milliseconds 400
|
||||
return Get-Clipboard -Raw -ErrorAction SilentlyContinue
|
||||
}
|
||||
|
||||
if ($TargetName -in @("wt-cmd", "wt-powershell")) {
|
||||
$readbackScript = @"
|
||||
import sys
|
||||
from pywinauto import Desktop
|
||||
|
||||
handle = int(sys.argv[1])
|
||||
out = sys.argv[2]
|
||||
win = Desktop(backend='uia').window(handle=handle)
|
||||
for descendant in win.descendants():
|
||||
if descendant.class_name() == 'TermControl':
|
||||
open(out, 'w', encoding='utf-8').write(descendant.window_text())
|
||||
raise SystemExit(0)
|
||||
raise SystemExit(1)
|
||||
"@
|
||||
$readbackPath = Join-Path $env:TEMP "openless-terminal-readback.py"
|
||||
$outputPath = Join-Path $env:TEMP "openless-terminal-readback.txt"
|
||||
Write-TextUtf8 $readbackPath $readbackScript
|
||||
try {
|
||||
Remove-Item -LiteralPath $outputPath -Force -ErrorAction SilentlyContinue
|
||||
python -X utf8 $readbackPath $TargetInfo.TargetHandle $outputPath | Out-Null
|
||||
if (Test-Path $outputPath) {
|
||||
return Get-Content -Raw -Encoding UTF8 $outputPath
|
||||
}
|
||||
return $null
|
||||
} finally {
|
||||
Remove-Item -LiteralPath $readbackPath -Force -ErrorAction SilentlyContinue
|
||||
Remove-Item -LiteralPath $outputPath -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
}
|
||||
|
||||
return $null
|
||||
}
|
||||
|
||||
function Send-CtrlChord($Vk) {
|
||||
@@ -338,6 +539,14 @@ $historyPath = Join-Path $env:APPDATA "OpenLess\history.json"
|
||||
$preferencesPath = Join-Path $env:APPDATA "OpenLess\preferences.json"
|
||||
$baselineCount = Get-HistoryCount $historyPath
|
||||
$previousPreferences = Set-HoldHotkeyPreference $preferencesPath
|
||||
$previousClipboard = Get-Clipboard -Raw -ErrorAction SilentlyContinue
|
||||
$clipboardSentinel = "OPENLESS_OLD_CLIPBOARD_SENTINEL_$(Get-Date -Format 'yyyyMMddHHmmssfff')"
|
||||
Restore-ClipboardValue $clipboardSentinel
|
||||
$debugTranscriptPath = $null
|
||||
if (-not [string]::IsNullOrWhiteSpace($InjectedTranscriptText)) {
|
||||
$debugTranscriptPath = Join-Path $env:TEMP "openless-debug-transcript.txt"
|
||||
Write-TextUtf8 $debugTranscriptPath $InjectedTranscriptText
|
||||
}
|
||||
|
||||
Get-Process openless -ErrorAction SilentlyContinue | Stop-Process -Force
|
||||
Remove-Item -LiteralPath $logPath -Force -ErrorAction SilentlyContinue
|
||||
@@ -348,12 +557,16 @@ $env:OPENLESS_ACCEPT_SYNTHETIC_HOTKEY_EVENTS = "1"
|
||||
if ($DebugHotkeyEvents) {
|
||||
$env:OPENLESS_DEBUG_HOTKEY_EVENTS = "1"
|
||||
}
|
||||
if ($debugTranscriptPath) {
|
||||
$env:OPENLESS_DEBUG_TRANSCRIPT_FILE = $debugTranscriptPath
|
||||
}
|
||||
try {
|
||||
$openless = Start-Process -FilePath $ExePath -WorkingDirectory (Split-Path $ExePath -Parent) -PassThru
|
||||
} finally {
|
||||
Remove-Item Env:OPENLESS_SHOW_MAIN_ON_START -ErrorAction SilentlyContinue
|
||||
Remove-Item Env:OPENLESS_ACCEPT_SYNTHETIC_HOTKEY_EVENTS -ErrorAction SilentlyContinue
|
||||
Remove-Item Env:OPENLESS_DEBUG_HOTKEY_EVENTS -ErrorAction SilentlyContinue
|
||||
Remove-Item Env:OPENLESS_DEBUG_TRANSCRIPT_FILE -ErrorAction SilentlyContinue
|
||||
}
|
||||
|
||||
$inputTarget = $null
|
||||
@@ -364,9 +577,18 @@ try {
|
||||
|
||||
$inputTarget = Start-InputTarget $Target
|
||||
|
||||
Press-Hotkey
|
||||
if (-not (Wait-LogPattern $logPath "\[hotkey\] Windows trigger pressed" 10)) {
|
||||
throw "Windows low-level hook did not observe the right Control press."
|
||||
$observedPress = $false
|
||||
for ($attempt = 1; $attempt -le 3 -and -not $observedPress; $attempt++) {
|
||||
Ensure-TargetFocused $inputTarget | Out-Null
|
||||
Press-Hotkey
|
||||
$observedPress = Wait-LogPattern $logPath "\[hotkey\] Windows trigger pressed" 4
|
||||
if (-not $observedPress) {
|
||||
Release-Hotkey
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
}
|
||||
if (-not $observedPress) {
|
||||
throw "Windows low-level hook did not observe the synthetic Control press."
|
||||
}
|
||||
if (-not (Wait-LogPattern $logPath "\[coord\] session started" 30)) {
|
||||
throw "OpenLess recording session did not start."
|
||||
@@ -399,18 +621,15 @@ try {
|
||||
throw "Expected Windows insertStatus pasteSent after guarded foreground restore, got '$($latest.insertStatus)'."
|
||||
}
|
||||
|
||||
Focus-Window $inputTarget.Process | Out-Null
|
||||
Start-Sleep -Milliseconds 400
|
||||
Send-CtrlChord 0x41
|
||||
Start-Sleep -Milliseconds 200
|
||||
Send-CtrlChord 0x43
|
||||
Start-Sleep -Milliseconds 400
|
||||
$targetText = Get-Clipboard -Raw -ErrorAction SilentlyContinue
|
||||
$targetText = Read-TargetContent $inputTarget $Target
|
||||
|
||||
if ([string]::IsNullOrWhiteSpace($targetText)) {
|
||||
throw "$Target clipboard readback is empty after Ctrl+A/C."
|
||||
throw "$Target readback is empty."
|
||||
}
|
||||
if (-not $targetText.Contains($latest.finalText)) {
|
||||
if ($targetText.Contains($clipboardSentinel)) {
|
||||
throw "$Target readback contains the pre-dictation clipboard sentinel instead of latest finalText."
|
||||
}
|
||||
throw "$Target readback does not contain latest finalText; insertion was not proven at the target caret."
|
||||
}
|
||||
|
||||
@@ -422,7 +641,7 @@ try {
|
||||
if ($null -ne $inputTarget) {
|
||||
if ($inputTarget.ProfilePath) {
|
||||
Stop-BrowserProfileProcesses $inputTarget.ProfilePath
|
||||
} else {
|
||||
} elseif ($null -ne $inputTarget.Process) {
|
||||
Stop-Process -Id $inputTarget.Process.Id -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
if ($inputTarget.FixturePath) {
|
||||
@@ -438,6 +657,10 @@ try {
|
||||
} else {
|
||||
Write-TextUtf8 $preferencesPath $previousPreferences
|
||||
}
|
||||
Restore-ClipboardValue $previousClipboard
|
||||
if ($debugTranscriptPath) {
|
||||
Remove-Item -LiteralPath $debugTranscriptPath -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host "Real ASR + insertion fallback smoke ($Target) passed."
|
||||
@@ -0,0 +1,184 @@
|
||||
param(
|
||||
[ValidateSet("notepad", "powershell", "cmd", "wt")]
|
||||
[string]$Target = "powershell",
|
||||
[int]$RestoreDelayMs = 150,
|
||||
[int]$PasteSettleDelayMs = 900
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
Add-Type @"
|
||||
using System;
|
||||
using System.Runtime.InteropServices;
|
||||
|
||||
public static class ClipboardRestoreSmokeWin32 {
|
||||
[DllImport("user32.dll")]
|
||||
public static extern bool ShowWindow(IntPtr hWnd, int nCmdShow);
|
||||
|
||||
[DllImport("user32.dll")]
|
||||
public static extern bool SetForegroundWindow(IntPtr hWnd);
|
||||
|
||||
[DllImport("user32.dll")]
|
||||
public static extern void keybd_event(byte bVk, byte bScan, int dwFlags, UIntPtr dwExtraInfo);
|
||||
|
||||
public const int KEYEVENTF_EXTENDEDKEY = 0x0001;
|
||||
public const int KEYEVENTF_KEYUP = 0x0002;
|
||||
}
|
||||
"@
|
||||
|
||||
function Send-KeyEdge($Vk, $KeyUp, $Extended = $false) {
|
||||
$flags = 0
|
||||
if ($Extended) {
|
||||
$flags = $flags -bor [ClipboardRestoreSmokeWin32]::KEYEVENTF_EXTENDEDKEY
|
||||
}
|
||||
if ($KeyUp) {
|
||||
$flags = $flags -bor [ClipboardRestoreSmokeWin32]::KEYEVENTF_KEYUP
|
||||
}
|
||||
$scanCode = if ($Vk -eq 0xA2 -or $Vk -eq 0xA3) { 0x1D } else { 0 }
|
||||
[ClipboardRestoreSmokeWin32]::keybd_event([byte]$Vk, [byte]$scanCode, $flags, [UIntPtr]::Zero)
|
||||
}
|
||||
|
||||
function Send-CtrlChord($Vk) {
|
||||
Send-KeyEdge 0xA2 $false $false
|
||||
Start-Sleep -Milliseconds 70
|
||||
Send-KeyEdge $Vk $false $false
|
||||
Start-Sleep -Milliseconds 70
|
||||
Send-KeyEdge $Vk $true $false
|
||||
Start-Sleep -Milliseconds 70
|
||||
Send-KeyEdge 0xA2 $true $false
|
||||
}
|
||||
|
||||
function Send-EnterKey {
|
||||
Send-KeyEdge 0x0D $false $false
|
||||
Start-Sleep -Milliseconds 70
|
||||
Send-KeyEdge 0x0D $true $false
|
||||
}
|
||||
|
||||
function Wait-ProcessWindow($ProcessName, $After, $TimeoutSeconds) {
|
||||
$deadline = (Get-Date).AddSeconds($TimeoutSeconds)
|
||||
while ((Get-Date) -lt $deadline) {
|
||||
$candidate = Get-Process $ProcessName -ErrorAction SilentlyContinue |
|
||||
Where-Object { $_.StartTime -ge $After -and $_.MainWindowHandle -ne 0 } |
|
||||
Sort-Object StartTime -Descending |
|
||||
Select-Object -First 1
|
||||
if ($null -ne $candidate) {
|
||||
return $candidate
|
||||
}
|
||||
Start-Sleep -Milliseconds 300
|
||||
}
|
||||
return $null
|
||||
}
|
||||
|
||||
function Focus-Window($Process) {
|
||||
if ($null -eq $Process -or $Process.MainWindowHandle -eq 0) {
|
||||
throw "Target window is unavailable."
|
||||
}
|
||||
[ClipboardRestoreSmokeWin32]::ShowWindow($Process.MainWindowHandle, 9) | Out-Null
|
||||
[ClipboardRestoreSmokeWin32]::SetForegroundWindow($Process.MainWindowHandle) | Out-Null
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
|
||||
function Start-TargetWindow($TargetName) {
|
||||
$startedAt = Get-Date
|
||||
switch ($TargetName) {
|
||||
"notepad" {
|
||||
Start-Process notepad.exe | Out-Null
|
||||
return Wait-ProcessWindow "notepad" $startedAt 15
|
||||
}
|
||||
"powershell" {
|
||||
Start-Process powershell.exe -ArgumentList "-NoLogo" | Out-Null
|
||||
return Wait-ProcessWindow "powershell" $startedAt 15
|
||||
}
|
||||
"cmd" {
|
||||
Start-Process cmd.exe | Out-Null
|
||||
return Wait-ProcessWindow "cmd" $startedAt 15
|
||||
}
|
||||
"wt" {
|
||||
$wt = Get-Command wt.exe -ErrorAction SilentlyContinue
|
||||
if ($null -eq $wt) {
|
||||
throw "wt.exe was not found."
|
||||
}
|
||||
Start-Process $wt.Source -ArgumentList "new-tab", "powershell.exe", "-NoLogo" | Out-Null
|
||||
return Wait-ProcessWindow "WindowsTerminal" $startedAt 20
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function Wait-FileText($Path, $Expected, $TimeoutSeconds) {
|
||||
$deadline = (Get-Date).AddSeconds($TimeoutSeconds)
|
||||
while ((Get-Date) -lt $deadline) {
|
||||
if (Test-Path $Path) {
|
||||
$content = Get-Content -Raw -Encoding UTF8 $Path
|
||||
if ($content.Contains($Expected)) {
|
||||
return $true
|
||||
}
|
||||
}
|
||||
Start-Sleep -Milliseconds 200
|
||||
}
|
||||
return $false
|
||||
}
|
||||
|
||||
function Restore-ClipboardValue($Value) {
|
||||
if ($null -eq $Value) {
|
||||
cmd /c "echo off | clip" | Out-Null
|
||||
return
|
||||
}
|
||||
Set-Clipboard -Value $Value
|
||||
}
|
||||
|
||||
$marker = "OPENLESS_CLIPBOARD_RESTORE_OK"
|
||||
$outputPath = Join-Path $env:TEMP "openless-clipboard-restore-$Target.txt"
|
||||
Remove-Item -LiteralPath $outputPath -Force -ErrorAction SilentlyContinue
|
||||
$previousClipboard = Get-Clipboard -Raw -ErrorAction SilentlyContinue
|
||||
|
||||
switch ($Target) {
|
||||
"notepad" {
|
||||
$payload = $marker
|
||||
}
|
||||
"cmd" {
|
||||
$payload = "echo $marker > `"$outputPath`""
|
||||
}
|
||||
default {
|
||||
$payload = "Set-Content -Path `"$outputPath`" -Value `"$marker`""
|
||||
}
|
||||
}
|
||||
|
||||
$targetProcess = $null
|
||||
try {
|
||||
$targetProcess = Start-TargetWindow $Target
|
||||
if ($null -eq $targetProcess) {
|
||||
throw "Failed to start target window: $Target"
|
||||
}
|
||||
Focus-Window $targetProcess
|
||||
Set-Clipboard -Value $payload
|
||||
Start-Sleep -Milliseconds 150
|
||||
|
||||
Send-CtrlChord 0x56
|
||||
Start-Sleep -Milliseconds $RestoreDelayMs
|
||||
Restore-ClipboardValue $previousClipboard
|
||||
|
||||
if ($Target -eq "notepad") {
|
||||
Start-Sleep -Milliseconds $PasteSettleDelayMs
|
||||
Send-CtrlChord 0x41
|
||||
Start-Sleep -Milliseconds 120
|
||||
Send-CtrlChord 0x43
|
||||
Start-Sleep -Milliseconds 200
|
||||
$result = Get-Clipboard -Raw -ErrorAction SilentlyContinue
|
||||
if (-not $result.Contains($marker)) {
|
||||
throw "Notepad readback did not contain the pasted marker."
|
||||
}
|
||||
} else {
|
||||
Start-Sleep -Milliseconds 120
|
||||
Send-EnterKey
|
||||
if (-not (Wait-FileText $outputPath $marker 6)) {
|
||||
throw "Terminal target did not execute the pasted command before clipboard restore."
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host "[ok] target=$Target restoreDelayMs=$RestoreDelayMs"
|
||||
} finally {
|
||||
Restore-ClipboardValue $previousClipboard
|
||||
if ($null -ne $targetProcess) {
|
||||
Stop-Process -Id $targetProcess.Id -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
}
|
||||
@@ -6,12 +6,48 @@ function assertEqual(actual, expected, name) {
|
||||
}
|
||||
}
|
||||
|
||||
const raw = await readFile(new URL('../src-tauri/tauri.conf.json', import.meta.url), 'utf-8');
|
||||
const config = JSON.parse(raw);
|
||||
const mainWindow = config.app.windows.find((window) => window.label === 'main');
|
||||
|
||||
if (!mainWindow) {
|
||||
throw new Error('main window config missing');
|
||||
function assertMatch(source, pattern, name) {
|
||||
if (!pattern.test(source)) {
|
||||
throw new Error(`${name}: pattern ${pattern} not found`);
|
||||
}
|
||||
}
|
||||
|
||||
assertEqual(mainWindow.decorations, false, 'windows main window should use only custom titlebar');
|
||||
const raw = await readFile(new URL('../src-tauri/tauri.conf.json', import.meta.url), 'utf-8');
|
||||
const config = JSON.parse(raw);
|
||||
const capsuleWindow = config.app.windows.find((window) => window.label === 'capsule');
|
||||
const libRs = await readFile(new URL('../src-tauri/src/lib.rs', import.meta.url), 'utf-8');
|
||||
const coordinatorRs = await readFile(new URL('../src-tauri/src/coordinator.rs', import.meta.url), 'utf-8');
|
||||
|
||||
if (!capsuleWindow) {
|
||||
throw new Error('capsule window config missing');
|
||||
}
|
||||
|
||||
assertEqual(capsuleWindow.width, 220, 'windows capsule config keeps translation-capable width baseline');
|
||||
assertEqual(capsuleWindow.height, 110, 'windows capsule config keeps translation-capable height baseline');
|
||||
assertEqual(capsuleWindow.transparent, true, 'capsule window should keep transparent visuals');
|
||||
assertEqual(capsuleWindow.alwaysOnTop, true, 'capsule window should stay above the focused app while recording');
|
||||
assertMatch(
|
||||
libRs,
|
||||
/#\[cfg\(target_os = "windows"\)\][\s\S]*?\(196\.0, height\)/,
|
||||
'windows runtime capsule width should collapse to the visible pill',
|
||||
);
|
||||
assertMatch(
|
||||
libRs,
|
||||
/let height = if translation_active \{ 110\.0 \} else \{ 52\.0 \};/,
|
||||
'windows runtime capsule height should shrink outside translation mode',
|
||||
);
|
||||
assertMatch(
|
||||
libRs,
|
||||
/window\.set_size\(LogicalSize::new\(cap_w, cap_h\)\)\?/,
|
||||
'capsule positioning should resync runtime size with the computed layout',
|
||||
);
|
||||
assertMatch(
|
||||
coordinatorRs,
|
||||
/let accepts_cursor_events = matches!\(state, CapsuleState::Recording\);/,
|
||||
'windows capsule should only accept clicks while actively recording',
|
||||
);
|
||||
assertMatch(
|
||||
coordinatorRs,
|
||||
/window\.set_ignore_cursor_events\(!accepts_cursor_events\)/,
|
||||
'windows capsule should pass clicks through in non-recording states',
|
||||
);
|
||||
@@ -516,9 +516,7 @@ fn qa_hotkey_supervisor_loop(inner: Arc<Inner>) {
|
||||
Err(e) => {
|
||||
attempts += 1;
|
||||
if attempts <= 3 || attempts % 10 == 0 {
|
||||
log::warn!(
|
||||
"[coord] QA hotkey 第 {attempts} 次注册失败: {e}; 3s 后重试"
|
||||
);
|
||||
log::warn!("[coord] QA hotkey 第 {attempts} 次注册失败: {e}; 3s 后重试");
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_secs(3));
|
||||
}
|
||||
@@ -756,6 +754,15 @@ async fn handle_window_hotkey_event(
|
||||
|
||||
#[cfg(target_os = "windows")]
|
||||
{
|
||||
if !window_hotkey_fallback_enabled() {
|
||||
if event_type == "keydown" && !repeat {
|
||||
log::info!(
|
||||
"[window-hotkey] ignored because Windows lifecycle owner is the low-level hook"
|
||||
);
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let trigger = inner.prefs.get().hotkey.trigger;
|
||||
if !window_key_matches_trigger(trigger, &key, &code) {
|
||||
return Ok(());
|
||||
@@ -781,6 +788,10 @@ async fn handle_window_hotkey_event(
|
||||
}
|
||||
}
|
||||
|
||||
fn window_hotkey_fallback_enabled() -> bool {
|
||||
crate::types::HotkeyCapability::current().explicit_fallback_available
|
||||
}
|
||||
|
||||
#[cfg(any(target_os = "windows", test))]
|
||||
fn window_key_matches_trigger(trigger: crate::types::HotkeyTrigger, key: &str, code: &str) -> bool {
|
||||
use crate::types::HotkeyTrigger;
|
||||
@@ -820,7 +831,9 @@ async fn begin_session(inner: &Arc<Inner>) -> Result<(), String> {
|
||||
}
|
||||
}
|
||||
// 翻译模式标志重置;hotkey 监听器在 Shift down 时再 set true。
|
||||
inner.translation_modifier_seen.store(false, Ordering::SeqCst);
|
||||
inner
|
||||
.translation_modifier_seen
|
||||
.store(false, Ordering::SeqCst);
|
||||
|
||||
#[cfg(any(debug_assertions, test))]
|
||||
if hotkey_injection_dry_run_enabled() {
|
||||
@@ -1192,6 +1205,19 @@ async fn end_session(inner: &Arc<Inner>) -> Result<(), String> {
|
||||
|
||||
// ASR 返回空转写护栏(来自 PR #66):写一条 emptyTranscript 失败历史 + 错误胶囊,
|
||||
// 与 main 上其它 error 路径保持一致(带 schedule_capsule_idle 让胶囊自动消失)。
|
||||
let mut raw = raw;
|
||||
|
||||
#[cfg(any(debug_assertions, test))]
|
||||
if raw.text.trim().is_empty() {
|
||||
if let Some(debug_text) = debug_transcript_override_text() {
|
||||
log::info!(
|
||||
"[coord] using debug transcript override (chars={})",
|
||||
debug_text.chars().count()
|
||||
);
|
||||
raw.text = debug_text;
|
||||
}
|
||||
}
|
||||
|
||||
if raw.text.trim().is_empty() {
|
||||
let session = DictationSession {
|
||||
id: Uuid::new_v4().to_string(),
|
||||
@@ -1230,8 +1256,8 @@ async fn end_session(inner: &Arc<Inner>) -> Result<(), String> {
|
||||
let working_languages = prefs.working_languages.clone();
|
||||
let front_app = inner.state.lock().front_app.clone();
|
||||
let translation_target = prefs.translation_target_language.trim().to_string();
|
||||
let translation_active = inner.translation_modifier_seen.load(Ordering::SeqCst)
|
||||
&& !translation_target.is_empty();
|
||||
let translation_active =
|
||||
inner.translation_modifier_seen.load(Ordering::SeqCst) && !translation_target.is_empty();
|
||||
let (polished, polish_error) = if translation_active {
|
||||
log::info!(
|
||||
"[coord] translation mode → target=\u{300C}{}\u{300D} working={:?} front_app={:?}",
|
||||
@@ -1411,6 +1437,18 @@ fn hotkey_injection_dry_run_enabled() -> bool {
|
||||
std::env::var_os("OPENLESS_HOTKEY_INJECTION_DRY_RUN").is_some()
|
||||
}
|
||||
|
||||
#[cfg(any(debug_assertions, test))]
|
||||
fn debug_transcript_override_text() -> Option<String> {
|
||||
let path = std::env::var_os("OPENLESS_DEBUG_TRANSCRIPT_FILE")?;
|
||||
let text = std::fs::read_to_string(path).ok()?;
|
||||
let trimmed = text.trim().to_string();
|
||||
if trimmed.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(trimmed)
|
||||
}
|
||||
}
|
||||
|
||||
fn ensure_microphone_permission(inner: &Arc<Inner>) -> Result<(), String> {
|
||||
use crate::permissions::{self, PermissionStatus};
|
||||
|
||||
@@ -1702,11 +1740,7 @@ async fn begin_qa_session(inner: &Arc<Inner>) -> Result<(), String> {
|
||||
*last = Some(now);
|
||||
}
|
||||
if let Some(app) = inner_for_level.app.lock().clone() {
|
||||
let _ = app.emit_to(
|
||||
"qa",
|
||||
"qa:level",
|
||||
serde_json::json!({ "level": level }),
|
||||
);
|
||||
let _ = app.emit_to("qa", "qa:level", serde_json::json!({ "level": level }));
|
||||
}
|
||||
// 同步把电平推给底部胶囊,让 QA 录音也有跟主听写一致的可视反馈。
|
||||
emit_capsule(
|
||||
@@ -1778,11 +1812,7 @@ async fn end_qa_session(inner: &Arc<Inner>) -> Result<(), String> {
|
||||
emit_capsule(inner, CapsuleState::Transcribing, 0.0, 0, None, None);
|
||||
|
||||
if let Some(app) = inner.app.lock().clone() {
|
||||
let _ = app.emit_to(
|
||||
"qa",
|
||||
"qa:state",
|
||||
serde_json::json!({ "kind": "loading" }),
|
||||
);
|
||||
let _ = app.emit_to("qa", "qa:state", serde_json::json!({ "kind": "loading" }));
|
||||
}
|
||||
|
||||
if let Some(rec) = inner.qa_recorder.lock().take() {
|
||||
@@ -1845,10 +1875,14 @@ async fn end_qa_session(inner: &Arc<Inner>) -> Result<(), String> {
|
||||
}
|
||||
};
|
||||
|
||||
inner.qa_state.lock().messages.push(crate::types::QaChatMessage {
|
||||
role: "user".to_string(),
|
||||
content: user_content,
|
||||
});
|
||||
inner
|
||||
.qa_state
|
||||
.lock()
|
||||
.messages
|
||||
.push(crate::types::QaChatMessage {
|
||||
role: "user".to_string(),
|
||||
content: user_content,
|
||||
});
|
||||
|
||||
if let Some(app) = inner.app.lock().clone() {
|
||||
let messages = inner.qa_state.lock().messages.clone();
|
||||
@@ -1914,10 +1948,14 @@ async fn end_qa_session(inner: &Arc<Inner>) -> Result<(), String> {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
inner.qa_state.lock().messages.push(crate::types::QaChatMessage {
|
||||
role: "assistant".to_string(),
|
||||
content: answer.clone(),
|
||||
});
|
||||
inner
|
||||
.qa_state
|
||||
.lock()
|
||||
.messages
|
||||
.push(crate::types::QaChatMessage {
|
||||
role: "assistant".to_string(),
|
||||
content: answer.clone(),
|
||||
});
|
||||
|
||||
if let Some(app) = inner.app.lock().clone() {
|
||||
let messages = inner.qa_state.lock().messages.clone();
|
||||
@@ -2195,6 +2233,14 @@ mod tests {
|
||||
);
|
||||
assert!(coordinator.inner.hotkey_trigger_held.load(Ordering::SeqCst));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn window_hotkey_fallback_is_disabled_when_no_explicit_fallback_is_advertised() {
|
||||
assert_eq!(
|
||||
window_hotkey_fallback_enabled(),
|
||||
crate::types::HotkeyCapability::current().explicit_fallback_available
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
fn enabled_phrases(inner: &Arc<Inner>) -> Vec<String> {
|
||||
@@ -2433,6 +2479,42 @@ fn show_capsule_window_no_activate() -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
#[cfg(target_os = "windows")]
|
||||
fn hide_capsule_window_if_present() {
|
||||
use std::iter::once;
|
||||
use windows::core::PCWSTR;
|
||||
use windows::Win32::Foundation::HWND;
|
||||
use windows::Win32::UI::WindowsAndMessaging::{
|
||||
FindWindowW, SetWindowPos, ShowWindow, HWND_NOTOPMOST, SWP_HIDEWINDOW, SWP_NOACTIVATE,
|
||||
SWP_NOMOVE, SWP_NOSIZE, SW_HIDE,
|
||||
};
|
||||
|
||||
let title: Vec<u16> = "OpenLess Capsule".encode_utf16().chain(once(0)).collect();
|
||||
let hwnd = match unsafe { FindWindowW(PCWSTR::null(), PCWSTR(title.as_ptr())) } {
|
||||
Ok(hwnd) => hwnd,
|
||||
Err(_) => return,
|
||||
};
|
||||
if hwnd == HWND::default() || hwnd.0.is_null() {
|
||||
return;
|
||||
}
|
||||
|
||||
let _ = unsafe { ShowWindow(hwnd, SW_HIDE) };
|
||||
let _ = unsafe {
|
||||
SetWindowPos(
|
||||
hwnd,
|
||||
HWND_NOTOPMOST,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
SWP_NOMOVE | SWP_NOSIZE | SWP_NOACTIVATE | SWP_HIDEWINDOW,
|
||||
)
|
||||
};
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "windows"))]
|
||||
fn hide_capsule_window_if_present() {}
|
||||
|
||||
fn emit_capsule(
|
||||
inner: &Arc<Inner>,
|
||||
state: CapsuleState,
|
||||
@@ -2454,7 +2536,10 @@ fn emit_capsule(
|
||||
|
||||
let show_capsule = inner.prefs.get().show_capsule;
|
||||
if let Some(window) = app.get_webview_window("capsule") {
|
||||
let visible = !matches!(state, CapsuleState::Idle);
|
||||
let visible = matches!(
|
||||
state,
|
||||
CapsuleState::Recording | CapsuleState::Transcribing | CapsuleState::Polishing
|
||||
);
|
||||
maybe_position_capsule_bottom_center(inner, &window, payload.translation);
|
||||
if show_capsule && visible {
|
||||
if cfg!(target_os = "windows") {
|
||||
@@ -2469,6 +2554,7 @@ fn emit_capsule(
|
||||
#[cfg(target_os = "macos")]
|
||||
crate::restore_main_window_key_if_active(&app);
|
||||
} else {
|
||||
hide_capsule_window_if_present();
|
||||
let _ = window.hide();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -14,7 +14,10 @@ use std::time::Duration;
|
||||
|
||||
use crate::types::InsertStatus;
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[cfg(target_os = "windows")]
|
||||
const CLIPBOARD_RESTORE_DELAY: Duration = Duration::from_millis(750);
|
||||
|
||||
#[cfg(all(not(target_os = "macos"), not(target_os = "windows")))]
|
||||
const CLIPBOARD_RESTORE_DELAY: Duration = Duration::from_millis(150);
|
||||
|
||||
pub struct TextInserter;
|
||||
@@ -131,19 +134,24 @@ fn insert_with_clipboard_restore(text: &str, restore_clipboard_after_paste: bool
|
||||
}
|
||||
|
||||
if restore_clipboard_after_paste {
|
||||
maybe_restore_clipboard(restore_plan);
|
||||
schedule_clipboard_restore(restore_plan);
|
||||
}
|
||||
// 关掉 → 听写文本留在剪贴板里,simulate_paste 没真正落地时用户能手动 Ctrl+V 找回。
|
||||
insertion_success_status()
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
fn maybe_restore_clipboard(plan: ClipboardRestorePlan) {
|
||||
fn schedule_clipboard_restore(plan: ClipboardRestorePlan) {
|
||||
std::thread::spawn(move || restore_clipboard_after_delay(plan, CLIPBOARD_RESTORE_DELAY));
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
fn restore_clipboard_after_delay(plan: ClipboardRestorePlan, delay: Duration) {
|
||||
if plan.previous_text.is_none() {
|
||||
return;
|
||||
}
|
||||
|
||||
std::thread::sleep(CLIPBOARD_RESTORE_DELAY);
|
||||
std::thread::sleep(delay);
|
||||
|
||||
let mut clipboard = match arboard::Clipboard::new() {
|
||||
Ok(clipboard) => clipboard,
|
||||
@@ -303,6 +311,9 @@ mod macos {
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::thread;
|
||||
use std::time::Duration;
|
||||
|
||||
#[test]
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
@@ -317,4 +328,34 @@ mod tests {
|
||||
));
|
||||
assert!(!should_restore_clipboard(None, "dictated text"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[cfg(target_os = "windows")]
|
||||
fn delayed_terminal_paste_must_see_dictated_text_before_clipboard_restore() {
|
||||
let inserted_text = "dictated text".to_string();
|
||||
let previous_text = "older clipboard".to_string();
|
||||
let clipboard = Arc::new(Mutex::new(inserted_text.clone()));
|
||||
let pasted = Arc::new(Mutex::new(None::<String>));
|
||||
|
||||
let clipboard_for_paste = Arc::clone(&clipboard);
|
||||
let pasted_for_paste = Arc::clone(&pasted);
|
||||
let reader = thread::spawn(move || {
|
||||
thread::sleep(Duration::from_millis(250));
|
||||
let seen = clipboard_for_paste.lock().unwrap().clone();
|
||||
*pasted_for_paste.lock().unwrap() = Some(seen);
|
||||
});
|
||||
|
||||
thread::sleep(CLIPBOARD_RESTORE_DELAY);
|
||||
let current_text = Some(clipboard.lock().unwrap().clone());
|
||||
if should_restore_clipboard(current_text.as_deref(), &inserted_text) {
|
||||
*clipboard.lock().unwrap() = previous_text;
|
||||
}
|
||||
|
||||
reader.join().unwrap();
|
||||
|
||||
assert_eq!(
|
||||
pasted.lock().unwrap().as_deref(),
|
||||
Some(inserted_text.as_str())
|
||||
);
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user