adustment
This commit is contained in:
1 parent
61756922e2
commit
5ad2da32df
210 files changed
+927
-3452
No files matched your search
@@ -25,7 +25,7 @@
|
||||
|
||||
- `doc/test-plan/ai-test.md`:涉及 AI 测试范围、正式或 live case 设计、执行摘要或 suite 覆盖维护时阅读。
|
||||
- `doc/test-plan/settings-test.md`:涉及 Settings 测试范围或测试边界时阅读。
|
||||
- `doc/test.md`:涉及测试体系、runner、renderer 辅助测试命令或正式测试流程时阅读。
|
||||
- `doc/test.md`:涉及测试体系、runner 或正式测试流程时阅读。
|
||||
|
||||
### 1.5 prd产品文档相关
|
||||
|
||||
@@ -39,7 +39,7 @@
|
||||
- 禁止使用 `Modal.confirm`、`Modal.info` 等静态方法
|
||||
|
||||
### 代码实现准则
|
||||
|
||||
0. 允许重构且不兼容旧逻辑
|
||||
1. 存在重复逻辑、重复类型,且适合抽象后提高阅读性、简单性、后期维护性的,要进行抽象封装,视情况看是否拆分文件
|
||||
2. 代码逻辑实现过于复杂,有更简单的实现,且有助提高阅读性、简单性、后期维护性的,要进行简单实现
|
||||
3. 文件拆得太碎,且拆后的小文件并不会继续在该文件扩展,那这些文件需要进行整合,有助提高阅读性、简单性、后期维护性
|
||||
|
||||
@@ -76,10 +76,10 @@ tests/ # 自动化测试
|
||||
- `tests/harness/**` 只放可复用的测试基础设施,不承载旧 suite 治理逻辑或旧分析文档契约。
|
||||
- `tests/fixtures/**` 只放长期复用的输入样本,不混入运行产物。
|
||||
- `tests/fixtures/workspace/` 是共享工作区输入目录;工作区语义测试统一从这里取样本。
|
||||
- `tests/ai/**` 只放 AI formal tests。
|
||||
- `tests/unit/**` 放长期保留、但不进入 formal test-plan 的 unit / helper / schema / prompt 类测试。
|
||||
- `tests/settings/**` 按 settings PRD 模块组织正式测试。
|
||||
- `tests/kb-search/**` 预留给检索链路测试;`tests/temp/**` 仅放按需临时排查用例。
|
||||
- `tests/case/ai/**` 只放 AI formal tests。
|
||||
- `tests/case/unit/**` 放长期保留、但不进入 formal test-plan 的 unit / helper / schema / prompt 类测试。
|
||||
- `tests/case/settings/**` 按 settings PRD 模块组织正式测试。
|
||||
- `tests/case/kb-search/**` 预留给检索链路测试。
|
||||
- `tests/logs/**` 只放每次运行的原始日志;raw logs 之外不再维护测试专用正式分析目录。
|
||||
- 测试目录统一使用 kebab-case,运行产物不得反向成为测试发现入口。
|
||||
|
||||
@@ -93,4 +93,4 @@ tests/ # 自动化测试
|
||||
- 用户可见功能行为变更时,应优先同步 `doc/prd/**`,再同步代码实现与测试。
|
||||
- `doc/test-plan/` 用于维护各类测试计划文档。
|
||||
- 测试计划按测试域与产品面拆分,例如 `doc/test-plan/ai-test.md`、`doc/test-plan/settings-test.md`。
|
||||
- 长期维护的正式测试用例,必须先在 `doc/test-plan/**` 中定义,再落到 `tests/**` 正式目录。
|
||||
- 长期维护的正式测试用例,必须先在 `doc/test-plan/**` 中定义,再落到 `tests/case/**` 正式目录。
|
||||
+135
-88
@@ -3,7 +3,7 @@
|
||||
## 1. 总体目标
|
||||
|
||||
- 本计划用于定义 AI 域正式测试的范围与领域特有规则。
|
||||
- 正式 / temp 准入、runner、raw logs 与正式 case 共享字段,统一遵循 `doc/test.md`。
|
||||
- 正式 / 临时准入、runner、raw logs 与正式 case 共享字段,统一遵循 `doc/test.md`。
|
||||
- AI 正式测试只围绕两类稳定对象治理:
|
||||
1. `tool`
|
||||
2. `human`
|
||||
@@ -13,29 +13,29 @@
|
||||
### 2.1 按 Suite / 分类汇总
|
||||
|
||||
- `tool / docx`
|
||||
- 通过率:`43 / 43`
|
||||
- 检测时间:`2026-04-29 16:39:22 +08:00`
|
||||
- 通过率:`42 / 42`
|
||||
- 检测时间:`2026-06-11 17:16:38 +08:00`
|
||||
- `tool / kb`
|
||||
- 通过率:`2 / 2`
|
||||
- 检测时间:`2026-04-16 16:16:05 +08:00`
|
||||
- 检测时间:`2026-06-11 17:16:38 +08:00`
|
||||
- `tool / terminal`
|
||||
- 通过率:`6 / 6`
|
||||
- 检测时间:`2026-06-06`
|
||||
- 通过率:`5 / 5`
|
||||
- 检测时间:`2026-06-11 17:16:38 +08:00`
|
||||
- 覆盖补充:terminal 正式测试按“安全指令集”语义维护;Windows 路径覆盖 PowerShell 语法、`EncodedCommand` 执行与 UTF-8 输出,macOS 路径覆盖 zsh 语法。
|
||||
- `tool / web-browsing`
|
||||
- 通过率:`5 / 5`
|
||||
- 检测时间:`2026-06-11`
|
||||
- 检测时间:`2026-06-11 17:16:38 +08:00`
|
||||
- 覆盖补充:`web_crawl` 已通过冻结 sidecar bin 和内置 headless shell 覆盖当前页抓取、导航页 topic 跟进、HTTP 失败与结果过大;`browser_use` 已覆盖缺少内置浏览器服务时的真实 diagnostic。
|
||||
- `tool / utility`
|
||||
- 通过率:`未执行`
|
||||
- 检测时间:`未执行`
|
||||
- 覆盖计划:`wait_time` 等待时长与返回 `waitedMs`。
|
||||
- 通过率:`1 / 1`
|
||||
- 检测时间:`2026-06-11 17:16:38 +08:00`
|
||||
- 覆盖补充:`wait_time` 等待时长与返回 `waitedMs`。
|
||||
- `tool / hand`
|
||||
- 通过率:`4 / 4`
|
||||
- 检测时间:`2026-04-16 16:16:05 +08:00`
|
||||
- 通过率:`9 / 9`
|
||||
- 检测时间:`2026-06-11 17:16:38 +08:00`
|
||||
- `human / mock-llm`
|
||||
- 通过率:`15 / 15`
|
||||
- 检测时间:`2026-04-29 17:02:09 +08:00`
|
||||
- 通过率:`22 / 22`
|
||||
- 检测时间:`2026-06-11 17:16:38 +08:00`
|
||||
- `human / live`
|
||||
- 通过率:`2 / 2`(2026-06-11 定向执行 `web_crawl` live case;2026-06-01 全量 `--runs 2` raw run 样本为 `10 / 31`,suite 最终失败)
|
||||
- 检测时间:`2026-06-11`
|
||||
@@ -95,7 +95,7 @@
|
||||
- 凡是会进入 `human` 的正式 run,必须在进入 `human` 前完成 embedding 基础服务 preflight;不允许把 embedding 初始化推迟到某个工具调用时才暴露。
|
||||
- KB fixture 必须走真实导入、真实索引、真实检索。
|
||||
- embedding 模型缺失时,harness 应优先通过真实生产下载链自动检查并下载,不得引入测试专用 embedding 替身。
|
||||
- 旧式 fake API 一律不得进入 formal harness;一次性排查如果确实需要 fake,只允许写入 `tests/temp/**`。
|
||||
- 旧式 fake API 一律不得进入 formal harness;一次性排查不得进入 `tests/case/**` 正式测试树。
|
||||
- backend/live/e2e 正式测试统一使用默认 `testTimeout=900_000`、`hookTimeout=900_000`;不再为单个正式 case 单独配置超时。
|
||||
|
||||
### 4.2 `mock-llm` 与 `live` 设计口径
|
||||
@@ -113,20 +113,19 @@
|
||||
- case 设计必须使用真实用户风格输入,不得把内部固定工具顺序写进用户输入。
|
||||
- 断言重点:完成态、关键工具调用、关键 artifact、关键事实是否成立。
|
||||
- 不对最终文案做精确文本断言。
|
||||
- transport / structured output 兼容性探针类 case,应至少分别覆盖一次 `force_non_stream` 与 `force_stream` 路径。
|
||||
- 每次 formal `human / live` 执行完成后,除更新通过率、检测时间、备注外,还要在执行摘要中补充本轮已执行样例的完成度评分及简短依据。
|
||||
|
||||
### 4.3 目录与命令
|
||||
|
||||
正式 AI 测试目录:
|
||||
|
||||
- `tests/ai/tool/<group>/<case>.test.ts`
|
||||
- `tests/ai/human/mock-llm/<case>.test.ts`
|
||||
- `tests/ai/human/live/<case>.test.ts`
|
||||
- `tests/case/ai/tool/<group>/<case>.test.ts`
|
||||
- `tests/case/ai/human/mock-llm/<case>.test.ts`
|
||||
- `tests/case/ai/human/live/<case>.test.ts`
|
||||
|
||||
长期非 formal 测试目录:
|
||||
|
||||
- `tests/unit/ai/<case>.test.ts`
|
||||
- `tests/case/unit/ai/<case>.test.ts`
|
||||
|
||||
推荐命令:
|
||||
|
||||
@@ -142,7 +141,6 @@
|
||||
|
||||
### 5.1 `tool`
|
||||
|
||||
|
||||
#### docx 类别
|
||||
|
||||
##### `docx-minimal-edit-flow`
|
||||
@@ -365,14 +363,64 @@
|
||||
- Oracle:各类 action 都能写入成功并返回最终文档状态下可继续消费的 `postWriteReadback`。
|
||||
- Not:不验证上层 op 选择策略。
|
||||
|
||||
##### `docx-edit-serial-actions-stability`
|
||||
##### `docx-edit-insert-blocks-order`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证单次 `docx_edit(actions[])` 内部按 action 顺序执行,并通过稳定 locator 避免前序写入造成后序 blockId 漂移。
|
||||
- Why:这是 docx 批量精细修改的核心写入契约,避免 AI 以读到的真实 blockId 连续编辑时被 blockIndex 改变误伤。
|
||||
- How:覆盖“先插入再替换原目标”、“replace 后继续引用同一原始 blockId 命中 successor”、“删除后再修改同一目标失败且不提交部分结果”。
|
||||
- Oracle:前两类成功并命中预期最终文本,失败类返回错误且 active 文档保持原状态。
|
||||
- What:验证 `docx_edit(insert_blocks)` 在同一 anchor 后插入多条段落与表格时保持输入顺序。
|
||||
- Why:结构插入顺序一旦漂移,后续 AI 读回确认和用户可见文档都会出现错位。
|
||||
- How:在段落 A 后一次插入 `P1 / P2 / Table / P3`,再读取完整 outline。
|
||||
- Oracle:最终顺序稳定为 `A / P1 / P2 / Table / P3 / B`,且返回 `createdBlocks` 与 `postWriteReadback`。
|
||||
- Not:不验证多 action 串行插入;该场景由 `docx-edit-serial-multiple-inserts-preserve-created-blocks` 覆盖。
|
||||
|
||||
##### `docx-edit-serial-multiple-inserts-preserve-created-blocks`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证一次 `docx_edit(actions[])` 内多个 `insert_blocks` 会按 action 顺序稳定写入,并分别返回新增 block。
|
||||
- Why:批量结构插入需要让后续读回和继续编辑能准确识别每个新增结构。
|
||||
- How:在同一文档里先在 A 后插入两个段落,再在 C 前插入段落和表格。
|
||||
- Oracle:最终顺序稳定,`createdBlocks` 能对应到本次新增的段落与表格,不误收尾部旧块。
|
||||
- Not:不验证 replace/delete 与 insert 的混合漂移;该场景由串行动作稳定性 case 覆盖。
|
||||
|
||||
##### `docx-structural-op-behavior`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证文本替换不会创建新结构,段落/表格对象编辑可以创建独立结构块。
|
||||
- Why:docx 工具必须清晰区分文本级修改与结构级修改,否则会污染后续 outline、read 和 createdBlocks 语义。
|
||||
- How:先执行 `replace_text`,再执行段落与表格 `insert_blocks`,分别读取 outline。
|
||||
- Oracle:`replace_text` 后 body 结构数量不变;对象插入后出现新增 paragraph/table 结构,且 changedBlocks 标记为结构插入。
|
||||
- Not:不验证 run 级样式保留。
|
||||
|
||||
##### `docx-edit-serial-actions-resolve-original-target-after-insert-drift`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证单次 `docx_edit(actions[])` 内前序插入导致 blockIndex 漂移后,后序 action 仍能命中最初读取到的原始 blockId。
|
||||
- Why:AI 常基于同一轮读取结果连续提交批量编辑,必须避免前序插入让后续目标误伤相邻旧块。
|
||||
- How:先在 A 后插入 A1,再继续用最初读取到的 C blockId 执行 `replace_block`。
|
||||
- Oracle:最终文档为 `A / A1 / B / C updated`,changedBlocks 同时记录插入与替换。
|
||||
- Not:不支持同一次 actions 内引用本次刚创建且尚未回读的新 blockId。
|
||||
|
||||
##### `docx-edit-serial-actions-follow-replacement-successor`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证同一次 `docx_edit(actions[])` 内 `replace_block` 后继续引用同一原始 blockId,会解析到替换后的 successor。
|
||||
- Why:批量编辑经常需要先替换整段,再继续对新段落做局部文字处理。
|
||||
- How:先把目标段落替换为含 `keyword` 的新段落,再用同一原始 blockId 执行 `replace_text`。
|
||||
- Oracle:最终文本包含二次修改后的 `styled-keyword`,并同时记录 `replace_block` 与 `replace_text`。
|
||||
- Not:不验证跨多轮引用新增 block 的策略。
|
||||
|
||||
##### `docx-edit-serial-actions-fail-after-delete-without-committing`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证同一次 `docx_edit(actions[])` 内删除后再修改同一原始 blockId 会失败,且不会提交部分结果。
|
||||
- Why:批量编辑失败必须保持 transaction 边界,不能留下半成功 working draft。
|
||||
- How:先删除目标 block,再尝试对同一原始 block 执行 `replace_block`。
|
||||
- Oracle:工具返回失败;active 文档仍保留原目标文本,不出现后续替换文本。
|
||||
- Not:不支持同一次 actions 内引用本次刚创建且尚未回读的新 blockId。
|
||||
|
||||
##### `docx-format-variants-contract`
|
||||
@@ -445,16 +493,6 @@
|
||||
- Oracle:未修改部分样式保留,修改目标样式正确。
|
||||
- Not:不验证 UI。
|
||||
|
||||
##### `docx-replay-open-and-continue-edit`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证同一 run 内 docx 自动 open 会基于 latest replay session 继续编辑。
|
||||
- Why:多轮编辑能否累计,依赖 replay session 的真实行为。
|
||||
- How:先执行一次 docx 写入,再执行 `docx_read_blocks / docx_edit(replace_text)`。
|
||||
- Oracle:读取内容来自最新 replay session,新增修改叠加在当前 revision 上。
|
||||
- Not:不验证 UI 与最终 direct apply。
|
||||
|
||||
##### `docx-working-draft-hides-deleted-content`
|
||||
|
||||
- 测试类型:`backend`
|
||||
@@ -467,16 +505,6 @@
|
||||
|
||||
#### docx hand 类别
|
||||
|
||||
##### `hand-serial-docx-chain`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 hand 在 `serial` 模式下会按顺序执行 `docx` 写入与后续读取。
|
||||
- Why:这是 hand 承接 docx 写工具返回复核信息后继续执行后续读取的最小基线。
|
||||
- How:真实执行 `docx_edit(replace_text) -> docx_read_blocks`。
|
||||
- Oracle:工具结果按顺序成功返回,写工具结果包含 `postWriteReadback.regions`,且不产生旧写后阻断 decision hint。
|
||||
- Not:不验证复杂定位恢复。
|
||||
|
||||
##### `hand-docx-single-file-multi-write-allowed`
|
||||
|
||||
- 测试类型:`backend`
|
||||
@@ -537,36 +565,6 @@
|
||||
- Oracle:新增段落与新增 cell 都可继续被定位和修改,最新 working draft 只暴露最新有效内容。
|
||||
- Not:不验证 live 模型如何规划多轮编辑。
|
||||
|
||||
##### `docx-committed-view-projection`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 direct apply 使用 committed view projection,且 projected docx 使用稳定表格布局。
|
||||
- Why:committed view 格式稳定性是 docx direct apply 的核心工程约束。
|
||||
- How:真实执行结构化编辑并 materialize committed view,再直接检查生成的 docx XML。
|
||||
- Oracle:committed view 不包含原生 `w:ins/w:del`,且插入表格带 `tblGrid / tblLayout / tcW`。
|
||||
- Not:不验证 live 渲染外观。
|
||||
|
||||
##### `docx-direct-apply-uses-committed-view`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 direct apply 时按 committed view 语义落盘,而不是以 working artifact 直接覆盖源文件。
|
||||
- Why:最终落盘必须来自 committed view,不能把仅供 AI 继续编辑的 working draft 直接写回源文件。
|
||||
- How:构造 replay session 后执行 `applyDocxRunEdits`,检查源文件采用 committed view 内容。
|
||||
- Oracle:写回后 source 文件采用 committed view 内容,不包含 working-only 展示痕迹。
|
||||
- Not:不验证冲突处理。
|
||||
|
||||
##### `docx-risk-report-direct-apply`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证风险周报场景能直接把最终内容落盘到源 `docx`,并生成可回退 checkpoint。
|
||||
- Why:通用 direct apply 契约之外,还需要一条贴近真实业务内容的 docx 写回样例。
|
||||
- How:构造与 live 风险周报 case 相同的模板;通过真实 docx 工具执行两处占位符替换,随后执行 `applyDocxRunEdits` 并检查源文档最终内容。
|
||||
- Oracle:源模板中两处占位符被最终内容替换,且不再保留 `RISK_TODO / OWNER_TODO`;生成 `DocxApplyRecord(status=applied)`。
|
||||
- Not:不覆盖 live 模型执行路径,不要求经过 KB 或 terminal。
|
||||
|
||||
##### `docx-history-and-rollback`
|
||||
|
||||
- 测试类型:`backend`
|
||||
@@ -737,15 +735,35 @@
|
||||
|
||||
#### human hand 类别
|
||||
|
||||
##### `hand-tool-call-order-contract`
|
||||
##### `hand-call-limit`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 `hand` 的多工具顺序执行、总调用数约束与结果聚合守卫。
|
||||
- Why:工具执行器本身必须在真实结果下稳定收敛,且不能再暴露并行分支复杂度。
|
||||
- How:通过真实 `hand` 执行多工具顺序调用、达到调用上限的批次,以及超过上限的批次。
|
||||
- Oracle:多工具调用会按输入顺序真实执行并正确聚合;超过 `HUMAN_HAND_CALL_LIMIT` 的批次会在执行前被 `hand_call_limit_exceeded` 拦截。
|
||||
- Not:不验证 `brain` 决策来源。
|
||||
- What:验证 `hand` 会在执行前拦截超过单轮调用上限的 `calls`。
|
||||
- Why:`calls` 数量上限是工具执行失控前的第一道硬边界,不能等到部分工具已执行后才发现超限。
|
||||
- How:直接构造 `HUMAN_HAND_CALL_LIMIT + 1` 个安全 terminal 调用交给真实 `hand` node。
|
||||
- Oracle:返回 `hand_call_limit_exceeded` decision hint,detail 中包含 limit 与 callCount,且不会产生 `lastToolResults`。
|
||||
- Not:不验证 hardActionLimit 的执行后收敛;该场景由 `hand-tool-limit-and-artifact-merge` 覆盖。
|
||||
|
||||
##### `hand-schema-preflight-invalid-input`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 `hand` preflight schema 失败时稳定返回 `invalid_input`,并保留具体 diagnostic。
|
||||
- Why:模型常见错误是把工具参数包错层级,preflight 必须在真实工具执行前给出可恢复的结构化失败。
|
||||
- How:向真实 `hand` node 提交双层嵌套的坏 `docx_edit` 参数。
|
||||
- Oracle:工具结果为失败,diagnostic 指向 `calls[0].args`,并提升为 `hand_call_invalid` decision hint。
|
||||
- Not:不验证 brain 如何根据该 hint 重写计划。
|
||||
|
||||
##### `hand-docx-edit-invalid-blockid-failed`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 `docx_edit` 使用不存在或过期 blockId 时失败,并返回统一 `hand_docx_edit_failed`。
|
||||
- Why:无效 blockId 是 docx 多轮编辑的高频失败入口,必须稳定进入可恢复 hint,而不是静默写错位置。
|
||||
- How:创建真实 docx 后,通过 `hand` 调用 `docx_edit(insert_blocks)`,anchorBlockId 使用不存在的 `b:99`。
|
||||
- Oracle:`docx_edit` 返回失败;pending decision hint 为 `hand_docx_edit_failed`,detail 包含 toolName、sourcePath、action 与具体错误。
|
||||
- Not:不验证 human 最终交付话术。
|
||||
|
||||
##### `hand-terminal-approval-batch`
|
||||
|
||||
@@ -773,15 +791,14 @@
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 `hand` 的工具次数限制与 artifact 合并行为。
|
||||
- Why:这是工具执行层的关键边界。
|
||||
- How:通过真实工具结果触发 limit 与 artifact merge 分支。
|
||||
- Oracle:超限被守卫拦截,artifact 合并结果稳定。
|
||||
- How:先执行会产生 file artifact 的 terminal 工具,再在保留既有 artifact 的情况下执行不产生 artifact 的 docx read,并触发 hardActionLimit 收敛。
|
||||
- Oracle:已有 artifact 不被后续无 artifact 工具覆盖;超过 hardActionLimit 后产生 `hand_tool_limit_exceeded` decision hint、交付说明和当前工具结果。
|
||||
- Not:不验证 `human` 全链路。
|
||||
|
||||
### 5.2 `human`
|
||||
|
||||
#### mock-llm 类别
|
||||
|
||||
|
||||
##### `ai-run-service-human-only`
|
||||
|
||||
- 测试类型:`backend`
|
||||
@@ -796,10 +813,20 @@
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 `AiRunService` 会在进入 `human` 前完成 embedding preflight,且 preflight 失败时会直接失败收口。
|
||||
- What:验证 `AiRunService` 会在进入 `human` 前完成 embedding preflight,并支撑后续 semantic docx 搜索。
|
||||
- Why:embedding 已被定义为 AI run 的基础设施,不应等到 semantic 工具调用时才第一次暴露未就绪问题。
|
||||
- How:分别覆盖 preflight 成功后执行真实 `docx_read_blocks`,以及 preflight 直接失败的分支。
|
||||
- Oracle:成功分支中 `docx_read_blocks` 正常返回结果;失败分支中 run 直接失败且不会进入 `human` stage。
|
||||
- How:通过 `AiRunService.getInstance().run(...)` 启动真实 run,在 preflight 成功后执行 `docx_search(mode=semantic)`。
|
||||
- Oracle:preflight 被调用一次;任务完成;semantic 搜索返回带 score 的真实命中。
|
||||
- Not:不验证 preflight 失败收口;该场景由 `ai-run-service-embedding-preflight-failure` 覆盖。
|
||||
|
||||
##### `ai-run-service-embedding-preflight-failure`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 embedding preflight 失败时,`AiRunService` 会在进入 `human` 前直接失败收口。
|
||||
- Why:基础服务不可用时继续进入 human graph 会制造后续工具层噪音,应该在入口阶段明确失败。
|
||||
- How:注入会抛错的 `ensureEmbeddingReady`,启动一次真实 `AiRunService` run。
|
||||
- Oracle:run 状态为 `failed`,错误包含 preflight 失败原因,事件流中没有任何 `human_step`。
|
||||
- Not:不验证模型下载进度展示。
|
||||
|
||||
##### `learning-hand-mouth-smoke`
|
||||
@@ -832,6 +859,16 @@
|
||||
- Oracle:事件流中真实出现 source 为 `learning` 的 `decision_hint`,并且在它之后仍发生真实 terminal 工具调用,任务最终完成。
|
||||
- Not:不验证 live 模型是否会自然产生这类重复决策。
|
||||
|
||||
##### `brain-output-schema-invalid-retry`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 `brain` 输出 hand plan schema 不合法时会产出 `brain_output_schema_invalid` decision hint,并在纠偏后继续完成任务。
|
||||
- Why:结构化输出偶发坏参数是 LLM 链路的真实风险,graph 必须把它转成可恢复 hint。
|
||||
- How:scripted brain 先返回带非法字段的 `docx_edit` hand plan,再返回完成任务决策。
|
||||
- Oracle:任务最终完成;事件流出现 source 为 `brain` 的 `brain_output_schema_invalid` decision hint,且 detail 中包含 `nextAction.calls` 相关 issue。
|
||||
- Not:不验证具体工具执行,因为坏 plan 应在执行前被拦截。
|
||||
|
||||
##### `decision-hint-contract-and-recovery`
|
||||
|
||||
- 测试类型:`backend`
|
||||
@@ -932,6 +969,16 @@
|
||||
- Oracle:真实 `docx_edit(replace_text)` 只产生一处 `changedBlocks`,真实 `docx_search` 与 `docx_read_blocks` 都能命中新文本,任务最终完成。
|
||||
- Not:不验证 UI 或最终落盘流程。
|
||||
|
||||
##### `complex-docx-insert-structured-content`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 mock-llm 下 `human` 可以按对象类型提交段落与表格编辑参数。
|
||||
- Why:复杂 docx 编辑不能只覆盖文本替换,还要保护 structured content 插入这类高价值写作场景。
|
||||
- How:先读取真实 docx outline,定位 anchor 后通过真实 `docx_edit(insert_blocks)` 插入段落和表格,再读回确认。
|
||||
- Oracle:任务最终完成;docx apply artifact 与 file artifact 正常生成;写入内容包含新增段落和表格结构。
|
||||
- Not:不验证 live 模型是否自然选择同样的结构化参数。
|
||||
|
||||
##### `docx-read-fact-exposure`
|
||||
|
||||
- 测试类型:`backend`
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
- 本计划用于定义 Settings 用户可见能力的正式测试范围。
|
||||
- Settings 正式测试统一对齐 [settings.md](/Users/test/work-space/NCurator-Desktop/doc/prd/settings.md),新增、调整或验收结论都应引用对应的 `SET-*` 需求编号与验收编号。
|
||||
- 姝e紡 / temp 鍑嗗叆銆乺unner銆乺aw logs 涓庢�寮� case 鍏变韩瀛楁�锛岀粺涓€閬靛惊 `doc/test.md`銆�
|
||||
- 正式 / 临时准入、runner、raw logs 与正式 case 共享字段,统一遵循 `doc/test.md`。
|
||||
|
||||
## 2. 执行摘要
|
||||
|
||||
@@ -12,28 +12,29 @@
|
||||
|
||||
- `model`
|
||||
- 登记 case:`8`
|
||||
- 閫氳繃鐜囷細`7 / 7`锛坋2e锛�
|
||||
- 妫€娴嬫椂闂达細`2026-06-02 18:32:06 +08:00`
|
||||
- 澶囨敞锛氭湰杞�畬鏁� Settings e2e 宸茶�鐩� model e2e锛沗model-config-list-contract` 鏈�湪鏈�疆鍗曠嫭鎵ц�銆�
|
||||
- 通过率:`1 / 1`(backend,2026-06-11) / `7 / 7`(e2e,2026-06-02)
|
||||
- 检测时间:`2026-06-11 17:16:38 +08:00`
|
||||
- 备注:本轮已通过 `model-config-list-contract` backend;model e2e 结论沿用 2026-06-02 完整 Settings e2e。
|
||||
- `workspace / core`
|
||||
- 登记 case:`6`
|
||||
- 閫氳繃鐜囷細`6 / 6`
|
||||
- 妫€娴嬫椂闂达細`2026-06-03 13:12:07 +08:00`
|
||||
- 通过率:`1 / 1`(backend,2026-06-11) / `5 / 5`(e2e,2026-06-03)
|
||||
- 检测时间:`2026-06-11 17:16:38 +08:00`
|
||||
- 备注:本轮已通过 `workspace-remove-source-contract` backend;workspace core e2e 结论沿用 2026-06-03。
|
||||
- `workspace / files`
|
||||
- 登记 case:`12`
|
||||
- 通过率:`12 / 12`
|
||||
- 妫€娴嬫椂闂达細`2026-06-04 13:51:28 +08:00`
|
||||
- 澶囨敞锛氭湰杞�柊澧炲苟閫氳繃 `workspace-index-worker-contract`锛涢噸璺戦€氳繃 `workspace-prepare-real-fixture-refresh`銆乣workspace-prepare-failed-retry`銆乣workspace-prepare-file-failure-isolated`銆乣workspace-detail-status-and-error` 鍥涙潯鍏抽敭 E2E锛屽叾浣� 7 鏉℃部鐢ㄤ笂涓€杞�€氳繃缁撹�銆�
|
||||
- 检测时间:`2026-06-11 17:16:38 +08:00`
|
||||
- 备注:本轮已通过 `workspace-prepare-contract`、`workspace-scan-contract`、`workspace-index-job-store-contract`、`workspace-index-worker-contract`、`workspace-interrupted-job-recovery-contract` 五条 backend;其余 E2E 结论沿用上一轮通过结论。
|
||||
- `workspace / web`
|
||||
- 登记 case:`7`
|
||||
- 通过率:`7 / 7`
|
||||
- 检测时间:`2026-06-03 13:12:07 +08:00`
|
||||
- 澶囨敞锛氭湰杞� `workspace` 鑼冨洿 E2E 涓� Settings backend 宸查€氳繃锛涘畬鏁� `settings:e2e` 鍦ㄥ綋鍓嶆満鍣ㄤ笂浠嶅洜 Electron 鍘熺敓鍚�姩 `SIGABRT` 涓�柇锛屾湭浣滀负閫氳繃缁撹�璁板綍銆�
|
||||
- 备注:本轮未重跑 `workspace / web` E2E;完整 `settings:e2e` 未作为本轮通过结论记录。
|
||||
- `vector`
|
||||
- 鐧昏� case锛歚4`
|
||||
- 閫氳繃鐜囷細`3 / 3`锛坋2e锛�
|
||||
- 妫€娴嬫椂闂达細`2026-06-02 18:32:06 +08:00`
|
||||
- 澶囨敞锛氭湰杞�畬鏁� Settings e2e 宸茶�鐩� vector e2e锛沗vector-download-single-task-contract` 鏈�湪鏈�疆鍗曠嫭鎵ц�銆�
|
||||
- 登记 case:`7`
|
||||
- 通过率:`2 / 2`(backend,2026-06-11) / `5 / 5`(e2e,2026-06-06)
|
||||
- 检测时间:`2026-06-11 17:16:38 +08:00`
|
||||
- 备注:本轮已通过 `vector-download-single-task-contract`、`vector-download-startup-resume-contract` backend;vector e2e 结论沿用 2026-06-06。
|
||||
|
||||
## 3. Suite 设计
|
||||
|
||||
@@ -47,7 +48,7 @@ Settings 的正式测试顶层 suite 直接对齐 PRD 模块:
|
||||
|
||||
- `model` 验证大模型连接配置的加载、创建、编辑、启用、删除和远端模型列表刷新。
|
||||
- `workspace` 验证工作空间容器及其下属 `files`、`web` 两类子区域的页面行为和 backend / IPC 支撑契约。
|
||||
- `vector` 灞炰簬 Settings 浜у搧鍩燂紝浣嗛〉闈�綅浜庣嫭绔嬭矾鐢� `/model-download`锛屾祴璇曠洰褰曚笌 runner 浠嶇粺涓€褰掑叆 `tests/settings/vector/**`銆�
|
||||
- `vector` 属于 Settings 产品域,但页面位于独立路由 `/model-download`,测试目录与 runner 仍统一归入 `tests/case/settings/vector/**`。
|
||||
- `workspace` 在这里是 Settings 下的 suite,不是独立测试 domain;其下统一按 `core / files / web` 组织子区域。
|
||||
- `workspace` 当前测试语义基于“具名工作空间容器 + 文件 tab + 网页 tab”,不再保留旧的“创建即绑定目录 / 全局状态 / 三种索引处理模式 / 打开路径 / 网页本地筛选”模型。
|
||||
- `workspace` 相关正式测试默认复用 `tests/fixtures/workspace` 作为共享输入目录。
|
||||
@@ -435,60 +436,32 @@ Settings 的正式测试顶层 suite 直接对齐 PRD 模块:
|
||||
- Oracle:下载完成后,对应模型显示为已安装,对齐 `SET-VECTOR-003-AC1`;下载失败时界面提示失败原因并退出下载中状态,对齐 `SET-VECTOR-003-AC2`。
|
||||
- Not:不验证真实 provider 侧文件列表变化,也不验证跨会话缓存复用。
|
||||
|
||||
#### `vector-download-navigation-resume`
|
||||
|
||||
- 测试类型:`e2e`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证用户离开向量模型页面后,再次进入时仍可恢复后台正在进行的下载进度。
|
||||
- Why:如果下载任务依赖页面生命周期,用户返回后会误以为任务中断或重新开始,影响对长时间下载任务的信心。
|
||||
- How:预置一个后台进行中的下载快照;进入向量模型页观察进度;切到其它 Settings 子页后再返回向量模型页。
|
||||
- Oracle:再次进入页面时,当前文件、完成数和总进度都能恢复显示,对齐 `SET-VECTOR-002-AC3`。
|
||||
- Not:不验证应用重启后的自动恢复;这里只覆盖页面切换下的后台继续下载体验。
|
||||
|
||||
#### `vector-delete-feedback`
|
||||
|
||||
- 测试类型:`e2e`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证已安装模型删除成功后切回可下载状态,删除失败时保留已安装状态并给出错误反馈。
|
||||
- Why:删除结果是用户确认模型已从本地移除的直接依据;若删除失败但界面未收敛错误,用户会误判本地模型状态。
|
||||
- How:分别覆盖已安装模型删除成功和删除失败两条路径,观察卡片操作区和提示反馈。
|
||||
- Oracle:删除成功后,对应模型回到可下载状态,对齐 `SET-VECTOR-004-AC1`;删除失败时提示失败原因并保留已安装状态,对齐 `SET-VECTOR-004-AC2`。
|
||||
- Not:不验证删除后磁盘占用的真实变化;这里只覆盖 Settings 页面可见状态与反馈。
|
||||
|
||||
## 5. 向量套件执行摘要更正(2026-06-06)
|
||||
|
||||
- `vector`
|
||||
- 登记 case:`7`
|
||||
- 通过率:`5 / 5`(e2e)
|
||||
- 检测时间:`2026-06-06 14:21:23 +08:00`
|
||||
- 备注:本轮新增并通过 `vector-download-startup-resume-contract`、`vector-download-navigation-resume`、`vector-delete-feedback`;`vector-download-single-task-contract` backend 也已单独通过。
|
||||
|
||||
#### `vector-download-startup-resume-contract`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证应用启动时若存在进行中的向量模型下载快照,后台服务会自动恢复该下载任务。
|
||||
- Why:如果应用重开后未能恢复下载,用户会误以为长时间下载任务已经丢失,且页面重新进入时无法看到一致的进度状态。
|
||||
- How:预置一个进行中的下载快照;启动向量模型下载服务初始化;观察是否恢复同一模型下载、继续上报进度,并在完成后清理快照。
|
||||
- Oracle:启动初始化后会自动恢复同一模型下载并继续上报进度,对齐 `SET-VECTOR-002-AC4`。
|
||||
- Not:不验证真实网络环境下的跨版本兼容;这里只覆盖本地快照恢复合同。
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证应用启动时遇到进行中的向量模型下载快照,会恢复后台下载并继续上报进度。
|
||||
- Why:如果启动恢复失效,用户会误以为长时间下载已经丢失,重新进入页面时也无法看到一致的进度状态。
|
||||
- How:预置一条进行中的下载快照,初始化向量模型下载服务,观察是否恢复同一模型下载、继续上报进度,并在完成后清理快照。
|
||||
- Oracle:服务初始化后自动恢复同一模型下载并持续上报进度,对齐 `SET-VECTOR-002-AC4`。
|
||||
- Not:不验证真实网络环境下的跨版本兼容;这里只覆盖本地快照恢复与同步。
|
||||
|
||||
#### `vector-download-navigation-resume`
|
||||
|
||||
- 测试类型:`e2e`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证用户离开向量模型页面后,再次进入时仍可恢复后台正在进行的下载进度。
|
||||
- Why:如果下载任务依赖页面生命周期,用户返回后会误以为任务中断或重新开始,影响对长时间下载任务的信心。
|
||||
- How:预置一个后台进行中的下载快照;进入向量模型页观察进度;切到其它 Settings 子页后再返回向量模型页。
|
||||
- Oracle:再次进入页面时,当前文件、完成数和总进度都能恢复显示,对齐 `SET-VECTOR-002-AC3`。
|
||||
- Not:不验证应用重启后的自动恢复;该场景由 `vector-download-startup-resume-contract` 覆盖。
|
||||
- 测试类型:`e2e`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证用户离开向量模型页后再次进入时,仍可恢复后台进行中的下载进度。
|
||||
- Why:下载不能依赖页面生命周期;用户切页后返回不应误判下载中断或重新开始。
|
||||
- How:预置一条后台下载中的任务快照,进入向量模型页观察进度,切到其他 Settings 页面后再返回向量模型页。
|
||||
- Oracle:再次进入页面时,当前文件、完成数和总进度都能恢复展示,对齐 `SET-VECTOR-002-AC3`。
|
||||
- Not:不验证应用重启后的自动恢复;该场景由 `vector-download-startup-resume-contract` 覆盖。
|
||||
|
||||
#### `vector-delete-feedback`
|
||||
|
||||
- 测试类型:`e2e`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证已安装模型删除前需要二次确认;取消删除时保持原状态;确认后成功切回可下载状态,失败时保留已安装状态并给出错误反馈。
|
||||
- Why:删除结果是用户确认模型已从本地移除的直接依据;如果没有确认步骤或失败反馈未收敛,用户会误判本地模型状态。
|
||||
- How:分别覆盖删除确认弹框出现、取消删除、确认删除成功和确认删除失败四条路径,观察卡片操作区和提示反馈。
|
||||
- Oracle:删除成功后,对应模型回到可下载状态,对齐 `SET-VECTOR-004-AC1`;删除失败时提示失败原因并保留已安装状态,对齐 `SET-VECTOR-004-AC2`;点击删除先出现确认弹框,对齐 `SET-VECTOR-004-AC3`;取消确认后保持已安装状态且不进入删除结果态,对齐 `SET-VECTOR-004-AC4`。
|
||||
- Not:不验证删除后磁盘占用的真实变化;这里只覆盖 Settings 页面可见状态与反馈。
|
||||
- 测试类型:`e2e`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证已安装模型删除前需要二次确认,取消删除时保持原状态,确认后成功切回可下载状态,失败时保留已安装状态并展示错误。
|
||||
- Why:删除操作需要给用户明确确认和结果反馈;取消确认或删除失败都不能让模型状态失真。
|
||||
- How:分别覆盖删除确认弹窗出现、取消删除、确认删除成功、确认删除失败四条路径,观察卡片状态和反馈提示。
|
||||
- Oracle:删除成功后,对应模型回到可下载状态,对齐 `SET-VECTOR-004-AC1`;删除失败时展示失败原因并保留已安装状态,对齐 `SET-VECTOR-004-AC2`;删除前出现确认弹窗,对齐 `SET-VECTOR-004-AC3`;取消确认后保持已安装状态且不进入删除中状态,对齐 `SET-VECTOR-004-AC4`。
|
||||
- Not:不验证删除后磁盘占用的真实变化;这里只覆盖 Settings 页面可见状态与反馈。
|
||||
+23
-27
@@ -23,10 +23,10 @@
|
||||
- 必须进入正式测试闭环。
|
||||
- 临时排查 case
|
||||
- 只用于一次性排查、验证猜想、临时复现或补上下文。
|
||||
- 固定放在 `tests/temp/**`。
|
||||
- 不进入 `tests/**`,也不作为长期测试资产提交。
|
||||
- 默认策略
|
||||
- 用户没有明确要求“补正式长期 case”时,默认先写 `tests/temp/**`。
|
||||
- 只要还在排查、验证猜想、试运行 fixture 或尚未确定要长期维护,一律先放 `tests/temp/**`。
|
||||
- 用户没有明确要求“补正式长期 case”时,默认不新增正式测试文件。
|
||||
- 只要还在排查、验证猜想、试运行 fixture 或尚未确定要长期维护,一律不写入正式测试目录。
|
||||
- 只有确认要长期维护、参与 gating 的 case,才进入正式测试目录。
|
||||
|
||||
### 2.2 目录结构
|
||||
@@ -35,25 +35,24 @@
|
||||
tests/
|
||||
harness/
|
||||
fixtures/
|
||||
ai/
|
||||
unit/
|
||||
settings/
|
||||
kb-search/
|
||||
temp/
|
||||
case/
|
||||
ai/
|
||||
unit/
|
||||
settings/
|
||||
kb-search/
|
||||
logs/
|
||||
```
|
||||
|
||||
规则:
|
||||
|
||||
- 目录统一使用 kebab-case。
|
||||
- 测试发现目录使用通用树形结构:`tests/<domain>/<suite>/<groups...>/<case>.test.ts`。
|
||||
- e2e 测试文件使用 `tests/<domain>/<suite>/<groups...>/<case>.e2e.test.ts`。
|
||||
- 测试发现目录使用通用树形结构:`tests/case/<domain>/<suite>/<groups...>/<case>.test.ts`。
|
||||
- e2e 测试文件使用 `tests/case/<domain>/<suite>/<groups...>/<case>.e2e.test.ts`。
|
||||
- `suite` 是第一层稳定分类;`groups...` 是 suite 下可选的任意深度子目录。
|
||||
- 各 domain 的 `suite` 列表与归属,只在对应 `doc/test-plan/**` 中定义。
|
||||
- 当某个 `suite` 不需要继续细分时,正式 case 可以直接放在该 `suite` 目录下。
|
||||
- `tests/unit/**` 用于长期保留、但不进入 `doc/test-plan/**` 正式治理的 unit / helper / schema / prompt 类测试。
|
||||
- `tests/temp/**` 默认不参与 gating,也不沉淀正式长期产物。
|
||||
- 除 `tests/temp/**` 外,`tests/**` 中的正式测试目录只允许放已在 `doc/test-plan/**` 登记的 case。
|
||||
- `tests/case/unit/**` 用于长期保留、但不进入 `doc/test-plan/**` 正式治理的 unit / helper / schema / prompt 类测试。
|
||||
- `tests/case/**` 中除 `unit/**` 外,只允许放已在 `doc/test-plan/**` 登记的正式 case。
|
||||
- `tests/fixtures/workspace/` 是共享工作区 fixture 根目录;凡是依赖“工作区里有什么文件”的测试,优先复用该目录。
|
||||
- 当 case 直接传入 `tests/fixtures/workspace` 目录时,runner 应把该目录下的子内容原样复制到临时 workspace 根目录,而不是保留 `tests/fixtures/**` 中间层路径。
|
||||
|
||||
@@ -79,8 +78,8 @@ tests/
|
||||
|
||||
- `doc/test-plan/**` 下每个领域文档,都是该领域正式测试用例定义的唯一规格来源。
|
||||
- 新增、修改或补齐正式 case 时,必须先更新对应 `doc/test-plan/**`,再修改测试代码。
|
||||
- 未登记的 case 不得直接写进 `tests/**` 非 `temp` 目录。
|
||||
- 当临时 case 需要转为正式长期 case 时,顺序必须是:先补 `doc/test-plan/**`,再迁入正式目录。
|
||||
- 未登记的 case 不得直接写进 `tests/case/**` 非 `unit` 目录。
|
||||
- 当临时验证需要转为正式长期 case 时,顺序必须是:先补 `doc/test-plan/**`,再迁入正式目录。
|
||||
- 如果实现行为变更导致正式 case 预期变化,必须先更新对应 `doc/test-plan/**`,再修改测试代码。
|
||||
- 每个正式 `caseId` 都必须稳定映射到唯一测试文件与唯一日志目录。
|
||||
|
||||
@@ -132,29 +131,28 @@ tests/
|
||||
### 3.4 长期价值与常见误区
|
||||
|
||||
- 进入正式 domain 目录的 case 默认都应具备长期回归价值。
|
||||
- `tests/unit/**` 默认具备长期维护价值,但不属于 formal test-plan case。
|
||||
- 一次性排查、临时验证、待定样例统一放在 `tests/temp/**`。
|
||||
- `tests/case/unit/**` 默认具备长期维护价值,但不属于 formal test-plan case。
|
||||
- 一次性排查、临时验证、待定样例不进入正式测试树。
|
||||
|
||||
常见反例:
|
||||
|
||||
- 为了排查某个猜想,直接在 `tests/ai/**` 或 `tests/settings/**` 新增正式 case。
|
||||
- 为了排查某个猜想,直接在 `tests/case/ai/**` 或 `tests/case/settings/**` 新增正式 case。
|
||||
- 还没补 `doc/test-plan/**`,就先写正式测试代码。
|
||||
- 正式 case 的 `How` 开始直接描述内部 route、内部工具顺序或固定 plan 条数。
|
||||
|
||||
常见正例:
|
||||
|
||||
- 为了确认某个上下文拼接问题是否真实存在,先写 `tests/temp/**` 做一次性验证。
|
||||
- 某条真实用户场景已经确定要长期回归,先补对应 `doc/test-plan/**`,再补正式测试代码。
|
||||
- 某个 helper / schema / prompt 需要长期回归,但不属于 formal 业务 case,放进 `tests/unit/**`。
|
||||
- 某个 helper / schema / prompt 需要长期回归,但不属于 formal 业务 case,放进 `tests/case/unit/**`。
|
||||
|
||||
## 4. 正式测试闭环
|
||||
|
||||
### 4.1 固定顺序
|
||||
|
||||
1. 先判断这是正式长期回归 case,还是临时排查 case。
|
||||
2. 如果只是临时排查,直接写入 `tests/temp/**`,不进入正式闭环。
|
||||
2. 如果只是临时排查,不写入 `tests/**`,也不进入正式闭环。
|
||||
3. 如果要进入正式目录,先更新对应 `doc/test-plan/**`。
|
||||
4. 再写或调整 `tests/**` 中的正式测试代码。
|
||||
4. 再写或调整 `tests/case/**` 中的正式测试代码。
|
||||
5. 通过 `scripts/test-workflow.mjs` 执行并产生日志。
|
||||
6. 先看 `timeline.jsonl`,不足时再看 `runtime-events.jsonl`、`meta.json` 与 `artifacts.json`。
|
||||
7. 最后基于 raw logs 回流结论,决定是否需要更新 `doc/test-plan/**` 或实现代码。
|
||||
@@ -175,7 +173,7 @@ runner 不负责:
|
||||
- 写失败原因分析。
|
||||
- 写优化建议。
|
||||
|
||||
当前常用命令按域收口如下。`workbench` 不再维护 formal test-plan,对应端到端体验验证改走 hand-test,不再提供 `test:workbench*` runner;Workbench renderer UI 的长期辅助契约可放入 `tests/renderer/**`,通过 `pnpm test:renderer` 执行;一次性视觉或交互排查放入 `tests/temp/**`,不得直接新增未登记的 formal workbench case。
|
||||
当前常用命令按域收口如下。`workbench` 不再维护 formal test-plan,对应端到端体验验证改走 hand-test,不再提供 `test:workbench*` 或 renderer 专用 runner;一次性视觉或交互排查不得直接新增未登记的 formal workbench case。
|
||||
|
||||
- AI
|
||||
- `pnpm test:ai`
|
||||
@@ -183,8 +181,6 @@ runner 不负责:
|
||||
- `pnpm test:ai:tool`
|
||||
- `pnpm test:ai:human:mock-llm`
|
||||
- `pnpm test:ai:human:live`
|
||||
- Renderer
|
||||
- `pnpm test:renderer`
|
||||
- Unit
|
||||
- `pnpm test:unit`
|
||||
- `pnpm test:unit:ai`
|
||||
@@ -413,7 +409,7 @@ tests/logs/<domain>/<suite>/<groups...>/<caseId>/<runId>/
|
||||
|
||||
常见误区:
|
||||
|
||||
- 把 `tests/unit/**` 当成 formal domain case 管理。
|
||||
- 把 `tests/temp/**` 的结果直接当成正式长期结论。
|
||||
- 把 `tests/case/unit/**` 当成 formal domain case 管理。
|
||||
- 把一次性排查结果直接当成正式长期结论。
|
||||
- 在 `timeline.jsonl` 中倾倒底层噪音,而不是保留高价值语义事件。
|
||||
- 在没有更新 `doc/test-plan/**` 的情况下直接修改正式 case 预期。
|
||||
@@ -17,7 +17,6 @@
|
||||
"test:ai:tool": "node scripts/test-workflow.mjs --domain ai --suite tool --type backend",
|
||||
"test:ai:human:mock-llm": "node scripts/test-workflow.mjs --domain ai --suite human --group mock-llm --type backend",
|
||||
"test:ai:human:live": "node scripts/test-workflow.mjs --domain ai --suite human --group live --type backend",
|
||||
"test:renderer": "vitest run --config vitest.renderer.config.mts",
|
||||
"test:unit": "node scripts/test-workflow.mjs --domain unit --type backend",
|
||||
"test:unit:ai": "node scripts/test-workflow.mjs --domain unit --suite ai --type backend",
|
||||
"test:settings": "node scripts/test-workflow.mjs --domain settings",
|
||||
|
||||
@@ -4,6 +4,7 @@ import { spawnSync } from 'node:child_process'
|
||||
|
||||
const ROOT = process.cwd()
|
||||
const TESTS_ROOT = path.join(ROOT, 'tests')
|
||||
const TEST_CASES_ROOT = path.join(TESTS_ROOT, 'case')
|
||||
const resolveLocalBin = (name) =>
|
||||
path.join(ROOT, 'node_modules', '.bin', process.platform === 'win32' ? `${name}.cmd` : name)
|
||||
const PNPM_BIN = process.platform === 'win32' ? 'pnpm.cmd' : 'pnpm'
|
||||
@@ -91,7 +92,7 @@ const walk = async (target) => {
|
||||
}
|
||||
|
||||
const resolveScopeDir = (input) => {
|
||||
const parts = [TESTS_ROOT]
|
||||
const parts = [TEST_CASES_ROOT]
|
||||
if (input.domain) parts.push(input.domain)
|
||||
if (input.suite) parts.push(input.suite)
|
||||
if (input.groups.length > 0) parts.push(...input.groups)
|
||||
|
||||
@@ -1,94 +0,0 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHandNode } from '../../../../src/main/services/ai/human/node/handNode'
|
||||
import { createHumanRuntimeCache } from '../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { ensureSkillTools } from '../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createInitialHumanState } from '../../../../src/main/services/ai/human/state/schema'
|
||||
import { writeTestDocxFile } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
|
||||
describe('ai tool hand backend', () => {
|
||||
it('hand-docx-write-produces-post-write-readback', async () => {
|
||||
await runBackendCase({
|
||||
caseId: 'hand-docx-write-post-write-readback',
|
||||
harnessMode: 'tool-real',
|
||||
domain: 'ai',
|
||||
suite: 'tool',
|
||||
groups: ['hand'],
|
||||
description: '验证一轮 hand 内发生 docx 写操作后,会返回 postWriteReadback 结果。',
|
||||
fixtures: [],
|
||||
runMode: 'single',
|
||||
execute: async (runtime) => {
|
||||
await writeTestDocxFile(runtime.workspaceRoot, 'post-write-readback.docx', [
|
||||
{ type: 'paragraph', text: 'Original content.' }
|
||||
])
|
||||
|
||||
const ctx = runtime.createAiRunContext()
|
||||
const cache = createHumanRuntimeCache()
|
||||
|
||||
ensureSkillTools({ cache, ctx, runId: runtime.runId, taskId: runtime.taskId, skillId: 'docx-read' })
|
||||
ensureSkillTools({ cache, ctx, runId: runtime.runId, taskId: runtime.taskId, skillId: 'docx-write' })
|
||||
|
||||
const handNode = createHandNode({ ctx, cache })
|
||||
const state = await handNode({
|
||||
...createInitialHumanState({
|
||||
taskId: runtime.taskId,
|
||||
userInput: '请修改文档'
|
||||
}),
|
||||
skills: {
|
||||
learnedSkillIds: ['docx-read', 'docx-write']
|
||||
},
|
||||
execution: {
|
||||
...createInitialHumanState({
|
||||
taskId: runtime.taskId,
|
||||
userInput: '请修改文档'
|
||||
}).execution,
|
||||
currentThought: {
|
||||
thought: {
|
||||
next: '执行 docx 写入'
|
||||
},
|
||||
nextAction: {
|
||||
action: 'hand',
|
||||
calls: [
|
||||
{
|
||||
toolName: 'docx_edit',
|
||||
args: {
|
||||
sourcePath: 'post-write-readback.docx',
|
||||
actions: [{
|
||||
type: 'replace_text',
|
||||
query: 'Original content.',
|
||||
replace: 'Updated content.'
|
||||
}]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
return {
|
||||
state
|
||||
}
|
||||
},
|
||||
assert: async (result, runtime) => {
|
||||
runtime.assertThat(
|
||||
result.state.execution.pendingDecisionHint === undefined &&
|
||||
result.state.execution.lastToolResults?.[0]?.toolName === 'docx_edit' &&
|
||||
result.state.execution.lastToolResults?.[0]?.ok === true &&
|
||||
typeof result.state.execution.lastToolResults?.[0]?.data === 'object' &&
|
||||
result.state.execution.lastToolResults?.[0]?.data !== null &&
|
||||
Array.isArray(
|
||||
(result.state.execution.lastToolResults?.[0]?.data as { postWriteReadback?: { regions?: unknown[] } }).postWriteReadback?.regions
|
||||
),
|
||||
{
|
||||
name: 'docx-write-produces-post-write-readback',
|
||||
message: '只要 hand 成功执行过 docx 写,就应返回 postWriteReadback,且不再产生旧写后阻断。',
|
||||
data: {
|
||||
lastToolResults: result.state.execution.lastToolResults
|
||||
}
|
||||
}
|
||||
)
|
||||
}
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -1,65 +0,0 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import { describe, it } from 'vitest'
|
||||
import { createTerminalToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { filterEventsByType } from '../testHelpers'
|
||||
|
||||
type TerminalRunCommandResult = {
|
||||
reason?: 'approval_denied' | 'workspace_required' | 'invalid_input' | 'failed'
|
||||
}
|
||||
|
||||
describe('ai tool terminal backend', () => {
|
||||
it('terminal-unsafe-requires-approval', async () => {
|
||||
await runBackendCase({
|
||||
caseId: 'terminal-unsafe-requires-approval',
|
||||
harnessMode: 'tool-real',
|
||||
domain: 'ai',
|
||||
suite: 'tool',
|
||||
groups: ['terminal'],
|
||||
description: '验证未命中安全指令集但并非高危的 terminal 命令仍必须审批。',
|
||||
fixtures: [],
|
||||
runMode: 'single',
|
||||
execute: async (runtime) => {
|
||||
const createdDir = path.join(runtime.workspaceRoot, 'blocked-dir')
|
||||
const harness = createTerminalToolHarness(runtime, {
|
||||
approvalPolicy: 'deny-all'
|
||||
})
|
||||
|
||||
return {
|
||||
mkdirDenied: await harness.invoke<TerminalRunCommandResult>('terminal_run', {
|
||||
command: 'mkdir blocked-dir'
|
||||
}),
|
||||
blockedDirExists: await fs
|
||||
.access(createdDir)
|
||||
.then(() => true)
|
||||
.catch(() => false),
|
||||
events: runtime.eventCollector.snapshot()
|
||||
}
|
||||
},
|
||||
assert: async (result, runtime) => {
|
||||
const approvalEvents = filterEventsByType(result.events, 'approval_batch_required')
|
||||
|
||||
runtime.assertThat(
|
||||
result.mkdirDenied.reason === 'approval_denied' &&
|
||||
result.blockedDirExists === false &&
|
||||
approvalEvents.some((event) =>
|
||||
event.items.some((item) => item.command === 'mkdir blocked-dir')
|
||||
),
|
||||
{
|
||||
name: 'unsafe-command-requires-approval-even-without-risk-tiering',
|
||||
message: '未命中安全指令集的 terminal 命令即使不是高危命令,也必须先审批。',
|
||||
data: {
|
||||
mkdirDenied: result.mkdirDenied,
|
||||
blockedDirExists: result.blockedDirExists,
|
||||
approvals: approvalEvents.map((event) => ({
|
||||
command: event.items[0]?.command,
|
||||
reason: event.items[0]?.reason
|
||||
}))
|
||||
}
|
||||
}
|
||||
)
|
||||
}
|
||||
})
|
||||
})
|
||||
})
|
||||
File renamed without changes.
+4
-4
@@ -1,13 +1,13 @@
|
||||
import path from 'node:path'
|
||||
import { describe, it } from 'vitest'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
CONTEXT_COMPACTION_SHARED_WORKSPACE_FIXTURE_PATH,
|
||||
HUMAN_CONTEXT_COMPACTION_USER_INPUT,
|
||||
loadContextCompactionWorkspaceSeed
|
||||
} from '../../../harness/backend/liveScenarios/contextCompactionScenario'
|
||||
} from '../../../../harness/backend/liveScenarios/contextCompactionScenario'
|
||||
import {
|
||||
filterAiRunEventsByType,
|
||||
hasDocxApplyArtifact,
|
||||
+5
-5
@@ -1,14 +1,14 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { updateConversationMemoryAfterRun } from '../../../../src/main/services/ai/memory'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { updateConversationMemoryAfterRun } from '../../../../../src/main/services/ai/memory'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
CONVERSATION_MEMORY_FIRST_RUN_USER_INPUT,
|
||||
CONVERSATION_MEMORY_SECOND_RUN_USER_INPUT,
|
||||
CONVERSATION_MEMORY_SHARED_WORKSPACE_FIXTURE_PATH,
|
||||
loadConversationMemoryWorkspaceSeed
|
||||
} from '../../../harness/backend/liveScenarios/conversationMemoryScenario'
|
||||
} from '../../../../harness/backend/liveScenarios/conversationMemoryScenario'
|
||||
import { filterAiRunEventsByType, runWithStructuredOutputRetry } from '../testUtils'
|
||||
|
||||
const SESSION_ID = 'live-conversation-memory-multi-run-session'
|
||||
+4
-4
@@ -1,14 +1,14 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildSelectedDocxFixtureReferences,
|
||||
DOCX_READING_FIXTURE_PATHS,
|
||||
HUMAN_DOCX_CAUSE_REMEDY_READING_USER_INPUT,
|
||||
SHARED_WORKSPACE_FIXTURE_PATH,
|
||||
resolveSelectedDocxFixtures
|
||||
} from '../../../harness/backend/aiLiveTestData'
|
||||
} from '../../../../harness/backend/aiLiveTestData'
|
||||
import {
|
||||
filterAiRunEventsByType,
|
||||
runWithStructuredOutputRetry
|
||||
+4
-4
@@ -1,8 +1,8 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import { describe, it } from 'vitest'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildSelectedDocxReferences,
|
||||
captureDocxFixtureBytes,
|
||||
@@ -11,7 +11,7 @@ import {
|
||||
HUMAN_DOCX_CROSS_SOURCE_SUMMARY_USER_INPUT,
|
||||
SHARED_WORKSPACE_FIXTURE_PATH,
|
||||
resolveSelectedDocxFixtures
|
||||
} from '../../../harness/backend/aiLiveTestData'
|
||||
} from '../../../../harness/backend/aiLiveTestData'
|
||||
import {
|
||||
collectDocxApplyOpsContentKinds,
|
||||
collectStartedDocxEditSourcePaths,
|
||||
+4
-4
@@ -1,14 +1,14 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildSelectedDocxFixtureReferences,
|
||||
DOCX_POSITIONED_FIXTURE_PATHS,
|
||||
HUMAN_DOCX_POSITIONED_STRUCTURAL_EDIT_FALLBACK_USER_INPUT,
|
||||
SHARED_WORKSPACE_FIXTURE_PATH,
|
||||
resolveSelectedDocxFixtures
|
||||
} from '../../../harness/backend/aiLiveTestData'
|
||||
} from '../../../../harness/backend/aiLiveTestData'
|
||||
import {
|
||||
countDocxApplyArtifacts,
|
||||
countDocxFileArtifacts,
|
||||
+5
-5
@@ -1,15 +1,15 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import { describe, it } from 'vitest'
|
||||
import type { AiRunEvent, JsonObject, JsonValue } from '../../../../src/shared/types'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import type { AiRunEvent, JsonObject, JsonValue } from '../../../../../src/shared/types'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildKbGuidedDocxWorkspaceSeed,
|
||||
HUMAN_KB_GUIDED_DOCX_COMPARISON_DRAFT_USER_INPUT,
|
||||
SHARED_WORKSPACE_FIXTURE_PATH
|
||||
} from '../../../harness/backend/aiLiveTestData'
|
||||
} from '../../../../harness/backend/aiLiveTestData'
|
||||
import {
|
||||
collectDocxApplyOpsContentKinds,
|
||||
collectStartedDocxEditSourcePaths,
|
||||
+5
-5
@@ -4,15 +4,15 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
buildBlockIndex,
|
||||
loadDocxBundle
|
||||
} from '../../../../src/main/services/ai/human/tools/docx/adapters/ooxml/blockIndex'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../../src/main/services/ai/human/tools/docx/adapters/ooxml/blockIndex'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
HUMAN_RISK_REPORT_DRAFT_USER_INPUT,
|
||||
loadRiskReportDraftWorkspaceSeed,
|
||||
RISK_REPORT_DRAFT_SHARED_WORKSPACE_FIXTURE_PATH
|
||||
} from '../../../harness/backend/liveScenarios/riskReportDraftScenario'
|
||||
} from '../../../../harness/backend/liveScenarios/riskReportDraftScenario'
|
||||
import {
|
||||
collectStartedDocxEditSourcePaths,
|
||||
collectStartedToolStringArgs,
|
||||
+8
-8
@@ -1,21 +1,21 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { AiRunService } from '../../../../src/main/services/ai/AiRunService'
|
||||
import { AiRunService } from '../../../../../src/main/services/ai/AiRunService'
|
||||
import {
|
||||
invokeStructuredChat,
|
||||
type AiLlmDriver,
|
||||
type AiLlmStructuredDriverResult,
|
||||
type AiLlmTextDriverResult
|
||||
} from '../../../../src/main/services/ai/llmExecutor'
|
||||
import type { AiRunDeps } from '../../../../src/main/services/ai/types'
|
||||
import type { AiRunEvent, JsonObject } from '../../../../src/shared/types'
|
||||
import { hasLiveLlmConfigInEnv, loadLiveLlmConfigFromEnv } from '../../../harness/backend/llmConfig'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import type { BackendTestRuntime } from '../../../harness/backend/test-types'
|
||||
} from '../../../../../src/main/services/ai/llmExecutor'
|
||||
import type { AiRunDeps } from '../../../../../src/main/services/ai/types'
|
||||
import type { AiRunEvent, JsonObject } from '../../../../../src/shared/types'
|
||||
import { hasLiveLlmConfigInEnv, loadLiveLlmConfigFromEnv } from '../../../../harness/backend/llmConfig'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { BackendTestRuntime } from '../../../../harness/backend/test-types'
|
||||
import {
|
||||
buildSelectedDocxFixtureReferences,
|
||||
DOCX_READING_FIXTURE_PATHS,
|
||||
SHARED_WORKSPACE_FIXTURE_PATH
|
||||
} from '../../../harness/backend/aiLiveTestData'
|
||||
} from '../../../../harness/backend/aiLiveTestData'
|
||||
import { filterAiRunEventsByType, runWithStructuredOutputRetry } from '../testUtils'
|
||||
|
||||
const SESSION_ID = 'live-session-resume-with-new-selection'
|
||||
+7
-7
@@ -1,12 +1,12 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { createWebCrawlerService } from '../../../harness/backend/webCrawlerTestUtils'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createMultiHopCrossDomainWebCrawlSite } from '../../../harness/backend/webCrawlLiveTestSites'
|
||||
import { buildHumanWebCrawlMultiHopCrossDomainUserInput } from '../../../harness/backend/aiLiveTestData'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { createWebCrawlerService } from '../../../../harness/backend/webCrawlerTestUtils'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import { createMultiHopCrossDomainWebCrawlSite } from '../../../../harness/backend/webCrawlLiveTestSites'
|
||||
import { buildHumanWebCrawlMultiHopCrossDomainUserInput } from '../../../../harness/backend/aiLiveTestData'
|
||||
import { filterAiRunEventsByType, runWithStructuredOutputRetry } from '../testUtils'
|
||||
import type { JsonObject, JsonValue } from '../../../../src/shared/types'
|
||||
import type { JsonObject, JsonValue } from '../../../../../src/shared/types'
|
||||
|
||||
const normalizeUrl = (value: string): string => value.replace(/\/$/, '')
|
||||
|
||||
+5
-5
@@ -1,9 +1,9 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { createWebCrawlerService } from '../../../harness/backend/webCrawlerTestUtils'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { HUMAN_WEB_CRAWL_URL_INFERENCE_USER_INPUT } from '../../../harness/backend/aiLiveTestData'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { createWebCrawlerService } from '../../../../harness/backend/webCrawlerTestUtils'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import { HUMAN_WEB_CRAWL_URL_INFERENCE_USER_INPUT } from '../../../../harness/backend/aiLiveTestData'
|
||||
import { filterAiRunEventsByType, runWithStructuredOutputRetry } from '../testUtils'
|
||||
|
||||
const isPandasHost = (url: string): boolean => new URL(url).hostname.endsWith('pandas.pydata.org')
|
||||
+4
-4
@@ -1,8 +1,8 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { AiRunService } from '../../../../src/main/services/ai/AiRunService'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createScriptedStructuredOutputDriver } from '../../../harness/backend/scriptedStructuredOutputDriver'
|
||||
import { writeTestDocxFile } from '../../../harness/backend/aiToolHarness'
|
||||
import { AiRunService } from '../../../../../src/main/services/ai/AiRunService'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import { createScriptedStructuredOutputDriver } from '../../../../harness/backend/scriptedStructuredOutputDriver'
|
||||
import { writeTestDocxFile } from '../../../../harness/backend/aiToolHarness'
|
||||
import type { DocxSearchResult } from '../../tool/docx/testUtils'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
+3
-3
@@ -1,7 +1,7 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { AiRunService } from '../../../../src/main/services/ai/AiRunService'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createScriptedStructuredOutputDriver } from '../../../harness/backend/scriptedStructuredOutputDriver'
|
||||
import { AiRunService } from '../../../../../src/main/services/ai/AiRunService'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import { createScriptedStructuredOutputDriver } from '../../../../harness/backend/scriptedStructuredOutputDriver'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
buildLearnSkillsDecision,
|
||||
+4
-4
@@ -1,8 +1,8 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { AiRunService } from '../../../../src/main/services/ai/AiRunService'
|
||||
import type { BackendTestRuntime } from '../../../harness/backend/test-types'
|
||||
import { createScriptedStructuredOutputDriver } from '../../../harness/backend/scriptedStructuredOutputDriver'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { AiRunService } from '../../../../../src/main/services/ai/AiRunService'
|
||||
import type { BackendTestRuntime } from '../../../../harness/backend/test-types'
|
||||
import { createScriptedStructuredOutputDriver } from '../../../../harness/backend/scriptedStructuredOutputDriver'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import { buildCompleteTaskDecision, buildLearnSkillsDecision } from '../testUtils'
|
||||
|
||||
const SESSION_ID = 'resume-snapshot-session'
|
||||
+2
-2
@@ -1,8 +1,8 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
buildToolDecision,
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import { buildCompleteTaskDecision } from '../testUtils'
|
||||
|
||||
describe('ai human mock-llm backend', () => {
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
filterAiRunEventsByType
|
||||
+3
-3
@@ -1,7 +1,7 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { writeTestDocxFile } from '../../../harness/backend/aiToolHarness'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { writeTestDocxFile } from '../../../../harness/backend/aiToolHarness'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxMutationResult, DocxReadResult, DocxSearchResult } from '../../tool/docx/testUtils'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
+7
-7
@@ -3,13 +3,13 @@ import {
|
||||
appendRoundRecord,
|
||||
buildHandExecution,
|
||||
buildLearningExecution
|
||||
} from '../../../../src/main/services/ai/human/memory/rounds'
|
||||
import { createBrainNode } from '../../../../src/main/services/ai/human/node/brainNode'
|
||||
import { createContextCompactionNode } from '../../../../src/main/services/ai/human/node/contextCompactionNode'
|
||||
import { createHumanRuntimeCache } from '../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../src/main/services/ai/human/state/schema'
|
||||
import type { BackendTestRuntime } from '../../../harness/backend/test-types'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../../src/main/services/ai/human/memory/rounds'
|
||||
import { createBrainNode } from '../../../../../src/main/services/ai/human/node/brainNode'
|
||||
import { createContextCompactionNode } from '../../../../../src/main/services/ai/human/node/contextCompactionNode'
|
||||
import { createHumanRuntimeCache } from '../../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../../src/main/services/ai/human/state/schema'
|
||||
import type { BackendTestRuntime } from '../../../../harness/backend/test-types'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
filterAiRunEventsByType
|
||||
+4
-4
@@ -1,8 +1,8 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { updateConversationMemoryAfterRun } from '../../../../src/main/services/ai/memory'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createScriptedStructuredOutputDriver } from '../../../harness/backend/scriptedStructuredOutputDriver'
|
||||
import { updateConversationMemoryAfterRun } from '../../../../../src/main/services/ai/memory'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import { createScriptedStructuredOutputDriver } from '../../../../harness/backend/scriptedStructuredOutputDriver'
|
||||
import { buildCompleteTaskDecision } from '../testUtils'
|
||||
|
||||
const SESSION_ID = 'conversation-memory-multi-run-session'
|
||||
+3
-3
@@ -1,8 +1,8 @@
|
||||
import path from 'node:path'
|
||||
import { describe, it } from 'vitest'
|
||||
import { createDocxToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createDocxToolHarness } from '../../../../harness/backend/aiToolHarness'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
buildLearnSkillsDecision,
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
buildLearnSkillsDecision,
|
||||
+3
-3
@@ -1,7 +1,7 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { writeTestDocxFile } from '../../../harness/backend/aiToolHarness'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { writeTestDocxFile } from '../../../../harness/backend/aiToolHarness'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxReadResult, DocxSearchResult } from '../../tool/docx/testUtils'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
+3
-3
@@ -1,7 +1,7 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { writeTestDocxFile } from '../../../harness/backend/aiToolHarness'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { writeTestDocxFile } from '../../../../harness/backend/aiToolHarness'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxReadResult, DocxSearchResult } from '../../tool/docx/testUtils'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
buildToolDecision,
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
buildLearnSkillsDecision,
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
buildLearnSkillsDecision,
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
filterAiRunEventsByType
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import { filterAiRunEventsByType } from '../testUtils'
|
||||
import { buildToolDecision } from '../testUtils'
|
||||
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
buildToolDecision,
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
buildToolDecision,
|
||||
@@ -3,8 +3,8 @@ import type {
|
||||
HumanState,
|
||||
HumanThought,
|
||||
RuntimeSkillId
|
||||
} from '../../../src/main/services/ai/human/types'
|
||||
import type { AiArtifactRef, AiRunEvent, JsonArray, JsonObject, JsonValue } from '../../../src/shared/types'
|
||||
} from '../../../../src/main/services/ai/human/types'
|
||||
import type { AiArtifactRef, AiRunEvent, JsonArray, JsonObject, JsonValue } from '../../../../src/shared/types'
|
||||
|
||||
export const filterAiRunEventsByType = <TType extends AiRunEvent['type']>(
|
||||
events: readonly AiRunEvent[],
|
||||
File renamed without changes.
+2
-2
@@ -5,8 +5,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
type DocxFailureResult,
|
||||
type DocxMutationResult,
|
||||
+3
-3
@@ -2,9 +2,9 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import type { AiRunEvent } from '../../../../src/shared/types'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { AiRunEvent } from '../../../../../src/shared/types'
|
||||
import type { DocxSearchResult } from './testUtils'
|
||||
|
||||
const filterToolResults = (events: readonly AiRunEvent[], toolName: string) =>
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
type DocxMutationResult,
|
||||
type DocxOutlineResult,
|
||||
+2
-2
@@ -3,8 +3,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxFailureResult } from './testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
+2
-2
@@ -4,8 +4,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxCreateResult } from './testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import { describe, it } from 'vitest'
|
||||
import { createDocxToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createDocxToolHarness } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type {
|
||||
DocxCreateResult,
|
||||
DocxMutationResult,
|
||||
+2
-2
@@ -1,8 +1,8 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import { describe, it } from 'vitest'
|
||||
import { createDocxToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createDocxToolHarness } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxCreateResult, DocxSearchResult } from './testUtils'
|
||||
|
||||
const fileExists = async (filePath: string): Promise<boolean> => {
|
||||
+2
-2
@@ -4,8 +4,8 @@ import JSZip from 'jszip'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxMutationResult, DocxSearchResult } from './testUtils'
|
||||
|
||||
const readDocumentXml = async (filePath: string): Promise<string> => {
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
type DocxMutationResult,
|
||||
type DocxOutlineResult,
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
type DocxFailureResult,
|
||||
type DocxMutationResult,
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
type DocxMutationResult,
|
||||
type DocxOutlineResult,
|
||||
+2
-2
@@ -4,8 +4,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
type DocxHistoryResult,
|
||||
type DocxSearchResult,
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
type DocxOutlineResult,
|
||||
type DocxReadResult,
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
import path from 'node:path'
|
||||
import { describe, it } from 'vitest'
|
||||
import { createDocxToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createDocxToolHarness } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
type DocxMutationResult,
|
||||
type DocxReadResult,
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeRichTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
type DocxMutationResult,
|
||||
type DocxOutlineResult,
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type {
|
||||
DocxMutationResult,
|
||||
DocxOutlineResult,
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxOutlineResult } from './testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxOutlineResult, DocxReadResult, DocxSearchResult } from './testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxOutlineResult } from './testUtils'
|
||||
import { filterAiRunEventsByType } from '../../human/testUtils'
|
||||
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createDocxToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createDocxToolHarness } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type {
|
||||
DocxFailureResult,
|
||||
DocxHistoryResult,
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createDocxToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createDocxToolHarness } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import { type DocxFailureResult } from './testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxOutlineResult, DocxReadResult } from './testUtils'
|
||||
import { findParagraphBlockIndexByText } from './testUtils'
|
||||
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type {
|
||||
DocxMutationResult,
|
||||
DocxOutlineResult,
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type {
|
||||
DocxMutationResult,
|
||||
DocxOutlineResult,
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxHistoryResult, DocxOutlineResult } from './testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxMutationResult, DocxOutlineResult } from './testUtils'
|
||||
import { findParagraphBlockIdByText } from './testUtils'
|
||||
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxSearchResult } from './testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxFailureResult, DocxSearchResult } from './testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxMutationResult, DocxOutlineResult, DocxSearchResult } from './testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxSearchResult } from './testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxSearchResult } from './testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
type DocxMutationResult,
|
||||
type DocxOutlineResult,
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type {
|
||||
DocxMutationResult,
|
||||
DocxOutlineResult,
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxMutationResult, DocxOutlineResult } from './testUtils'
|
||||
import { findParagraphBlockIdByText } from './testUtils'
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import type { DocxChangedBlock, DocxOutlineBlock } from '../../../../src/main/services/ai/human/tools/docx/types'
|
||||
import type { DocxChangedBlock, DocxOutlineBlock } from '../../../../../src/main/services/ai/human/tools/docx/types'
|
||||
|
||||
export type DocxOutlineResult = {
|
||||
sourcePath: string
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
} from '../../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../../harness/backend/run-backend-case'
|
||||
import type { DocxMutationResult, DocxOutlineResult } from '../testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeRichTestDocxFile
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
} from '../../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../../harness/backend/run-backend-case'
|
||||
import type {
|
||||
DocxMutationResult,
|
||||
DocxOutlineResult,
|
||||
+166
@@ -0,0 +1,166 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import { describe, it } from 'vitest'
|
||||
import {
|
||||
buildBlockIndex,
|
||||
loadDocxBundle
|
||||
} from '../../../../../../src/main/services/ai/human/tools/docx/adapters/ooxml/blockIndex'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeRichTestDocxFile
|
||||
} from '../../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../../harness/backend/run-backend-case'
|
||||
import type { DocxMutationResult, DocxOutlineResult, DocxReadResult } from '../testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
it('docx-insert-does-not-duplicate-downstream-blocks', async () => {
|
||||
await runBackendCase({
|
||||
caseId: 'docx-insert-does-not-duplicate-downstream-blocks',
|
||||
harnessMode: 'tool-real',
|
||||
domain: 'ai',
|
||||
suite: 'tool',
|
||||
groups: ['docx', 'write-stability'],
|
||||
description: '验证结构插入后的 review projection 不会重复下游旧段落。',
|
||||
fixtures: [],
|
||||
runMode: 'single',
|
||||
execute: async (runtime) => {
|
||||
const sourcePath = await writeRichTestDocxFile(
|
||||
runtime.workspaceRoot,
|
||||
'insert-after-no-duplicate-tail.docx',
|
||||
[
|
||||
{
|
||||
type: 'paragraph',
|
||||
paragraph: {
|
||||
runs: [
|
||||
{ text: '学生可以通过', bold: true },
|
||||
{ text: '点击对话中的按钮', color: '2563EB' },
|
||||
{ text: '进入引导流程。', underline: true }
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
type: 'paragraph',
|
||||
paragraph: {
|
||||
runs: [{ text: '学生端脚手架引导:' }]
|
||||
}
|
||||
},
|
||||
{
|
||||
type: 'paragraph',
|
||||
paragraph: {
|
||||
runs: [{ text: '引导完成后进入正式练习。' }]
|
||||
}
|
||||
}
|
||||
]
|
||||
)
|
||||
|
||||
const harness = createDocxToolHarness(runtime)
|
||||
try {
|
||||
harness.ctx.beginDocxApplyRun()
|
||||
const outlineBefore = await harness.invoke<DocxOutlineResult>('docx_outline', {
|
||||
sourcePath: 'insert-after-no-duplicate-tail.docx',
|
||||
ranges: [{ type: 'block_index', fromBlockIndex: 0, toBlockIndex: 49 }]
|
||||
})
|
||||
const anchorBlockId =
|
||||
outlineBefore.blocks.find(
|
||||
(block) =>
|
||||
block.type === 'paragraph' &&
|
||||
block.text.includes('学生可以通过点击对话中的按钮进入引导流程。')
|
||||
)?.blockId || ''
|
||||
|
||||
const anchorBefore = await harness.invoke<DocxReadResult>('docx_read_blocks', {
|
||||
sourcePath: 'insert-after-no-duplicate-tail.docx',
|
||||
range: { type: 'block_ids', blockIds: [anchorBlockId] }
|
||||
})
|
||||
const insertedParagraph = await harness.invoke<DocxMutationResult>('docx_edit', {
|
||||
sourcePath: 'insert-after-no-duplicate-tail.docx',
|
||||
actions: [
|
||||
{
|
||||
type: 'insert_blocks',
|
||||
anchorBlockId,
|
||||
position: 'after',
|
||||
blocks: [
|
||||
{
|
||||
type: 'paragraph',
|
||||
text: '学生可以根据按钮提示逐步完成脚手架操作。'
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
})
|
||||
const insertedParagraphBlockId =
|
||||
insertedParagraph.createdBlocks?.find((block) => block.type === 'paragraph')?.blockId ||
|
||||
''
|
||||
const insertedTable = await harness.invoke<DocxMutationResult>('docx_edit', {
|
||||
sourcePath: insertedParagraph.sourcePath,
|
||||
actions: [
|
||||
{
|
||||
type: 'insert_blocks',
|
||||
anchorBlockId: insertedParagraphBlockId,
|
||||
position: 'after',
|
||||
blocks: [
|
||||
{
|
||||
type: 'table',
|
||||
rows: [
|
||||
['入口按钮', '对应动作'],
|
||||
['开始引导', '进入脚手架']
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
})
|
||||
const anchorAfter = await harness.invoke<DocxReadResult>('docx_read_blocks', {
|
||||
sourcePath: insertedTable.sourcePath,
|
||||
range: { type: 'block_ids', blockIds: [anchorBlockId] }
|
||||
})
|
||||
const records = await harness.ctx.applyDocxRunEdits()
|
||||
if (records.length < 1) throw new Error('expected docx apply record')
|
||||
|
||||
const reviewBundle = await loadDocxBundle(await fs.readFile(sourcePath))
|
||||
const reviewOutline = buildBlockIndex(reviewBundle.documentDom)
|
||||
.outline.filter((block) => block.type !== 'table_cell')
|
||||
.map((block) => ({
|
||||
type: block.type,
|
||||
text: block.type === 'table' ? block.preview : block.text
|
||||
}))
|
||||
|
||||
return {
|
||||
anchorBefore: anchorBefore.blocks[0],
|
||||
anchorAfter: anchorAfter.blocks[0],
|
||||
records,
|
||||
reviewOutline
|
||||
}
|
||||
} finally {
|
||||
await harness.release()
|
||||
}
|
||||
},
|
||||
assert: async (result, runtime) => {
|
||||
runtime.assertThat(
|
||||
result.anchorBefore?.type === 'paragraph' &&
|
||||
result.anchorAfter?.type === 'paragraph' &&
|
||||
JSON.stringify(result.anchorBefore.runs) === JSON.stringify(result.anchorAfter.runs),
|
||||
{
|
||||
name: 'anchor-runs-remain-identical-after-structural-insert',
|
||||
message: '结构插入后,anchor 原段落的 run 样式不应发生漂移。',
|
||||
data: {
|
||||
anchorBefore: result.anchorBefore,
|
||||
anchorAfter: result.anchorAfter
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
runtime.assertThat(
|
||||
result.reviewOutline.filter((block) =>
|
||||
String(block.text || '').includes('学生端脚手架引导:')
|
||||
).length === 1,
|
||||
{
|
||||
name: 'review-does-not-duplicate-downstream-tail-block',
|
||||
message: 'review artifact 中下游旧段落只应出现一次,不得因插入重绑错误而重复。',
|
||||
data: {
|
||||
reviewOutline: result.reviewOutline
|
||||
}
|
||||
}
|
||||
)
|
||||
}
|
||||
})
|
||||
})
|
||||
})
|
||||
+2
-2
@@ -2,8 +2,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createDocxToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
} from '../../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../../harness/backend/run-backend-case'
|
||||
import type { DocxMutationResult, DocxOutlineResult } from '../testUtils'
|
||||
|
||||
describe('ai tool docx backend', () => {
|
||||
+6
-6
@@ -1,10 +1,10 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { HUMAN_HAND_CALL_LIMIT } from '../../../../src/main/services/ai/human/constants'
|
||||
import { createHandNode } from '../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../src/main/services/ai/human/state/schema'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { HUMAN_HAND_CALL_LIMIT } from '../../../../../src/main/services/ai/human/constants'
|
||||
import { createHandNode } from '../../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../../src/main/services/ai/human/state/schema'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
|
||||
const isWindows = process.platform === 'win32'
|
||||
|
||||
+6
-6
@@ -1,10 +1,10 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHandNode } from '../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../src/main/services/ai/human/state/schema'
|
||||
import { writeTestDocxFile } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHandNode } from '../../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../../src/main/services/ai/human/state/schema'
|
||||
import { writeTestDocxFile } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
|
||||
describe('ai tool hand backend', () => {
|
||||
it('hand-docx-edit-invalid-blockid-failed', async () => {
|
||||
+7
-7
@@ -1,11 +1,11 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHandNode } from '../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { executeToolCall } from '../../../../src/main/services/ai/human/node/handNode/toolExecutor'
|
||||
import { createInitialHumanState } from '../../../../src/main/services/ai/human/state/schema'
|
||||
import { createHumanRuntimeCache } from '../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { writeTestDocxFile } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHandNode } from '../../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { executeToolCall } from '../../../../../src/main/services/ai/human/node/handNode/toolExecutor'
|
||||
import { createInitialHumanState } from '../../../../../src/main/services/ai/human/state/schema'
|
||||
import { createHumanRuntimeCache } from '../../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { writeTestDocxFile } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import type { DocxOutlineResult } from '../docx/testUtils'
|
||||
|
||||
describe('ai tool hand backend', () => {
|
||||
+6
-6
@@ -1,10 +1,10 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHandNode } from '../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../src/main/services/ai/human/state/schema'
|
||||
import { writeTestDocxFile } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHandNode } from '../../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../../src/main/services/ai/human/state/schema'
|
||||
import { writeTestDocxFile } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
|
||||
describe('ai tool hand backend', () => {
|
||||
it('hand-docx-same-file-read-write-read-allowed', async () => {
|
||||
+6
-6
@@ -1,10 +1,10 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHandNode } from '../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../src/main/services/ai/human/state/schema'
|
||||
import { writeTestDocxFile } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHandNode } from '../../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../../src/main/services/ai/human/state/schema'
|
||||
import { writeTestDocxFile } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
|
||||
describe('ai tool hand backend', () => {
|
||||
it('hand-docx-single-file-multi-write-allowed', async () => {
|
||||
+5
-5
@@ -1,9 +1,9 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHandNode } from '../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../src/main/services/ai/human/state/schema'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHandNode } from '../../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../../src/main/services/ai/human/state/schema'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
|
||||
describe('ai tool hand backend', () => {
|
||||
it('hand-schema-preflight-invalid-input', async () => {
|
||||
+6
-6
@@ -1,10 +1,10 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHandNode } from '../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../src/main/services/ai/human/state/schema'
|
||||
import { writeTestDocxFile } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHandNode } from '../../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../../src/main/services/ai/human/state/schema'
|
||||
import { writeTestDocxFile } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
|
||||
describe('ai tool hand backend', () => {
|
||||
it('hand-serial-docx-chain', async () => {
|
||||
+2
-2
@@ -1,8 +1,8 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildCompleteTaskDecision,
|
||||
filterAiRunEventsByType
|
||||
+7
-91
@@ -1,11 +1,11 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createHandNode } from '../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../src/main/services/ai/human/state/schema'
|
||||
import type { AiArtifactRef } from '../../../../src/shared/types'
|
||||
import { writeTestDocxFile } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createHandNode } from '../../../../../src/main/services/ai/human/node/handNode'
|
||||
import { ensureSkillTools } from '../../../../../src/main/services/ai/human/node/learningNode/skillLoader'
|
||||
import { createHumanRuntimeCache } from '../../../../../src/main/services/ai/human/node/utils/runtimeCache'
|
||||
import { createInitialHumanState } from '../../../../../src/main/services/ai/human/state/schema'
|
||||
import type { AiArtifactRef } from '../../../../../src/shared/types'
|
||||
import { writeTestDocxFile } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
|
||||
const isWindows = process.platform === 'win32'
|
||||
|
||||
@@ -48,90 +48,6 @@ const buildHandState = (input: {
|
||||
})
|
||||
|
||||
describe('ai tool hand backend', () => {
|
||||
it('hand-tool-limit-near-exhausted', async () => {
|
||||
await runBackendCase({
|
||||
caseId: 'hand-tool-limit-near-exhausted',
|
||||
harnessMode: 'tool-real',
|
||||
domain: 'ai',
|
||||
suite: 'tool',
|
||||
groups: ['hand'],
|
||||
description: '验证 hand 在达到工具预算 0.6 阈值后会发出尽快收尾的提示。',
|
||||
fixtures: [],
|
||||
runMode: 'single',
|
||||
execute: async (runtime) => {
|
||||
const ctx = runtime.createAiRunContext({
|
||||
humanRuntimeOverrides: {
|
||||
hardActionLimit: 5
|
||||
}
|
||||
})
|
||||
const cache = createHumanRuntimeCache()
|
||||
|
||||
ensureSkillTools({
|
||||
cache,
|
||||
ctx,
|
||||
runId: runtime.runId,
|
||||
taskId: runtime.taskId,
|
||||
skillId: 'terminal'
|
||||
})
|
||||
|
||||
const handNode = createHandNode({ ctx, cache })
|
||||
const firstState = await handNode(
|
||||
buildHandState({
|
||||
taskId: runtime.taskId,
|
||||
userInput: '第一次安全指令',
|
||||
learnedSkillIds: [],
|
||||
toolName: 'terminal_run',
|
||||
args: {
|
||||
command: isWindows ? 'Get-Location' : 'pwd'
|
||||
}
|
||||
})
|
||||
)
|
||||
const secondState = await handNode({
|
||||
...buildHandState({
|
||||
taskId: runtime.taskId,
|
||||
userInput: '第二次安全指令',
|
||||
learnedSkillIds: [],
|
||||
toolName: 'terminal_run',
|
||||
args: {
|
||||
command: isWindows ? 'Get-ChildItem -LiteralPath .' : 'ls'
|
||||
}
|
||||
}),
|
||||
trace: firstState.trace
|
||||
})
|
||||
const thirdState = await handNode({
|
||||
...buildHandState({
|
||||
taskId: runtime.taskId,
|
||||
userInput: '第三次安全指令',
|
||||
learnedSkillIds: [],
|
||||
toolName: 'terminal_run',
|
||||
args: {
|
||||
command: isWindows ? 'Get-Location' : 'pwd'
|
||||
}
|
||||
}),
|
||||
trace: secondState.trace
|
||||
})
|
||||
|
||||
return {
|
||||
thirdState
|
||||
}
|
||||
},
|
||||
assert: async (result, runtime) => {
|
||||
runtime.assertThat(
|
||||
result.thirdState.execution.pendingDecisionHint?.code === 'hand_tool_limit_near_exhausted' &&
|
||||
result.thirdState.execution.pendingDecisionHint.detail.limit === 5 &&
|
||||
result.thirdState.execution.pendingDecisionHint.detail.toolCallCount === 3,
|
||||
{
|
||||
name: 'tool-limit-near-exhausted-produces-soft-converging-decision-hint',
|
||||
message: '达到工具预算 0.6 阈值后,hand 应写入尽快收尾的软提示。',
|
||||
data: {
|
||||
decisionHint: result.thirdState.execution.pendingDecisionHint
|
||||
}
|
||||
}
|
||||
)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
it('hand-tool-limit-and-artifact-merge', async () => {
|
||||
await runBackendCase({
|
||||
caseId: 'hand-tool-limit-and-artifact-merge',
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createRetrievalToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createRetrievalToolHarness } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
|
||||
type KbSearchRow = {
|
||||
id: string
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createRetrievalToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createRetrievalToolHarness } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
|
||||
type KbListDocumentsResult = {
|
||||
items: Array<{
|
||||
@@ -1,7 +1,7 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import { resolveRunLogDir } from '../../harness/backend/write-run-log'
|
||||
import type { TestArtifactsIndex, TestRunMeta } from '../../harness/core/run-log-contract'
|
||||
import { resolveRunLogDir } from '../../../harness/backend/write-run-log'
|
||||
import type { TestArtifactsIndex, TestRunMeta } from '../../../harness/core/run-log-contract'
|
||||
|
||||
export const readCapturedRunLog = async (input: {
|
||||
scope: string
|
||||
+2
-2
@@ -1,8 +1,8 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import { describe, it } from 'vitest'
|
||||
import { createTerminalToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createTerminalToolHarness } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import { filterEventsByType } from '../testHelpers'
|
||||
|
||||
type TerminalRunCommandResult = {
|
||||
+2
-2
@@ -1,8 +1,8 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { createTerminalToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createTerminalToolHarness } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import { filterEventsByType } from '../testHelpers'
|
||||
import { readCapturedRunLog } from '../logHelpers'
|
||||
|
||||
+2
-2
@@ -3,8 +3,8 @@ import { describe, it } from 'vitest'
|
||||
import {
|
||||
createTerminalToolHarness,
|
||||
writeTestDocxFile
|
||||
} from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
} from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
|
||||
type TerminalRunCommandResult = {
|
||||
reason?: 'result_unusable'
|
||||
+40
-3
@@ -1,8 +1,8 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import { describe, it } from 'vitest'
|
||||
import { createTerminalToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createTerminalToolHarness } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import { filterEventsByType } from '../testHelpers'
|
||||
|
||||
type TerminalRunCommandResult = {
|
||||
@@ -25,6 +25,7 @@ describe('ai tool terminal backend', () => {
|
||||
runMode: 'single',
|
||||
execute: async (runtime) => {
|
||||
const guardedDir = path.join(runtime.workspaceRoot, 'guarded')
|
||||
const blockedDir = path.join(runtime.workspaceRoot, 'blocked-dir')
|
||||
const outsidePath = path.join(path.dirname(runtime.workspaceRoot), 'outside.txt')
|
||||
await fs.mkdir(guardedDir, { recursive: true })
|
||||
await fs.writeFile(path.join(guardedDir, 'keep.txt'), 'keep\n')
|
||||
@@ -37,6 +38,11 @@ describe('ai tool terminal backend', () => {
|
||||
unsafeDenied: await harness.invoke<TerminalRunCommandResult>('terminal_run', {
|
||||
command: isWindows ? 'Remove-Item -Recurse -Force guarded' : 'rm -rf guarded'
|
||||
}),
|
||||
unlistedWriteDenied: await harness.invoke<TerminalRunCommandResult>('terminal_run', {
|
||||
command: isWindows
|
||||
? 'New-Item -ItemType Directory -Path blocked-dir'
|
||||
: 'mkdir blocked-dir'
|
||||
}),
|
||||
relativeEscape: await harness.invoke<TerminalRunCommandResult>('terminal_run', {
|
||||
command: isWindows ? 'Get-Content ../outside.txt' : 'cat ../outside.txt'
|
||||
}),
|
||||
@@ -47,6 +53,10 @@ describe('ai tool terminal backend', () => {
|
||||
.access(path.join(guardedDir, 'keep.txt'))
|
||||
.then(() => true)
|
||||
.catch(() => false),
|
||||
blockedDirExists: await fs
|
||||
.access(blockedDir)
|
||||
.then(() => true)
|
||||
.catch(() => false),
|
||||
events: runtime.eventCollector.snapshot()
|
||||
}
|
||||
},
|
||||
@@ -59,7 +69,8 @@ describe('ai tool terminal backend', () => {
|
||||
approvalEvents.some((event) =>
|
||||
event.items.some(
|
||||
(item) =>
|
||||
item.command === (isWindows ? 'Remove-Item -Recurse -Force guarded' : 'rm -rf guarded')
|
||||
item.command ===
|
||||
(isWindows ? 'Remove-Item -Recurse -Force guarded' : 'rm -rf guarded')
|
||||
)
|
||||
) &&
|
||||
result.guardedStillExists,
|
||||
@@ -77,6 +88,32 @@ describe('ai tool terminal backend', () => {
|
||||
}
|
||||
)
|
||||
|
||||
runtime.assertThat(
|
||||
result.unlistedWriteDenied.reason === 'approval_denied' &&
|
||||
result.blockedDirExists === false &&
|
||||
approvalEvents.some((event) =>
|
||||
event.items.some(
|
||||
(item) =>
|
||||
item.command ===
|
||||
(isWindows
|
||||
? 'New-Item -ItemType Directory -Path blocked-dir'
|
||||
: 'mkdir blocked-dir')
|
||||
)
|
||||
),
|
||||
{
|
||||
name: 'unlisted-write-command-requires-approval',
|
||||
message: '未命中安全指令集的低风险写命令也必须先审批;审批拒绝后不得创建目录。',
|
||||
data: {
|
||||
unlistedWriteDenied: result.unlistedWriteDenied,
|
||||
blockedDirExists: result.blockedDirExists,
|
||||
approvals: approvalEvents.map((event) => ({
|
||||
command: event.items[0]?.command,
|
||||
reason: event.items[0]?.reason
|
||||
}))
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
runtime.assertThat(
|
||||
result.relativeEscape.reason === 'workspace_required' &&
|
||||
result.absoluteEscape.reason === 'workspace_required' &&
|
||||
+2
-2
@@ -1,8 +1,8 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import { describe, it } from 'vitest'
|
||||
import { createTerminalToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createTerminalToolHarness } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import { filterEventsByType } from '../testHelpers'
|
||||
|
||||
type TerminalRunCommandResult = {
|
||||
@@ -1,5 +1,5 @@
|
||||
import type { DocumentRecord, SearchResult } from '../../../src/shared/types'
|
||||
import type { AiRunEvent } from '../../../src/shared/types'
|
||||
import type { DocumentRecord, SearchResult } from '../../../../src/shared/types'
|
||||
import type { AiRunEvent } from '../../../../src/shared/types'
|
||||
|
||||
export const createTestDocumentRecord = (input: {
|
||||
id: string
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { createUtilityToolHarness } from '../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import { createUtilityToolHarness } from '../../../../harness/backend/aiToolHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
|
||||
type WaitTimeResult = {
|
||||
waitedMs: number
|
||||
Loaded 100 of 210 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user