diff --git a/AGENTS.md b/AGENTS.md index 85e47152..1be9bda6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ ### 1.4 测试相关 - `doc/test-plan/ai-test.md`：涉及 AI 回归测试范围、live case 设计或 suite 覆盖维护时阅读。 -- `doc/test-plan/knowledge-test.md`：涉及知识库专项时阅读。当前尚未登记 case。 +- `doc/test-plan/knowledge-test.md`：涉及知识库专项（foundation 功能与 optimization 效果）时阅读。 - `doc/test-plan/settings-test.md`：涉及 Settings 测试范围或测试边界时阅读。 - `doc/test-plan/workbench-test.md`：涉及 Workbench 回归测试范围、真实 e2e case 设计、发送前置条件验证或 suite 覆盖维护时阅读。 - `doc/test.md`：涉及测试体系、UI E2E 回归规则、runner 或回归测试流程时阅读。 @@ -61,7 +61,7 @@ │ │ └── workbench.md # Workbench 产品需求（含定时任务） │ ├── test-plan/ # 专项测试计划 │ │ ├── ai-test.md # AI 回归测试计划 -│ │ ├── knowledge-test.md # 知识库专项测试计划（当前无 case） +│ │ ├── knowledge-test.md # 知识库专项测试计划 │ │ ├── settings-test.md # Settings 测试计划 │ │ └── workbench-test.md # Workbench 测试计划 │ └── tidy.md # 周期性文档、测试与代码整理流程 @@ -78,7 +78,7 @@ ├── tests/ # 自动化测试 │ ├── case/ # 测试用例 │ │ ├── ai/ # AI 测试用例 -│ │ ├── knowledge/ # 知识库专项测试（当前无 case） +│ │ ├── knowledge/ # 知识库专项测试 │ │ ├── showcase/ # 独立官网样例采集 │ │ ├── settings/ # Settings 测试用例 │ │ ├── unit/ # backend 单元测试 diff --git a/doc/test-plan/ai-test.md b/doc/test-plan/ai-test.md index 2d9f2626..20c9cf7c 100644 --- a/doc/test-plan/ai-test.md +++ b/doc/test-plan/ai-test.md @@ -434,7 +434,7 @@ DOCX tool 回归以 7 个公开工具、story 内容树、稳定 `nodeId`、segm - Why：KB tool family 必须显式覆盖每个回归工具。 - How：对已 seed 的文件与网页数据分别执行三种检索，并覆盖 `documentIds` 缩圈。 - Oracle：三种检索都返回稳定 `items`；`documentIds` 缩圈语义成立（含全部不存在时返回空）；文件命中保留 `sourceType=file` 与 `filePath`，网页命中保留 `sourceType=web` 与 `sourceUrl`，供最终引用生成可追溯地址。 -- Not：不比较召回优劣；不测人造跨 workspace 脏数据隔离。设置页「更新索引」期间的拦截由 unit `kb-indexing-blocks-search` 覆盖 `SET-WORKSPACE-004-AC9`、`SET-WEB-004-AC8`（工具失败，模型侧结果为 `Error: 正在更新知识库索引，请完成后再检索。`）；「检查更新」不拦截。 +- Not：不比较召回优劣； #### terminal 类别 diff --git a/doc/test-plan/knowledge-test.md b/doc/test-plan/knowledge-test.md index 1512c85b..7583d7b2 100644 --- a/doc/test-plan/knowledge-test.md +++ b/doc/test-plan/knowledge-test.md @@ -2,22 +2,110 @@ ## 1. 总体目标 -- 本计划定义知识库专项回归，对标 `ai-test`：召回、检索、索引质量与后续专项优化。 +- 本计划定义知识库专项回归，对标 `ai-test`，全部是 backend。 - 通用测试、runner 与断言规则遵循 `doc/test.md`。 -- 正式 case 放在 `tests/case/knowledge/**`，全部是 backend。 +- 正式 case 放在 `tests/case/knowledge//**`。 -当前没有登记 case。专项 suite 与 case 在后续做召回、检索或索引优化时再设计。 +本计划覆盖两类： + +1. `foundation`：知识库功能（扫描、建索引、job、索引中拦截检索、索引速度线程预算、多地址顺序更新）。 +2. `optimization`：召回率、检索与索引效果。当前无 case，后续做优化时再设计。 边界： -- 不替代 Settings「更新索引」产品 E2E。 +- 不替代 Settings「更新索引」产品 E2E。E2E 在页面上改慢/高速后再走真实更新索引；本计划只测 backend 公式与服务契约。 - 不替代 `ai-test` 中的 kb tool / live。 -- 不登记 unit。函数级契约仍在 `tests/case/unit/**`（含 `tests/case/unit/knowledge/**`）。 +- 不登记 unit。函数级补足仍在 `tests/case/unit/knowledge/**`。 ## 2. Suite 设计 -暂不定义。 +### 2.1 `foundation` -## 3. 回归清单 +- 目标：守住知识库可回归的功能主链。 +- 约束：走生产 store / job / 扫描 / worker；索引速度 case 只断言 ORT 线程预算，不测 UI、不测真实 CPU 占用。 +- 目录：`tests/case/knowledge/foundation/.test.ts` + +### 2.2 `optimization` + +- 目标：召回率、检索与索引效果评测。 +- 当前无登记 case。 + +## 3. 领域规则 + +- 全部是 backend，禁止导入 Renderer。 +- 线程预算以逻辑核 `N` 为准（Mac / Windows 相同）： + - 慢速：`max(1, floor(N/2))`，关闭 spinning + - 高速：`max(2, N - reserve)`，`reserve = N≤8 ? 1 : 2` +- 多文件地址更新索引必须顺序 kickoff。 +- 不在 `foundation` 比较召回优劣。 + +## 4. 回归清单 + +### 4.1 `foundation` + +#### `workspace-scan-contract` + +- 测试类型：`backend` +- What：扫描真实目录后生成文件索引记录，并由派生摘要支撑刷新。 +- Why：检查更新的 backend 主链；漏扫或摘要错会导致页面可索引数失真。 +- How：隔离 workspace fixture，写入可见/隐藏目录后调用扫描服务并读取索引记录与摘要。 +- Oracle：支持文件进入索引记录；忽略目录不入库；摘要与扫描进度可支撑页面刷新。 +- Not：不验证 Settings 按钮与进度块 UI。 + +#### `workspace-index-job-store-contract` + +- 测试类型：`backend` +- What：`workspace_index_job` 支持创建、active 查询、阶段进度更新和失败收敛。 +- Why：更新索引的进度恢复与中断收敛依赖 job store。 +- How：真实 job store 写入 running/failed 记录并查询 active、进度与批量失败。 +- Oracle：active 取最新 running；进度可持久化读取；running job 可批量失败，对齐 `SET-WORKSPACE-004-AC3`、`SET-WORKSPACE-004-AC4`、`SET-WORKSPACE-004-AC5`。 +- Not：不验证页面进度块文案。 + +#### `workspace-index-worker-contract` + +- 测试类型：`backend` +- What：文件索引 worker 协议与 embedding batch 限制保持稳定。 +- Why：worker 输入输出或切批失真会让更新索引静默丢文件或打爆嵌入。 +- How：真实 worker 协议与切批函数覆盖支持文件与超限文本。 +- Oracle：协议可执行；batch 遵守条数与字符上限。 +- Not：不验证向量召回质量。 + +#### `workspace-interrupted-job-recovery-contract` + +- 测试类型：`backend` +- What：启动恢复把遗留 running job 和条目级执行中状态收敛为失败。 +- Why：进程被杀后必须能退出“执行中”，否则无法再次更新索引。 +- How：写入遗留 running job 与执行中文件状态后走启动恢复。 +- Oracle：job 与条目收敛为失败，对齐 `SET-WORKSPACE-004-AC5`。 +- Not：不模拟操作系统强杀 UI。 + +#### `kb-indexing-blocks-search` + +- 测试类型：`backend` +- What：prepare 索引进行中时 `kb_search` / `kb_list_documents` 失败，并投递模型可读错误。 +- Why：更新索引期间检索必须明确失败，避免半库结果。 +- How：构造索引中上下文调用两个工具，并检查 model-facing 投递文案。 +- Oracle：两工具抛出正在更新知识库错误；模型侧结果为 `Error: 正在更新知识库索引，请完成后再检索。`，对齐 `SET-WORKSPACE-004-AC9`、`SET-WEB-004-AC8`。 +- Not：不验证检查更新是否拦截（检查更新不拦截）；不替代 `ai-test` kb tool 召回契约。 + +#### `index-speed-thread-budget` + +- 测试类型：`backend` +- What：慢速/高速按逻辑核数映射 ORT `intraOpNumThreads` 与 spinning。 +- Why：索引速度档位的 backend 效果是线程预算，不是页面文案。 +- How：对 4/8/16 核分别计算慢速与高速预算。 +- Oracle：慢速 2/4/8 且 `allowSpinning=false`；高速 3/7/14 且 `allowSpinning=true`。 +- Not：不测 Settings 切换 UI、不问号文案、不测真实 CPU 占用。页面改档后再更新索引由 `workspace-prepare-refresh` E2E 覆盖。 + +#### `prepare-workspaces-sequentially` + +- 测试类型：`backend` +- What：多地址更新索引按顺序等待前一个完成再启动下一个。 +- Why：并行 kickoff 会叠满嵌入线程，和慢/高速档位冲突。 +- How：对三个 workspace id 顺序调用 prepare，记录 start/end。 +- Oracle：事件顺序为每个地址 start→end 后再开始下一个，对齐 `SET-WORKSPACE-004-AC11`。 +- Not：不验证 Settings 多地址按钮 UI。 + +### 4.2 `optimization` 无。 diff --git a/doc/test-plan/settings-test.md b/doc/test-plan/settings-test.md index e96ab20d..802c113f 100644 --- a/doc/test-plan/settings-test.md +++ b/doc/test-plan/settings-test.md @@ -19,7 +19,7 @@ Settings 顶层 suite 对齐 PRD 模块： - `model` 验证大模型可用服务配置的加载、创建、编辑、删除，以及预设服务商与自定义配置的类型、锁定和去重语义。 - `general` 验证 AI 运行系统通知开关的加载、保存和关闭后的静默行为，以及错误日志目录入口；系统通知中心的真实视觉展示按 macOS 与 Windows 发布前手动验证。 -- `workspace` 验证文件地址森林及其并列的全局网页资料：添加/删除地址、包含关系、树顶层节点、合计索引进度、网页全局列表。索引速度档位的线程公式与多地址顺序由 unit 覆盖，不在本计划重复测算力。`SET-WORKSPACE-009` 两 tab 共用全局档位；默认慢速与问号说明不进日常 E2E。 +- `workspace` 验证文件地址森林及其并列的全局网页资料：添加/删除地址、包含关系、树顶层节点、合计索引进度、网页全局列表。`SET-WORKSPACE-009` 由 `workspace-prepare-refresh` 在页面切换档位后走真实更新索引覆盖；线程公式与多地址顺序由 knowledge `foundation` 覆盖。问号文案与网页 tab 同步不单独测。 - `vector` 属于 Settings 产品域，页面路由为 `/settings/vector`（`/model-download` 仅为历史重定向），测试目录与 runner 仍统一归入 `tests/case/settings/vector/**`。 - `workspace` 是 Settings 下的 suite，按 `core / files / web` 组织，语义为“文件地址 + 文件 tab + 全局网页 tab”，不是具名容器。 - `workspace` 相关回归测试默认复用 `tests/fixtures/workspace` 作为共享输入目录。 @@ -122,9 +122,9 @@ Settings 顶层 suite 对齐 PRD 模块： - 测试类型：`e2e` - What：点击“更新索引”后，文件状态会刷新为已索引并同步更新摘要。 - Why：“索引已经可用”的核心用户反馈链路。 -- How：复制共享 workspace 输入的隔离副本，通过 Settings UI 添加为文件地址；随后点击更新索引，读取刷新后的文件树和全部支持文件状态。共享 fixture 含 `.doc`、`docx`、`pdf`、`md`、`txt`、`xlsx`、`xlsm` 等全部受支持类型，其中 excel 只以元数据方式索引进库。 -- Oracle：更新索引过程中出现持久化总进度，且检查更新/更新索引按钮保持可用，对齐 `SET-WORKSPACE-004-AC3`；运行期间目录读取次数为 0，终态只刷新一次且不因逐文件完成而闪烁或折叠，对齐 `SET-WORKSPACE-004-AC7`；共享 fixture 中全部支持文件真实进入处理范围，对齐 `SET-WORKSPACE-004-AC1`；更新后所有支持文件收敛为 `indexed / failed`，摘要和节点状态同步刷新，对齐 `SET-WORKSPACE-004-AC2`。 -- Not：不验证向量内容质量和搜索召回结果。更新索引执行中 Workbench `kb_search` / `kb_list_documents` 返回错误、检查更新不拦截，由 unit `kb-indexing-blocks-search` 覆盖 `SET-WORKSPACE-004-AC9`。多地址顺序执行由 unit `prepare-workspaces-sequentially` 覆盖 `SET-WORKSPACE-004-AC11`。索引速度档位的线程预算由 unit `index-speed-thread-budget` 覆盖。 +- How：复制共享 workspace 输入的隔离副本，通过 Settings UI 添加为文件地址；确认默认慢速后切换为高速并保存，再点击更新索引，读取刷新后的文件树和全部支持文件状态。共享 fixture 含 `.doc`、`docx`、`pdf`、`md`、`txt`、`xlsx`、`xlsm` 等全部受支持类型，其中 excel 只以元数据方式索引进库。 +- Oracle：默认慢速后可切换高速，本次更新索引在切换后的档位上执行，对齐 `SET-WORKSPACE-009-AC1`；更新索引过程中出现持久化总进度，且检查更新/更新索引按钮保持可用，对齐 `SET-WORKSPACE-004-AC3`；运行期间目录读取次数为 0，终态只刷新一次且不因逐文件完成而闪烁或折叠，对齐 `SET-WORKSPACE-004-AC7`；共享 fixture 中全部支持文件真实进入处理范围，对齐 `SET-WORKSPACE-004-AC1`；更新后所有支持文件收敛为 `indexed / failed`，摘要和节点状态同步刷新，对齐 `SET-WORKSPACE-004-AC2`。 +- Not：不验证向量内容质量和搜索召回结果，也不测 CPU 占用公式。更新索引执行中 Workbench `kb_search` / `kb_list_documents` 返回错误、检查更新不拦截，由 knowledge `kb-indexing-blocks-search` 覆盖 `SET-WORKSPACE-004-AC9`。多地址顺序执行由 knowledge `prepare-workspaces-sequentially` 覆盖 `SET-WORKSPACE-004-AC11`。线程预算由 knowledge `index-speed-thread-budget` 覆盖。问号说明与网页 tab 同步不单独测。 #### `workspace-prepare-failed-retry` @@ -180,7 +180,7 @@ Settings 顶层 suite 对齐 PRD 模块： - Why：网页索引是“爬取 + 建索引”两阶段链路，失败类型必须可区分。 - How：预置一条成功网页和一条带失败标记的网页，先执行“检查更新”，再执行“更新索引”。 - Oracle：网页列表、摘要、失败数量和操作区初始化对齐 `SET-WEB-001-AC1`、`SET-WEB-001-AC2`；检查进度和完成状态对齐 `SET-WEB-003-AC1`、`SET-WEB-003-AC2`；更新时网页进入抓取/索引状态，对齐 `SET-WEB-004-AC1`；成功与失败分别收敛，对齐 `SET-WEB-004-AC2`、`SET-WEB-004-AC3`；分阶段进度对齐 `SET-WEB-004-AC4`。 -- Not：不验证真实网页正文切块、embedding 写入质量和索引失败以外的网络细节；向量模型未安装时「更新索引」禁用与说明（`SET-WEB-004-AC7`，文件侧姊妹 `SET-WORKSPACE-004-AC6`）标为手动 / 暂缓，不进日常自动化回归。网页更新索引执行中的 kb 拦截由 unit `kb-indexing-blocks-search` 覆盖 `SET-WEB-004-AC8`。 +- Not：不验证真实网页正文切块、embedding 写入质量和索引失败以外的网络细节；向量模型未安装时「更新索引」禁用与说明（`SET-WEB-004-AC7`，文件侧姊妹 `SET-WORKSPACE-004-AC6`）标为手动 / 暂缓，不进日常自动化回归。网页更新索引执行中的 kb 拦截由 knowledge `kb-indexing-blocks-search` 覆盖 `SET-WEB-004-AC8`。 #### `web-prepare-progress-restore` diff --git a/doc/test.md b/doc/test.md index 37b6ab51..4dfe358c 100644 --- a/doc/test.md +++ b/doc/test.md @@ -4,7 +4,7 @@ - 本文是 NCurator-Desktop 的唯一测试总规范。 - 各业务域的 suite、case 和领域规则只在对应 `doc/test-plan/**` 中维护，本文不重复登记 case。 -- `ai-test` 专门覆盖 AI 架构及其工具，全部是 backend；`knowledge-test` 专门覆盖知识库的功能测试（如搜索和索引等），优化设计测试（如召回率、检索与索引优化等），全部是 backend，当前尚未登记 case；`settings-test` 和 `workbench-test` 专门覆盖对应 PRD 功能点，全部是 E2E；Unit 只补足前述范围未覆盖的 backend 函数、类和底层逻辑；`showcase` 是独立官网样例采集，不进入领域测试计划。 +- `ai-test` 专门覆盖 AI 架构及其工具，全部是 backend；`knowledge-test` 专门覆盖知识库的功能测试（如搜索和索引等），优化设计测试（如召回率、检索与索引优化等），全部是 backend；`settings-test` 和 `workbench-test` 专门覆盖对应 PRD 功能点，全部是 E2E；Unit 只补足前述范围未覆盖的 backend 函数、类和底层逻辑；`showcase` 是独立官网样例采集，不进入领域测试计划。 测试体系只沉淀一层长期执行产物： @@ -144,7 +144,7 @@ runner 不负责： - `pnpm test:ai:human:mock-llm` - `pnpm test:ai:human:live` - Knowledge - - `pnpm test:knowledge:backend`：知识库专项；当前尚未登记 case。 + - `pnpm test:knowledge:backend` - Unit - `pnpm test:unit` - `pnpm test:unit:ai` diff --git a/tests/case/knowledge/foundation/index-speed-thread-budget.test.ts b/tests/case/knowledge/foundation/index-speed-thread-budget.test.ts index 11edfcee..9326da25 100644 --- a/tests/case/knowledge/foundation/index-speed-thread-budget.test.ts +++ b/tests/case/knowledge/foundation/index-speed-thread-budget.test.ts @@ -1,31 +1,75 @@ -import { describe, expect, it } from 'vitest' -import { resolveOrtThreadBudget } from '../../../../../src/main/services/vector/indexSpeed' +import { describe, it } from 'vitest' +import { resolveOrtThreadBudget } from '../../../../src/main/services/vector/indexSpeed' +import { runBackendCase } from '../../../harness/backend/run-backend-case' -describe('index speed thread budget', () => { - it('maps slow and fast ORT budgets for 4, 8 and 16 logical cores', () => { - expect(resolveOrtThreadBudget('slow', 4)).toEqual({ - intraOpNumThreads: 2, - allowSpinning: false - }) - expect(resolveOrtThreadBudget('slow', 8)).toEqual({ - intraOpNumThreads: 4, - allowSpinning: false - }) - expect(resolveOrtThreadBudget('slow', 16)).toEqual({ - intraOpNumThreads: 8, - allowSpinning: false - }) - expect(resolveOrtThreadBudget('fast', 4)).toEqual({ - intraOpNumThreads: 3, - allowSpinning: true - }) - expect(resolveOrtThreadBudget('fast', 8)).toEqual({ - intraOpNumThreads: 7, - allowSpinning: true - }) - expect(resolveOrtThreadBudget('fast', 16)).toEqual({ - intraOpNumThreads: 14, - allowSpinning: true +describe('knowledge foundation', () => { + it('index-speed-thread-budget', async () => { + await runBackendCase({ + caseId: 'index-speed-thread-budget', + harnessMode: 'tool-real', + domain: 'knowledge', + suite: 'foundation', + description: '慢速/高速按逻辑核数映射 ORT 线程预算与 spinning。', + fixtures: [], + runMode: 'single', + execute: async () => ({ + slow4: resolveOrtThreadBudget('slow', 4), + slow8: resolveOrtThreadBudget('slow', 8), + slow16: resolveOrtThreadBudget('slow', 16), + fast4: resolveOrtThreadBudget('fast', 4), + fast8: resolveOrtThreadBudget('fast', 8), + fast16: resolveOrtThreadBudget('fast', 16) + }), + assert: async (result, runtime) => { + runtime.assertThat( + result.slow4.intraOpNumThreads === 2 && result.slow4.allowSpinning === false, + { + name: 'slow-4-cores', + message: '4 核慢速应为 2 线程且关闭 spinning。', + data: result.slow4 + } + ) + runtime.assertThat( + result.slow8.intraOpNumThreads === 4 && result.slow8.allowSpinning === false, + { + name: 'slow-8-cores', + message: '8 核慢速应为 4 线程且关闭 spinning。', + data: result.slow8 + } + ) + runtime.assertThat( + result.slow16.intraOpNumThreads === 8 && result.slow16.allowSpinning === false, + { + name: 'slow-16-cores', + message: '16 核慢速应为 8 线程且关闭 spinning。', + data: result.slow16 + } + ) + runtime.assertThat( + result.fast4.intraOpNumThreads === 3 && result.fast4.allowSpinning === true, + { + name: 'fast-4-cores', + message: '4 核高速应为 3 线程且开启 spinning。', + data: result.fast4 + } + ) + runtime.assertThat( + result.fast8.intraOpNumThreads === 7 && result.fast8.allowSpinning === true, + { + name: 'fast-8-cores', + message: '8 核高速应为 7 线程且开启 spinning。', + data: result.fast8 + } + ) + runtime.assertThat( + result.fast16.intraOpNumThreads === 14 && result.fast16.allowSpinning === true, + { + name: 'fast-16-cores', + message: '16 核高速应为 14 线程且开启 spinning。', + data: result.fast16 + } + ) + } }) }) }) diff --git a/tests/case/knowledge/foundation/kb-indexing-blocks-search.test.ts b/tests/case/knowledge/foundation/kb-indexing-blocks-search.test.ts index a96f1aa0..38f18fa7 100644 --- a/tests/case/knowledge/foundation/kb-indexing-blocks-search.test.ts +++ b/tests/case/knowledge/foundation/kb-indexing-blocks-search.test.ts @@ -1,10 +1,11 @@ -import { describe, expect, it } from 'vitest' -import { buildModelToolResultDelivery } from '../../../../../src/main/services/ai/human/shared/modelToolResult' +import { describe, it } from 'vitest' +import { buildModelToolResultDelivery } from '../../../../src/main/services/ai/human/shared/modelToolResult' import { buildKbCapabilityTools, KB_INDEXING_IN_PROGRESS_ERROR -} from '../../../../../src/main/services/ai/human/tools/kb/retrievalTools' -import type { AiRunContext, KnowledgeRetrievalContext } from '../../../../../src/main/services/ai/types' +} from '../../../../src/main/services/ai/human/tools/kb/retrievalTools' +import type { AiRunContext, KnowledgeRetrievalContext } from '../../../../src/main/services/ai/types' +import { runBackendCase } from '../../../harness/backend/run-backend-case' const MODEL_FACING_INDEXING_ERROR = `Error: ${KB_INDEXING_IN_PROGRESS_ERROR}` @@ -19,41 +20,81 @@ const invokeTool = async ( return await tool.invoke(args) } -describe('ai unit kb indexing blocks search', () => { - it('fails kb_search and kb_list_documents while a prepare index job is running', async () => { - const ctx = { - runId: 'run-kb-indexing', - checkCancelled: () => undefined, - sendEvent: () => undefined, - documentsStore: {} as KnowledgeRetrievalContext['documentsStore'], - embeddingService: {} as KnowledgeRetrievalContext['embeddingService'], - assertKbSearchAvailable: async () => { - throw new Error(KB_INDEXING_IN_PROGRESS_ERROR) - } - } as KnowledgeRetrievalContext +describe('knowledge foundation', () => { + it('kb-indexing-blocks-search', async () => { + await runBackendCase({ + caseId: 'kb-indexing-blocks-search', + harnessMode: 'tool-real', + domain: 'knowledge', + suite: 'foundation', + description: 'prepare 索引进行中时 kb_search / kb_list_documents 失败并投递模型可读错误。', + fixtures: [], + runMode: 'single', + execute: async () => { + const ctx = { + runId: 'run-kb-indexing', + checkCancelled: () => undefined, + sendEvent: () => undefined, + documentsStore: {} as KnowledgeRetrievalContext['documentsStore'], + embeddingService: {} as KnowledgeRetrievalContext['embeddingService'], + assertKbSearchAvailable: async () => { + throw new Error(KB_INDEXING_IN_PROGRESS_ERROR) + } + } as KnowledgeRetrievalContext - await expect(invokeTool(ctx, 'kb_search', { queryText: '合同' })).rejects.toThrow( - KB_INDEXING_IN_PROGRESS_ERROR - ) - await expect(invokeTool(ctx, 'kb_list_documents', {})).rejects.toThrow( - KB_INDEXING_IN_PROGRESS_ERROR - ) - }) + const searchError = await invokeTool(ctx, 'kb_search', { queryText: '合同' }).then( + () => '', + (error: unknown) => (error instanceof Error ? error.message : String(error)) + ) + const listError = await invokeTool(ctx, 'kb_list_documents', {}).then( + () => '', + (error: unknown) => (error instanceof Error ? error.message : String(error)) + ) - it('delivers the indexing error into the model-facing kb tool result', async () => { - for (const toolName of ['kb_search', 'kb_list_documents'] as const) { - const delivery = await buildModelToolResultDelivery({ - ctx: { platform: 'win32' } as AiRunContext, - toolCallId: `call-${toolName}`, - args: toolName === 'kb_search' ? { queryText: '合同' } : {}, - result: { - toolName, - ok: false, - data: { error: KB_INDEXING_IN_PROGRESS_ERROR } + const deliveries = [] + for (const toolName of ['kb_search', 'kb_list_documents'] as const) { + const delivery = await buildModelToolResultDelivery({ + ctx: { platform: 'win32' } as AiRunContext, + toolCallId: `call-${toolName}`, + args: toolName === 'kb_search' ? { queryText: '合同' } : {}, + result: { + toolName, + ok: false, + data: { error: KB_INDEXING_IN_PROGRESS_ERROR } + } + }) + deliveries.push({ + toolName, + ok: delivery.ok, + content: delivery.content + }) } - }) - expect(delivery.ok).toBe(false) - expect(delivery.content).toBe(MODEL_FACING_INDEXING_ERROR) - } + + return { searchError, listError, deliveries } + }, + assert: async (result, runtime) => { + runtime.assertThat( + result.searchError === KB_INDEXING_IN_PROGRESS_ERROR && + result.listError === KB_INDEXING_IN_PROGRESS_ERROR, + { + name: 'kb-tools-fail-while-indexing', + message: + '更新索引执行中 kb_search / kb_list_documents 应失败，对齐 SET-WORKSPACE-004-AC9、SET-WEB-004-AC8。', + data: result + } + ) + runtime.assertThat( + result.deliveries.length === 2 && + result.deliveries.every( + (item) => item.ok === false && item.content === MODEL_FACING_INDEXING_ERROR + ), + { + name: 'indexing-error-is-model-facing', + message: '索引中错误应投递为模型可读的 Error: 正在更新知识库索引，请完成后再检索。', + data: result.deliveries + } + ) + } + }) }) }) diff --git a/tests/case/knowledge/foundation/prepare-workspaces-sequentially.test.ts b/tests/case/knowledge/foundation/prepare-workspaces-sequentially.test.ts index 480f9a0d..0de3ad61 100644 --- a/tests/case/knowledge/foundation/prepare-workspaces-sequentially.test.ts +++ b/tests/case/knowledge/foundation/prepare-workspaces-sequentially.test.ts @@ -1,24 +1,42 @@ -import { describe, expect, it } from 'vitest' -import { prepareWorkspacesSequentially } from '../../../../../src/shared/prepareWorkspacesSequentially' +import { describe, it } from 'vitest' +import { prepareWorkspacesSequentially } from '../../../../src/shared/prepareWorkspacesSequentially' +import { runBackendCase } from '../../../harness/backend/run-backend-case' -describe('prepareWorkspacesSequentially', () => { - it('starts the next workspace only after the previous prepare settles', async () => { - const events: string[] = [] - const results = await prepareWorkspacesSequentially(['ws-a', 'ws-b', 'ws-c'], async (workspaceId) => { - events.push(`start:${workspaceId}`) - await Promise.resolve() - events.push(`end:${workspaceId}`) - return workspaceId +describe('knowledge foundation', () => { + it('prepare-workspaces-sequentially', async () => { + await runBackendCase({ + caseId: 'prepare-workspaces-sequentially', + harnessMode: 'tool-real', + domain: 'knowledge', + suite: 'foundation', + description: '多地址更新索引按顺序等待前一个完成再启动下一个。', + fixtures: [], + runMode: 'single', + execute: async () => { + const events: string[] = [] + const results = await prepareWorkspacesSequentially( + ['ws-a', 'ws-b', 'ws-c'], + async (workspaceId) => { + events.push(`start:${workspaceId}`) + await Promise.resolve() + events.push(`end:${workspaceId}`) + return workspaceId + } + ) + return { results, events } + }, + assert: async (result, runtime) => { + runtime.assertThat( + result.results.join(',') === 'ws-a,ws-b,ws-c' && + result.events.join(',') === + 'start:ws-a,end:ws-a,start:ws-b,end:ws-b,start:ws-c,end:ws-c', + { + name: 'prepare-is-sequential', + message: '多地址更新索引必须等前一个完成后再启动下一个，对齐 SET-WORKSPACE-004-AC11。', + data: result + } + ) + } }) - - expect(results).toEqual(['ws-a', 'ws-b', 'ws-c']) - expect(events).toEqual([ - 'start:ws-a', - 'end:ws-a', - 'start:ws-b', - 'end:ws-b', - 'start:ws-c', - 'end:ws-c' - ]) }) }) diff --git a/tests/case/knowledge/foundation/workspace-index-job-store-contract.test.ts b/tests/case/knowledge/foundation/workspace-index-job-store-contract.test.ts index c4194442..60d13622 100644 --- a/tests/case/knowledge/foundation/workspace-index-job-store-contract.test.ts +++ b/tests/case/knowledge/foundation/workspace-index-job-store-contract.test.ts @@ -1,6 +1,6 @@ import { describe, it } from 'vitest' -import type { WorkspaceIndexJobRecord } from '../../../../../src/shared/types' -import { runBackendCase } from '../../../../harness/backend/run-backend-case' +import type { WorkspaceIndexJobRecord } from '../../../../src/shared/types' +import { runBackendCase } from '../../../harness/backend/run-backend-case' const NOW = 1_716_200_000_000 @@ -32,9 +32,8 @@ describe('unit workspace backend', () => { await runBackendCase({ caseId: 'workspace-index-job-store-contract', harnessMode: 'tool-real', - domain: 'unit', - suite: 'knowledge', - groups: ['indexing'], + domain: 'knowledge', + suite: 'foundation', description: 'workspace_index_job store 支持创建、active 查询、阶段进度更新和失败收敛。', fixtures: [], runMode: 'single', diff --git a/tests/case/knowledge/foundation/workspace-index-worker-contract.test.ts b/tests/case/knowledge/foundation/workspace-index-worker-contract.test.ts index 886b781b..20c5ec3d 100644 --- a/tests/case/knowledge/foundation/workspace-index-worker-contract.test.ts +++ b/tests/case/knowledge/foundation/workspace-index-worker-contract.test.ts @@ -6,20 +6,20 @@ import { WORKSPACE_EMBEDDING_BATCH_CHUNK_COUNT, WORKSPACE_EMBEDDING_BATCH_MAX_CHARS, WORKSPACE_EMBEDDING_CHUNK_MAX_CHARS -} from '../../../../../src/main/services/workspace/constants' +} from '../../../../src/main/services/workspace/constants' import { IngestionService, isSupportedLocalFilePath -} from '../../../../../src/main/services/ingestion/loader' +} from '../../../../src/main/services/ingestion/loader' import { buildWorkspaceKnowledgeSplitSnapshot, computeWorkspaceFileContentHash, createWorkspaceEmbeddingBatches -} from '../../../../../src/main/services/workspace/utils/knowledge' -import { handleWorkspacePrepareWorkerRequest } from '../../../../../src/main/services/workspace/worker/workspacePrepareWorker' -import { WorkspacePrepareWorkerClient } from '../../../../../src/main/services/workspace/worker/workspacePrepareWorkerClient' -import type { WorkspacePrepareWorkerRequest } from '../../../../../src/main/services/workspace/types' -import { runBackendCase } from '../../../../harness/backend/run-backend-case' +} from '../../../../src/main/services/workspace/utils/knowledge' +import { handleWorkspacePrepareWorkerRequest } from '../../../../src/main/services/workspace/worker/workspacePrepareWorker' +import { WorkspacePrepareWorkerClient } from '../../../../src/main/services/workspace/worker/workspacePrepareWorkerClient' +import type { WorkspacePrepareWorkerRequest } from '../../../../src/main/services/workspace/types' +import { runBackendCase } from '../../../harness/backend/run-backend-case' const createChunk = (length: number, page: number): { text: string; page: number } => ({ text: 'x'.repeat(length), @@ -31,9 +31,8 @@ describe('unit workspace backend', () => { await runBackendCase({ caseId: 'workspace-index-worker-contract', harnessMode: 'tool-real', - domain: 'unit', - suite: 'knowledge', - groups: ['indexing'], + domain: 'knowledge', + suite: 'foundation', description: 'workspace 文件索引 worker 协议与 embedding batch 限制保持稳定。', fixtures: ['tests/fixtures/workspace'], runMode: 'single', diff --git a/tests/case/knowledge/foundation/workspace-interrupted-job-recovery-contract.test.ts b/tests/case/knowledge/foundation/workspace-interrupted-job-recovery-contract.test.ts index 274b8fcf..79afffaa 100644 --- a/tests/case/knowledge/foundation/workspace-interrupted-job-recovery-contract.test.ts +++ b/tests/case/knowledge/foundation/workspace-interrupted-job-recovery-contract.test.ts @@ -4,8 +4,8 @@ import type { WorkspaceFileIndexRecord, WorkspaceIndexJobRecord, WorkspaceSource -} from '../../../../../src/shared/types' -import { runBackendCase } from '../../../../harness/backend/run-backend-case' +} from '../../../../src/shared/types' +import { runBackendCase } from '../../../harness/backend/run-backend-case' const NOW = 1_716_200_000_000 const WORKSPACE: WorkspaceSource = { @@ -69,9 +69,8 @@ describe('unit workspace backend', () => { await runBackendCase({ caseId: 'workspace-interrupted-job-recovery-contract', harnessMode: 'tool-real', - domain: 'unit', - suite: 'knowledge', - groups: ['indexing'], + domain: 'knowledge', + suite: 'foundation', description: '启动恢复会把遗留 running job 和条目级执行中状态收敛为失败。', fixtures: [], runMode: 'single', diff --git a/tests/case/knowledge/foundation/workspace-scan-contract.test.ts b/tests/case/knowledge/foundation/workspace-scan-contract.test.ts index 64a9fcfb..cc04f1c8 100644 --- a/tests/case/knowledge/foundation/workspace-scan-contract.test.ts +++ b/tests/case/knowledge/foundation/workspace-scan-contract.test.ts @@ -1,16 +1,15 @@ import { describe, it } from 'vitest' import path from 'node:path' import { mkdir, writeFile } from 'node:fs/promises' -import { runBackendCase } from '../../../../harness/backend/run-backend-case' +import { runBackendCase } from '../../../harness/backend/run-backend-case' describe('unit workspace backend', () => { it('workspace-scan-contract', async () => { await runBackendCase({ caseId: 'workspace-scan-contract', harnessMode: 'tool-real', - domain: 'unit', - suite: 'knowledge', - groups: ['indexing'], + domain: 'knowledge', + suite: 'foundation', description: '工作空间扫描服务生成文件索引记录，并由派生摘要支撑页面刷新。', fixtures: ['tests/fixtures/workspace'], runMode: 'single', diff --git a/tests/case/knowledge/.gitkeep b/tests/case/knowledge/optimization/.gitkeep similarity index 100% rename from tests/case/knowledge/.gitkeep rename to tests/case/knowledge/optimization/.gitkeep diff --git a/tests/case/settings/workspace/files/workspace-prepare-refresh.e2e.test.ts b/tests/case/settings/workspace/files/workspace-prepare-refresh.e2e.test.ts index 089dbbe5..a8719cb2 100644 --- a/tests/case/settings/workspace/files/workspace-prepare-refresh.e2e.test.ts +++ b/tests/case/settings/workspace/files/workspace-prepare-refresh.e2e.test.ts @@ -38,13 +38,15 @@ describe('settings workspace e2e', () => { const workspaceId = workspace.workspace!.id let prepareProgressVisible = false let buttonsDisabled = false + let defaultSlowSelected = false + let knowledgeIndexSpeed = '' let directoryCallBaseline = 0 let directoryCallsWhileRunning = -1 let directoryCallsAfterCompletion = -1 await runtime.step({ name: 'prepare-workspace-through-ui', - message: '通过 UI 点击更新索引，并确认执行中进度块和按钮禁用状态。', + message: '通过 UI 先切换索引速度，再点击更新索引，并确认执行中进度块和按钮状态。', run: async () => { await navigateToSettingsSection(page, 'workspace') await getSettingsLocator(page, importSelectors.workspaceQuickPrepareButton).waitFor({ @@ -53,6 +55,29 @@ describe('settings workspace e2e', () => { await getSettingsLocator(page, importSelectors.workspaceTree).waitFor({ state: 'visible' }) + const slowSpeed = getSettingsLocator(page, importSelectors.workspaceIndexSpeedSlow) + const fastSpeed = getSettingsLocator(page, importSelectors.workspaceIndexSpeedFast) + await slowSpeed.waitFor({ state: 'visible' }) + await page.waitForFunction( + (testId) => { + const element = document.querySelector(`[data-testid="${testId}"]`) + return element instanceof HTMLButtonElement && !element.disabled + }, + importSelectors.workspaceIndexSpeedSlow + ) + defaultSlowSelected = await slowSpeed.evaluate((element) => + element.classList.contains('ant-btn-primary') + ) + await fastSpeed.click() + await page.waitForFunction( + (testId) => + document.querySelector(`[data-testid="${testId}"]`)?.classList.contains( + 'ant-btn-primary' + ) === true, + importSelectors.workspaceIndexSpeedFast + ) + knowledgeIndexSpeed = (await page.evaluate(() => window.api.appSettingsGet())) + .knowledgeIndexSpeed directoryCallBaseline = await page.evaluate( () => (window as typeof window & { @@ -102,6 +127,8 @@ describe('settings workspace e2e', () => { return { prepareProgressVisible, buttonsDisabled, + defaultSlowSelected, + knowledgeIndexSpeed, directoryCallsWhileRunning, directoryCallsAfterCompletion, summary: result.tree.summary, @@ -109,6 +136,16 @@ describe('settings workspace e2e', () => { } }, assert: async (result, runtime) => { + await runtime.assertThat( + result.defaultSlowSelected === true && result.knowledgeIndexSpeed === 'fast', + { + name: 'prepare-runs-after-switching-index-speed', + message: + '默认慢速，切换高速并保存后再更新索引，对齐 SET-WORKSPACE-009-AC1；本次索引走切换后的档位。', + data: result + } + ) + await runtime.assertThat(result.prepareProgressVisible && !result.buttonsDisabled, { name: 'prepare-progress-is-visible', message: diff --git a/tests/harness/backend/test-types.ts b/tests/harness/backend/test-types.ts index 477afb0b..5feb269a 100644 --- a/tests/harness/backend/test-types.ts +++ b/tests/harness/backend/test-types.ts @@ -221,7 +221,7 @@ export type TestCaseDefinition = { caseId: string harnessMode: BackendHarnessMode testType?: Extract - domain: Extract + domain: Extract suite: string groups?: readonly string[] description: string diff --git a/tests/harness/core/run-log-contract.ts b/tests/harness/core/run-log-contract.ts index 5328b861..cd465c79 100644 --- a/tests/harness/core/run-log-contract.ts +++ b/tests/harness/core/run-log-contract.ts @@ -2,7 +2,7 @@ import type { JsonValue } from '../../../src/shared/types' export const TEST_LOG_SCHEMA_VERSION = 5 as const -export type TestDomain = 'ai' | 'settings' | 'workbench' | 'unit' | 'showcase' +export type TestDomain = 'ai' | 'knowledge' | 'settings' | 'workbench' | 'unit' | 'showcase' export type TestType = 'backend' | 'e2e' export type TestRunStatus = 'running' | 'passed' | 'failed' export type TestLogStatus = 'info' | 'ok' | 'failed' | 'skipped'