refactor: update knowledge test plans and related documentation

- Enhanced `knowledge-test.md` to clarify the scope and objectives of the knowledge base testing, including the introduction of `foundation` and `optimization` categories.
- Updated references in `AGENTS.md` to reflect the changes in knowledge testing documentation.
- Improved test case structure in `index-speed-thread-budget.test.ts`, `kb-indexing-blocks-search.test.ts`, and `prepare-workspaces-sequentially.test.ts` to utilize `runBackendCase` for better consistency.
- Adjusted assertions in various test cases to ensure accurate validation of expected outcomes during indexing operations.
- Modified `run-log-contract.ts` and `test-types.ts` to include 'knowledge' as a valid domain for test cases.
- Updated `workspace-prepare-refresh.e2e.test.ts` to validate index speed switching functionality during workspace preparation.
This commit is contained in:
hy committed 2026-09-20 12:49:43 +08:00
1 parent a5fce348b2
commit 4fab66e1ea
16 files changed
+353 -129

No files matched your search

+3 -3
View File
@@ -24,7 +24,7 @@
### 1.4 测试相关
- `doc/test-plan/ai-test.md`:涉及 AI 回归测试范围、live case 设计或 suite 覆盖维护时阅读。
- `doc/test-plan/knowledge-test.md`:涉及知识库专项时阅读。当前尚未登记 case。
- `doc/test-plan/knowledge-test.md`:涉及知识库专项(foundation 功能与 optimization 效果)时阅读。
- `doc/test-plan/settings-test.md`:涉及 Settings 测试范围或测试边界时阅读。
- `doc/test-plan/workbench-test.md`:涉及 Workbench 回归测试范围、真实 e2e case 设计、发送前置条件验证或 suite 覆盖维护时阅读。
- `doc/test.md`:涉及测试体系、UI E2E 回归规则、runner 或回归测试流程时阅读。
@@ -61,7 +61,7 @@
│ │ └── workbench.md # Workbench 产品需求(含定时任务)
│ ├── test-plan/ # 专项测试计划
│ │ ├── ai-test.md # AI 回归测试计划
│ │ ├── knowledge-test.md # 知识库专项测试计划(当前无 case)
│ │ ├── knowledge-test.md # 知识库专项测试计划
│ │ ├── settings-test.md # Settings 测试计划
│ │ └── workbench-test.md # Workbench 测试计划
│ └── tidy.md # 周期性文档、测试与代码整理流程
@@ -78,7 +78,7 @@
├── tests/ # 自动化测试
│ ├── case/ # 测试用例
│ │ ├── ai/ # AI 测试用例
│ │ ├── knowledge/ # 知识库专项测试(当前无 case)
│ │ ├── knowledge/ # 知识库专项测试
│ │ ├── showcase/ # 独立官网样例采集
│ │ ├── settings/ # Settings 测试用例
│ │ ├── unit/ # backend 单元测试
+1 -1
View File
@@ -434,7 +434,7 @@ DOCX tool 回归以 7 个公开工具、story 内容树、稳定 `nodeId`、segm
- Why:KB tool family 必须显式覆盖每个回归工具。
- How:对已 seed 的文件与网页数据分别执行三种检索,并覆盖 `documentIds` 缩圈。
- Oracle:三种检索都返回稳定 `items`;`documentIds` 缩圈语义成立(含全部不存在时返回空);文件命中保留 `sourceType=file` 与 `filePath`,网页命中保留 `sourceType=web` 与 `sourceUrl`,供最终引用生成可追溯地址。
- Not:不比较召回优劣;不测人造跨 workspace 脏数据隔离。设置页「更新索引」期间的拦截由 unit `kb-indexing-blocks-search` 覆盖 `SET-WORKSPACE-004-AC9`、`SET-WEB-004-AC8`(工具失败,模型侧结果为 `Error: 正在更新知识库索引,请完成后再检索。`);「检查更新」不拦截。
- Not:不比较召回优劣;
#### terminal 类别
+95 -7
View File
@@ -2,22 +2,110 @@
## 1. 总体目标
- 本计划定义知识库专项回归,对标 `ai-test`:召回、检索、索引质量与后续专项优化。
- 本计划定义知识库专项回归,对标 `ai-test`,全部是 backend。
- 通用测试、runner 与断言规则遵循 `doc/test.md`。
- 正式 case 放在 `tests/case/knowledge/**`,全部是 backend。
- 正式 case 放在 `tests/case/knowledge/<suite>/**`。
当前没有登记 case。专项 suite 与 case 在后续做召回、检索或索引优化时再设计。
本计划覆盖两类:
1. `foundation`:知识库功能(扫描、建索引、job、索引中拦截检索、索引速度线程预算、多地址顺序更新)。
2. `optimization`:召回率、检索与索引效果。当前无 case,后续做优化时再设计。
边界:
- 不替代 Settings「更新索引」产品 E2E。
- 不替代 Settings「更新索引」产品 E2E。E2E 在页面上改慢/高速后再走真实更新索引;本计划只测 backend 公式与服务契约。
- 不替代 `ai-test` 中的 kb tool / live。
- 不登记 unit。函数级契约仍在 `tests/case/unit/**`(含 `tests/case/unit/knowledge/**`)。
- 不登记 unit。函数级补足仍在 `tests/case/unit/knowledge/**`。
## 2. Suite 设计
暂不定义。
### 2.1 `foundation`
## 3. 回归清单
- 目标:守住知识库可回归的功能主链。
- 约束:走生产 store / job / 扫描 / worker;索引速度 case 只断言 ORT 线程预算,不测 UI、不测真实 CPU 占用。
- 目录:`tests/case/knowledge/foundation/<case>.test.ts`
### 2.2 `optimization`
- 目标:召回率、检索与索引效果评测。
- 当前无登记 case。
## 3. 领域规则
- 全部是 backend,禁止导入 Renderer。
- 线程预算以逻辑核 `N` 为准(Mac / Windows 相同):
- 慢速:`max(1, floor(N/2))`,关闭 spinning
- 高速:`max(2, N - reserve)`,`reserve = N≤8 ? 1 : 2`
- 多文件地址更新索引必须顺序 kickoff。
- 不在 `foundation` 比较召回优劣。
## 4. 回归清单
### 4.1 `foundation`
#### `workspace-scan-contract`
- 测试类型:`backend`
- What:扫描真实目录后生成文件索引记录,并由派生摘要支撑刷新。
- Why:检查更新的 backend 主链;漏扫或摘要错会导致页面可索引数失真。
- How:隔离 workspace fixture,写入可见/隐藏目录后调用扫描服务并读取索引记录与摘要。
- Oracle:支持文件进入索引记录;忽略目录不入库;摘要与扫描进度可支撑页面刷新。
- Not:不验证 Settings 按钮与进度块 UI。
#### `workspace-index-job-store-contract`
- 测试类型:`backend`
- What:`workspace_index_job` 支持创建、active 查询、阶段进度更新和失败收敛。
- Why:更新索引的进度恢复与中断收敛依赖 job store。
- How:真实 job store 写入 running/failed 记录并查询 active、进度与批量失败。
- Oracle:active 取最新 running;进度可持久化读取;running job 可批量失败,对齐 `SET-WORKSPACE-004-AC3`、`SET-WORKSPACE-004-AC4`、`SET-WORKSPACE-004-AC5`。
- Not:不验证页面进度块文案。
#### `workspace-index-worker-contract`
- 测试类型:`backend`
- What:文件索引 worker 协议与 embedding batch 限制保持稳定。
- Why:worker 输入输出或切批失真会让更新索引静默丢文件或打爆嵌入。
- How:真实 worker 协议与切批函数覆盖支持文件与超限文本。
- Oracle:协议可执行;batch 遵守条数与字符上限。
- Not:不验证向量召回质量。
#### `workspace-interrupted-job-recovery-contract`
- 测试类型:`backend`
- What:启动恢复把遗留 running job 和条目级执行中状态收敛为失败。
- Why:进程被杀后必须能退出“执行中”,否则无法再次更新索引。
- How:写入遗留 running job 与执行中文件状态后走启动恢复。
- Oracle:job 与条目收敛为失败,对齐 `SET-WORKSPACE-004-AC5`。
- Not:不模拟操作系统强杀 UI。
#### `kb-indexing-blocks-search`
- 测试类型:`backend`
- What:prepare 索引进行中时 `kb_search` / `kb_list_documents` 失败,并投递模型可读错误。
- Why:更新索引期间检索必须明确失败,避免半库结果。
- How:构造索引中上下文调用两个工具,并检查 model-facing 投递文案。
- Oracle:两工具抛出正在更新知识库错误;模型侧结果为 `Error: 正在更新知识库索引,请完成后再检索。`,对齐 `SET-WORKSPACE-004-AC9`、`SET-WEB-004-AC8`。
- Not:不验证检查更新是否拦截(检查更新不拦截);不替代 `ai-test` kb tool 召回契约。
#### `index-speed-thread-budget`
- 测试类型:`backend`
- What:慢速/高速按逻辑核数映射 ORT `intraOpNumThreads` 与 spinning。
- Why:索引速度档位的 backend 效果是线程预算,不是页面文案。
- How:对 4/8/16 核分别计算慢速与高速预算。
- Oracle:慢速 2/4/8 且 `allowSpinning=false`;高速 3/7/14 且 `allowSpinning=true`。
- Not:不测 Settings 切换 UI、不问号文案、不测真实 CPU 占用。页面改档后再更新索引由 `workspace-prepare-refresh` E2E 覆盖。
#### `prepare-workspaces-sequentially`
- 测试类型:`backend`
- What:多地址更新索引按顺序等待前一个完成再启动下一个。
- Why:并行 kickoff 会叠满嵌入线程,和慢/高速档位冲突。
- How:对三个 workspace id 顺序调用 prepare,记录 start/end。
- Oracle:事件顺序为每个地址 start→end 后再开始下一个,对齐 `SET-WORKSPACE-004-AC11`。
- Not:不验证 Settings 多地址按钮 UI。
### 4.2 `optimization`
无。
+5 -5
View File
@@ -19,7 +19,7 @@ Settings 顶层 suite 对齐 PRD 模块:
- `model` 验证大模型可用服务配置的加载、创建、编辑、删除,以及预设服务商与自定义配置的类型、锁定和去重语义。
- `general` 验证 AI 运行系统通知开关的加载、保存和关闭后的静默行为,以及错误日志目录入口;系统通知中心的真实视觉展示按 macOS 与 Windows 发布前手动验证。
- `workspace` 验证文件地址森林及其并列的全局网页资料:添加/删除地址、包含关系、树顶层节点、合计索引进度、网页全局列表。索引速度档位的线程公式与多地址顺序由 unit 覆盖,不在本计划重复测算力。`SET-WORKSPACE-009` 两 tab 共用全局档位;默认慢速与问号说明不进日常 E2E。
- `workspace` 验证文件地址森林及其并列的全局网页资料:添加/删除地址、包含关系、树顶层节点、合计索引进度、网页全局列表。`SET-WORKSPACE-009` 由 `workspace-prepare-refresh` 在页面切换档位后走真实更新索引覆盖;线程公式与多地址顺序由 knowledge `foundation` 覆盖。问号文案与网页 tab 同步不单独测。
- `vector` 属于 Settings 产品域,页面路由为 `/settings/vector`(`/model-download` 仅为历史重定向),测试目录与 runner 仍统一归入 `tests/case/settings/vector/**`。
- `workspace` 是 Settings 下的 suite,按 `core / files / web` 组织,语义为“文件地址 + 文件 tab + 全局网页 tab”,不是具名容器。
- `workspace` 相关回归测试默认复用 `tests/fixtures/workspace` 作为共享输入目录。
@@ -122,9 +122,9 @@ Settings 顶层 suite 对齐 PRD 模块:
- 测试类型:`e2e`
- What:点击“更新索引”后,文件状态会刷新为已索引并同步更新摘要。
- Why:“索引已经可用”的核心用户反馈链路。
- How:复制共享 workspace 输入的隔离副本,通过 Settings UI 添加为文件地址;随后点击更新索引,读取刷新后的文件树和全部支持文件状态。共享 fixture 含 `.doc`、`docx`、`pdf`、`md`、`txt`、`xlsx`、`xlsm` 等全部受支持类型,其中 excel 只以元数据方式索引进库。
- Oracle:更新索引过程中出现持久化总进度,且检查更新/更新索引按钮保持可用,对齐 `SET-WORKSPACE-004-AC3`;运行期间目录读取次数为 0,终态只刷新一次且不因逐文件完成而闪烁或折叠,对齐 `SET-WORKSPACE-004-AC7`;共享 fixture 中全部支持文件真实进入处理范围,对齐 `SET-WORKSPACE-004-AC1`;更新后所有支持文件收敛为 `indexed / failed`,摘要和节点状态同步刷新,对齐 `SET-WORKSPACE-004-AC2`。
- Not:不验证向量内容质量和搜索召回结果。更新索引执行中 Workbench `kb_search` / `kb_list_documents` 返回错误、检查更新不拦截,由 unit `kb-indexing-blocks-search` 覆盖 `SET-WORKSPACE-004-AC9`。多地址顺序执行由 unit `prepare-workspaces-sequentially` 覆盖 `SET-WORKSPACE-004-AC11`。索引速度档位的线程预算由 unit `index-speed-thread-budget` 覆盖。
- How:复制共享 workspace 输入的隔离副本,通过 Settings UI 添加为文件地址;确认默认慢速后切换为高速并保存,再点击更新索引,读取刷新后的文件树和全部支持文件状态。共享 fixture 含 `.doc`、`docx`、`pdf`、`md`、`txt`、`xlsx`、`xlsm` 等全部受支持类型,其中 excel 只以元数据方式索引进库。
- Oracle:默认慢速后可切换高速,本次更新索引在切换后的档位上执行,对齐 `SET-WORKSPACE-009-AC1`;更新索引过程中出现持久化总进度,且检查更新/更新索引按钮保持可用,对齐 `SET-WORKSPACE-004-AC3`;运行期间目录读取次数为 0,终态只刷新一次且不因逐文件完成而闪烁或折叠,对齐 `SET-WORKSPACE-004-AC7`;共享 fixture 中全部支持文件真实进入处理范围,对齐 `SET-WORKSPACE-004-AC1`;更新后所有支持文件收敛为 `indexed / failed`,摘要和节点状态同步刷新,对齐 `SET-WORKSPACE-004-AC2`。
- Not:不验证向量内容质量和搜索召回结果,也不测 CPU 占用公式。更新索引执行中 Workbench `kb_search` / `kb_list_documents` 返回错误、检查更新不拦截,由 knowledge `kb-indexing-blocks-search` 覆盖 `SET-WORKSPACE-004-AC9`。多地址顺序执行由 knowledge `prepare-workspaces-sequentially` 覆盖 `SET-WORKSPACE-004-AC11`。线程预算由 knowledge `index-speed-thread-budget` 覆盖。问号说明与网页 tab 同步不单独测。
#### `workspace-prepare-failed-retry`
@@ -180,7 +180,7 @@ Settings 顶层 suite 对齐 PRD 模块:
- Why:网页索引是“爬取 + 建索引”两阶段链路,失败类型必须可区分。
- How:预置一条成功网页和一条带失败标记的网页,先执行“检查更新”,再执行“更新索引”。
- Oracle:网页列表、摘要、失败数量和操作区初始化对齐 `SET-WEB-001-AC1`、`SET-WEB-001-AC2`;检查进度和完成状态对齐 `SET-WEB-003-AC1`、`SET-WEB-003-AC2`;更新时网页进入抓取/索引状态,对齐 `SET-WEB-004-AC1`;成功与失败分别收敛,对齐 `SET-WEB-004-AC2`、`SET-WEB-004-AC3`;分阶段进度对齐 `SET-WEB-004-AC4`。
- Not:不验证真实网页正文切块、embedding 写入质量和索引失败以外的网络细节;向量模型未安装时「更新索引」禁用与说明(`SET-WEB-004-AC7`,文件侧姊妹 `SET-WORKSPACE-004-AC6`)标为手动 / 暂缓,不进日常自动化回归。网页更新索引执行中的 kb 拦截由 unit `kb-indexing-blocks-search` 覆盖 `SET-WEB-004-AC8`。
- Not:不验证真实网页正文切块、embedding 写入质量和索引失败以外的网络细节;向量模型未安装时「更新索引」禁用与说明(`SET-WEB-004-AC7`,文件侧姊妹 `SET-WORKSPACE-004-AC6`)标为手动 / 暂缓,不进日常自动化回归。网页更新索引执行中的 kb 拦截由 knowledge `kb-indexing-blocks-search` 覆盖 `SET-WEB-004-AC8`。
#### `web-prepare-progress-restore`
+2 -2
View File
@@ -4,7 +4,7 @@
- 本文是 NCurator-Desktop 的唯一测试总规范。
- 各业务域的 suite、case 和领域规则只在对应 `doc/test-plan/**` 中维护,本文不重复登记 case。
- `ai-test` 专门覆盖 AI 架构及其工具,全部是 backend;`knowledge-test` 专门覆盖知识库的功能测试(如搜索和索引等),优化设计测试(如召回率、检索与索引优化等),全部是 backend,当前尚未登记 case;`settings-test` 和 `workbench-test` 专门覆盖对应 PRD 功能点,全部是 E2E;Unit 只补足前述范围未覆盖的 backend 函数、类和底层逻辑;`showcase` 是独立官网样例采集,不进入领域测试计划。
- `ai-test` 专门覆盖 AI 架构及其工具,全部是 backend;`knowledge-test` 专门覆盖知识库的功能测试(如搜索和索引等),优化设计测试(如召回率、检索与索引优化等),全部是 backend;`settings-test` 和 `workbench-test` 专门覆盖对应 PRD 功能点,全部是 E2E;Unit 只补足前述范围未覆盖的 backend 函数、类和底层逻辑;`showcase` 是独立官网样例采集,不进入领域测试计划。
测试体系只沉淀一层长期执行产物:
@@ -144,7 +144,7 @@ runner 不负责:
- `pnpm test:ai:human:mock-llm`
- `pnpm test:ai:human:live`
- Knowledge
- `pnpm test:knowledge:backend`:知识库专项;当前尚未登记 case。
- `pnpm test:knowledge:backend`
- Unit
- `pnpm test:unit`
- `pnpm test:unit:ai`
@@ -1,31 +1,75 @@
import { describe, expect, it } from 'vitest'
import { resolveOrtThreadBudget } from '../../../../../src/main/services/vector/indexSpeed'
import { describe, it } from 'vitest'
import { resolveOrtThreadBudget } from '../../../../src/main/services/vector/indexSpeed'
import { runBackendCase } from '../../../harness/backend/run-backend-case'
describe('index speed thread budget', () => {
it('maps slow and fast ORT budgets for 4, 8 and 16 logical cores', () => {
expect(resolveOrtThreadBudget('slow', 4)).toEqual({
intraOpNumThreads: 2,
allowSpinning: false
})
expect(resolveOrtThreadBudget('slow', 8)).toEqual({
intraOpNumThreads: 4,
allowSpinning: false
})
expect(resolveOrtThreadBudget('slow', 16)).toEqual({
intraOpNumThreads: 8,
allowSpinning: false
})
expect(resolveOrtThreadBudget('fast', 4)).toEqual({
intraOpNumThreads: 3,
allowSpinning: true
})
expect(resolveOrtThreadBudget('fast', 8)).toEqual({
intraOpNumThreads: 7,
allowSpinning: true
})
expect(resolveOrtThreadBudget('fast', 16)).toEqual({
intraOpNumThreads: 14,
allowSpinning: true
describe('knowledge foundation', () => {
it('index-speed-thread-budget', async () => {
await runBackendCase({
caseId: 'index-speed-thread-budget',
harnessMode: 'tool-real',
domain: 'knowledge',
suite: 'foundation',
description: '慢速/高速按逻辑核数映射 ORT 线程预算与 spinning。',
fixtures: [],
runMode: 'single',
execute: async () => ({
slow4: resolveOrtThreadBudget('slow', 4),
slow8: resolveOrtThreadBudget('slow', 8),
slow16: resolveOrtThreadBudget('slow', 16),
fast4: resolveOrtThreadBudget('fast', 4),
fast8: resolveOrtThreadBudget('fast', 8),
fast16: resolveOrtThreadBudget('fast', 16)
}),
assert: async (result, runtime) => {
runtime.assertThat(
result.slow4.intraOpNumThreads === 2 && result.slow4.allowSpinning === false,
{
name: 'slow-4-cores',
message: '4 核慢速应为 2 线程且关闭 spinning。',
data: result.slow4
}
)
runtime.assertThat(
result.slow8.intraOpNumThreads === 4 && result.slow8.allowSpinning === false,
{
name: 'slow-8-cores',
message: '8 核慢速应为 4 线程且关闭 spinning。',
data: result.slow8
}
)
runtime.assertThat(
result.slow16.intraOpNumThreads === 8 && result.slow16.allowSpinning === false,
{
name: 'slow-16-cores',
message: '16 核慢速应为 8 线程且关闭 spinning。',
data: result.slow16
}
)
runtime.assertThat(
result.fast4.intraOpNumThreads === 3 && result.fast4.allowSpinning === true,
{
name: 'fast-4-cores',
message: '4 核高速应为 3 线程且开启 spinning。',
data: result.fast4
}
)
runtime.assertThat(
result.fast8.intraOpNumThreads === 7 && result.fast8.allowSpinning === true,
{
name: 'fast-8-cores',
message: '8 核高速应为 7 线程且开启 spinning。',
data: result.fast8
}
)
runtime.assertThat(
result.fast16.intraOpNumThreads === 14 && result.fast16.allowSpinning === true,
{
name: 'fast-16-cores',
message: '16 核高速应为 14 线程且开启 spinning。',
data: result.fast16
}
)
}
})
})
})
@@ -1,10 +1,11 @@
import { describe, expect, it } from 'vitest'
import { buildModelToolResultDelivery } from '../../../../../src/main/services/ai/human/shared/modelToolResult'
import { describe, it } from 'vitest'
import { buildModelToolResultDelivery } from '../../../../src/main/services/ai/human/shared/modelToolResult'
import {
buildKbCapabilityTools,
KB_INDEXING_IN_PROGRESS_ERROR
} from '../../../../../src/main/services/ai/human/tools/kb/retrievalTools'
import type { AiRunContext, KnowledgeRetrievalContext } from '../../../../../src/main/services/ai/types'
} from '../../../../src/main/services/ai/human/tools/kb/retrievalTools'
import type { AiRunContext, KnowledgeRetrievalContext } from '../../../../src/main/services/ai/types'
import { runBackendCase } from '../../../harness/backend/run-backend-case'
const MODEL_FACING_INDEXING_ERROR = `Error: ${KB_INDEXING_IN_PROGRESS_ERROR}`
@@ -19,41 +20,81 @@ const invokeTool = async (
return await tool.invoke(args)
}
describe('ai unit kb indexing blocks search', () => {
it('fails kb_search and kb_list_documents while a prepare index job is running', async () => {
const ctx = {
runId: 'run-kb-indexing',
checkCancelled: () => undefined,
sendEvent: () => undefined,
documentsStore: {} as KnowledgeRetrievalContext['documentsStore'],
embeddingService: {} as KnowledgeRetrievalContext['embeddingService'],
assertKbSearchAvailable: async () => {
throw new Error(KB_INDEXING_IN_PROGRESS_ERROR)
}
} as KnowledgeRetrievalContext
describe('knowledge foundation', () => {
it('kb-indexing-blocks-search', async () => {
await runBackendCase({
caseId: 'kb-indexing-blocks-search',
harnessMode: 'tool-real',
domain: 'knowledge',
suite: 'foundation',
description: 'prepare 索引进行中时 kb_search / kb_list_documents 失败并投递模型可读错误。',
fixtures: [],
runMode: 'single',
execute: async () => {
const ctx = {
runId: 'run-kb-indexing',
checkCancelled: () => undefined,
sendEvent: () => undefined,
documentsStore: {} as KnowledgeRetrievalContext['documentsStore'],
embeddingService: {} as KnowledgeRetrievalContext['embeddingService'],
assertKbSearchAvailable: async () => {
throw new Error(KB_INDEXING_IN_PROGRESS_ERROR)
}
} as KnowledgeRetrievalContext
await expect(invokeTool(ctx, 'kb_search', { queryText: '合同' })).rejects.toThrow(
KB_INDEXING_IN_PROGRESS_ERROR
)
await expect(invokeTool(ctx, 'kb_list_documents', {})).rejects.toThrow(
KB_INDEXING_IN_PROGRESS_ERROR
)
})
const searchError = await invokeTool(ctx, 'kb_search', { queryText: '合同' }).then(
() => '',
(error: unknown) => (error instanceof Error ? error.message : String(error))
)
const listError = await invokeTool(ctx, 'kb_list_documents', {}).then(
() => '',
(error: unknown) => (error instanceof Error ? error.message : String(error))
)
it('delivers the indexing error into the model-facing kb tool result', async () => {
for (const toolName of ['kb_search', 'kb_list_documents'] as const) {
const delivery = await buildModelToolResultDelivery({
ctx: { platform: 'win32' } as AiRunContext,
toolCallId: `call-${toolName}`,
args: toolName === 'kb_search' ? { queryText: '合同' } : {},
result: {
toolName,
ok: false,
data: { error: KB_INDEXING_IN_PROGRESS_ERROR }
const deliveries = []
for (const toolName of ['kb_search', 'kb_list_documents'] as const) {
const delivery = await buildModelToolResultDelivery({
ctx: { platform: 'win32' } as AiRunContext,
toolCallId: `call-${toolName}`,
args: toolName === 'kb_search' ? { queryText: '合同' } : {},
result: {
toolName,
ok: false,
data: { error: KB_INDEXING_IN_PROGRESS_ERROR }
}
})
deliveries.push({
toolName,
ok: delivery.ok,
content: delivery.content
})
}
})
expect(delivery.ok).toBe(false)
expect(delivery.content).toBe(MODEL_FACING_INDEXING_ERROR)
}
return { searchError, listError, deliveries }
},
assert: async (result, runtime) => {
runtime.assertThat(
result.searchError === KB_INDEXING_IN_PROGRESS_ERROR &&
result.listError === KB_INDEXING_IN_PROGRESS_ERROR,
{
name: 'kb-tools-fail-while-indexing',
message:
'更新索引执行中 kb_search / kb_list_documents 应失败,对齐 SET-WORKSPACE-004-AC9、SET-WEB-004-AC8。',
data: result
}
)
runtime.assertThat(
result.deliveries.length === 2 &&
result.deliveries.every(
(item) => item.ok === false && item.content === MODEL_FACING_INDEXING_ERROR
),
{
name: 'indexing-error-is-model-facing',
message: '索引中错误应投递为模型可读的 Error: 正在更新知识库索引,请完成后再检索。',
data: result.deliveries
}
)
}
})
})
})
@@ -1,24 +1,42 @@
import { describe, expect, it } from 'vitest'
import { prepareWorkspacesSequentially } from '../../../../../src/shared/prepareWorkspacesSequentially'
import { describe, it } from 'vitest'
import { prepareWorkspacesSequentially } from '../../../../src/shared/prepareWorkspacesSequentially'
import { runBackendCase } from '../../../harness/backend/run-backend-case'
describe('prepareWorkspacesSequentially', () => {
it('starts the next workspace only after the previous prepare settles', async () => {
const events: string[] = []
const results = await prepareWorkspacesSequentially(['ws-a', 'ws-b', 'ws-c'], async (workspaceId) => {
events.push(`start:${workspaceId}`)
await Promise.resolve()
events.push(`end:${workspaceId}`)
return workspaceId
describe('knowledge foundation', () => {
it('prepare-workspaces-sequentially', async () => {
await runBackendCase({
caseId: 'prepare-workspaces-sequentially',
harnessMode: 'tool-real',
domain: 'knowledge',
suite: 'foundation',
description: '多地址更新索引按顺序等待前一个完成再启动下一个。',
fixtures: [],
runMode: 'single',
execute: async () => {
const events: string[] = []
const results = await prepareWorkspacesSequentially(
['ws-a', 'ws-b', 'ws-c'],
async (workspaceId) => {
events.push(`start:${workspaceId}`)
await Promise.resolve()
events.push(`end:${workspaceId}`)
return workspaceId
}
)
return { results, events }
},
assert: async (result, runtime) => {
runtime.assertThat(
result.results.join(',') === 'ws-a,ws-b,ws-c' &&
result.events.join(',') ===
'start:ws-a,end:ws-a,start:ws-b,end:ws-b,start:ws-c,end:ws-c',
{
name: 'prepare-is-sequential',
message: '多地址更新索引必须等前一个完成后再启动下一个,对齐 SET-WORKSPACE-004-AC11。',
data: result
}
)
}
})
expect(results).toEqual(['ws-a', 'ws-b', 'ws-c'])
expect(events).toEqual([
'start:ws-a',
'end:ws-a',
'start:ws-b',
'end:ws-b',
'start:ws-c',
'end:ws-c'
])
})
})
@@ -1,6 +1,6 @@
import { describe, it } from 'vitest'
import type { WorkspaceIndexJobRecord } from '../../../../../src/shared/types'
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
import type { WorkspaceIndexJobRecord } from '../../../../src/shared/types'
import { runBackendCase } from '../../../harness/backend/run-backend-case'
const NOW = 1_716_200_000_000
@@ -32,9 +32,8 @@ describe('unit workspace backend', () => {
await runBackendCase({
caseId: 'workspace-index-job-store-contract',
harnessMode: 'tool-real',
domain: 'unit',
suite: 'knowledge',
groups: ['indexing'],
domain: 'knowledge',
suite: 'foundation',
description: 'workspace_index_job store 支持创建、active 查询、阶段进度更新和失败收敛。',
fixtures: [],
runMode: 'single',
@@ -6,20 +6,20 @@ import {
WORKSPACE_EMBEDDING_BATCH_CHUNK_COUNT,
WORKSPACE_EMBEDDING_BATCH_MAX_CHARS,
WORKSPACE_EMBEDDING_CHUNK_MAX_CHARS
} from '../../../../../src/main/services/workspace/constants'
} from '../../../../src/main/services/workspace/constants'
import {
IngestionService,
isSupportedLocalFilePath
} from '../../../../../src/main/services/ingestion/loader'
} from '../../../../src/main/services/ingestion/loader'
import {
buildWorkspaceKnowledgeSplitSnapshot,
computeWorkspaceFileContentHash,
createWorkspaceEmbeddingBatches
} from '../../../../../src/main/services/workspace/utils/knowledge'
import { handleWorkspacePrepareWorkerRequest } from '../../../../../src/main/services/workspace/worker/workspacePrepareWorker'
import { WorkspacePrepareWorkerClient } from '../../../../../src/main/services/workspace/worker/workspacePrepareWorkerClient'
import type { WorkspacePrepareWorkerRequest } from '../../../../../src/main/services/workspace/types'
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
} from '../../../../src/main/services/workspace/utils/knowledge'
import { handleWorkspacePrepareWorkerRequest } from '../../../../src/main/services/workspace/worker/workspacePrepareWorker'
import { WorkspacePrepareWorkerClient } from '../../../../src/main/services/workspace/worker/workspacePrepareWorkerClient'
import type { WorkspacePrepareWorkerRequest } from '../../../../src/main/services/workspace/types'
import { runBackendCase } from '../../../harness/backend/run-backend-case'
const createChunk = (length: number, page: number): { text: string; page: number } => ({
text: 'x'.repeat(length),
@@ -31,9 +31,8 @@ describe('unit workspace backend', () => {
await runBackendCase({
caseId: 'workspace-index-worker-contract',
harnessMode: 'tool-real',
domain: 'unit',
suite: 'knowledge',
groups: ['indexing'],
domain: 'knowledge',
suite: 'foundation',
description: 'workspace 文件索引 worker 协议与 embedding batch 限制保持稳定。',
fixtures: ['tests/fixtures/workspace'],
runMode: 'single',
@@ -4,8 +4,8 @@ import type {
WorkspaceFileIndexRecord,
WorkspaceIndexJobRecord,
WorkspaceSource
} from '../../../../../src/shared/types'
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
} from '../../../../src/shared/types'
import { runBackendCase } from '../../../harness/backend/run-backend-case'
const NOW = 1_716_200_000_000
const WORKSPACE: WorkspaceSource = {
@@ -69,9 +69,8 @@ describe('unit workspace backend', () => {
await runBackendCase({
caseId: 'workspace-interrupted-job-recovery-contract',
harnessMode: 'tool-real',
domain: 'unit',
suite: 'knowledge',
groups: ['indexing'],
domain: 'knowledge',
suite: 'foundation',
description: '启动恢复会把遗留 running job 和条目级执行中状态收敛为失败。',
fixtures: [],
runMode: 'single',
@@ -1,16 +1,15 @@
import { describe, it } from 'vitest'
import path from 'node:path'
import { mkdir, writeFile } from 'node:fs/promises'
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
import { runBackendCase } from '../../../harness/backend/run-backend-case'
describe('unit workspace backend', () => {
it('workspace-scan-contract', async () => {
await runBackendCase({
caseId: 'workspace-scan-contract',
harnessMode: 'tool-real',
domain: 'unit',
suite: 'knowledge',
groups: ['indexing'],
domain: 'knowledge',
suite: 'foundation',
description: '工作空间扫描服务生成文件索引记录,并由派生摘要支撑页面刷新。',
fixtures: ['tests/fixtures/workspace'],
runMode: 'single',
File renamed without changes.
@@ -38,13 +38,15 @@ describe('settings workspace e2e', () => {
const workspaceId = workspace.workspace!.id
let prepareProgressVisible = false
let buttonsDisabled = false
let defaultSlowSelected = false
let knowledgeIndexSpeed = ''
let directoryCallBaseline = 0
let directoryCallsWhileRunning = -1
let directoryCallsAfterCompletion = -1
await runtime.step({
name: 'prepare-workspace-through-ui',
message: '通过 UI 点击更新索引,并确认执行中进度块和按钮禁用状态。',
message: '通过 UI 先切换索引速度,再点击更新索引,并确认执行中进度块和按钮状态。',
run: async () => {
await navigateToSettingsSection(page, 'workspace')
await getSettingsLocator(page, importSelectors.workspaceQuickPrepareButton).waitFor({
@@ -53,6 +55,29 @@ describe('settings workspace e2e', () => {
await getSettingsLocator(page, importSelectors.workspaceTree).waitFor({
state: 'visible'
})
const slowSpeed = getSettingsLocator(page, importSelectors.workspaceIndexSpeedSlow)
const fastSpeed = getSettingsLocator(page, importSelectors.workspaceIndexSpeedFast)
await slowSpeed.waitFor({ state: 'visible' })
await page.waitForFunction(
(testId) => {
const element = document.querySelector(`[data-testid="${testId}"]`)
return element instanceof HTMLButtonElement && !element.disabled
},
importSelectors.workspaceIndexSpeedSlow
)
defaultSlowSelected = await slowSpeed.evaluate((element) =>
element.classList.contains('ant-btn-primary')
)
await fastSpeed.click()
await page.waitForFunction(
(testId) =>
document.querySelector(`[data-testid="${testId}"]`)?.classList.contains(
'ant-btn-primary'
) === true,
importSelectors.workspaceIndexSpeedFast
)
knowledgeIndexSpeed = (await page.evaluate(() => window.api.appSettingsGet()))
.knowledgeIndexSpeed
directoryCallBaseline = await page.evaluate(
() =>
(window as typeof window & {
@@ -102,6 +127,8 @@ describe('settings workspace e2e', () => {
return {
prepareProgressVisible,
buttonsDisabled,
defaultSlowSelected,
knowledgeIndexSpeed,
directoryCallsWhileRunning,
directoryCallsAfterCompletion,
summary: result.tree.summary,
@@ -109,6 +136,16 @@ describe('settings workspace e2e', () => {
}
},
assert: async (result, runtime) => {
await runtime.assertThat(
result.defaultSlowSelected === true && result.knowledgeIndexSpeed === 'fast',
{
name: 'prepare-runs-after-switching-index-speed',
message:
'默认慢速,切换高速并保存后再更新索引,对齐 SET-WORKSPACE-009-AC1;本次索引走切换后的档位。',
data: result
}
)
await runtime.assertThat(result.prepareProgressVisible && !result.buttonsDisabled, {
name: 'prepare-progress-is-visible',
message:
+1 -1
View File
@@ -221,7 +221,7 @@ export type TestCaseDefinition<TResult> = {
caseId: string
harnessMode: BackendHarnessMode
testType?: Extract<TestType, 'backend'>
domain: Extract<TestDomain, 'ai' | 'unit'>
domain: Extract<TestDomain, 'ai' | 'knowledge' | 'unit'>
suite: string
groups?: readonly string[]
description: string
+1 -1
View File
@@ -2,7 +2,7 @@ import type { JsonValue } from '../../../src/shared/types'
export const TEST_LOG_SCHEMA_VERSION = 5 as const
export type TestDomain = 'ai' | 'settings' | 'workbench' | 'unit' | 'showcase'
export type TestDomain = 'ai' | 'knowledge' | 'settings' | 'workbench' | 'unit' | 'showcase'
export type TestType = 'backend' | 'e2e'
export type TestRunStatus = 'running' | 'passed' | 'failed'
export type TestLogStatus = 'info' | 'ok' | 'failed' | 'skipped'