diff --git a/AGENTS.md b/AGENTS.md index 97fe9fdb..3881147f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,7 +3,7 @@ ## 1. 文档引用 ### 1.1 AI相关 - 当涉及 AI 架构、主流程或 human-only runtime 时，必须阅读 `doc/ai-design/architecture.md` -- 当涉及 AI 实现、运行入口、跨层边界、LLM 执行入口、流式传输、runtime settings、运行数据、`execution.watchHint` / `toolUsePlan` / `execution.lastLearningOutcome` / `memory.history.roundHistory` / `history_recall` / `recent_progress` / `earlier_attempts` / `recalled_history` 契约或工作区 KB 范围时，必须阅读 `doc/ai-design/implementation-spec.md` 与 `doc/ai-design/data-definition-and-flow.md` +- 当涉及 AI 实现、运行入口、跨层边界、LLM 执行入口、流式传输、runtime settings、运行数据、`execution.watchHint` / `toolUsePlan` / `execution.lastLearningOutcome` / `memory.history.roundHistory` / `history_recall` / `mouth` 统一 `history` prompt 输入契约或工作区 KB 范围时，必须阅读 `doc/ai-design/implementation-spec.md` 与 `doc/ai-design/data-definition-and-flow.md` - 当涉及 AI 最小运行样例、默认链路或想快速理解 human 各环节如何衔接时，必须阅读 `doc/ai-design/example.md` - 当涉及 human 的 skill 分类、拆分边界、builtin/external 区分或当前 skill 基线时，必须阅读 `doc/ai-design/human-skill-design.md` - 当涉及 prompt 的通用结构、输入组织、输出规则、约束表达或风格一致性时，必须阅读 `doc/ai-design/prompt-design.md` @@ -16,6 +16,7 @@ ### 1.3 测试相关 - 当涉及 AI 测试，必须阅读 `doc/test-plan/ai-test.md` +- 当涉及 human live 的 docx / KB 混合场景测试设计时，必须阅读 `doc/test-plan/ai-test.md` - 当涉及 Workbench 测试时，必须阅读 `doc/test-plan/workbench-test.md` - 当涉及 Settings 测试时，必须阅读 `doc/test-plan/settings-test.md` - 当涉及测试用例编写，测试规范时，必须阅读 `doc/test.md` diff --git a/doc/ai-design/data-definition-and-flow.md b/doc/ai-design/data-definition-and-flow.md index c69f6575..71f1e474 100644 --- a/doc/ai-design/data-definition-and-flow.md +++ b/doc/ai-design/data-definition-and-flow.md @@ -1100,20 +1100,21 @@ flowchart LR flowchart LR T[("task")] SM[("memory.stable")] - PM[("memory.history.roundHistory")] + PM[("memory.history.roundHistory / loadedHistoryContext")] CD[("execution.currentDecision")] A[("execution.artifacts")] - WH[("execution.watchHint")] M["mouth"] R[("result")] TR[("trace.steps")] T -->|读取 taskId + userInput| M SM -->|读取任务笔记| M - PM -->|读取上一轮完整闭环| M + PM -->|读取统一 history 事实视图| M CD -->|读取当前最终决策| M A -->|读取已收集的结构化 artifacts| M - WH -->|必要时整理为交付提示| M M -->|输出 deliveryMarkdown；node 派生 summary、回写 completed result，并透传 artifacts| R M -->|追加 mouth 执行轨迹| TR ``` + +- `mouth` 不再依赖 `recent_progress / earlier_attempts / recalled_history` 这类拆分块；统一基于 `roundHistory + loadedHistoryContext` 渲染单一 `history` 视图。 +- `mouth` 的 prompt 输入仍以 `your_notes` 为优先事实来源；当 `your_notes` 不够具体时，再从统一 `history` 与 `runtime_artifacts` 补足用户回答所需细节，而不是依赖额外的 `watchHint` 说明。 diff --git a/doc/ai-design/implementation-spec.md b/doc/ai-design/implementation-spec.md index aefee433..30f28776 100644 --- a/doc/ai-design/implementation-spec.md +++ b/doc/ai-design/implementation-spec.md @@ -94,10 +94,10 @@ stateDiagram-v2 ### 5.4 `mouth` -- 职责：把当前已经确定的状态整理成最终用户交付，并回写 `HumanResult`。 -- 主要读取：`task`、`memory.stable`、`execution.currentDecision`、`execution.artifacts`。 +- 职责：围绕 `task_goal` 把当前已经确定的状态组织成最终用户交付，并回写 `HumanResult`。 +- 主要读取：`task`、`memory.stable`、`memory.history.roundHistory`、`memory.history.loadedHistoryContext`、`execution.currentDecision`、`execution.artifacts`。 - 主要回写：`result`、`execution.status`、`execution.stage`、`trace.steps`。 -- 约束：模型输出必须通过 `ctx.chatText`；输出固定面向最终用户；若开启 `emitAnswerEvents`，增量文本通过 `answer_token` 发出，完成后通过 `answer_completed` 收口。 +- 约束：模型输出必须通过 `ctx.chatText`；输出面向最终用户，优先直接完成用户原始诉求；可使用自由段落或列表组织，不要求固定报告模板；若开启 `emitAnswerEvents`，增量文本通过 `answer_token` 发出，完成后通过 `answer_completed` 收口。 ## 6. 事件与交付 diff --git a/doc/prd/workbench.md b/doc/prd/workbench.md index caa70d7d..5e5388fc 100644 --- a/doc/prd/workbench.md +++ b/doc/prd/workbench.md @@ -99,13 +99,15 @@ - 验收： - `WB-CORE-005-AC1` `terminal` 执行成功后，后续用户可见输出会体现关键命令结果，而不是只表现为“命令执行过”。 - `WB-CORE-005-AC2` `terminal` 输出不足或执行失败时，assistant 会明确说明无法从当前结果得到所需信息，而不是编造结论或文档内容。 + - `WB-CORE-005-AC3` 用户可见输出优先回答用户原始诉求，不要求固定报告体；若引用执行结果，应使用用户能理解的表达，而不是暴露内部工具或过程标签。 #### WB-CORE-006 标准任务应能闭环到可交付结果 -- 系统行为：当用户提供了足够上下文的标准任务，且当前不需要额外阻塞或确认时，workbench 应支持 AI 沿同一条任务链完成资料使用、必要执行和最终交付，而不是只停留在搜索结果、工具调用或过程提示。 +- 系统行为：当用户提供了足够上下文的标准任务，且当前不需要额外阻塞或确认时，workbench 应支持 AI 沿同一条任务链完成资料使用、必要执行和最终交付，而不是只停留在搜索结果、工具调用或过程提示；最终收口应围绕用户原始诉求组织，而不是强制套用固定报告模板。 - 验收： - `WB-CORE-006-AC1` 用户已提供足够上下文且无需额外阻塞时，系统能把标准任务推进到可消费的最终结果或明确产物，而不是停留在中间态。 - `WB-CORE-006-AC2` 最终结果会体现用户显式给定的资料范围与执行得到的关键信息，不会把范围、执行结果和最终交付割裂开。 + - `WB-CORE-006-AC3` 最终结果默认直接回答用户显式提出的问题；若同时需要说明产物、依据或限制，也应以用户可理解的语言表达，而不是依赖内部 block id、tool 名或固定章节标题。 ### 8.2 会话管理 diff --git a/doc/test-plan/ai-test.md b/doc/test-plan/ai-test.md index 09d6807f..79578cce 100644 --- a/doc/test-plan/ai-test.md +++ b/doc/test-plan/ai-test.md @@ -247,10 +247,10 @@ - 测试类型：`backend` - 实现状态：`已实现` -- What：验证 `hand` 的并行工具执行与聚合守卫。 +- What：验证 `hand` 的并行工具执行上限、只读安全约束与结果聚合守卫。 - Why：工具执行器本身必须在真实结果下稳定收敛。 -- How：通过真实 `hand` 执行多工具调用并检查状态收敛。 -- Oracle：并行工具结果被正确聚合，运行状态无串扰。 +- How：通过真实 `hand` 执行安全并行批次、`6` 个只读安全并行批次、超过上限的 `7` 个并行批次以及包含 mutating 调用的不安全批次。 +- Oracle：`6` 个只读且 `parallelSafe` 的工具调用会被真实执行并正确聚合；`7` 个并行调用会在执行前被 `parallel_limit_exceeded` 拦截；包含 mutating 或非并行安全工具的批次会在执行前被 `parallel_requires_read_only` 拦截。 - Not：不验证 `brain` 决策来源。 ##### `hand-serial-docx-chain` @@ -416,12 +416,42 @@ - What：验证 live 模式下 `human` 会在一个真实长任务里串起模板定位、资料取证、docx 编辑与预览交付。 - Why：`human/live` 需要至少一条长期高价值复杂样例，覆盖 workspace、KB、docx 与最终交付链路。 - How：注入真实 live LLM 配置；workspace 中动态写入带 `RISK_TODO / OWNER_TODO` 占位符的周报模板和干扰文件；seed 两份真实 KB 文档分别提供风险结论与负责人说明并限制在已选范围内；直接输入“找到上周 Word 周报模板，按已选资料补全风险结论与负责人，生成预览且不要覆盖原文件”的自然语言诉求，让真实 `human` 自主完成。 -- Oracle：任务最终完成；事件流中至少出现一次 `terminal_run_command`、一次以 `kb_` 开头的检索工具调用、一次 `docx_open`、一次 `docx_apply_ops` 与一次 `docx_render_preview`；存在 preview/draft artifact，且原模板未被覆盖。 +- Oracle：任务最终完成；事件流中至少出现一次 `terminal_run_command`、一次以 `kb_` 开头的检索工具调用、一次 `docx_open`、一次 `docx_apply_ops` 与一次 `docx_render_preview`；存在 preview/draft artifact，且原模板未被覆盖；最终答复能从用户视角说明改动的是“风险结论”和“负责人说明”两处、依据分别来自 `Weekly Risk Summary.md` 与 `Owner Note.md`，并明确已经生成预览且未覆盖原文件。 - Not：不绑定最终答案措辞，也不要求固定检索工具顺序。 +##### `human-docx-cause-remedy-reading-live` + +- 测试类型：`backend` +- 实现状态：`已实现` +- What：验证 live 模式下 `human` 会基于两份已选 `docx` 做跨文件精读，并准确回答“过拟合 / 反走样”的成因与处理思路。 +- Why：真实用户在复习场景里经常已经选中了资料，但仍会把两个概念混在一起；这条 case 要长期守住“已选资料范围 + 跨文件准确回答”的正式能力，对齐 `WB-CORE-002-AC2`、`WB-CORE-004-AC1`、`WB-CORE-006-AC1`、`WB-CORE-006-AC2`。 +- How：注入真实 live LLM 配置；把 `tests/fixtures/ai/docx/深度学习.docx` 与 `tests/fixtures/ai/docx/图形学.docx` 作为已选资料复制进 workspace；直接输入“我做题老把这两个点混在一起。你就按我刚选的这两份资料，帮我说清楚：过拟合和反走样各自是什么、为什么会出现、一般怎么处理。别补充课外知识，哪份文档没写你就直接说没写。”的自然语言诉求，让真实 `human` 自主完成。 +- Oracle：任务最终完成；事件流中至少两次 `docx_open` 覆盖两份目标文档，且至少出现一次 `docx_search`、`docx_outline` 或 `docx_get_block`；最终回答同时命中“过拟合”的成因/处理事实组和“反走样”的成因/处理事实组；不出现 `docx_apply_ops` 或 `docx_save`。 +- Not：不要求固定回答结构，也不对最终文案做逐字断言。 + +##### `human-docx-contextual-comparison-draft-live` + +- 测试类型：`backend` +- 实现状态：`已实现` +- What：验证 live 模式下 `human` 会基于两份已选 `docx`，在目标 `docx` 的相关位置追加解释性段落和对照表，并生成不覆盖原文件的预览稿。 +- Why：真实用户常见诉求不是“改占位符”，而是“在这块后面补一段更好懂的说明”；这条 case 覆盖跨文件取材、相关位置补写、结构化新增与 preview-only 交付，对齐 `WB-CORE-002-AC2`、`WB-CORE-003-AC1`、`WB-CORE-004-AC1`、`WB-CORE-006-AC1`、`WB-CORE-006-AC2`。 +- How：注入真实 live LLM 配置；把 `tests/fixtures/ai/docx/深度学习.docx` 与 `tests/fixtures/ai/docx/图形学.docx` 作为已选资料复制进 workspace；直接输入“《图形学》里‘反走样’这一节我还是记不住。你参考我刚选的这两份 Word，直接在这一节最后补一小段更好理解的说明，再加个简单对照表，一列写反走样，一列写《深度学习》里的过拟合。直接给我可预览的版本就行，别改原文件，也不用先回我文字草稿。”的自然语言诉求，让真实 `human` 自主完成。 +- Oracle：任务最终完成；事件流中至少两次 `docx_open` 覆盖两份目标文档；`docx_apply_ops` 前至少出现一次针对“反走样”的读取定位；至少出现一次 `docx_apply_ops` 与一次 `docx_render_preview`；存在 preview/draft artifact；不出现 `docx_save`；原始两份 `docx` 文件未被覆盖；写入请求里能看到新增说明段落与表格结构。 +- Not：不要求固定 op 组合，也不对最终说明文案做逐字断言。 + +##### `human-kb-guided-docx-comparison-draft-live` + +- 测试类型：`backend` +- 实现状态：`已实现` +- What：验证 live 模式下 `human` 在本地资料较多时，会先用 KB 缩圈定位相关资料，再回到目标 `docx` 做跨文件补写并交付预览。 +- Why：真实用户常常记得主题、不记得文件名；这条 case 要长期守住“工作区默认范围 + KB 缩圈 + docx 继续执行”的正式能力，对齐 `WB-CORE-002-AC2`、`WB-CORE-004-AC1`、`WB-CORE-006-AC1`、`WB-CORE-006-AC2`。 +- How：注入真实 live LLM 配置；workspace 中动态写入 `深度学习.docx`、`图形学.docx` 和若干干扰文件，并把目标 `docx` 与干扰资料同时 seed 到真实 KB；不传 `selectedReferences`，只绑定 workspace；直接输入“我工作区资料有点多，我忘了是哪份课件在讲反走样了。你先帮我找到那份，再参考讲过拟合的那份资料，直接在‘反走样’那一节最后补一段更好记的说明，再加个简单对照表，一列写反走样，一列写过拟合。直接给我可预览的版本就行，别改原文件，也不用先回我文字草稿。”的自然语言诉求，让真实 `human` 自主完成。 +- Oracle：任务最终完成；事件流中至少出现一次 KB 工具调用；后续出现 `docx_open` 并覆盖最终参与编辑的目标 `docx`；至少出现一次 `docx_apply_ops` 与一次 `docx_render_preview`；存在 preview/draft artifact；不出现 `docx_save`；目标原始 `docx` 未被覆盖；新增内容同时涉及“反走样”和“过拟合”。 +- Not：不强制固定执行路径，也不要求 KB 缩圈后必须再打开所有候选资料。 + ## 5. 验收口径 - `tool` suite 必须持续守住真实工具输入输出、审批边界和 artifact 契约。 - `human/mock-llm` 必须覆盖 `AiRunService -> human` 入口、`brain / learning / hand / mouth` 主链、memory 延续、watch hint 收敛与复杂工具协作。 - `human/live` 必须至少保留 1 条简单任务和 1 条复杂任务正式 case。 -- `human` suite 必须继续支撑 `WB-CORE-002`、`WB-CORE-005`、`WB-CORE-006`。 +- `human` suite 必须继续支撑 `WB-CORE-002`、`WB-CORE-003`、`WB-CORE-004`、`WB-CORE-005`、`WB-CORE-006`。 diff --git a/src/main/services/ai/human/constants.ts b/src/main/services/ai/human/constants.ts index 37a3561b..315fdd65 100644 --- a/src/main/services/ai/human/constants.ts +++ b/src/main/services/ai/human/constants.ts @@ -63,6 +63,7 @@ export const HUMAN_WATCH_HINT_CODES = [ export const HUMAN_MAX_ACTIVE_TOOLS = 20 export const HUMAN_DEFAULT_TOOL_CALL_LIMIT = 50 +export const HUMAN_MAX_PARALLEL_TOOL_CALLS = 6 export const HUMAN_ROUND_EXECUTION_KINDS = [ 'pre_execution_blocked', 'learning', diff --git a/src/main/services/ai/human/node/handNode/index.ts b/src/main/services/ai/human/node/handNode/index.ts index d3550645..f273fb87 100644 --- a/src/main/services/ai/human/node/handNode/index.ts +++ b/src/main/services/ai/human/node/handNode/index.ts @@ -1,5 +1,5 @@ import type { JsonObject } from '../../../../../../shared/types' -import { HUMAN_DEFAULT_TOOL_CALL_LIMIT } from '../../constants' +import { HUMAN_DEFAULT_TOOL_CALL_LIMIT, HUMAN_MAX_PARALLEL_TOOL_CALLS } from '../../constants' import type { AiRunContext } from '../../../types' import { extractArtifactRefsFromToolResults, mergeArtifactRefs } from '../../../utils/artifacts' import { @@ -14,8 +14,6 @@ import type { HumanRuntimeCache, HumanState, HumanWatchHint, ToolCallResult, Too import { executeToolCall } from './toolExecutor' import { returnToBrain, returnToBrainAfterTrace, returnToBrainWithWatchHint } from '../utils/stateTransitions' -const MAX_PARALLEL_TOOL_CALLS = 4 - export const getToolExecutionMeta = (input: { toolName: string args: JsonObject @@ -70,13 +68,13 @@ const buildToolLimitWatchHint = (toolCallCount: number, limit: number): HumanWat const validateToolUsePlan = (toolUsePlan: ToolUsePlan): HumanWatchHint | undefined => { if (toolUsePlan.mode === 'serial') return undefined - if (toolUsePlan.calls.length > MAX_PARALLEL_TOOL_CALLS) { + if (toolUsePlan.calls.length > HUMAN_MAX_PARALLEL_TOOL_CALLS) { return createHandPlanInvalidWatchHint({ mode: 'parallel', reason: 'parallel_limit_exceeded', offendingToolNames: toolUsePlan.calls.map((item) => item.toolName), callCount: toolUsePlan.calls.length, - limit: MAX_PARALLEL_TOOL_CALLS + limit: HUMAN_MAX_PARALLEL_TOOL_CALLS }) } diff --git a/src/main/services/ai/human/prompts/brain.ts b/src/main/services/ai/human/prompts/brain.ts index 2023c7af..5993ea5d 100644 --- a/src/main/services/ai/human/prompts/brain.ts +++ b/src/main/services/ai/human/prompts/brain.ts @@ -13,10 +13,9 @@ import { listExternalSkillDefinitions } from '../skills' import { renderHumanStableNotes } from '../shared/memoryRenderers' +import { renderLoadedHistoryText, renderRoundSection } from '../shared/historyRenderers' import { getToolPromptSpec } from '../tools/registry' import type { - HumanLoadedHistoryContext, - HumanRoundRecord, HumanState, ToolPromptSpec } from '../types' @@ -43,168 +42,6 @@ const renderLearnedToolSpecLine = (spec: ToolPromptSpec): string => { return `- 工具 ${spec.name}：用途=${spec.purpose}；必填参数=[${requiredArgs || 'none'}]；${example}` } -const renderDecisionNextAction = (record: HumanRoundRecord): string => { - switch (record.decision.nextAction.action) { - case 'learning': - return `- 你准备下一步做：准备学习 ${renderQuotedNames( - record.decision.nextAction.wantLearnSkillIds - )}。` - case 'hand': - return `- 你准备下一步做：准备进入 hand，${ - record.decision.nextAction.toolUsePlan.mode === 'parallel' - ? `以 parallel 模式并行执行 ${renderQuotedNames( - record.decision.nextAction.toolUsePlan.calls.map((call) => call.toolName) - )}。` - : `以 serial 模式按顺序执行 ${renderQuotedNames( - record.decision.nextAction.toolUsePlan.calls.map((call) => call.toolName) - )}。` - }` - case 'mouth': - return '- 你准备下一步做：准备进入 mouth 组织最终交付。' - } -} - -const renderRoundDecision = (record: HumanRoundRecord): string => - ` -- 你的判断：${record.decision.decision.summary} -- 你这样判断的原因：${record.decision.decision.reason} -- 你当时期待的结果：${record.decision.decision.expectedOutcome} -- ${record.decision.readyToDeliver ? '你当时认为已经可以交付。' : '你当时认为还不能直接交付。'} -${renderDecisionNextAction(record)} -`.trim() - -const renderRoundExecution = ( - record: HumanRoundRecord, - detail: 'focus' | 'chain' | 'loaded' -): string => { - switch (record.execution.kind) { - case 'pre_execution_blocked': - return '- 没有进入真实执行，系统在执行前阻断了这轮动作。' - case 'learning': - return '- 进入 learning，按请求尝试补齐外置技能。' - case 'hand': - return ` -- 进入 hand，执行模式是 ${record.execution.toolUsePlan.mode}。 -- ${ - record.execution.toolUsePlan.mode === 'parallel' - ? `实际并行执行了 ${renderQuotedNames(record.execution.toolUsePlan.calls.map((call) => call.toolName))}。` - : `实际按顺序执行了 ${renderQuotedNames(record.execution.toolUsePlan.calls.map((call) => call.toolName))}。` - } -${detail !== 'chain' ? `- 本轮计划包含 ${record.execution.toolUsePlan.calls.length} 个工具调用。` : ''} -` - .trim() - .replace(/\n\n+/g, '\n') - case 'mouth': - return '- 进入 mouth，组织最终说明。' - } -} - -const renderRoundResult = (record: HumanRoundRecord): string => { - switch (record.execution.kind) { - case 'pre_execution_blocked': - return renderWatchHintForBrain(record.execution.watchHint) - case 'learning': - return ` -- 已加载的技能：${renderQuotedNames(record.execution.learningOutcome.loadedSkillIds)} -- 已跳过的重复技能：${renderQuotedNames(record.execution.learningOutcome.skippedDuplicateSkillIds)} -- 未识别的技能：${renderQuotedNames(record.execution.learningOutcome.skippedUnknownSkillIds)} -- 当前 active tool 数：${String(record.execution.learningOutcome.activeToolCount)} -${record.execution.watchHint ? `- 收敛提示：${renderWatchHintForBrain(record.execution.watchHint)}` : ''} -` - .trim() - .replace(/\n\n+/g, '\n') - case 'hand': { - const toolResultLines = record.execution.toolResults - .map((result) => - [ - `- \`${result.toolName}\`：${result.summary}`, - result.dataPreview ? `结果：${result.dataPreview}` : '', - result.diagnosticText ? `诊断：${result.diagnosticText}` : '' - ] - .filter(Boolean) - .join('；') - ) - .join('\n') - const artifactLine = - record.execution.artifactRefs && record.execution.artifactRefs.length > 0 - ? `- 产物：${renderInlineList( - record.execution.artifactRefs.map((artifact) => - artifact.kind === 'file' - ? `\`${artifact.fileName}\`` - : `draft review ${artifact.review.reviewId}` - ), - { - separator: '、', - fallback: 'none' - } - )}` - : '' - const watchHintLine = record.execution.watchHint - ? `- 收敛提示：${renderWatchHintForBrain(record.execution.watchHint)}` - : '' - - return [toolResultLines, artifactLine, watchHintLine].filter(Boolean).join('\n') - } - case 'mouth': - return ` -- 输出摘要：${record.execution.summary} -${record.execution.completed ? '- 任务已完成。' : '- 任务未完成。'} -`.trim() - } -} - -const renderRoundRecord = ( - record: HumanRoundRecord, - detail: 'focus' | 'chain' | 'loaded' -): string => - ` -- 第 ${record.roundId} 轮 - -决策了什么： -${renderRoundDecision(record)} - -做了什么： -${renderRoundExecution(record, detail)} - -结果是什么： -${renderRoundResult(record)} -`.trim() - -const renderRoundSection = (input: { - records: HumanRoundRecord[] - detail: 'focus' | 'chain' | 'loaded' -}): string => - input.records.length > 0 - ? input.records.map((record) => renderRoundRecord(record, input.detail)).join('\n\n') - : 'none' - -const renderLoadedHistoryText = ( - loadedHistoryContext: HumanLoadedHistoryContext | undefined -): string => - loadedHistoryContext - ? ` -过滤条件：roundIds=${renderInlineList((loadedHistoryContext.filters.roundIds || []).map(String), { - separator: '、', - fallback: 'none' - })}；executionKinds=${renderInlineList(loadedHistoryContext.filters.executionKinds || [], { - separator: '、', - fallback: 'none' - })}；toolNames=${renderInlineList(loadedHistoryContext.filters.toolNames || [], { - separator: '、', - fallback: 'none' - })}；watchHintCodes=${renderInlineList(loadedHistoryContext.filters.watchHintCodes || [], { - separator: '、', - fallback: 'none' - })}；limit=${String(loadedHistoryContext.filters.limit || loadedHistoryContext.rounds.length)} - -${renderRoundSection({ - records: loadedHistoryContext.rounds, - detail: 'loaded' -})} -` - .trim() - : 'none' - const renderAvailableSkillsSection = (state: HumanState): string => { const effectiveSkillIds = getEffectiveLearnedSkillIds(state.skills.learnedSkillIds) const effectiveSkillDefinitions = effectiveSkillIds.map((skillId) => getSkillDefinition(skillId)) diff --git a/src/main/services/ai/human/prompts/mouth.ts b/src/main/services/ai/human/prompts/mouth.ts index 0de3adf4..65ef2eaa 100644 --- a/src/main/services/ai/human/prompts/mouth.ts +++ b/src/main/services/ai/human/prompts/mouth.ts @@ -1,132 +1,41 @@ -import { renderInlineList } from '../../utils/render' -import { getLastRoundFocus } from '../memory/roundHistory' +import { getLoadedHistoryContext } from '../memory/roundHistory' +import { renderUnifiedHistoryText } from '../shared/historyRenderers' import { renderHumanStableNotes } from '../shared/memoryRenderers' import { renderHumanArtifactLines } from '../shared/runtimeRenderers' -import type { HumanRoundRecord, HumanState } from '../types' -import { renderWatchHintForMouth } from './watchHints' - -const renderRecentProgressSection = (record: HumanRoundRecord | undefined): string => { - if (!record) { - return 'none' - } - - const progressText = - record.execution.kind === 'pre_execution_blocked' - ? `- 执行前阻断：${renderWatchHintForMouth(record.execution.watchHint)}` - : record.execution.kind === 'learning' - ? `- learning 结果：已加载 ${renderInlineList( - record.execution.learningOutcome.loadedSkillIds.map((skillId) => `\`${skillId}\``), - { - separator: '、', - fallback: 'none' - } - )}；重复 ${renderInlineList( - record.execution.learningOutcome.skippedDuplicateSkillIds.map( - (skillId) => `\`${skillId}\`` - ), - { - separator: '、', - fallback: 'none' - } - )}；未知 ${renderInlineList( - record.execution.learningOutcome.skippedUnknownSkillIds.map( - (skillId) => `\`${skillId}\`` - ), - { - separator: '、', - fallback: 'none' - } - )}。` - : record.execution.kind === 'hand' - ? ` -- hand 执行模式：${record.execution.toolUsePlan.mode} -${record.execution.toolResults.map((result) => `- \`${result.toolName}\`：${result.summary}`).join('\n')} -` - .trim() - .replace(/\n\n+/g, '\n') - : `- mouth 输出摘要：${record.execution.summary}` - - return ` -- 第 ${record.roundId} 轮判断：${record.decision.decision.summary} -- 原因：${record.decision.decision.reason} -- 预期结果：${record.decision.decision.expectedOutcome} -${progressText} -`.trim() -} +import type { HumanState } from '../types' const buildFinalDecisionText = (state: HumanState): string => state.execution.currentDecision ? `最终决定（readyToDeliver=${String(state.execution.currentDecision.readyToDeliver)}）：“${state.execution.currentDecision.decision.summary}”；原因：${state.execution.currentDecision.decision.reason}；预期结果：${state.execution.currentDecision.decision.expectedOutcome}` : 'none' -const OUTPUT_EXAMPLE = ` -# 处理结果 - -已完成当前任务，下面是基于现有事实和结果整理出的最终结论。 - -## 关键依据 -- 依据一 - -## 产物与引用 -- /path/to/file.docx - -## 当前限制 -无 -`.trim() - export const buildMouthPrompt = (input: { state: HumanState }) => { const system = ` - 你现在扮演一个负责对外表达最终结果的人。 - 你的任务是把已经确定下来的判断、任务笔记和产物整理成最终交付。 + 你现在负责根据已有事实回答用户这次任务。 + 你的目标不是复述内部过程，而是直接完成 task_goal 里用户真正要的回复。 - - 请像一个表达清晰、判断克制的人一样输出结果： - 不夸大、不补编、不引入新的结论， - 只把已经确认的信息组织成最终交付。 - - - - - final_decision 是 brain 已经确认的最终判断；你负责表达，不负责重新推翻或改写它。 - - your_notes 是当前任务的稳定事实来源，包括已确认事实、关键依据和未决问题；正文中的事实与依据优先从这里取。 - - recent_progress 只用于补充最近完成到哪里、最近结果如何，不必把内部过程完整复述出来。 - - runtime_artifacts 只允许准确引用已经给出的路径、文件名、状态和摘要，不要补编额外元数据。 - - correction_hint 只用于如实说明限制、阻断或未完成点，不要把它扩写成新的事实。 - - - - - task_goal：用户原始诉求与限制。 - - final_decision：brain 已经确认的最终判断与收口方向。 - - your_notes：稳定任务笔记，按“已确认 / 关键依据 / 未决问题”组织。 - - recent_progress：最近一轮过程记录，用于补充当前完成到哪里。 - - runtime_artifacts：运行中已经产生的真实产物列表。 - - correction_hint：当前额外的收敛或纠偏提醒；若无则为 none。 - - - - 1. 如果任务已经完成，先直接给出结论，再给关键依据，再给结果或产物说明。 - 2. 如果当前结果有限，先说明已经完成到哪里，再说明限制和继续需要什么。 - 3. 只使用已经给你的任务笔记、最终判断和产物，不要补充新的事实或推理。 - 4. 语言要清楚、克制、可执行，不要堆内部过程。 - 5. 如果存在运行时产物或最近过程记录，只能准确引用已经给出的路径、状态和摘要。 - 6. 你不会调用工具，也不会继续推进未完成步骤；如果仍有步骤没做，只能如实说明当前停在哪。 - + + - task_goal：用户最初要你完成什么，以及用户显式要求你回答哪些点。 + - final_decision：brain 最后确认的收口方向。 + - your_notes：当前稳定事实来源。其中： + - 已确认：当前稳定成立、可以直接告诉用户的事实。 + - 关键依据：支撑这些事实的证据、来源或命中结果。 + - 未决问题：仍会影响结论、或需要用户补充确认的点。 + - history：任务过程中真实发生过的决策、执行和结果，可用于补足 your_notes 不够具体的地方。 + - runtime_artifacts：运行中已经产生的真实产物，可按给定路径、文件名、状态和摘要引用。 + - - 你必须只输出最终 markdown 正文，不要输出任何额外说明、JSON 或代码块。 - - 不要引入新的事实、依据、步骤或承诺。 - - 不要暴露内部字段名、节点名或系统术语。 - - 如果依据不足，就保持保守表达，不要写成确定结论。 - - 不要把未来可能执行的工具动作写成“后续会自动完成”或“已经在进行中”。 - - 输出必须是完整结果，而不是零散句子或局部片段。 - - 默认使用最终用户答复的组织方式：先给结论，再给关键依据，再给产物或当前限制。 + - 只输出最终 markdown 正文，不要输出 JSON 或额外说明。 + - 直接回答 task_goal，优先覆盖用户显式提出的问题。 + - your_notes 优先；如果 your_notes 不够具体，可结合 history 与 runtime_artifacts 补足。 + - 只基于提供的信息回答，不要补编新的事实。 + - 标题、分节、列表都按需要使用，不固定。 + - 默认使用用户能理解的语言，不原样照抄内部标签、block id、tool 名，除非用户明确要求这些内部细节。 - - -${OUTPUT_EXAMPLE} - `.trim() @@ -144,17 +53,16 @@ ${buildFinalDecisionText(input.state)} ${renderHumanStableNotes(input.state, 'mouth_prompt')} - -${renderRecentProgressSection(getLastRoundFocus(input.state))} - + +${renderUnifiedHistoryText({ + roundHistory: input.state.memory.history.roundHistory, + loadedHistoryContext: getLoadedHistoryContext(input.state) +})} + ${renderHumanArtifactLines(input.state, 'prompt')} - - -${input.state.execution.watchHint ? renderWatchHintForMouth(input.state.execution.watchHint) : 'none'} - `.trim() diff --git a/src/main/services/ai/human/prompts/watchHints.ts b/src/main/services/ai/human/prompts/watchHints.ts index d39e743a..6db03e50 100644 --- a/src/main/services/ai/human/prompts/watchHints.ts +++ b/src/main/services/ai/human/prompts/watchHints.ts @@ -78,8 +78,4 @@ export const renderWatchHintForBrain = (hint: HumanWatchHint): string => { } } -export const renderWatchHintForMouth = (hint: HumanWatchHint): string => { - return summarizeWatchHint(hint) -} - export const renderWatchHintDetail = (hint: HumanWatchHint): JsonObject => asJsonObject(hint.detail) diff --git a/src/main/services/ai/human/shared/historyRenderers.ts b/src/main/services/ai/human/shared/historyRenderers.ts new file mode 100644 index 00000000..ec18e134 --- /dev/null +++ b/src/main/services/ai/human/shared/historyRenderers.ts @@ -0,0 +1,191 @@ +import { renderInlineList } from '../../utils/render' +import type { HumanLoadedHistoryContext, HumanRoundRecord } from '../types' +import { renderWatchHintForBrain } from '../prompts/watchHints' + +const renderQuotedNames = (items: readonly string[]): string => + renderInlineList( + items.map((item) => `\`${item}\``), + { + separator: '、', + fallback: 'none' + } + ) + +const renderDecisionNextAction = (record: HumanRoundRecord): string => { + switch (record.decision.nextAction.action) { + case 'learning': + return `- 你准备下一步做：准备学习 ${renderQuotedNames( + record.decision.nextAction.wantLearnSkillIds + )}。` + case 'hand': + return `- 你准备下一步做：准备进入 hand，${ + record.decision.nextAction.toolUsePlan.mode === 'parallel' + ? `以 parallel 模式并行执行 ${renderQuotedNames( + record.decision.nextAction.toolUsePlan.calls.map((call) => call.toolName) + )}。` + : `以 serial 模式按顺序执行 ${renderQuotedNames( + record.decision.nextAction.toolUsePlan.calls.map((call) => call.toolName) + )}。` + }` + case 'mouth': + return '- 你准备下一步做：准备进入 mouth 组织最终交付。' + } +} + +const renderRoundDecision = (record: HumanRoundRecord): string => + ` +- 你的判断：${record.decision.decision.summary} +- 你这样判断的原因：${record.decision.decision.reason} +- 你当时期待的结果：${record.decision.decision.expectedOutcome} +- ${record.decision.readyToDeliver ? '你当时认为已经可以交付。' : '你当时认为还不能直接交付。'} +${renderDecisionNextAction(record)} +`.trim() + +const renderRoundExecution = ( + record: HumanRoundRecord, + detail: 'focus' | 'chain' | 'loaded' +): string => { + switch (record.execution.kind) { + case 'pre_execution_blocked': + return '- 没有进入真实执行，系统在执行前阻断了这轮动作。' + case 'learning': + return '- 进入 learning，按请求尝试补齐外置技能。' + case 'hand': + return ` +- 进入 hand，执行模式是 ${record.execution.toolUsePlan.mode}。 +- ${ + record.execution.toolUsePlan.mode === 'parallel' + ? `实际并行执行了 ${renderQuotedNames(record.execution.toolUsePlan.calls.map((call) => call.toolName))}。` + : `实际按顺序执行了 ${renderQuotedNames(record.execution.toolUsePlan.calls.map((call) => call.toolName))}。` + } +${detail !== 'chain' ? `- 本轮计划包含 ${record.execution.toolUsePlan.calls.length} 个工具调用。` : ''} +` + .trim() + .replace(/\n\n+/g, '\n') + case 'mouth': + return '- 进入 mouth，组织最终说明。' + } +} + +const renderRoundResult = (record: HumanRoundRecord): string => { + switch (record.execution.kind) { + case 'pre_execution_blocked': + return renderWatchHintForBrain(record.execution.watchHint) + case 'learning': + return ` +- 已加载的技能：${renderQuotedNames(record.execution.learningOutcome.loadedSkillIds)} +- 已跳过的重复技能：${renderQuotedNames(record.execution.learningOutcome.skippedDuplicateSkillIds)} +- 未识别的技能：${renderQuotedNames(record.execution.learningOutcome.skippedUnknownSkillIds)} +- 当前 active tool 数：${String(record.execution.learningOutcome.activeToolCount)} +${record.execution.watchHint ? `- 收敛提示：${renderWatchHintForBrain(record.execution.watchHint)}` : ''} +` + .trim() + .replace(/\n\n+/g, '\n') + case 'hand': { + const toolResultLines = record.execution.toolResults + .map((result) => + [ + `- \`${result.toolName}\`：${result.summary}`, + result.dataPreview ? `结果：${result.dataPreview}` : '', + result.diagnosticText ? `诊断：${result.diagnosticText}` : '' + ] + .filter(Boolean) + .join('；') + ) + .join('\n') + const artifactLine = + record.execution.artifactRefs && record.execution.artifactRefs.length > 0 + ? `- 产物：${renderInlineList( + record.execution.artifactRefs.map((artifact) => + artifact.kind === 'file' + ? `\`${artifact.fileName}\`` + : `draft review ${artifact.review.reviewId}` + ), + { + separator: '、', + fallback: 'none' + } + )}` + : '' + const watchHintLine = record.execution.watchHint + ? `- 收敛提示：${renderWatchHintForBrain(record.execution.watchHint)}` + : '' + + return [toolResultLines, artifactLine, watchHintLine].filter(Boolean).join('\n') + } + case 'mouth': + return ` +- 输出摘要：${record.execution.summary} +${record.execution.completed ? '- 任务已完成。' : '- 任务未完成。'} +`.trim() + } +} + +const renderRoundRecord = ( + record: HumanRoundRecord, + detail: 'focus' | 'chain' | 'loaded' +): string => + ` +- 第 ${record.roundId} 轮 + +决策了什么： +${renderRoundDecision(record)} + +做了什么： +${renderRoundExecution(record, detail)} + +结果是什么： +${renderRoundResult(record)} +`.trim() + +export const renderRoundSection = (input: { + records: HumanRoundRecord[] + detail: 'focus' | 'chain' | 'loaded' +}): string => + input.records.length > 0 + ? input.records.map((record) => renderRoundRecord(record, input.detail)).join('\n\n') + : 'none' + +export const renderLoadedHistoryText = ( + loadedHistoryContext: HumanLoadedHistoryContext | undefined +): string => + loadedHistoryContext + ? ` +过滤条件：roundIds=${renderInlineList((loadedHistoryContext.filters.roundIds || []).map(String), { + separator: '、', + fallback: 'none' + })}；executionKinds=${renderInlineList(loadedHistoryContext.filters.executionKinds || [], { + separator: '、', + fallback: 'none' + })}；toolNames=${renderInlineList(loadedHistoryContext.filters.toolNames || [], { + separator: '、', + fallback: 'none' + })}；watchHintCodes=${renderInlineList(loadedHistoryContext.filters.watchHintCodes || [], { + separator: '、', + fallback: 'none' + })}；limit=${String(loadedHistoryContext.filters.limit || loadedHistoryContext.rounds.length)} + +${renderRoundSection({ + records: loadedHistoryContext.rounds, + detail: 'loaded' +})} +` + .trim() + : 'none' + +export const renderUnifiedHistoryText = (input: { + roundHistory: HumanRoundRecord[] + loadedHistoryContext: HumanLoadedHistoryContext | undefined +}): string => + ` +过程历史： +${renderRoundSection({ + records: input.roundHistory, + detail: 'loaded' +})} + +主动回看历史： +${renderLoadedHistoryText(input.loadedHistoryContext)} +` + .trim() + .replace(/\n{3,}/g, '\n\n') diff --git a/tests/ai/human/live/human-docx-cause-remedy-reading-live.test.ts b/tests/ai/human/live/human-docx-cause-remedy-reading-live.test.ts new file mode 100644 index 00000000..85830510 --- /dev/null +++ b/tests/ai/human/live/human-docx-cause-remedy-reading-live.test.ts @@ -0,0 +1,121 @@ +import { describe, it } from 'vitest' +import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig' +import { createHumanTestRunner } from '../../../harness/backend/humanHarness' +import { runBackendCase } from '../../../harness/backend/run-backend-case' +import { + buildSelectedDocxFixtureReferences, + DOCX_CROSS_FILE_FIXTURE_PATHS, + HUMAN_DOCX_CAUSE_REMEDY_READING_USER_INPUT, + resolveSelectedDocxFixtures +} from '../../../harness/backend/aiLiveTestData' +import { + collectStartedToolStringArgs, + filterAiRunEventsByType, + runWithStructuredOutputRetry +} from '../testUtils' + +const openedExpectedDoc = ( + openedSourcePaths: readonly string[], + expectedRelativePath: string, + expectedName: string +): boolean => + openedSourcePaths.some( + (sourcePath) => + sourcePath.endsWith(expectedRelativePath) || sourcePath.endsWith(`/${expectedName}`) + ) + +describe('ai human live backend', () => { + it('human-docx-cause-remedy-reading-live', async () => { + await runWithStructuredOutputRetry(async () => { + await runBackendCase({ + caseId: 'human-docx-cause-remedy-reading-live', + harnessMode: 'live', + domain: 'ai', + suite: 'human', + groups: ['live'], + description: '验证 live 模式下 human 会基于两份已选 docx 准确回答过拟合与反走样的成因和处理思路。', + fixtures: DOCX_CROSS_FILE_FIXTURE_PATHS, + runMode: 'single', + runtimeModelName: hasLiveLlmConfigInEnv('primary') + ? loadLiveLlmConfigFromEnv('primary').model_name + : null, + execute: async (runtime) => { + if (!hasLiveLlmConfigInEnv('primary')) { + return { + skipped: true, + output: null, + fixtureDocs: [], + events: [] + } + } + + await runtime.seedLlmConfig(loadLiveLlmConfigFromEnv('primary')) + const fixtureDocs = resolveSelectedDocxFixtures(runtime) + const runner = createHumanTestRunner(runtime, { + selectedReferences: buildSelectedDocxFixtureReferences(runtime) + }) + const output = await runner.run({ + taskId: runtime.taskId, + userInput: HUMAN_DOCX_CAUSE_REMEDY_READING_USER_INPUT + }) + + return { + skipped: false, + output, + fixtureDocs, + events: runtime.eventCollector.snapshot() + } + }, + assert: async (result, runtime) => { + if (result.skipped) { + runtime.assertThat(true, { + name: 'live-config-not-provided', + message: '未提供 live 模型配置，已跳过 human live docx cause/remedy reading case。' + }) + return + } + + const toolResults = filterAiRunEventsByType(result.events, 'tool_call_result') + const toolNames = toolResults.map((event) => event.toolName) + const openedSourcePaths = collectStartedToolStringArgs( + result.events, + 'docx_open', + 'sourcePath' + ) + const deliveryText = `${result.output?.summary || ''}\n${result.output?.deliveryMarkdown || ''}` + + runtime.assertThat( + result.output?.status === 'completed' && + result.fixtureDocs.every((fixtureDoc) => + openedExpectedDoc(openedSourcePaths, fixtureDoc.relativePath, fixtureDoc.name) + ) && + toolNames.some( + (toolName) => + toolName === 'docx_search' || + toolName === 'docx_outline' || + toolName === 'docx_get_block' + ) && + !toolNames.includes('docx_apply_ops') && + !toolNames.includes('docx_save') && + /(过拟合|过度拟合)/.test(deliveryText) && + /(记住过多训练数据中的特征值|训练时间过久)/.test(deliveryText) && + /(数据增强|未使用过的数据|验证数据)/.test(deliveryText) && + /反走样/.test(deliveryText) && + /(采样频率低于信号频率|像素.*边界)/.test(deliveryText) && + /(低通滤波|多个采样点|多个像素点|多次采样)/.test(deliveryText), + { + name: 'human-live-reads-both-selected-docx-and-explains-cause-and-remedy', + message: 'live human 读 case 应打开两份已选 docx，并准确回答过拟合与反走样的成因和处理思路。', + data: { + openedSourcePaths, + toolNames, + summary: result.output?.summary, + deliveryMarkdown: result.output?.deliveryMarkdown + } + } + ) + } + }) + }) + }, 600_000) +}) diff --git a/tests/ai/human/live/human-docx-contextual-comparison-draft-live.test.ts b/tests/ai/human/live/human-docx-contextual-comparison-draft-live.test.ts new file mode 100644 index 00000000..b6d4f4ad --- /dev/null +++ b/tests/ai/human/live/human-docx-contextual-comparison-draft-live.test.ts @@ -0,0 +1,160 @@ +import fs from 'node:fs/promises' +import { describe, it } from 'vitest' +import type { AiRunEvent } from '../../../../src/shared/types' +import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig' +import { createHumanTestRunner } from '../../../harness/backend/humanHarness' +import { runBackendCase } from '../../../harness/backend/run-backend-case' +import { + buildSelectedDocxFixtureReferences, + DOCX_CROSS_FILE_FIXTURE_PATHS, + hasUnchangedFileBytes, + HUMAN_DOCX_CONTEXTUAL_COMPARISON_DRAFT_USER_INPUT, + resolveSelectedDocxFixtures +} from '../../../harness/backend/aiLiveTestData' +import { + collectDocxApplyOpsContentKinds, + collectStartedToolStringArgs, + filterAiRunEventsByType, + runWithStructuredOutputRetry +} from '../testUtils' + +const openedExpectedDoc = ( + openedSourcePaths: readonly string[], + expectedRelativePath: string, + expectedName: string +): boolean => + openedSourcePaths.some( + (sourcePath) => + sourcePath.endsWith(expectedRelativePath) || sourcePath.endsWith(`/${expectedName}`) + ) + +const hasTargetReadBeforeApply = (events: readonly AiRunEvent[]): boolean => { + const firstApplyIndex = events.findIndex( + (event) => event.type === 'tool_call_started' && event.toolName === 'docx_apply_ops' + ) + if (firstApplyIndex <= 0) { + return false + } + + return events.slice(0, firstApplyIndex).some((event) => { + if (event.type !== 'tool_call_started') { + return false + } + if (event.toolName === 'docx_search') { + const query = event.input.query + return typeof query === 'string' && query.includes('反走样') + } + return event.toolName === 'docx_outline' || event.toolName === 'docx_get_block' + }) +} + +describe('ai human live backend', () => { + it('human-docx-contextual-comparison-draft-live', async () => { + await runWithStructuredOutputRetry(async () => { + await runBackendCase({ + caseId: 'human-docx-contextual-comparison-draft-live', + harnessMode: 'live', + domain: 'ai', + suite: 'human', + groups: ['live'], + description: '验证 live 模式下 human 会在图形学相关位置后补写说明和对照表，并生成预览。', + fixtures: DOCX_CROSS_FILE_FIXTURE_PATHS, + runMode: 'single', + runtimeModelName: hasLiveLlmConfigInEnv('primary') + ? loadLiveLlmConfigFromEnv('primary').model_name + : null, + execute: async (runtime) => { + if (!hasLiveLlmConfigInEnv('primary')) { + return { + skipped: true, + output: null, + fixtureDocs: [], + originalFixtureBytes: [], + events: [] + } + } + + await runtime.seedLlmConfig(loadLiveLlmConfigFromEnv('primary')) + const fixtureDocs = resolveSelectedDocxFixtures(runtime) + const originalFixtureBytes = await Promise.all( + fixtureDocs.map(async (fixtureDoc) => ({ + name: fixtureDoc.name, + relativePath: fixtureDoc.relativePath, + workspacePath: fixtureDoc.workspacePath, + bytes: await fs.readFile(fixtureDoc.workspacePath) + })) + ) + const runner = createHumanTestRunner(runtime, { + selectedReferences: buildSelectedDocxFixtureReferences(runtime) + }) + const output = await runner.run({ + taskId: runtime.taskId, + userInput: HUMAN_DOCX_CONTEXTUAL_COMPARISON_DRAFT_USER_INPUT + }) + + return { + skipped: false, + output, + fixtureDocs, + originalFixtureBytes, + events: runtime.eventCollector.snapshot() + } + }, + assert: async (result, runtime) => { + if (result.skipped) { + runtime.assertThat(true, { + name: 'live-config-not-provided', + message: '未提供 live 模型配置，已跳过 human live contextual docx comparison draft case。' + }) + return + } + + const events = result.events + const toolResults = filterAiRunEventsByType(events, 'tool_call_result') + const toolNames = toolResults.map((event) => event.toolName) + const openedSourcePaths = collectStartedToolStringArgs(events, 'docx_open', 'sourcePath') + const changedContentKinds = collectDocxApplyOpsContentKinds(events) + const docxArtifacts = + result.output?.artifacts?.filter( + (artifact) => + artifact.kind === 'file' && + artifact.capability === 'docx' && + (artifact.artifactRole === 'preview' || artifact.artifactRole === 'draft') + ) || [] + const unchangedFixtures = await Promise.all( + result.originalFixtureBytes.map(async (fixture) => + await hasUnchangedFileBytes(fixture.workspacePath, fixture.bytes) + ) + ) + + runtime.assertThat( + result.output?.status === 'completed' && + result.fixtureDocs.every((fixtureDoc) => + openedExpectedDoc(openedSourcePaths, fixtureDoc.relativePath, fixtureDoc.name) + ) && + hasTargetReadBeforeApply(events) && + toolNames.includes('docx_apply_ops') && + toolNames.includes('docx_render_preview') && + !toolNames.includes('docx_save') && + docxArtifacts.length > 0 && + changedContentKinds.includes('paragraph') && + changedContentKinds.includes('table') && + unchangedFixtures.every(Boolean), + { + name: 'human-live-contextual-docx-write-produces-preview-without-overwriting-source', + message: 'live human 定位补写 case 应在读取定位后补写说明和表格，并仅生成预览稿。', + data: { + openedSourcePaths, + toolNames, + changedContentKinds, + summary: result.output?.summary, + deliveryMarkdown: result.output?.deliveryMarkdown, + artifacts: result.output?.artifacts + } + } + ) + } + }) + }) + }, 600_000) +}) diff --git a/tests/ai/human/live/human-kb-guided-docx-comparison-draft-live.test.ts b/tests/ai/human/live/human-kb-guided-docx-comparison-draft-live.test.ts new file mode 100644 index 00000000..c91303e0 --- /dev/null +++ b/tests/ai/human/live/human-kb-guided-docx-comparison-draft-live.test.ts @@ -0,0 +1,128 @@ +import path from 'node:path' +import { describe, it } from 'vitest' +import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig' +import { createHumanTestRunner } from '../../../harness/backend/humanHarness' +import { runBackendCase } from '../../../harness/backend/run-backend-case' +import { + hasUnchangedFileBytes, + HUMAN_KB_GUIDED_DOCX_COMPARISON_DRAFT_USER_INPUT, + seedKbGuidedDocxKb, + seedKbGuidedDocxWorkspace +} from '../../../harness/backend/aiLiveTestData' +import { + collectDocxApplyOpsContentKinds, + collectStartedToolStringArgs, + filterAiRunEventsByType, + hasKbToolCall, + runWithStructuredOutputRetry +} from '../testUtils' + +describe('ai human live backend', () => { + it('human-kb-guided-docx-comparison-draft-live', async () => { + await runWithStructuredOutputRetry(async () => { + await runBackendCase({ + caseId: 'human-kb-guided-docx-comparison-draft-live', + harnessMode: 'live', + domain: 'ai', + suite: 'human', + groups: ['live'], + description: '验证 live 模式下 human 会先用 KB 缩圈定位课件，再继续对目标 docx 做预览型补写。', + fixtures: [], + runMode: 'single', + runtimeModelName: hasLiveLlmConfigInEnv('primary') + ? loadLiveLlmConfigFromEnv('primary').model_name + : null, + execute: async (runtime) => { + if (!hasLiveLlmConfigInEnv('primary')) { + return { + skipped: true, + output: null, + workspaceSeed: null, + events: [] + } + } + + await runtime.seedLlmConfig(loadLiveLlmConfigFromEnv('primary')) + const workspaceSeed = await seedKbGuidedDocxWorkspace(runtime) + await seedKbGuidedDocxKb(runtime, workspaceSeed) + + const runner = createHumanTestRunner(runtime) + const output = await runner.run({ + taskId: runtime.taskId, + userInput: HUMAN_KB_GUIDED_DOCX_COMPARISON_DRAFT_USER_INPUT + }) + + return { + skipped: false, + output, + workspaceSeed, + events: runtime.eventCollector.snapshot() + } + }, + assert: async (result, runtime) => { + if (result.skipped || !result.workspaceSeed) { + runtime.assertThat(true, { + name: 'live-config-not-provided', + message: '未提供 live 模型配置，已跳过 human live KB-guided docx comparison draft case。' + }) + return + } + + const events = result.events + const toolResults = filterAiRunEventsByType(events, 'tool_call_result') + const toolNames = toolResults.map((event) => event.toolName) + const openedSourcePaths = collectStartedToolStringArgs(events, 'docx_open', 'sourcePath') + const changedContentKinds = collectDocxApplyOpsContentKinds(events) + const docxArtifacts = + result.output?.artifacts?.filter( + (artifact) => + artifact.kind === 'file' && + artifact.capability === 'docx' && + (artifact.artifactRole === 'preview' || artifact.artifactRole === 'draft') + ) || [] + const deepLearningUnchanged = await hasUnchangedFileBytes( + result.workspaceSeed.deepLearningPath, + result.workspaceSeed.originalDeepLearningBytes + ) + const graphicsUnchanged = await hasUnchangedFileBytes( + result.workspaceSeed.graphicsPath, + result.workspaceSeed.originalGraphicsBytes + ) + const deliveryText = `${result.output?.summary || ''}\n${result.output?.deliveryMarkdown || ''}` + + runtime.assertThat( + result.output?.status === 'completed' && + hasKbToolCall(events) && + toolNames.includes('docx_apply_ops') && + toolNames.includes('docx_render_preview') && + !toolNames.includes('docx_save') && + docxArtifacts.length > 0 && + changedContentKinds.includes('paragraph') && + changedContentKinds.includes('table') && + openedSourcePaths.some( + (sourcePath) => + sourcePath.endsWith(result.workspaceSeed.graphicsRelativePath) || + sourcePath.endsWith(`/${path.basename(result.workspaceSeed.graphicsRelativePath)}`) + ) && + /(过拟合|过度拟合)/.test(deliveryText) && + /反走样/.test(deliveryText) && + deepLearningUnchanged && + graphicsUnchanged, + { + name: 'human-live-kb-guided-docx-write-uses-kb-and-preserves-source-files', + message: 'live human KB-guided 写 case 应先触发 KB 缩圈，再对目标 docx 生成预览稿且不覆盖原文件。', + data: { + openedSourcePaths, + toolNames, + changedContentKinds, + summary: result.output?.summary, + deliveryMarkdown: result.output?.deliveryMarkdown, + artifacts: result.output?.artifacts + } + } + ) + } + }) + }) + }, 600_000) +}) diff --git a/tests/ai/human/live/human-risk-report-draft-live.test.ts b/tests/ai/human/live/human-risk-report-draft-live.test.ts index df92f8ca..1d39796e 100644 --- a/tests/ai/human/live/human-risk-report-draft-live.test.ts +++ b/tests/ai/human/live/human-risk-report-draft-live.test.ts @@ -83,8 +83,12 @@ describe('ai human live backend', () => { toolNames.includes('docx_render_preview') && docxArtifacts.length > 0 && templateUnchanged && - /(待跟进|合规签署)/.test(deliveryText) && - /Alice/.test(deliveryText), + /风险结论/.test(deliveryText) && + /负责人(说明)?/.test(deliveryText) && + /Weekly Risk Summary\.md/.test(deliveryText) && + /Owner Note\.md/.test(deliveryText) && + /(预览|preview)/i.test(deliveryText) && + /(未覆盖原文件|原文件未被覆盖|未覆盖原件)/.test(deliveryText), { name: 'human-live-completes-risk-report-draft-through-real-tool-chain', message: 'live human complex case 应通过 terminal、KB 与 docx 真实工具链完成周报草稿任务。', diff --git a/tests/ai/human/testUtils.ts b/tests/ai/human/testUtils.ts index 77061b62..63ac920a 100644 --- a/tests/ai/human/testUtils.ts +++ b/tests/ai/human/testUtils.ts @@ -4,7 +4,7 @@ import type { HumanStableMemoryPatch, RuntimeSkillId } from '../../../src/main/services/ai/human/types' -import type { AiRunEvent, JsonObject, JsonValue } from '../../../src/shared/types' +import type { AiRunEvent, JsonArray, JsonObject, JsonValue } from '../../../src/shared/types' export const filterAiRunEventsByType = ( events: readonly AiRunEvent[], @@ -98,6 +98,83 @@ export const filterToolCallStartedEventsByName = ( ): Extract[] => filterAiRunEventsByType(events, 'tool_call_started').filter((event) => event.toolName === toolName) +export const collectStartedToolInputs = ( + events: readonly AiRunEvent[], + toolName: string +): JsonObject[] => filterToolCallStartedEventsByName(events, toolName).map((event) => event.input) + +export const collectStartedToolStringArgs = ( + events: readonly AiRunEvent[], + toolName: string, + argName: string +): string[] => + collectStartedToolInputs(events, toolName) + .map((input) => input[argName]) + .filter((value): value is string => typeof value === 'string') + +export const hasKbToolCall = (events: readonly AiRunEvent[]): boolean => + filterAiRunEventsByType(events, 'tool_call_result').some((event) => event.toolName.startsWith('kb_')) + +const isJsonObject = (value: JsonValue | undefined): value is JsonObject => + typeof value === 'object' && value !== null && !Array.isArray(value) + +const isJsonArray = (value: JsonValue | undefined): value is JsonArray => Array.isArray(value) + +const collectDocxBlockKindsFromValue = ( + value: JsonValue | undefined, + kinds: Set<'paragraph' | 'table'> +): void => { + if (isJsonArray(value)) { + value.forEach((item) => collectDocxBlockKindsFromValue(item, kinds)) + return + } + + if (!isJsonObject(value)) { + return + } + + const kind = value.kind + if (kind === 'paragraph' || kind === 'table') { + kinds.add(kind) + } + + collectDocxBlockKindsFromValue(value.content, kinds) + collectDocxBlockKindsFromValue(value.ops, kinds) + collectDocxBlockKindsFromValue(value.rows, kinds) +} + +export const collectDocxApplyOpsContentKinds = ( + events: readonly AiRunEvent[] +): Array<'paragraph' | 'table'> => { + const kinds = new Set<'paragraph' | 'table'>() + collectStartedToolInputs(events, 'docx_apply_ops').forEach((input) => { + collectDocxBlockKindsFromValue(input, kinds) + }) + return [...kinds] +} + +const isStructuredOutputEmptyResponseError = (error: unknown): boolean => + error instanceof Error && error.message.includes('structured_output_empty_response') + +export const runWithStructuredOutputRetry = async ( + action: () => Promise, + maxAttempts = 2 +): Promise => { + let lastError: unknown = null + for (let attempt = 1; attempt <= maxAttempts; attempt += 1) { + try { + await action() + return + } catch (error) { + lastError = error + if (!isStructuredOutputEmptyResponseError(error) || attempt >= maxAttempts) { + throw error + } + } + } + throw lastError +} + export const readToolCallResultPreview = ( events: readonly AiRunEvent[], toolName: string, diff --git a/tests/ai/tool/hand/hand-parallel-safety-contract.test.ts b/tests/ai/tool/hand/hand-parallel-safety-contract.test.ts index e0c5b737..2646a57f 100644 --- a/tests/ai/tool/hand/hand-parallel-safety-contract.test.ts +++ b/tests/ai/tool/hand/hand-parallel-safety-contract.test.ts @@ -126,18 +126,182 @@ describe('ai tool hand backend', () => { } }) + const safeLimitState = await handNode({ + ...safeState, + skills: { + learnedSkillIds: ['docx-read'] + }, + execution: { + ...safeState.execution, + currentDecision: { + decision: { + summary: '执行 6 个只读并行调用', + reason: '验证新的并行上限允许 6 个安全调用。', + expectedOutcome: '六个 read_only 且 parallelSafe 的调用全部真正执行。' + }, + readyToDeliver: false, + nextAction: { + action: 'hand', + toolUsePlan: { + mode: 'parallel', + calls: [ + { + toolName: 'docx_outline', + args: { + docId: openResult.docId, + maxBlocks: 20 + } + }, + { + toolName: 'docx_search', + args: { + docId: openResult.docId, + query: 'Safety', + mode: 'literal' + } + }, + { + toolName: 'docx_search', + args: { + docId: openResult.docId, + query: 'Risk', + mode: 'literal' + } + }, + { + toolName: 'docx_search', + args: { + docId: openResult.docId, + query: 'monitor', + mode: 'literal' + } + }, + { + toolName: 'docx_search', + args: { + docId: openResult.docId, + query: 'approval', + mode: 'literal' + } + }, + { + toolName: 'docx_search', + args: { + docId: openResult.docId, + query: 'edge', + mode: 'literal' + } + } + ] + } + } + } + } + }) + + const toolStartedBeforeLimitExceeded = filterEventsByType( + runtime.eventCollector.snapshot(), + 'tool_call_started' + ).length + + const limitExceededState = await handNode({ + ...safeLimitState, + skills: { + learnedSkillIds: ['docx-read'] + }, + execution: { + ...safeLimitState.execution, + currentDecision: { + decision: { + summary: '执行 7 个只读并行调用', + reason: '验证超过并行上限时会直接回到 brain。', + expectedOutcome: '写入 parallel_limit_exceeded watchHint，且不继续执行工具。' + }, + readyToDeliver: false, + nextAction: { + action: 'hand', + toolUsePlan: { + mode: 'parallel', + calls: [ + { + toolName: 'docx_outline', + args: { + docId: openResult.docId, + maxBlocks: 20 + } + }, + { + toolName: 'docx_search', + args: { + docId: openResult.docId, + query: 'Safety', + mode: 'literal' + } + }, + { + toolName: 'docx_search', + args: { + docId: openResult.docId, + query: 'Risk', + mode: 'literal' + } + }, + { + toolName: 'docx_search', + args: { + docId: openResult.docId, + query: 'monitor', + mode: 'literal' + } + }, + { + toolName: 'docx_search', + args: { + docId: openResult.docId, + query: 'approval', + mode: 'literal' + } + }, + { + toolName: 'docx_search', + args: { + docId: openResult.docId, + query: 'edge', + mode: 'literal' + } + }, + { + toolName: 'docx_search', + args: { + docId: openResult.docId, + query: 'cases', + mode: 'literal' + } + } + ] + } + } + } + } + }) + + const toolStartedAfterLimitExceeded = filterEventsByType( + runtime.eventCollector.snapshot(), + 'tool_call_started' + ).length + const toolStartedBeforeUnsafe = filterEventsByType( runtime.eventCollector.snapshot(), 'tool_call_started' ).length const unsafeState = await handNode({ - ...safeState, + ...limitExceededState, skills: { learnedSkillIds: ['docx-read', 'docx-write', 'docx-delivery'] }, execution: { - ...safeState.execution, + ...limitExceededState.execution, currentDecision: { decision: { summary: '并行写入和 validate', @@ -183,7 +347,11 @@ describe('ai tool hand backend', () => { return { safeState, + safeLimitState, + limitExceededState, unsafeState, + toolStartedBeforeLimitExceeded, + toolStartedAfterLimitExceeded, toolStartedBeforeUnsafe, toolStartedAfterUnsafe, events: runtime.eventCollector.snapshot() @@ -201,6 +369,33 @@ describe('ai tool hand backend', () => { } ) + runtime.assertThat( + result.safeLimitState.execution.lastToolResults?.length === 6 && + result.safeLimitState.execution.lastToolResults.every((item) => item.code === 'ok') && + !result.safeLimitState.execution.watchHint, + { + name: 'six-safe-parallel-calls-are-allowed', + message: '6 个只读且 parallelSafe 的工具调用应真正执行并写回结果。', + data: result.safeLimitState.execution.lastToolResults + } + ) + + runtime.assertThat( + result.limitExceededState.execution.watchHint?.code === 'hand_plan_invalid' && + result.limitExceededState.execution.watchHint.detail.reason === 'parallel_limit_exceeded' && + result.limitExceededState.execution.watchHint.detail.limit === 6 && + result.toolStartedAfterLimitExceeded === result.toolStartedBeforeLimitExceeded, + { + name: 'seven-parallel-calls-stop-before-execution', + message: '超过 6 个并行工具调用时应直接返回 brain 并写入 parallel_limit_exceeded。', + data: { + watchHint: result.limitExceededState.execution.watchHint, + toolStartedBeforeLimitExceeded: result.toolStartedBeforeLimitExceeded, + toolStartedAfterLimitExceeded: result.toolStartedAfterLimitExceeded + } + } + ) + runtime.assertThat( result.unsafeState.execution.watchHint?.code === 'hand_plan_invalid' && result.unsafeState.execution.watchHint.detail.reason === 'parallel_requires_read_only' && diff --git a/tests/harness/backend/aiLiveTestData.ts b/tests/harness/backend/aiLiveTestData.ts index 077bde75..47a10a79 100644 --- a/tests/harness/backend/aiLiveTestData.ts +++ b/tests/harness/backend/aiLiveTestData.ts @@ -1,5 +1,6 @@ import fs from 'node:fs/promises' import path from 'node:path' +import type { SelectedReference } from '../../../src/shared/types' import { writeTestDocxFile } from './aiToolHarness' import type { BackendTestRuntime, SeedKbDocumentInput } from './test-types' @@ -9,6 +10,23 @@ export const HUMAN_FILE_RETENTION_USER_INPUT = export const HUMAN_RISK_REPORT_DRAFT_USER_INPUT = '我下午要把本周风险周报发出去。工作区里应该有一份上周用过的 Word 周报模板，但我忘了文件名。你先帮我找到那份模板，再按我刚才选的资料，把本周风险结论和负责人补进去，给我一个可预览的版本，先别覆盖原文件。最后告诉我你改了哪几处，依据分别来自哪里。' +export const DOCX_CROSS_FILE_FIXTURE_PATHS = [ + 'tests/fixtures/ai/docx/深度学习.docx', + 'tests/fixtures/ai/docx/图形学.docx' +] as const + +export const HUMAN_DOCX_CAUSE_REMEDY_READING_USER_INPUT = + '我做题老把这两个点混在一起。你就按我刚选的这两份资料，帮我说清楚：过拟合和反走样各自是什么、为什么会出现、一般怎么处理。别补充课外知识，哪份文档没写你就直接说没写。' + +export const HUMAN_DOCX_CONTEXTUAL_COMPARISON_DRAFT_USER_INPUT = + '《图形学》里“反走样”这一节我还是记不住。你参考我刚选的这两份 Word，直接在这一节最后补一小段更好理解的说明，再加个简单对照表，一列写反走样，一列写《深度学习》里的过拟合。直接给我可预览的版本就行，别改原文件，也不用先回我文字草稿。' + +export const HUMAN_KB_GUIDED_DOCX_COMPARISON_DRAFT_USER_INPUT = + '我工作区资料有点多，我忘了是哪份课件在讲反走样了。你先帮我找到那份，再参考讲过拟合的那份资料，直接在“反走样”那一节最后补一段更好记的说明，再加个简单对照表，一列写反走样，一列写过拟合。直接给我可预览的版本就行，别改原文件，也不用先回我文字草稿。' + +export const HUMAN_KB_GUIDED_DOCX_READING_USER_INPUT = + '我工作区里应该有两份课件分别讲过过拟合和反走样，但我忘了文件名。你先帮我找出来，再告诉我这两个问题一般怎么处理。' + export const FILE_RETENTION_DOCUMENTS: readonly SeedKbDocumentInput[] = [ { id: 'doc-retention', @@ -51,6 +69,22 @@ export type RiskReportWorkspaceSeed = { originalTemplateBytes: Uint8Array } +export type SelectedDocxFixture = { + sourcePath: string + workspacePath: string + relativePath: string + name: string +} + +export type KbGuidedDocxWorkspaceSeed = { + deepLearningPath: string + deepLearningRelativePath: string + graphicsPath: string + graphicsRelativePath: string + originalDeepLearningBytes: Uint8Array + originalGraphicsBytes: Uint8Array +} + export const seedFileRetentionKb = async (runtime: BackendTestRuntime): Promise => { await runtime.seedKb(FILE_RETENTION_DOCUMENTS) return FILE_RETENTION_DOCUMENTS.map((document) => document.id) @@ -61,6 +95,236 @@ export const seedRiskReportKb = async (runtime: BackendTestRuntime): Promise document.id) } +export const resolveSelectedDocxFixtures = ( + runtime: BackendTestRuntime, + fixturePaths: readonly string[] = DOCX_CROSS_FILE_FIXTURE_PATHS +): SelectedDocxFixture[] => + fixturePaths.map((fixturePath) => { + const workspacePath = runtime.getFixtureWorkspacePath(fixturePath) + return { + sourcePath: fixturePath, + workspacePath, + relativePath: path.relative(runtime.workspaceRoot, workspacePath), + name: path.basename(workspacePath) + } + }) + +export const buildSelectedDocxFixtureReferences = ( + runtime: BackendTestRuntime, + fixturePaths: readonly string[] = DOCX_CROSS_FILE_FIXTURE_PATHS +): SelectedReference[] => + resolveSelectedDocxFixtures(runtime, fixturePaths).map((fixture, index) => ({ + kind: 'file', + workspaceId: 'test-workspace', + nodeId: `fixture-docx-${String(index + 1)}`, + name: fixture.name, + relativePath: fixture.relativePath, + absolutePath: fixture.workspacePath, + fileExt: '.docx', + kbDocumentIds: [], + kbAvailability: 'not_indexed' + })) + +const copyDocxFixtureToWorkspaceRoot = async (input: { + runtime: BackendTestRuntime + sourcePath: string + targetName: string + summary: string +}): Promise => { + const absoluteSourcePath = path.resolve(input.sourcePath) + const targetPath = path.join(input.runtime.workspaceRoot, input.targetName) + await fs.cp(absoluteSourcePath, targetPath) + input.runtime.recordSeedWorkspaceFile({ + relativePath: input.targetName, + absolutePath: targetPath, + summary: input.summary, + metadata: { + format: 'docx' + } + }) + return targetPath +} + +export const seedKbGuidedDocxWorkspace = async ( + runtime: BackendTestRuntime +): Promise => { + const deepLearningRelativePath = '深度学习.docx' + const graphicsRelativePath = '图形学.docx' + const deepLearningPath = await copyDocxFixtureToWorkspaceRoot({ + runtime, + sourcePath: DOCX_CROSS_FILE_FIXTURE_PATHS[0], + targetName: deepLearningRelativePath, + summary: '目标课件：深度学习' + }) + const graphicsPath = await copyDocxFixtureToWorkspaceRoot({ + runtime, + sourcePath: DOCX_CROSS_FILE_FIXTURE_PATHS[1], + targetName: graphicsRelativePath, + summary: '目标课件：图形学' + }) + + const scheduleRelativePath = '课程安排.docx' + const schedulePath = await writeTestDocxFile(runtime.workspaceRoot, scheduleRelativePath, [ + { + type: 'heading', + text: '课程安排', + level: 1 + }, + { + type: 'paragraph', + text: '这里记录的是考试时间、教室和分组安排。' + } + ]) + runtime.recordSeedWorkspaceFile({ + relativePath: scheduleRelativePath, + absolutePath: schedulePath, + summary: '干扰项：课程安排', + metadata: { + format: 'docx' + } + }) + + const labNoteRelativePath = '机器学习实验记录.docx' + const labNotePath = await writeTestDocxFile(runtime.workspaceRoot, labNoteRelativePath, [ + { + type: 'heading', + text: '机器学习实验记录', + level: 1 + }, + { + type: 'paragraph', + text: '本周实验主要关注学习率和 batch size 的调参过程。' + } + ]) + runtime.recordSeedWorkspaceFile({ + relativePath: labNoteRelativePath, + absolutePath: labNotePath, + summary: '干扰项：机器学习实验记录', + metadata: { + format: 'docx' + } + }) + + const readingListRelativePath = '参考书目.md' + const readingListPath = path.join(runtime.workspaceRoot, readingListRelativePath) + const readingListText = '- 线性代数导引\n- 计算机图形学基础\n' + await fs.writeFile(readingListPath, readingListText, 'utf8') + runtime.recordSeedWorkspaceFile({ + relativePath: readingListRelativePath, + absolutePath: readingListPath, + summary: '干扰项：参考书目', + metadata: { + format: 'text', + text: readingListText + } + }) + + const progressRelativePath = '复习进度.txt' + const progressPath = path.join(runtime.workspaceRoot, progressRelativePath) + const progressText = '今天先看图形学，再看深度学习。\n' + await fs.writeFile(progressPath, progressText, 'utf8') + runtime.recordSeedWorkspaceFile({ + relativePath: progressRelativePath, + absolutePath: progressPath, + summary: '干扰项：复习进度', + metadata: { + format: 'text', + text: progressText + } + }) + + return { + deepLearningPath, + deepLearningRelativePath, + graphicsPath, + graphicsRelativePath, + originalDeepLearningBytes: await fs.readFile(deepLearningPath), + originalGraphicsBytes: await fs.readFile(graphicsPath) + } +} + +export const seedKbGuidedDocxKb = async ( + runtime: BackendTestRuntime, + workspaceSeed: KbGuidedDocxWorkspaceSeed +): Promise => { + const kbDocuments: readonly SeedKbDocumentInput[] = [ + { + id: 'doc-deep-learning', + name: '深度学习.docx', + filePath: workspaceSeed.deepLearningPath, + chunks: [ + { + id: 'chunk-deep-learning-1', + text: '过度拟合是模型记住了过多训练数据中的特征值，使训练后的模型只对这部分训练数据有效，不太具有展望性。' + }, + { + id: 'chunk-deep-learning-2', + text: '当训练时间过久时，会逐渐记住过多训练数据中的特征值，从而出现过度拟合。' + }, + { + id: 'chunk-deep-learning-3', + text: '缓解过度拟合的思路包括：使用训练时未使用过的数据做验证，以及通过数据增强把同一图像拆成不同亮度、旋转方向等多张图像来训练。' + } + ] + }, + { + id: 'doc-graphics', + name: '图形学.docx', + filePath: workspaceSeed.graphicsPath, + chunks: [ + { + id: 'chunk-graphics-1', + text: '反走样出现的原因是采样频率低于信号频率。' + }, + { + id: 'chunk-graphics-2', + text: '由于一个像素可能同时落在图形边界内和边界外，边缘会出现锯齿。' + }, + { + id: 'chunk-graphics-3', + text: '反走样的处理思路包括：在一个像素内模拟多个采样点，或者先把图形转换成信号后做低通滤波。' + } + ] + }, + { + id: 'doc-course-schedule', + name: '课程安排.docx', + filePath: path.join(runtime.workspaceRoot, '课程安排.docx'), + chunks: [ + { + id: 'chunk-course-schedule-1', + text: '课程安排只记录考试时间、教室和分组信息。' + } + ] + }, + { + id: 'doc-ml-lab', + name: '机器学习实验记录.docx', + filePath: path.join(runtime.workspaceRoot, '机器学习实验记录.docx'), + chunks: [ + { + id: 'chunk-ml-lab-1', + text: '实验记录主要关注学习率和 batch size 的调参过程。' + } + ] + }, + { + id: 'doc-reading-list', + name: '参考书目.md', + filePath: path.join(runtime.workspaceRoot, '参考书目.md'), + chunks: [ + { + id: 'chunk-reading-list-1', + text: '参考书目里只有课程参考资料列表，没有过拟合或反走样的讲解。' + } + ] + } + ] + + await runtime.seedKb(kbDocuments) + return kbDocuments.map((document) => document.id) +} + export const seedRiskReportWorkspace = async ( runtime: BackendTestRuntime ): Promise => { diff --git a/tests/temp/ai/human/live/human-kb-guided-docx-reading-live.test.ts b/tests/temp/ai/human/live/human-kb-guided-docx-reading-live.test.ts new file mode 100644 index 00000000..5608e7a3 --- /dev/null +++ b/tests/temp/ai/human/live/human-kb-guided-docx-reading-live.test.ts @@ -0,0 +1,94 @@ +import { describe, it } from 'vitest' +import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../../harness/backend/llmConfig' +import { createHumanTestRunner } from '../../../../harness/backend/humanHarness' +import { runBackendCase } from '../../../../harness/backend/run-backend-case' +import { + HUMAN_KB_GUIDED_DOCX_READING_USER_INPUT, + seedKbGuidedDocxKb, + seedKbGuidedDocxWorkspace +} from '../../../../harness/backend/aiLiveTestData' +import { + collectStartedToolStringArgs, + filterAiRunEventsByType, + hasKbToolCall, + runWithStructuredOutputRetry +} from '../../../../ai/human/testUtils' + +describe('temp ai human live backend', () => { + it('human-kb-guided-docx-reading-live', async () => { + await runWithStructuredOutputRetry(async () => { + await runBackendCase({ + caseId: 'human-kb-guided-docx-reading-live', + harnessMode: 'live', + domain: 'temp', + suite: 'ai', + groups: ['human', 'live'], + description: '观察 live 模式下 human 在工作区资料较多时，会如何用 KB 缩圈并回答过拟合与反走样的处理思路。', + fixtures: [], + runMode: 'single', + runtimeModelName: hasLiveLlmConfigInEnv('primary') + ? loadLiveLlmConfigFromEnv('primary').model_name + : null, + execute: async (runtime) => { + if (!hasLiveLlmConfigInEnv('primary')) { + return { + skipped: true, + output: null, + events: [] + } + } + + await runtime.seedLlmConfig(loadLiveLlmConfigFromEnv('primary')) + const workspaceSeed = await seedKbGuidedDocxWorkspace(runtime) + await seedKbGuidedDocxKb(runtime, workspaceSeed) + + const runner = createHumanTestRunner(runtime) + const output = await runner.run({ + taskId: runtime.taskId, + userInput: HUMAN_KB_GUIDED_DOCX_READING_USER_INPUT + }) + + return { + skipped: false, + output, + events: runtime.eventCollector.snapshot() + } + }, + assert: async (result, runtime) => { + if (result.skipped) { + runtime.assertThat(true, { + name: 'live-config-not-provided', + message: '未提供 live 模型配置，已跳过 temp human live KB-guided docx reading case。' + }) + return + } + + const toolResults = filterAiRunEventsByType(result.events, 'tool_call_result') + const deliveryText = `${result.output?.summary || ''}\n${result.output?.deliveryMarkdown || ''}` + const openedSourcePaths = collectStartedToolStringArgs( + result.events, + 'docx_open', + 'sourcePath' + ) + + runtime.assertThat( + result.output?.status === 'completed' && + hasKbToolCall(result.events) && + /(数据增强|未使用过的数据|验证数据)/.test(deliveryText) && + /(低通滤波|多个采样点|多个像素点|多次采样)/.test(deliveryText), + { + name: 'temp-human-live-kb-guided-reading-covers-kb-and-answer-facts', + message: 'temp KB-guided 读 case 应至少触发 KB，并回答过拟合与反走样的处理思路。', + data: { + openedSourcePaths, + toolNames: toolResults.map((event) => event.toolName), + summary: result.output?.summary, + deliveryMarkdown: result.output?.deliveryMarkdown + } + } + ) + } + }) + }) + }, 600_000) +}) diff --git a/tests/unit/ai/human-history-renderers.test.ts b/tests/unit/ai/human-history-renderers.test.ts new file mode 100644 index 00000000..1a015622 --- /dev/null +++ b/tests/unit/ai/human-history-renderers.test.ts @@ -0,0 +1,229 @@ +import { describe, expect, it } from 'vitest' +import type { AiRunContext } from '../../../src/main/services/ai/types' +import { + appendRoundRecord, + buildHandExecution, + buildLearningExecution +} from '../../../src/main/services/ai/human/memory/roundHistory' +import { buildBrainPrompt } from '../../../src/main/services/ai/human/prompts/brain' +import { buildMouthPrompt } from '../../../src/main/services/ai/human/prompts/mouth' +import { + renderRoundSection, + renderUnifiedHistoryText +} from '../../../src/main/services/ai/human/shared/historyRenderers' +import { buildReferenceScope } from '../../../src/main/services/ai/prompts/referenceScope' +import { createInitialHumanState } from '../../../src/main/services/ai/human/state/schema' +import type { HumanDecision } from '../../../src/main/services/ai/human/types' + +const createStubAiRunContext = (referenceScope: AiRunContext['referenceScope']): AiRunContext => ({ + runId: 'run-history-render', + referenceScope, + checkCancelled: () => {}, + chatStructured: async () => { + throw new Error('chatStructured should not be called in prompt unit tests') + }, + chatText: async () => { + throw new Error('chatText should not be called in prompt unit tests') + }, + sendEvent: () => {}, + requestApproval: async () => ({ + approved: false + }), + loadHistory: async () => ({ + recentTurnsText: '', + sessionSummaryText: '' + }), + trackDocxOpen: () => {}, + trackDocxChanges: () => {}, + resolveDocxOpenTarget: async () => { + throw new Error('resolveDocxOpenTarget should not be called in prompt unit tests') + }, + documentsStore: undefined as never, + embeddingService: undefined as never +}) + +const buildDecision = (input: { + summary: string + reason: string + expectedOutcome: string + nextAction: HumanDecision['nextAction'] +}): HumanDecision => ({ + decision: { + summary: input.summary, + reason: input.reason, + expectedOutcome: input.expectedOutcome + }, + readyToDeliver: false, + nextAction: input.nextAction +}) + +describe('ai unit human history renderers', () => { + it('renders shared history facts for both brain and mouth prompts', () => { + const initialState = createInitialHumanState({ + taskId: 'task-history-shared', + userInput: '请总结这次任务做了什么' + }) + + const learningState = { + ...initialState, + skills: { + learnedSkillIds: ['workspace-cli'] + }, + memory: { + ...initialState.memory, + history: appendRoundRecord({ + state: initialState, + decision: buildDecision({ + summary: '补齐 workspace 能力', + reason: '需要 terminal 工具', + expectedOutcome: '拿到 workspace 读取能力', + nextAction: { + action: 'learning', + wantLearnSkillIds: ['workspace-cli'] + } + }), + execution: buildLearningExecution({ + learningOutcome: { + requestedSkillIds: ['workspace-cli'], + loadedSkillIds: ['workspace-cli'], + skippedDuplicateSkillIds: [], + skippedUnknownSkillIds: [], + evictedSkillIds: [], + activeToolCount: 1 + } + }), + createdAt: 1 + }) + } + } + + const handState = { + ...learningState, + memory: { + ...learningState.memory, + history: appendRoundRecord({ + state: learningState, + decision: buildDecision({ + summary: '读取资料并生成预览', + reason: '已经具备需要的能力', + expectedOutcome: '拿到 docx 预览产物', + nextAction: { + action: 'hand', + toolUsePlan: { + mode: 'serial', + calls: [ + { + toolName: 'kb_hybrid_search_chunks', + args: { + queryText: 'approval denied' + } + }, + { + toolName: 'docx_render_preview', + args: { + format: 'html' + } + } + ] + } + } + }), + execution: buildHandExecution({ + toolUsePlan: { + mode: 'serial', + calls: [ + { + toolName: 'kb_hybrid_search_chunks', + args: { + queryText: 'approval denied' + } + }, + { + toolName: 'docx_render_preview', + args: { + format: 'html' + } + } + ] + }, + toolResults: [ + { + toolName: 'kb_hybrid_search_chunks', + ok: true, + code: 'ok', + data: { + items: [ + { + document_name: 'Weekly Risk Summary.md', + text: '审批未通过时原文件保留' + } + ] + } + }, + { + toolName: 'docx_render_preview', + ok: true, + code: 'ok', + data: { + previewPath: '/tmp/preview.docx', + summary: 'preview generated (html)' + } + } + ] + }), + createdAt: 2 + }) + } + } + + const state = { + ...handState, + memory: { + ...handState.memory, + history: { + ...handState.memory.history, + loadedHistoryContext: { + filters: { + executionKinds: ['learning'], + limit: 1 + }, + rounds: [handState.memory.history.roundHistory[0]] + } + } + } + } + + const focusHistory = renderRoundSection({ + records: [state.memory.history.roundHistory[0]], + detail: 'focus' + }) + const unifiedHistory = renderUnifiedHistoryText({ + roundHistory: state.memory.history.roundHistory, + loadedHistoryContext: state.memory.history.loadedHistoryContext + }) + + const brainPrompt = buildBrainPrompt({ + state, + ctx: createStubAiRunContext( + buildReferenceScope({ + workspace: { + workspaceId: 'workspace-1', + workspaceName: 'workspace', + rootPath: '/tmp/workspace-root' + } + }) + ) + }) + const mouthPrompt = buildMouthPrompt({ state }) + + expect(focusHistory).toContain('决策了什么：') + expect(focusHistory).toContain('做了什么：') + expect(focusHistory).toContain('结果是什么：') + expect(unifiedHistory).toContain('过程历史：') + expect(unifiedHistory).toContain('主动回看历史：') + expect(unifiedHistory).toContain('Weekly Risk Summary.md') + expect(unifiedHistory).toContain('preview generated (html)') + expect(brainPrompt.user).toContain(focusHistory) + expect(mouthPrompt.user).toContain(unifiedHistory) + }) +}) diff --git a/tests/unit/ai/mouth-ready-to-deliver.test.ts b/tests/unit/ai/mouth-ready-to-deliver.test.ts index 901bc521..82210e39 100644 --- a/tests/unit/ai/mouth-ready-to-deliver.test.ts +++ b/tests/unit/ai/mouth-ready-to-deliver.test.ts @@ -71,16 +71,16 @@ describe('ai unit mouth readyToDeliver guard', () => { }) }).system - expect(prompt).toContain('') - expect(prompt).toContain('') + expect(prompt).toContain('') expect(prompt).toContain('') - expect(prompt).toContain('final_decision 是 brain 已经确认的最终判断') - expect(prompt).toContain('runtime_artifacts 只允许准确引用已经给出的路径') - expect(prompt).toContain('你不会调用工具，也不会继续推进未完成步骤') - expect(prompt).toContain('不要把未来可能执行的工具动作写成“后续会自动完成”') - expect(prompt).toContain('你必须只输出最终 markdown 正文') - expect(prompt).not.toContain('"deliveryMarkdown"') - expect(prompt).not.toContain('"summary"') + expect(prompt).toContain('直接回答 task_goal') + expect(prompt).toContain('your_notes 优先') + expect(prompt).toContain('history：任务过程中真实发生过的决策、执行和结果') + expect(prompt).toContain('只输出最终 markdown 正文') + expect(prompt).toContain('标题、分节、列表都按需要使用，不固定') + expect(prompt).toContain('不原样照抄内部标签、block id、tool 名') + expect(prompt).not.toContain('# 处理结果') + expect(prompt).not.toContain('## 关键依据') }) it('keeps mouth user-context blocks data-only and moves explanations into system', () => { @@ -91,14 +91,21 @@ describe('ai unit mouth readyToDeliver guard', () => { }) }) - expect(result.system).toContain('final_decision：brain 已经确认的最终判断') - expect(result.system).toContain('默认使用最终用户答复的组织方式') + expect(result.system).toContain('task_goal：用户最初要你完成什么') + expect(result.system).toContain('已确认：当前稳定成立、可以直接告诉用户的事实') + expect(result.system).toContain('关键依据：支撑这些事实的证据') + expect(result.system).toContain('未决问题：仍会影响结论') expect(result.user).not.toContain('这是当前交付模式。') expect(result.user).not.toContain('这是用户这次原始诉求与限制。') expect(result.user).not.toContain('这是 brain 已经做出的最终决定。') expect(result.user).not.toContain('这是任务的稳定笔记') expect(result.user).not.toContain('这是最近一轮最相关的过程记录。') + expect(result.user).not.toContain('这是更早几轮的过程链。') expect(result.user).not.toContain('这是运行中已经产生的真实产物列表。') - expect(result.user).not.toContain('这是当前额外的收敛或纠偏提醒。') + expect(result.user).toContain('') + expect(result.user).not.toContain('') + expect(result.user).not.toContain('') + expect(result.user).not.toContain('') + expect(result.user).not.toContain('') }) })