Refine mouth delivery around task goals and history

This commit is contained in:
huangyuan committed 2026-04-09 18:26:19 +08:00
1 parent 2cef65ec7f
commit 117ce68e71
21 files changed
+1567 -323

No files matched your search

+2 -1
View File
@@ -3,7 +3,7 @@
## 1. 文档引用
### 1.1 AI相关
- 当涉及 AI 架构、主流程或 human-only runtime 时,必须阅读 `doc/ai-design/architecture.md`
- 当涉及 AI 实现、运行入口、跨层边界、LLM 执行入口、流式传输、runtime settings、运行数据、`execution.watchHint` / `toolUsePlan` / `execution.lastLearningOutcome` / `memory.history.roundHistory` / `history_recall` / `recent_progress` / `earlier_attempts` / `recalled_history` 契约或工作区 KB 范围时,必须阅读 `doc/ai-design/implementation-spec.md` 与 `doc/ai-design/data-definition-and-flow.md`
- 当涉及 AI 实现、运行入口、跨层边界、LLM 执行入口、流式传输、runtime settings、运行数据、`execution.watchHint` / `toolUsePlan` / `execution.lastLearningOutcome` / `memory.history.roundHistory` / `history_recall` / `mouth` 统一 `history` prompt 输入契约或工作区 KB 范围时,必须阅读 `doc/ai-design/implementation-spec.md` 与 `doc/ai-design/data-definition-and-flow.md`
- 当涉及 AI 最小运行样例、默认链路或想快速理解 human 各环节如何衔接时,必须阅读 `doc/ai-design/example.md`
- 当涉及 human 的 skill 分类、拆分边界、builtin/external 区分或当前 skill 基线时,必须阅读 `doc/ai-design/human-skill-design.md`
- 当涉及 prompt 的通用结构、输入组织、输出规则、约束表达或风格一致性时,必须阅读 `doc/ai-design/prompt-design.md`
@@ -16,6 +16,7 @@
### 1.3 测试相关
- 当涉及 AI 测试,必须阅读 `doc/test-plan/ai-test.md`
- 当涉及 human live 的 docx / KB 混合场景测试设计时,必须阅读 `doc/test-plan/ai-test.md`
- 当涉及 Workbench 测试时,必须阅读 `doc/test-plan/workbench-test.md`
- 当涉及 Settings 测试时,必须阅读 `doc/test-plan/settings-test.md`
- 当涉及测试用例编写,测试规范时,必须阅读 `doc/test.md`
+5 -4
View File
@@ -1100,20 +1100,21 @@ flowchart LR
flowchart LR
T[("task")]
SM[("memory.stable")]
PM[("memory.history.roundHistory")]
PM[("memory.history.roundHistory / loadedHistoryContext")]
CD[("execution.currentDecision")]
A[("execution.artifacts")]
WH[("execution.watchHint")]
M["mouth"]
R[("result")]
TR[("trace.steps")]
T -->|读取 taskId + userInput| M
SM -->|读取任务笔记| M
PM -->|读取上一轮完整闭环| M
PM -->|读取统一 history 事实视图| M
CD -->|读取当前最终决策| M
A -->|读取已收集的结构化 artifacts| M
WH -->|必要时整理为交付提示| M
M -->|输出 deliveryMarkdown;node 派生 summary、回写 completed result,并透传 artifacts| R
M -->|追加 mouth 执行轨迹| TR
```
- `mouth` 不再依赖 `recent_progress / earlier_attempts / recalled_history` 这类拆分块;统一基于 `roundHistory + loadedHistoryContext` 渲染单一 `history` 视图。
- `mouth` 的 prompt 输入仍以 `your_notes` 为优先事实来源;当 `your_notes` 不够具体时,再从统一 `history` 与 `runtime_artifacts` 补足用户回答所需细节,而不是依赖额外的 `watchHint` 说明。
+3 -3
View File
@@ -94,10 +94,10 @@ stateDiagram-v2
### 5.4 `mouth`
- 职责:把当前已经确定的状态整理成最终用户交付,并回写 `HumanResult`。
- 主要读取:`task`、`memory.stable`、`execution.currentDecision`、`execution.artifacts`。
- 职责:围绕 `task_goal` 把当前已经确定的状态组织成最终用户交付,并回写 `HumanResult`。
- 主要读取:`task`、`memory.stable`、`memory.history.roundHistory`、`memory.history.loadedHistoryContext`、`execution.currentDecision`、`execution.artifacts`。
- 主要回写:`result`、`execution.status`、`execution.stage`、`trace.steps`。
- 约束:模型输出必须通过 `ctx.chatText`;输出固定面向最终用户;若开启 `emitAnswerEvents`,增量文本通过 `answer_token` 发出,完成后通过 `answer_completed` 收口。
- 约束:模型输出必须通过 `ctx.chatText`;输出面向最终用户,优先直接完成用户原始诉求;可使用自由段落或列表组织,不要求固定报告模板;若开启 `emitAnswerEvents`,增量文本通过 `answer_token` 发出,完成后通过 `answer_completed` 收口。
## 6. 事件与交付
+3 -1
View File
@@ -99,13 +99,15 @@
- 验收:
- `WB-CORE-005-AC1` `terminal` 执行成功后,后续用户可见输出会体现关键命令结果,而不是只表现为“命令执行过”。
- `WB-CORE-005-AC2` `terminal` 输出不足或执行失败时,assistant 会明确说明无法从当前结果得到所需信息,而不是编造结论或文档内容。
- `WB-CORE-005-AC3` 用户可见输出优先回答用户原始诉求,不要求固定报告体;若引用执行结果,应使用用户能理解的表达,而不是暴露内部工具或过程标签。
#### WB-CORE-006 标准任务应能闭环到可交付结果
- 系统行为:当用户提供了足够上下文的标准任务,且当前不需要额外阻塞或确认时,workbench 应支持 AI 沿同一条任务链完成资料使用、必要执行和最终交付,而不是只停留在搜索结果、工具调用或过程提示。
- 系统行为:当用户提供了足够上下文的标准任务,且当前不需要额外阻塞或确认时,workbench 应支持 AI 沿同一条任务链完成资料使用、必要执行和最终交付,而不是只停留在搜索结果、工具调用或过程提示;最终收口应围绕用户原始诉求组织,而不是强制套用固定报告模板。
- 验收:
- `WB-CORE-006-AC1` 用户已提供足够上下文且无需额外阻塞时,系统能把标准任务推进到可消费的最终结果或明确产物,而不是停留在中间态。
- `WB-CORE-006-AC2` 最终结果会体现用户显式给定的资料范围与执行得到的关键信息,不会把范围、执行结果和最终交付割裂开。
- `WB-CORE-006-AC3` 最终结果默认直接回答用户显式提出的问题;若同时需要说明产物、依据或限制,也应以用户可理解的语言表达,而不是依赖内部 block id、tool 名或固定章节标题。
### 8.2 会话管理
+35 -5
View File
@@ -247,10 +247,10 @@
- 测试类型:`backend`
- 实现状态:`已实现`
- What:验证 `hand` 的并行工具执行与聚合守卫。
- What:验证 `hand` 的并行工具执行上限、只读安全约束与结果聚合守卫。
- Why:工具执行器本身必须在真实结果下稳定收敛。
- How:通过真实 `hand` 执行多工具调用并检查状态收敛。
- Oracle:并行工具结果被正确聚合,运行状态无串扰。
- How:通过真实 `hand` 执行安全并行批次、`6` 个只读安全并行批次、超过上限的 `7` 个并行批次以及包含 mutating 调用的不安全批次。
- Oracle:`6` 个只读且 `parallelSafe` 的工具调用会被真实执行并正确聚合;`7` 个并行调用会在执行前被 `parallel_limit_exceeded` 拦截;包含 mutating 或非并行安全工具的批次会在执行前被 `parallel_requires_read_only` 拦截。
- Not:不验证 `brain` 决策来源。
##### `hand-serial-docx-chain`
@@ -416,12 +416,42 @@
- What:验证 live 模式下 `human` 会在一个真实长任务里串起模板定位、资料取证、docx 编辑与预览交付。
- Why:`human/live` 需要至少一条长期高价值复杂样例,覆盖 workspace、KB、docx 与最终交付链路。
- How:注入真实 live LLM 配置;workspace 中动态写入带 `RISK_TODO / OWNER_TODO` 占位符的周报模板和干扰文件;seed 两份真实 KB 文档分别提供风险结论与负责人说明并限制在已选范围内;直接输入“找到上周 Word 周报模板,按已选资料补全风险结论与负责人,生成预览且不要覆盖原文件”的自然语言诉求,让真实 `human` 自主完成。
- Oracle:任务最终完成;事件流中至少出现一次 `terminal_run_command`、一次以 `kb_` 开头的检索工具调用、一次 `docx_open`、一次 `docx_apply_ops` 与一次 `docx_render_preview`;存在 preview/draft artifact,且原模板未被覆盖。
- Oracle:任务最终完成;事件流中至少出现一次 `terminal_run_command`、一次以 `kb_` 开头的检索工具调用、一次 `docx_open`、一次 `docx_apply_ops` 与一次 `docx_render_preview`;存在 preview/draft artifact,且原模板未被覆盖;最终答复能从用户视角说明改动的是“风险结论”和“负责人说明”两处、依据分别来自 `Weekly Risk Summary.md` 与 `Owner Note.md`,并明确已经生成预览且未覆盖原文件。
- Not:不绑定最终答案措辞,也不要求固定检索工具顺序。
##### `human-docx-cause-remedy-reading-live`
- 测试类型:`backend`
- 实现状态:`已实现`
- What:验证 live 模式下 `human` 会基于两份已选 `docx` 做跨文件精读,并准确回答“过拟合 / 反走样”的成因与处理思路。
- Why:真实用户在复习场景里经常已经选中了资料,但仍会把两个概念混在一起;这条 case 要长期守住“已选资料范围 + 跨文件准确回答”的正式能力,对齐 `WB-CORE-002-AC2`、`WB-CORE-004-AC1`、`WB-CORE-006-AC1`、`WB-CORE-006-AC2`。
- How:注入真实 live LLM 配置;把 `tests/fixtures/ai/docx/深度学习.docx` 与 `tests/fixtures/ai/docx/图形学.docx` 作为已选资料复制进 workspace;直接输入“我做题老把这两个点混在一起。你就按我刚选的这两份资料,帮我说清楚:过拟合和反走样各自是什么、为什么会出现、一般怎么处理。别补充课外知识,哪份文档没写你就直接说没写。”的自然语言诉求,让真实 `human` 自主完成。
- Oracle:任务最终完成;事件流中至少两次 `docx_open` 覆盖两份目标文档,且至少出现一次 `docx_search`、`docx_outline` 或 `docx_get_block`;最终回答同时命中“过拟合”的成因/处理事实组和“反走样”的成因/处理事实组;不出现 `docx_apply_ops` 或 `docx_save`。
- Not:不要求固定回答结构,也不对最终文案做逐字断言。
##### `human-docx-contextual-comparison-draft-live`
- 测试类型:`backend`
- 实现状态:`已实现`
- What:验证 live 模式下 `human` 会基于两份已选 `docx`,在目标 `docx` 的相关位置追加解释性段落和对照表,并生成不覆盖原文件的预览稿。
- Why:真实用户常见诉求不是“改占位符”,而是“在这块后面补一段更好懂的说明”;这条 case 覆盖跨文件取材、相关位置补写、结构化新增与 preview-only 交付,对齐 `WB-CORE-002-AC2`、`WB-CORE-003-AC1`、`WB-CORE-004-AC1`、`WB-CORE-006-AC1`、`WB-CORE-006-AC2`。
- How:注入真实 live LLM 配置;把 `tests/fixtures/ai/docx/深度学习.docx` 与 `tests/fixtures/ai/docx/图形学.docx` 作为已选资料复制进 workspace;直接输入“《图形学》里‘反走样’这一节我还是记不住。你参考我刚选的这两份 Word,直接在这一节最后补一小段更好理解的说明,再加个简单对照表,一列写反走样,一列写《深度学习》里的过拟合。直接给我可预览的版本就行,别改原文件,也不用先回我文字草稿。”的自然语言诉求,让真实 `human` 自主完成。
- Oracle:任务最终完成;事件流中至少两次 `docx_open` 覆盖两份目标文档;`docx_apply_ops` 前至少出现一次针对“反走样”的读取定位;至少出现一次 `docx_apply_ops` 与一次 `docx_render_preview`;存在 preview/draft artifact;不出现 `docx_save`;原始两份 `docx` 文件未被覆盖;写入请求里能看到新增说明段落与表格结构。
- Not:不要求固定 op 组合,也不对最终说明文案做逐字断言。
##### `human-kb-guided-docx-comparison-draft-live`
- 测试类型:`backend`
- 实现状态:`已实现`
- What:验证 live 模式下 `human` 在本地资料较多时,会先用 KB 缩圈定位相关资料,再回到目标 `docx` 做跨文件补写并交付预览。
- Why:真实用户常常记得主题、不记得文件名;这条 case 要长期守住“工作区默认范围 + KB 缩圈 + docx 继续执行”的正式能力,对齐 `WB-CORE-002-AC2`、`WB-CORE-004-AC1`、`WB-CORE-006-AC1`、`WB-CORE-006-AC2`。
- How:注入真实 live LLM 配置;workspace 中动态写入 `深度学习.docx`、`图形学.docx` 和若干干扰文件,并把目标 `docx` 与干扰资料同时 seed 到真实 KB;不传 `selectedReferences`,只绑定 workspace;直接输入“我工作区资料有点多,我忘了是哪份课件在讲反走样了。你先帮我找到那份,再参考讲过拟合的那份资料,直接在‘反走样’那一节最后补一段更好记的说明,再加个简单对照表,一列写反走样,一列写过拟合。直接给我可预览的版本就行,别改原文件,也不用先回我文字草稿。”的自然语言诉求,让真实 `human` 自主完成。
- Oracle:任务最终完成;事件流中至少出现一次 KB 工具调用;后续出现 `docx_open` 并覆盖最终参与编辑的目标 `docx`;至少出现一次 `docx_apply_ops` 与一次 `docx_render_preview`;存在 preview/draft artifact;不出现 `docx_save`;目标原始 `docx` 未被覆盖;新增内容同时涉及“反走样”和“过拟合”。
- Not:不强制固定执行路径,也不要求 KB 缩圈后必须再打开所有候选资料。
## 5. 验收口径
- `tool` suite 必须持续守住真实工具输入输出、审批边界和 artifact 契约。
- `human/mock-llm` 必须覆盖 `AiRunService -> human` 入口、`brain / learning / hand / mouth` 主链、memory 延续、watch hint 收敛与复杂工具协作。
- `human/live` 必须至少保留 1 条简单任务和 1 条复杂任务正式 case。
- `human` suite 必须继续支撑 `WB-CORE-002`、`WB-CORE-005`、`WB-CORE-006`。
- `human` suite 必须继续支撑 `WB-CORE-002`、`WB-CORE-003`、`WB-CORE-004`、`WB-CORE-005`、`WB-CORE-006`。
+1
View File
@@ -63,6 +63,7 @@ export const HUMAN_WATCH_HINT_CODES = [
export const HUMAN_MAX_ACTIVE_TOOLS = 20
export const HUMAN_DEFAULT_TOOL_CALL_LIMIT = 50
export const HUMAN_MAX_PARALLEL_TOOL_CALLS = 6
export const HUMAN_ROUND_EXECUTION_KINDS = [
'pre_execution_blocked',
'learning',
@@ -1,5 +1,5 @@
import type { JsonObject } from '../../../../../../shared/types'
import { HUMAN_DEFAULT_TOOL_CALL_LIMIT } from '../../constants'
import { HUMAN_DEFAULT_TOOL_CALL_LIMIT, HUMAN_MAX_PARALLEL_TOOL_CALLS } from '../../constants'
import type { AiRunContext } from '../../../types'
import { extractArtifactRefsFromToolResults, mergeArtifactRefs } from '../../../utils/artifacts'
import {
@@ -14,8 +14,6 @@ import type { HumanRuntimeCache, HumanState, HumanWatchHint, ToolCallResult, Too
import { executeToolCall } from './toolExecutor'
import { returnToBrain, returnToBrainAfterTrace, returnToBrainWithWatchHint } from '../utils/stateTransitions'
const MAX_PARALLEL_TOOL_CALLS = 4
export const getToolExecutionMeta = (input: {
toolName: string
args: JsonObject
@@ -70,13 +68,13 @@ const buildToolLimitWatchHint = (toolCallCount: number, limit: number): HumanWat
const validateToolUsePlan = (toolUsePlan: ToolUsePlan): HumanWatchHint | undefined => {
if (toolUsePlan.mode === 'serial') return undefined
if (toolUsePlan.calls.length > MAX_PARALLEL_TOOL_CALLS) {
if (toolUsePlan.calls.length > HUMAN_MAX_PARALLEL_TOOL_CALLS) {
return createHandPlanInvalidWatchHint({
mode: 'parallel',
reason: 'parallel_limit_exceeded',
offendingToolNames: toolUsePlan.calls.map((item) => item.toolName),
callCount: toolUsePlan.calls.length,
limit: MAX_PARALLEL_TOOL_CALLS
limit: HUMAN_MAX_PARALLEL_TOOL_CALLS
})
}
+1 -164
View File
@@ -13,10 +13,9 @@ import {
listExternalSkillDefinitions
} from '../skills'
import { renderHumanStableNotes } from '../shared/memoryRenderers'
import { renderLoadedHistoryText, renderRoundSection } from '../shared/historyRenderers'
import { getToolPromptSpec } from '../tools/registry'
import type {
HumanLoadedHistoryContext,
HumanRoundRecord,
HumanState,
ToolPromptSpec
} from '../types'
@@ -43,168 +42,6 @@ const renderLearnedToolSpecLine = (spec: ToolPromptSpec): string => {
return `- 工具 ${spec.name}:用途=${spec.purpose};必填参数=[${requiredArgs || 'none'}];${example}`
}
const renderDecisionNextAction = (record: HumanRoundRecord): string => {
switch (record.decision.nextAction.action) {
case 'learning':
return `- 你准备下一步做:准备学习 ${renderQuotedNames(
record.decision.nextAction.wantLearnSkillIds
)}。`
case 'hand':
return `- 你准备下一步做:准备进入 hand,${
record.decision.nextAction.toolUsePlan.mode === 'parallel'
? `以 parallel 模式并行执行 ${renderQuotedNames(
record.decision.nextAction.toolUsePlan.calls.map((call) => call.toolName)
)}。`
: `以 serial 模式按顺序执行 ${renderQuotedNames(
record.decision.nextAction.toolUsePlan.calls.map((call) => call.toolName)
)}。`
}`
case 'mouth':
return '- 你准备下一步做:准备进入 mouth 组织最终交付。'
}
}
const renderRoundDecision = (record: HumanRoundRecord): string =>
`
- 你的判断:${record.decision.decision.summary}
- 你这样判断的原因:${record.decision.decision.reason}
- 你当时期待的结果:${record.decision.decision.expectedOutcome}
- ${record.decision.readyToDeliver ? '你当时认为已经可以交付。' : '你当时认为还不能直接交付。'}
${renderDecisionNextAction(record)}
`.trim()
const renderRoundExecution = (
record: HumanRoundRecord,
detail: 'focus' | 'chain' | 'loaded'
): string => {
switch (record.execution.kind) {
case 'pre_execution_blocked':
return '- 没有进入真实执行,系统在执行前阻断了这轮动作。'
case 'learning':
return '- 进入 learning,按请求尝试补齐外置技能。'
case 'hand':
return `
- 进入 hand,执行模式是 ${record.execution.toolUsePlan.mode}。
- ${
record.execution.toolUsePlan.mode === 'parallel'
? `实际并行执行了 ${renderQuotedNames(record.execution.toolUsePlan.calls.map((call) => call.toolName))}。`
: `实际按顺序执行了 ${renderQuotedNames(record.execution.toolUsePlan.calls.map((call) => call.toolName))}。`
}
${detail !== 'chain' ? `- 本轮计划包含 ${record.execution.toolUsePlan.calls.length} 个工具调用。` : ''}
`
.trim()
.replace(/\n\n+/g, '\n')
case 'mouth':
return '- 进入 mouth,组织最终说明。'
}
}
const renderRoundResult = (record: HumanRoundRecord): string => {
switch (record.execution.kind) {
case 'pre_execution_blocked':
return renderWatchHintForBrain(record.execution.watchHint)
case 'learning':
return `
- 已加载的技能:${renderQuotedNames(record.execution.learningOutcome.loadedSkillIds)}
- 已跳过的重复技能:${renderQuotedNames(record.execution.learningOutcome.skippedDuplicateSkillIds)}
- 未识别的技能:${renderQuotedNames(record.execution.learningOutcome.skippedUnknownSkillIds)}
- 当前 active tool 数:${String(record.execution.learningOutcome.activeToolCount)}
${record.execution.watchHint ? `- 收敛提示:${renderWatchHintForBrain(record.execution.watchHint)}` : ''}
`
.trim()
.replace(/\n\n+/g, '\n')
case 'hand': {
const toolResultLines = record.execution.toolResults
.map((result) =>
[
`- \`${result.toolName}\`:${result.summary}`,
result.dataPreview ? `结果:${result.dataPreview}` : '',
result.diagnosticText ? `诊断:${result.diagnosticText}` : ''
]
.filter(Boolean)
.join(';')
)
.join('\n')
const artifactLine =
record.execution.artifactRefs && record.execution.artifactRefs.length > 0
? `- 产物:${renderInlineList(
record.execution.artifactRefs.map((artifact) =>
artifact.kind === 'file'
? `\`${artifact.fileName}\``
: `draft review ${artifact.review.reviewId}`
),
{
separator: '、',
fallback: 'none'
}
)}`
: ''
const watchHintLine = record.execution.watchHint
? `- 收敛提示:${renderWatchHintForBrain(record.execution.watchHint)}`
: ''
return [toolResultLines, artifactLine, watchHintLine].filter(Boolean).join('\n')
}
case 'mouth':
return `
- 输出摘要:${record.execution.summary}
${record.execution.completed ? '- 任务已完成。' : '- 任务未完成。'}
`.trim()
}
}
const renderRoundRecord = (
record: HumanRoundRecord,
detail: 'focus' | 'chain' | 'loaded'
): string =>
`
- 第 ${record.roundId} 轮
决策了什么:
${renderRoundDecision(record)}
做了什么:
${renderRoundExecution(record, detail)}
结果是什么:
${renderRoundResult(record)}
`.trim()
const renderRoundSection = (input: {
records: HumanRoundRecord[]
detail: 'focus' | 'chain' | 'loaded'
}): string =>
input.records.length > 0
? input.records.map((record) => renderRoundRecord(record, input.detail)).join('\n\n')
: 'none'
const renderLoadedHistoryText = (
loadedHistoryContext: HumanLoadedHistoryContext | undefined
): string =>
loadedHistoryContext
? `
过滤条件:roundIds=${renderInlineList((loadedHistoryContext.filters.roundIds || []).map(String), {
separator: '、',
fallback: 'none'
})};executionKinds=${renderInlineList(loadedHistoryContext.filters.executionKinds || [], {
separator: '、',
fallback: 'none'
})};toolNames=${renderInlineList(loadedHistoryContext.filters.toolNames || [], {
separator: '、',
fallback: 'none'
})};watchHintCodes=${renderInlineList(loadedHistoryContext.filters.watchHintCodes || [], {
separator: '、',
fallback: 'none'
})};limit=${String(loadedHistoryContext.filters.limit || loadedHistoryContext.rounds.length)}
${renderRoundSection({
records: loadedHistoryContext.rounds,
detail: 'loaded'
})}
`
.trim()
: 'none'
const renderAvailableSkillsSection = (state: HumanState): string => {
const effectiveSkillIds = getEffectiveLearnedSkillIds(state.skills.learnedSkillIds)
const effectiveSkillDefinitions = effectiveSkillIds.map((skillId) => getSkillDefinition(skillId))
+27 -119
View File
@@ -1,132 +1,41 @@
import { renderInlineList } from '../../utils/render'
import { getLastRoundFocus } from '../memory/roundHistory'
import { getLoadedHistoryContext } from '../memory/roundHistory'
import { renderUnifiedHistoryText } from '../shared/historyRenderers'
import { renderHumanStableNotes } from '../shared/memoryRenderers'
import { renderHumanArtifactLines } from '../shared/runtimeRenderers'
import type { HumanRoundRecord, HumanState } from '../types'
import { renderWatchHintForMouth } from './watchHints'
const renderRecentProgressSection = (record: HumanRoundRecord | undefined): string => {
if (!record) {
return 'none'
}
const progressText =
record.execution.kind === 'pre_execution_blocked'
? `- 执行前阻断:${renderWatchHintForMouth(record.execution.watchHint)}`
: record.execution.kind === 'learning'
? `- learning 结果:已加载 ${renderInlineList(
record.execution.learningOutcome.loadedSkillIds.map((skillId) => `\`${skillId}\``),
{
separator: '、',
fallback: 'none'
}
)};重复 ${renderInlineList(
record.execution.learningOutcome.skippedDuplicateSkillIds.map(
(skillId) => `\`${skillId}\``
),
{
separator: '、',
fallback: 'none'
}
)};未知 ${renderInlineList(
record.execution.learningOutcome.skippedUnknownSkillIds.map(
(skillId) => `\`${skillId}\``
),
{
separator: '、',
fallback: 'none'
}
)}。`
: record.execution.kind === 'hand'
? `
- hand 执行模式:${record.execution.toolUsePlan.mode}
${record.execution.toolResults.map((result) => `- \`${result.toolName}\`:${result.summary}`).join('\n')}
`
.trim()
.replace(/\n\n+/g, '\n')
: `- mouth 输出摘要:${record.execution.summary}`
return `
- 第 ${record.roundId} 轮判断:${record.decision.decision.summary}
- 原因:${record.decision.decision.reason}
- 预期结果:${record.decision.decision.expectedOutcome}
${progressText}
`.trim()
}
import type { HumanState } from '../types'
const buildFinalDecisionText = (state: HumanState): string =>
state.execution.currentDecision
? `最终决定(readyToDeliver=${String(state.execution.currentDecision.readyToDeliver)}):“${state.execution.currentDecision.decision.summary}”;原因:${state.execution.currentDecision.decision.reason};预期结果:${state.execution.currentDecision.decision.expectedOutcome}`
: 'none'
const OUTPUT_EXAMPLE = `
# 处理结果
已完成当前任务,下面是基于现有事实和结果整理出的最终结论。
## 关键依据
- 依据一
## 产物与引用
- /path/to/file.docx
## 当前限制
无
`.trim()
export const buildMouthPrompt = (input: { state: HumanState }) => {
const system = `
<system>
<role>
你现在扮演一个负责对外表达最终结果的人。
你的任务是把已经确定下来的判断、任务笔记和产物整理成最终交付。
你现在负责根据已有事实回答用户这次任务。
你的目标不是复述内部过程,而是直接完成 task_goal 里用户真正要的回复。
</role>
<working_style>
请像一个表达清晰、判断克制的人一样输出结果:
不夸大、不补编、不引入新的结论,
只把已经确认的信息组织成最终交付。
</working_style>
<source_of_truth>
- final_decision 是 brain 已经确认的最终判断;你负责表达,不负责重新推翻或改写它。
- your_notes 是当前任务的稳定事实来源,包括已确认事实、关键依据和未决问题;正文中的事实与依据优先从这里取。
- recent_progress 只用于补充最近完成到哪里、最近结果如何,不必把内部过程完整复述出来。
- runtime_artifacts 只允许准确引用已经给出的路径、文件名、状态和摘要,不要补编额外元数据。
- correction_hint 只用于如实说明限制、阻断或未完成点,不要把它扩写成新的事实。
</source_of_truth>
<context_block_guide>
- task_goal:用户原始诉求与限制。
- final_decision:brain 已经确认的最终判断与收口方向。
- your_notes:稳定任务笔记,按“已确认 / 关键依据 / 未决问题”组织。
- recent_progress:最近一轮过程记录,用于补充当前完成到哪里。
- runtime_artifacts:运行中已经产生的真实产物列表。
- correction_hint:当前额外的收敛或纠偏提醒;若无则为 none。
</context_block_guide>
<delivery_principles>
1. 如果任务已经完成,先直接给出结论,再给关键依据,再给结果或产物说明。
2. 如果当前结果有限,先说明已经完成到哪里,再说明限制和继续需要什么。
3. 只使用已经给你的任务笔记、最终判断和产物,不要补充新的事实或推理。
4. 语言要清楚、克制、可执行,不要堆内部过程。
5. 如果存在运行时产物或最近过程记录,只能准确引用已经给出的路径、状态和摘要。
6. 你不会调用工具,也不会继续推进未完成步骤;如果仍有步骤没做,只能如实说明当前停在哪。
</delivery_principles>
<available_information>
- task_goal:用户最初要你完成什么,以及用户显式要求你回答哪些点。
- final_decision:brain 最后确认的收口方向。
- your_notes:当前稳定事实来源。其中:
- 已确认:当前稳定成立、可以直接告诉用户的事实。
- 关键依据:支撑这些事实的证据、来源或命中结果。
- 未决问题:仍会影响结论、或需要用户补充确认的点。
- history:任务过程中真实发生过的决策、执行和结果,可用于补足 your_notes 不够具体的地方。
- runtime_artifacts:运行中已经产生的真实产物,可按给定路径、文件名、状态和摘要引用。
</available_information>
<output_contract>
- 你必须只输出最终 markdown 正文,不要输出任何额外说明、JSON 或代码块。
- 不要引入新的事实、依据、步骤或承诺。
- 不要暴露内部字段名、节点名或系统术语。
- 如果依据不足,就保持保守表达,不要写成确定结论。
- 不要把未来可能执行的工具动作写成“后续会自动完成”或“已经在进行中”。
- 输出必须是完整结果,而不是零散句子或局部片段。
- 默认使用最终用户答复的组织方式:先给结论,再给关键依据,再给产物或当前限制。
- 只输出最终 markdown 正文,不要输出 JSON 或额外说明。
- 直接回答 task_goal,优先覆盖用户显式提出的问题。
- your_notes 优先;如果 your_notes 不够具体,可结合 history 与 runtime_artifacts 补足。
- 只基于提供的信息回答,不要补编新的事实。
- 标题、分节、列表都按需要使用,不固定。
- 默认使用用户能理解的语言,不原样照抄内部标签、block id、tool 名,除非用户明确要求这些内部细节。
</output_contract>
<output_example>
${OUTPUT_EXAMPLE}
</output_example>
</system>
`.trim()
@@ -144,17 +53,16 @@ ${buildFinalDecisionText(input.state)}
${renderHumanStableNotes(input.state, 'mouth_prompt')}
</your_notes>
<recent_progress>
${renderRecentProgressSection(getLastRoundFocus(input.state))}
</recent_progress>
<history>
${renderUnifiedHistoryText({
roundHistory: input.state.memory.history.roundHistory,
loadedHistoryContext: getLoadedHistoryContext(input.state)
})}
</history>
<runtime_artifacts>
${renderHumanArtifactLines(input.state, 'prompt')}
</runtime_artifacts>
<correction_hint>
${input.state.execution.watchHint ? renderWatchHintForMouth(input.state.execution.watchHint) : 'none'}
</correction_hint>
</user_context>
`.trim()
@@ -78,8 +78,4 @@ export const renderWatchHintForBrain = (hint: HumanWatchHint): string => {
}
}
export const renderWatchHintForMouth = (hint: HumanWatchHint): string => {
return summarizeWatchHint(hint)
}
export const renderWatchHintDetail = (hint: HumanWatchHint): JsonObject => asJsonObject(hint.detail)
@@ -0,0 +1,191 @@
import { renderInlineList } from '../../utils/render'
import type { HumanLoadedHistoryContext, HumanRoundRecord } from '../types'
import { renderWatchHintForBrain } from '../prompts/watchHints'
const renderQuotedNames = (items: readonly string[]): string =>
renderInlineList(
items.map((item) => `\`${item}\``),
{
separator: '、',
fallback: 'none'
}
)
const renderDecisionNextAction = (record: HumanRoundRecord): string => {
switch (record.decision.nextAction.action) {
case 'learning':
return `- 你准备下一步做:准备学习 ${renderQuotedNames(
record.decision.nextAction.wantLearnSkillIds
)}。`
case 'hand':
return `- 你准备下一步做:准备进入 hand,${
record.decision.nextAction.toolUsePlan.mode === 'parallel'
? `以 parallel 模式并行执行 ${renderQuotedNames(
record.decision.nextAction.toolUsePlan.calls.map((call) => call.toolName)
)}。`
: `以 serial 模式按顺序执行 ${renderQuotedNames(
record.decision.nextAction.toolUsePlan.calls.map((call) => call.toolName)
)}。`
}`
case 'mouth':
return '- 你准备下一步做:准备进入 mouth 组织最终交付。'
}
}
const renderRoundDecision = (record: HumanRoundRecord): string =>
`
- 你的判断:${record.decision.decision.summary}
- 你这样判断的原因:${record.decision.decision.reason}
- 你当时期待的结果:${record.decision.decision.expectedOutcome}
- ${record.decision.readyToDeliver ? '你当时认为已经可以交付。' : '你当时认为还不能直接交付。'}
${renderDecisionNextAction(record)}
`.trim()
const renderRoundExecution = (
record: HumanRoundRecord,
detail: 'focus' | 'chain' | 'loaded'
): string => {
switch (record.execution.kind) {
case 'pre_execution_blocked':
return '- 没有进入真实执行,系统在执行前阻断了这轮动作。'
case 'learning':
return '- 进入 learning,按请求尝试补齐外置技能。'
case 'hand':
return `
- 进入 hand,执行模式是 ${record.execution.toolUsePlan.mode}。
- ${
record.execution.toolUsePlan.mode === 'parallel'
? `实际并行执行了 ${renderQuotedNames(record.execution.toolUsePlan.calls.map((call) => call.toolName))}。`
: `实际按顺序执行了 ${renderQuotedNames(record.execution.toolUsePlan.calls.map((call) => call.toolName))}。`
}
${detail !== 'chain' ? `- 本轮计划包含 ${record.execution.toolUsePlan.calls.length} 个工具调用。` : ''}
`
.trim()
.replace(/\n\n+/g, '\n')
case 'mouth':
return '- 进入 mouth,组织最终说明。'
}
}
const renderRoundResult = (record: HumanRoundRecord): string => {
switch (record.execution.kind) {
case 'pre_execution_blocked':
return renderWatchHintForBrain(record.execution.watchHint)
case 'learning':
return `
- 已加载的技能:${renderQuotedNames(record.execution.learningOutcome.loadedSkillIds)}
- 已跳过的重复技能:${renderQuotedNames(record.execution.learningOutcome.skippedDuplicateSkillIds)}
- 未识别的技能:${renderQuotedNames(record.execution.learningOutcome.skippedUnknownSkillIds)}
- 当前 active tool 数:${String(record.execution.learningOutcome.activeToolCount)}
${record.execution.watchHint ? `- 收敛提示:${renderWatchHintForBrain(record.execution.watchHint)}` : ''}
`
.trim()
.replace(/\n\n+/g, '\n')
case 'hand': {
const toolResultLines = record.execution.toolResults
.map((result) =>
[
`- \`${result.toolName}\`:${result.summary}`,
result.dataPreview ? `结果:${result.dataPreview}` : '',
result.diagnosticText ? `诊断:${result.diagnosticText}` : ''
]
.filter(Boolean)
.join(';')
)
.join('\n')
const artifactLine =
record.execution.artifactRefs && record.execution.artifactRefs.length > 0
? `- 产物:${renderInlineList(
record.execution.artifactRefs.map((artifact) =>
artifact.kind === 'file'
? `\`${artifact.fileName}\``
: `draft review ${artifact.review.reviewId}`
),
{
separator: '、',
fallback: 'none'
}
)}`
: ''
const watchHintLine = record.execution.watchHint
? `- 收敛提示:${renderWatchHintForBrain(record.execution.watchHint)}`
: ''
return [toolResultLines, artifactLine, watchHintLine].filter(Boolean).join('\n')
}
case 'mouth':
return `
- 输出摘要:${record.execution.summary}
${record.execution.completed ? '- 任务已完成。' : '- 任务未完成。'}
`.trim()
}
}
const renderRoundRecord = (
record: HumanRoundRecord,
detail: 'focus' | 'chain' | 'loaded'
): string =>
`
- 第 ${record.roundId} 轮
决策了什么:
${renderRoundDecision(record)}
做了什么:
${renderRoundExecution(record, detail)}
结果是什么:
${renderRoundResult(record)}
`.trim()
export const renderRoundSection = (input: {
records: HumanRoundRecord[]
detail: 'focus' | 'chain' | 'loaded'
}): string =>
input.records.length > 0
? input.records.map((record) => renderRoundRecord(record, input.detail)).join('\n\n')
: 'none'
export const renderLoadedHistoryText = (
loadedHistoryContext: HumanLoadedHistoryContext | undefined
): string =>
loadedHistoryContext
? `
过滤条件:roundIds=${renderInlineList((loadedHistoryContext.filters.roundIds || []).map(String), {
separator: '、',
fallback: 'none'
})};executionKinds=${renderInlineList(loadedHistoryContext.filters.executionKinds || [], {
separator: '、',
fallback: 'none'
})};toolNames=${renderInlineList(loadedHistoryContext.filters.toolNames || [], {
separator: '、',
fallback: 'none'
})};watchHintCodes=${renderInlineList(loadedHistoryContext.filters.watchHintCodes || [], {
separator: '、',
fallback: 'none'
})};limit=${String(loadedHistoryContext.filters.limit || loadedHistoryContext.rounds.length)}
${renderRoundSection({
records: loadedHistoryContext.rounds,
detail: 'loaded'
})}
`
.trim()
: 'none'
export const renderUnifiedHistoryText = (input: {
roundHistory: HumanRoundRecord[]
loadedHistoryContext: HumanLoadedHistoryContext | undefined
}): string =>
`
过程历史:
${renderRoundSection({
records: input.roundHistory,
detail: 'loaded'
})}
主动回看历史:
${renderLoadedHistoryText(input.loadedHistoryContext)}
`
.trim()
.replace(/\n{3,}/g, '\n\n')
@@ -0,0 +1,121 @@
import { describe, it } from 'vitest'
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
import { runBackendCase } from '../../../harness/backend/run-backend-case'
import {
buildSelectedDocxFixtureReferences,
DOCX_CROSS_FILE_FIXTURE_PATHS,
HUMAN_DOCX_CAUSE_REMEDY_READING_USER_INPUT,
resolveSelectedDocxFixtures
} from '../../../harness/backend/aiLiveTestData'
import {
collectStartedToolStringArgs,
filterAiRunEventsByType,
runWithStructuredOutputRetry
} from '../testUtils'
const openedExpectedDoc = (
openedSourcePaths: readonly string[],
expectedRelativePath: string,
expectedName: string
): boolean =>
openedSourcePaths.some(
(sourcePath) =>
sourcePath.endsWith(expectedRelativePath) || sourcePath.endsWith(`/${expectedName}`)
)
describe('ai human live backend', () => {
it('human-docx-cause-remedy-reading-live', async () => {
await runWithStructuredOutputRetry(async () => {
await runBackendCase({
caseId: 'human-docx-cause-remedy-reading-live',
harnessMode: 'live',
domain: 'ai',
suite: 'human',
groups: ['live'],
description: '验证 live 模式下 human 会基于两份已选 docx 准确回答过拟合与反走样的成因和处理思路。',
fixtures: DOCX_CROSS_FILE_FIXTURE_PATHS,
runMode: 'single',
runtimeModelName: hasLiveLlmConfigInEnv('primary')
? loadLiveLlmConfigFromEnv('primary').model_name
: null,
execute: async (runtime) => {
if (!hasLiveLlmConfigInEnv('primary')) {
return {
skipped: true,
output: null,
fixtureDocs: [],
events: []
}
}
await runtime.seedLlmConfig(loadLiveLlmConfigFromEnv('primary'))
const fixtureDocs = resolveSelectedDocxFixtures(runtime)
const runner = createHumanTestRunner(runtime, {
selectedReferences: buildSelectedDocxFixtureReferences(runtime)
})
const output = await runner.run({
taskId: runtime.taskId,
userInput: HUMAN_DOCX_CAUSE_REMEDY_READING_USER_INPUT
})
return {
skipped: false,
output,
fixtureDocs,
events: runtime.eventCollector.snapshot()
}
},
assert: async (result, runtime) => {
if (result.skipped) {
runtime.assertThat(true, {
name: 'live-config-not-provided',
message: '未提供 live 模型配置,已跳过 human live docx cause/remedy reading case。'
})
return
}
const toolResults = filterAiRunEventsByType(result.events, 'tool_call_result')
const toolNames = toolResults.map((event) => event.toolName)
const openedSourcePaths = collectStartedToolStringArgs(
result.events,
'docx_open',
'sourcePath'
)
const deliveryText = `${result.output?.summary || ''}\n${result.output?.deliveryMarkdown || ''}`
runtime.assertThat(
result.output?.status === 'completed' &&
result.fixtureDocs.every((fixtureDoc) =>
openedExpectedDoc(openedSourcePaths, fixtureDoc.relativePath, fixtureDoc.name)
) &&
toolNames.some(
(toolName) =>
toolName === 'docx_search' ||
toolName === 'docx_outline' ||
toolName === 'docx_get_block'
) &&
!toolNames.includes('docx_apply_ops') &&
!toolNames.includes('docx_save') &&
/(过拟合|过度拟合)/.test(deliveryText) &&
/(记住过多训练数据中的特征值|训练时间过久)/.test(deliveryText) &&
/(数据增强|未使用过的数据|验证数据)/.test(deliveryText) &&
/反走样/.test(deliveryText) &&
/(采样频率低于信号频率|像素.*边界)/.test(deliveryText) &&
/(低通滤波|多个采样点|多个像素点|多次采样)/.test(deliveryText),
{
name: 'human-live-reads-both-selected-docx-and-explains-cause-and-remedy',
message: 'live human 读 case 应打开两份已选 docx,并准确回答过拟合与反走样的成因和处理思路。',
data: {
openedSourcePaths,
toolNames,
summary: result.output?.summary,
deliveryMarkdown: result.output?.deliveryMarkdown
}
}
)
}
})
})
}, 600_000)
})
@@ -0,0 +1,160 @@
import fs from 'node:fs/promises'
import { describe, it } from 'vitest'
import type { AiRunEvent } from '../../../../src/shared/types'
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
import { runBackendCase } from '../../../harness/backend/run-backend-case'
import {
buildSelectedDocxFixtureReferences,
DOCX_CROSS_FILE_FIXTURE_PATHS,
hasUnchangedFileBytes,
HUMAN_DOCX_CONTEXTUAL_COMPARISON_DRAFT_USER_INPUT,
resolveSelectedDocxFixtures
} from '../../../harness/backend/aiLiveTestData'
import {
collectDocxApplyOpsContentKinds,
collectStartedToolStringArgs,
filterAiRunEventsByType,
runWithStructuredOutputRetry
} from '../testUtils'
const openedExpectedDoc = (
openedSourcePaths: readonly string[],
expectedRelativePath: string,
expectedName: string
): boolean =>
openedSourcePaths.some(
(sourcePath) =>
sourcePath.endsWith(expectedRelativePath) || sourcePath.endsWith(`/${expectedName}`)
)
const hasTargetReadBeforeApply = (events: readonly AiRunEvent[]): boolean => {
const firstApplyIndex = events.findIndex(
(event) => event.type === 'tool_call_started' && event.toolName === 'docx_apply_ops'
)
if (firstApplyIndex <= 0) {
return false
}
return events.slice(0, firstApplyIndex).some((event) => {
if (event.type !== 'tool_call_started') {
return false
}
if (event.toolName === 'docx_search') {
const query = event.input.query
return typeof query === 'string' && query.includes('反走样')
}
return event.toolName === 'docx_outline' || event.toolName === 'docx_get_block'
})
}
describe('ai human live backend', () => {
it('human-docx-contextual-comparison-draft-live', async () => {
await runWithStructuredOutputRetry(async () => {
await runBackendCase({
caseId: 'human-docx-contextual-comparison-draft-live',
harnessMode: 'live',
domain: 'ai',
suite: 'human',
groups: ['live'],
description: '验证 live 模式下 human 会在图形学相关位置后补写说明和对照表,并生成预览。',
fixtures: DOCX_CROSS_FILE_FIXTURE_PATHS,
runMode: 'single',
runtimeModelName: hasLiveLlmConfigInEnv('primary')
? loadLiveLlmConfigFromEnv('primary').model_name
: null,
execute: async (runtime) => {
if (!hasLiveLlmConfigInEnv('primary')) {
return {
skipped: true,
output: null,
fixtureDocs: [],
originalFixtureBytes: [],
events: []
}
}
await runtime.seedLlmConfig(loadLiveLlmConfigFromEnv('primary'))
const fixtureDocs = resolveSelectedDocxFixtures(runtime)
const originalFixtureBytes = await Promise.all(
fixtureDocs.map(async (fixtureDoc) => ({
name: fixtureDoc.name,
relativePath: fixtureDoc.relativePath,
workspacePath: fixtureDoc.workspacePath,
bytes: await fs.readFile(fixtureDoc.workspacePath)
}))
)
const runner = createHumanTestRunner(runtime, {
selectedReferences: buildSelectedDocxFixtureReferences(runtime)
})
const output = await runner.run({
taskId: runtime.taskId,
userInput: HUMAN_DOCX_CONTEXTUAL_COMPARISON_DRAFT_USER_INPUT
})
return {
skipped: false,
output,
fixtureDocs,
originalFixtureBytes,
events: runtime.eventCollector.snapshot()
}
},
assert: async (result, runtime) => {
if (result.skipped) {
runtime.assertThat(true, {
name: 'live-config-not-provided',
message: '未提供 live 模型配置,已跳过 human live contextual docx comparison draft case。'
})
return
}
const events = result.events
const toolResults = filterAiRunEventsByType(events, 'tool_call_result')
const toolNames = toolResults.map((event) => event.toolName)
const openedSourcePaths = collectStartedToolStringArgs(events, 'docx_open', 'sourcePath')
const changedContentKinds = collectDocxApplyOpsContentKinds(events)
const docxArtifacts =
result.output?.artifacts?.filter(
(artifact) =>
artifact.kind === 'file' &&
artifact.capability === 'docx' &&
(artifact.artifactRole === 'preview' || artifact.artifactRole === 'draft')
) || []
const unchangedFixtures = await Promise.all(
result.originalFixtureBytes.map(async (fixture) =>
await hasUnchangedFileBytes(fixture.workspacePath, fixture.bytes)
)
)
runtime.assertThat(
result.output?.status === 'completed' &&
result.fixtureDocs.every((fixtureDoc) =>
openedExpectedDoc(openedSourcePaths, fixtureDoc.relativePath, fixtureDoc.name)
) &&
hasTargetReadBeforeApply(events) &&
toolNames.includes('docx_apply_ops') &&
toolNames.includes('docx_render_preview') &&
!toolNames.includes('docx_save') &&
docxArtifacts.length > 0 &&
changedContentKinds.includes('paragraph') &&
changedContentKinds.includes('table') &&
unchangedFixtures.every(Boolean),
{
name: 'human-live-contextual-docx-write-produces-preview-without-overwriting-source',
message: 'live human 定位补写 case 应在读取定位后补写说明和表格,并仅生成预览稿。',
data: {
openedSourcePaths,
toolNames,
changedContentKinds,
summary: result.output?.summary,
deliveryMarkdown: result.output?.deliveryMarkdown,
artifacts: result.output?.artifacts
}
}
)
}
})
})
}, 600_000)
})
@@ -0,0 +1,128 @@
import path from 'node:path'
import { describe, it } from 'vitest'
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
import { runBackendCase } from '../../../harness/backend/run-backend-case'
import {
hasUnchangedFileBytes,
HUMAN_KB_GUIDED_DOCX_COMPARISON_DRAFT_USER_INPUT,
seedKbGuidedDocxKb,
seedKbGuidedDocxWorkspace
} from '../../../harness/backend/aiLiveTestData'
import {
collectDocxApplyOpsContentKinds,
collectStartedToolStringArgs,
filterAiRunEventsByType,
hasKbToolCall,
runWithStructuredOutputRetry
} from '../testUtils'
describe('ai human live backend', () => {
it('human-kb-guided-docx-comparison-draft-live', async () => {
await runWithStructuredOutputRetry(async () => {
await runBackendCase({
caseId: 'human-kb-guided-docx-comparison-draft-live',
harnessMode: 'live',
domain: 'ai',
suite: 'human',
groups: ['live'],
description: '验证 live 模式下 human 会先用 KB 缩圈定位课件,再继续对目标 docx 做预览型补写。',
fixtures: [],
runMode: 'single',
runtimeModelName: hasLiveLlmConfigInEnv('primary')
? loadLiveLlmConfigFromEnv('primary').model_name
: null,
execute: async (runtime) => {
if (!hasLiveLlmConfigInEnv('primary')) {
return {
skipped: true,
output: null,
workspaceSeed: null,
events: []
}
}
await runtime.seedLlmConfig(loadLiveLlmConfigFromEnv('primary'))
const workspaceSeed = await seedKbGuidedDocxWorkspace(runtime)
await seedKbGuidedDocxKb(runtime, workspaceSeed)
const runner = createHumanTestRunner(runtime)
const output = await runner.run({
taskId: runtime.taskId,
userInput: HUMAN_KB_GUIDED_DOCX_COMPARISON_DRAFT_USER_INPUT
})
return {
skipped: false,
output,
workspaceSeed,
events: runtime.eventCollector.snapshot()
}
},
assert: async (result, runtime) => {
if (result.skipped || !result.workspaceSeed) {
runtime.assertThat(true, {
name: 'live-config-not-provided',
message: '未提供 live 模型配置,已跳过 human live KB-guided docx comparison draft case。'
})
return
}
const events = result.events
const toolResults = filterAiRunEventsByType(events, 'tool_call_result')
const toolNames = toolResults.map((event) => event.toolName)
const openedSourcePaths = collectStartedToolStringArgs(events, 'docx_open', 'sourcePath')
const changedContentKinds = collectDocxApplyOpsContentKinds(events)
const docxArtifacts =
result.output?.artifacts?.filter(
(artifact) =>
artifact.kind === 'file' &&
artifact.capability === 'docx' &&
(artifact.artifactRole === 'preview' || artifact.artifactRole === 'draft')
) || []
const deepLearningUnchanged = await hasUnchangedFileBytes(
result.workspaceSeed.deepLearningPath,
result.workspaceSeed.originalDeepLearningBytes
)
const graphicsUnchanged = await hasUnchangedFileBytes(
result.workspaceSeed.graphicsPath,
result.workspaceSeed.originalGraphicsBytes
)
const deliveryText = `${result.output?.summary || ''}\n${result.output?.deliveryMarkdown || ''}`
runtime.assertThat(
result.output?.status === 'completed' &&
hasKbToolCall(events) &&
toolNames.includes('docx_apply_ops') &&
toolNames.includes('docx_render_preview') &&
!toolNames.includes('docx_save') &&
docxArtifacts.length > 0 &&
changedContentKinds.includes('paragraph') &&
changedContentKinds.includes('table') &&
openedSourcePaths.some(
(sourcePath) =>
sourcePath.endsWith(result.workspaceSeed.graphicsRelativePath) ||
sourcePath.endsWith(`/${path.basename(result.workspaceSeed.graphicsRelativePath)}`)
) &&
/(过拟合|过度拟合)/.test(deliveryText) &&
/反走样/.test(deliveryText) &&
deepLearningUnchanged &&
graphicsUnchanged,
{
name: 'human-live-kb-guided-docx-write-uses-kb-and-preserves-source-files',
message: 'live human KB-guided 写 case 应先触发 KB 缩圈,再对目标 docx 生成预览稿且不覆盖原文件。',
data: {
openedSourcePaths,
toolNames,
changedContentKinds,
summary: result.output?.summary,
deliveryMarkdown: result.output?.deliveryMarkdown,
artifacts: result.output?.artifacts
}
}
)
}
})
})
}, 600_000)
})
@@ -83,8 +83,12 @@ describe('ai human live backend', () => {
toolNames.includes('docx_render_preview') &&
docxArtifacts.length > 0 &&
templateUnchanged &&
/(待跟进|合规签署)/.test(deliveryText) &&
/Alice/.test(deliveryText),
/风险结论/.test(deliveryText) &&
/负责人(说明)?/.test(deliveryText) &&
/Weekly Risk Summary\.md/.test(deliveryText) &&
/Owner Note\.md/.test(deliveryText) &&
/(预览|preview)/i.test(deliveryText) &&
/(未覆盖原文件|原文件未被覆盖|未覆盖原件)/.test(deliveryText),
{
name: 'human-live-completes-risk-report-draft-through-real-tool-chain',
message: 'live human complex case 应通过 terminal、KB 与 docx 真实工具链完成周报草稿任务。',
+78 -1
View File
@@ -4,7 +4,7 @@ import type {
HumanStableMemoryPatch,
RuntimeSkillId
} from '../../../src/main/services/ai/human/types'
import type { AiRunEvent, JsonObject, JsonValue } from '../../../src/shared/types'
import type { AiRunEvent, JsonArray, JsonObject, JsonValue } from '../../../src/shared/types'
export const filterAiRunEventsByType = <TType extends AiRunEvent['type']>(
events: readonly AiRunEvent[],
@@ -98,6 +98,83 @@ export const filterToolCallStartedEventsByName = (
): Extract<AiRunEvent, { type: 'tool_call_started' }>[] =>
filterAiRunEventsByType(events, 'tool_call_started').filter((event) => event.toolName === toolName)
export const collectStartedToolInputs = (
events: readonly AiRunEvent[],
toolName: string
): JsonObject[] => filterToolCallStartedEventsByName(events, toolName).map((event) => event.input)
export const collectStartedToolStringArgs = (
events: readonly AiRunEvent[],
toolName: string,
argName: string
): string[] =>
collectStartedToolInputs(events, toolName)
.map((input) => input[argName])
.filter((value): value is string => typeof value === 'string')
export const hasKbToolCall = (events: readonly AiRunEvent[]): boolean =>
filterAiRunEventsByType(events, 'tool_call_result').some((event) => event.toolName.startsWith('kb_'))
const isJsonObject = (value: JsonValue | undefined): value is JsonObject =>
typeof value === 'object' && value !== null && !Array.isArray(value)
const isJsonArray = (value: JsonValue | undefined): value is JsonArray => Array.isArray(value)
const collectDocxBlockKindsFromValue = (
value: JsonValue | undefined,
kinds: Set<'paragraph' | 'table'>
): void => {
if (isJsonArray(value)) {
value.forEach((item) => collectDocxBlockKindsFromValue(item, kinds))
return
}
if (!isJsonObject(value)) {
return
}
const kind = value.kind
if (kind === 'paragraph' || kind === 'table') {
kinds.add(kind)
}
collectDocxBlockKindsFromValue(value.content, kinds)
collectDocxBlockKindsFromValue(value.ops, kinds)
collectDocxBlockKindsFromValue(value.rows, kinds)
}
export const collectDocxApplyOpsContentKinds = (
events: readonly AiRunEvent[]
): Array<'paragraph' | 'table'> => {
const kinds = new Set<'paragraph' | 'table'>()
collectStartedToolInputs(events, 'docx_apply_ops').forEach((input) => {
collectDocxBlockKindsFromValue(input, kinds)
})
return [...kinds]
}
const isStructuredOutputEmptyResponseError = (error: unknown): boolean =>
error instanceof Error && error.message.includes('structured_output_empty_response')
export const runWithStructuredOutputRetry = async (
action: () => Promise<void>,
maxAttempts = 2
): Promise<void> => {
let lastError: unknown = null
for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
try {
await action()
return
} catch (error) {
lastError = error
if (!isStructuredOutputEmptyResponseError(error) || attempt >= maxAttempts) {
throw error
}
}
}
throw lastError
}
export const readToolCallResultPreview = <TValue extends JsonValue>(
events: readonly AiRunEvent[],
toolName: string,
@@ -126,18 +126,182 @@ describe('ai tool hand backend', () => {
}
})
const safeLimitState = await handNode({
...safeState,
skills: {
learnedSkillIds: ['docx-read']
},
execution: {
...safeState.execution,
currentDecision: {
decision: {
summary: '执行 6 个只读并行调用',
reason: '验证新的并行上限允许 6 个安全调用。',
expectedOutcome: '六个 read_only 且 parallelSafe 的调用全部真正执行。'
},
readyToDeliver: false,
nextAction: {
action: 'hand',
toolUsePlan: {
mode: 'parallel',
calls: [
{
toolName: 'docx_outline',
args: {
docId: openResult.docId,
maxBlocks: 20
}
},
{
toolName: 'docx_search',
args: {
docId: openResult.docId,
query: 'Safety',
mode: 'literal'
}
},
{
toolName: 'docx_search',
args: {
docId: openResult.docId,
query: 'Risk',
mode: 'literal'
}
},
{
toolName: 'docx_search',
args: {
docId: openResult.docId,
query: 'monitor',
mode: 'literal'
}
},
{
toolName: 'docx_search',
args: {
docId: openResult.docId,
query: 'approval',
mode: 'literal'
}
},
{
toolName: 'docx_search',
args: {
docId: openResult.docId,
query: 'edge',
mode: 'literal'
}
}
]
}
}
}
}
})
const toolStartedBeforeLimitExceeded = filterEventsByType(
runtime.eventCollector.snapshot(),
'tool_call_started'
).length
const limitExceededState = await handNode({
...safeLimitState,
skills: {
learnedSkillIds: ['docx-read']
},
execution: {
...safeLimitState.execution,
currentDecision: {
decision: {
summary: '执行 7 个只读并行调用',
reason: '验证超过并行上限时会直接回到 brain。',
expectedOutcome: '写入 parallel_limit_exceeded watchHint,且不继续执行工具。'
},
readyToDeliver: false,
nextAction: {
action: 'hand',
toolUsePlan: {
mode: 'parallel',
calls: [
{
toolName: 'docx_outline',
args: {
docId: openResult.docId,
maxBlocks: 20
}
},
{
toolName: 'docx_search',
args: {
docId: openResult.docId,
query: 'Safety',
mode: 'literal'
}
},
{
toolName: 'docx_search',
args: {
docId: openResult.docId,
query: 'Risk',
mode: 'literal'
}
},
{
toolName: 'docx_search',
args: {
docId: openResult.docId,
query: 'monitor',
mode: 'literal'
}
},
{
toolName: 'docx_search',
args: {
docId: openResult.docId,
query: 'approval',
mode: 'literal'
}
},
{
toolName: 'docx_search',
args: {
docId: openResult.docId,
query: 'edge',
mode: 'literal'
}
},
{
toolName: 'docx_search',
args: {
docId: openResult.docId,
query: 'cases',
mode: 'literal'
}
}
]
}
}
}
}
})
const toolStartedAfterLimitExceeded = filterEventsByType(
runtime.eventCollector.snapshot(),
'tool_call_started'
).length
const toolStartedBeforeUnsafe = filterEventsByType(
runtime.eventCollector.snapshot(),
'tool_call_started'
).length
const unsafeState = await handNode({
...safeState,
...limitExceededState,
skills: {
learnedSkillIds: ['docx-read', 'docx-write', 'docx-delivery']
},
execution: {
...safeState.execution,
...limitExceededState.execution,
currentDecision: {
decision: {
summary: '并行写入和 validate',
@@ -183,7 +347,11 @@ describe('ai tool hand backend', () => {
return {
safeState,
safeLimitState,
limitExceededState,
unsafeState,
toolStartedBeforeLimitExceeded,
toolStartedAfterLimitExceeded,
toolStartedBeforeUnsafe,
toolStartedAfterUnsafe,
events: runtime.eventCollector.snapshot()
@@ -201,6 +369,33 @@ describe('ai tool hand backend', () => {
}
)
runtime.assertThat(
result.safeLimitState.execution.lastToolResults?.length === 6 &&
result.safeLimitState.execution.lastToolResults.every((item) => item.code === 'ok') &&
!result.safeLimitState.execution.watchHint,
{
name: 'six-safe-parallel-calls-are-allowed',
message: '6 个只读且 parallelSafe 的工具调用应真正执行并写回结果。',
data: result.safeLimitState.execution.lastToolResults
}
)
runtime.assertThat(
result.limitExceededState.execution.watchHint?.code === 'hand_plan_invalid' &&
result.limitExceededState.execution.watchHint.detail.reason === 'parallel_limit_exceeded' &&
result.limitExceededState.execution.watchHint.detail.limit === 6 &&
result.toolStartedAfterLimitExceeded === result.toolStartedBeforeLimitExceeded,
{
name: 'seven-parallel-calls-stop-before-execution',
message: '超过 6 个并行工具调用时应直接返回 brain 并写入 parallel_limit_exceeded。',
data: {
watchHint: result.limitExceededState.execution.watchHint,
toolStartedBeforeLimitExceeded: result.toolStartedBeforeLimitExceeded,
toolStartedAfterLimitExceeded: result.toolStartedAfterLimitExceeded
}
}
)
runtime.assertThat(
result.unsafeState.execution.watchHint?.code === 'hand_plan_invalid' &&
result.unsafeState.execution.watchHint.detail.reason === 'parallel_requires_read_only' &&
+264
View File
@@ -1,5 +1,6 @@
import fs from 'node:fs/promises'
import path from 'node:path'
import type { SelectedReference } from '../../../src/shared/types'
import { writeTestDocxFile } from './aiToolHarness'
import type { BackendTestRuntime, SeedKbDocumentInput } from './test-types'
@@ -9,6 +10,23 @@ export const HUMAN_FILE_RETENTION_USER_INPUT =
export const HUMAN_RISK_REPORT_DRAFT_USER_INPUT =
'我下午要把本周风险周报发出去。工作区里应该有一份上周用过的 Word 周报模板,但我忘了文件名。你先帮我找到那份模板,再按我刚才选的资料,把本周风险结论和负责人补进去,给我一个可预览的版本,先别覆盖原文件。最后告诉我你改了哪几处,依据分别来自哪里。'
export const DOCX_CROSS_FILE_FIXTURE_PATHS = [
'tests/fixtures/ai/docx/深度学习.docx',
'tests/fixtures/ai/docx/图形学.docx'
] as const
export const HUMAN_DOCX_CAUSE_REMEDY_READING_USER_INPUT =
'我做题老把这两个点混在一起。你就按我刚选的这两份资料,帮我说清楚:过拟合和反走样各自是什么、为什么会出现、一般怎么处理。别补充课外知识,哪份文档没写你就直接说没写。'
export const HUMAN_DOCX_CONTEXTUAL_COMPARISON_DRAFT_USER_INPUT =
'《图形学》里“反走样”这一节我还是记不住。你参考我刚选的这两份 Word,直接在这一节最后补一小段更好理解的说明,再加个简单对照表,一列写反走样,一列写《深度学习》里的过拟合。直接给我可预览的版本就行,别改原文件,也不用先回我文字草稿。'
export const HUMAN_KB_GUIDED_DOCX_COMPARISON_DRAFT_USER_INPUT =
'我工作区资料有点多,我忘了是哪份课件在讲反走样了。你先帮我找到那份,再参考讲过拟合的那份资料,直接在“反走样”那一节最后补一段更好记的说明,再加个简单对照表,一列写反走样,一列写过拟合。直接给我可预览的版本就行,别改原文件,也不用先回我文字草稿。'
export const HUMAN_KB_GUIDED_DOCX_READING_USER_INPUT =
'我工作区里应该有两份课件分别讲过过拟合和反走样,但我忘了文件名。你先帮我找出来,再告诉我这两个问题一般怎么处理。'
export const FILE_RETENTION_DOCUMENTS: readonly SeedKbDocumentInput[] = [
{
id: 'doc-retention',
@@ -51,6 +69,22 @@ export type RiskReportWorkspaceSeed = {
originalTemplateBytes: Uint8Array
}
export type SelectedDocxFixture = {
sourcePath: string
workspacePath: string
relativePath: string
name: string
}
export type KbGuidedDocxWorkspaceSeed = {
deepLearningPath: string
deepLearningRelativePath: string
graphicsPath: string
graphicsRelativePath: string
originalDeepLearningBytes: Uint8Array
originalGraphicsBytes: Uint8Array
}
export const seedFileRetentionKb = async (runtime: BackendTestRuntime): Promise<string[]> => {
await runtime.seedKb(FILE_RETENTION_DOCUMENTS)
return FILE_RETENTION_DOCUMENTS.map((document) => document.id)
@@ -61,6 +95,236 @@ export const seedRiskReportKb = async (runtime: BackendTestRuntime): Promise<str
return RISK_REPORT_DOCUMENTS.map((document) => document.id)
}
export const resolveSelectedDocxFixtures = (
runtime: BackendTestRuntime,
fixturePaths: readonly string[] = DOCX_CROSS_FILE_FIXTURE_PATHS
): SelectedDocxFixture[] =>
fixturePaths.map((fixturePath) => {
const workspacePath = runtime.getFixtureWorkspacePath(fixturePath)
return {
sourcePath: fixturePath,
workspacePath,
relativePath: path.relative(runtime.workspaceRoot, workspacePath),
name: path.basename(workspacePath)
}
})
export const buildSelectedDocxFixtureReferences = (
runtime: BackendTestRuntime,
fixturePaths: readonly string[] = DOCX_CROSS_FILE_FIXTURE_PATHS
): SelectedReference[] =>
resolveSelectedDocxFixtures(runtime, fixturePaths).map((fixture, index) => ({
kind: 'file',
workspaceId: 'test-workspace',
nodeId: `fixture-docx-${String(index + 1)}`,
name: fixture.name,
relativePath: fixture.relativePath,
absolutePath: fixture.workspacePath,
fileExt: '.docx',
kbDocumentIds: [],
kbAvailability: 'not_indexed'
}))
const copyDocxFixtureToWorkspaceRoot = async (input: {
runtime: BackendTestRuntime
sourcePath: string
targetName: string
summary: string
}): Promise<string> => {
const absoluteSourcePath = path.resolve(input.sourcePath)
const targetPath = path.join(input.runtime.workspaceRoot, input.targetName)
await fs.cp(absoluteSourcePath, targetPath)
input.runtime.recordSeedWorkspaceFile({
relativePath: input.targetName,
absolutePath: targetPath,
summary: input.summary,
metadata: {
format: 'docx'
}
})
return targetPath
}
export const seedKbGuidedDocxWorkspace = async (
runtime: BackendTestRuntime
): Promise<KbGuidedDocxWorkspaceSeed> => {
const deepLearningRelativePath = '深度学习.docx'
const graphicsRelativePath = '图形学.docx'
const deepLearningPath = await copyDocxFixtureToWorkspaceRoot({
runtime,
sourcePath: DOCX_CROSS_FILE_FIXTURE_PATHS[0],
targetName: deepLearningRelativePath,
summary: '目标课件:深度学习'
})
const graphicsPath = await copyDocxFixtureToWorkspaceRoot({
runtime,
sourcePath: DOCX_CROSS_FILE_FIXTURE_PATHS[1],
targetName: graphicsRelativePath,
summary: '目标课件:图形学'
})
const scheduleRelativePath = '课程安排.docx'
const schedulePath = await writeTestDocxFile(runtime.workspaceRoot, scheduleRelativePath, [
{
type: 'heading',
text: '课程安排',
level: 1
},
{
type: 'paragraph',
text: '这里记录的是考试时间、教室和分组安排。'
}
])
runtime.recordSeedWorkspaceFile({
relativePath: scheduleRelativePath,
absolutePath: schedulePath,
summary: '干扰项:课程安排',
metadata: {
format: 'docx'
}
})
const labNoteRelativePath = '机器学习实验记录.docx'
const labNotePath = await writeTestDocxFile(runtime.workspaceRoot, labNoteRelativePath, [
{
type: 'heading',
text: '机器学习实验记录',
level: 1
},
{
type: 'paragraph',
text: '本周实验主要关注学习率和 batch size 的调参过程。'
}
])
runtime.recordSeedWorkspaceFile({
relativePath: labNoteRelativePath,
absolutePath: labNotePath,
summary: '干扰项:机器学习实验记录',
metadata: {
format: 'docx'
}
})
const readingListRelativePath = '参考书目.md'
const readingListPath = path.join(runtime.workspaceRoot, readingListRelativePath)
const readingListText = '- 线性代数导引\n- 计算机图形学基础\n'
await fs.writeFile(readingListPath, readingListText, 'utf8')
runtime.recordSeedWorkspaceFile({
relativePath: readingListRelativePath,
absolutePath: readingListPath,
summary: '干扰项:参考书目',
metadata: {
format: 'text',
text: readingListText
}
})
const progressRelativePath = '复习进度.txt'
const progressPath = path.join(runtime.workspaceRoot, progressRelativePath)
const progressText = '今天先看图形学,再看深度学习。\n'
await fs.writeFile(progressPath, progressText, 'utf8')
runtime.recordSeedWorkspaceFile({
relativePath: progressRelativePath,
absolutePath: progressPath,
summary: '干扰项:复习进度',
metadata: {
format: 'text',
text: progressText
}
})
return {
deepLearningPath,
deepLearningRelativePath,
graphicsPath,
graphicsRelativePath,
originalDeepLearningBytes: await fs.readFile(deepLearningPath),
originalGraphicsBytes: await fs.readFile(graphicsPath)
}
}
export const seedKbGuidedDocxKb = async (
runtime: BackendTestRuntime,
workspaceSeed: KbGuidedDocxWorkspaceSeed
): Promise<string[]> => {
const kbDocuments: readonly SeedKbDocumentInput[] = [
{
id: 'doc-deep-learning',
name: '深度学习.docx',
filePath: workspaceSeed.deepLearningPath,
chunks: [
{
id: 'chunk-deep-learning-1',
text: '过度拟合是模型记住了过多训练数据中的特征值,使训练后的模型只对这部分训练数据有效,不太具有展望性。'
},
{
id: 'chunk-deep-learning-2',
text: '当训练时间过久时,会逐渐记住过多训练数据中的特征值,从而出现过度拟合。'
},
{
id: 'chunk-deep-learning-3',
text: '缓解过度拟合的思路包括:使用训练时未使用过的数据做验证,以及通过数据增强把同一图像拆成不同亮度、旋转方向等多张图像来训练。'
}
]
},
{
id: 'doc-graphics',
name: '图形学.docx',
filePath: workspaceSeed.graphicsPath,
chunks: [
{
id: 'chunk-graphics-1',
text: '反走样出现的原因是采样频率低于信号频率。'
},
{
id: 'chunk-graphics-2',
text: '由于一个像素可能同时落在图形边界内和边界外,边缘会出现锯齿。'
},
{
id: 'chunk-graphics-3',
text: '反走样的处理思路包括:在一个像素内模拟多个采样点,或者先把图形转换成信号后做低通滤波。'
}
]
},
{
id: 'doc-course-schedule',
name: '课程安排.docx',
filePath: path.join(runtime.workspaceRoot, '课程安排.docx'),
chunks: [
{
id: 'chunk-course-schedule-1',
text: '课程安排只记录考试时间、教室和分组信息。'
}
]
},
{
id: 'doc-ml-lab',
name: '机器学习实验记录.docx',
filePath: path.join(runtime.workspaceRoot, '机器学习实验记录.docx'),
chunks: [
{
id: 'chunk-ml-lab-1',
text: '实验记录主要关注学习率和 batch size 的调参过程。'
}
]
},
{
id: 'doc-reading-list',
name: '参考书目.md',
filePath: path.join(runtime.workspaceRoot, '参考书目.md'),
chunks: [
{
id: 'chunk-reading-list-1',
text: '参考书目里只有课程参考资料列表,没有过拟合或反走样的讲解。'
}
]
}
]
await runtime.seedKb(kbDocuments)
return kbDocuments.map((document) => document.id)
}
export const seedRiskReportWorkspace = async (
runtime: BackendTestRuntime
): Promise<RiskReportWorkspaceSeed> => {
@@ -0,0 +1,94 @@
import { describe, it } from 'vitest'
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../../harness/backend/llmConfig'
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
import {
HUMAN_KB_GUIDED_DOCX_READING_USER_INPUT,
seedKbGuidedDocxKb,
seedKbGuidedDocxWorkspace
} from '../../../../harness/backend/aiLiveTestData'
import {
collectStartedToolStringArgs,
filterAiRunEventsByType,
hasKbToolCall,
runWithStructuredOutputRetry
} from '../../../../ai/human/testUtils'
describe('temp ai human live backend', () => {
it('human-kb-guided-docx-reading-live', async () => {
await runWithStructuredOutputRetry(async () => {
await runBackendCase({
caseId: 'human-kb-guided-docx-reading-live',
harnessMode: 'live',
domain: 'temp',
suite: 'ai',
groups: ['human', 'live'],
description: '观察 live 模式下 human 在工作区资料较多时,会如何用 KB 缩圈并回答过拟合与反走样的处理思路。',
fixtures: [],
runMode: 'single',
runtimeModelName: hasLiveLlmConfigInEnv('primary')
? loadLiveLlmConfigFromEnv('primary').model_name
: null,
execute: async (runtime) => {
if (!hasLiveLlmConfigInEnv('primary')) {
return {
skipped: true,
output: null,
events: []
}
}
await runtime.seedLlmConfig(loadLiveLlmConfigFromEnv('primary'))
const workspaceSeed = await seedKbGuidedDocxWorkspace(runtime)
await seedKbGuidedDocxKb(runtime, workspaceSeed)
const runner = createHumanTestRunner(runtime)
const output = await runner.run({
taskId: runtime.taskId,
userInput: HUMAN_KB_GUIDED_DOCX_READING_USER_INPUT
})
return {
skipped: false,
output,
events: runtime.eventCollector.snapshot()
}
},
assert: async (result, runtime) => {
if (result.skipped) {
runtime.assertThat(true, {
name: 'live-config-not-provided',
message: '未提供 live 模型配置,已跳过 temp human live KB-guided docx reading case。'
})
return
}
const toolResults = filterAiRunEventsByType(result.events, 'tool_call_result')
const deliveryText = `${result.output?.summary || ''}\n${result.output?.deliveryMarkdown || ''}`
const openedSourcePaths = collectStartedToolStringArgs(
result.events,
'docx_open',
'sourcePath'
)
runtime.assertThat(
result.output?.status === 'completed' &&
hasKbToolCall(result.events) &&
/(数据增强|未使用过的数据|验证数据)/.test(deliveryText) &&
/(低通滤波|多个采样点|多个像素点|多次采样)/.test(deliveryText),
{
name: 'temp-human-live-kb-guided-reading-covers-kb-and-answer-facts',
message: 'temp KB-guided 读 case 应至少触发 KB,并回答过拟合与反走样的处理思路。',
data: {
openedSourcePaths,
toolNames: toolResults.map((event) => event.toolName),
summary: result.output?.summary,
deliveryMarkdown: result.output?.deliveryMarkdown
}
}
)
}
})
})
}, 600_000)
})
@@ -0,0 +1,229 @@
import { describe, expect, it } from 'vitest'
import type { AiRunContext } from '../../../src/main/services/ai/types'
import {
appendRoundRecord,
buildHandExecution,
buildLearningExecution
} from '../../../src/main/services/ai/human/memory/roundHistory'
import { buildBrainPrompt } from '../../../src/main/services/ai/human/prompts/brain'
import { buildMouthPrompt } from '../../../src/main/services/ai/human/prompts/mouth'
import {
renderRoundSection,
renderUnifiedHistoryText
} from '../../../src/main/services/ai/human/shared/historyRenderers'
import { buildReferenceScope } from '../../../src/main/services/ai/prompts/referenceScope'
import { createInitialHumanState } from '../../../src/main/services/ai/human/state/schema'
import type { HumanDecision } from '../../../src/main/services/ai/human/types'
const createStubAiRunContext = (referenceScope: AiRunContext['referenceScope']): AiRunContext => ({
runId: 'run-history-render',
referenceScope,
checkCancelled: () => {},
chatStructured: async () => {
throw new Error('chatStructured should not be called in prompt unit tests')
},
chatText: async () => {
throw new Error('chatText should not be called in prompt unit tests')
},
sendEvent: () => {},
requestApproval: async () => ({
approved: false
}),
loadHistory: async () => ({
recentTurnsText: '',
sessionSummaryText: ''
}),
trackDocxOpen: () => {},
trackDocxChanges: () => {},
resolveDocxOpenTarget: async () => {
throw new Error('resolveDocxOpenTarget should not be called in prompt unit tests')
},
documentsStore: undefined as never,
embeddingService: undefined as never
})
const buildDecision = (input: {
summary: string
reason: string
expectedOutcome: string
nextAction: HumanDecision['nextAction']
}): HumanDecision => ({
decision: {
summary: input.summary,
reason: input.reason,
expectedOutcome: input.expectedOutcome
},
readyToDeliver: false,
nextAction: input.nextAction
})
describe('ai unit human history renderers', () => {
it('renders shared history facts for both brain and mouth prompts', () => {
const initialState = createInitialHumanState({
taskId: 'task-history-shared',
userInput: '请总结这次任务做了什么'
})
const learningState = {
...initialState,
skills: {
learnedSkillIds: ['workspace-cli']
},
memory: {
...initialState.memory,
history: appendRoundRecord({
state: initialState,
decision: buildDecision({
summary: '补齐 workspace 能力',
reason: '需要 terminal 工具',
expectedOutcome: '拿到 workspace 读取能力',
nextAction: {
action: 'learning',
wantLearnSkillIds: ['workspace-cli']
}
}),
execution: buildLearningExecution({
learningOutcome: {
requestedSkillIds: ['workspace-cli'],
loadedSkillIds: ['workspace-cli'],
skippedDuplicateSkillIds: [],
skippedUnknownSkillIds: [],
evictedSkillIds: [],
activeToolCount: 1
}
}),
createdAt: 1
})
}
}
const handState = {
...learningState,
memory: {
...learningState.memory,
history: appendRoundRecord({
state: learningState,
decision: buildDecision({
summary: '读取资料并生成预览',
reason: '已经具备需要的能力',
expectedOutcome: '拿到 docx 预览产物',
nextAction: {
action: 'hand',
toolUsePlan: {
mode: 'serial',
calls: [
{
toolName: 'kb_hybrid_search_chunks',
args: {
queryText: 'approval denied'
}
},
{
toolName: 'docx_render_preview',
args: {
format: 'html'
}
}
]
}
}
}),
execution: buildHandExecution({
toolUsePlan: {
mode: 'serial',
calls: [
{
toolName: 'kb_hybrid_search_chunks',
args: {
queryText: 'approval denied'
}
},
{
toolName: 'docx_render_preview',
args: {
format: 'html'
}
}
]
},
toolResults: [
{
toolName: 'kb_hybrid_search_chunks',
ok: true,
code: 'ok',
data: {
items: [
{
document_name: 'Weekly Risk Summary.md',
text: '审批未通过时原文件保留'
}
]
}
},
{
toolName: 'docx_render_preview',
ok: true,
code: 'ok',
data: {
previewPath: '/tmp/preview.docx',
summary: 'preview generated (html)'
}
}
]
}),
createdAt: 2
})
}
}
const state = {
...handState,
memory: {
...handState.memory,
history: {
...handState.memory.history,
loadedHistoryContext: {
filters: {
executionKinds: ['learning'],
limit: 1
},
rounds: [handState.memory.history.roundHistory[0]]
}
}
}
}
const focusHistory = renderRoundSection({
records: [state.memory.history.roundHistory[0]],
detail: 'focus'
})
const unifiedHistory = renderUnifiedHistoryText({
roundHistory: state.memory.history.roundHistory,
loadedHistoryContext: state.memory.history.loadedHistoryContext
})
const brainPrompt = buildBrainPrompt({
state,
ctx: createStubAiRunContext(
buildReferenceScope({
workspace: {
workspaceId: 'workspace-1',
workspaceName: 'workspace',
rootPath: '/tmp/workspace-root'
}
})
)
})
const mouthPrompt = buildMouthPrompt({ state })
expect(focusHistory).toContain('决策了什么:')
expect(focusHistory).toContain('做了什么:')
expect(focusHistory).toContain('结果是什么:')
expect(unifiedHistory).toContain('过程历史:')
expect(unifiedHistory).toContain('主动回看历史:')
expect(unifiedHistory).toContain('Weekly Risk Summary.md')
expect(unifiedHistory).toContain('preview generated (html)')
expect(brainPrompt.user).toContain(focusHistory)
expect(mouthPrompt.user).toContain(unifiedHistory)
})
})
+19 -12
View File
@@ -71,16 +71,16 @@ describe('ai unit mouth readyToDeliver guard', () => {
})
}).system
expect(prompt).toContain('<source_of_truth>')
expect(prompt).toContain('<context_block_guide>')
expect(prompt).toContain('<available_information>')
expect(prompt).toContain('<output_contract>')
expect(prompt).toContain('final_decision 是 brain 已经确认的最终判断')
expect(prompt).toContain('runtime_artifacts 只允许准确引用已经给出的路径')
expect(prompt).toContain('你不会调用工具,也不会继续推进未完成步骤')
expect(prompt).toContain('不要把未来可能执行的工具动作写成“后续会自动完成”')
expect(prompt).toContain('你必须只输出最终 markdown 正文')
expect(prompt).not.toContain('"deliveryMarkdown"')
expect(prompt).not.toContain('"summary"')
expect(prompt).toContain('直接回答 task_goal')
expect(prompt).toContain('your_notes 优先')
expect(prompt).toContain('history:任务过程中真实发生过的决策、执行和结果')
expect(prompt).toContain('只输出最终 markdown 正文')
expect(prompt).toContain('标题、分节、列表都按需要使用,不固定')
expect(prompt).toContain('不原样照抄内部标签、block id、tool 名')
expect(prompt).not.toContain('# 处理结果')
expect(prompt).not.toContain('## 关键依据')
})
it('keeps mouth user-context blocks data-only and moves explanations into system', () => {
@@ -91,14 +91,21 @@ describe('ai unit mouth readyToDeliver guard', () => {
})
})
expect(result.system).toContain('final_decision:brain 已经确认的最终判断')
expect(result.system).toContain('默认使用最终用户答复的组织方式')
expect(result.system).toContain('task_goal:用户最初要你完成什么')
expect(result.system).toContain('已确认:当前稳定成立、可以直接告诉用户的事实')
expect(result.system).toContain('关键依据:支撑这些事实的证据')
expect(result.system).toContain('未决问题:仍会影响结论')
expect(result.user).not.toContain('这是当前交付模式。')
expect(result.user).not.toContain('这是用户这次原始诉求与限制。')
expect(result.user).not.toContain('这是 brain 已经做出的最终决定。')
expect(result.user).not.toContain('这是任务的稳定笔记')
expect(result.user).not.toContain('这是最近一轮最相关的过程记录。')
expect(result.user).not.toContain('这是更早几轮的过程链。')
expect(result.user).not.toContain('这是运行中已经产生的真实产物列表。')
expect(result.user).not.toContain('这是当前额外的收敛或纠偏提醒。')
expect(result.user).toContain('<history>')
expect(result.user).not.toContain('<correction_hint>')
expect(result.user).not.toContain('<recent_progress>')
expect(result.user).not.toContain('<earlier_attempts>')
expect(result.user).not.toContain('<recalled_history>')
})
})