Refine mouth delivery around task goals and history
This commit is contained in:
1 parent
2cef65ec7f
commit
117ce68e71
21 files changed
+1567
-323
No files matched your search
@@ -3,7 +3,7 @@
|
||||
## 1. 文档引用
|
||||
### 1.1 AI相关
|
||||
- 当涉及 AI 架构、主流程或 human-only runtime 时,必须阅读 `doc/ai-design/architecture.md`
|
||||
- 当涉及 AI 实现、运行入口、跨层边界、LLM 执行入口、流式传输、runtime settings、运行数据、`execution.watchHint` / `toolUsePlan` / `execution.lastLearningOutcome` / `memory.history.roundHistory` / `history_recall` / `recent_progress` / `earlier_attempts` / `recalled_history` 契约或工作区 KB 范围时,必须阅读 `doc/ai-design/implementation-spec.md` 与 `doc/ai-design/data-definition-and-flow.md`
|
||||
- 当涉及 AI 实现、运行入口、跨层边界、LLM 执行入口、流式传输、runtime settings、运行数据、`execution.watchHint` / `toolUsePlan` / `execution.lastLearningOutcome` / `memory.history.roundHistory` / `history_recall` / `mouth` 统一 `history` prompt 输入契约或工作区 KB 范围时,必须阅读 `doc/ai-design/implementation-spec.md` 与 `doc/ai-design/data-definition-and-flow.md`
|
||||
- 当涉及 AI 最小运行样例、默认链路或想快速理解 human 各环节如何衔接时,必须阅读 `doc/ai-design/example.md`
|
||||
- 当涉及 human 的 skill 分类、拆分边界、builtin/external 区分或当前 skill 基线时,必须阅读 `doc/ai-design/human-skill-design.md`
|
||||
- 当涉及 prompt 的通用结构、输入组织、输出规则、约束表达或风格一致性时,必须阅读 `doc/ai-design/prompt-design.md`
|
||||
@@ -16,6 +16,7 @@
|
||||
|
||||
### 1.3 测试相关
|
||||
- 当涉及 AI 测试,必须阅读 `doc/test-plan/ai-test.md`
|
||||
- 当涉及 human live 的 docx / KB 混合场景测试设计时,必须阅读 `doc/test-plan/ai-test.md`
|
||||
- 当涉及 Workbench 测试时,必须阅读 `doc/test-plan/workbench-test.md`
|
||||
- 当涉及 Settings 测试时,必须阅读 `doc/test-plan/settings-test.md`
|
||||
- 当涉及测试用例编写,测试规范时,必须阅读 `doc/test.md`
|
||||
|
||||
@@ -1100,20 +1100,21 @@ flowchart LR
|
||||
flowchart LR
|
||||
T[("task")]
|
||||
SM[("memory.stable")]
|
||||
PM[("memory.history.roundHistory")]
|
||||
PM[("memory.history.roundHistory / loadedHistoryContext")]
|
||||
CD[("execution.currentDecision")]
|
||||
A[("execution.artifacts")]
|
||||
WH[("execution.watchHint")]
|
||||
M["mouth"]
|
||||
R[("result")]
|
||||
TR[("trace.steps")]
|
||||
|
||||
T -->|读取 taskId + userInput| M
|
||||
SM -->|读取任务笔记| M
|
||||
PM -->|读取上一轮完整闭环| M
|
||||
PM -->|读取统一 history 事实视图| M
|
||||
CD -->|读取当前最终决策| M
|
||||
A -->|读取已收集的结构化 artifacts| M
|
||||
WH -->|必要时整理为交付提示| M
|
||||
M -->|输出 deliveryMarkdown;node 派生 summary、回写 completed result,并透传 artifacts| R
|
||||
M -->|追加 mouth 执行轨迹| TR
|
||||
```
|
||||
|
||||
- `mouth` 不再依赖 `recent_progress / earlier_attempts / recalled_history` 这类拆分块;统一基于 `roundHistory + loadedHistoryContext` 渲染单一 `history` 视图。
|
||||
- `mouth` 的 prompt 输入仍以 `your_notes` 为优先事实来源;当 `your_notes` 不够具体时,再从统一 `history` 与 `runtime_artifacts` 补足用户回答所需细节,而不是依赖额外的 `watchHint` 说明。
|
||||
@@ -94,10 +94,10 @@ stateDiagram-v2
|
||||
|
||||
### 5.4 `mouth`
|
||||
|
||||
- 职责:把当前已经确定的状态整理成最终用户交付,并回写 `HumanResult`。
|
||||
- 主要读取:`task`、`memory.stable`、`execution.currentDecision`、`execution.artifacts`。
|
||||
- 职责:围绕 `task_goal` 把当前已经确定的状态组织成最终用户交付,并回写 `HumanResult`。
|
||||
- 主要读取:`task`、`memory.stable`、`memory.history.roundHistory`、`memory.history.loadedHistoryContext`、`execution.currentDecision`、`execution.artifacts`。
|
||||
- 主要回写:`result`、`execution.status`、`execution.stage`、`trace.steps`。
|
||||
- 约束:模型输出必须通过 `ctx.chatText`;输出固定面向最终用户;若开启 `emitAnswerEvents`,增量文本通过 `answer_token` 发出,完成后通过 `answer_completed` 收口。
|
||||
- 约束:模型输出必须通过 `ctx.chatText`;输出面向最终用户,优先直接完成用户原始诉求;可使用自由段落或列表组织,不要求固定报告模板;若开启 `emitAnswerEvents`,增量文本通过 `answer_token` 发出,完成后通过 `answer_completed` 收口。
|
||||
|
||||
## 6. 事件与交付
|
||||
|
||||
|
||||
@@ -99,13 +99,15 @@
|
||||
- 验收:
|
||||
- `WB-CORE-005-AC1` `terminal` 执行成功后,后续用户可见输出会体现关键命令结果,而不是只表现为“命令执行过”。
|
||||
- `WB-CORE-005-AC2` `terminal` 输出不足或执行失败时,assistant 会明确说明无法从当前结果得到所需信息,而不是编造结论或文档内容。
|
||||
- `WB-CORE-005-AC3` 用户可见输出优先回答用户原始诉求,不要求固定报告体;若引用执行结果,应使用用户能理解的表达,而不是暴露内部工具或过程标签。
|
||||
|
||||
#### WB-CORE-006 标准任务应能闭环到可交付结果
|
||||
|
||||
- 系统行为:当用户提供了足够上下文的标准任务,且当前不需要额外阻塞或确认时,workbench 应支持 AI 沿同一条任务链完成资料使用、必要执行和最终交付,而不是只停留在搜索结果、工具调用或过程提示。
|
||||
- 系统行为:当用户提供了足够上下文的标准任务,且当前不需要额外阻塞或确认时,workbench 应支持 AI 沿同一条任务链完成资料使用、必要执行和最终交付,而不是只停留在搜索结果、工具调用或过程提示;最终收口应围绕用户原始诉求组织,而不是强制套用固定报告模板。
|
||||
- 验收:
|
||||
- `WB-CORE-006-AC1` 用户已提供足够上下文且无需额外阻塞时,系统能把标准任务推进到可消费的最终结果或明确产物,而不是停留在中间态。
|
||||
- `WB-CORE-006-AC2` 最终结果会体现用户显式给定的资料范围与执行得到的关键信息,不会把范围、执行结果和最终交付割裂开。
|
||||
- `WB-CORE-006-AC3` 最终结果默认直接回答用户显式提出的问题;若同时需要说明产物、依据或限制,也应以用户可理解的语言表达,而不是依赖内部 block id、tool 名或固定章节标题。
|
||||
|
||||
### 8.2 会话管理
|
||||
|
||||
|
||||
@@ -247,10 +247,10 @@
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 `hand` 的并行工具执行与聚合守卫。
|
||||
- What:验证 `hand` 的并行工具执行上限、只读安全约束与结果聚合守卫。
|
||||
- Why:工具执行器本身必须在真实结果下稳定收敛。
|
||||
- How:通过真实 `hand` 执行多工具调用并检查状态收敛。
|
||||
- Oracle:并行工具结果被正确聚合,运行状态无串扰。
|
||||
- How:通过真实 `hand` 执行安全并行批次、`6` 个只读安全并行批次、超过上限的 `7` 个并行批次以及包含 mutating 调用的不安全批次。
|
||||
- Oracle:`6` 个只读且 `parallelSafe` 的工具调用会被真实执行并正确聚合;`7` 个并行调用会在执行前被 `parallel_limit_exceeded` 拦截;包含 mutating 或非并行安全工具的批次会在执行前被 `parallel_requires_read_only` 拦截。
|
||||
- Not:不验证 `brain` 决策来源。
|
||||
|
||||
##### `hand-serial-docx-chain`
|
||||
@@ -416,12 +416,42 @@
|
||||
- What:验证 live 模式下 `human` 会在一个真实长任务里串起模板定位、资料取证、docx 编辑与预览交付。
|
||||
- Why:`human/live` 需要至少一条长期高价值复杂样例,覆盖 workspace、KB、docx 与最终交付链路。
|
||||
- How:注入真实 live LLM 配置;workspace 中动态写入带 `RISK_TODO / OWNER_TODO` 占位符的周报模板和干扰文件;seed 两份真实 KB 文档分别提供风险结论与负责人说明并限制在已选范围内;直接输入“找到上周 Word 周报模板,按已选资料补全风险结论与负责人,生成预览且不要覆盖原文件”的自然语言诉求,让真实 `human` 自主完成。
|
||||
- Oracle:任务最终完成;事件流中至少出现一次 `terminal_run_command`、一次以 `kb_` 开头的检索工具调用、一次 `docx_open`、一次 `docx_apply_ops` 与一次 `docx_render_preview`;存在 preview/draft artifact,且原模板未被覆盖。
|
||||
- Oracle:任务最终完成;事件流中至少出现一次 `terminal_run_command`、一次以 `kb_` 开头的检索工具调用、一次 `docx_open`、一次 `docx_apply_ops` 与一次 `docx_render_preview`;存在 preview/draft artifact,且原模板未被覆盖;最终答复能从用户视角说明改动的是“风险结论”和“负责人说明”两处、依据分别来自 `Weekly Risk Summary.md` 与 `Owner Note.md`,并明确已经生成预览且未覆盖原文件。
|
||||
- Not:不绑定最终答案措辞,也不要求固定检索工具顺序。
|
||||
|
||||
##### `human-docx-cause-remedy-reading-live`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 live 模式下 `human` 会基于两份已选 `docx` 做跨文件精读,并准确回答“过拟合 / 反走样”的成因与处理思路。
|
||||
- Why:真实用户在复习场景里经常已经选中了资料,但仍会把两个概念混在一起;这条 case 要长期守住“已选资料范围 + 跨文件准确回答”的正式能力,对齐 `WB-CORE-002-AC2`、`WB-CORE-004-AC1`、`WB-CORE-006-AC1`、`WB-CORE-006-AC2`。
|
||||
- How:注入真实 live LLM 配置;把 `tests/fixtures/ai/docx/深度学习.docx` 与 `tests/fixtures/ai/docx/图形学.docx` 作为已选资料复制进 workspace;直接输入“我做题老把这两个点混在一起。你就按我刚选的这两份资料,帮我说清楚:过拟合和反走样各自是什么、为什么会出现、一般怎么处理。别补充课外知识,哪份文档没写你就直接说没写。”的自然语言诉求,让真实 `human` 自主完成。
|
||||
- Oracle:任务最终完成;事件流中至少两次 `docx_open` 覆盖两份目标文档,且至少出现一次 `docx_search`、`docx_outline` 或 `docx_get_block`;最终回答同时命中“过拟合”的成因/处理事实组和“反走样”的成因/处理事实组;不出现 `docx_apply_ops` 或 `docx_save`。
|
||||
- Not:不要求固定回答结构,也不对最终文案做逐字断言。
|
||||
|
||||
##### `human-docx-contextual-comparison-draft-live`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 live 模式下 `human` 会基于两份已选 `docx`,在目标 `docx` 的相关位置追加解释性段落和对照表,并生成不覆盖原文件的预览稿。
|
||||
- Why:真实用户常见诉求不是“改占位符”,而是“在这块后面补一段更好懂的说明”;这条 case 覆盖跨文件取材、相关位置补写、结构化新增与 preview-only 交付,对齐 `WB-CORE-002-AC2`、`WB-CORE-003-AC1`、`WB-CORE-004-AC1`、`WB-CORE-006-AC1`、`WB-CORE-006-AC2`。
|
||||
- How:注入真实 live LLM 配置;把 `tests/fixtures/ai/docx/深度学习.docx` 与 `tests/fixtures/ai/docx/图形学.docx` 作为已选资料复制进 workspace;直接输入“《图形学》里‘反走样’这一节我还是记不住。你参考我刚选的这两份 Word,直接在这一节最后补一小段更好理解的说明,再加个简单对照表,一列写反走样,一列写《深度学习》里的过拟合。直接给我可预览的版本就行,别改原文件,也不用先回我文字草稿。”的自然语言诉求,让真实 `human` 自主完成。
|
||||
- Oracle:任务最终完成;事件流中至少两次 `docx_open` 覆盖两份目标文档;`docx_apply_ops` 前至少出现一次针对“反走样”的读取定位;至少出现一次 `docx_apply_ops` 与一次 `docx_render_preview`;存在 preview/draft artifact;不出现 `docx_save`;原始两份 `docx` 文件未被覆盖;写入请求里能看到新增说明段落与表格结构。
|
||||
- Not:不要求固定 op 组合,也不对最终说明文案做逐字断言。
|
||||
|
||||
##### `human-kb-guided-docx-comparison-draft-live`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 live 模式下 `human` 在本地资料较多时,会先用 KB 缩圈定位相关资料,再回到目标 `docx` 做跨文件补写并交付预览。
|
||||
- Why:真实用户常常记得主题、不记得文件名;这条 case 要长期守住“工作区默认范围 + KB 缩圈 + docx 继续执行”的正式能力,对齐 `WB-CORE-002-AC2`、`WB-CORE-004-AC1`、`WB-CORE-006-AC1`、`WB-CORE-006-AC2`。
|
||||
- How:注入真实 live LLM 配置;workspace 中动态写入 `深度学习.docx`、`图形学.docx` 和若干干扰文件,并把目标 `docx` 与干扰资料同时 seed 到真实 KB;不传 `selectedReferences`,只绑定 workspace;直接输入“我工作区资料有点多,我忘了是哪份课件在讲反走样了。你先帮我找到那份,再参考讲过拟合的那份资料,直接在‘反走样’那一节最后补一段更好记的说明,再加个简单对照表,一列写反走样,一列写过拟合。直接给我可预览的版本就行,别改原文件,也不用先回我文字草稿。”的自然语言诉求,让真实 `human` 自主完成。
|
||||
- Oracle:任务最终完成;事件流中至少出现一次 KB 工具调用;后续出现 `docx_open` 并覆盖最终参与编辑的目标 `docx`;至少出现一次 `docx_apply_ops` 与一次 `docx_render_preview`;存在 preview/draft artifact;不出现 `docx_save`;目标原始 `docx` 未被覆盖;新增内容同时涉及“反走样”和“过拟合”。
|
||||
- Not:不强制固定执行路径,也不要求 KB 缩圈后必须再打开所有候选资料。
|
||||
|
||||
## 5. 验收口径
|
||||
|
||||
- `tool` suite 必须持续守住真实工具输入输出、审批边界和 artifact 契约。
|
||||
- `human/mock-llm` 必须覆盖 `AiRunService -> human` 入口、`brain / learning / hand / mouth` 主链、memory 延续、watch hint 收敛与复杂工具协作。
|
||||
- `human/live` 必须至少保留 1 条简单任务和 1 条复杂任务正式 case。
|
||||
- `human` suite 必须继续支撑 `WB-CORE-002`、`WB-CORE-005`、`WB-CORE-006`。
|
||||
- `human` suite 必须继续支撑 `WB-CORE-002`、`WB-CORE-003`、`WB-CORE-004`、`WB-CORE-005`、`WB-CORE-006`。
|
||||
@@ -63,6 +63,7 @@ export const HUMAN_WATCH_HINT_CODES = [
|
||||
|
||||
export const HUMAN_MAX_ACTIVE_TOOLS = 20
|
||||
export const HUMAN_DEFAULT_TOOL_CALL_LIMIT = 50
|
||||
export const HUMAN_MAX_PARALLEL_TOOL_CALLS = 6
|
||||
export const HUMAN_ROUND_EXECUTION_KINDS = [
|
||||
'pre_execution_blocked',
|
||||
'learning',
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import type { JsonObject } from '../../../../../../shared/types'
|
||||
import { HUMAN_DEFAULT_TOOL_CALL_LIMIT } from '../../constants'
|
||||
import { HUMAN_DEFAULT_TOOL_CALL_LIMIT, HUMAN_MAX_PARALLEL_TOOL_CALLS } from '../../constants'
|
||||
import type { AiRunContext } from '../../../types'
|
||||
import { extractArtifactRefsFromToolResults, mergeArtifactRefs } from '../../../utils/artifacts'
|
||||
import {
|
||||
@@ -14,8 +14,6 @@ import type { HumanRuntimeCache, HumanState, HumanWatchHint, ToolCallResult, Too
|
||||
import { executeToolCall } from './toolExecutor'
|
||||
import { returnToBrain, returnToBrainAfterTrace, returnToBrainWithWatchHint } from '../utils/stateTransitions'
|
||||
|
||||
const MAX_PARALLEL_TOOL_CALLS = 4
|
||||
|
||||
export const getToolExecutionMeta = (input: {
|
||||
toolName: string
|
||||
args: JsonObject
|
||||
@@ -70,13 +68,13 @@ const buildToolLimitWatchHint = (toolCallCount: number, limit: number): HumanWat
|
||||
|
||||
const validateToolUsePlan = (toolUsePlan: ToolUsePlan): HumanWatchHint | undefined => {
|
||||
if (toolUsePlan.mode === 'serial') return undefined
|
||||
if (toolUsePlan.calls.length > MAX_PARALLEL_TOOL_CALLS) {
|
||||
if (toolUsePlan.calls.length > HUMAN_MAX_PARALLEL_TOOL_CALLS) {
|
||||
return createHandPlanInvalidWatchHint({
|
||||
mode: 'parallel',
|
||||
reason: 'parallel_limit_exceeded',
|
||||
offendingToolNames: toolUsePlan.calls.map((item) => item.toolName),
|
||||
callCount: toolUsePlan.calls.length,
|
||||
limit: MAX_PARALLEL_TOOL_CALLS
|
||||
limit: HUMAN_MAX_PARALLEL_TOOL_CALLS
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -13,10 +13,9 @@ import {
|
||||
listExternalSkillDefinitions
|
||||
} from '../skills'
|
||||
import { renderHumanStableNotes } from '../shared/memoryRenderers'
|
||||
import { renderLoadedHistoryText, renderRoundSection } from '../shared/historyRenderers'
|
||||
import { getToolPromptSpec } from '../tools/registry'
|
||||
import type {
|
||||
HumanLoadedHistoryContext,
|
||||
HumanRoundRecord,
|
||||
HumanState,
|
||||
ToolPromptSpec
|
||||
} from '../types'
|
||||
@@ -43,168 +42,6 @@ const renderLearnedToolSpecLine = (spec: ToolPromptSpec): string => {
|
||||
return `- 工具 ${spec.name}:用途=${spec.purpose};必填参数=[${requiredArgs || 'none'}];${example}`
|
||||
}
|
||||
|
||||
const renderDecisionNextAction = (record: HumanRoundRecord): string => {
|
||||
switch (record.decision.nextAction.action) {
|
||||
case 'learning':
|
||||
return `- 你准备下一步做:准备学习 ${renderQuotedNames(
|
||||
record.decision.nextAction.wantLearnSkillIds
|
||||
)}。`
|
||||
case 'hand':
|
||||
return `- 你准备下一步做:准备进入 hand,${
|
||||
record.decision.nextAction.toolUsePlan.mode === 'parallel'
|
||||
? `以 parallel 模式并行执行 ${renderQuotedNames(
|
||||
record.decision.nextAction.toolUsePlan.calls.map((call) => call.toolName)
|
||||
)}。`
|
||||
: `以 serial 模式按顺序执行 ${renderQuotedNames(
|
||||
record.decision.nextAction.toolUsePlan.calls.map((call) => call.toolName)
|
||||
)}。`
|
||||
}`
|
||||
case 'mouth':
|
||||
return '- 你准备下一步做:准备进入 mouth 组织最终交付。'
|
||||
}
|
||||
}
|
||||
|
||||
const renderRoundDecision = (record: HumanRoundRecord): string =>
|
||||
`
|
||||
- 你的判断:${record.decision.decision.summary}
|
||||
- 你这样判断的原因:${record.decision.decision.reason}
|
||||
- 你当时期待的结果:${record.decision.decision.expectedOutcome}
|
||||
- ${record.decision.readyToDeliver ? '你当时认为已经可以交付。' : '你当时认为还不能直接交付。'}
|
||||
${renderDecisionNextAction(record)}
|
||||
`.trim()
|
||||
|
||||
const renderRoundExecution = (
|
||||
record: HumanRoundRecord,
|
||||
detail: 'focus' | 'chain' | 'loaded'
|
||||
): string => {
|
||||
switch (record.execution.kind) {
|
||||
case 'pre_execution_blocked':
|
||||
return '- 没有进入真实执行,系统在执行前阻断了这轮动作。'
|
||||
case 'learning':
|
||||
return '- 进入 learning,按请求尝试补齐外置技能。'
|
||||
case 'hand':
|
||||
return `
|
||||
- 进入 hand,执行模式是 ${record.execution.toolUsePlan.mode}。
|
||||
- ${
|
||||
record.execution.toolUsePlan.mode === 'parallel'
|
||||
? `实际并行执行了 ${renderQuotedNames(record.execution.toolUsePlan.calls.map((call) => call.toolName))}。`
|
||||
: `实际按顺序执行了 ${renderQuotedNames(record.execution.toolUsePlan.calls.map((call) => call.toolName))}。`
|
||||
}
|
||||
${detail !== 'chain' ? `- 本轮计划包含 ${record.execution.toolUsePlan.calls.length} 个工具调用。` : ''}
|
||||
`
|
||||
.trim()
|
||||
.replace(/\n\n+/g, '\n')
|
||||
case 'mouth':
|
||||
return '- 进入 mouth,组织最终说明。'
|
||||
}
|
||||
}
|
||||
|
||||
const renderRoundResult = (record: HumanRoundRecord): string => {
|
||||
switch (record.execution.kind) {
|
||||
case 'pre_execution_blocked':
|
||||
return renderWatchHintForBrain(record.execution.watchHint)
|
||||
case 'learning':
|
||||
return `
|
||||
- 已加载的技能:${renderQuotedNames(record.execution.learningOutcome.loadedSkillIds)}
|
||||
- 已跳过的重复技能:${renderQuotedNames(record.execution.learningOutcome.skippedDuplicateSkillIds)}
|
||||
- 未识别的技能:${renderQuotedNames(record.execution.learningOutcome.skippedUnknownSkillIds)}
|
||||
- 当前 active tool 数:${String(record.execution.learningOutcome.activeToolCount)}
|
||||
${record.execution.watchHint ? `- 收敛提示:${renderWatchHintForBrain(record.execution.watchHint)}` : ''}
|
||||
`
|
||||
.trim()
|
||||
.replace(/\n\n+/g, '\n')
|
||||
case 'hand': {
|
||||
const toolResultLines = record.execution.toolResults
|
||||
.map((result) =>
|
||||
[
|
||||
`- \`${result.toolName}\`:${result.summary}`,
|
||||
result.dataPreview ? `结果:${result.dataPreview}` : '',
|
||||
result.diagnosticText ? `诊断:${result.diagnosticText}` : ''
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join(';')
|
||||
)
|
||||
.join('\n')
|
||||
const artifactLine =
|
||||
record.execution.artifactRefs && record.execution.artifactRefs.length > 0
|
||||
? `- 产物:${renderInlineList(
|
||||
record.execution.artifactRefs.map((artifact) =>
|
||||
artifact.kind === 'file'
|
||||
? `\`${artifact.fileName}\``
|
||||
: `draft review ${artifact.review.reviewId}`
|
||||
),
|
||||
{
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
}
|
||||
)}`
|
||||
: ''
|
||||
const watchHintLine = record.execution.watchHint
|
||||
? `- 收敛提示:${renderWatchHintForBrain(record.execution.watchHint)}`
|
||||
: ''
|
||||
|
||||
return [toolResultLines, artifactLine, watchHintLine].filter(Boolean).join('\n')
|
||||
}
|
||||
case 'mouth':
|
||||
return `
|
||||
- 输出摘要:${record.execution.summary}
|
||||
${record.execution.completed ? '- 任务已完成。' : '- 任务未完成。'}
|
||||
`.trim()
|
||||
}
|
||||
}
|
||||
|
||||
const renderRoundRecord = (
|
||||
record: HumanRoundRecord,
|
||||
detail: 'focus' | 'chain' | 'loaded'
|
||||
): string =>
|
||||
`
|
||||
- 第 ${record.roundId} 轮
|
||||
|
||||
决策了什么:
|
||||
${renderRoundDecision(record)}
|
||||
|
||||
做了什么:
|
||||
${renderRoundExecution(record, detail)}
|
||||
|
||||
结果是什么:
|
||||
${renderRoundResult(record)}
|
||||
`.trim()
|
||||
|
||||
const renderRoundSection = (input: {
|
||||
records: HumanRoundRecord[]
|
||||
detail: 'focus' | 'chain' | 'loaded'
|
||||
}): string =>
|
||||
input.records.length > 0
|
||||
? input.records.map((record) => renderRoundRecord(record, input.detail)).join('\n\n')
|
||||
: 'none'
|
||||
|
||||
const renderLoadedHistoryText = (
|
||||
loadedHistoryContext: HumanLoadedHistoryContext | undefined
|
||||
): string =>
|
||||
loadedHistoryContext
|
||||
? `
|
||||
过滤条件:roundIds=${renderInlineList((loadedHistoryContext.filters.roundIds || []).map(String), {
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
})};executionKinds=${renderInlineList(loadedHistoryContext.filters.executionKinds || [], {
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
})};toolNames=${renderInlineList(loadedHistoryContext.filters.toolNames || [], {
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
})};watchHintCodes=${renderInlineList(loadedHistoryContext.filters.watchHintCodes || [], {
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
})};limit=${String(loadedHistoryContext.filters.limit || loadedHistoryContext.rounds.length)}
|
||||
|
||||
${renderRoundSection({
|
||||
records: loadedHistoryContext.rounds,
|
||||
detail: 'loaded'
|
||||
})}
|
||||
`
|
||||
.trim()
|
||||
: 'none'
|
||||
|
||||
const renderAvailableSkillsSection = (state: HumanState): string => {
|
||||
const effectiveSkillIds = getEffectiveLearnedSkillIds(state.skills.learnedSkillIds)
|
||||
const effectiveSkillDefinitions = effectiveSkillIds.map((skillId) => getSkillDefinition(skillId))
|
||||
|
||||
@@ -1,132 +1,41 @@
|
||||
import { renderInlineList } from '../../utils/render'
|
||||
import { getLastRoundFocus } from '../memory/roundHistory'
|
||||
import { getLoadedHistoryContext } from '../memory/roundHistory'
|
||||
import { renderUnifiedHistoryText } from '../shared/historyRenderers'
|
||||
import { renderHumanStableNotes } from '../shared/memoryRenderers'
|
||||
import { renderHumanArtifactLines } from '../shared/runtimeRenderers'
|
||||
import type { HumanRoundRecord, HumanState } from '../types'
|
||||
import { renderWatchHintForMouth } from './watchHints'
|
||||
|
||||
const renderRecentProgressSection = (record: HumanRoundRecord | undefined): string => {
|
||||
if (!record) {
|
||||
return 'none'
|
||||
}
|
||||
|
||||
const progressText =
|
||||
record.execution.kind === 'pre_execution_blocked'
|
||||
? `- 执行前阻断:${renderWatchHintForMouth(record.execution.watchHint)}`
|
||||
: record.execution.kind === 'learning'
|
||||
? `- learning 结果:已加载 ${renderInlineList(
|
||||
record.execution.learningOutcome.loadedSkillIds.map((skillId) => `\`${skillId}\``),
|
||||
{
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
}
|
||||
)};重复 ${renderInlineList(
|
||||
record.execution.learningOutcome.skippedDuplicateSkillIds.map(
|
||||
(skillId) => `\`${skillId}\``
|
||||
),
|
||||
{
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
}
|
||||
)};未知 ${renderInlineList(
|
||||
record.execution.learningOutcome.skippedUnknownSkillIds.map(
|
||||
(skillId) => `\`${skillId}\``
|
||||
),
|
||||
{
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
}
|
||||
)}。`
|
||||
: record.execution.kind === 'hand'
|
||||
? `
|
||||
- hand 执行模式:${record.execution.toolUsePlan.mode}
|
||||
${record.execution.toolResults.map((result) => `- \`${result.toolName}\`:${result.summary}`).join('\n')}
|
||||
`
|
||||
.trim()
|
||||
.replace(/\n\n+/g, '\n')
|
||||
: `- mouth 输出摘要:${record.execution.summary}`
|
||||
|
||||
return `
|
||||
- 第 ${record.roundId} 轮判断:${record.decision.decision.summary}
|
||||
- 原因:${record.decision.decision.reason}
|
||||
- 预期结果:${record.decision.decision.expectedOutcome}
|
||||
${progressText}
|
||||
`.trim()
|
||||
}
|
||||
import type { HumanState } from '../types'
|
||||
|
||||
const buildFinalDecisionText = (state: HumanState): string =>
|
||||
state.execution.currentDecision
|
||||
? `最终决定(readyToDeliver=${String(state.execution.currentDecision.readyToDeliver)}):“${state.execution.currentDecision.decision.summary}”;原因:${state.execution.currentDecision.decision.reason};预期结果:${state.execution.currentDecision.decision.expectedOutcome}`
|
||||
: 'none'
|
||||
|
||||
const OUTPUT_EXAMPLE = `
|
||||
# 处理结果
|
||||
|
||||
已完成当前任务,下面是基于现有事实和结果整理出的最终结论。
|
||||
|
||||
## 关键依据
|
||||
- 依据一
|
||||
|
||||
## 产物与引用
|
||||
- /path/to/file.docx
|
||||
|
||||
## 当前限制
|
||||
无
|
||||
`.trim()
|
||||
|
||||
export const buildMouthPrompt = (input: { state: HumanState }) => {
|
||||
const system = `
|
||||
<system>
|
||||
<role>
|
||||
你现在扮演一个负责对外表达最终结果的人。
|
||||
你的任务是把已经确定下来的判断、任务笔记和产物整理成最终交付。
|
||||
你现在负责根据已有事实回答用户这次任务。
|
||||
你的目标不是复述内部过程,而是直接完成 task_goal 里用户真正要的回复。
|
||||
</role>
|
||||
|
||||
<working_style>
|
||||
请像一个表达清晰、判断克制的人一样输出结果:
|
||||
不夸大、不补编、不引入新的结论,
|
||||
只把已经确认的信息组织成最终交付。
|
||||
</working_style>
|
||||
|
||||
<source_of_truth>
|
||||
- final_decision 是 brain 已经确认的最终判断;你负责表达,不负责重新推翻或改写它。
|
||||
- your_notes 是当前任务的稳定事实来源,包括已确认事实、关键依据和未决问题;正文中的事实与依据优先从这里取。
|
||||
- recent_progress 只用于补充最近完成到哪里、最近结果如何,不必把内部过程完整复述出来。
|
||||
- runtime_artifacts 只允许准确引用已经给出的路径、文件名、状态和摘要,不要补编额外元数据。
|
||||
- correction_hint 只用于如实说明限制、阻断或未完成点,不要把它扩写成新的事实。
|
||||
</source_of_truth>
|
||||
|
||||
<context_block_guide>
|
||||
- task_goal:用户原始诉求与限制。
|
||||
- final_decision:brain 已经确认的最终判断与收口方向。
|
||||
- your_notes:稳定任务笔记,按“已确认 / 关键依据 / 未决问题”组织。
|
||||
- recent_progress:最近一轮过程记录,用于补充当前完成到哪里。
|
||||
- runtime_artifacts:运行中已经产生的真实产物列表。
|
||||
- correction_hint:当前额外的收敛或纠偏提醒;若无则为 none。
|
||||
</context_block_guide>
|
||||
|
||||
<delivery_principles>
|
||||
1. 如果任务已经完成,先直接给出结论,再给关键依据,再给结果或产物说明。
|
||||
2. 如果当前结果有限,先说明已经完成到哪里,再说明限制和继续需要什么。
|
||||
3. 只使用已经给你的任务笔记、最终判断和产物,不要补充新的事实或推理。
|
||||
4. 语言要清楚、克制、可执行,不要堆内部过程。
|
||||
5. 如果存在运行时产物或最近过程记录,只能准确引用已经给出的路径、状态和摘要。
|
||||
6. 你不会调用工具,也不会继续推进未完成步骤;如果仍有步骤没做,只能如实说明当前停在哪。
|
||||
</delivery_principles>
|
||||
<available_information>
|
||||
- task_goal:用户最初要你完成什么,以及用户显式要求你回答哪些点。
|
||||
- final_decision:brain 最后确认的收口方向。
|
||||
- your_notes:当前稳定事实来源。其中:
|
||||
- 已确认:当前稳定成立、可以直接告诉用户的事实。
|
||||
- 关键依据:支撑这些事实的证据、来源或命中结果。
|
||||
- 未决问题:仍会影响结论、或需要用户补充确认的点。
|
||||
- history:任务过程中真实发生过的决策、执行和结果,可用于补足 your_notes 不够具体的地方。
|
||||
- runtime_artifacts:运行中已经产生的真实产物,可按给定路径、文件名、状态和摘要引用。
|
||||
</available_information>
|
||||
|
||||
<output_contract>
|
||||
- 你必须只输出最终 markdown 正文,不要输出任何额外说明、JSON 或代码块。
|
||||
- 不要引入新的事实、依据、步骤或承诺。
|
||||
- 不要暴露内部字段名、节点名或系统术语。
|
||||
- 如果依据不足,就保持保守表达,不要写成确定结论。
|
||||
- 不要把未来可能执行的工具动作写成“后续会自动完成”或“已经在进行中”。
|
||||
- 输出必须是完整结果,而不是零散句子或局部片段。
|
||||
- 默认使用最终用户答复的组织方式:先给结论,再给关键依据,再给产物或当前限制。
|
||||
- 只输出最终 markdown 正文,不要输出 JSON 或额外说明。
|
||||
- 直接回答 task_goal,优先覆盖用户显式提出的问题。
|
||||
- your_notes 优先;如果 your_notes 不够具体,可结合 history 与 runtime_artifacts 补足。
|
||||
- 只基于提供的信息回答,不要补编新的事实。
|
||||
- 标题、分节、列表都按需要使用,不固定。
|
||||
- 默认使用用户能理解的语言,不原样照抄内部标签、block id、tool 名,除非用户明确要求这些内部细节。
|
||||
</output_contract>
|
||||
|
||||
<output_example>
|
||||
${OUTPUT_EXAMPLE}
|
||||
</output_example>
|
||||
</system>
|
||||
`.trim()
|
||||
|
||||
@@ -144,17 +53,16 @@ ${buildFinalDecisionText(input.state)}
|
||||
${renderHumanStableNotes(input.state, 'mouth_prompt')}
|
||||
</your_notes>
|
||||
|
||||
<recent_progress>
|
||||
${renderRecentProgressSection(getLastRoundFocus(input.state))}
|
||||
</recent_progress>
|
||||
<history>
|
||||
${renderUnifiedHistoryText({
|
||||
roundHistory: input.state.memory.history.roundHistory,
|
||||
loadedHistoryContext: getLoadedHistoryContext(input.state)
|
||||
})}
|
||||
</history>
|
||||
|
||||
<runtime_artifacts>
|
||||
${renderHumanArtifactLines(input.state, 'prompt')}
|
||||
</runtime_artifacts>
|
||||
|
||||
<correction_hint>
|
||||
${input.state.execution.watchHint ? renderWatchHintForMouth(input.state.execution.watchHint) : 'none'}
|
||||
</correction_hint>
|
||||
</user_context>
|
||||
`.trim()
|
||||
|
||||
|
||||
@@ -78,8 +78,4 @@ export const renderWatchHintForBrain = (hint: HumanWatchHint): string => {
|
||||
}
|
||||
}
|
||||
|
||||
export const renderWatchHintForMouth = (hint: HumanWatchHint): string => {
|
||||
return summarizeWatchHint(hint)
|
||||
}
|
||||
|
||||
export const renderWatchHintDetail = (hint: HumanWatchHint): JsonObject => asJsonObject(hint.detail)
|
||||
@@ -0,0 +1,191 @@
|
||||
import { renderInlineList } from '../../utils/render'
|
||||
import type { HumanLoadedHistoryContext, HumanRoundRecord } from '../types'
|
||||
import { renderWatchHintForBrain } from '../prompts/watchHints'
|
||||
|
||||
const renderQuotedNames = (items: readonly string[]): string =>
|
||||
renderInlineList(
|
||||
items.map((item) => `\`${item}\``),
|
||||
{
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
}
|
||||
)
|
||||
|
||||
const renderDecisionNextAction = (record: HumanRoundRecord): string => {
|
||||
switch (record.decision.nextAction.action) {
|
||||
case 'learning':
|
||||
return `- 你准备下一步做:准备学习 ${renderQuotedNames(
|
||||
record.decision.nextAction.wantLearnSkillIds
|
||||
)}。`
|
||||
case 'hand':
|
||||
return `- 你准备下一步做:准备进入 hand,${
|
||||
record.decision.nextAction.toolUsePlan.mode === 'parallel'
|
||||
? `以 parallel 模式并行执行 ${renderQuotedNames(
|
||||
record.decision.nextAction.toolUsePlan.calls.map((call) => call.toolName)
|
||||
)}。`
|
||||
: `以 serial 模式按顺序执行 ${renderQuotedNames(
|
||||
record.decision.nextAction.toolUsePlan.calls.map((call) => call.toolName)
|
||||
)}。`
|
||||
}`
|
||||
case 'mouth':
|
||||
return '- 你准备下一步做:准备进入 mouth 组织最终交付。'
|
||||
}
|
||||
}
|
||||
|
||||
const renderRoundDecision = (record: HumanRoundRecord): string =>
|
||||
`
|
||||
- 你的判断:${record.decision.decision.summary}
|
||||
- 你这样判断的原因:${record.decision.decision.reason}
|
||||
- 你当时期待的结果:${record.decision.decision.expectedOutcome}
|
||||
- ${record.decision.readyToDeliver ? '你当时认为已经可以交付。' : '你当时认为还不能直接交付。'}
|
||||
${renderDecisionNextAction(record)}
|
||||
`.trim()
|
||||
|
||||
const renderRoundExecution = (
|
||||
record: HumanRoundRecord,
|
||||
detail: 'focus' | 'chain' | 'loaded'
|
||||
): string => {
|
||||
switch (record.execution.kind) {
|
||||
case 'pre_execution_blocked':
|
||||
return '- 没有进入真实执行,系统在执行前阻断了这轮动作。'
|
||||
case 'learning':
|
||||
return '- 进入 learning,按请求尝试补齐外置技能。'
|
||||
case 'hand':
|
||||
return `
|
||||
- 进入 hand,执行模式是 ${record.execution.toolUsePlan.mode}。
|
||||
- ${
|
||||
record.execution.toolUsePlan.mode === 'parallel'
|
||||
? `实际并行执行了 ${renderQuotedNames(record.execution.toolUsePlan.calls.map((call) => call.toolName))}。`
|
||||
: `实际按顺序执行了 ${renderQuotedNames(record.execution.toolUsePlan.calls.map((call) => call.toolName))}。`
|
||||
}
|
||||
${detail !== 'chain' ? `- 本轮计划包含 ${record.execution.toolUsePlan.calls.length} 个工具调用。` : ''}
|
||||
`
|
||||
.trim()
|
||||
.replace(/\n\n+/g, '\n')
|
||||
case 'mouth':
|
||||
return '- 进入 mouth,组织最终说明。'
|
||||
}
|
||||
}
|
||||
|
||||
const renderRoundResult = (record: HumanRoundRecord): string => {
|
||||
switch (record.execution.kind) {
|
||||
case 'pre_execution_blocked':
|
||||
return renderWatchHintForBrain(record.execution.watchHint)
|
||||
case 'learning':
|
||||
return `
|
||||
- 已加载的技能:${renderQuotedNames(record.execution.learningOutcome.loadedSkillIds)}
|
||||
- 已跳过的重复技能:${renderQuotedNames(record.execution.learningOutcome.skippedDuplicateSkillIds)}
|
||||
- 未识别的技能:${renderQuotedNames(record.execution.learningOutcome.skippedUnknownSkillIds)}
|
||||
- 当前 active tool 数:${String(record.execution.learningOutcome.activeToolCount)}
|
||||
${record.execution.watchHint ? `- 收敛提示:${renderWatchHintForBrain(record.execution.watchHint)}` : ''}
|
||||
`
|
||||
.trim()
|
||||
.replace(/\n\n+/g, '\n')
|
||||
case 'hand': {
|
||||
const toolResultLines = record.execution.toolResults
|
||||
.map((result) =>
|
||||
[
|
||||
`- \`${result.toolName}\`:${result.summary}`,
|
||||
result.dataPreview ? `结果:${result.dataPreview}` : '',
|
||||
result.diagnosticText ? `诊断:${result.diagnosticText}` : ''
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join(';')
|
||||
)
|
||||
.join('\n')
|
||||
const artifactLine =
|
||||
record.execution.artifactRefs && record.execution.artifactRefs.length > 0
|
||||
? `- 产物:${renderInlineList(
|
||||
record.execution.artifactRefs.map((artifact) =>
|
||||
artifact.kind === 'file'
|
||||
? `\`${artifact.fileName}\``
|
||||
: `draft review ${artifact.review.reviewId}`
|
||||
),
|
||||
{
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
}
|
||||
)}`
|
||||
: ''
|
||||
const watchHintLine = record.execution.watchHint
|
||||
? `- 收敛提示:${renderWatchHintForBrain(record.execution.watchHint)}`
|
||||
: ''
|
||||
|
||||
return [toolResultLines, artifactLine, watchHintLine].filter(Boolean).join('\n')
|
||||
}
|
||||
case 'mouth':
|
||||
return `
|
||||
- 输出摘要:${record.execution.summary}
|
||||
${record.execution.completed ? '- 任务已完成。' : '- 任务未完成。'}
|
||||
`.trim()
|
||||
}
|
||||
}
|
||||
|
||||
const renderRoundRecord = (
|
||||
record: HumanRoundRecord,
|
||||
detail: 'focus' | 'chain' | 'loaded'
|
||||
): string =>
|
||||
`
|
||||
- 第 ${record.roundId} 轮
|
||||
|
||||
决策了什么:
|
||||
${renderRoundDecision(record)}
|
||||
|
||||
做了什么:
|
||||
${renderRoundExecution(record, detail)}
|
||||
|
||||
结果是什么:
|
||||
${renderRoundResult(record)}
|
||||
`.trim()
|
||||
|
||||
export const renderRoundSection = (input: {
|
||||
records: HumanRoundRecord[]
|
||||
detail: 'focus' | 'chain' | 'loaded'
|
||||
}): string =>
|
||||
input.records.length > 0
|
||||
? input.records.map((record) => renderRoundRecord(record, input.detail)).join('\n\n')
|
||||
: 'none'
|
||||
|
||||
export const renderLoadedHistoryText = (
|
||||
loadedHistoryContext: HumanLoadedHistoryContext | undefined
|
||||
): string =>
|
||||
loadedHistoryContext
|
||||
? `
|
||||
过滤条件:roundIds=${renderInlineList((loadedHistoryContext.filters.roundIds || []).map(String), {
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
})};executionKinds=${renderInlineList(loadedHistoryContext.filters.executionKinds || [], {
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
})};toolNames=${renderInlineList(loadedHistoryContext.filters.toolNames || [], {
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
})};watchHintCodes=${renderInlineList(loadedHistoryContext.filters.watchHintCodes || [], {
|
||||
separator: '、',
|
||||
fallback: 'none'
|
||||
})};limit=${String(loadedHistoryContext.filters.limit || loadedHistoryContext.rounds.length)}
|
||||
|
||||
${renderRoundSection({
|
||||
records: loadedHistoryContext.rounds,
|
||||
detail: 'loaded'
|
||||
})}
|
||||
`
|
||||
.trim()
|
||||
: 'none'
|
||||
|
||||
export const renderUnifiedHistoryText = (input: {
|
||||
roundHistory: HumanRoundRecord[]
|
||||
loadedHistoryContext: HumanLoadedHistoryContext | undefined
|
||||
}): string =>
|
||||
`
|
||||
过程历史:
|
||||
${renderRoundSection({
|
||||
records: input.roundHistory,
|
||||
detail: 'loaded'
|
||||
})}
|
||||
|
||||
主动回看历史:
|
||||
${renderLoadedHistoryText(input.loadedHistoryContext)}
|
||||
`
|
||||
.trim()
|
||||
.replace(/\n{3,}/g, '\n\n')
|
||||
@@ -0,0 +1,121 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildSelectedDocxFixtureReferences,
|
||||
DOCX_CROSS_FILE_FIXTURE_PATHS,
|
||||
HUMAN_DOCX_CAUSE_REMEDY_READING_USER_INPUT,
|
||||
resolveSelectedDocxFixtures
|
||||
} from '../../../harness/backend/aiLiveTestData'
|
||||
import {
|
||||
collectStartedToolStringArgs,
|
||||
filterAiRunEventsByType,
|
||||
runWithStructuredOutputRetry
|
||||
} from '../testUtils'
|
||||
|
||||
const openedExpectedDoc = (
|
||||
openedSourcePaths: readonly string[],
|
||||
expectedRelativePath: string,
|
||||
expectedName: string
|
||||
): boolean =>
|
||||
openedSourcePaths.some(
|
||||
(sourcePath) =>
|
||||
sourcePath.endsWith(expectedRelativePath) || sourcePath.endsWith(`/${expectedName}`)
|
||||
)
|
||||
|
||||
describe('ai human live backend', () => {
|
||||
it('human-docx-cause-remedy-reading-live', async () => {
|
||||
await runWithStructuredOutputRetry(async () => {
|
||||
await runBackendCase({
|
||||
caseId: 'human-docx-cause-remedy-reading-live',
|
||||
harnessMode: 'live',
|
||||
domain: 'ai',
|
||||
suite: 'human',
|
||||
groups: ['live'],
|
||||
description: '验证 live 模式下 human 会基于两份已选 docx 准确回答过拟合与反走样的成因和处理思路。',
|
||||
fixtures: DOCX_CROSS_FILE_FIXTURE_PATHS,
|
||||
runMode: 'single',
|
||||
runtimeModelName: hasLiveLlmConfigInEnv('primary')
|
||||
? loadLiveLlmConfigFromEnv('primary').model_name
|
||||
: null,
|
||||
execute: async (runtime) => {
|
||||
if (!hasLiveLlmConfigInEnv('primary')) {
|
||||
return {
|
||||
skipped: true,
|
||||
output: null,
|
||||
fixtureDocs: [],
|
||||
events: []
|
||||
}
|
||||
}
|
||||
|
||||
await runtime.seedLlmConfig(loadLiveLlmConfigFromEnv('primary'))
|
||||
const fixtureDocs = resolveSelectedDocxFixtures(runtime)
|
||||
const runner = createHumanTestRunner(runtime, {
|
||||
selectedReferences: buildSelectedDocxFixtureReferences(runtime)
|
||||
})
|
||||
const output = await runner.run({
|
||||
taskId: runtime.taskId,
|
||||
userInput: HUMAN_DOCX_CAUSE_REMEDY_READING_USER_INPUT
|
||||
})
|
||||
|
||||
return {
|
||||
skipped: false,
|
||||
output,
|
||||
fixtureDocs,
|
||||
events: runtime.eventCollector.snapshot()
|
||||
}
|
||||
},
|
||||
assert: async (result, runtime) => {
|
||||
if (result.skipped) {
|
||||
runtime.assertThat(true, {
|
||||
name: 'live-config-not-provided',
|
||||
message: '未提供 live 模型配置,已跳过 human live docx cause/remedy reading case。'
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
const toolResults = filterAiRunEventsByType(result.events, 'tool_call_result')
|
||||
const toolNames = toolResults.map((event) => event.toolName)
|
||||
const openedSourcePaths = collectStartedToolStringArgs(
|
||||
result.events,
|
||||
'docx_open',
|
||||
'sourcePath'
|
||||
)
|
||||
const deliveryText = `${result.output?.summary || ''}\n${result.output?.deliveryMarkdown || ''}`
|
||||
|
||||
runtime.assertThat(
|
||||
result.output?.status === 'completed' &&
|
||||
result.fixtureDocs.every((fixtureDoc) =>
|
||||
openedExpectedDoc(openedSourcePaths, fixtureDoc.relativePath, fixtureDoc.name)
|
||||
) &&
|
||||
toolNames.some(
|
||||
(toolName) =>
|
||||
toolName === 'docx_search' ||
|
||||
toolName === 'docx_outline' ||
|
||||
toolName === 'docx_get_block'
|
||||
) &&
|
||||
!toolNames.includes('docx_apply_ops') &&
|
||||
!toolNames.includes('docx_save') &&
|
||||
/(过拟合|过度拟合)/.test(deliveryText) &&
|
||||
/(记住过多训练数据中的特征值|训练时间过久)/.test(deliveryText) &&
|
||||
/(数据增强|未使用过的数据|验证数据)/.test(deliveryText) &&
|
||||
/反走样/.test(deliveryText) &&
|
||||
/(采样频率低于信号频率|像素.*边界)/.test(deliveryText) &&
|
||||
/(低通滤波|多个采样点|多个像素点|多次采样)/.test(deliveryText),
|
||||
{
|
||||
name: 'human-live-reads-both-selected-docx-and-explains-cause-and-remedy',
|
||||
message: 'live human 读 case 应打开两份已选 docx,并准确回答过拟合与反走样的成因和处理思路。',
|
||||
data: {
|
||||
openedSourcePaths,
|
||||
toolNames,
|
||||
summary: result.output?.summary,
|
||||
deliveryMarkdown: result.output?.deliveryMarkdown
|
||||
}
|
||||
}
|
||||
)
|
||||
}
|
||||
})
|
||||
})
|
||||
}, 600_000)
|
||||
})
|
||||
@@ -0,0 +1,160 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import { describe, it } from 'vitest'
|
||||
import type { AiRunEvent } from '../../../../src/shared/types'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildSelectedDocxFixtureReferences,
|
||||
DOCX_CROSS_FILE_FIXTURE_PATHS,
|
||||
hasUnchangedFileBytes,
|
||||
HUMAN_DOCX_CONTEXTUAL_COMPARISON_DRAFT_USER_INPUT,
|
||||
resolveSelectedDocxFixtures
|
||||
} from '../../../harness/backend/aiLiveTestData'
|
||||
import {
|
||||
collectDocxApplyOpsContentKinds,
|
||||
collectStartedToolStringArgs,
|
||||
filterAiRunEventsByType,
|
||||
runWithStructuredOutputRetry
|
||||
} from '../testUtils'
|
||||
|
||||
const openedExpectedDoc = (
|
||||
openedSourcePaths: readonly string[],
|
||||
expectedRelativePath: string,
|
||||
expectedName: string
|
||||
): boolean =>
|
||||
openedSourcePaths.some(
|
||||
(sourcePath) =>
|
||||
sourcePath.endsWith(expectedRelativePath) || sourcePath.endsWith(`/${expectedName}`)
|
||||
)
|
||||
|
||||
const hasTargetReadBeforeApply = (events: readonly AiRunEvent[]): boolean => {
|
||||
const firstApplyIndex = events.findIndex(
|
||||
(event) => event.type === 'tool_call_started' && event.toolName === 'docx_apply_ops'
|
||||
)
|
||||
if (firstApplyIndex <= 0) {
|
||||
return false
|
||||
}
|
||||
|
||||
return events.slice(0, firstApplyIndex).some((event) => {
|
||||
if (event.type !== 'tool_call_started') {
|
||||
return false
|
||||
}
|
||||
if (event.toolName === 'docx_search') {
|
||||
const query = event.input.query
|
||||
return typeof query === 'string' && query.includes('反走样')
|
||||
}
|
||||
return event.toolName === 'docx_outline' || event.toolName === 'docx_get_block'
|
||||
})
|
||||
}
|
||||
|
||||
describe('ai human live backend', () => {
|
||||
it('human-docx-contextual-comparison-draft-live', async () => {
|
||||
await runWithStructuredOutputRetry(async () => {
|
||||
await runBackendCase({
|
||||
caseId: 'human-docx-contextual-comparison-draft-live',
|
||||
harnessMode: 'live',
|
||||
domain: 'ai',
|
||||
suite: 'human',
|
||||
groups: ['live'],
|
||||
description: '验证 live 模式下 human 会在图形学相关位置后补写说明和对照表,并生成预览。',
|
||||
fixtures: DOCX_CROSS_FILE_FIXTURE_PATHS,
|
||||
runMode: 'single',
|
||||
runtimeModelName: hasLiveLlmConfigInEnv('primary')
|
||||
? loadLiveLlmConfigFromEnv('primary').model_name
|
||||
: null,
|
||||
execute: async (runtime) => {
|
||||
if (!hasLiveLlmConfigInEnv('primary')) {
|
||||
return {
|
||||
skipped: true,
|
||||
output: null,
|
||||
fixtureDocs: [],
|
||||
originalFixtureBytes: [],
|
||||
events: []
|
||||
}
|
||||
}
|
||||
|
||||
await runtime.seedLlmConfig(loadLiveLlmConfigFromEnv('primary'))
|
||||
const fixtureDocs = resolveSelectedDocxFixtures(runtime)
|
||||
const originalFixtureBytes = await Promise.all(
|
||||
fixtureDocs.map(async (fixtureDoc) => ({
|
||||
name: fixtureDoc.name,
|
||||
relativePath: fixtureDoc.relativePath,
|
||||
workspacePath: fixtureDoc.workspacePath,
|
||||
bytes: await fs.readFile(fixtureDoc.workspacePath)
|
||||
}))
|
||||
)
|
||||
const runner = createHumanTestRunner(runtime, {
|
||||
selectedReferences: buildSelectedDocxFixtureReferences(runtime)
|
||||
})
|
||||
const output = await runner.run({
|
||||
taskId: runtime.taskId,
|
||||
userInput: HUMAN_DOCX_CONTEXTUAL_COMPARISON_DRAFT_USER_INPUT
|
||||
})
|
||||
|
||||
return {
|
||||
skipped: false,
|
||||
output,
|
||||
fixtureDocs,
|
||||
originalFixtureBytes,
|
||||
events: runtime.eventCollector.snapshot()
|
||||
}
|
||||
},
|
||||
assert: async (result, runtime) => {
|
||||
if (result.skipped) {
|
||||
runtime.assertThat(true, {
|
||||
name: 'live-config-not-provided',
|
||||
message: '未提供 live 模型配置,已跳过 human live contextual docx comparison draft case。'
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
const events = result.events
|
||||
const toolResults = filterAiRunEventsByType(events, 'tool_call_result')
|
||||
const toolNames = toolResults.map((event) => event.toolName)
|
||||
const openedSourcePaths = collectStartedToolStringArgs(events, 'docx_open', 'sourcePath')
|
||||
const changedContentKinds = collectDocxApplyOpsContentKinds(events)
|
||||
const docxArtifacts =
|
||||
result.output?.artifacts?.filter(
|
||||
(artifact) =>
|
||||
artifact.kind === 'file' &&
|
||||
artifact.capability === 'docx' &&
|
||||
(artifact.artifactRole === 'preview' || artifact.artifactRole === 'draft')
|
||||
) || []
|
||||
const unchangedFixtures = await Promise.all(
|
||||
result.originalFixtureBytes.map(async (fixture) =>
|
||||
await hasUnchangedFileBytes(fixture.workspacePath, fixture.bytes)
|
||||
)
|
||||
)
|
||||
|
||||
runtime.assertThat(
|
||||
result.output?.status === 'completed' &&
|
||||
result.fixtureDocs.every((fixtureDoc) =>
|
||||
openedExpectedDoc(openedSourcePaths, fixtureDoc.relativePath, fixtureDoc.name)
|
||||
) &&
|
||||
hasTargetReadBeforeApply(events) &&
|
||||
toolNames.includes('docx_apply_ops') &&
|
||||
toolNames.includes('docx_render_preview') &&
|
||||
!toolNames.includes('docx_save') &&
|
||||
docxArtifacts.length > 0 &&
|
||||
changedContentKinds.includes('paragraph') &&
|
||||
changedContentKinds.includes('table') &&
|
||||
unchangedFixtures.every(Boolean),
|
||||
{
|
||||
name: 'human-live-contextual-docx-write-produces-preview-without-overwriting-source',
|
||||
message: 'live human 定位补写 case 应在读取定位后补写说明和表格,并仅生成预览稿。',
|
||||
data: {
|
||||
openedSourcePaths,
|
||||
toolNames,
|
||||
changedContentKinds,
|
||||
summary: result.output?.summary,
|
||||
deliveryMarkdown: result.output?.deliveryMarkdown,
|
||||
artifacts: result.output?.artifacts
|
||||
}
|
||||
}
|
||||
)
|
||||
}
|
||||
})
|
||||
})
|
||||
}, 600_000)
|
||||
})
|
||||
@@ -0,0 +1,128 @@
|
||||
import path from 'node:path'
|
||||
import { describe, it } from 'vitest'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
hasUnchangedFileBytes,
|
||||
HUMAN_KB_GUIDED_DOCX_COMPARISON_DRAFT_USER_INPUT,
|
||||
seedKbGuidedDocxKb,
|
||||
seedKbGuidedDocxWorkspace
|
||||
} from '../../../harness/backend/aiLiveTestData'
|
||||
import {
|
||||
collectDocxApplyOpsContentKinds,
|
||||
collectStartedToolStringArgs,
|
||||
filterAiRunEventsByType,
|
||||
hasKbToolCall,
|
||||
runWithStructuredOutputRetry
|
||||
} from '../testUtils'
|
||||
|
||||
describe('ai human live backend', () => {
|
||||
it('human-kb-guided-docx-comparison-draft-live', async () => {
|
||||
await runWithStructuredOutputRetry(async () => {
|
||||
await runBackendCase({
|
||||
caseId: 'human-kb-guided-docx-comparison-draft-live',
|
||||
harnessMode: 'live',
|
||||
domain: 'ai',
|
||||
suite: 'human',
|
||||
groups: ['live'],
|
||||
description: '验证 live 模式下 human 会先用 KB 缩圈定位课件,再继续对目标 docx 做预览型补写。',
|
||||
fixtures: [],
|
||||
runMode: 'single',
|
||||
runtimeModelName: hasLiveLlmConfigInEnv('primary')
|
||||
? loadLiveLlmConfigFromEnv('primary').model_name
|
||||
: null,
|
||||
execute: async (runtime) => {
|
||||
if (!hasLiveLlmConfigInEnv('primary')) {
|
||||
return {
|
||||
skipped: true,
|
||||
output: null,
|
||||
workspaceSeed: null,
|
||||
events: []
|
||||
}
|
||||
}
|
||||
|
||||
await runtime.seedLlmConfig(loadLiveLlmConfigFromEnv('primary'))
|
||||
const workspaceSeed = await seedKbGuidedDocxWorkspace(runtime)
|
||||
await seedKbGuidedDocxKb(runtime, workspaceSeed)
|
||||
|
||||
const runner = createHumanTestRunner(runtime)
|
||||
const output = await runner.run({
|
||||
taskId: runtime.taskId,
|
||||
userInput: HUMAN_KB_GUIDED_DOCX_COMPARISON_DRAFT_USER_INPUT
|
||||
})
|
||||
|
||||
return {
|
||||
skipped: false,
|
||||
output,
|
||||
workspaceSeed,
|
||||
events: runtime.eventCollector.snapshot()
|
||||
}
|
||||
},
|
||||
assert: async (result, runtime) => {
|
||||
if (result.skipped || !result.workspaceSeed) {
|
||||
runtime.assertThat(true, {
|
||||
name: 'live-config-not-provided',
|
||||
message: '未提供 live 模型配置,已跳过 human live KB-guided docx comparison draft case。'
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
const events = result.events
|
||||
const toolResults = filterAiRunEventsByType(events, 'tool_call_result')
|
||||
const toolNames = toolResults.map((event) => event.toolName)
|
||||
const openedSourcePaths = collectStartedToolStringArgs(events, 'docx_open', 'sourcePath')
|
||||
const changedContentKinds = collectDocxApplyOpsContentKinds(events)
|
||||
const docxArtifacts =
|
||||
result.output?.artifacts?.filter(
|
||||
(artifact) =>
|
||||
artifact.kind === 'file' &&
|
||||
artifact.capability === 'docx' &&
|
||||
(artifact.artifactRole === 'preview' || artifact.artifactRole === 'draft')
|
||||
) || []
|
||||
const deepLearningUnchanged = await hasUnchangedFileBytes(
|
||||
result.workspaceSeed.deepLearningPath,
|
||||
result.workspaceSeed.originalDeepLearningBytes
|
||||
)
|
||||
const graphicsUnchanged = await hasUnchangedFileBytes(
|
||||
result.workspaceSeed.graphicsPath,
|
||||
result.workspaceSeed.originalGraphicsBytes
|
||||
)
|
||||
const deliveryText = `${result.output?.summary || ''}\n${result.output?.deliveryMarkdown || ''}`
|
||||
|
||||
runtime.assertThat(
|
||||
result.output?.status === 'completed' &&
|
||||
hasKbToolCall(events) &&
|
||||
toolNames.includes('docx_apply_ops') &&
|
||||
toolNames.includes('docx_render_preview') &&
|
||||
!toolNames.includes('docx_save') &&
|
||||
docxArtifacts.length > 0 &&
|
||||
changedContentKinds.includes('paragraph') &&
|
||||
changedContentKinds.includes('table') &&
|
||||
openedSourcePaths.some(
|
||||
(sourcePath) =>
|
||||
sourcePath.endsWith(result.workspaceSeed.graphicsRelativePath) ||
|
||||
sourcePath.endsWith(`/${path.basename(result.workspaceSeed.graphicsRelativePath)}`)
|
||||
) &&
|
||||
/(过拟合|过度拟合)/.test(deliveryText) &&
|
||||
/反走样/.test(deliveryText) &&
|
||||
deepLearningUnchanged &&
|
||||
graphicsUnchanged,
|
||||
{
|
||||
name: 'human-live-kb-guided-docx-write-uses-kb-and-preserves-source-files',
|
||||
message: 'live human KB-guided 写 case 应先触发 KB 缩圈,再对目标 docx 生成预览稿且不覆盖原文件。',
|
||||
data: {
|
||||
openedSourcePaths,
|
||||
toolNames,
|
||||
changedContentKinds,
|
||||
summary: result.output?.summary,
|
||||
deliveryMarkdown: result.output?.deliveryMarkdown,
|
||||
artifacts: result.output?.artifacts
|
||||
}
|
||||
}
|
||||
)
|
||||
}
|
||||
})
|
||||
})
|
||||
}, 600_000)
|
||||
})
|
||||
@@ -83,8 +83,12 @@ describe('ai human live backend', () => {
|
||||
toolNames.includes('docx_render_preview') &&
|
||||
docxArtifacts.length > 0 &&
|
||||
templateUnchanged &&
|
||||
/(待跟进|合规签署)/.test(deliveryText) &&
|
||||
/Alice/.test(deliveryText),
|
||||
/风险结论/.test(deliveryText) &&
|
||||
/负责人(说明)?/.test(deliveryText) &&
|
||||
/Weekly Risk Summary\.md/.test(deliveryText) &&
|
||||
/Owner Note\.md/.test(deliveryText) &&
|
||||
/(预览|preview)/i.test(deliveryText) &&
|
||||
/(未覆盖原文件|原文件未被覆盖|未覆盖原件)/.test(deliveryText),
|
||||
{
|
||||
name: 'human-live-completes-risk-report-draft-through-real-tool-chain',
|
||||
message: 'live human complex case 应通过 terminal、KB 与 docx 真实工具链完成周报草稿任务。',
|
||||
|
||||
@@ -4,7 +4,7 @@ import type {
|
||||
HumanStableMemoryPatch,
|
||||
RuntimeSkillId
|
||||
} from '../../../src/main/services/ai/human/types'
|
||||
import type { AiRunEvent, JsonObject, JsonValue } from '../../../src/shared/types'
|
||||
import type { AiRunEvent, JsonArray, JsonObject, JsonValue } from '../../../src/shared/types'
|
||||
|
||||
export const filterAiRunEventsByType = <TType extends AiRunEvent['type']>(
|
||||
events: readonly AiRunEvent[],
|
||||
@@ -98,6 +98,83 @@ export const filterToolCallStartedEventsByName = (
|
||||
): Extract<AiRunEvent, { type: 'tool_call_started' }>[] =>
|
||||
filterAiRunEventsByType(events, 'tool_call_started').filter((event) => event.toolName === toolName)
|
||||
|
||||
export const collectStartedToolInputs = (
|
||||
events: readonly AiRunEvent[],
|
||||
toolName: string
|
||||
): JsonObject[] => filterToolCallStartedEventsByName(events, toolName).map((event) => event.input)
|
||||
|
||||
export const collectStartedToolStringArgs = (
|
||||
events: readonly AiRunEvent[],
|
||||
toolName: string,
|
||||
argName: string
|
||||
): string[] =>
|
||||
collectStartedToolInputs(events, toolName)
|
||||
.map((input) => input[argName])
|
||||
.filter((value): value is string => typeof value === 'string')
|
||||
|
||||
export const hasKbToolCall = (events: readonly AiRunEvent[]): boolean =>
|
||||
filterAiRunEventsByType(events, 'tool_call_result').some((event) => event.toolName.startsWith('kb_'))
|
||||
|
||||
const isJsonObject = (value: JsonValue | undefined): value is JsonObject =>
|
||||
typeof value === 'object' && value !== null && !Array.isArray(value)
|
||||
|
||||
const isJsonArray = (value: JsonValue | undefined): value is JsonArray => Array.isArray(value)
|
||||
|
||||
const collectDocxBlockKindsFromValue = (
|
||||
value: JsonValue | undefined,
|
||||
kinds: Set<'paragraph' | 'table'>
|
||||
): void => {
|
||||
if (isJsonArray(value)) {
|
||||
value.forEach((item) => collectDocxBlockKindsFromValue(item, kinds))
|
||||
return
|
||||
}
|
||||
|
||||
if (!isJsonObject(value)) {
|
||||
return
|
||||
}
|
||||
|
||||
const kind = value.kind
|
||||
if (kind === 'paragraph' || kind === 'table') {
|
||||
kinds.add(kind)
|
||||
}
|
||||
|
||||
collectDocxBlockKindsFromValue(value.content, kinds)
|
||||
collectDocxBlockKindsFromValue(value.ops, kinds)
|
||||
collectDocxBlockKindsFromValue(value.rows, kinds)
|
||||
}
|
||||
|
||||
export const collectDocxApplyOpsContentKinds = (
|
||||
events: readonly AiRunEvent[]
|
||||
): Array<'paragraph' | 'table'> => {
|
||||
const kinds = new Set<'paragraph' | 'table'>()
|
||||
collectStartedToolInputs(events, 'docx_apply_ops').forEach((input) => {
|
||||
collectDocxBlockKindsFromValue(input, kinds)
|
||||
})
|
||||
return [...kinds]
|
||||
}
|
||||
|
||||
const isStructuredOutputEmptyResponseError = (error: unknown): boolean =>
|
||||
error instanceof Error && error.message.includes('structured_output_empty_response')
|
||||
|
||||
export const runWithStructuredOutputRetry = async (
|
||||
action: () => Promise<void>,
|
||||
maxAttempts = 2
|
||||
): Promise<void> => {
|
||||
let lastError: unknown = null
|
||||
for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
|
||||
try {
|
||||
await action()
|
||||
return
|
||||
} catch (error) {
|
||||
lastError = error
|
||||
if (!isStructuredOutputEmptyResponseError(error) || attempt >= maxAttempts) {
|
||||
throw error
|
||||
}
|
||||
}
|
||||
}
|
||||
throw lastError
|
||||
}
|
||||
|
||||
export const readToolCallResultPreview = <TValue extends JsonValue>(
|
||||
events: readonly AiRunEvent[],
|
||||
toolName: string,
|
||||
|
||||
@@ -126,18 +126,182 @@ describe('ai tool hand backend', () => {
|
||||
}
|
||||
})
|
||||
|
||||
const safeLimitState = await handNode({
|
||||
...safeState,
|
||||
skills: {
|
||||
learnedSkillIds: ['docx-read']
|
||||
},
|
||||
execution: {
|
||||
...safeState.execution,
|
||||
currentDecision: {
|
||||
decision: {
|
||||
summary: '执行 6 个只读并行调用',
|
||||
reason: '验证新的并行上限允许 6 个安全调用。',
|
||||
expectedOutcome: '六个 read_only 且 parallelSafe 的调用全部真正执行。'
|
||||
},
|
||||
readyToDeliver: false,
|
||||
nextAction: {
|
||||
action: 'hand',
|
||||
toolUsePlan: {
|
||||
mode: 'parallel',
|
||||
calls: [
|
||||
{
|
||||
toolName: 'docx_outline',
|
||||
args: {
|
||||
docId: openResult.docId,
|
||||
maxBlocks: 20
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_search',
|
||||
args: {
|
||||
docId: openResult.docId,
|
||||
query: 'Safety',
|
||||
mode: 'literal'
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_search',
|
||||
args: {
|
||||
docId: openResult.docId,
|
||||
query: 'Risk',
|
||||
mode: 'literal'
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_search',
|
||||
args: {
|
||||
docId: openResult.docId,
|
||||
query: 'monitor',
|
||||
mode: 'literal'
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_search',
|
||||
args: {
|
||||
docId: openResult.docId,
|
||||
query: 'approval',
|
||||
mode: 'literal'
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_search',
|
||||
args: {
|
||||
docId: openResult.docId,
|
||||
query: 'edge',
|
||||
mode: 'literal'
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
const toolStartedBeforeLimitExceeded = filterEventsByType(
|
||||
runtime.eventCollector.snapshot(),
|
||||
'tool_call_started'
|
||||
).length
|
||||
|
||||
const limitExceededState = await handNode({
|
||||
...safeLimitState,
|
||||
skills: {
|
||||
learnedSkillIds: ['docx-read']
|
||||
},
|
||||
execution: {
|
||||
...safeLimitState.execution,
|
||||
currentDecision: {
|
||||
decision: {
|
||||
summary: '执行 7 个只读并行调用',
|
||||
reason: '验证超过并行上限时会直接回到 brain。',
|
||||
expectedOutcome: '写入 parallel_limit_exceeded watchHint,且不继续执行工具。'
|
||||
},
|
||||
readyToDeliver: false,
|
||||
nextAction: {
|
||||
action: 'hand',
|
||||
toolUsePlan: {
|
||||
mode: 'parallel',
|
||||
calls: [
|
||||
{
|
||||
toolName: 'docx_outline',
|
||||
args: {
|
||||
docId: openResult.docId,
|
||||
maxBlocks: 20
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_search',
|
||||
args: {
|
||||
docId: openResult.docId,
|
||||
query: 'Safety',
|
||||
mode: 'literal'
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_search',
|
||||
args: {
|
||||
docId: openResult.docId,
|
||||
query: 'Risk',
|
||||
mode: 'literal'
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_search',
|
||||
args: {
|
||||
docId: openResult.docId,
|
||||
query: 'monitor',
|
||||
mode: 'literal'
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_search',
|
||||
args: {
|
||||
docId: openResult.docId,
|
||||
query: 'approval',
|
||||
mode: 'literal'
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_search',
|
||||
args: {
|
||||
docId: openResult.docId,
|
||||
query: 'edge',
|
||||
mode: 'literal'
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_search',
|
||||
args: {
|
||||
docId: openResult.docId,
|
||||
query: 'cases',
|
||||
mode: 'literal'
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
const toolStartedAfterLimitExceeded = filterEventsByType(
|
||||
runtime.eventCollector.snapshot(),
|
||||
'tool_call_started'
|
||||
).length
|
||||
|
||||
const toolStartedBeforeUnsafe = filterEventsByType(
|
||||
runtime.eventCollector.snapshot(),
|
||||
'tool_call_started'
|
||||
).length
|
||||
|
||||
const unsafeState = await handNode({
|
||||
...safeState,
|
||||
...limitExceededState,
|
||||
skills: {
|
||||
learnedSkillIds: ['docx-read', 'docx-write', 'docx-delivery']
|
||||
},
|
||||
execution: {
|
||||
...safeState.execution,
|
||||
...limitExceededState.execution,
|
||||
currentDecision: {
|
||||
decision: {
|
||||
summary: '并行写入和 validate',
|
||||
@@ -183,7 +347,11 @@ describe('ai tool hand backend', () => {
|
||||
|
||||
return {
|
||||
safeState,
|
||||
safeLimitState,
|
||||
limitExceededState,
|
||||
unsafeState,
|
||||
toolStartedBeforeLimitExceeded,
|
||||
toolStartedAfterLimitExceeded,
|
||||
toolStartedBeforeUnsafe,
|
||||
toolStartedAfterUnsafe,
|
||||
events: runtime.eventCollector.snapshot()
|
||||
@@ -201,6 +369,33 @@ describe('ai tool hand backend', () => {
|
||||
}
|
||||
)
|
||||
|
||||
runtime.assertThat(
|
||||
result.safeLimitState.execution.lastToolResults?.length === 6 &&
|
||||
result.safeLimitState.execution.lastToolResults.every((item) => item.code === 'ok') &&
|
||||
!result.safeLimitState.execution.watchHint,
|
||||
{
|
||||
name: 'six-safe-parallel-calls-are-allowed',
|
||||
message: '6 个只读且 parallelSafe 的工具调用应真正执行并写回结果。',
|
||||
data: result.safeLimitState.execution.lastToolResults
|
||||
}
|
||||
)
|
||||
|
||||
runtime.assertThat(
|
||||
result.limitExceededState.execution.watchHint?.code === 'hand_plan_invalid' &&
|
||||
result.limitExceededState.execution.watchHint.detail.reason === 'parallel_limit_exceeded' &&
|
||||
result.limitExceededState.execution.watchHint.detail.limit === 6 &&
|
||||
result.toolStartedAfterLimitExceeded === result.toolStartedBeforeLimitExceeded,
|
||||
{
|
||||
name: 'seven-parallel-calls-stop-before-execution',
|
||||
message: '超过 6 个并行工具调用时应直接返回 brain 并写入 parallel_limit_exceeded。',
|
||||
data: {
|
||||
watchHint: result.limitExceededState.execution.watchHint,
|
||||
toolStartedBeforeLimitExceeded: result.toolStartedBeforeLimitExceeded,
|
||||
toolStartedAfterLimitExceeded: result.toolStartedAfterLimitExceeded
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
runtime.assertThat(
|
||||
result.unsafeState.execution.watchHint?.code === 'hand_plan_invalid' &&
|
||||
result.unsafeState.execution.watchHint.detail.reason === 'parallel_requires_read_only' &&
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import type { SelectedReference } from '../../../src/shared/types'
|
||||
import { writeTestDocxFile } from './aiToolHarness'
|
||||
import type { BackendTestRuntime, SeedKbDocumentInput } from './test-types'
|
||||
|
||||
@@ -9,6 +10,23 @@ export const HUMAN_FILE_RETENTION_USER_INPUT =
|
||||
export const HUMAN_RISK_REPORT_DRAFT_USER_INPUT =
|
||||
'我下午要把本周风险周报发出去。工作区里应该有一份上周用过的 Word 周报模板,但我忘了文件名。你先帮我找到那份模板,再按我刚才选的资料,把本周风险结论和负责人补进去,给我一个可预览的版本,先别覆盖原文件。最后告诉我你改了哪几处,依据分别来自哪里。'
|
||||
|
||||
export const DOCX_CROSS_FILE_FIXTURE_PATHS = [
|
||||
'tests/fixtures/ai/docx/深度学习.docx',
|
||||
'tests/fixtures/ai/docx/图形学.docx'
|
||||
] as const
|
||||
|
||||
export const HUMAN_DOCX_CAUSE_REMEDY_READING_USER_INPUT =
|
||||
'我做题老把这两个点混在一起。你就按我刚选的这两份资料,帮我说清楚:过拟合和反走样各自是什么、为什么会出现、一般怎么处理。别补充课外知识,哪份文档没写你就直接说没写。'
|
||||
|
||||
export const HUMAN_DOCX_CONTEXTUAL_COMPARISON_DRAFT_USER_INPUT =
|
||||
'《图形学》里“反走样”这一节我还是记不住。你参考我刚选的这两份 Word,直接在这一节最后补一小段更好理解的说明,再加个简单对照表,一列写反走样,一列写《深度学习》里的过拟合。直接给我可预览的版本就行,别改原文件,也不用先回我文字草稿。'
|
||||
|
||||
export const HUMAN_KB_GUIDED_DOCX_COMPARISON_DRAFT_USER_INPUT =
|
||||
'我工作区资料有点多,我忘了是哪份课件在讲反走样了。你先帮我找到那份,再参考讲过拟合的那份资料,直接在“反走样”那一节最后补一段更好记的说明,再加个简单对照表,一列写反走样,一列写过拟合。直接给我可预览的版本就行,别改原文件,也不用先回我文字草稿。'
|
||||
|
||||
export const HUMAN_KB_GUIDED_DOCX_READING_USER_INPUT =
|
||||
'我工作区里应该有两份课件分别讲过过拟合和反走样,但我忘了文件名。你先帮我找出来,再告诉我这两个问题一般怎么处理。'
|
||||
|
||||
export const FILE_RETENTION_DOCUMENTS: readonly SeedKbDocumentInput[] = [
|
||||
{
|
||||
id: 'doc-retention',
|
||||
@@ -51,6 +69,22 @@ export type RiskReportWorkspaceSeed = {
|
||||
originalTemplateBytes: Uint8Array
|
||||
}
|
||||
|
||||
export type SelectedDocxFixture = {
|
||||
sourcePath: string
|
||||
workspacePath: string
|
||||
relativePath: string
|
||||
name: string
|
||||
}
|
||||
|
||||
export type KbGuidedDocxWorkspaceSeed = {
|
||||
deepLearningPath: string
|
||||
deepLearningRelativePath: string
|
||||
graphicsPath: string
|
||||
graphicsRelativePath: string
|
||||
originalDeepLearningBytes: Uint8Array
|
||||
originalGraphicsBytes: Uint8Array
|
||||
}
|
||||
|
||||
export const seedFileRetentionKb = async (runtime: BackendTestRuntime): Promise<string[]> => {
|
||||
await runtime.seedKb(FILE_RETENTION_DOCUMENTS)
|
||||
return FILE_RETENTION_DOCUMENTS.map((document) => document.id)
|
||||
@@ -61,6 +95,236 @@ export const seedRiskReportKb = async (runtime: BackendTestRuntime): Promise<str
|
||||
return RISK_REPORT_DOCUMENTS.map((document) => document.id)
|
||||
}
|
||||
|
||||
export const resolveSelectedDocxFixtures = (
|
||||
runtime: BackendTestRuntime,
|
||||
fixturePaths: readonly string[] = DOCX_CROSS_FILE_FIXTURE_PATHS
|
||||
): SelectedDocxFixture[] =>
|
||||
fixturePaths.map((fixturePath) => {
|
||||
const workspacePath = runtime.getFixtureWorkspacePath(fixturePath)
|
||||
return {
|
||||
sourcePath: fixturePath,
|
||||
workspacePath,
|
||||
relativePath: path.relative(runtime.workspaceRoot, workspacePath),
|
||||
name: path.basename(workspacePath)
|
||||
}
|
||||
})
|
||||
|
||||
export const buildSelectedDocxFixtureReferences = (
|
||||
runtime: BackendTestRuntime,
|
||||
fixturePaths: readonly string[] = DOCX_CROSS_FILE_FIXTURE_PATHS
|
||||
): SelectedReference[] =>
|
||||
resolveSelectedDocxFixtures(runtime, fixturePaths).map((fixture, index) => ({
|
||||
kind: 'file',
|
||||
workspaceId: 'test-workspace',
|
||||
nodeId: `fixture-docx-${String(index + 1)}`,
|
||||
name: fixture.name,
|
||||
relativePath: fixture.relativePath,
|
||||
absolutePath: fixture.workspacePath,
|
||||
fileExt: '.docx',
|
||||
kbDocumentIds: [],
|
||||
kbAvailability: 'not_indexed'
|
||||
}))
|
||||
|
||||
const copyDocxFixtureToWorkspaceRoot = async (input: {
|
||||
runtime: BackendTestRuntime
|
||||
sourcePath: string
|
||||
targetName: string
|
||||
summary: string
|
||||
}): Promise<string> => {
|
||||
const absoluteSourcePath = path.resolve(input.sourcePath)
|
||||
const targetPath = path.join(input.runtime.workspaceRoot, input.targetName)
|
||||
await fs.cp(absoluteSourcePath, targetPath)
|
||||
input.runtime.recordSeedWorkspaceFile({
|
||||
relativePath: input.targetName,
|
||||
absolutePath: targetPath,
|
||||
summary: input.summary,
|
||||
metadata: {
|
||||
format: 'docx'
|
||||
}
|
||||
})
|
||||
return targetPath
|
||||
}
|
||||
|
||||
export const seedKbGuidedDocxWorkspace = async (
|
||||
runtime: BackendTestRuntime
|
||||
): Promise<KbGuidedDocxWorkspaceSeed> => {
|
||||
const deepLearningRelativePath = '深度学习.docx'
|
||||
const graphicsRelativePath = '图形学.docx'
|
||||
const deepLearningPath = await copyDocxFixtureToWorkspaceRoot({
|
||||
runtime,
|
||||
sourcePath: DOCX_CROSS_FILE_FIXTURE_PATHS[0],
|
||||
targetName: deepLearningRelativePath,
|
||||
summary: '目标课件:深度学习'
|
||||
})
|
||||
const graphicsPath = await copyDocxFixtureToWorkspaceRoot({
|
||||
runtime,
|
||||
sourcePath: DOCX_CROSS_FILE_FIXTURE_PATHS[1],
|
||||
targetName: graphicsRelativePath,
|
||||
summary: '目标课件:图形学'
|
||||
})
|
||||
|
||||
const scheduleRelativePath = '课程安排.docx'
|
||||
const schedulePath = await writeTestDocxFile(runtime.workspaceRoot, scheduleRelativePath, [
|
||||
{
|
||||
type: 'heading',
|
||||
text: '课程安排',
|
||||
level: 1
|
||||
},
|
||||
{
|
||||
type: 'paragraph',
|
||||
text: '这里记录的是考试时间、教室和分组安排。'
|
||||
}
|
||||
])
|
||||
runtime.recordSeedWorkspaceFile({
|
||||
relativePath: scheduleRelativePath,
|
||||
absolutePath: schedulePath,
|
||||
summary: '干扰项:课程安排',
|
||||
metadata: {
|
||||
format: 'docx'
|
||||
}
|
||||
})
|
||||
|
||||
const labNoteRelativePath = '机器学习实验记录.docx'
|
||||
const labNotePath = await writeTestDocxFile(runtime.workspaceRoot, labNoteRelativePath, [
|
||||
{
|
||||
type: 'heading',
|
||||
text: '机器学习实验记录',
|
||||
level: 1
|
||||
},
|
||||
{
|
||||
type: 'paragraph',
|
||||
text: '本周实验主要关注学习率和 batch size 的调参过程。'
|
||||
}
|
||||
])
|
||||
runtime.recordSeedWorkspaceFile({
|
||||
relativePath: labNoteRelativePath,
|
||||
absolutePath: labNotePath,
|
||||
summary: '干扰项:机器学习实验记录',
|
||||
metadata: {
|
||||
format: 'docx'
|
||||
}
|
||||
})
|
||||
|
||||
const readingListRelativePath = '参考书目.md'
|
||||
const readingListPath = path.join(runtime.workspaceRoot, readingListRelativePath)
|
||||
const readingListText = '- 线性代数导引\n- 计算机图形学基础\n'
|
||||
await fs.writeFile(readingListPath, readingListText, 'utf8')
|
||||
runtime.recordSeedWorkspaceFile({
|
||||
relativePath: readingListRelativePath,
|
||||
absolutePath: readingListPath,
|
||||
summary: '干扰项:参考书目',
|
||||
metadata: {
|
||||
format: 'text',
|
||||
text: readingListText
|
||||
}
|
||||
})
|
||||
|
||||
const progressRelativePath = '复习进度.txt'
|
||||
const progressPath = path.join(runtime.workspaceRoot, progressRelativePath)
|
||||
const progressText = '今天先看图形学,再看深度学习。\n'
|
||||
await fs.writeFile(progressPath, progressText, 'utf8')
|
||||
runtime.recordSeedWorkspaceFile({
|
||||
relativePath: progressRelativePath,
|
||||
absolutePath: progressPath,
|
||||
summary: '干扰项:复习进度',
|
||||
metadata: {
|
||||
format: 'text',
|
||||
text: progressText
|
||||
}
|
||||
})
|
||||
|
||||
return {
|
||||
deepLearningPath,
|
||||
deepLearningRelativePath,
|
||||
graphicsPath,
|
||||
graphicsRelativePath,
|
||||
originalDeepLearningBytes: await fs.readFile(deepLearningPath),
|
||||
originalGraphicsBytes: await fs.readFile(graphicsPath)
|
||||
}
|
||||
}
|
||||
|
||||
export const seedKbGuidedDocxKb = async (
|
||||
runtime: BackendTestRuntime,
|
||||
workspaceSeed: KbGuidedDocxWorkspaceSeed
|
||||
): Promise<string[]> => {
|
||||
const kbDocuments: readonly SeedKbDocumentInput[] = [
|
||||
{
|
||||
id: 'doc-deep-learning',
|
||||
name: '深度学习.docx',
|
||||
filePath: workspaceSeed.deepLearningPath,
|
||||
chunks: [
|
||||
{
|
||||
id: 'chunk-deep-learning-1',
|
||||
text: '过度拟合是模型记住了过多训练数据中的特征值,使训练后的模型只对这部分训练数据有效,不太具有展望性。'
|
||||
},
|
||||
{
|
||||
id: 'chunk-deep-learning-2',
|
||||
text: '当训练时间过久时,会逐渐记住过多训练数据中的特征值,从而出现过度拟合。'
|
||||
},
|
||||
{
|
||||
id: 'chunk-deep-learning-3',
|
||||
text: '缓解过度拟合的思路包括:使用训练时未使用过的数据做验证,以及通过数据增强把同一图像拆成不同亮度、旋转方向等多张图像来训练。'
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
id: 'doc-graphics',
|
||||
name: '图形学.docx',
|
||||
filePath: workspaceSeed.graphicsPath,
|
||||
chunks: [
|
||||
{
|
||||
id: 'chunk-graphics-1',
|
||||
text: '反走样出现的原因是采样频率低于信号频率。'
|
||||
},
|
||||
{
|
||||
id: 'chunk-graphics-2',
|
||||
text: '由于一个像素可能同时落在图形边界内和边界外,边缘会出现锯齿。'
|
||||
},
|
||||
{
|
||||
id: 'chunk-graphics-3',
|
||||
text: '反走样的处理思路包括:在一个像素内模拟多个采样点,或者先把图形转换成信号后做低通滤波。'
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
id: 'doc-course-schedule',
|
||||
name: '课程安排.docx',
|
||||
filePath: path.join(runtime.workspaceRoot, '课程安排.docx'),
|
||||
chunks: [
|
||||
{
|
||||
id: 'chunk-course-schedule-1',
|
||||
text: '课程安排只记录考试时间、教室和分组信息。'
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
id: 'doc-ml-lab',
|
||||
name: '机器学习实验记录.docx',
|
||||
filePath: path.join(runtime.workspaceRoot, '机器学习实验记录.docx'),
|
||||
chunks: [
|
||||
{
|
||||
id: 'chunk-ml-lab-1',
|
||||
text: '实验记录主要关注学习率和 batch size 的调参过程。'
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
id: 'doc-reading-list',
|
||||
name: '参考书目.md',
|
||||
filePath: path.join(runtime.workspaceRoot, '参考书目.md'),
|
||||
chunks: [
|
||||
{
|
||||
id: 'chunk-reading-list-1',
|
||||
text: '参考书目里只有课程参考资料列表,没有过拟合或反走样的讲解。'
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
await runtime.seedKb(kbDocuments)
|
||||
return kbDocuments.map((document) => document.id)
|
||||
}
|
||||
|
||||
export const seedRiskReportWorkspace = async (
|
||||
runtime: BackendTestRuntime
|
||||
): Promise<RiskReportWorkspaceSeed> => {
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
import { describe, it } from 'vitest'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
HUMAN_KB_GUIDED_DOCX_READING_USER_INPUT,
|
||||
seedKbGuidedDocxKb,
|
||||
seedKbGuidedDocxWorkspace
|
||||
} from '../../../../harness/backend/aiLiveTestData'
|
||||
import {
|
||||
collectStartedToolStringArgs,
|
||||
filterAiRunEventsByType,
|
||||
hasKbToolCall,
|
||||
runWithStructuredOutputRetry
|
||||
} from '../../../../ai/human/testUtils'
|
||||
|
||||
describe('temp ai human live backend', () => {
|
||||
it('human-kb-guided-docx-reading-live', async () => {
|
||||
await runWithStructuredOutputRetry(async () => {
|
||||
await runBackendCase({
|
||||
caseId: 'human-kb-guided-docx-reading-live',
|
||||
harnessMode: 'live',
|
||||
domain: 'temp',
|
||||
suite: 'ai',
|
||||
groups: ['human', 'live'],
|
||||
description: '观察 live 模式下 human 在工作区资料较多时,会如何用 KB 缩圈并回答过拟合与反走样的处理思路。',
|
||||
fixtures: [],
|
||||
runMode: 'single',
|
||||
runtimeModelName: hasLiveLlmConfigInEnv('primary')
|
||||
? loadLiveLlmConfigFromEnv('primary').model_name
|
||||
: null,
|
||||
execute: async (runtime) => {
|
||||
if (!hasLiveLlmConfigInEnv('primary')) {
|
||||
return {
|
||||
skipped: true,
|
||||
output: null,
|
||||
events: []
|
||||
}
|
||||
}
|
||||
|
||||
await runtime.seedLlmConfig(loadLiveLlmConfigFromEnv('primary'))
|
||||
const workspaceSeed = await seedKbGuidedDocxWorkspace(runtime)
|
||||
await seedKbGuidedDocxKb(runtime, workspaceSeed)
|
||||
|
||||
const runner = createHumanTestRunner(runtime)
|
||||
const output = await runner.run({
|
||||
taskId: runtime.taskId,
|
||||
userInput: HUMAN_KB_GUIDED_DOCX_READING_USER_INPUT
|
||||
})
|
||||
|
||||
return {
|
||||
skipped: false,
|
||||
output,
|
||||
events: runtime.eventCollector.snapshot()
|
||||
}
|
||||
},
|
||||
assert: async (result, runtime) => {
|
||||
if (result.skipped) {
|
||||
runtime.assertThat(true, {
|
||||
name: 'live-config-not-provided',
|
||||
message: '未提供 live 模型配置,已跳过 temp human live KB-guided docx reading case。'
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
const toolResults = filterAiRunEventsByType(result.events, 'tool_call_result')
|
||||
const deliveryText = `${result.output?.summary || ''}\n${result.output?.deliveryMarkdown || ''}`
|
||||
const openedSourcePaths = collectStartedToolStringArgs(
|
||||
result.events,
|
||||
'docx_open',
|
||||
'sourcePath'
|
||||
)
|
||||
|
||||
runtime.assertThat(
|
||||
result.output?.status === 'completed' &&
|
||||
hasKbToolCall(result.events) &&
|
||||
/(数据增强|未使用过的数据|验证数据)/.test(deliveryText) &&
|
||||
/(低通滤波|多个采样点|多个像素点|多次采样)/.test(deliveryText),
|
||||
{
|
||||
name: 'temp-human-live-kb-guided-reading-covers-kb-and-answer-facts',
|
||||
message: 'temp KB-guided 读 case 应至少触发 KB,并回答过拟合与反走样的处理思路。',
|
||||
data: {
|
||||
openedSourcePaths,
|
||||
toolNames: toolResults.map((event) => event.toolName),
|
||||
summary: result.output?.summary,
|
||||
deliveryMarkdown: result.output?.deliveryMarkdown
|
||||
}
|
||||
}
|
||||
)
|
||||
}
|
||||
})
|
||||
})
|
||||
}, 600_000)
|
||||
})
|
||||
@@ -0,0 +1,229 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import type { AiRunContext } from '../../../src/main/services/ai/types'
|
||||
import {
|
||||
appendRoundRecord,
|
||||
buildHandExecution,
|
||||
buildLearningExecution
|
||||
} from '../../../src/main/services/ai/human/memory/roundHistory'
|
||||
import { buildBrainPrompt } from '../../../src/main/services/ai/human/prompts/brain'
|
||||
import { buildMouthPrompt } from '../../../src/main/services/ai/human/prompts/mouth'
|
||||
import {
|
||||
renderRoundSection,
|
||||
renderUnifiedHistoryText
|
||||
} from '../../../src/main/services/ai/human/shared/historyRenderers'
|
||||
import { buildReferenceScope } from '../../../src/main/services/ai/prompts/referenceScope'
|
||||
import { createInitialHumanState } from '../../../src/main/services/ai/human/state/schema'
|
||||
import type { HumanDecision } from '../../../src/main/services/ai/human/types'
|
||||
|
||||
const createStubAiRunContext = (referenceScope: AiRunContext['referenceScope']): AiRunContext => ({
|
||||
runId: 'run-history-render',
|
||||
referenceScope,
|
||||
checkCancelled: () => {},
|
||||
chatStructured: async () => {
|
||||
throw new Error('chatStructured should not be called in prompt unit tests')
|
||||
},
|
||||
chatText: async () => {
|
||||
throw new Error('chatText should not be called in prompt unit tests')
|
||||
},
|
||||
sendEvent: () => {},
|
||||
requestApproval: async () => ({
|
||||
approved: false
|
||||
}),
|
||||
loadHistory: async () => ({
|
||||
recentTurnsText: '',
|
||||
sessionSummaryText: ''
|
||||
}),
|
||||
trackDocxOpen: () => {},
|
||||
trackDocxChanges: () => {},
|
||||
resolveDocxOpenTarget: async () => {
|
||||
throw new Error('resolveDocxOpenTarget should not be called in prompt unit tests')
|
||||
},
|
||||
documentsStore: undefined as never,
|
||||
embeddingService: undefined as never
|
||||
})
|
||||
|
||||
const buildDecision = (input: {
|
||||
summary: string
|
||||
reason: string
|
||||
expectedOutcome: string
|
||||
nextAction: HumanDecision['nextAction']
|
||||
}): HumanDecision => ({
|
||||
decision: {
|
||||
summary: input.summary,
|
||||
reason: input.reason,
|
||||
expectedOutcome: input.expectedOutcome
|
||||
},
|
||||
readyToDeliver: false,
|
||||
nextAction: input.nextAction
|
||||
})
|
||||
|
||||
describe('ai unit human history renderers', () => {
|
||||
it('renders shared history facts for both brain and mouth prompts', () => {
|
||||
const initialState = createInitialHumanState({
|
||||
taskId: 'task-history-shared',
|
||||
userInput: '请总结这次任务做了什么'
|
||||
})
|
||||
|
||||
const learningState = {
|
||||
...initialState,
|
||||
skills: {
|
||||
learnedSkillIds: ['workspace-cli']
|
||||
},
|
||||
memory: {
|
||||
...initialState.memory,
|
||||
history: appendRoundRecord({
|
||||
state: initialState,
|
||||
decision: buildDecision({
|
||||
summary: '补齐 workspace 能力',
|
||||
reason: '需要 terminal 工具',
|
||||
expectedOutcome: '拿到 workspace 读取能力',
|
||||
nextAction: {
|
||||
action: 'learning',
|
||||
wantLearnSkillIds: ['workspace-cli']
|
||||
}
|
||||
}),
|
||||
execution: buildLearningExecution({
|
||||
learningOutcome: {
|
||||
requestedSkillIds: ['workspace-cli'],
|
||||
loadedSkillIds: ['workspace-cli'],
|
||||
skippedDuplicateSkillIds: [],
|
||||
skippedUnknownSkillIds: [],
|
||||
evictedSkillIds: [],
|
||||
activeToolCount: 1
|
||||
}
|
||||
}),
|
||||
createdAt: 1
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
const handState = {
|
||||
...learningState,
|
||||
memory: {
|
||||
...learningState.memory,
|
||||
history: appendRoundRecord({
|
||||
state: learningState,
|
||||
decision: buildDecision({
|
||||
summary: '读取资料并生成预览',
|
||||
reason: '已经具备需要的能力',
|
||||
expectedOutcome: '拿到 docx 预览产物',
|
||||
nextAction: {
|
||||
action: 'hand',
|
||||
toolUsePlan: {
|
||||
mode: 'serial',
|
||||
calls: [
|
||||
{
|
||||
toolName: 'kb_hybrid_search_chunks',
|
||||
args: {
|
||||
queryText: 'approval denied'
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_render_preview',
|
||||
args: {
|
||||
format: 'html'
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}),
|
||||
execution: buildHandExecution({
|
||||
toolUsePlan: {
|
||||
mode: 'serial',
|
||||
calls: [
|
||||
{
|
||||
toolName: 'kb_hybrid_search_chunks',
|
||||
args: {
|
||||
queryText: 'approval denied'
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_render_preview',
|
||||
args: {
|
||||
format: 'html'
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
toolResults: [
|
||||
{
|
||||
toolName: 'kb_hybrid_search_chunks',
|
||||
ok: true,
|
||||
code: 'ok',
|
||||
data: {
|
||||
items: [
|
||||
{
|
||||
document_name: 'Weekly Risk Summary.md',
|
||||
text: '审批未通过时原文件保留'
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
toolName: 'docx_render_preview',
|
||||
ok: true,
|
||||
code: 'ok',
|
||||
data: {
|
||||
previewPath: '/tmp/preview.docx',
|
||||
summary: 'preview generated (html)'
|
||||
}
|
||||
}
|
||||
]
|
||||
}),
|
||||
createdAt: 2
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
const state = {
|
||||
...handState,
|
||||
memory: {
|
||||
...handState.memory,
|
||||
history: {
|
||||
...handState.memory.history,
|
||||
loadedHistoryContext: {
|
||||
filters: {
|
||||
executionKinds: ['learning'],
|
||||
limit: 1
|
||||
},
|
||||
rounds: [handState.memory.history.roundHistory[0]]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const focusHistory = renderRoundSection({
|
||||
records: [state.memory.history.roundHistory[0]],
|
||||
detail: 'focus'
|
||||
})
|
||||
const unifiedHistory = renderUnifiedHistoryText({
|
||||
roundHistory: state.memory.history.roundHistory,
|
||||
loadedHistoryContext: state.memory.history.loadedHistoryContext
|
||||
})
|
||||
|
||||
const brainPrompt = buildBrainPrompt({
|
||||
state,
|
||||
ctx: createStubAiRunContext(
|
||||
buildReferenceScope({
|
||||
workspace: {
|
||||
workspaceId: 'workspace-1',
|
||||
workspaceName: 'workspace',
|
||||
rootPath: '/tmp/workspace-root'
|
||||
}
|
||||
})
|
||||
)
|
||||
})
|
||||
const mouthPrompt = buildMouthPrompt({ state })
|
||||
|
||||
expect(focusHistory).toContain('决策了什么:')
|
||||
expect(focusHistory).toContain('做了什么:')
|
||||
expect(focusHistory).toContain('结果是什么:')
|
||||
expect(unifiedHistory).toContain('过程历史:')
|
||||
expect(unifiedHistory).toContain('主动回看历史:')
|
||||
expect(unifiedHistory).toContain('Weekly Risk Summary.md')
|
||||
expect(unifiedHistory).toContain('preview generated (html)')
|
||||
expect(brainPrompt.user).toContain(focusHistory)
|
||||
expect(mouthPrompt.user).toContain(unifiedHistory)
|
||||
})
|
||||
})
|
||||
@@ -71,16 +71,16 @@ describe('ai unit mouth readyToDeliver guard', () => {
|
||||
})
|
||||
}).system
|
||||
|
||||
expect(prompt).toContain('<source_of_truth>')
|
||||
expect(prompt).toContain('<context_block_guide>')
|
||||
expect(prompt).toContain('<available_information>')
|
||||
expect(prompt).toContain('<output_contract>')
|
||||
expect(prompt).toContain('final_decision 是 brain 已经确认的最终判断')
|
||||
expect(prompt).toContain('runtime_artifacts 只允许准确引用已经给出的路径')
|
||||
expect(prompt).toContain('你不会调用工具,也不会继续推进未完成步骤')
|
||||
expect(prompt).toContain('不要把未来可能执行的工具动作写成“后续会自动完成”')
|
||||
expect(prompt).toContain('你必须只输出最终 markdown 正文')
|
||||
expect(prompt).not.toContain('"deliveryMarkdown"')
|
||||
expect(prompt).not.toContain('"summary"')
|
||||
expect(prompt).toContain('直接回答 task_goal')
|
||||
expect(prompt).toContain('your_notes 优先')
|
||||
expect(prompt).toContain('history:任务过程中真实发生过的决策、执行和结果')
|
||||
expect(prompt).toContain('只输出最终 markdown 正文')
|
||||
expect(prompt).toContain('标题、分节、列表都按需要使用,不固定')
|
||||
expect(prompt).toContain('不原样照抄内部标签、block id、tool 名')
|
||||
expect(prompt).not.toContain('# 处理结果')
|
||||
expect(prompt).not.toContain('## 关键依据')
|
||||
})
|
||||
|
||||
it('keeps mouth user-context blocks data-only and moves explanations into system', () => {
|
||||
@@ -91,14 +91,21 @@ describe('ai unit mouth readyToDeliver guard', () => {
|
||||
})
|
||||
})
|
||||
|
||||
expect(result.system).toContain('final_decision:brain 已经确认的最终判断')
|
||||
expect(result.system).toContain('默认使用最终用户答复的组织方式')
|
||||
expect(result.system).toContain('task_goal:用户最初要你完成什么')
|
||||
expect(result.system).toContain('已确认:当前稳定成立、可以直接告诉用户的事实')
|
||||
expect(result.system).toContain('关键依据:支撑这些事实的证据')
|
||||
expect(result.system).toContain('未决问题:仍会影响结论')
|
||||
expect(result.user).not.toContain('这是当前交付模式。')
|
||||
expect(result.user).not.toContain('这是用户这次原始诉求与限制。')
|
||||
expect(result.user).not.toContain('这是 brain 已经做出的最终决定。')
|
||||
expect(result.user).not.toContain('这是任务的稳定笔记')
|
||||
expect(result.user).not.toContain('这是最近一轮最相关的过程记录。')
|
||||
expect(result.user).not.toContain('这是更早几轮的过程链。')
|
||||
expect(result.user).not.toContain('这是运行中已经产生的真实产物列表。')
|
||||
expect(result.user).not.toContain('这是当前额外的收敛或纠偏提醒。')
|
||||
expect(result.user).toContain('<history>')
|
||||
expect(result.user).not.toContain('<correction_hint>')
|
||||
expect(result.user).not.toContain('<recent_progress>')
|
||||
expect(result.user).not.toContain('<earlier_attempts>')
|
||||
expect(result.user).not.toContain('<recalled_history>')
|
||||
})
|
||||
})
|
||||
Reference in new issue
Block a user