Add cross-source summary live case and retune KB prompt
This commit is contained in:
1 parent
1cb4911acc
commit
fc04dbdadf
5 files changed
+189
-6
No files matched your search
@@ -752,6 +752,16 @@
|
||||
- Oracle:任务最终完成;事件流中至少两次 `docx_search` / `docx_read_blocks` / `docx_outline` 覆盖两份目标文档;至少出现一次 docx 编辑工具;存在 `draft_review_ready` 事件,最终输出 artifacts 中同时包含 `draft_review` 与 `draft`;原始两份 `docx` 文件未被覆盖;写入请求只应落在 `人工智能探究课堂PRD.docx` 上,并包含新增说明与表格结构。
|
||||
- Not:不要求固定 op 组合,也不对最终说明文案做逐字断言。
|
||||
|
||||
##### `human-docx-cross-source-summary-draft-live`
|
||||
|
||||
- 测试类型:`backend`
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 live 模式下 `human` 会读取两份已选源资料,并把跨源总结内容回写到一份独立的 docx 草稿上。
|
||||
- Why:真实用户常见诉求不只是“参考另一份文档补主稿”,还包括“把两份资料压成一份复习总结”;这条 case 要长期守住跨源读取、单目标总结成稿与不覆盖原文件的正式能力,对齐 `WB-CORE-002-AC2`、`WB-CORE-003-AC1`、`WB-CORE-004-AC1`、`WB-CORE-006-AC1`、`WB-CORE-006-AC2`。
|
||||
- How:注入真实 live LLM 配置;把 `深度学习.docx`、`图形学.docx` 和 `sample-edit.docx` 作为已选资料复制进 workspace;直接输入“我想把刚选的这两份课件整理成一份发给同学复习的总结。我也放了一份 Word 草稿在旁边,你把关键概念、出现原因和常见处理方式压成一版清楚好记的总结,直接整理进那份草稿里。”的自然语言诉求,让真实 `human` 自主完成。
|
||||
- Oracle:任务最终完成;事件流中至少两次 `docx_search` / `docx_read_blocks` / `docx_outline` 覆盖两份源资料;至少出现一次 docx 编辑工具;存在 `draft_review_ready` 事件,最终输出 artifacts 中同时包含 `draft_review` 与 `draft`;三份原始 `docx` 文件未被覆盖;写入请求只应落在 `sample-edit.docx` 上;最终答复应包含“过拟合/过度拟合”与“反走样”两个主题。
|
||||
- Not:不要求固定总结结构,也不要求一定生成表格。
|
||||
|
||||
##### `human-docx-positioned-structural-edit-fallback-live`
|
||||
|
||||
- 测试类型:`backend`
|
||||
@@ -768,7 +778,7 @@
|
||||
- 实现状态:`已实现`
|
||||
- What:验证 live 模式下 `human` 在不传 `selectedReferences` 且 workspace 存在整组 docx 时,会先用 KB 缩圈找出讲“反走样 / 过拟合”的两份源资料,再把内容分别回写到另外两份目标 docx。
|
||||
- Why:真实用户常常记得主题、不记得文件名,而且希望把资料内容整理回自己的教学文档;这条 case 要长期守住“工作区默认范围 + KB 缩圈 + 多源读取 + 多目标 docx 回写”的正式能力,对齐 `WB-CORE-002-AC2`、`WB-CORE-004-AC1`、`WB-CORE-006-AC1`、`WB-CORE-006-AC2`。
|
||||
- How:注入真实 live LLM 配置;workspace 直接使用 `tests/fixtures/ai/docx` 目录下整组 docx,不传 `selectedReferences`;同时将 `图形学.docx`、`深度学习.docx`、`人工智能探究课堂PRD.docx` 与 `交互式探究学习案例:产业转移.docx` seed 到真实 KB;直接输入“我工作区里这几份课件和教学材料有点多,你别先靠文件名猜,直接在知识库里帮我找哪两份在讲反走样和过拟合。找到以后先看一遍,再把里面最适合老师讲‘概念区分’的内容分别补到课堂 PRD 和交互式案例里,写法尽量像备课备注和课堂引导语,不要像教材摘抄。”的自然语言诉求,让真实 `human` 自主完成。
|
||||
- How:注入真实 live LLM 配置;workspace 直接使用 `tests/fixtures/ai/docx` 目录下整组 docx,不传 `selectedReferences`;同时将 `图形学.docx`、`深度学习.docx`、`人工智能探究课堂PRD.docx` 与 `交互式探究学习案例:产业转移.docx` seed 到真实 KB;直接输入“我工作区里这几份课件和教学材料有点多,我想把里面跟反走样和过拟合有关的内容整理进两份教学文档里。你先把相关资料找出来看一遍,再把适合老师讲‘概念区分’的内容分别补到课堂 PRD 和交互式案例里,写法尽量像备课备注和课堂引导语,不要像教材摘抄。”的自然语言诉求,让真实 `human` 自主完成。
|
||||
- Oracle:任务最终完成;事件流中至少出现一次 KB 工具调用;后续 docx 读取覆盖 `图形学.docx` 与 `深度学习.docx` 两份源资料;至少两次允许的 docx 编辑成功,且写入目标落在 `人工智能探究课堂PRD.docx` 与 `交互式探究学习案例:产业转移.docx` 两份目标稿上;存在至少两次 `draft_review_ready` 事件,最终输出 artifacts 中同时包含两组 `draft_review` 与 `draft`;四份原始 `docx` 文件未被覆盖;最终答复明确说明找到了哪两份源资料,并分别更新了哪两份教学文档。
|
||||
- Not:不强制固定执行路径,也不要求 KB 缩圈后必须再打开所有候选资料。
|
||||
|
||||
|
||||
@@ -8,8 +8,8 @@ export const buildMouthPrompt = (input: { state: HumanState }) => {
|
||||
const system = `
|
||||
<system>
|
||||
<role>
|
||||
你现在负责根据已有事实回答用户这次任务。
|
||||
你的目标不是复述内部过程,而是直接完成 task_goal 里用户真正要的回复。
|
||||
你现在负责吸收当前任务在完整上下文下已经成立的信息,并把它整理成面对用户的最终回答。
|
||||
你的目标不是复述内部过程,也不是替内部节点续做推理,而是把用户诉求、当前事实、已有结果与可引用产物自然组织成回复。
|
||||
</role>
|
||||
|
||||
<available_information>
|
||||
@@ -24,7 +24,8 @@ export const buildMouthPrompt = (input: { state: HumanState }) => {
|
||||
|
||||
<output_contract>
|
||||
- 只输出最终 markdown 正文,不要输出 JSON 或额外说明。
|
||||
- 直接回答 task_goal,优先覆盖用户显式提出的问题。
|
||||
- 面向用户回答 task_goal,优先覆盖用户显式提出的问题。
|
||||
- 回答应基于当前整体状态自然组织,不能脱离已经成立的信息。
|
||||
- 当存在工作区或显式参考资料时,优先基于这些本地信息组织答案。
|
||||
- 优先依据 your_notes 组织答案或交付内容。
|
||||
- 如果 your_notes 过于概括、与更具体的证据记录存在张力,或不足以支撑用户要求的细节,可结合 history 与 runtime_artifacts 补足。
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import { describe, it } from 'vitest'
|
||||
import { loadLiveLlmConfigFromEnv, hasLiveLlmConfigInEnv } from '../../../harness/backend/llmConfig'
|
||||
import { createHumanTestRunner } from '../../../harness/backend/humanHarness'
|
||||
import { runBackendCase } from '../../../harness/backend/run-backend-case'
|
||||
import {
|
||||
buildSelectedDocxFixtureReferences,
|
||||
DOCX_CROSS_SOURCE_SUMMARY_FIXTURE_PATHS,
|
||||
HUMAN_DOCX_CROSS_SOURCE_SUMMARY_DRAFT_USER_INPUT,
|
||||
resolveSelectedDocxFixtures
|
||||
} from '../../../harness/backend/aiLiveTestData'
|
||||
import {
|
||||
collectDocxApplyOpsContentKinds,
|
||||
filterAiRunEventsByType,
|
||||
hasDraftFileArtifact,
|
||||
hasDraftReviewArtifact,
|
||||
runWithStructuredOutputRetry
|
||||
} from '../testUtils'
|
||||
|
||||
const openedExpectedDoc = (
|
||||
openedSourcePaths: readonly string[],
|
||||
expectedRelativePath: string,
|
||||
expectedName: string
|
||||
): boolean =>
|
||||
openedSourcePaths.some(
|
||||
(sourcePath) =>
|
||||
sourcePath.endsWith(expectedRelativePath) || sourcePath.endsWith(`/${expectedName}`)
|
||||
)
|
||||
|
||||
describe('ai human live backend', () => {
|
||||
it('human-docx-cross-source-summary-draft-live', async () => {
|
||||
await runWithStructuredOutputRetry(async () => {
|
||||
await runBackendCase({
|
||||
caseId: 'human-docx-cross-source-summary-draft-live',
|
||||
harnessMode: 'live',
|
||||
domain: 'ai',
|
||||
suite: 'human',
|
||||
groups: ['live'],
|
||||
description: '验证 live 模式下 human 会读取两份已选源资料,并把总结内容回写到一份总结稿 docx 上。',
|
||||
fixtures: DOCX_CROSS_SOURCE_SUMMARY_FIXTURE_PATHS,
|
||||
runMode: 'single',
|
||||
runtimeModelName: hasLiveLlmConfigInEnv('primary')
|
||||
? loadLiveLlmConfigFromEnv('primary').model_name
|
||||
: null,
|
||||
execute: async (runtime) => {
|
||||
if (!hasLiveLlmConfigInEnv('primary')) {
|
||||
return {
|
||||
skipped: true,
|
||||
output: null,
|
||||
fixtureDocs: [],
|
||||
originalFixtureBytes: [],
|
||||
events: []
|
||||
}
|
||||
}
|
||||
|
||||
await runtime.seedLlmConfig(loadLiveLlmConfigFromEnv('primary'))
|
||||
const fixtureDocs = resolveSelectedDocxFixtures(runtime, DOCX_CROSS_SOURCE_SUMMARY_FIXTURE_PATHS)
|
||||
const originalFixtureBytes = await Promise.all(
|
||||
fixtureDocs.map(async (fixtureDoc) => ({
|
||||
name: fixtureDoc.name,
|
||||
relativePath: fixtureDoc.relativePath,
|
||||
workspacePath: fixtureDoc.workspacePath,
|
||||
bytes: await fs.readFile(fixtureDoc.workspacePath)
|
||||
}))
|
||||
)
|
||||
const runner = createHumanTestRunner(runtime, {
|
||||
selectedReferences: buildSelectedDocxFixtureReferences(
|
||||
runtime,
|
||||
DOCX_CROSS_SOURCE_SUMMARY_FIXTURE_PATHS
|
||||
)
|
||||
})
|
||||
const output = await runner.run({
|
||||
taskId: runtime.taskId,
|
||||
userInput: HUMAN_DOCX_CROSS_SOURCE_SUMMARY_DRAFT_USER_INPUT
|
||||
})
|
||||
|
||||
return {
|
||||
skipped: false,
|
||||
output,
|
||||
fixtureDocs,
|
||||
originalFixtureBytes,
|
||||
events: runtime.eventCollector.snapshot()
|
||||
}
|
||||
},
|
||||
assert: async (result, runtime) => {
|
||||
if (result.skipped) {
|
||||
runtime.assertThat(true, {
|
||||
name: 'live-config-not-provided',
|
||||
message: '未提供 live 模型配置,已跳过 human live cross-source summary draft case。'
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
const events = result.events
|
||||
const toolResults = filterAiRunEventsByType(events, 'tool_call_result')
|
||||
const toolNames = toolResults.map((event) => event.toolName)
|
||||
const openedSourcePaths = filterAiRunEventsByType(events, 'tool_call_started')
|
||||
.filter((event) => event.toolName.startsWith('docx_'))
|
||||
.map((event) => event.input.sourcePath)
|
||||
.filter((value): value is string => typeof value === 'string')
|
||||
const writeSourcePaths = filterAiRunEventsByType(events, 'tool_call_started')
|
||||
.filter((event) =>
|
||||
event.toolName === 'docx_paragraph_edit' ||
|
||||
event.toolName === 'docx_heading_edit' ||
|
||||
event.toolName === 'docx_table_edit' ||
|
||||
event.toolName === 'docx_text_edit' ||
|
||||
event.toolName === 'docx_text_style_edit'
|
||||
)
|
||||
.map((event) => event.input.sourcePath)
|
||||
.filter((value): value is string => typeof value === 'string')
|
||||
const changedContentKinds = collectDocxApplyOpsContentKinds(events)
|
||||
const unchangedFixtures = await Promise.all(
|
||||
result.originalFixtureBytes.map(async (fixture) => {
|
||||
const currentBytes = await fs.readFile(fixture.workspacePath)
|
||||
return Buffer.compare(currentBytes, Buffer.from(fixture.bytes)) === 0
|
||||
})
|
||||
)
|
||||
const sourceFixtures = result.fixtureDocs.filter((fixture) => fixture.name !== 'sample-edit.docx')
|
||||
const targetFixture = result.fixtureDocs.find((fixture) => fixture.name === 'sample-edit.docx')
|
||||
const deliveryText = `${result.output?.summary || ''}\n${result.output?.deliveryMarkdown || ''}`
|
||||
|
||||
runtime.assertThat(
|
||||
result.output?.status === 'completed' &&
|
||||
sourceFixtures.every((fixtureDoc) =>
|
||||
openedExpectedDoc(openedSourcePaths, fixtureDoc.relativePath, fixtureDoc.name)
|
||||
) &&
|
||||
Boolean(targetFixture) &&
|
||||
toolNames.some((toolName) => toolName.endsWith('_edit')) &&
|
||||
writeSourcePaths.length > 0 &&
|
||||
writeSourcePaths.every(
|
||||
(sourcePath) =>
|
||||
sourcePath.endsWith(targetFixture!.relativePath) ||
|
||||
sourcePath.endsWith(`/${targetFixture!.name}`)
|
||||
) &&
|
||||
changedContentKinds.includes('paragraph') &&
|
||||
filterAiRunEventsByType(events, 'draft_review_ready').length >= 1 &&
|
||||
hasDraftReviewArtifact(result.output?.artifacts) &&
|
||||
hasDraftFileArtifact(result.output?.artifacts) &&
|
||||
unchangedFixtures.every(Boolean) &&
|
||||
/(过拟合|过度拟合)/.test(deliveryText) &&
|
||||
/反走样/.test(deliveryText),
|
||||
{
|
||||
name: 'human-live-cross-source-summary-writes-only-target-draft-doc',
|
||||
message:
|
||||
'live human cross-source summary case 应读取两份源资料,并只把总结内容落到 sample-edit 草稿上。',
|
||||
data: {
|
||||
openedSourcePaths,
|
||||
writeSourcePaths,
|
||||
toolNames,
|
||||
changedContentKinds,
|
||||
summary: result.output?.summary,
|
||||
deliveryMarkdown: result.output?.deliveryMarkdown,
|
||||
artifacts: result.output?.artifacts
|
||||
}
|
||||
}
|
||||
)
|
||||
}
|
||||
})
|
||||
})
|
||||
}, 600_000)
|
||||
})
|
||||
@@ -19,6 +19,11 @@ export const DOCX_MULTI_EDIT_FIXTURE_PATHS = [
|
||||
'tests/fixtures/ai/docx/交互式探究学习案例:产业转移.docx'
|
||||
] as const
|
||||
|
||||
export const DOCX_CROSS_SOURCE_SUMMARY_FIXTURE_PATHS = [
|
||||
...DOCX_READING_FIXTURE_PATHS,
|
||||
'tests/fixtures/ai/docx/sample-edit.docx'
|
||||
] as const
|
||||
|
||||
export const DOCX_ALL_FIXTURE_PATHS = [
|
||||
...DOCX_READING_FIXTURE_PATHS,
|
||||
...DOCX_MULTI_EDIT_FIXTURE_PATHS,
|
||||
@@ -31,11 +36,14 @@ export const HUMAN_DOCX_CAUSE_REMEDY_READING_USER_INPUT =
|
||||
export const HUMAN_DOCX_MULTI_DOC_SUPPORTED_EDIT_USER_INPUT =
|
||||
'我在整理这版课堂方案,想把主稿里那段交互设计说明写得更落地一点。你参考我刚选的那份案例文档,把对应章节补得更清楚一些,再顺手加一个很短的对照表,方便老师一眼看懂重点。'
|
||||
|
||||
export const HUMAN_DOCX_CROSS_SOURCE_SUMMARY_DRAFT_USER_INPUT =
|
||||
'我想把刚选的这两份课件整理成一份发给同学复习的总结。我也放了一份 Word 草稿在旁边,你把关键概念、出现原因和常见处理方式压成一版清楚好记的总结,直接整理进那份草稿里。'
|
||||
|
||||
export const HUMAN_DOCX_POSITIONED_STRUCTURAL_EDIT_FALLBACK_USER_INPUT =
|
||||
'我准备把这份图形学讲义发给同学复习,想把“反走样”这一节补得更容易记一些。你直接在这一节后面加一段更通俗的说明,再加一个很短的总结表,帮助大家快速区分两种常见做法。'
|
||||
|
||||
export const HUMAN_KB_GUIDED_DOCX_COMPARISON_DRAFT_USER_INPUT =
|
||||
'我工作区里这几份课件和教学材料有点多,你别先靠文件名猜,直接在知识库里帮我找哪两份在讲反走样和过拟合。找到以后先看一遍,再把里面最适合老师讲“概念区分”的内容分别补到课堂 PRD 和交互式案例里,写法尽量像备课备注和课堂引导语,不要像教材摘抄。'
|
||||
'我工作区里这几份课件和教学材料有点多,我想把里面跟反走样和过拟合有关的内容整理进两份教学文档里。你先把相关资料找出来看一遍,再把适合老师讲“概念区分”的内容分别补到课堂 PRD 和交互式案例里,写法尽量像备课备注和课堂引导语,不要像教材摘抄。'
|
||||
|
||||
export const HUMAN_KB_GUIDED_DOCX_READING_USER_INPUT =
|
||||
'我工作区里应该有两份课件分别讲过过拟合和反走样,但我忘了文件名。你先帮我找出来,再告诉我这两个问题一般怎么处理。'
|
||||
|
||||
@@ -77,7 +77,10 @@ describe('ai unit mouth readyToDeliver guard', () => {
|
||||
|
||||
expect(prompt).toContain('<available_information>')
|
||||
expect(prompt).toContain('<output_contract>')
|
||||
expect(prompt).toContain('直接回答 task_goal')
|
||||
expect(prompt).toContain('吸收当前任务在完整上下文下已经成立的信息')
|
||||
expect(prompt).toContain('把用户诉求、当前事实、已有结果与可引用产物自然组织成回复')
|
||||
expect(prompt).toContain('面向用户回答 task_goal')
|
||||
expect(prompt).toContain('回答应基于当前整体状态自然组织')
|
||||
expect(prompt).toContain('优先依据 your_notes 组织答案或交付内容')
|
||||
expect(prompt).toContain('与更具体的证据记录存在张力')
|
||||
expect(prompt).toContain('只输出最终 markdown 正文')
|
||||
|
||||
Reference in new issue
Block a user