feat: add ask_user_question tool for user interactions
- Introduced ASK_USER_QUESTION_TOOL_SPEC and related schemas for user questions. - Implemented agentControlToolFamily to include the new ask_user_question tool. - Enhanced AiRunStore to handle pending interactions and updated database schema. - Created HumanInteractionCard component for rendering user questions in the UI. - Updated ChatPage to manage human interaction states and submit answers. - Added tests for ask_user_question tool contract to ensure validation and constraints.
This commit is contained in:
1 parent
3add2f5ae7
commit
5dfbec24ad
37 files changed
+663
-82
No files matched your search
@@ -43,7 +43,7 @@ graph 从 `context_budget` 进入;首次进入才写本轮 `user`。
|
||||
| `dispatch` | 否 | 否 | 消费 `tool_calls` 队列,路由下一 stage |
|
||||
| `learning` | 否 | `learning_*` + `tool` + **`skill_state_updated`** | 加载 skill 工具;**不**改 build user |
|
||||
| `plan` | 否 | `tool` + `plan_updated` | 更新 state.plan;**不**改 build user |
|
||||
| `hand` | 否 | `tool`(+ 可选 `images[]`)+ 审批后 `approval_batch_resolved` | append ToolMessage;`read_image` 后 append 多模态 HumanMessage |
|
||||
| `hand` / `human_interaction` | 否 | `tool`(+ 可选 `images[]`)+ `human_interaction_resolved` annotation | append ToolMessage;`read_image` 后 append 多模态 HumanMessage;问题答案作为结构化 tool result 回到 brain |
|
||||
|
||||
交付收尾由 brain 内的 `finalizeDelivery` 完成(**不是**独立图节点):docx/ppt 落盘 side effect → `finalizing_started` → 写交付 `assistant`(`toolCalls: []`)→ 立即 `answer_completed` / `run_completed`。下一轮同伙伴发送可立刻创建。
|
||||
无 `tool_calls` 的正常交付路径上,**brain 不向 log 写 `assistant`**(流式 Answer 只走事件 / `partial_answer`)。
|
||||
@@ -128,7 +128,7 @@ append-only 伙伴消息转录;模型上下文、UI、resume 均由此投影
|
||||
| `plan_updated` | `{ plan: [{ title, status }] }` | UI 计划条;新 run 从图启动扫描该伙伴剩余 log 最新一条 |
|
||||
| `compaction_checkpoint` | `{ compactBlockId, throughSeq, sourceMessageCount, summary }` | 压缩书签 + 摘要正文;`seq <= historyFloorSeq` 的原文不再进 brain;发给主模型时再合成 Compact User |
|
||||
| `finalizing_started` | `{}` | 投影时**清空** plan |
|
||||
| `approval_batch_resolved` | `{ batchId, items: [{ toolCallId, toolName, command, commandExplanation?, action, message? }] }` | UI Activity:走过审批的工具行图标 / 悬停揭示文案;`hand` 在审批决策后写入 |
|
||||
| `human_interaction_resolved` | `{ interactionId, kind, result }` | UI Activity:在对应工具行下按顺序展示问题答案或审批结果 |
|
||||
|
||||
#### `entry_json` 示例
|
||||
|
||||
@@ -277,11 +277,11 @@ append-only 伙伴消息转录;模型上下文、UI、resume 均由此投影
|
||||
|
||||
| 字段(要点) | 用途 |
|
||||
|--------------|------|
|
||||
| `status` | `running` / `waiting_approval` / `completed` / `failed` / `cancelled` |
|
||||
| `status` | `running` / `waiting_human` / `completed` / `failed` / `cancelled` |
|
||||
| `input` | 本轮用户输入原文 |
|
||||
| `partial_answer` / `partial_answer_role` | 流式 UI 草稿;**不参与** resume 与下次 brain |
|
||||
| `final_answer` / `error` / `failure_code` / `failure_raw_message` | 终态展示(短标题 + 原文详情) |
|
||||
| `pending_approval` | 等待批量审批 |
|
||||
| `pending_interaction` | 等待用户问题回答或批量审批;`full_access` 只自动通过审批,不自动回答问题 |
|
||||
| `artifacts` | run 级产物引用(Workbench 展示) |
|
||||
| `selectedReferences` / `attachments` | run 创建时 Composer 快照(与 log `user.context` 对应) |
|
||||
|
||||
|
||||
@@ -135,6 +135,7 @@ Terminal:`terminal_read` 输出上限与 inline 字符预算挂钩(见 termi
|
||||
| `learn_skill` | 由 `learning` 节点直接写 ToolMessage(loaded / duplicate / unknown / evicted 摘要),不走 `buildModelToolResultDelivery` |
|
||||
| `update_plan` | 由 `plan` 节点直接写 ToolMessage(计划项列表 `[status] title`),不走 `buildModelToolResultDelivery` |
|
||||
| `read_image` | 路径列表 + 尺寸;`images[]` 进 log 供多模态(human-harness 第 3.5 节) |
|
||||
| `ask_user_question` | 只询问会改变执行结果的信息;每批 1 至 3 个问题,选项具体互斥,用户始终可选择“其他,请填写”;问题调用不得与副作用工具混用。答案通过 tool result 回到 brain。 |
|
||||
|
||||
### 6.2 写工具输出
|
||||
|
||||
|
||||
@@ -75,11 +75,10 @@ prepared → ready → published
|
||||
git diff <previous-platform-tag>..<current-platform-tag>
|
||||
```
|
||||
|
||||
评估版本实际改动涉及的产品能力、数据升级、平台构建、安装更新和跨模块风险,然后从正式包 E2E 回归用例中确定范围:
|
||||
评估版本实际改动涉及的产品能力、数据升级、平台构建、安装更新和跨模块风险,然后从正式包 E2E 回归用例中确定回归测试范围:
|
||||
|
||||
- 影响边界可以可靠收敛时,只执行受影响能力对应的用例。
|
||||
- 影响范围较广,或无法可靠判断是否遗漏时,执行全部发版回归用例。
|
||||
- 修改了 SQLite schema 时,至少包含 `sidebar-app-update`、`database-version-update-required`,并使用上一公开 schema 的可审查 fixture 验证原地升级。
|
||||
- 评估结论必须记录版本、影响范围、实际执行的用例、结果和剩余风险。
|
||||
- 同一次发布中,已通过且后续修复未波及的用例不要重跑;每次跑正式包 E2E 回归时,应仔细评估改动影响范围来跑对应用例,禁止偷懒直接全量重跑。
|
||||
|
||||
|
||||
@@ -473,13 +473,14 @@
|
||||
|
||||
### 8.5 AI 运行中的阻塞与提示
|
||||
|
||||
#### WB-BLOCKER-001 审批阻塞
|
||||
#### WB-BLOCKER-001 人机交互阻塞
|
||||
|
||||
- 系统行为:
|
||||
- AI 提问与审批共用 `waiting_human` 和 composer 上方交互卡片。提问支持单选、多选及始终可用的“其他,请填写”;一批最多 3 个问题,回答在 tool result 中回到 AI。
|
||||
- 审核权限下,AI 运行收到批量审批事件后,在输入框上方展示紧贴 composer 的审批卡片;卡片一次只展示当前一条审批项,用户可通过上下切换浏览整批,并且必须先完成整批审核后系统才会继续决策是否执行。
|
||||
- 审批卡片不展示内部策略原因,命令文本与同意/拒绝图标按钮左右排列,指导输入框与提交按钮同排展示;审批不自动超时,必须等待用户明确操作或运行被取消/关闭。
|
||||
- 等待审批期间,当前 assistant turn 的 Activity header 显示「等待用户审核中」;审核结束后 header 恢复为常规工具汇总短句。
|
||||
- 审完后,本批走过审批的工具行在 Activity 展开列表文案前仅加结果图标(同意绿勾、拒绝红叉、指导琥珀色对话气泡),不额外包灰底卡片;悬停图标时原工具文案隐藏并滑入揭示文案:同意/拒绝显示 `commandExplanation`(缺省回退 `command`),指导显示 `{commandExplanation} · 已反馈:{指导语}`。
|
||||
- Activity 展开后,在对应工具行下按时间顺序显示问题、选项文字、自定义答案和审批结果;不再使用审核图标前缀或 hover 替换工具文案。
|
||||
- 不在过程流展示只读 blockers 提示,也不弹出「有 n 条命令需要审核 / 审核已通过」toast。决策结果写入 `chat_session_message_log` 的 `approval_batch_resolved` annotation,刷新后可恢复。
|
||||
- 完全访问权限下,统一审批入口直接批准该 run 的所有审批项,不创建等待审批状态、不发送审批事件、不展示审批卡片与审批叙事。完全访问只跳过审批,不绕过命令安全限制、取消、工作地址内文件变更记录或其他运行时保护。
|
||||
- 验收:
|
||||
|
||||
@@ -571,6 +571,12 @@ DOCX tool 回归以 7 个公开工具、story 内容树、稳定 `nodeId`、segm
|
||||
|
||||
#### human hand 类别
|
||||
|
||||
##### `human-ask-user-question`
|
||||
|
||||
- 测试类型:`backend / mock-llm`
|
||||
- What:提问工具校验 1 至 3 个问题,支持单选、多选和自定义答案,并在回答前阻止副作用工具。
|
||||
- Oracle:run 进入 `waiting_human`;回答前没有写操作;结构化 `{ answers: [...] }` tool result 回到下一轮 brain;同一 run 不能有两个 pending interaction;`full_access` 不会自动回答。
|
||||
|
||||
##### `hand-invalid-tool-input`
|
||||
|
||||
- 测试类型:`backend`
|
||||
|
||||
@@ -600,6 +600,12 @@ pnpm test:update:e2e
|
||||
- How:真实 AI 修改一个 Markdown 文件并打开审核模式;折叠面板后创建第二个伙伴,再切回原伙伴,等待异步记录刷新窗口;确认面板保持折叠,再手动展开并检查审核内容与文件 Tab。
|
||||
- Oracle:切回后 shell 不存在且仅保留右缘展开按钮,对齐 `WB-PANEL-001-AC5`;手动展开后展示目标文件的最新审核内容且文件 Tab 保留,对齐 `WB-PANEL-001-AC1`。
|
||||
|
||||
#### `human-interaction-question-card`
|
||||
|
||||
- 测试类型:`e2e / live`
|
||||
- What:模糊任务触发 AI 提问,用户完成 A/B/C/D、其他输入和多问题切换;回答前不显示写入结果,提交后 run 恢复。
|
||||
- Oracle:composer 上方出现统一交互卡片;刷新后 pending 问题可恢复;Activity 对应工具行下显示问题与答案,审批结果以文字显示且没有旧图标前缀或 hover 替换。
|
||||
|
||||
### 5.4 `scheduled-task`
|
||||
|
||||
定时任务测试单独归类,Oracle 引用 [`workbench.md`](../prd/workbench.md) 的 `ST-*` 编号。调度触发时点由真实程序运行与系统时间决定,case 通过「下一分钟」级别的短周期调度验证端到端触发,不伪造系统时钟。下列 AC 由 unit `tests/case/unit/scheduled-task-service.test.ts` 承担,不进入 Workbench E2E 发现范围:`ST-PARTNER-001-AC2`、`ST-RUN-001-AC3`、`ST-RUN-001-AC4`、`ST-RUN-003-AC1/AC2`、`ST-RUN-004-AC2/AC3`、`ST-RUN-005-AC1/AC2/AC3`。`ST-PARTNER-001-AC1`(删除专属伙伴后任务仍在)标为不测。
|
||||
|
||||
@@ -39,7 +39,9 @@ import type {
|
||||
ScheduledTaskRunNowRequest,
|
||||
ScheduledTaskDismissBusyNoticeRequest,
|
||||
WriteTextFileRequest,
|
||||
WriteTextFileResponse
|
||||
WriteTextFileResponse,
|
||||
AiRunHumanInteractionRequest,
|
||||
AiRunHumanInteractionResponse
|
||||
} from '../../shared/types'
|
||||
import { listReferenceItemsPage } from '../services/sourceScope/referenceList'
|
||||
import { WorkspaceSourceService } from '../services/workspace/WorkspaceSourceService'
|
||||
@@ -2932,6 +2934,12 @@ export function registerHandlers(services: {
|
||||
}
|
||||
)
|
||||
|
||||
ipcMain.handle(
|
||||
'ai-run-human-interaction-submit',
|
||||
async (_event, payload: AiRunHumanInteractionRequest): Promise<AiRunHumanInteractionResponse> =>
|
||||
await chatRunService.submitHumanInteraction(payload)
|
||||
)
|
||||
|
||||
ipcMain.handle(
|
||||
'docx-apply-get',
|
||||
async (_event, payload: GetDocxApplyRecordRequest): Promise<GetDocxApplyRecordResponse> => {
|
||||
@@ -3032,7 +3040,7 @@ export function registerHandlers(services: {
|
||||
const [partners, activePartnerIds, unreadCompletedPartnerIds, latestUserInputs] =
|
||||
await Promise.all([
|
||||
partnerStore.listPartners(),
|
||||
aiRunStore.listPartnerIdsByRunStatuses(['running', 'waiting_approval']),
|
||||
aiRunStore.listPartnerIdsByRunStatuses(['running', 'waiting_human', 'waiting_approval']),
|
||||
partnerStore.listUnreadCompletedPartnerIds(),
|
||||
aiRunStore.listLatestUserInputs()
|
||||
])
|
||||
|
||||
@@ -46,7 +46,13 @@ import type {
|
||||
ChatTurnRetractLatestRequest,
|
||||
ChatTurnRetractLatestResponse,
|
||||
ChatSubmitRequest,
|
||||
ChatSubmitResponse
|
||||
ChatSubmitResponse,
|
||||
AskUserQuestionResult,
|
||||
AiRunHumanInteractionRequest,
|
||||
AiRunHumanInteractionResponse,
|
||||
PendingHumanInteraction,
|
||||
UserQuestion,
|
||||
JsonObject
|
||||
} from '../../../shared/types'
|
||||
import { projectAiConversationMessages } from '../../../shared/workbench/conversationProjection'
|
||||
|
||||
@@ -67,6 +73,14 @@ type ActiveApproval = {
|
||||
) => void
|
||||
}
|
||||
|
||||
type ActiveQuestion = {
|
||||
interactionId: string
|
||||
taskId: string
|
||||
toolCallId: string
|
||||
questions: UserQuestion[]
|
||||
resolve: (result: { accepted: true; result: AskUserQuestionResult } | { accepted: false; error: string }) => void
|
||||
}
|
||||
|
||||
type ActiveRun = {
|
||||
run: AiRunRecord
|
||||
cancelled: boolean
|
||||
@@ -106,7 +120,7 @@ type ChatRunServiceDeps = {
|
||||
) => void | Promise<void>
|
||||
}
|
||||
|
||||
const ACTIVE_RUN_STATUSES: AiRunRecord['status'][] = ['running', 'waiting_approval']
|
||||
const ACTIVE_RUN_STATUSES: AiRunRecord['status'][] = ['running', 'waiting_human', 'waiting_approval']
|
||||
const INTERRUPTED_RUN_MESSAGE = '运行已中断,请重新发送消息。'
|
||||
const PARTIAL_SAVE_DELAY_MS = 500
|
||||
|
||||
@@ -129,6 +143,7 @@ const appendArtifact = (artifacts: AiArtifactRef[] | undefined, artifact: AiArti
|
||||
export class ChatRunService {
|
||||
private readonly activeRuns = new Map<string, ActiveRun>()
|
||||
private readonly activeApprovals = new Map<string, ActiveApproval>()
|
||||
private readonly activeQuestions = new Map<string, ActiveQuestion>()
|
||||
private readonly pendingPartnerSubmissions = new Set<string>()
|
||||
|
||||
public constructor(private readonly deps: ChatRunServiceDeps) {}
|
||||
@@ -175,7 +190,7 @@ export class ChatRunService {
|
||||
}
|
||||
|
||||
public async failInterruptedRuns(): Promise<void> {
|
||||
const interrupted = await this.deps.aiRunStore.listRunsByStatuses(ACTIVE_RUN_STATUSES)
|
||||
const interrupted = await this.deps.aiRunStore.listRunsByStatuses(['running'])
|
||||
for (const run of interrupted) {
|
||||
const now = Date.now()
|
||||
const nextRun: AiRunRecord = {
|
||||
@@ -186,6 +201,7 @@ export class ChatRunService {
|
||||
partialAnswer: run.partialAnswer || INTERRUPTED_RUN_MESSAGE,
|
||||
finalAnswer: run.finalAnswer || INTERRUPTED_RUN_MESSAGE,
|
||||
pendingApproval: undefined,
|
||||
pendingInteraction: undefined,
|
||||
updatedAt: now,
|
||||
finishedAt: now
|
||||
}
|
||||
@@ -398,6 +414,51 @@ export class ChatRunService {
|
||||
const approval = this.activeApprovals.get(normalizedRunId)
|
||||
approval?.resolve({ accepted: false, error: '已取消' })
|
||||
this.activeApprovals.delete(normalizedRunId)
|
||||
const question = this.activeQuestions.get(normalizedRunId)
|
||||
question?.resolve({ accepted: false, error: '已取消' })
|
||||
this.activeQuestions.delete(normalizedRunId)
|
||||
return { success: true }
|
||||
}
|
||||
|
||||
public async submitHumanInteraction(
|
||||
payload: AiRunHumanInteractionRequest
|
||||
): Promise<AiRunHumanInteractionResponse> {
|
||||
const runId = String(payload?.runId || '').trim()
|
||||
const pending = this.activeQuestions.get(runId)
|
||||
if (!pending) return { success: false, error: '当前运行没有待回答问题' }
|
||||
if (pending.interactionId !== payload.interactionId) {
|
||||
return { success: false, error: 'interaction not found' }
|
||||
}
|
||||
const answers = Array.isArray(payload.answers?.answers) ? payload.answers.answers : []
|
||||
if (answers.length !== pending.questions.length) return { success: false, error: '每个问题都必须回答' }
|
||||
const questionIds = new Set(pending.questions.map((question) => question.id))
|
||||
const seenAnswerIds = new Set<string>()
|
||||
for (const answer of answers) {
|
||||
if (!answer || !questionIds.has(answer.id) || seenAnswerIds.has(answer.id) || !Array.isArray(answer.selected)) {
|
||||
return { success: false, error: '答案格式无效' }
|
||||
}
|
||||
seenAnswerIds.add(answer.id)
|
||||
const question = pending.questions.find((item) => item.id === answer.id)
|
||||
const optionIds = new Set(question?.options.map((option) => option.id) || [])
|
||||
if (answer.selected.some((id) => !optionIds.has(id))) return { success: false, error: '存在无效选项' }
|
||||
if (question && !question.multiSelect && answer.selected.length > 1) {
|
||||
return { success: false, error: '单选问题只能选择一个选项' }
|
||||
}
|
||||
if (answer.selected.length === 0 && !String(answer.custom || '').trim()) {
|
||||
return { success: false, error: '请选择选项或填写其他内容' }
|
||||
}
|
||||
}
|
||||
await this.queueRunEvent(runId, {
|
||||
type: 'human_interaction_resolved',
|
||||
runId,
|
||||
taskId: pending.taskId,
|
||||
interactionId: pending.interactionId,
|
||||
kind: 'question',
|
||||
result: payload.answers as unknown as JsonObject,
|
||||
resolvedAt: Date.now()
|
||||
})
|
||||
this.activeQuestions.delete(runId)
|
||||
pending.resolve({ accepted: true, result: payload.answers })
|
||||
return { success: true }
|
||||
}
|
||||
|
||||
@@ -460,7 +521,8 @@ export class ChatRunService {
|
||||
...entry.run,
|
||||
...(persisted.run || {}),
|
||||
status: 'running',
|
||||
pendingApproval: undefined
|
||||
pendingApproval: undefined,
|
||||
pendingInteraction: undefined
|
||||
}
|
||||
}
|
||||
|
||||
@@ -479,7 +541,8 @@ export class ChatRunService {
|
||||
return {
|
||||
run,
|
||||
messageLogRecords: records,
|
||||
pendingApproval: run.pendingApproval
|
||||
pendingApproval: run.pendingApproval,
|
||||
pendingInteraction: run.pendingInteraction
|
||||
}
|
||||
}
|
||||
|
||||
@@ -651,6 +714,7 @@ export class ChatRunService {
|
||||
sendEvent,
|
||||
recordLlmIo: this.deps.recordLlmIo,
|
||||
requestApprovalBatch: async (request) => await this.requestApprovalBatch(runId, request),
|
||||
requestUserQuestion: async (request) => await this.requestUserQuestion(runId, request),
|
||||
browserUseService: this.deps.browserUseService,
|
||||
webCrawlerService: this.deps.webCrawlerService,
|
||||
humanRuntimeOverrides: this.deps.resolveHumanRuntimeOverrides?.(run)
|
||||
@@ -727,6 +791,9 @@ export class ChatRunService {
|
||||
const approval = this.activeApprovals.get(runId)
|
||||
approval?.resolve({ accepted: false, error: 'run closed' })
|
||||
this.activeApprovals.delete(runId)
|
||||
const question = this.activeQuestions.get(runId)
|
||||
question?.resolve({ accepted: false, error: 'run closed' })
|
||||
this.activeQuestions.delete(runId)
|
||||
await this.flushRunEvents(runId)
|
||||
this.clearProjectionFlush(runId)
|
||||
this.activeRuns.delete(runId)
|
||||
@@ -757,8 +824,8 @@ export class ChatRunService {
|
||||
decisions: request.items.map((item) => ({ itemId: item.itemId, action: 'approve' }))
|
||||
}
|
||||
}
|
||||
if (this.activeApprovals.has(runId)) {
|
||||
return { accepted: false, error: 'approval batch already pending' }
|
||||
if (this.activeApprovals.has(runId) || this.activeQuestions.has(runId)) {
|
||||
return { accepted: false, error: '已有待处理的人机交互' }
|
||||
}
|
||||
|
||||
await this.queueRunEvent(runId, {
|
||||
@@ -804,6 +871,56 @@ export class ChatRunService {
|
||||
})
|
||||
}
|
||||
|
||||
private async requestUserQuestion(
|
||||
runId: string,
|
||||
request: {
|
||||
taskId: string
|
||||
interactionId: string
|
||||
toolCallId: string
|
||||
questions: UserQuestion[]
|
||||
stage?: AiRunEvent['stage']
|
||||
}
|
||||
): Promise<{ accepted: true; result: AskUserQuestionResult } | { accepted: false; error: string }> {
|
||||
if (this.activeQuestions.has(runId) || this.activeApprovals.has(runId)) {
|
||||
return { accepted: false, error: '已有待处理的人机交互' }
|
||||
}
|
||||
const interaction: PendingHumanInteraction = {
|
||||
kind: 'question',
|
||||
interactionId: request.interactionId,
|
||||
toolCallId: request.toolCallId,
|
||||
questions: request.questions
|
||||
}
|
||||
const resultPromise = new Promise<
|
||||
{ accepted: true; result: AskUserQuestionResult } | { accepted: false; error: string }
|
||||
>((resolve) => {
|
||||
this.activeQuestions.set(runId, {
|
||||
interactionId: request.interactionId,
|
||||
taskId: request.taskId,
|
||||
toolCallId: request.toolCallId,
|
||||
questions: request.questions,
|
||||
resolve: (result) => {
|
||||
this.activeQuestions.delete(runId)
|
||||
resolve(result)
|
||||
}
|
||||
})
|
||||
})
|
||||
try {
|
||||
await this.queueRunEvent(runId, {
|
||||
type: 'human_interaction_required',
|
||||
runId,
|
||||
taskId: request.taskId,
|
||||
interactionId: request.interactionId,
|
||||
interaction,
|
||||
createdAt: Date.now(),
|
||||
stage: request.stage
|
||||
})
|
||||
} catch (error) {
|
||||
const pending = this.activeQuestions.get(runId)
|
||||
pending?.resolve({ accepted: false, error: error instanceof Error ? error.message : String(error) })
|
||||
}
|
||||
return await resultPromise
|
||||
}
|
||||
|
||||
private queueRunEvent(runId: string, event: AiRunEvent): Promise<void> {
|
||||
const entry = this.activeRuns.get(runId)
|
||||
if (entry?.deleted) return Promise.resolve()
|
||||
@@ -903,20 +1020,37 @@ export class ChatRunService {
|
||||
} else if (event.type === 'approval_batch_required') {
|
||||
run.status = 'waiting_approval'
|
||||
run.pendingApproval = event
|
||||
run.pendingInteraction = {
|
||||
kind: 'approval',
|
||||
interactionId: event.batchId,
|
||||
batchId: event.batchId,
|
||||
items: event.items
|
||||
}
|
||||
run.updatedAt = now
|
||||
} else if (event.type === 'approval_batch_decision') {
|
||||
run.status = 'running'
|
||||
run.pendingApproval = undefined
|
||||
run.pendingInteraction = undefined
|
||||
run.updatedAt = now
|
||||
} else if (event.type === 'human_interaction_required') {
|
||||
run.status = 'waiting_human'
|
||||
run.pendingInteraction = event.interaction
|
||||
run.updatedAt = now
|
||||
} else if (event.type === 'human_interaction_resolved') {
|
||||
run.status = 'running'
|
||||
run.pendingInteraction = undefined
|
||||
run.updatedAt = now
|
||||
} else if (event.type === 'run_completed') {
|
||||
run.status = run.status === 'failed' || run.status === 'cancelled' ? run.status : 'completed'
|
||||
run.pendingApproval = undefined
|
||||
run.pendingInteraction = undefined
|
||||
run.updatedAt = now
|
||||
run.finishedAt = now
|
||||
} else if (event.type === 'run_failed') {
|
||||
run.status = 'failed'
|
||||
run.failureCode = event.code
|
||||
run.pendingApproval = undefined
|
||||
run.pendingInteraction = undefined
|
||||
run.error = event.message
|
||||
run.failureRawMessage = event.rawMessage
|
||||
run.updatedAt = now
|
||||
@@ -930,6 +1064,7 @@ export class ChatRunService {
|
||||
run.status = 'cancelled'
|
||||
run.failureCode = 'user_cancelled'
|
||||
run.pendingApproval = undefined
|
||||
run.pendingInteraction = undefined
|
||||
run.error = '已取消'
|
||||
run.updatedAt = now
|
||||
run.finishedAt = now
|
||||
|
||||
@@ -8,6 +8,7 @@ export const HUMAN_STAGES = [
|
||||
'learning',
|
||||
'plan',
|
||||
'hand',
|
||||
'interaction',
|
||||
'context_budget',
|
||||
'context_compaction'
|
||||
] as const
|
||||
|
||||
@@ -9,6 +9,7 @@ import { createContextBudgetNode } from '../node/contextBudgetNode'
|
||||
import { createLearningNode } from '../node/learningNode'
|
||||
import { createPlanNode } from '../node/planNode'
|
||||
import { createHandNode } from '../node/handNode'
|
||||
import { createHumanInteractionNode } from '../node/humanInteractionNode'
|
||||
import { createHumanRuntimeCache } from '../node/utils/runtimeCache'
|
||||
import { advanceDispatchState } from '../node/dispatch'
|
||||
import { createHumanRunStateSchema, createInitialHumanState } from '../state/schema'
|
||||
@@ -50,6 +51,7 @@ export const createHumanGraphRunner = (input: { ctx: AiRunContext }) => {
|
||||
const learningNode = createLearningNode({ ctx, cache })
|
||||
const planNode = createPlanNode({ ctx, cache })
|
||||
const handNode = createHandNode({ ctx, cache })
|
||||
const humanInteractionNode = createHumanInteractionNode({ ctx, cache })
|
||||
|
||||
const routeToolReturn = (state: HumanState): string => {
|
||||
if (state.execution.stage === 'context_budget') return 'human_context_budget_node'
|
||||
@@ -74,6 +76,9 @@ export const createHumanGraphRunner = (input: { ctx: AiRunContext }) => {
|
||||
)
|
||||
.addNode('human_plan_node', (state) => runNode(state as HumanState, planNode))
|
||||
.addNode('human_hand_node', (state) => runNode(state as HumanState, handNode))
|
||||
.addNode('human_interaction_node', (state) =>
|
||||
runNode(state as HumanState, humanInteractionNode)
|
||||
)
|
||||
.addEdge(START, 'human_context_budget_node')
|
||||
.addConditionalEdges('human_context_budget_node', (state) => {
|
||||
const humanState = state as HumanState
|
||||
@@ -94,11 +99,13 @@ export const createHumanGraphRunner = (input: { ctx: AiRunContext }) => {
|
||||
if (humanState.execution.stage === 'learning') return 'human_learning_node'
|
||||
if (humanState.execution.stage === 'plan') return 'human_plan_node'
|
||||
if (humanState.execution.stage === 'hand') return 'human_hand_node'
|
||||
if (humanState.execution.stage === 'interaction') return 'human_interaction_node'
|
||||
return 'human_context_budget_node'
|
||||
})
|
||||
.addConditionalEdges('human_learning_node', (state) => routeToolReturn(state as HumanState))
|
||||
.addConditionalEdges('human_plan_node', (state) => routeToolReturn(state as HumanState))
|
||||
.addConditionalEdges('human_hand_node', (state) => routeToolReturn(state as HumanState))
|
||||
.addEdge('human_interaction_node', 'human_brain_node')
|
||||
.compile()
|
||||
|
||||
const baseInitialState = options.initialState || createInitialHumanState(taskInput)
|
||||
|
||||
@@ -7,6 +7,7 @@ const classifyToolCallTarget = (toolName: string): HumanDispatchTarget => {
|
||||
const normalized = String(toolName || '').trim()
|
||||
if (normalized === 'learn_skill') return 'learning'
|
||||
if (normalized === 'update_plan') return 'plan'
|
||||
if (normalized === 'ask_user_question') return 'interaction'
|
||||
return 'hand'
|
||||
}
|
||||
|
||||
@@ -16,8 +17,11 @@ type RawToolCall = {
|
||||
args?: Record<string, unknown>
|
||||
}
|
||||
|
||||
export const buildDispatchQueue = (toolCalls: RawToolCall[]): HumanDispatchItem[] =>
|
||||
toolCalls
|
||||
export const buildDispatchQueue = (toolCalls: RawToolCall[]): HumanDispatchItem[] => {
|
||||
if (toolCalls.some((call) => call.name.trim() === 'ask_user_question') && toolCalls.length !== 1) {
|
||||
throw new Error('ask_user_question must be the only tool call in a model response')
|
||||
}
|
||||
return toolCalls
|
||||
.filter((call) => call.name.trim().length > 0)
|
||||
.map((call) => ({
|
||||
toolCallId: call.id,
|
||||
@@ -25,10 +29,19 @@ export const buildDispatchQueue = (toolCalls: RawToolCall[]): HumanDispatchItem[
|
||||
args: (call.args || {}) as JsonObject,
|
||||
target: classifyToolCallTarget(call.name.trim())
|
||||
}))
|
||||
}
|
||||
|
||||
const dispatchGroupToNextAction = (group: HumanDispatchItem[]): HumanNextAction => {
|
||||
const first = group[0]!
|
||||
switch (first.target) {
|
||||
case 'interaction':
|
||||
if (group.length !== 1) throw new Error('ask_user_question must be called alone')
|
||||
return {
|
||||
action: 'interaction',
|
||||
toolCallId: first.toolCallId,
|
||||
toolName: 'ask_user_question',
|
||||
args: first.args
|
||||
}
|
||||
case 'learning': {
|
||||
const skillIds = Array.isArray(first.args.skillIds)
|
||||
? first.args.skillIds.filter((item): item is string => typeof item === 'string')
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import type { JsonObject, UserQuestion } from '../../../../../../shared/types'
|
||||
import type { AiRunContext } from '../../../types'
|
||||
import type { HumanRuntimeCache, HumanState, ToolCallResult } from '../../types'
|
||||
import { appendLogEntriesToCache } from '../../memory/conversationLogMessages'
|
||||
import { returnToBrain } from '../utils/stateTransitions'
|
||||
|
||||
export const createHumanInteractionNode = (input: {
|
||||
ctx: AiRunContext
|
||||
cache: HumanRuntimeCache
|
||||
}) => {
|
||||
return async (state: HumanState): Promise<HumanState> => {
|
||||
input.ctx.checkCancelled()
|
||||
const action = state.execution.pendingRound?.nextAction
|
||||
if (!action || action.action !== 'interaction') return returnToBrain(state)
|
||||
const questions = Array.isArray(action.args.questions) ? action.args.questions : []
|
||||
if (questions.length < 1 || questions.length > 3) {
|
||||
throw new Error('ask_user_question requires 1 to 3 questions')
|
||||
}
|
||||
const interactionId = randomUUID()
|
||||
const response = await input.ctx.requestUserQuestion({
|
||||
taskId: state.task.taskId,
|
||||
interactionId,
|
||||
toolCallId: action.toolCallId,
|
||||
questions: questions as unknown as UserQuestion[]
|
||||
})
|
||||
if (!response.accepted) throw new Error(response.error)
|
||||
const result: ToolCallResult = {
|
||||
toolName: 'ask_user_question',
|
||||
ok: true,
|
||||
data: response.result as unknown as JsonObject
|
||||
}
|
||||
const content = JSON.stringify(response.result)
|
||||
appendLogEntriesToCache(input.cache, [
|
||||
{ kind: 'tool', toolCallId: action.toolCallId, toolName: 'ask_user_question', ok: true, content },
|
||||
{
|
||||
kind: 'annotation',
|
||||
eventType: 'human_interaction_resolved',
|
||||
payload: { interactionId, kind: 'question', result: response.result as unknown as JsonObject }
|
||||
}
|
||||
])
|
||||
await input.ctx.appendMessageLog([
|
||||
{ kind: 'tool', toolCallId: action.toolCallId, toolName: 'ask_user_question', ok: true, content },
|
||||
{
|
||||
kind: 'annotation',
|
||||
eventType: 'human_interaction_resolved',
|
||||
payload: { interactionId, kind: 'question', result: response.result as unknown as JsonObject }
|
||||
}
|
||||
])
|
||||
return returnToBrain(state, {
|
||||
execution: { pendingRound: undefined, lastToolResults: [result] }
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -93,7 +93,13 @@ export const returnToBrainAfterTraceState = (input: {
|
||||
}
|
||||
|
||||
const resolveBrainNextStage = (action: HumanNextAction['action']): HumanState['execution']['stage'] =>
|
||||
action === 'learning' ? 'learning' : action === 'hand' ? 'hand' : 'plan'
|
||||
action === 'learning'
|
||||
? 'learning'
|
||||
: action === 'hand'
|
||||
? 'hand'
|
||||
: action === 'interaction'
|
||||
? 'interaction'
|
||||
: 'plan'
|
||||
|
||||
export const applyBrainDecisionState = (input: {
|
||||
state: HumanState
|
||||
|
||||
@@ -107,6 +107,12 @@ export const humanReferenceSchema = brainEvidenceReferenceSchema.extend({
|
||||
})
|
||||
|
||||
export const humanNextActionSchema = z.discriminatedUnion('action', [
|
||||
z.object({
|
||||
action: z.literal('interaction'),
|
||||
toolCallId: z.string().min(1),
|
||||
toolName: z.literal('ask_user_question'),
|
||||
args: jsonObjectSchema
|
||||
}),
|
||||
z.object({
|
||||
action: z.literal('learning'),
|
||||
wantLearnSkillIds: z.array(z.string().min(1)).min(1),
|
||||
@@ -162,7 +168,7 @@ export const humanDispatchItemSchema = z.object({
|
||||
toolCallId: z.string().min(1),
|
||||
toolName: z.string().min(1),
|
||||
args: jsonObjectSchema,
|
||||
target: z.enum(['learning', 'plan', 'hand'])
|
||||
target: z.enum(['learning', 'plan', 'hand', 'interaction'])
|
||||
})
|
||||
|
||||
export const humanExecutionSchema = z.object({
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
import { tool } from '@langchain/core/tools'
|
||||
import type { ToolFamilyDefinition } from '../types'
|
||||
import { LEARN_SKILL_TOOL_SPEC, READ_IMAGE_TOOL_SPEC, UPDATE_PLAN_TOOL_SPEC } from './toolSpecs'
|
||||
import {
|
||||
ASK_USER_QUESTION_TOOL_SPEC,
|
||||
LEARN_SKILL_TOOL_SPEC,
|
||||
READ_IMAGE_TOOL_SPEC,
|
||||
UPDATE_PLAN_TOOL_SPEC
|
||||
} from './toolSpecs'
|
||||
import { prepareImagesForMainModel } from './readImage'
|
||||
|
||||
// agent-control 工具只用于让模型通过原生 tool call 表达控制意图(学习技能、更新计划),
|
||||
@@ -35,6 +40,12 @@ export const agentControlToolFamily: ToolFamilyDefinition = {
|
||||
}
|
||||
)
|
||||
|
||||
return [learnSkill, updatePlan, readImage]
|
||||
const askUserQuestion = tool(async () => ({}), {
|
||||
name: ASK_USER_QUESTION_TOOL_SPEC.name,
|
||||
description: ASK_USER_QUESTION_TOOL_SPEC.promptSpec.description,
|
||||
schema: ASK_USER_QUESTION_TOOL_SPEC.inputSchema
|
||||
})
|
||||
|
||||
return [learnSkill, updatePlan, readImage, askUserQuestion]
|
||||
}
|
||||
}
|
||||
@@ -17,6 +17,33 @@ export const READ_IMAGE_SCHEMA = z.object({
|
||||
.describe('本地或远程图片路径,1 至 10 个')
|
||||
})
|
||||
|
||||
export const ASK_USER_QUESTION_SCHEMA = z.object({
|
||||
questions: z
|
||||
.array(
|
||||
z.object({
|
||||
id: z.string().min(1).describe('问题的稳定标识'),
|
||||
question: z.string().min(1).describe('需要用户回答的问题'),
|
||||
options: z
|
||||
.array(
|
||||
z.object({
|
||||
id: z.string().min(1).describe('选项的稳定标识'),
|
||||
label: z.string().min(1).describe('用户看到的选项文字'),
|
||||
description: z.string().min(1).optional().describe('选项影响说明'),
|
||||
recommended: z.boolean().optional().describe('是否为推荐方案')
|
||||
})
|
||||
)
|
||||
.min(1),
|
||||
multiSelect: z.boolean().optional().describe('是否允许同时选择多个选项')
|
||||
})
|
||||
)
|
||||
.min(1)
|
||||
.max(3)
|
||||
.describe('最多 3 个会改变执行结果的问题')
|
||||
})
|
||||
|
||||
export type AskUserQuestionInput = z.infer<typeof ASK_USER_QUESTION_SCHEMA>
|
||||
export type AskUserQuestionResult = { answers: Array<{ id: string; selected: string[]; custom?: string }> }
|
||||
|
||||
export type {
|
||||
LearnSkillOutput,
|
||||
ReadImageItem,
|
||||
|
||||
@@ -9,9 +9,30 @@ import {
|
||||
import {
|
||||
LEARN_SKILL_SCHEMA,
|
||||
READ_IMAGE_SCHEMA,
|
||||
UPDATE_PLAN_SCHEMA
|
||||
UPDATE_PLAN_SCHEMA,
|
||||
ASK_USER_QUESTION_SCHEMA
|
||||
} from './toolSchemas'
|
||||
|
||||
const ASK_USER_QUESTION_DESCRIPTION =
|
||||
'向用户询问会实际改变任务定义或执行结果的关键信息。能从资料确定的信息不要询问;每批最多 3 个相关问题,选项要具体、互斥、可执行,并在有合理默认方案时标记 recommended。用户始终可以选择“其他,请填写”。必须单独调用,不能与写文件、运行命令等工具调用混在同一批。'
|
||||
|
||||
export const ASK_USER_QUESTION_TOOL_SPEC: HumanToolSpec<
|
||||
z.infer<typeof ASK_USER_QUESTION_SCHEMA>,
|
||||
'ask_user_question',
|
||||
'agent-control'
|
||||
> = {
|
||||
name: 'ask_user_question',
|
||||
familyId: 'agent-control',
|
||||
skillIds: ['agent-control'],
|
||||
inputSchema: ASK_USER_QUESTION_SCHEMA,
|
||||
promptSpec: { name: 'ask_user_question', description: ASK_USER_QUESTION_DESCRIPTION },
|
||||
resolveExecutionMeta: createStaticExecutionMetaResolver({
|
||||
mutability: 'read_only',
|
||||
defaultResourceKey: 'agent-control'
|
||||
}),
|
||||
modelResult: { renderSuccess: (result) => JSON.stringify(result) }
|
||||
}
|
||||
|
||||
export const LEARN_SKILL_TOOL_SPEC: HumanToolSpec<
|
||||
z.infer<typeof LEARN_SKILL_SCHEMA>,
|
||||
'learn_skill',
|
||||
@@ -75,5 +96,6 @@ export const READ_IMAGE_TOOL_SPEC: HumanToolSpec<
|
||||
export const AGENT_CONTROL_TOOL_SPECS = [
|
||||
LEARN_SKILL_TOOL_SPEC,
|
||||
UPDATE_PLAN_TOOL_SPEC,
|
||||
READ_IMAGE_TOOL_SPEC
|
||||
READ_IMAGE_TOOL_SPEC,
|
||||
ASK_USER_QUESTION_TOOL_SPEC
|
||||
] as const
|
||||
@@ -119,6 +119,10 @@ export const createAiRunContext = (payload: AiRunStartPayload, deps: AiRunDeps):
|
||||
sendEvent: deps.sendEvent,
|
||||
recordLlmIo: deps.recordLlmIo,
|
||||
requestApprovalBatch: deps.requestApprovalBatch,
|
||||
requestUserQuestion: deps.requestUserQuestion || (async () => ({
|
||||
accepted: false as const,
|
||||
error: '当前运行不支持用户提问'
|
||||
})),
|
||||
getLlmConfig: loadRequestedConfig,
|
||||
browserUseService: deps.browserUseService,
|
||||
webCrawlerService: deps.webCrawlerService,
|
||||
|
||||
@@ -17,6 +17,8 @@ import type { z } from 'zod/v4'
|
||||
import type {
|
||||
AiApprovalBatchDecision,
|
||||
AiApprovalBatchItem,
|
||||
AskUserQuestionInput,
|
||||
AskUserQuestionResult,
|
||||
AiLlmIoRecord,
|
||||
AiLlmStreamingMode,
|
||||
AiLlmUsage,
|
||||
@@ -308,6 +310,13 @@ export type AiRunDeps = {
|
||||
sendEvent: (event: AiRunEvent) => void
|
||||
recordLlmIo?: (record: AiLlmIoRecord) => void
|
||||
requestApprovalBatch: (request: AiApprovalBatchRequest) => Promise<AiApprovalBatchResult>
|
||||
requestUserQuestion?: (request: {
|
||||
taskId: string
|
||||
interactionId: string
|
||||
toolCallId: string
|
||||
questions: AskUserQuestionInput['questions']
|
||||
stage?: AiRunEventStage
|
||||
}) => Promise<{ accepted: true; result: AskUserQuestionResult } | { accepted: false; error: string }>
|
||||
llmDriver?: AiLlmDriver
|
||||
chatToolsWrapper?: (real: AiRunContext['chatTools']) => AiRunContext['chatTools']
|
||||
browserUseService?: BrowserUseService
|
||||
@@ -367,6 +376,13 @@ export type AiRunContext = {
|
||||
sendEvent: (event: AiRunEvent) => void
|
||||
recordLlmIo?: (record: AiLlmIoRecord) => void
|
||||
requestApprovalBatch: (request: AiApprovalBatchRequest) => Promise<AiApprovalBatchResult>
|
||||
requestUserQuestion: (request: {
|
||||
taskId: string
|
||||
interactionId: string
|
||||
toolCallId: string
|
||||
questions: AskUserQuestionInput['questions']
|
||||
stage?: AiRunEventStage
|
||||
}) => Promise<{ accepted: true; result: AskUserQuestionResult } | { accepted: false; error: string }>
|
||||
getLlmConfig: () => Promise<LlmRuntimeConfig>
|
||||
browserUseService?: BrowserUseService
|
||||
webCrawlerService?: WebCrawlerService
|
||||
|
||||
@@ -38,6 +38,7 @@ type AiRunRow = {
|
||||
failure_code?: unknown
|
||||
failure_raw_message?: unknown
|
||||
pending_approval_json?: unknown
|
||||
pending_interaction_json?: unknown
|
||||
artifacts_json?: unknown
|
||||
created_at?: unknown
|
||||
updated_at?: unknown
|
||||
@@ -48,6 +49,7 @@ const normalizeRunStatus = (value: unknown): AiRunStatus => {
|
||||
const status = String(value || '')
|
||||
if (
|
||||
status === 'running' ||
|
||||
status === 'waiting_human' ||
|
||||
status === 'waiting_approval' ||
|
||||
status === 'completed' ||
|
||||
status === 'failed' ||
|
||||
@@ -82,6 +84,9 @@ const toRunRecord = (row: AiRunRow): AiRunRecord => ({
|
||||
: undefined,
|
||||
failureRawMessage: row.failure_raw_message ? String(row.failure_raw_message) : undefined,
|
||||
pendingApproval: parseJsonObject<AiRunRecord['pendingApproval']>(row.pending_approval_json),
|
||||
pendingInteraction: parseJsonObject<AiRunRecord['pendingInteraction']>(
|
||||
row.pending_interaction_json || row.pending_approval_json
|
||||
),
|
||||
artifacts: parseJsonArray<AiArtifactRef>(row.artifacts_json),
|
||||
createdAt: toRequiredNumber(row.created_at),
|
||||
updatedAt: toRequiredNumber(row.updated_at),
|
||||
@@ -100,13 +105,13 @@ export class AiRunStore {
|
||||
selected_references_json, attachments_json, model_name,
|
||||
main_model_json, execution_permission,
|
||||
partial_answer, partial_answer_role, final_answer, error, failure_code, failure_raw_message,
|
||||
pending_approval_json, artifacts_json, created_at, updated_at, finished_at
|
||||
pending_approval_json, pending_interaction_json, artifacts_json, created_at, updated_at, finished_at
|
||||
) VALUES (
|
||||
@id, @partner_id, @user_message_id, @assistant_message_id, @status, @input,
|
||||
@selected_references_json, @attachments_json, @model_name,
|
||||
@main_model_json, @execution_permission,
|
||||
@partial_answer, @partial_answer_role, @final_answer, @error, @failure_code, @failure_raw_message,
|
||||
@pending_approval_json, @artifacts_json, @created_at, @updated_at, @finished_at
|
||||
@pending_approval_json, @pending_interaction_json, @artifacts_json, @created_at, @updated_at, @finished_at
|
||||
)
|
||||
ON CONFLICT(id) DO UPDATE SET
|
||||
partner_id = excluded.partner_id,
|
||||
@@ -126,6 +131,7 @@ export class AiRunStore {
|
||||
failure_code = excluded.failure_code,
|
||||
failure_raw_message = excluded.failure_raw_message,
|
||||
pending_approval_json = excluded.pending_approval_json,
|
||||
pending_interaction_json = excluded.pending_interaction_json,
|
||||
artifacts_json = excluded.artifacts_json,
|
||||
updated_at = excluded.updated_at,
|
||||
finished_at = excluded.finished_at
|
||||
@@ -149,7 +155,8 @@ export class AiRunStore {
|
||||
error: record.error || null,
|
||||
failure_code: record.failureCode || null,
|
||||
failure_raw_message: record.failureRawMessage || null,
|
||||
pending_approval_json: jsonStringify(record.pendingApproval),
|
||||
pending_approval_json: jsonStringify(record.pendingInteraction || record.pendingApproval),
|
||||
pending_interaction_json: jsonStringify(record.pendingInteraction),
|
||||
artifacts_json: jsonStringify(record.artifacts),
|
||||
created_at: record.createdAt,
|
||||
updated_at: record.updatedAt,
|
||||
@@ -281,7 +288,10 @@ export class AiRunStore {
|
||||
| undefined
|
||||
if (!runRow) return { ok: false, error: 'run not found' }
|
||||
const run = toRunRecord(runRow)
|
||||
if (run.status !== 'waiting_approval' || !run.pendingApproval) {
|
||||
if (
|
||||
(run.status !== 'waiting_human' && run.status !== 'waiting_approval') ||
|
||||
!run.pendingApproval
|
||||
) {
|
||||
return { ok: false, error: 'approval batch not found' }
|
||||
}
|
||||
if (run.pendingApproval.batchId !== input.request.batchId) {
|
||||
@@ -295,6 +305,7 @@ export class AiRunStore {
|
||||
UPDATE ai_run
|
||||
SET status = 'running',
|
||||
pending_approval_json = NULL,
|
||||
pending_interaction_json = NULL,
|
||||
updated_at = @updated_at
|
||||
WHERE id = @id
|
||||
`
|
||||
|
||||
@@ -15,6 +15,7 @@ import { repairAiRuntimeSystemNotificationsColumnMigration } from './migrations/
|
||||
import { upgradeFromV9Migration } from './migrations/010_upgrade_from_v9'
|
||||
import { dropTerminalFileChangeRecordsMigration } from './migrations/011_drop_terminal_file_change_records'
|
||||
import { appSettingsMigration } from './migrations/012_app_settings'
|
||||
import { aiRunPendingInteractionMigration } from './migrations/013_ai_run_pending_interaction'
|
||||
|
||||
export type SqliteDatabase = Database.Database
|
||||
export type SqliteStatement = Database.Statement
|
||||
@@ -31,10 +32,11 @@ const MIGRATIONS: readonly Migration[] = [
|
||||
repairAiRuntimeSystemNotificationsColumnMigration,
|
||||
upgradeFromV9Migration,
|
||||
dropTerminalFileChangeRecordsMigration,
|
||||
appSettingsMigration
|
||||
appSettingsMigration,
|
||||
aiRunPendingInteractionMigration
|
||||
]
|
||||
|
||||
export const CURRENT_SCHEMA_VERSION = 12
|
||||
export const CURRENT_SCHEMA_VERSION = 13
|
||||
|
||||
const MIGRATION_NAME_ALIASES = new Map<number, ReadonlySet<string>>([
|
||||
[2, new Set(['workspace_change_sets'])]
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
import type { Migration } from './types'
|
||||
|
||||
export const aiRunPendingInteractionMigration: Migration = {
|
||||
version: 13,
|
||||
name: 'ai_run_pending_interaction',
|
||||
sql: `ALTER TABLE ai_run ADD COLUMN pending_interaction_json TEXT;`
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"publishedThrough": 12,
|
||||
"publishedThrough": 13,
|
||||
"migrations": [
|
||||
{
|
||||
"version": 1,
|
||||
@@ -72,6 +72,12 @@
|
||||
"name": "app_settings",
|
||||
"file": "012_app_settings.ts",
|
||||
"sha256": "0fd373bf26854cb7c6816cc60621242691b2fb9fd93200c40d33c8651e584230"
|
||||
},
|
||||
{
|
||||
"version": 13,
|
||||
"name": "ai_run_pending_interaction",
|
||||
"file": "013_ai_run_pending_interaction.ts",
|
||||
"sha256": "e214a7f29b3354dd824eb94d581811dc8b8155b199348c7c0490d7be6500ba22"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -23,6 +23,8 @@ import type {
|
||||
BrowserTabState,
|
||||
AiRunApprovalBatchDecisionRequest,
|
||||
AiRunApprovalBatchDecisionResponse,
|
||||
AiRunHumanInteractionRequest,
|
||||
AiRunHumanInteractionResponse,
|
||||
AiRunCancelResponse,
|
||||
AiContextUsageRequest,
|
||||
AiContextUsageResponse,
|
||||
@@ -287,6 +289,9 @@ export type DesktopApi = {
|
||||
aiRunApprovalBatchDecide: (
|
||||
payload: AiRunApprovalBatchDecisionRequest
|
||||
) => Promise<AiRunApprovalBatchDecisionResponse>
|
||||
aiRunHumanInteractionSubmit: (
|
||||
payload: AiRunHumanInteractionRequest
|
||||
) => Promise<AiRunHumanInteractionResponse>
|
||||
aiContextUsageGet: (payload: AiContextUsageRequest) => Promise<AiContextUsageResponse>
|
||||
browserPanelSetBounds: (
|
||||
payload: BrowserPanelBoundsRequest
|
||||
|
||||
@@ -392,6 +392,10 @@ const api: DesktopApi = {
|
||||
emitE2EIpcCall('aiRunApprovalBatchDecide')
|
||||
return ipcRenderer.invoke('ai-run-approval-batch-decide', payload)
|
||||
},
|
||||
aiRunHumanInteractionSubmit: (payload) => {
|
||||
emitE2EIpcCall('aiRunHumanInteractionSubmit')
|
||||
return ipcRenderer.invoke('ai-run-human-interaction-submit', payload)
|
||||
},
|
||||
aiContextUsageGet: (payload) => ipcRenderer.invoke('ai-context-usage-get', payload),
|
||||
browserPanelSetBounds: (payload) => ipcRenderer.invoke('browser-panel-set-bounds', payload),
|
||||
browserPanelSetFollow: (payload) => ipcRenderer.send('browser-panel-set-follow', payload),
|
||||
|
||||
@@ -1,7 +1,4 @@
|
||||
import React, { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
|
||||
import { FiCheckCircle } from 'react-icons/fi'
|
||||
import { MdOutlineSmsFailed } from 'react-icons/md'
|
||||
import { RiForbidLine } from 'react-icons/ri'
|
||||
import { FaSpider } from 'react-icons/fa'
|
||||
import { PiBrowsers, PiEyes } from 'react-icons/pi'
|
||||
import {
|
||||
@@ -35,7 +32,6 @@ import type { WorkspaceSource } from '../../../../shared/types'
|
||||
import type { OpenableMarkdownReference } from '../markdownReference'
|
||||
import {
|
||||
findApprovalReviewForProcessItem,
|
||||
formatApprovalReviewRevealText,
|
||||
type AssistantApprovalReview
|
||||
} from '../../../../shared/workbench/approvalReview'
|
||||
import { workbenchSelectors } from '../../testing/selectors'
|
||||
@@ -377,29 +373,15 @@ const buildAssistantTimeline = (input: {
|
||||
return timeline
|
||||
}
|
||||
|
||||
const WAITING_APPROVAL_SUMMARY: AssistantRichTextSegment[] = [{ text: '等待用户审核中' }]
|
||||
|
||||
const ApprovalReviewIcon: React.FC<{ action: AssistantApprovalReview['action'] }> = ({
|
||||
action
|
||||
}) => {
|
||||
if (action === 'approve') {
|
||||
return <FiCheckCircle className="h-4 w-4 shrink-0 text-[#2F6F4E]" aria-hidden />
|
||||
}
|
||||
if (action === 'reject') {
|
||||
return <RiForbidLine className="h-4 w-4 shrink-0 text-[#B3261E]" aria-hidden />
|
||||
}
|
||||
return <MdOutlineSmsFailed className="h-4 w-4 shrink-0 text-[#B45309]" aria-hidden />
|
||||
}
|
||||
const WAITING_APPROVAL_SUMMARY: AssistantRichTextSegment[] = [{ text: '等待用户处理' }]
|
||||
|
||||
const AssistantActivityItemView: React.FC<{
|
||||
item: AssistantProcessGroup['items'][number]
|
||||
review?: AssistantApprovalReview
|
||||
onPreviewImage?: (image: ImageLightboxImage) => void
|
||||
}> = ({ item, review, onPreviewImage }) => {
|
||||
const [revealing, setRevealing] = useState(false)
|
||||
const failed = item.status === 'failed' || item.status === 'partial_failed'
|
||||
const toolText = getItemText(item)
|
||||
const revealText = review ? formatApprovalReviewRevealText(review) : ''
|
||||
const readImagePaths =
|
||||
item.toolName === 'read_image' ? readImagePathsFromInput(item.input) : []
|
||||
const thumbs =
|
||||
@@ -411,6 +393,7 @@ const AssistantActivityItemView: React.FC<{
|
||||
return (
|
||||
<div className={failed ? 'text-[#B3261E]' : undefined}>
|
||||
{renderSegments(toolText)}
|
||||
{item.interaction ? <InteractionResultView interaction={item.interaction} /> : null}
|
||||
{thumbs}
|
||||
</div>
|
||||
)
|
||||
@@ -421,36 +404,24 @@ const AssistantActivityItemView: React.FC<{
|
||||
data-testid={workbenchSelectors.assistantApprovalReviewedItem}
|
||||
className={failed ? 'text-[#B3261E]' : undefined}
|
||||
>
|
||||
<div className="flex min-w-0 items-center gap-1.5 leading-[22px]">
|
||||
<span
|
||||
className="inline-flex h-5 w-5 shrink-0 cursor-help items-center justify-center rounded-[4px] transition-colors duration-150 hover:bg-[#EFEFEE]"
|
||||
onMouseEnter={() => setRevealing(true)}
|
||||
onMouseLeave={() => setRevealing(false)}
|
||||
>
|
||||
<ApprovalReviewIcon action={review.action} />
|
||||
</span>
|
||||
<span className="relative min-w-0 flex-1 overflow-hidden">
|
||||
<span
|
||||
className={`block truncate transition-all duration-150 ${
|
||||
revealing ? 'pointer-events-none translate-x-1 opacity-0' : 'translate-x-0 opacity-100'
|
||||
}`}
|
||||
>
|
||||
{renderSegments(toolText)}
|
||||
</span>
|
||||
<span
|
||||
className={`absolute inset-0 flex items-center truncate text-[#787774] transition-all duration-150 ${
|
||||
revealing ? 'translate-x-0 opacity-100' : 'pointer-events-none -translate-x-1 opacity-0'
|
||||
}`}
|
||||
>
|
||||
{revealText}
|
||||
</span>
|
||||
</span>
|
||||
</div>
|
||||
<div className="min-w-0 leading-[22px]">{renderSegments(toolText)}</div>
|
||||
<div className="mt-1 text-xs text-[#787774]">审核结果:{review.action === 'approve' ? '已允许' : review.action === 'reject' ? '已拒绝' : `已提供指导${review.message ? `:${review.message}` : ''}`}</div>
|
||||
{thumbs}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
const InteractionResultView: React.FC<{ interaction: NonNullable<AssistantProcessGroup['items'][number]['interaction']> }> = ({ interaction }) => (
|
||||
<div className="mt-1 space-y-1 border-l border-[#E0DFDC] pl-3 text-xs text-[#787774]">
|
||||
{interaction.questions.map((question) => {
|
||||
const answer = interaction.answers?.find((item) => item.id === question.id)
|
||||
const labels = question.options.filter((option) => answer?.selected.includes(option.id)).map((option) => option.label)
|
||||
if (answer?.custom) labels.push(answer.custom)
|
||||
return <div key={question.id}><div>{question.question}</div><div className="text-[#37352F]">{labels.join('、') || '未填写'}</div></div>
|
||||
})}
|
||||
</div>
|
||||
)
|
||||
|
||||
const AssistantActivityGroupView: React.FC<{
|
||||
groups: AssistantProcessGroup[]
|
||||
id: string
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
import React, { useMemo, useState } from 'react'
|
||||
import { Button, Input } from 'antd'
|
||||
import { HiArrowUp, HiChevronLeft, HiChevronRight } from 'react-icons/hi2'
|
||||
import type { PendingHumanInteraction, AskUserQuestionAnswer } from '../../../../shared/types'
|
||||
|
||||
type Props = {
|
||||
interaction: Extract<PendingHumanInteraction, { kind: 'question' }>
|
||||
onSubmit: (answers: { answers: AskUserQuestionAnswer[] }) => Promise<void>
|
||||
onCancel: () => void
|
||||
}
|
||||
|
||||
const OTHER_ID = '__custom__'
|
||||
|
||||
const HumanInteractionCard: React.FC<Props> = ({ interaction, onSubmit, onCancel }) => {
|
||||
const [index, setIndex] = useState(0)
|
||||
const [selected, setSelected] = useState<Record<string, string[]>>({})
|
||||
const [custom, setCustom] = useState<Record<string, string>>({})
|
||||
const [submitting, setSubmitting] = useState(false)
|
||||
const question = interaction.questions[index]
|
||||
const currentSelected = selected[question.id] || []
|
||||
const isCustom = currentSelected.includes(OTHER_ID)
|
||||
const canSubmit = useMemo(
|
||||
() => interaction.questions.every((item) => {
|
||||
const values = selected[item.id] || []
|
||||
return values.length > 0 && (!values.includes(OTHER_ID) || Boolean(custom[item.id]?.trim()))
|
||||
}),
|
||||
[custom, interaction.questions, selected]
|
||||
)
|
||||
const toggle = (id: string): void => {
|
||||
setSelected((previous) => {
|
||||
const values = previous[question.id] || []
|
||||
if (id === OTHER_ID) return { ...previous, [question.id]: values.includes(id) ? values.filter((value) => value !== id) : question.multiSelect ? [...values, id] : [id] }
|
||||
if (!question.multiSelect) return { ...previous, [question.id]: [id] }
|
||||
return { ...previous, [question.id]: values.includes(id) ? values.filter((value) => value !== id) : [...values.filter((value) => value !== OTHER_ID), id] }
|
||||
})
|
||||
}
|
||||
const submit = async (): Promise<void> => {
|
||||
if (!canSubmit || submitting) return
|
||||
setSubmitting(true)
|
||||
try {
|
||||
await onSubmit({
|
||||
answers: interaction.questions.map((item) => {
|
||||
const values = selected[item.id] || []
|
||||
return { id: item.id, selected: values.filter((value) => value !== OTHER_ID), custom: custom[item.id]?.trim() || undefined }
|
||||
})
|
||||
})
|
||||
} finally {
|
||||
setSubmitting(false)
|
||||
}
|
||||
}
|
||||
return (
|
||||
<div className="nc-timeline-layout-anchor min-w-0 overflow-hidden rounded-[12px] border border-[#E0DFDC] bg-white shadow-[0_8px_24px_rgba(15,15,15,0.08)]" data-testid="workbench-human-question-card">
|
||||
<div className="flex items-center justify-between px-5 pt-4">
|
||||
<div className="text-[15px] font-semibold text-[#37352F]">需要补充一点信息</div>
|
||||
<div className="flex items-center gap-1 text-xs text-[#9B9A97]">
|
||||
<button type="button" aria-label="上一个问题" disabled={index === 0 || submitting} onClick={() => setIndex((value) => Math.max(0, value - 1))}><HiChevronLeft /></button>
|
||||
<span>{index + 1} / {interaction.questions.length}</span>
|
||||
<button type="button" aria-label="下一个问题" disabled={index === interaction.questions.length - 1 || submitting} onClick={() => setIndex((value) => Math.min(interaction.questions.length - 1, value + 1))}><HiChevronRight /></button>
|
||||
</div>
|
||||
</div>
|
||||
<div className="px-5 pt-3 text-[14px] leading-6 text-[#37352F]">{question.question}</div>
|
||||
<div className="mx-2 mt-3 space-y-1">
|
||||
{question.options.map((option, optionIndex) => {
|
||||
const active = currentSelected.includes(option.id)
|
||||
return <button key={option.id} type="button" className={`grid min-h-11 w-full grid-cols-[auto_1fr] gap-2 rounded-[6px] px-3 text-left text-[13px] ${active ? 'bg-[#F4F4F2] shadow-[inset_3px_0_0_#37352F]' : 'hover:bg-[#F8F8F7]'}`} onClick={() => toggle(option.id)}><span className="font-medium">{String.fromCharCode(65 + optionIndex)}.</span><span><span>{option.label}{option.recommended ? <span className="ml-2 text-xs text-[#9B9A97]">推荐</span> : null}</span>{option.description ? <span className="block text-xs text-[#9B9A97]">{option.description}</span> : null}</span></button>
|
||||
})}
|
||||
<button type="button" className={`grid min-h-11 w-full grid-cols-[auto_1fr] gap-2 rounded-[6px] px-3 text-left text-[13px] ${isCustom ? 'bg-[#F4F4F2] shadow-[inset_3px_0_0_#37352F]' : 'hover:bg-[#F8F8F7]'}`} onClick={() => toggle(OTHER_ID)}><span className="font-medium">{String.fromCharCode(65 + question.options.length)}.</span><span>其他,请填写</span></button>
|
||||
</div>
|
||||
{isCustom ? <div className="px-4 pt-3"><Input value={custom[question.id] || ''} onChange={(event) => setCustom((previous) => ({ ...previous, [question.id]: event.target.value }))} placeholder="请填写你的要求" /></div> : null}
|
||||
<div className="flex justify-end gap-2 border-t border-[#EEEEEC] px-4 py-4"><Button type="text" onClick={onCancel}>停止</Button><Button type="primary" icon={<HiArrowUp />} loading={submitting} disabled={!canSubmit} onClick={() => void submit()}>提交</Button></div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
export default HumanInteractionCard
|
||||
@@ -101,6 +101,17 @@ export const projectProcessGroups = (input: {
|
||||
createdAt,
|
||||
sourceEventType: 'assistant_process_item_started'
|
||||
})
|
||||
const interaction = entry.toolName === 'ask_user_question'
|
||||
? (() => {
|
||||
try {
|
||||
const parsed = JSON.parse(entry.content) as { answers?: Array<{ id: string; selected: string[]; custom?: string }> }
|
||||
const questions = Array.isArray((toolArgsById.get(entry.toolCallId) as { questions?: unknown })?.questions)
|
||||
? ((toolArgsById.get(entry.toolCallId) as { questions: Array<{ id: string; question: string; options: Array<{ id: string; label: string }> }> }).questions)
|
||||
: []
|
||||
return { questions, answers: parsed.answers }
|
||||
} catch { return undefined }
|
||||
})()
|
||||
: undefined
|
||||
const group = createDisplayProcessGroup({
|
||||
groupId: `${input.runId}:tool:${entry.toolCallId}`,
|
||||
kind: 'skill',
|
||||
@@ -109,7 +120,7 @@ export const projectProcessGroups = (input: {
|
||||
createdAt,
|
||||
sourceEventType: 'assistant_process_group_started'
|
||||
})
|
||||
upsertGroup({ ...group, items: [item] })
|
||||
upsertGroup({ ...group, items: [interaction ? { ...item, interaction } : item] })
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -127,6 +127,10 @@ export type AssistantProcessGroupItem = {
|
||||
toolName?: AssistantToolName
|
||||
targetLabel?: string
|
||||
input?: IpcJsonObject
|
||||
interaction?: {
|
||||
questions: Array<{ id: string; question: string; options: Array<{ id: string; label: string }> }>
|
||||
answers?: Array<{ id: string; selected: string[]; custom?: string }>
|
||||
}
|
||||
}
|
||||
|
||||
export type AssistantProcessGroup = {
|
||||
|
||||
@@ -375,6 +375,7 @@ const TOOL_ITEM_RENDERERS = {
|
||||
return toolCopy('操作浏览器', sentence(text('操作浏览器 '), param(instructionLabel)))
|
||||
},
|
||||
read_image: () => toolCopy('读取图片', sentence(text('读取图片'))),
|
||||
ask_user_question: () => toolCopy('询问用户', sentence(text('等待用户确认关键信息'))),
|
||||
image_generate: (input) =>
|
||||
toolCopy('生成图片', sentence(text('生成图片 '), param(readToolFileLabel(input)))),
|
||||
learn_skill: (input, status) => {
|
||||
|
||||
@@ -38,7 +38,10 @@ import type {
|
||||
ScheduledTask,
|
||||
SelectedReference,
|
||||
WorkbenchModelSelection,
|
||||
WorkspaceSource
|
||||
WorkspaceSource,
|
||||
PendingHumanInteraction,
|
||||
AskUserQuestionAnswer,
|
||||
AiRunHumanInteractionRequest
|
||||
} from '../../../shared/types'
|
||||
import { PARTNER_AVATAR_SHAPES, PARTNER_AVATAR_COLORS } from '../../../shared/types'
|
||||
import {
|
||||
@@ -59,6 +62,7 @@ import { HiOutlineCog6Tooth } from 'react-icons/hi2'
|
||||
import AssistantTurnCard from '../components/workbench/AssistantTurnCard'
|
||||
import { FullAccessConfirmModal } from '../components/workbench/FullAccessConfirmModal'
|
||||
import WorkbenchComposer from '../components/workbench/WorkbenchComposer'
|
||||
import HumanInteractionCard from '../components/workbench/HumanInteractionCard'
|
||||
import { buildApprovalGuidePlaceholder } from '../components/workbench/composer/constants'
|
||||
import { MessageTimeline } from '../components/workbench/MessageTimeline'
|
||||
import {
|
||||
@@ -457,6 +461,14 @@ const reduceAssistantTurnByRunEvent = (
|
||||
}
|
||||
}
|
||||
|
||||
if (event.type === 'human_interaction_required' && event.interaction.kind === 'question') {
|
||||
next = { ...next, isBrainThinking: false, waitingApproval: true }
|
||||
}
|
||||
|
||||
if (event.type === 'human_interaction_resolved' && event.kind === 'question') {
|
||||
next = { ...next, waitingApproval: false }
|
||||
}
|
||||
|
||||
if (event.type === 'assistant_brain_stream_started') {
|
||||
const retryRunning = next.processGroups.some(
|
||||
(group) => group.kind === 'retry' && group.status === 'running'
|
||||
@@ -642,7 +654,7 @@ const buildAssistantTurnFromRunSnapshot = (
|
||||
processThoughts: projected.processThoughts,
|
||||
processGroups: projected.processGroups,
|
||||
isBrainThinking: projected.isBrainThinking,
|
||||
waitingApproval: Boolean(snapshot.run.pendingApproval),
|
||||
waitingApproval: Boolean(snapshot.run.pendingApproval || snapshot.run.pendingInteraction),
|
||||
pendingApprovalItems: snapshot.run.pendingApproval?.items || [],
|
||||
approvalReviews
|
||||
}
|
||||
@@ -877,6 +889,10 @@ const ChatPage: React.FC = () => {
|
||||
Record<string, ApprovalDraftDecision>
|
||||
>({})
|
||||
const [approvalSubmitting, setApprovalSubmitting] = useState(false)
|
||||
const [pendingQuestion, setPendingQuestion] = useState<{
|
||||
runId: string
|
||||
interaction: Extract<PendingHumanInteraction, { kind: 'question' }>
|
||||
} | null>(null)
|
||||
|
||||
const activeRunMessageIdsRef = useRef<Record<string, string>>({})
|
||||
const activeRunPartnerIdsRef = useRef<Record<string, string>>({})
|
||||
@@ -2002,6 +2018,12 @@ const ChatPage: React.FC = () => {
|
||||
activeRuns.find((item) => item.pendingApproval && !item.pendingApproval.internal)
|
||||
?.pendingApproval || null
|
||||
setPendingApprovalBatch(approval)
|
||||
const questionRun = activeRuns.find((item) => item.pendingInteraction?.kind === 'question')
|
||||
setPendingQuestion(
|
||||
questionRun?.pendingInteraction?.kind === 'question'
|
||||
? { runId: questionRun.run.id, interaction: questionRun.pendingInteraction }
|
||||
: null
|
||||
)
|
||||
setApprovalCurrentIndex(0)
|
||||
setApprovalDraftDecisions({})
|
||||
setApprovalSubmitting(false)
|
||||
@@ -3330,6 +3352,22 @@ const ChatPage: React.FC = () => {
|
||||
return
|
||||
}
|
||||
|
||||
if (event.type === 'human_interaction_required') {
|
||||
if (event.interaction.kind === 'question') {
|
||||
setLoading(false)
|
||||
setPendingQuestion({ runId: event.runId, interaction: event.interaction })
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if (event.type === 'human_interaction_resolved') {
|
||||
if (event.kind === 'question') {
|
||||
setPendingQuestion(null)
|
||||
setLoading(true)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if (event.type === 'approval_batch_decision') {
|
||||
if (event.internal) return
|
||||
setApprovalSubmitting(false)
|
||||
@@ -3370,6 +3408,7 @@ const ChatPage: React.FC = () => {
|
||||
setAiRunId(null)
|
||||
setCancellingRun(false)
|
||||
setPendingApprovalBatch(null)
|
||||
setPendingQuestion(null)
|
||||
setApprovalSubmitting(false)
|
||||
setCurrentMessages((previous) =>
|
||||
previous.map((item) =>
|
||||
@@ -3396,6 +3435,7 @@ const ChatPage: React.FC = () => {
|
||||
setAiRunId(null)
|
||||
setCancellingRun(false)
|
||||
setPendingApprovalBatch(null)
|
||||
setPendingQuestion(null)
|
||||
setApprovalSubmitting(false)
|
||||
setCurrentMessages((previous) =>
|
||||
previous.map((item) =>
|
||||
@@ -3420,6 +3460,7 @@ const ChatPage: React.FC = () => {
|
||||
setAiRunId(null)
|
||||
setCancellingRun(false)
|
||||
setPendingApprovalBatch(null)
|
||||
setPendingQuestion(null)
|
||||
setApprovalSubmitting(false)
|
||||
if (eventPartnerId && eventPartnerId === currentPartnerIdRef.current) {
|
||||
void loadFileChangesForPartnerRef.current(eventPartnerId, event.runId, {
|
||||
@@ -3576,6 +3617,7 @@ const ChatPage: React.FC = () => {
|
||||
scrollToBottom(true)
|
||||
setCancellingRun(false)
|
||||
setPendingApprovalBatch(null)
|
||||
setPendingQuestion(null)
|
||||
setApprovalSubmitting(false)
|
||||
lastSubmittedInputRef.current = userQuery
|
||||
setPartners((items) =>
|
||||
@@ -3832,6 +3874,21 @@ const ChatPage: React.FC = () => {
|
||||
setApprovalDraftDecisions({})
|
||||
}
|
||||
|
||||
const submitHumanQuestion = async (answers: { answers: AskUserQuestionAnswer[] }): Promise<void> => {
|
||||
if (!pendingQuestion) return
|
||||
const response = await window.api.aiRunHumanInteractionSubmit({
|
||||
runId: pendingQuestion.runId,
|
||||
interactionId: pendingQuestion.interaction.interactionId,
|
||||
answers
|
||||
} satisfies AiRunHumanInteractionRequest)
|
||||
if (!response.success) {
|
||||
emitErrorMessage(response.error || '提交问题回答失败')
|
||||
return
|
||||
}
|
||||
setPendingQuestion(null)
|
||||
setLoading(true)
|
||||
}
|
||||
|
||||
const flushApprovalBatch = async (
|
||||
draftDecisions: Record<string, ApprovalDraftDecision>
|
||||
): Promise<void> => {
|
||||
@@ -4714,6 +4771,12 @@ const ChatPage: React.FC = () => {
|
||||
? '还没有伙伴。从左侧创建一个开始对话。'
|
||||
: '请选择一个伙伴开始对话。'}
|
||||
</div>
|
||||
) : pendingQuestion ? (
|
||||
<HumanInteractionCard
|
||||
interaction={pendingQuestion.interaction}
|
||||
onSubmit={submitHumanQuestion}
|
||||
onCancel={() => void handleCancel()}
|
||||
/>
|
||||
) : pendingApprovalBatch ? (
|
||||
<div
|
||||
className="nc-timeline-layout-anchor min-w-0 overflow-hidden rounded-[12px] border border-[#E0DFDC] bg-[#FFFFFF] shadow-[0_8px_24px_rgba(15,15,15,0.08)]"
|
||||
|
||||
@@ -240,6 +240,7 @@ export const workbenchSelectors = {
|
||||
readImageThumbPrefix: 'workbench-read-image-thumb',
|
||||
assistantApprovalWaiting: 'workbench-assistant-approval-waiting',
|
||||
assistantApprovalReviewedItem: 'workbench-assistant-approval-reviewed-item',
|
||||
humanQuestionCard: 'workbench-human-question-card',
|
||||
assistantFinalAnswerPrefix: 'workbench-assistant-final-answer',
|
||||
assistantRunFailurePrefix: 'workbench-assistant-run-failure-card',
|
||||
assistantRunFailureDetailTogglePrefix: 'workbench-assistant-run-failure-detail-toggle',
|
||||
|
||||
@@ -110,7 +110,8 @@ const TOOL_UI_KINDS = {
|
||||
image_generate: 'image-generation',
|
||||
learn_skill: 'learning',
|
||||
update_plan: 'generic',
|
||||
read_image: 'image-read'
|
||||
read_image: 'image-read',
|
||||
ask_user_question: 'generic'
|
||||
} as const satisfies Record<RuntimeToolName, AssistantExecutionSkillKey>
|
||||
|
||||
export const getAssistantToolUiKind = (
|
||||
|
||||
@@ -48,6 +48,7 @@ export const RUNTIME_TOOL_CATALOG = {
|
||||
learn_skill: { familyId: 'agent-control', skillId: 'agent-control' },
|
||||
update_plan: { familyId: 'agent-control', skillId: 'agent-control' },
|
||||
read_image: { familyId: 'agent-control', skillId: 'agent-control' }
|
||||
,ask_user_question: { familyId: 'agent-control', skillId: 'agent-control' }
|
||||
} as const satisfies Record<string, RuntimeToolCatalogEntry>
|
||||
|
||||
export type RuntimeToolName = keyof typeof RUNTIME_TOOL_CATALOG
|
||||
|
||||
+77
-1
@@ -1048,7 +1048,7 @@ export interface OpenFileWithApplicationResponse {
|
||||
error?: string
|
||||
}
|
||||
|
||||
export type AiRunStatus = 'running' | 'waiting_approval' | 'completed' | 'failed' | 'cancelled'
|
||||
export type AiRunStatus = 'running' | 'waiting_human' | 'waiting_approval' | 'completed' | 'failed' | 'cancelled'
|
||||
export type AiRunWarningCode =
|
||||
| 'workspace_binding_missing'
|
||||
| 'workspace_missing'
|
||||
@@ -1203,6 +1203,52 @@ export type AiArtifactRef = AiFileArtifactRef
|
||||
export type AiPreference = 'default' | 'local_sources_relaxed' | 'local_sources_strict'
|
||||
export type AiExecutionPermission = 'review' | 'full_access'
|
||||
|
||||
export type UserQuestionOption = {
|
||||
id: string
|
||||
label: string
|
||||
description?: string
|
||||
recommended?: boolean
|
||||
}
|
||||
|
||||
export type UserQuestion = {
|
||||
id: string
|
||||
question: string
|
||||
options: UserQuestionOption[]
|
||||
multiSelect?: boolean
|
||||
}
|
||||
|
||||
export type AskUserQuestionInput = { questions: UserQuestion[] }
|
||||
|
||||
export type AskUserQuestionAnswer = {
|
||||
id: string
|
||||
selected: string[]
|
||||
custom?: string
|
||||
}
|
||||
|
||||
export type AskUserQuestionResult = { answers: AskUserQuestionAnswer[] }
|
||||
|
||||
export type PendingHumanInteraction =
|
||||
| {
|
||||
kind: 'question'
|
||||
interactionId: string
|
||||
toolCallId: string
|
||||
questions: UserQuestion[]
|
||||
}
|
||||
| {
|
||||
kind: 'approval'
|
||||
interactionId: string
|
||||
items: AiApprovalBatchItem[]
|
||||
batchId?: string
|
||||
}
|
||||
|
||||
export type AiRunHumanInteractionRequest = {
|
||||
runId: string
|
||||
interactionId: string
|
||||
answers: AskUserQuestionResult
|
||||
}
|
||||
|
||||
export type AiRunHumanInteractionResponse = { success: boolean; error?: string }
|
||||
|
||||
export type ChatSubmitRequest = {
|
||||
preference?: AiPreference
|
||||
executionPermission?: AiExecutionPermission
|
||||
@@ -1286,6 +1332,7 @@ export type AiRunRecord = {
|
||||
failureCode?: AiRunFailureCode
|
||||
failureRawMessage?: string
|
||||
pendingApproval?: Extract<AiRunEvent, { type: 'approval_batch_required' }>
|
||||
pendingInteraction?: PendingHumanInteraction
|
||||
artifacts?: AiArtifactRef[]
|
||||
createdAt: number
|
||||
updatedAt: number
|
||||
@@ -1296,6 +1343,7 @@ export type PartnerRunSnapshot = {
|
||||
run: AiRunRecord
|
||||
messageLogRecords?: ChatMessageLogRecord[]
|
||||
pendingApproval?: Extract<AiRunEvent, { type: 'approval_batch_required' }>
|
||||
pendingInteraction?: PendingHumanInteraction
|
||||
}
|
||||
|
||||
export type PartnerSnapshotRequest = {
|
||||
@@ -1860,6 +1908,25 @@ type AiRunEventPayload =
|
||||
type: 'run_cancelled'
|
||||
runId: string
|
||||
}
|
||||
| {
|
||||
type: 'human_interaction_required'
|
||||
runId: string
|
||||
taskId: string
|
||||
interactionId: string
|
||||
interaction: PendingHumanInteraction
|
||||
createdAt: number
|
||||
stage?: AiRunEventStage
|
||||
}
|
||||
| {
|
||||
type: 'human_interaction_resolved'
|
||||
runId: string
|
||||
taskId: string
|
||||
interactionId: string
|
||||
kind: 'question' | 'approval'
|
||||
result: JsonObject
|
||||
resolvedAt: number
|
||||
stage?: AiRunEventStage
|
||||
}
|
||||
|
||||
export type AiRunEvent = AiRunEventPayload & {
|
||||
internal?: boolean
|
||||
@@ -1936,6 +2003,15 @@ export type ChatMessageLogAnnotationEntry =
|
||||
}>
|
||||
}
|
||||
}
|
||||
| {
|
||||
kind: 'annotation'
|
||||
eventType: 'human_interaction_resolved'
|
||||
payload: {
|
||||
interactionId: string
|
||||
kind: 'question' | 'approval'
|
||||
result: JsonObject
|
||||
}
|
||||
}
|
||||
|
||||
export type ChatMessageLogEntry =
|
||||
| {
|
||||
|
||||
@@ -7,7 +7,7 @@ import type {
|
||||
SelectedReference
|
||||
} from '../types'
|
||||
|
||||
const ACTIVE_RUN_STATUSES = new Set<AiRunRecord['status']>(['running', 'waiting_approval'])
|
||||
const ACTIVE_RUN_STATUSES = new Set<AiRunRecord['status']>(['running', 'waiting_human', 'waiting_approval'])
|
||||
|
||||
const isDeliveryAssistant = (record: ChatMessageLogRecord): boolean =>
|
||||
record.entry.kind === 'assistant' && record.entry.toolCalls.length === 0
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { ASK_USER_QUESTION_SCHEMA } from '../../../../src/main/services/ai/human/tools/agentControl/toolSchemas'
|
||||
import { buildDispatchQueue } from '../../../../src/main/services/ai/human/node/dispatch'
|
||||
|
||||
describe('ask_user_question contract', () => {
|
||||
it('accepts minimal questions and rejects more than three', () => {
|
||||
const question = { id: 'goal', question: '选择目标', options: [{ id: 'a', label: '方案 A' }] }
|
||||
expect(ASK_USER_QUESTION_SCHEMA.safeParse({ questions: [question] }).success).toBe(true)
|
||||
expect(ASK_USER_QUESTION_SCHEMA.safeParse({ questions: [question, question, question, question] }).success).toBe(false)
|
||||
})
|
||||
|
||||
it('must be the only tool call in a model response', () => {
|
||||
expect(() => buildDispatchQueue([{ id: 'q', name: 'ask_user_question', args: {} }])).not.toThrow()
|
||||
expect(() => buildDispatchQueue([
|
||||
{ id: 'q', name: 'ask_user_question', args: {} },
|
||||
{ id: 'w', name: 'text_file_edit', args: {} }
|
||||
])).toThrow(/must be the only tool call/)
|
||||
})
|
||||
})
|
||||
Reference in new issue
Block a user