Files
Termind/TermindTests/AgentUserInputTests.swift
T
Kaede AkinoandClaude Opus 5 22258e3f2f Run a turn's tools at once, not one after another
`AgentRuntime` was one actor doing four jobs: holding the run's state,
driving the loop, executing tools, and publishing events. Adding an
audit, a rewrite, or a new tool source meant editing it. The loop now
keeps only the state and the control flow, and each phase is a value it
drives — `AgentContextPipeline` at the model boundary, `AgentTurnDriver`
for one assistant turn, `AgentToolScheduler` for the batch,
`AgentToolExecutor` for a single call, `AgentToolRegistry` for where
tools come from. Each is testable without a run around it. Four seams are
protocols with real conformers: `AgentToolProviding`, `AgentLoopHook`,
`AgentContextTransforming`, `AgentModelStreaming`. Authorization is
deliberately not among them — `AgentApprovalHandling` stays a structural
stage of the executor, because a permission gate you can remove by
leaving an element out of an array is not a gate.

A turn's calls run concurrently only when every one of them is marked
parallel; a single sequential call takes the whole turn serial rather
than being fenced into its own segment. Segmenting would buy back only
the cross-host case and cost semantics nobody can hold in their head
while reading an incident transcript. Concurrency is declared per tool
and may be overridden per call by preflight, which is how `ssh_exec`
decides: parallel exactly when `CommandRiskClassifier` proves the command
safe, the same locally-proven evidence that already decides its safety.
MCP tools are parallel by default, and a server's destructive hint or Ask
First makes them sequential — the hints can only tighten concurrency,
never grant it, so remote annotations remain risk hints rather than proof
of safety. The ceiling is worth stating plainly: `SSHExecSession` and
`SFTPClient` are actors, so calls aimed at one host still queue inside
their session. The win is several hosts at once, and host work
overlapping `fetch`, `web_search`, and MCP.

Whatever order they finish in, results reach the transcript in the order
the model asked. `toolFinished` still follows completion, because that is
when a card should stop spinning, so the two orders now disagree by
design — and `AgentToolResultOrdering` repairs it at the model boundary,
where both the run's snapshot and the conversation's own copy pass
through one implementation.

Three defects that serial execution was hiding, in rising order of harm.
A denied call was relabelled `Failed` the instant it was refused, because
the denial is an error to the model and the card followed the result;
`Denied` therefore never reached the screen. A run that gave up halfway —
cancelled, out of tool budget, or handed a response truncated
mid-arguments — left an assistant turn whose tool calls nothing answered,
and every provider rejects that outright on the *next* request, so an
interrupted run became a permanently unusable conversation. Every call
now gets a card and a result on every path, refusals included. And
`pendingApproval` was a single slot: two calls needing approval in one
turn overwrote each other, the first continuation was never resumed, and
the run stopped with no error and nothing on screen to answer. Approvals
and agent questions are queues now, presented one at a time with a
`1 of N` count, and run state is derived from what is outstanding rather
than asserted by whichever call moved last.

Events travel one ordered path. The runtime and the approval broker used
to publish independently, each wrapping its durable write in its own
unstructured `Task`, so a review's events had no defined order against
the tool events they belong between. `AgentEventChannel` fans out to the
UI and to a single serial recorder. `messageStarted` now carries its
message rather than just an id, so the visible turn is stamped with the
run's clock instead of whenever the UI got around to the event — a card
proposed later could otherwise carry an earlier timestamp than the turn
that asked for it, and sort above it.

`AgentContextPipeline` is also where trimming finally has somewhere to
live. `ssh_exec` returns up to 256 KB and `sftp_read` up to 128 KB, and
each is replayed on every later turn; two log reads were enough to crowd
out the conversation they were meant to explain. At most one turn's tool
output is now replayed in full, and older results keep their head and
their tail, because a truncated command output missing its ending has
lost the error message. It shapes only the request: snapshots and stored
history keep everything.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 09:28:07 +08:00

257 lines
9.1 KiB
Swift

import Foundation
import Testing
@testable import Termind
@MainActor
@Suite
struct AgentUserInputTests {
// MARK: - Parsing the tool's options
@Test
func optionsCarryTheirDescriptionAndRecommendation() throws {
let options = try arguments([
"prompt": .string("Which database?"),
"options": .array([
.object([
"id": .string("postgres"), "label": .string("Postgres"),
"description": .string("JSONB, and it matches the existing schema."),
"recommended": .bool(true),
]),
.object(["id": .string("sqlite"), "label": .string("SQLite")]),
]),
]).userInputOptions()
#expect(options.count == 2)
#expect(options[0].id == "postgres")
#expect(options[0].detail == "JSONB, and it matches the existing schema.")
#expect(options[0].isRecommended)
// Absent is absent, not false-y noise.
#expect(options[1].detail.isEmpty)
#expect(!options[1].isRecommended)
}
@Test
func noOptionsMeansAFreeFormQuestion() throws {
#expect(try arguments(["prompt": .string("Which URL?")]).userInputOptions().isEmpty)
#expect(
try arguments(["prompt": .string("Which URL?"), "options": .null])
.userInputOptions().isEmpty
)
}
/// A malformed row is rejected rather than dropped: a menu quietly missing
/// the option the user wanted is worse than an error the model can retry.
@Test
func malformedOptionsAreRejectedRatherThanDropped() {
#expect(throws: AgentToolError.self) {
try arguments([
"prompt": .string("?"),
"options": .array([.object(["label": .string("No id here")])]),
]).userInputOptions()
}
#expect(throws: AgentToolError.self) {
try arguments([
"prompt": .string("?"),
"options": .array([.string("a bare string")]),
]).userInputOptions()
}
}
@Test
func duplicateOptionIdsAreRejected() {
#expect(throws: AgentToolError.self) {
try arguments([
"prompt": .string("?"),
"options": .array([
.object(["id": .string("a"), "label": .string("First")]),
.object(["id": .string("a"), "label": .string("Second")]),
]),
]).userInputOptions()
}
}
// MARK: - One answer, or several
@Test
func nothingIsPreselectedEvenWhenAnOptionIsRecommended() {
let conversation = ConversationModel()
conversation.present(request(multiple: false))
#expect(conversation.userInputSelection.isEmpty)
// A recommendation is a label, not a default — Continue stays off until
// the user actually answers.
#expect(!conversation.canSubmitUserInput)
}
@Test
func choosingAnOptionInASingleAnswerQuestionClearsATypedOne() {
let conversation = ConversationModel()
conversation.present(request(multiple: false))
conversation.setUserInputCustomAnswer("MySQL")
#expect(conversation.canSubmitUserInput)
conversation.toggleUserInputOption("postgres")
#expect(conversation.userInputSelection == ["postgres"])
#expect(conversation.userInputDraft.isEmpty)
}
@Test
func typingInASingleAnswerQuestionClearsTheChosenOption() {
let conversation = ConversationModel()
conversation.present(request(multiple: false))
conversation.toggleUserInputOption("postgres")
conversation.setUserInputCustomAnswer("MySQL")
#expect(conversation.userInputSelection.isEmpty)
#expect(conversation.userInputDraft == "MySQL")
}
@Test
func singleAnswerSelectionReplacesRatherThanAccumulates() {
let conversation = ConversationModel()
conversation.present(request(multiple: false))
conversation.toggleUserInputOption("postgres")
conversation.toggleUserInputOption("sqlite")
#expect(conversation.userInputSelection == ["sqlite"])
// Clicking the chosen one again takes the answer back.
conversation.toggleUserInputOption("sqlite")
#expect(conversation.userInputSelection.isEmpty)
}
/// When several answers are allowed the two routes are additive, not
/// exclusive — "these two, plus this other thing" is a real answer.
@Test
func severalAnswersMayIncludeATypedOne() {
let conversation = ConversationModel()
conversation.present(request(multiple: true))
conversation.toggleUserInputOption("postgres")
conversation.toggleUserInputOption("sqlite")
conversation.setUserInputCustomAnswer("and DuckDB")
#expect(conversation.userInputSelection == ["postgres", "sqlite"])
#expect(conversation.userInputDraft == "and DuckDB")
}
// MARK: - What the tool gets back
@Test
func answerReportsIdsInTheOrderTheRequestListedThem() async {
let conversation = ConversationModel()
let pending = request(multiple: true)
let answering = Task { await conversation.request(pending) }
await settle(conversation)
// Clicked back to front; the answer still reads front to back, because
// the model wrote the list and reads the reply against it.
conversation.toggleUserInputOption("duckdb")
conversation.toggleUserInputOption("postgres")
conversation.submitUserInput()
#expect(await answering.value == .choice(selectedIDs: ["postgres", "duckdb"], custom: nil))
}
@Test
func aTypedAnswerComesBackAsCustomTextWithNoSelection() async {
let conversation = ConversationModel()
let answering = Task { await conversation.request(request(multiple: false)) }
await settle(conversation)
conversation.setUserInputCustomAnswer(" MySQL ")
conversation.submitUserInput()
// Trimmed, because trailing whitespace is not part of anyone's answer.
#expect(await answering.value == .choice(selectedIDs: [], custom: "MySQL"))
}
@Test
func cancellingReportsCancelledRatherThanAnEmptyAnswer() async {
let conversation = ConversationModel()
let answering = Task { await conversation.request(request(multiple: false)) }
await settle(conversation)
conversation.cancelUserInput()
#expect(await answering.value == .cancelled)
#expect(conversation.pendingUserInput == nil)
#expect(conversation.userInputSelection.isEmpty)
}
@Test
func submittingNothingIsRefusedRatherThanSentAsAnEmptyChoice() async {
let conversation = ConversationModel()
let answering = Task { await conversation.request(request(multiple: false)) }
await settle(conversation)
conversation.submitUserInput()
#expect(conversation.pendingUserInput != nil, "the request must still be waiting")
conversation.toggleUserInputOption("sqlite")
conversation.submitUserInput()
#expect(await answering.value == .choice(selectedIDs: ["sqlite"], custom: nil))
}
@Test
func aFreeFormQuestionStillAnswersAsPlainText() async {
let conversation = ConversationModel()
let answering = Task {
await conversation.request(AgentUserInputRequest(prompt: "Base URL?", kind: .text))
}
await settle(conversation)
conversation.userInputDraft = "https://api.example.com"
conversation.submitUserInput()
#expect(await answering.value == .text("https://api.example.com"))
}
// MARK: - Helpers
private func arguments(_ object: [String: AgentJSONValue]) throws -> Arguments {
try Arguments(
AgentToolInvocation(
runID: UUID(),
call: AgentToolCall(
id: "call-1", name: "request_user_input", arguments: .object(object)
)
)
)
}
private func request(multiple: Bool) -> AgentUserInputRequest {
AgentUserInputRequest(
prompt: "Which database?",
kind: .choice(
AgentUserInputChoice(
options: [
AgentUserInputOption(
id: "postgres", label: "Postgres",
detail: "Matches the existing schema.", isRecommended: true
),
AgentUserInputOption(id: "sqlite", label: "SQLite"),
AgentUserInputOption(id: "duckdb", label: "DuckDB"),
],
allowsMultipleAnswers: multiple
)
),
header: "Storage"
)
}
/// `request(_:)` publishes the pending question and then suspends on its
/// continuation. Waiting for that to be installed is what keeps these tests
/// from racing the answer in ahead of the question.
private func settle(_ conversation: ConversationModel) async {
for _ in 0..<100 where conversation.pendingUserInput == nil {
await Task.yield()
}
#expect(conversation.pendingUserInput != nil, "the request never became pending")
}
}