diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md index 915465d2..07be6f10 100644 --- a/docs/subsystems/gateway-routing.md +++ b/docs/subsystems/gateway-routing.md @@ -65,6 +65,7 @@ served by a plugin are in [Provider and plugin ownership](provider-plugins.md). - Group members are `provider/model[:effort]`. A member with a fixed effort is asked for that effort, bounded by the group's levels (#671). - An agent whose config can't carry an effort has one picked in magpie (`provider.AgentEffort`, settings' `agentEfforts`; Cursor Private Inference's row, #1003). It is asked for on that agent's requests to a model that reasons (`Provider.Thinks`), in place of the level the request sent, fitted to the model's levels and bounded by the group's. Order: a member's fixed effort, then a model's `:effort` suffix, then the turn's pick (a routing rule's), then the agent's pick, then the request's own. A model that doesn't reason is sent as asked; the default (none picked) leaves every request as it came. The pick beats the request's own because Cursor sends its own default as often as a real choice. Picking Default hands the choice back to Cursor. - Gemini's OpenAI-compatible API (AI Studio's, or a proxy in front of Gemini on this computer or the LAN: `geminiCompat`) is asked for the model's thoughts in `extra_body.google.thinking_config`. A 400 naming it (`refusesThinkingConfig`) marks the provider unfit for the field (`thinking_config`), and it is asked again with `reasoning_effort` alone. Gemini 3 can't stop thinking, so reasoning turned off goes at `minimal`, or at the model's lowest level where its levels are known and have no `minimal` (`geminiOffLevel`). Google's 400 for `minimal` (`geminiMinimalRefused`; gemini-3.8-flash's "Thinking level is unsupported: THINKING_LEVEL_MINIMAL") names thinking too, but isn't the field turned away: the model is marked as refusing reasoning off (`offRefused`) and asked again at its lowest level, and so from then on, still asked for its thoughts. Vertex AI's OpenAI-compatible API marks each streamed chunk of thoughts in the delta's `extra_content.google.thought`, and such a chunk is read as thinking. +- A request to an Anthropic endpoint that turns thinking off is sent as the model takes it (`thinkingOffAsTaken`, in `forwardOnce`, so relayed requests, count_tokens, magpie's own thinking-off and retries all get it; the model is read as the vendor names it, from the body, or from the path for Bedrock and Vertex). Claude Sonnet 5.5 refuses `thinking: {type: "disabled"}` and is sent `{type: "between_tools"}` alone, at effort `high` at most. Opus 5.5, Fable 5 and Mythos 5 refuse it at every effort and are sent no thinking, at effort `low` unless one was asked, with `classifierRoom` added to a reply limit below it, as Claude Code does. Opus 5 and Haiku 5.5 take it only up to `high`, so `xhigh` and `max` are asked as `high`. `between_tools` sent to any other Claude goes as `disabled`, or as none where that is refused too. Other vendors' models, older Claudes and later lines than these go as they came (#1454). A Sonnet 5.5 by a name these rules can't read, answering 400 that asks for `between_tools` (`asksBetweenTools`), is asked again with it, once; a refusal of `disabled` at any effort (`thinkingOffRefused`) is asked again with thinking left to the model, as `alwaysThinks` is. - A model that refuses a conversation ending with the assistant's message (`refusesPrefill`: OpenCode Go's Claude models over Messages, "This model does not support assistant message prefill"; Volcengine's Agent Plan, "…for a model that does not support prefill") is marked for that provider, model and API (`prefillRefused`, #1447); the provider's other models keep their prefill ([`prefill.go`](../../internal/gateway/prefill.go)). A Chat or Responses client's conversation ending with the assistant's message, which those APIs read as a turn already said (`Request.LastIsTurn`), is asked again with a user turn after it (`continueText`), translated (`forwardTranslated`) or relayed (`passthrough`, `userLastBody`), and so from then on. An Anthropic client's prefill is its own ask and gets the refusal as sent. magpie's own continuation of a cut reply (`continuation.go`) isn't asked of that model again; the try the model refused ends the reply with what cut it, not with the refusal. - A model's reply limit is published within the window beside it (#1438, `provider.OutputWithin`, `Entry.PublishedOutput`): `/v1/models`' `max_output_tokens` (`modelObject`), Cursor Private Inference's `capabilities.max_output_tokens` (`cursorLocalModel`), Muse Code's `limit.output` (`museModels`), the GUI's model and group rows (`providerInfo`, `providersState`), `magpie model output`, and every agent's file (`agent.maxTokens`, #338). The window and the output can come from different sources: Grok's backend lists grok-4.7's 256000 window, and models.dev fills its 500000 output (`catalog.Decorate` and `Provider.replyLimit` fill an output only where the vendor gives none). An unknown window or output is published as it is. The catalog keeps the unclamped `Entry.Output`, and a group's member is still asked for no more than that (`outputLimit`, `withMaxOutput`), so a request isn't lowered by the clamp. A limit the user set stays in their settings as they set it (`TestModelListsKeepOutputWithinTheWindow`). @@ -107,7 +108,7 @@ and Volcengine's polls and downloads as well. ```sh go test -tags nogui ./internal/gateway -run 'TestRemoteMagpie' -go test -tags nogui ./internal/gateway -run 'TestLoop|TestLongReplyThatDoesNotLoop|TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestModelNotInPlanRestsTheModelNotTheKey|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestWindowCapsHoldTheAccount|TestAccountCap|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestProtection|TestSearchedConversation|TestUnsearchedConversationUnchanged|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestClaudeAccountBackOnceItsWindowRenews|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused|TestGeminiThinkingOffAtItsLowest|TestGeminiThoughtsMarkedInExtraContent|TestClaudeSignInNotToldItsSignInFailed|TestSignInWriterReleasesA403|TestSignInBetaStaysWithMagpie|TestStructuredOutput|TestResponsesTextKeepsVerbosityBesideFormat|TestRefusedFormatToldInWords|RPM|TestQueued|TestCustomGeminiUpstream|TestWholeNums|TestGrokCalls|TestOnlyGrokCalls|TestGrokModel|TestGrokPluginCalls' +go test -tags nogui ./internal/gateway -run 'TestLoop|TestLongReplyThatDoesNotLoop|TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestModelNotInPlanRestsTheModelNotTheKey|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestWindowCapsHoldTheAccount|TestAccountCap|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestProtection|TestSearchedConversation|TestUnsearchedConversationUnchanged|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestClaudeAccountBackOnceItsWindowRenews|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused|TestGeminiThinkingOffAtItsLowest|TestThinkingOff|TestClaudeFiveFiveThinkingOffOnTheWire|TestGeminiThoughtsMarkedInExtraContent|TestClaudeSignInNotToldItsSignInFailed|TestSignInWriterReleasesA403|TestSignInBetaStaysWithMagpie|TestStructuredOutput|TestResponsesTextKeepsVerbosityBesideFormat|TestRefusedFormatToldInWords|RPM|TestQueued|TestCustomGeminiUpstream|TestWholeNums|TestGrokCalls|TestOnlyGrokCalls|TestGrokModel|TestGrokPluginCalls' go test -tags nogui ./internal/gateway -run 'TestCodexModelList|TestCodexModelsEtag' go test -tags nogui ./internal/provider -run 'TestGroup|TestCodexCreditsSwitch|TestRenewedAccountForgetsItsAllowance|TestDeclareSearch|TestNative|TestKeyAllowance|TestPlanKeyAllowance|TestClaudeAccountToldRenewed|TestPluginAccountToldRenewed|TestSkipsRedaction|TestSetRPM' ``` diff --git a/internal/gateway/adaptive_wire_test.go b/internal/gateway/adaptive_wire_test.go index a2de88cf..9fd3d1b5 100644 --- a/internal/gateway/adaptive_wire_test.go +++ b/internal/gateway/adaptive_wire_test.go @@ -91,9 +91,13 @@ func TestAdaptiveOnlyModelsNeverGetABudget(t *testing.T) { {"chat's effort, name mapped after", "/v1/chat/completions", `{"model":"relay/opus","reasoning_effort":"low","messages":[{"role":"user","content":"hi"}]}`, `{"type":"adaptive"}`, "low"}, - {"disabled stays", "/v1/messages", - `{"model":"relay/claude-opus-5.5","max_tokens":100,"thinking":{"type":"disabled"},"messages":[{"role":"user","content":"hi"}]}`, + {"disabled stays on an older Claude", "/v1/messages", + `{"model":"relay/claude-sonnet-4-5","max_tokens":100,"thinking":{"type":"disabled"},"messages":[{"role":"user","content":"hi"}]}`, `{"type":"disabled"}`, ""}, + // Opus 5.5 refuses disabled at every effort (#1454) + {"disabled left out for Opus 5.5", "/v1/messages", + `{"model":"relay/claude-opus-5.5","max_tokens":100,"thinking":{"type":"disabled"},"messages":[{"role":"user","content":"hi"}]}`, + `null`, "low"}, {"an older Claude keeps its budget", "/v1/messages", `{"model":"relay/claude-sonnet-4-5","max_tokens":32000,"thinking":{"type":"enabled","budget_tokens":10000},"messages":[{"role":"user","content":"hi"}]}`, `{"budget_tokens":10000,"type":"enabled"}`, ""}, diff --git a/internal/gateway/anthropic.go b/internal/gateway/anthropic.go index fc086b38..a5d6cc57 100644 --- a/internal/gateway/anthropic.go +++ b/internal/gateway/anthropic.go @@ -312,6 +312,182 @@ func adaptiveThinking(body []byte) []byte { return withFields(body, fields) } +// claudeLine finds a Claude model's family and version however a relay +// spells it: claude-sonnet-5-5, claude-sonnet-5.5, a relay's sonnet-5.5, +// Bedrock's anthropic.claude-sonnet-5-5-v1:0, Vertex's +// claude-sonnet-5-5@20261001, claude-fable-5-1. +var claudeLine = regexp.MustCompile(`(?:^|[^a-z0-9])(?:claude-)?(opus|sonnet|haiku|fable|mythos)-(\d{1,2})(?:[-.](\d{1,2}))?(?:[^0-9]|$)`) + +// thinkingOff is how a Claude model takes thinking turned off. +type thinkingOff int + +const ( + offUnknown thinkingOff = iota // not a model these rules know: as sent + offTaken // thinking.type=disabled at any effort + offUpToHigh // disabled at effort high or below: Opus 5, Haiku 5.5 + offBetweenTools // disabled refused, between_tools asked for: Sonnet 5.5 + offNever // thinking can't be turned off: Opus 5.5, Fable 5, Mythos 5 +) + +// thinkingOffOf is how model takes thinking turned off, from Anthropic's +// own rules: Sonnet 5.5 answers disabled with 400 "To turn thinking off on +// this model, send "thinking": {"type": "between_tools"}…" and takes +// between_tools at effort high or below; Opus 5.5 and the Fable and +// Mythos models refuse disabled at every effort, so thinking is left out +// (they think adaptively); Opus 5 and Haiku 5.5 refuse it at xhigh and +// max. Every other Claude takes it as sent, and between_tools on none of +// them. A model of a later line than these is offUnknown, left as sent. +func thinkingOffOf(model string) thinkingOff { + m := claudeLine.FindStringSubmatch(strings.ToLower(model)) + if m == nil { + return offUnknown + } + major, _ := strconv.Atoi(m[2]) + minor, _ := strconv.Atoi(m[3]) + family := m[1] + switch { + case family == "fable" || family == "mythos": + if major == 5 { + return offNever + } + case major < 5: + return offTaken + case major == 5 && minor == 0: + if family == "opus" { + return offUpToHigh + } + return offTaken // Sonnet 5, Haiku 5 + case major == 5 && minor == 5: + switch family { + case "sonnet": + return offBetweenTools + case "opus": + return offNever + case "haiku": + return offUpToHigh + } + } + return offUnknown +} + +// thinkingOffAsTaken is an Anthropic request that turns thinking off, as +// the model it is sent to takes that (thinkingOffOf). Claude Code turns +// thinking off for auto mode's classifier and its side queries whenever +// it can't tell the model refuses it — a model reached through a gateway +// is one — and magpie turns it off itself when fitting an effort of none +// or a budget that doesn't fit; claude-sonnet-5-5 answered every one of +// them 400 (#1454). Sonnet 5.5 is sent between_tools, alone in thinking, +// with an effort above high asked as high; Opus 5.5, Fable and Mythos are +// sent no thinking, at effort low unless one is asked, with the room a +// thinking model needs on top of a short reply, as Claude Code gives it +// (classifierRoom); Opus 5 and Haiku 5.5 are asked effort high in place +// of xhigh or max. between_tools sent to a model that doesn't take it goes +// as disabled, or as no thinking where disabled is refused too. model is +// the vendor's name of it: the body's, or, for Bedrock and Vertex, whose +// body names none, the request path's. Any other request, other vendors' +// models' and older Claudes' included, goes byte for byte as it came. +func thinkingOffAsTaken(body []byte, path string) []byte { + t := gjson.GetBytes(body, "thinking.type").String() + if t != "disabled" && t != "between_tools" { + return body + } + model := gjson.GetBytes(body, "model").String() + if model == "" { + model = path + } + rule := thinkingOffOf(model) + effort := gjson.GetBytes(body, "output_config.effort").String() + aboveHigh := effort == "xhigh" || effort == "max" + var thinking map[string]any // nil: left out + effortTo := "" + switch rule { + case offTaken: + if t == "disabled" { + return body + } + thinking = map[string]any{"type": "disabled"} + case offUpToHigh: + if t == "disabled" && !aboveHigh { + return body + } + thinking = map[string]any{"type": "disabled"} + if aboveHigh { + effortTo = "high" + } + case offBetweenTools: + if th := gjson.GetBytes(body, "thinking"); t == "between_tools" && len(th.Map()) == 1 && !aboveHigh { + return body + } + // between_tools takes no other field (display, budget_tokens) + thinking = map[string]any{"type": "between_tools"} + if aboveHigh { + effortTo = "high" + } + case offNever: + if effort == "" { + effortTo = "low" + } + default: + return body + } + dec := json.NewDecoder(bytes.NewReader(body)) + dec.UseNumber() + var q map[string]any + if dec.Decode(&q) != nil { + return body + } + if thinking != nil { + q["thinking"] = thinking + } else { + delete(q, "thinking") + if n := gjson.GetBytes(body, "max_tokens").Int(); n > 0 && n < classifierRoom { + q["max_tokens"] = n + classifierRoom + } + } + if effortTo != "" { + oc, _ := q["output_config"].(map[string]any) + if oc == nil { + oc = map[string]any{} + } + oc["effort"] = effortTo + q["output_config"] = oc + } + var out bytes.Buffer + enc := json.NewEncoder(&out) + enc.SetEscapeHTML(false) // <, > and & as the agent wrote them + if enc.Encode(q) != nil { + return body + } + return bytes.TrimSuffix(out.Bytes(), []byte("\n")) +} + +// asksBetweenTools is Sonnet 5.5's answer to thinking turned off, asking +// for between_tools: "To turn thinking off on this model, send "thinking": +// {"type": "between_tools"} instead of {"type": "disabled"}" as relays +// pass it on (#1454), or the API's "Use "thinking.type.between_tools" for +// the lowest thinking setting". +var asksBetweenTools = regexp.MustCompile(`(?i)between_tools\W+instead of|use\W+thinking\.type\.between_tools`) + +// thinkingOffRefused is a Claude refusing thinking turned off at any +// effort, as Opus 5.5 does: ""thinking.type.disabled" is not supported for +// this model. Use "thinking.type.adaptive"…". +var thinkingOffRefused = regexp.MustCompile(`(?i)thinking\.type\.disabled\W+is not supported`) + +// withBetweenTools is body with its thinking turned off as Sonnet 5.5 +// asks it, at effort high at most; false when body doesn't turn it off. +func withBetweenTools(body []byte) ([]byte, bool) { + if gjson.GetBytes(body, "thinking.type").String() != "disabled" { + return nil, false + } + fields := map[string]any{"thinking": map[string]any{"type": "between_tools"}} + if e := gjson.GetBytes(body, "output_config.effort").String(); e == "xhigh" || e == "max" { + oc, _ := gjson.GetBytes(body, "output_config").Value().(map[string]any) + oc["effort"] = "high" + fields["output_config"] = oc + } + return withFields(body, fields), true +} + // AdaptiveThinking is adaptiveOnly for agents told how to ask a model: a // Claude that takes thinking.type=adaptive and an effort, never a budget. func AdaptiveThinking(model string) bool { return adaptiveOnly(model) } diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go index e50ba632..e17d01cb 100644 --- a/internal/gateway/gateway.go +++ b/internal/gateway/gateway.go @@ -2880,6 +2880,7 @@ func (s *Server) forwardOnce(ctx context.Context, p provider.Provider, to provid // what every path to an Anthropic endpoint sends, relayed or // built, with the model named as the vendor names it body = adaptiveThinking(body) + body = thinkingOffAsTaken(body, path) asked := askedBetas(in) if gjson.GetBytes(body, "speed").String() == "fast" && provider.HostOf(p.Base(to)) == "api.anthropic.com" { asked = append(slices.Clone(asked), claudeFastBeta) // a group's member sent fast @@ -3308,7 +3309,13 @@ func (s *Server) passthrough(w http.ResponseWriter, r *http.Request, p provider. b, _ := io.ReadAll(io.LimitReader(res.Body, 1<<20)) res.Body.Close() res.Body = io.NopCloser(bytes.NewReader(b)) - if nb, ok := withoutThinkingOff(body); ok && (alwaysThinks.Match(b) || mandatoryReasoning.Match(b)) { + // Sonnet 5.5 by a name thinkingOffAsTaken doesn't know asks for + // between_tools in place of disabled (#1454) + if nb, ok := withBetweenTools(body); ok && asksBetweenTools.Match(b) { + if res, err = s.forward(r.Context(), p, proto, path, p.Prepare(nb), r.Header); err != nil { + return writeError(w, proto, 502, p.Name+": "+err.Error()), err.Error(), true + } + } else if nb, ok := withoutThinkingOff(body); ok && (alwaysThinks.Match(b) || mandatoryReasoning.Match(b) || thinkingOffRefused.Match(b)) { if res, err = s.forward(r.Context(), p, proto, path, p.Prepare(nb), r.Header); err != nil { return writeError(w, proto, 502, p.Name+": "+err.Error()), err.Error(), true } diff --git a/internal/gateway/thinking_off_test.go b/internal/gateway/thinking_off_test.go new file mode 100644 index 00000000..1bec1dfe --- /dev/null +++ b/internal/gateway/thinking_off_test.go @@ -0,0 +1,230 @@ +package gateway + +import ( + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + + "github.com/tidwall/gjson" + "github.com/yetone/magpie/internal/provider" +) + +// #1454: Claude Code's auto mode classifier, through magpie, asked +// claude-sonnet-5-5 with thinking turned off and got 400 "To turn thinking +// off on this model, send "thinking": {"type": "between_tools"} instead of +// {"type": "disabled"}". Each model that refuses disabled is sent what its +// vendor asks for; every other request goes byte for byte as it came. +func TestThinkingOffAsTheModelTakesIt(t *testing.T) { + for _, c := range []struct { + name, path, body string + thinking string // the thinking sent, "" for none + effort string + maxTokens int64 + }{ + {"Sonnet 5.5 is sent between_tools", "/v1/messages", + `{"model":"claude-sonnet-5-5","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`, + `{"type":"between_tools"}`, "", 64}, + {"Sonnet 5.5 at xhigh is asked high", "/v1/messages", + `{"model":"claude-sonnet-5-5","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"xhigh"},"messages":[]}`, + `{"type":"between_tools"}`, "high", 64}, + {"Sonnet 5.5's between_tools takes no other field", "/v1/messages", + `{"model":"claude-sonnet-5-5","max_tokens":64,"thinking":{"type":"between_tools","display":"omitted"},"messages":[]}`, + `{"type":"between_tools"}`, "", 64}, + {"a relay's claude-sonnet-5.5", "/v1/messages", + `{"model":"claude-sonnet-5.5","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`, + `{"type":"between_tools"}`, "", 64}, + {"a relay's sonnet-5.5", "/v1/messages", + `{"model":"sonnet-5.5","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`, + `{"type":"between_tools"}`, "", 64}, + {"Bedrock names the model in the path", "/model/anthropic.claude-sonnet-5-5-v1:0/invoke", + `{"anthropic_version":"bedrock-2023-05-31","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`, + `{"type":"between_tools"}`, "", 64}, + {"Opus 5.5 is sent no thinking, at effort low, with room", "/v1/messages", + `{"model":"claude-opus-5-5","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`, + "", "low", 64 + classifierRoom}, + {"Opus 5.5 keeps an effort asked", "/v1/messages", + `{"model":"claude-opus-5-5","max_tokens":8000,"thinking":{"type":"disabled"},"output_config":{"effort":"high"},"messages":[]}`, + "", "high", 8000}, + {"Fable 5.1 is sent no thinking", "/v1/messages", + `{"model":"claude-fable-5-1","max_tokens":256,"thinking":{"type":"disabled"},"messages":[]}`, + "", "low", 256 + classifierRoom}, + {"Mythos 5.1 is sent no thinking", "/v1/messages", + `{"model":"claude-mythos-5-1","max_tokens":4096,"thinking":{"type":"disabled"},"messages":[]}`, + "", "low", 4096}, + {"Opus 5 off at max is asked high", "/v1/messages", + `{"model":"claude-opus-5","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"max"},"messages":[]}`, + `{"type":"disabled"}`, "high", 64}, + {"Haiku 5.5 off at xhigh is asked high", "/v1/messages", + `{"model":"claude-haiku-5-5","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"xhigh"},"messages":[]}`, + `{"type":"disabled"}`, "high", 64}, + {"between_tools to Sonnet 5 goes as disabled", "/v1/messages", + `{"model":"claude-sonnet-5","max_tokens":64,"thinking":{"type":"between_tools"},"messages":[]}`, + `{"type":"disabled"}`, "", 64}, + {"between_tools to Opus 5.5 goes as none", "/v1/messages", + `{"model":"claude-opus-5-5","max_tokens":64,"thinking":{"type":"between_tools"},"messages":[]}`, + "", "low", 64 + classifierRoom}, + } { + t.Run(c.name, func(t *testing.T) { + got := thinkingOffAsTaken([]byte(c.body), c.path) + if !json.Valid(got) { + t.Fatalf("not JSON: %s", got) + } + th := gjson.GetBytes(got, "thinking").Raw + if th != c.thinking { + t.Errorf("thinking = %s, want %s (sent %s)", th, c.thinking, got) + } + if e := gjson.GetBytes(got, "output_config.effort").String(); e != c.effort { + t.Errorf("effort = %q, want %q (sent %s)", e, c.effort, got) + } + if n := gjson.GetBytes(got, "max_tokens").Int(); n != c.maxTokens { + t.Errorf("max_tokens = %d, want %d", n, c.maxTokens) + } + if gjson.GetBytes(got, "messages").Raw != "[]" { + t.Errorf("the rest of the request changed: %s", got) + } + }) + } +} + +// Another vendor's model and every Claude that takes thinking turned off +// get the request byte for byte as it came. +func TestThinkingOffLeavesOtherModelsAlone(t *testing.T) { + for _, body := range []string{ + `{"model":"deepseek-v4-pro","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`, + `{"model":"glm-5.3","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"max"},"messages":[]}`, + `{"model":"kimi-k2.5","max_tokens":64,"thinking":{"type":"between_tools"},"messages":[]}`, + `{"model":"claude-sonnet-5","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"max"},"messages":[]}`, + `{"model":"claude-opus-5","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"high"},"messages":[]}`, + `{"model":"claude-haiku-5-5","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`, + `{"model":"claude-opus-4-8","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"max"},"messages":[]}`, + `{"model":"claude-sonnet-4-6","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`, + `{"model":"claude-haiku-4-5-20251001","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`, + `{"model":"claude-3-7-sonnet-20250219","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`, + `{"model":"claude-sonnet-5-20261001","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`, + // Sonnet 5.5 thinking, or not told: nothing to turn off + `{"model":"claude-sonnet-5-5","max_tokens":64,"thinking":{"type":"adaptive"},"output_config":{"effort":"max"},"messages":[]}`, + `{"model":"claude-sonnet-5-5","max_tokens":64,"messages":[]}`, + `{"model":"claude-sonnet-5-5","max_tokens":64,"thinking":{"type":"between_tools"},"output_config":{"effort":"high"},"messages":[]}`, + // a later line than these rules know + `{"model":"claude-sonnet-6","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`, + } { + if got := thinkingOffAsTaken([]byte(body), "/v1/messages"); string(got) != body { + t.Errorf("%s\n sent as %s", body, got) + } + } +} + +// claudeFiveVendor answers as Anthropic does for the 5.5 models: Sonnet +// 5.5 refuses disabled in the words the reporter got through a relay, and +// between_tools above high; Opus 5.5 refuses disabled; every other model +// refuses between_tools. +type claudeFiveVendor struct { + mu sync.Mutex + paths []string + sent []string +} + +func (v *claudeFiveVendor) ServeHTTP(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + v.mu.Lock() + v.paths = append(v.paths, r.URL.Path) + v.sent = append(v.sent, string(b)) + v.mu.Unlock() + model := gjson.GetBytes(b, "model").String() + t := gjson.GetBytes(b, "thinking.type").String() + effort := gjson.GetBytes(b, "output_config.effort").String() + refuse := func(msg string) { + w.WriteHeader(400) + io.WriteString(w, `{"error":{"type":"invalid_request_error","message":`+msg+`},"type":"error"}`) + } + sonnet55 := strings.Contains(model, "sonnet-5-5") || strings.Contains(model, "sonnet-5.5") || strings.Contains(model, "s55") + switch { + case sonnet55 && t == "disabled": + refuse(`"To turn thinking off on this model, send \"thinking\": {\"type\": \"between_tools\"} instead of {\"type\": \"disabled\"}. The model does not think before responding. The short updates it writes between tool calls come back as thinking blocks. (tid: 2026101002490321168731647902503)"`) + return + case sonnet55 && t == "between_tools" && (effort == "xhigh" || effort == "max"): + refuse(`"output_config.effort 'xhigh' is not supported when thinking is disabled on this model. Use effort 'high' or below."`) + return + case strings.Contains(model, "opus-5-5") && t == "disabled": + refuse(`"\"thinking.type.disabled\" is not supported for this model. Use \"thinking.type.adaptive\" and \"output_config.effort\" to control thinking behavior."`) + return + case !sonnet55 && t == "between_tools": + refuse(`"thinking.type.between_tools is not supported for this model."`) + return + } + if gjson.GetBytes(b, "stream").Bool() { + w.Header().Set("Content-Type", "text/event-stream") + io.WriteString(w, "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"id\":\"msg_1\",\"type\":\"message\",\"role\":\"assistant\",\"model\":\""+model+"\",\"content\":[],\"usage\":{\"input_tokens\":1,\"output_tokens\":0}}}\n\n"+ + "event: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"}}\n\n"+ + "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"ok\"}}\n\n"+ + "event: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0}\n\n"+ + "event: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\"},\"usage\":{\"output_tokens\":1}}\n\n"+ + "event: message_stop\ndata: {\"type\":\"message_stop\"}\n\n") + return + } + w.Header().Set("Content-Type", "application/json") + if strings.HasSuffix(r.URL.Path, "/count_tokens") { + io.WriteString(w, `{"input_tokens":7}`) + return + } + io.WriteString(w, `{"id":"msg_1","type":"message","role":"assistant","model":"`+model+`","content":[{"type":"text","text":"no"}],"stop_reason":"end_turn","usage":{"input_tokens":1,"output_tokens":1}}`) +} + +func (v *claudeFiveVendor) last() string { + v.mu.Lock() + defer v.mu.Unlock() + return v.sent[len(v.sent)-1] +} + +// Every route to an Anthropic endpoint that turns thinking off — the +// classifier relayed as Claude Code sends it, count_tokens, and magpie's +// own thinking-off (an effort of none fitted, a budget that doesn't fit), +// all made before forwardOnce — reaches the 5.5 models as they take it, and a Sonnet 5.5 by a name magpie can't read is asked again with +// between_tools, once. +func TestClaudeFiveFiveThinkingOffOnTheWire(t *testing.T) { + fresh(t) + up := &claudeFiveVendor{} + srv := httptest.NewServer(up) + t.Cleanup(srv.Close) + if err := provider.Save(provider.Provider{ID: "relay", Name: "Relay", Key: "k", Anthropic: srv.URL, + Models: []string{"claude-sonnet-5-5", "claude-opus-5-5", "claude-sonnet-5", "s55"}}); err != nil { + t.Fatal(err) + } + classifier := `{"model":"relay/%s","max_tokens":64,"temperature":0,"stop_sequences":[""],"thinking":{"type":"disabled"},"system":[{"type":"text","text":"Answer yes or no."}],"messages":[{"role":"user","content":"ls"}]}` + for _, c := range []struct { + name, path, body string + calls int + thinking string + }{ + {"classifier to Sonnet 5.5", "/v1/messages", strings.ReplaceAll(classifier, "%s", "claude-sonnet-5-5"), 1, `{"type":"between_tools"}`}, + {"classifier to Opus 5.5", "/v1/messages", strings.ReplaceAll(classifier, "%s", "claude-opus-5-5"), 1, ""}, + {"classifier to Sonnet 5 as it came", "/v1/messages", strings.ReplaceAll(classifier, "%s", "claude-sonnet-5"), 1, `{"type":"disabled"}`}, + {"count_tokens to Sonnet 5.5", "/v1/messages/count_tokens", + `{"model":"relay/claude-sonnet-5-5","thinking":{"type":"disabled"},"messages":[{"role":"user","content":"hi"}]}`, 1, `{"type":"between_tools"}`}, + {"a name magpie can't read is asked again", "/v1/messages", strings.ReplaceAll(classifier, "%s", "s55"), 2, `{"type":"between_tools"}`}, + } { + t.Run(c.name, func(t *testing.T) { + up.mu.Lock() + before := len(up.sent) + up.mu.Unlock() + code, body := post(t, c.path, c.body) + if code != 200 { + t.Fatalf("status %d: %s", code, body) + } + up.mu.Lock() + calls := len(up.sent) - before + up.mu.Unlock() + if calls != c.calls { + t.Errorf("%d calls upstream, want %d", calls, c.calls) + } + got := up.last() + if th := gjson.Get(got, "thinking").Raw; th != c.thinking { + t.Errorf("thinking = %s, want %s: %s", th, c.thinking, got) + } + }) + } +}