gateway: Claude Sonnet 5.5 is sent thinking between_tools in place of disabled, and Opus 5.5, Fable and Mythos no thinking, so Claude Code's auto mode classifier no longer gets a 400 (#1454)
911Liam's auto mode, on claude-sonnet-5.5 through a relay, got 400 "To
turn thinking off on this model, send "thinking": {"type":
"between_tools"} instead of {"type": "disabled"}". Claude Code turns
thinking off for its classifier and side queries unless it knows the
model refuses that, and a model reached through a gateway can fall
outside what it knows. magpie forwarded the disabled as it came, and
turns thinking off itself too (an effort of none fitted, a budget that
doesn't fit, thinkingOffUnlessAsked for other vendors' models).
thinkingOffAsTaken, in forwardOnce next to adaptiveThinking, so that
relayed requests, count_tokens, magpie's own thinking-off and retries
all go through it, follows Anthropic's rules for each model. It reads
the model's name as the vendor gives it, from the body, or from the path
for Bedrock and Vertex.
- Sonnet 5.5: between_tools with no other field, and xhigh or max asked
as high (between_tools is a 400 above high).
- Opus 5.5, Fable 5.x and Mythos 5.x refuse disabled at every effort.
Thinking is left out, effort is low unless one was asked, and
classifierRoom is added to a reply limit below it, as Claude Code
itself does for these models.
- Opus 5 and Haiku 5.5: disabled is kept, with xhigh or max asked as
high.
- between_tools sent to any other Claude goes as disabled, or with no
thinking where disabled is refused too.
Other vendors' models, Sonnet 5, Opus 4.8 and older, and lines later than
these go byte for byte as they came. A Sonnet 5.5 under a name these
rules can't read is asked again with between_tools, once, on the 400
that asks for it (asksBetweenTools). A refusal of disabled at any effort
(thinkingOffRefused) is asked again with thinking left to the model,
beside alwaysThinks.
TestAdaptiveOnlyModelsNeverGetABudget's "disabled stays" case pinned
disabled for claude-opus-5.5, which the vendor refuses. It now checks an
older Claude keeps disabled and Opus 5.5 is sent none.
Verified:
- Tests:
- TestThinkingOffAsTheModelTakesIt and
TestThinkingOffLeavesOtherModelsAlone (deepseek, glm-5.3, kimi,
Sonnet 5, Opus 5 at high, Haiku 5.5, Opus 4.8 at max, Sonnet 4.6,
Haiku 4.5, Claude 3.7, dated Sonnet 5 and Sonnet 6 unchanged).
- TestClaudeFiveFiveThinkingOffOnTheWire: a fake vendor answers with
the reporter's literal 400, covering the classifier body,
count_tokens and the retry.
- Without the forwardOnce call, the wire test's Sonnet 5.5, Opus 5.5 and
count_tokens cases and the Opus 5.5 adaptive case FAIL. Without the
retry, its unreadable-name case FAILs.
- Live, through this build's `magpie serve` under a sandbox HOME with
the aihubmix relay key, in the classifier's shape (max_tokens 64,
temperature 0, stop </block>, thinking disabled):
- claude-sonnet-5-5 answers 200 at default and xhigh effort (sent
straight to aihubmix with disabled it gives the reporter's 400);
- count_tokens 200;
- claude-opus-5-5 200;
- claude-haiku-5-5 at max 200;
- claude-sonnet-5 200.
- gofmt -l, GOOS=linux and GOOS=windows go vet -tags nogui
./internal/gateway, and go test -tags nogui ./internal/gateway under a
temp HOME with pinned caches.
This commit is contained in:
yetone committed
1 parent
16539c2ea9
commit
a9cb0ef988
5 files changed
+422
-4
No files matched your search
@@ -65,6 +65,7 @@ served by a plugin are in [Provider and plugin ownership](provider-plugins.md).
|
||||
- Group members are `provider/model[:effort]`. A member with a fixed effort is asked for that effort, bounded by the group's levels (#671).
|
||||
- An agent whose config can't carry an effort has one picked in magpie (`provider.AgentEffort`, settings' `agentEfforts`; Cursor Private Inference's row, #1003). It is asked for on that agent's requests to a model that reasons (`Provider.Thinks`), in place of the level the request sent, fitted to the model's levels and bounded by the group's. Order: a member's fixed effort, then a model's `:effort` suffix, then the turn's pick (a routing rule's), then the agent's pick, then the request's own. A model that doesn't reason is sent as asked; the default (none picked) leaves every request as it came. The pick beats the request's own because Cursor sends its own default as often as a real choice. Picking Default hands the choice back to Cursor.
|
||||
- Gemini's OpenAI-compatible API (AI Studio's, or a proxy in front of Gemini on this computer or the LAN: `geminiCompat`) is asked for the model's thoughts in `extra_body.google.thinking_config`. A 400 naming it (`refusesThinkingConfig`) marks the provider unfit for the field (`thinking_config`), and it is asked again with `reasoning_effort` alone. Gemini 3 can't stop thinking, so reasoning turned off goes at `minimal`, or at the model's lowest level where its levels are known and have no `minimal` (`geminiOffLevel`). Google's 400 for `minimal` (`geminiMinimalRefused`; gemini-3.8-flash's "Thinking level is unsupported: THINKING_LEVEL_MINIMAL") names thinking too, but isn't the field turned away: the model is marked as refusing reasoning off (`offRefused`) and asked again at its lowest level, and so from then on, still asked for its thoughts. Vertex AI's OpenAI-compatible API marks each streamed chunk of thoughts in the delta's `extra_content.google.thought`, and such a chunk is read as thinking.
|
||||
- A request to an Anthropic endpoint that turns thinking off is sent as the model takes it (`thinkingOffAsTaken`, in `forwardOnce`, so relayed requests, count_tokens, magpie's own thinking-off and retries all get it; the model is read as the vendor names it, from the body, or from the path for Bedrock and Vertex). Claude Sonnet 5.5 refuses `thinking: {type: "disabled"}` and is sent `{type: "between_tools"}` alone, at effort `high` at most. Opus 5.5, Fable 5 and Mythos 5 refuse it at every effort and are sent no thinking, at effort `low` unless one was asked, with `classifierRoom` added to a reply limit below it, as Claude Code does. Opus 5 and Haiku 5.5 take it only up to `high`, so `xhigh` and `max` are asked as `high`. `between_tools` sent to any other Claude goes as `disabled`, or as none where that is refused too. Other vendors' models, older Claudes and later lines than these go as they came (#1454). A Sonnet 5.5 by a name these rules can't read, answering 400 that asks for `between_tools` (`asksBetweenTools`), is asked again with it, once; a refusal of `disabled` at any effort (`thinkingOffRefused`) is asked again with thinking left to the model, as `alwaysThinks` is.
|
||||
- A model that refuses a conversation ending with the assistant's message (`refusesPrefill`: OpenCode Go's Claude models over Messages, "This model does not support assistant message prefill"; Volcengine's Agent Plan, "…for a model that does not support prefill") is marked for that provider, model and API (`prefillRefused`, #1447); the provider's other models keep their prefill ([`prefill.go`](../../internal/gateway/prefill.go)). A Chat or Responses client's conversation ending with the assistant's message, which those APIs read as a turn already said (`Request.LastIsTurn`), is asked again with a user turn after it (`continueText`), translated (`forwardTranslated`) or relayed (`passthrough`, `userLastBody`), and so from then on. An Anthropic client's prefill is its own ask and gets the refusal as sent. magpie's own continuation of a cut reply (`continuation.go`) isn't asked of that model again; the try the model refused ends the reply with what cut it, not with the refusal.
|
||||
- A model's reply limit is published within the window beside it (#1438, `provider.OutputWithin`, `Entry.PublishedOutput`): `/v1/models`' `max_output_tokens` (`modelObject`), Cursor Private Inference's `capabilities.max_output_tokens` (`cursorLocalModel`), Muse Code's `limit.output` (`museModels`), the GUI's model and group rows (`providerInfo`, `providersState`), `magpie model output`, and every agent's file (`agent.maxTokens`, #338). The window and the output can come from different sources: Grok's backend lists grok-4.7's 256000 window, and models.dev fills its 500000 output (`catalog.Decorate` and `Provider.replyLimit` fill an output only where the vendor gives none). An unknown window or output is published as it is. The catalog keeps the unclamped `Entry.Output`, and a group's member is still asked for no more than that (`outputLimit`, `withMaxOutput`), so a request isn't lowered by the clamp. A limit the user set stays in their settings as they set it (`TestModelListsKeepOutputWithinTheWindow`).
|
||||
|
||||
@@ -107,7 +108,7 @@ and Volcengine's polls and downloads as well.
|
||||
|
||||
```sh
|
||||
go test -tags nogui ./internal/gateway -run 'TestRemoteMagpie'
|
||||
go test -tags nogui ./internal/gateway -run 'TestLoop|TestLongReplyThatDoesNotLoop|TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestModelNotInPlanRestsTheModelNotTheKey|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestWindowCapsHoldTheAccount|TestAccountCap|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestProtection|TestSearchedConversation|TestUnsearchedConversationUnchanged|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestClaudeAccountBackOnceItsWindowRenews|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused|TestGeminiThinkingOffAtItsLowest|TestGeminiThoughtsMarkedInExtraContent|TestClaudeSignInNotToldItsSignInFailed|TestSignInWriterReleasesA403|TestSignInBetaStaysWithMagpie|TestStructuredOutput|TestResponsesTextKeepsVerbosityBesideFormat|TestRefusedFormatToldInWords|RPM|TestQueued|TestCustomGeminiUpstream|TestWholeNums|TestGrokCalls|TestOnlyGrokCalls|TestGrokModel|TestGrokPluginCalls'
|
||||
go test -tags nogui ./internal/gateway -run 'TestLoop|TestLongReplyThatDoesNotLoop|TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestModelNotInPlanRestsTheModelNotTheKey|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestWindowCapsHoldTheAccount|TestAccountCap|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestProtection|TestSearchedConversation|TestUnsearchedConversationUnchanged|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestClaudeAccountBackOnceItsWindowRenews|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused|TestGeminiThinkingOffAtItsLowest|TestThinkingOff|TestClaudeFiveFiveThinkingOffOnTheWire|TestGeminiThoughtsMarkedInExtraContent|TestClaudeSignInNotToldItsSignInFailed|TestSignInWriterReleasesA403|TestSignInBetaStaysWithMagpie|TestStructuredOutput|TestResponsesTextKeepsVerbosityBesideFormat|TestRefusedFormatToldInWords|RPM|TestQueued|TestCustomGeminiUpstream|TestWholeNums|TestGrokCalls|TestOnlyGrokCalls|TestGrokModel|TestGrokPluginCalls'
|
||||
go test -tags nogui ./internal/gateway -run 'TestCodexModelList|TestCodexModelsEtag'
|
||||
go test -tags nogui ./internal/provider -run 'TestGroup|TestCodexCreditsSwitch|TestRenewedAccountForgetsItsAllowance|TestDeclareSearch|TestNative|TestKeyAllowance|TestPlanKeyAllowance|TestClaudeAccountToldRenewed|TestPluginAccountToldRenewed|TestSkipsRedaction|TestSetRPM'
|
||||
```
|
||||
|
||||
@@ -91,9 +91,13 @@ func TestAdaptiveOnlyModelsNeverGetABudget(t *testing.T) {
|
||||
{"chat's effort, name mapped after", "/v1/chat/completions",
|
||||
`{"model":"relay/opus","reasoning_effort":"low","messages":[{"role":"user","content":"hi"}]}`,
|
||||
`{"type":"adaptive"}`, "low"},
|
||||
{"disabled stays", "/v1/messages",
|
||||
`{"model":"relay/claude-opus-5.5","max_tokens":100,"thinking":{"type":"disabled"},"messages":[{"role":"user","content":"hi"}]}`,
|
||||
{"disabled stays on an older Claude", "/v1/messages",
|
||||
`{"model":"relay/claude-sonnet-4-5","max_tokens":100,"thinking":{"type":"disabled"},"messages":[{"role":"user","content":"hi"}]}`,
|
||||
`{"type":"disabled"}`, ""},
|
||||
// Opus 5.5 refuses disabled at every effort (#1454)
|
||||
{"disabled left out for Opus 5.5", "/v1/messages",
|
||||
`{"model":"relay/claude-opus-5.5","max_tokens":100,"thinking":{"type":"disabled"},"messages":[{"role":"user","content":"hi"}]}`,
|
||||
`null`, "low"},
|
||||
{"an older Claude keeps its budget", "/v1/messages",
|
||||
`{"model":"relay/claude-sonnet-4-5","max_tokens":32000,"thinking":{"type":"enabled","budget_tokens":10000},"messages":[{"role":"user","content":"hi"}]}`,
|
||||
`{"budget_tokens":10000,"type":"enabled"}`, ""},
|
||||
|
||||
@@ -312,6 +312,182 @@ func adaptiveThinking(body []byte) []byte {
|
||||
return withFields(body, fields)
|
||||
}
|
||||
|
||||
// claudeLine finds a Claude model's family and version however a relay
|
||||
// spells it: claude-sonnet-5-5, claude-sonnet-5.5, a relay's sonnet-5.5,
|
||||
// Bedrock's anthropic.claude-sonnet-5-5-v1:0, Vertex's
|
||||
// claude-sonnet-5-5@20261001, claude-fable-5-1.
|
||||
var claudeLine = regexp.MustCompile(`(?:^|[^a-z0-9])(?:claude-)?(opus|sonnet|haiku|fable|mythos)-(\d{1,2})(?:[-.](\d{1,2}))?(?:[^0-9]|$)`)
|
||||
|
||||
// thinkingOff is how a Claude model takes thinking turned off.
|
||||
type thinkingOff int
|
||||
|
||||
const (
|
||||
offUnknown thinkingOff = iota // not a model these rules know: as sent
|
||||
offTaken // thinking.type=disabled at any effort
|
||||
offUpToHigh // disabled at effort high or below: Opus 5, Haiku 5.5
|
||||
offBetweenTools // disabled refused, between_tools asked for: Sonnet 5.5
|
||||
offNever // thinking can't be turned off: Opus 5.5, Fable 5, Mythos 5
|
||||
)
|
||||
|
||||
// thinkingOffOf is how model takes thinking turned off, from Anthropic's
|
||||
// own rules: Sonnet 5.5 answers disabled with 400 "To turn thinking off on
|
||||
// this model, send "thinking": {"type": "between_tools"}…" and takes
|
||||
// between_tools at effort high or below; Opus 5.5 and the Fable and
|
||||
// Mythos models refuse disabled at every effort, so thinking is left out
|
||||
// (they think adaptively); Opus 5 and Haiku 5.5 refuse it at xhigh and
|
||||
// max. Every other Claude takes it as sent, and between_tools on none of
|
||||
// them. A model of a later line than these is offUnknown, left as sent.
|
||||
func thinkingOffOf(model string) thinkingOff {
|
||||
m := claudeLine.FindStringSubmatch(strings.ToLower(model))
|
||||
if m == nil {
|
||||
return offUnknown
|
||||
}
|
||||
major, _ := strconv.Atoi(m[2])
|
||||
minor, _ := strconv.Atoi(m[3])
|
||||
family := m[1]
|
||||
switch {
|
||||
case family == "fable" || family == "mythos":
|
||||
if major == 5 {
|
||||
return offNever
|
||||
}
|
||||
case major < 5:
|
||||
return offTaken
|
||||
case major == 5 && minor == 0:
|
||||
if family == "opus" {
|
||||
return offUpToHigh
|
||||
}
|
||||
return offTaken // Sonnet 5, Haiku 5
|
||||
case major == 5 && minor == 5:
|
||||
switch family {
|
||||
case "sonnet":
|
||||
return offBetweenTools
|
||||
case "opus":
|
||||
return offNever
|
||||
case "haiku":
|
||||
return offUpToHigh
|
||||
}
|
||||
}
|
||||
return offUnknown
|
||||
}
|
||||
|
||||
// thinkingOffAsTaken is an Anthropic request that turns thinking off, as
|
||||
// the model it is sent to takes that (thinkingOffOf). Claude Code turns
|
||||
// thinking off for auto mode's classifier and its side queries whenever
|
||||
// it can't tell the model refuses it — a model reached through a gateway
|
||||
// is one — and magpie turns it off itself when fitting an effort of none
|
||||
// or a budget that doesn't fit; claude-sonnet-5-5 answered every one of
|
||||
// them 400 (#1454). Sonnet 5.5 is sent between_tools, alone in thinking,
|
||||
// with an effort above high asked as high; Opus 5.5, Fable and Mythos are
|
||||
// sent no thinking, at effort low unless one is asked, with the room a
|
||||
// thinking model needs on top of a short reply, as Claude Code gives it
|
||||
// (classifierRoom); Opus 5 and Haiku 5.5 are asked effort high in place
|
||||
// of xhigh or max. between_tools sent to a model that doesn't take it goes
|
||||
// as disabled, or as no thinking where disabled is refused too. model is
|
||||
// the vendor's name of it: the body's, or, for Bedrock and Vertex, whose
|
||||
// body names none, the request path's. Any other request, other vendors'
|
||||
// models' and older Claudes' included, goes byte for byte as it came.
|
||||
func thinkingOffAsTaken(body []byte, path string) []byte {
|
||||
t := gjson.GetBytes(body, "thinking.type").String()
|
||||
if t != "disabled" && t != "between_tools" {
|
||||
return body
|
||||
}
|
||||
model := gjson.GetBytes(body, "model").String()
|
||||
if model == "" {
|
||||
model = path
|
||||
}
|
||||
rule := thinkingOffOf(model)
|
||||
effort := gjson.GetBytes(body, "output_config.effort").String()
|
||||
aboveHigh := effort == "xhigh" || effort == "max"
|
||||
var thinking map[string]any // nil: left out
|
||||
effortTo := ""
|
||||
switch rule {
|
||||
case offTaken:
|
||||
if t == "disabled" {
|
||||
return body
|
||||
}
|
||||
thinking = map[string]any{"type": "disabled"}
|
||||
case offUpToHigh:
|
||||
if t == "disabled" && !aboveHigh {
|
||||
return body
|
||||
}
|
||||
thinking = map[string]any{"type": "disabled"}
|
||||
if aboveHigh {
|
||||
effortTo = "high"
|
||||
}
|
||||
case offBetweenTools:
|
||||
if th := gjson.GetBytes(body, "thinking"); t == "between_tools" && len(th.Map()) == 1 && !aboveHigh {
|
||||
return body
|
||||
}
|
||||
// between_tools takes no other field (display, budget_tokens)
|
||||
thinking = map[string]any{"type": "between_tools"}
|
||||
if aboveHigh {
|
||||
effortTo = "high"
|
||||
}
|
||||
case offNever:
|
||||
if effort == "" {
|
||||
effortTo = "low"
|
||||
}
|
||||
default:
|
||||
return body
|
||||
}
|
||||
dec := json.NewDecoder(bytes.NewReader(body))
|
||||
dec.UseNumber()
|
||||
var q map[string]any
|
||||
if dec.Decode(&q) != nil {
|
||||
return body
|
||||
}
|
||||
if thinking != nil {
|
||||
q["thinking"] = thinking
|
||||
} else {
|
||||
delete(q, "thinking")
|
||||
if n := gjson.GetBytes(body, "max_tokens").Int(); n > 0 && n < classifierRoom {
|
||||
q["max_tokens"] = n + classifierRoom
|
||||
}
|
||||
}
|
||||
if effortTo != "" {
|
||||
oc, _ := q["output_config"].(map[string]any)
|
||||
if oc == nil {
|
||||
oc = map[string]any{}
|
||||
}
|
||||
oc["effort"] = effortTo
|
||||
q["output_config"] = oc
|
||||
}
|
||||
var out bytes.Buffer
|
||||
enc := json.NewEncoder(&out)
|
||||
enc.SetEscapeHTML(false) // <, > and & as the agent wrote them
|
||||
if enc.Encode(q) != nil {
|
||||
return body
|
||||
}
|
||||
return bytes.TrimSuffix(out.Bytes(), []byte("\n"))
|
||||
}
|
||||
|
||||
// asksBetweenTools is Sonnet 5.5's answer to thinking turned off, asking
|
||||
// for between_tools: "To turn thinking off on this model, send "thinking":
|
||||
// {"type": "between_tools"} instead of {"type": "disabled"}" as relays
|
||||
// pass it on (#1454), or the API's "Use "thinking.type.between_tools" for
|
||||
// the lowest thinking setting".
|
||||
var asksBetweenTools = regexp.MustCompile(`(?i)between_tools\W+instead of|use\W+thinking\.type\.between_tools`)
|
||||
|
||||
// thinkingOffRefused is a Claude refusing thinking turned off at any
|
||||
// effort, as Opus 5.5 does: ""thinking.type.disabled" is not supported for
|
||||
// this model. Use "thinking.type.adaptive"…".
|
||||
var thinkingOffRefused = regexp.MustCompile(`(?i)thinking\.type\.disabled\W+is not supported`)
|
||||
|
||||
// withBetweenTools is body with its thinking turned off as Sonnet 5.5
|
||||
// asks it, at effort high at most; false when body doesn't turn it off.
|
||||
func withBetweenTools(body []byte) ([]byte, bool) {
|
||||
if gjson.GetBytes(body, "thinking.type").String() != "disabled" {
|
||||
return nil, false
|
||||
}
|
||||
fields := map[string]any{"thinking": map[string]any{"type": "between_tools"}}
|
||||
if e := gjson.GetBytes(body, "output_config.effort").String(); e == "xhigh" || e == "max" {
|
||||
oc, _ := gjson.GetBytes(body, "output_config").Value().(map[string]any)
|
||||
oc["effort"] = "high"
|
||||
fields["output_config"] = oc
|
||||
}
|
||||
return withFields(body, fields), true
|
||||
}
|
||||
|
||||
// AdaptiveThinking is adaptiveOnly for agents told how to ask a model: a
|
||||
// Claude that takes thinking.type=adaptive and an effort, never a budget.
|
||||
func AdaptiveThinking(model string) bool { return adaptiveOnly(model) }
|
||||
|
||||
@@ -2880,6 +2880,7 @@ func (s *Server) forwardOnce(ctx context.Context, p provider.Provider, to provid
|
||||
// what every path to an Anthropic endpoint sends, relayed or
|
||||
// built, with the model named as the vendor names it
|
||||
body = adaptiveThinking(body)
|
||||
body = thinkingOffAsTaken(body, path)
|
||||
asked := askedBetas(in)
|
||||
if gjson.GetBytes(body, "speed").String() == "fast" && provider.HostOf(p.Base(to)) == "api.anthropic.com" {
|
||||
asked = append(slices.Clone(asked), claudeFastBeta) // a group's member sent fast
|
||||
@@ -3308,7 +3309,13 @@ func (s *Server) passthrough(w http.ResponseWriter, r *http.Request, p provider.
|
||||
b, _ := io.ReadAll(io.LimitReader(res.Body, 1<<20))
|
||||
res.Body.Close()
|
||||
res.Body = io.NopCloser(bytes.NewReader(b))
|
||||
if nb, ok := withoutThinkingOff(body); ok && (alwaysThinks.Match(b) || mandatoryReasoning.Match(b)) {
|
||||
// Sonnet 5.5 by a name thinkingOffAsTaken doesn't know asks for
|
||||
// between_tools in place of disabled (#1454)
|
||||
if nb, ok := withBetweenTools(body); ok && asksBetweenTools.Match(b) {
|
||||
if res, err = s.forward(r.Context(), p, proto, path, p.Prepare(nb), r.Header); err != nil {
|
||||
return writeError(w, proto, 502, p.Name+": "+err.Error()), err.Error(), true
|
||||
}
|
||||
} else if nb, ok := withoutThinkingOff(body); ok && (alwaysThinks.Match(b) || mandatoryReasoning.Match(b) || thinkingOffRefused.Match(b)) {
|
||||
if res, err = s.forward(r.Context(), p, proto, path, p.Prepare(nb), r.Header); err != nil {
|
||||
return writeError(w, proto, 502, p.Name+": "+err.Error()), err.Error(), true
|
||||
}
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
package gateway
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/tidwall/gjson"
|
||||
"github.com/yetone/magpie/internal/provider"
|
||||
)
|
||||
|
||||
// #1454: Claude Code's auto mode classifier, through magpie, asked
|
||||
// claude-sonnet-5-5 with thinking turned off and got 400 "To turn thinking
|
||||
// off on this model, send "thinking": {"type": "between_tools"} instead of
|
||||
// {"type": "disabled"}". Each model that refuses disabled is sent what its
|
||||
// vendor asks for; every other request goes byte for byte as it came.
|
||||
func TestThinkingOffAsTheModelTakesIt(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name, path, body string
|
||||
thinking string // the thinking sent, "" for none
|
||||
effort string
|
||||
maxTokens int64
|
||||
}{
|
||||
{"Sonnet 5.5 is sent between_tools", "/v1/messages",
|
||||
`{"model":"claude-sonnet-5-5","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
`{"type":"between_tools"}`, "", 64},
|
||||
{"Sonnet 5.5 at xhigh is asked high", "/v1/messages",
|
||||
`{"model":"claude-sonnet-5-5","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"xhigh"},"messages":[]}`,
|
||||
`{"type":"between_tools"}`, "high", 64},
|
||||
{"Sonnet 5.5's between_tools takes no other field", "/v1/messages",
|
||||
`{"model":"claude-sonnet-5-5","max_tokens":64,"thinking":{"type":"between_tools","display":"omitted"},"messages":[]}`,
|
||||
`{"type":"between_tools"}`, "", 64},
|
||||
{"a relay's claude-sonnet-5.5", "/v1/messages",
|
||||
`{"model":"claude-sonnet-5.5","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
`{"type":"between_tools"}`, "", 64},
|
||||
{"a relay's sonnet-5.5", "/v1/messages",
|
||||
`{"model":"sonnet-5.5","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
`{"type":"between_tools"}`, "", 64},
|
||||
{"Bedrock names the model in the path", "/model/anthropic.claude-sonnet-5-5-v1:0/invoke",
|
||||
`{"anthropic_version":"bedrock-2023-05-31","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
`{"type":"between_tools"}`, "", 64},
|
||||
{"Opus 5.5 is sent no thinking, at effort low, with room", "/v1/messages",
|
||||
`{"model":"claude-opus-5-5","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
"", "low", 64 + classifierRoom},
|
||||
{"Opus 5.5 keeps an effort asked", "/v1/messages",
|
||||
`{"model":"claude-opus-5-5","max_tokens":8000,"thinking":{"type":"disabled"},"output_config":{"effort":"high"},"messages":[]}`,
|
||||
"", "high", 8000},
|
||||
{"Fable 5.1 is sent no thinking", "/v1/messages",
|
||||
`{"model":"claude-fable-5-1","max_tokens":256,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
"", "low", 256 + classifierRoom},
|
||||
{"Mythos 5.1 is sent no thinking", "/v1/messages",
|
||||
`{"model":"claude-mythos-5-1","max_tokens":4096,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
"", "low", 4096},
|
||||
{"Opus 5 off at max is asked high", "/v1/messages",
|
||||
`{"model":"claude-opus-5","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"max"},"messages":[]}`,
|
||||
`{"type":"disabled"}`, "high", 64},
|
||||
{"Haiku 5.5 off at xhigh is asked high", "/v1/messages",
|
||||
`{"model":"claude-haiku-5-5","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"xhigh"},"messages":[]}`,
|
||||
`{"type":"disabled"}`, "high", 64},
|
||||
{"between_tools to Sonnet 5 goes as disabled", "/v1/messages",
|
||||
`{"model":"claude-sonnet-5","max_tokens":64,"thinking":{"type":"between_tools"},"messages":[]}`,
|
||||
`{"type":"disabled"}`, "", 64},
|
||||
{"between_tools to Opus 5.5 goes as none", "/v1/messages",
|
||||
`{"model":"claude-opus-5-5","max_tokens":64,"thinking":{"type":"between_tools"},"messages":[]}`,
|
||||
"", "low", 64 + classifierRoom},
|
||||
} {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got := thinkingOffAsTaken([]byte(c.body), c.path)
|
||||
if !json.Valid(got) {
|
||||
t.Fatalf("not JSON: %s", got)
|
||||
}
|
||||
th := gjson.GetBytes(got, "thinking").Raw
|
||||
if th != c.thinking {
|
||||
t.Errorf("thinking = %s, want %s (sent %s)", th, c.thinking, got)
|
||||
}
|
||||
if e := gjson.GetBytes(got, "output_config.effort").String(); e != c.effort {
|
||||
t.Errorf("effort = %q, want %q (sent %s)", e, c.effort, got)
|
||||
}
|
||||
if n := gjson.GetBytes(got, "max_tokens").Int(); n != c.maxTokens {
|
||||
t.Errorf("max_tokens = %d, want %d", n, c.maxTokens)
|
||||
}
|
||||
if gjson.GetBytes(got, "messages").Raw != "[]" {
|
||||
t.Errorf("the rest of the request changed: %s", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Another vendor's model and every Claude that takes thinking turned off
|
||||
// get the request byte for byte as it came.
|
||||
func TestThinkingOffLeavesOtherModelsAlone(t *testing.T) {
|
||||
for _, body := range []string{
|
||||
`{"model":"deepseek-v4-pro","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
`{"model":"glm-5.3","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"max"},"messages":[]}`,
|
||||
`{"model":"kimi-k2.5","max_tokens":64,"thinking":{"type":"between_tools"},"messages":[]}`,
|
||||
`{"model":"claude-sonnet-5","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"max"},"messages":[]}`,
|
||||
`{"model":"claude-opus-5","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"high"},"messages":[]}`,
|
||||
`{"model":"claude-haiku-5-5","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
`{"model":"claude-opus-4-8","max_tokens":64,"thinking":{"type":"disabled"},"output_config":{"effort":"max"},"messages":[]}`,
|
||||
`{"model":"claude-sonnet-4-6","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
`{"model":"claude-haiku-4-5-20251001","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
`{"model":"claude-3-7-sonnet-20250219","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
`{"model":"claude-sonnet-5-20261001","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
// Sonnet 5.5 thinking, or not told: nothing to turn off
|
||||
`{"model":"claude-sonnet-5-5","max_tokens":64,"thinking":{"type":"adaptive"},"output_config":{"effort":"max"},"messages":[]}`,
|
||||
`{"model":"claude-sonnet-5-5","max_tokens":64,"messages":[]}`,
|
||||
`{"model":"claude-sonnet-5-5","max_tokens":64,"thinking":{"type":"between_tools"},"output_config":{"effort":"high"},"messages":[]}`,
|
||||
// a later line than these rules know
|
||||
`{"model":"claude-sonnet-6","max_tokens":64,"thinking":{"type":"disabled"},"messages":[]}`,
|
||||
} {
|
||||
if got := thinkingOffAsTaken([]byte(body), "/v1/messages"); string(got) != body {
|
||||
t.Errorf("%s\n sent as %s", body, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// claudeFiveVendor answers as Anthropic does for the 5.5 models: Sonnet
|
||||
// 5.5 refuses disabled in the words the reporter got through a relay, and
|
||||
// between_tools above high; Opus 5.5 refuses disabled; every other model
|
||||
// refuses between_tools.
|
||||
type claudeFiveVendor struct {
|
||||
mu sync.Mutex
|
||||
paths []string
|
||||
sent []string
|
||||
}
|
||||
|
||||
func (v *claudeFiveVendor) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
b, _ := io.ReadAll(r.Body)
|
||||
v.mu.Lock()
|
||||
v.paths = append(v.paths, r.URL.Path)
|
||||
v.sent = append(v.sent, string(b))
|
||||
v.mu.Unlock()
|
||||
model := gjson.GetBytes(b, "model").String()
|
||||
t := gjson.GetBytes(b, "thinking.type").String()
|
||||
effort := gjson.GetBytes(b, "output_config.effort").String()
|
||||
refuse := func(msg string) {
|
||||
w.WriteHeader(400)
|
||||
io.WriteString(w, `{"error":{"type":"invalid_request_error","message":`+msg+`},"type":"error"}`)
|
||||
}
|
||||
sonnet55 := strings.Contains(model, "sonnet-5-5") || strings.Contains(model, "sonnet-5.5") || strings.Contains(model, "s55")
|
||||
switch {
|
||||
case sonnet55 && t == "disabled":
|
||||
refuse(`"To turn thinking off on this model, send \"thinking\": {\"type\": \"between_tools\"} instead of {\"type\": \"disabled\"}. The model does not think before responding. The short updates it writes between tool calls come back as thinking blocks. (tid: 2026101002490321168731647902503)"`)
|
||||
return
|
||||
case sonnet55 && t == "between_tools" && (effort == "xhigh" || effort == "max"):
|
||||
refuse(`"output_config.effort 'xhigh' is not supported when thinking is disabled on this model. Use effort 'high' or below."`)
|
||||
return
|
||||
case strings.Contains(model, "opus-5-5") && t == "disabled":
|
||||
refuse(`"\"thinking.type.disabled\" is not supported for this model. Use \"thinking.type.adaptive\" and \"output_config.effort\" to control thinking behavior."`)
|
||||
return
|
||||
case !sonnet55 && t == "between_tools":
|
||||
refuse(`"thinking.type.between_tools is not supported for this model."`)
|
||||
return
|
||||
}
|
||||
if gjson.GetBytes(b, "stream").Bool() {
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
io.WriteString(w, "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"id\":\"msg_1\",\"type\":\"message\",\"role\":\"assistant\",\"model\":\""+model+"\",\"content\":[],\"usage\":{\"input_tokens\":1,\"output_tokens\":0}}}\n\n"+
|
||||
"event: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"}}\n\n"+
|
||||
"event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"ok\"}}\n\n"+
|
||||
"event: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0}\n\n"+
|
||||
"event: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\"},\"usage\":{\"output_tokens\":1}}\n\n"+
|
||||
"event: message_stop\ndata: {\"type\":\"message_stop\"}\n\n")
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
if strings.HasSuffix(r.URL.Path, "/count_tokens") {
|
||||
io.WriteString(w, `{"input_tokens":7}`)
|
||||
return
|
||||
}
|
||||
io.WriteString(w, `{"id":"msg_1","type":"message","role":"assistant","model":"`+model+`","content":[{"type":"text","text":"<block>no</block>"}],"stop_reason":"end_turn","usage":{"input_tokens":1,"output_tokens":1}}`)
|
||||
}
|
||||
|
||||
func (v *claudeFiveVendor) last() string {
|
||||
v.mu.Lock()
|
||||
defer v.mu.Unlock()
|
||||
return v.sent[len(v.sent)-1]
|
||||
}
|
||||
|
||||
// Every route to an Anthropic endpoint that turns thinking off — the
|
||||
// classifier relayed as Claude Code sends it, count_tokens, and magpie's
|
||||
// own thinking-off (an effort of none fitted, a budget that doesn't fit),
|
||||
// all made before forwardOnce — reaches the 5.5 models as they take it, and a Sonnet 5.5 by a name magpie can't read is asked again with
|
||||
// between_tools, once.
|
||||
func TestClaudeFiveFiveThinkingOffOnTheWire(t *testing.T) {
|
||||
fresh(t)
|
||||
up := &claudeFiveVendor{}
|
||||
srv := httptest.NewServer(up)
|
||||
t.Cleanup(srv.Close)
|
||||
if err := provider.Save(provider.Provider{ID: "relay", Name: "Relay", Key: "k", Anthropic: srv.URL,
|
||||
Models: []string{"claude-sonnet-5-5", "claude-opus-5-5", "claude-sonnet-5", "s55"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
classifier := `{"model":"relay/%s","max_tokens":64,"temperature":0,"stop_sequences":["</block>"],"thinking":{"type":"disabled"},"system":[{"type":"text","text":"Answer <block>yes</block> or <block>no</block>."}],"messages":[{"role":"user","content":"<transcript>ls</transcript>"}]}`
|
||||
for _, c := range []struct {
|
||||
name, path, body string
|
||||
calls int
|
||||
thinking string
|
||||
}{
|
||||
{"classifier to Sonnet 5.5", "/v1/messages", strings.ReplaceAll(classifier, "%s", "claude-sonnet-5-5"), 1, `{"type":"between_tools"}`},
|
||||
{"classifier to Opus 5.5", "/v1/messages", strings.ReplaceAll(classifier, "%s", "claude-opus-5-5"), 1, ""},
|
||||
{"classifier to Sonnet 5 as it came", "/v1/messages", strings.ReplaceAll(classifier, "%s", "claude-sonnet-5"), 1, `{"type":"disabled"}`},
|
||||
{"count_tokens to Sonnet 5.5", "/v1/messages/count_tokens",
|
||||
`{"model":"relay/claude-sonnet-5-5","thinking":{"type":"disabled"},"messages":[{"role":"user","content":"hi"}]}`, 1, `{"type":"between_tools"}`},
|
||||
{"a name magpie can't read is asked again", "/v1/messages", strings.ReplaceAll(classifier, "%s", "s55"), 2, `{"type":"between_tools"}`},
|
||||
} {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
up.mu.Lock()
|
||||
before := len(up.sent)
|
||||
up.mu.Unlock()
|
||||
code, body := post(t, c.path, c.body)
|
||||
if code != 200 {
|
||||
t.Fatalf("status %d: %s", code, body)
|
||||
}
|
||||
up.mu.Lock()
|
||||
calls := len(up.sent) - before
|
||||
up.mu.Unlock()
|
||||
if calls != c.calls {
|
||||
t.Errorf("%d calls upstream, want %d", calls, c.calls)
|
||||
}
|
||||
got := up.last()
|
||||
if th := gjson.Get(got, "thinking").Raw; th != c.thinking {
|
||||
t.Errorf("thinking = %s, want %s: %s", th, c.thinking, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user