diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md index 0a898fc8..3eefd154 100644 --- a/docs/subsystems/gateway-routing.md +++ b/docs/subsystems/gateway-routing.md @@ -69,6 +69,7 @@ served by a plugin are in [Provider and plugin ownership](provider-plugins.md). - Group members are `provider/model[:effort]`. A member with a fixed effort is asked for that effort, bounded by the group's levels (#671). - An agent whose config can't carry an effort has one picked in magpie (`provider.AgentEffort`, settings' `agentEfforts`; Cursor Private Inference's row, #1003). It is asked for on that agent's requests to a model that reasons (`Provider.Thinks`), in place of the level the request sent, fitted to the model's levels and bounded by the group's. Order: a member's fixed effort, then a model's `:effort` suffix, then the turn's pick (a routing rule's), then the agent's pick, then the request's own. A model that doesn't reason is sent as asked; the default (none picked) leaves every request as it came. The pick beats the request's own because Cursor sends its own default as often as a real choice. Picking Default hands the choice back to Cursor. - Gemini's OpenAI-compatible API (AI Studio's, or a proxy in front of Gemini on this computer or the LAN: `geminiCompat`) is asked for the model's thoughts in `extra_body.google.thinking_config`. A 400 naming it (`refusesThinkingConfig`) marks the provider unfit for the field (`thinking_config`), and it is asked again with `reasoning_effort` alone. Gemini 3 can't stop thinking, so reasoning turned off goes at `minimal`, or at the model's lowest level where its levels are known and have no `minimal` (`geminiOffLevel`). Google's 400 for `minimal` (`geminiMinimalRefused`; gemini-3.8-flash's "Thinking level is unsupported: THINKING_LEVEL_MINIMAL") names thinking too, but isn't the field turned away: the model is marked as refusing reasoning off (`offRefused`) and asked again at its lowest level, and so from then on, still asked for its thoughts. Vertex AI's OpenAI-compatible API marks each streamed chunk of thoughts in the delta's `extra_content.google.thought`, and such a chunk is read as thinking. +- A Claude that thinks only adaptively (`adaptiveOnly` in [`anthropic.go`](../../internal/gateway/anthropic.go): Opus, Sonnet and Haiku from 4.6 on, Fable and Mythos 5 and 5.1, read by `claudeVersion` however a relay spells them) is never sent a thinking budget, which they answer with 400. magpie's own request from a Chat or Responses client (`buildAnthropic`) goes as `thinking: {type: "adaptive"}` with the effort in `output_config.effort`; a client's `thinking: {type: "enabled", budget_tokens}` is turned into the same in `forwardOnce` (`adaptiveThinking`), the budget as its nearest effort, above `high` as `max`; and Pi, dsh and omp are written to ask it adaptively (`AdaptiveThinking`: `forceAdaptiveThinking`, omp's `anthropic-adaptive`; #1470). A client's `xhigh` goes as `xhigh`, except to Opus and Sonnet 4.6, which have no `xhigh` and are asked `max` (`noXhigh`, #1471). - A request to an Anthropic endpoint that turns thinking off is sent as the model takes it (`thinkingOffAsTaken`, in `forwardOnce`, so relayed requests, count_tokens, magpie's own thinking-off and retries all get it; the model is read as the vendor names it, from the body, or from the path for Bedrock and Vertex). Claude Sonnet 5.5 refuses `thinking: {type: "disabled"}` and is sent `{type: "between_tools"}` alone, at effort `high` at most. Opus 5.5, Fable 5 and Mythos 5 refuse it at every effort and are sent no thinking, at effort `low` unless one was asked, with `classifierRoom` added to a reply limit below it, as Claude Code does. Opus 5 and Haiku 5.5 take it only up to `high`, so `xhigh` and `max` are asked as `high`. `between_tools` sent to any other Claude goes as `disabled`, or as none where that is refused too. Other vendors' models, older Claudes and later lines than these go as they came (#1454). A Sonnet 5.5 by a name these rules can't read, answering 400 that asks for `between_tools` (`asksBetweenTools`), is asked again with it, once; a refusal of `disabled` at any effort (`thinkingOffRefused`) is asked again with thinking left to the model, as `alwaysThinks` is. - A request to an Anthropic endpoint with `temperature`, `top_p` or `top_k` reaches a model that refuses them without them (`samplingAsTaken`, beside `thinkingOffAsTaken` in `forwardOnce`): Opus 4.7, 4.8, 5 and 5.5, Fable 5, Mythos 5, Sonnet 5 and 5.5, and Haiku 5.5, whose 400 is "`temperature` is deprecated for this model" (Claude Code's auto mode classifier and magpie's own classifiers ask at temperature 0). Older Claudes, Haiku 4.5 among them, other vendors' models and later lines go as they came (#1454). A Claude by a name these rules can't read, answering that a sampling parameter is deprecated (`samplingDeprecated`), is asked again without them, once. - A model that refuses a conversation ending with the assistant's message (`refusesPrefill`: OpenCode Go's Claude models over Messages, "This model does not support assistant message prefill"; Volcengine's Agent Plan, "…for a model that does not support prefill") is marked for that provider, model and API (`prefillRefused`, #1447); the provider's other models keep their prefill ([`prefill.go`](../../internal/gateway/prefill.go)). A Chat or Responses client's conversation ending with the assistant's message, which those APIs read as a turn already said (`Request.LastIsTurn`), is asked again with a user turn after it (`continueText`), translated (`forwardTranslated`) or relayed (`passthrough`, `userLastBody`), and so from then on. An Anthropic client's prefill is its own ask and gets the refusal as sent. magpie's own continuation of a cut reply (`continuation.go`) isn't asked of that model again; the try the model refused ends the reply with what cut it, not with the refusal. diff --git a/internal/gateway/adaptive_test.go b/internal/gateway/adaptive_test.go index e5ff7778..7ea7b8d0 100644 --- a/internal/gateway/adaptive_test.go +++ b/internal/gateway/adaptive_test.go @@ -66,9 +66,13 @@ func TestAdaptiveThinking(t *testing.T) { if th, _ := json.Marshal(out["thinking"]); string(th) != `{"type":"adaptive"}` { t.Errorf("thinking = %s", th) } - if oc, _ := json.Marshal(out["output_config"]); string(oc) != `{"effort":"max"}` { + if oc, _ := json.Marshal(out["output_config"]); string(oc) != `{"effort":"xhigh"}` { t.Errorf("output_config = %s", oc) } + json.Unmarshal(buildAnthropic(&Request{Thinking: true, Effort: "xhigh"}, "claude-opus-4-6"), &out) + if oc, _ := json.Marshal(out["output_config"]); string(oc) != `{"effort":"max"}` { + t.Errorf("4.6 output_config = %s", oc) + } json.Unmarshal(buildAnthropic(&Request{Thinking: true, Effort: "low"}, "claude-sonnet-4-5"), &out) if th, _ := json.Marshal(out["thinking"]); string(th) != `{"budget_tokens":4096,"type":"enabled"}` { t.Errorf("old model thinking = %s", th) diff --git a/internal/gateway/anthropic.go b/internal/gateway/anthropic.go index 20217e96..478cf48c 100644 --- a/internal/gateway/anthropic.go +++ b/internal/gateway/anthropic.go @@ -283,17 +283,33 @@ var claudeVersion = regexp.MustCompile(`(?:^|[^a-z0-9])(?:claude-)?(?:opus|sonne // adaptive thinking"), as do Fable and Mythos 5 and 5.1, whose adaptive // thinking is always on, so how hard it thinks goes in output_config.effort. func adaptiveOnly(model string) bool { + major, minor, ok := claudeVersionOf(model) + return ok && (major > 4 || major == 4 && minor >= 6) +} + +// noXhigh is a Claude that thinks only adaptively but has no xhigh effort, +// Opus and Sonnet 4.6, so it is asked max in its place. Every later one +// takes xhigh (platform.claude.com/docs/en/build-with-claude/effort), and +// max there spends without limit. +func noXhigh(model string) bool { + major, minor, ok := claudeVersionOf(model) + return ok && major == 4 && minor == 6 +} + +// claudeVersionOf is a Claude model's version as claudeVersion reads it; +// ok is false for a model it doesn't read as a Claude. +func claudeVersionOf(model string) (major, minor int, ok bool) { m := claudeVersion.FindStringSubmatch(strings.ToLower(model)) if m == nil { - return false + return 0, 0, false } v := m[1:3] if m[3] != "" { v = m[3:5] } - major, _ := strconv.Atoi(v[0]) - minor, _ := strconv.Atoi(v[1]) - return major > 4 || major == 4 && minor >= 6 + major, _ = strconv.Atoi(v[0]) + minor, _ = strconv.Atoi(v[1]) + return major, minor, true } // adaptiveThinking is an Anthropic request as a model that thinks only @@ -318,7 +334,7 @@ func adaptiveThinking(body []byte) []byte { if gjson.GetBytes(body, "output_config.effort").String() == "" { if e := effortOfBudget(int(th.Get("budget_tokens").Int())); e != "" { if e == "xhigh" { - e = "max" // as buildAnthropic asks it: 4.6 has no xhigh + e = "max" // a budget can't tell xhigh from max (budgetOf), and 4.6 has no xhigh } oc, _ := gjson.GetBytes(body, "output_config").Value().(map[string]any) if oc == nil { @@ -712,7 +728,7 @@ func buildAnthropic(r *Request, model string) []byte { if (r.Thinking || r.Effort != "") && adaptiveOnly(model) { out["thinking"] = map[string]any{"type": "adaptive"} if e := r.Effort; e != "" { - if e == "xhigh" { + if e == "xhigh" && noXhigh(model) { e = "max" } out["output_config"] = map[string]any{"effort": e} diff --git a/internal/gateway/xhigh_effort_test.go b/internal/gateway/xhigh_effort_test.go new file mode 100644 index 00000000..0149f230 --- /dev/null +++ b/internal/gateway/xhigh_effort_test.go @@ -0,0 +1,48 @@ +package gateway + +import ( + "net/http/httptest" + "testing" + + "github.com/yetone/magpie/internal/provider" +) + +// A client that asks xhigh of a Claude that takes it (Opus 4.7 on, Sonnet +// 5 on, Haiku 5.5, Fable: platform.claude.com/docs/en/build-with-claude/effort) +// gets xhigh, not max, which spends without limit; Opus and Sonnet 4.6, +// which have no xhigh, are asked max. +func TestXhighStaysXhighWhereClaudeTakesIt(t *testing.T) { + fresh(t) + up := &adaptiveVendor{} + srv := httptest.NewServer(up) + t.Cleanup(srv.Close) + if err := provider.Save(provider.Provider{ID: "anth", Name: "Anth", Key: "k", Anthropic: srv.URL, + Models: []string{"claude-opus-5-5", "claude-opus-4-7", "claude-sonnet-5", "claude-haiku-5-5", "claude-fable-5-1", "claude-opus-4-6", "claude-sonnet-4-6"}}); err != nil { + t.Fatal(err) + } + for _, c := range []struct{ model, path, body, want string }{ + {"claude-opus-5-5", "/v1/chat/completions", `"reasoning_effort":"xhigh"`, "xhigh"}, + {"claude-opus-5-5", "/v1/responses", `"reasoning":{"effort":"xhigh"}`, "xhigh"}, + {"claude-opus-4-7", "/v1/chat/completions", `"reasoning_effort":"xhigh"`, "xhigh"}, + {"claude-sonnet-5", "/v1/chat/completions", `"reasoning_effort":"xhigh"`, "xhigh"}, + {"claude-haiku-5-5", "/v1/chat/completions", `"reasoning_effort":"xhigh"`, "xhigh"}, + {"claude-fable-5-1", "/v1/chat/completions", `"reasoning_effort":"xhigh"`, "xhigh"}, + {"claude-opus-4-6", "/v1/chat/completions", `"reasoning_effort":"xhigh"`, "max"}, + {"claude-sonnet-4-6", "/v1/chat/completions", `"reasoning_effort":"xhigh"`, "max"}, + } { + t.Run(c.model+c.path, func(t *testing.T) { + body := `{"model":"anth/` + c.model + `",` + c.body + `,"messages":[{"role":"user","content":"hi"}]}` + if c.path == "/v1/responses" { + body = `{"model":"anth/` + c.model + `",` + c.body + `,"input":"hi"}` + } + code, out := post(t, c.path, body) + if code != 200 { + t.Fatalf("status %d: %s", code, out) + } + oc, _ := up.last()["output_config"].(map[string]any) + if e, _ := oc["effort"].(string); e != c.want { + t.Errorf("output_config = %v, want effort %q", up.last()["output_config"], c.want) + } + }) + } +}