From bed306ae60459c76155d8d6a65b75fa3458104d3 Mon Sep 17 00:00:00 2001 From: forever Date: Wed, 7 Oct 2026 03:51:01 -0700 Subject: [PATCH] gateway: reasoning off reaches Gemini 3 at its lowest level, and Google refusing minimal no longer stops a provider's thoughts (#1062) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * gateway: reasoning off goes to Gemini 3 at its lowest level where it has no minimal, and Google refusing minimal no longer stops the provider's thoughts until restart With reasoning off, a request to Gemini's OpenAI-compatible API (AI Studio's, or a proxy in front of it on this machine or the LAN) went with reasoning_effort minimal, since Gemini 3 can't stop thinking. gemini-3.8-flash has no minimal: models.dev gives its levels as low, medium and high, and Google answers minimal with 400 "Thinking level is unsupported: THINKING_LEVEL_MINIMAL". The message names thinking, so refusesThinkingConfig took it for thinking_config turned away and marked that field unfit for the provider. Until magpie restarted, nothing sent to the provider asked for Gemini's thoughts again, on any model, and a client that asked to see them got none. The turn itself was asked again with no reasoning_effort, at the model's default level. * Reasoning off goes at the model's lowest level where its levels are known and have no minimal, and at minimal where they have it or aren't known (Request.OffLevel, geminiOffLevel). gemini-3.8-flash on AI Studio goes at low, in one request. * Where the levels aren't known (a proxy's own name for the model), Google's 400 for minimal is told apart from thinking_config turned away (geminiMinimalRefused, which also matches gemini-3.1-pro-preview's "thinking_level MINIMAL is not supported by this model" at generateContent). It is remembered as the model refusing reasoning off (offRefused, as for other vendors), and the model is asked again at its lowest level, low when none are known, and so from then on. thinking_config is kept, and the next turn that thinks gets the model's thoughts. * An effort of minimal set on the request, which fitFor keeps where the levels aren't known, is turned away the same way and asked again at low by the existing offEffort step, with its thinking_config. Passthrough (a Chat client calling AI Studio itself) is unchanged: it fits the effort to the model's levels where they're known, and has no retry for a proxy whose levels aren't. Tests, with what upstream's gateway gives: * TestGeminiThinkingOffAtItsLowest: an Anthropic client's turn with thinking disabled, then one thinking at high, then disabled again, to a fake Gemini that answers minimal with the 400 above, in its literal bytes. With the levels known, the first turn goes at [low]; not known, at [minimal low]. Either way the thinking turn then sends thinking_config {include_thoughts: true, thinking_level: high} and gets a thinking block, and the last turn goes at low in one request. Upstream sends [minimal ] both ways. * TestAIStudioThinking: "off at its lowest" sends reasoning_effort low. Upstream sends minimal. Each part was also broken on its own, and each break fails a test: buildChat ignoring OffLevel ("reasoning_effort = "minimal", want "low"", and [minimal minimal ]), the loop never setting it ([minimal minimal ] both ways), no check for the refusal of minimal ([minimal ] with the levels not known), and geminiOffLevel ignoring the refusal ([minimal minimal ]) or the known levels ([minimal low]). Verified on 443f1cbc: gofmt -l, go vet ./..., go build, GOOS=linux go build -tags nogui and GOOS=windows go build pass; go test -tags nogui ./... passes, and -race -count=20 on the tests above. Live, with gemini-3.8-flash through a proxy on this machine in front of Vertex AI's OpenAI-compatible API, the gateway in a temporary HOME (AI Studio itself wasn't tried: there was no key for it). Upstream sent the off turn at minimal, got the 400 and asked again with no reasoning_effort; the thinking turn after it went as reasoning_effort high with no thinking_config, and no thoughts came back. With this change the off turn went at low in one request, the thinking turn with thinking_config at high, and the last off turn at low. Under a name whose levels aren't known, the first off turn went minimal, then low, and the next at low at once. The thoughts reached the client as thinking with the next commit, which reads how that endpoint marks them. * gateway: Gemini's thoughts streamed from Vertex AI's OpenAI-compatible API reach the client as thinking, not as the start of the answer A proxy on this machine or the LAN in front of Gemini is asked for the model's thoughts as AI Studio is, in thinking_config. In front of Vertex AI's OpenAI-compatible API, the thoughts don't come in AI Studio's tags: each comes in a chunk of its own, the thought as the delta's content, and the chunk is marked "extra_content": {"google": {"thought": true}}. chatDecoder read only the content, so the thoughts were streamed to the client as the first part of the answer. * A chunk marked as a thought is thinking (KThink). A whole reply from that API carries its thoughts in tags instead. That isn't changed: the gateway streams from the upstream when it translates, and a Chat client's own request is passed through as it is. Tests, with what upstream's gateway gives: * TestGeminiThoughtsMarkedInExtraContent: that API's stream for "你好" at high with the thoughts asked for, in its literal bytes (testdata/vertex_openai_thoughts.sse), to an Anthropic client: a thinking block that starts "**Acknowledging Greeting**", then the text "你好！很高兴与你交流。请问有什么我可以帮你的吗？". Upstream gives thinking "" and the thought at the head of the text. Verified on 443f1cbc: gofmt -l, go vet ./..., go build, GOOS=linux go build -tags nogui and GOOS=windows go build pass; go test -tags nogui ./... passes, and -race -count=20 on the tests of this and the previous commit. Live, through the proxy and temporary HOME of the previous commit, a turn thinking at high gave a thinking block of 238 characters, and the answer apart from it. --- docs/subsystems/gateway-routing.md | 3 +- internal/gateway/aistudio_thinking_test.go | 165 ++++++++++++++++++ internal/gateway/chat.go | 25 ++- internal/gateway/gateway.go | 25 +++ internal/gateway/ir.go | 5 + .../testdata/vertex_openai_thoughts.sse | 10 ++ 6 files changed, 230 insertions(+), 3 deletions(-) create mode 100644 internal/gateway/testdata/vertex_openai_thoughts.sse diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md index 36351101..1ea523ed 100644 --- a/docs/subsystems/gateway-routing.md +++ b/docs/subsystems/gateway-routing.md @@ -55,11 +55,12 @@ served by a plugin are in [Provider and plugin ownership](provider-plugins.md). - A translated Responses reply gives Codex reasoning items magpie made: an `rs_` id and a summary, nothing sealed. Codex hands them back on every turn. A request relayed as is to OpenAI's API or Azure OpenAI goes without them (`withoutBareReasoning`) unless `store` is true, because both look the id up among stored items and answer 400 "Item with id 'rs_…' not found" (#1008). Any other Responses upstream gets them, because a DeepSeek model wants its reasoning back (#388). When one answers 400 naming the input (`refusesInput`: `param` is `input`, or the error names an item by id), as a relay in front of OpenAI does with a message of its own, `passthrough` asks once more without them and, once that works, leaves them out for that provider and model from then on (`bareReasoningRefused`, #1044). Codex's compaction offers no tools, so it is relayed even when its turns were translated. When a provider answers its summary request 404, or 400 for an item it didn't find (`itemNotFound`, a relay in front of either), `codexCompact` asks once more with the conversation as text (`plainCompact`), then compacts locally (#866, #1008). - Group members are `provider/model[:effort]`. A member with a fixed effort is asked for that effort, bounded by the group's levels (#671). - An agent whose config can't carry an effort has one picked in magpie (`provider.AgentEffort`, settings' `agentEfforts`; Cursor Private Inference's row, #1003). It is asked for on that agent's requests to a model that reasons (`Provider.Thinks`), in place of the level the request sent, fitted to the model's levels and bounded by the group's. Order: a member's fixed effort, then a model's `:effort` suffix, then the turn's pick (a routing rule's), then the agent's pick, then the request's own. A model that doesn't reason is sent as asked; the default (none picked) leaves every request as it came. The pick beats the request's own because Cursor sends its own default as often as a real choice. Picking Default hands the choice back to Cursor. +- Gemini's OpenAI-compatible API (AI Studio's, or a proxy in front of Gemini on this computer or the LAN: `geminiCompat`) is asked for the model's thoughts in `extra_body.google.thinking_config`. A 400 naming it (`refusesThinkingConfig`) marks the provider unfit for the field (`thinking_config`), and it is asked again with `reasoning_effort` alone. Gemini 3 can't stop thinking, so reasoning turned off goes at `minimal`, or at the model's lowest level where its levels are known and have no `minimal` (`geminiOffLevel`). Google's 400 for `minimal` (`geminiMinimalRefused`; gemini-3.8-flash's "Thinking level is unsupported: THINKING_LEVEL_MINIMAL") names thinking too, but isn't the field turned away: the model is marked as refusing reasoning off (`offRefused`) and asked again at its lowest level, and so from then on, still asked for its thoughts. Vertex AI's OpenAI-compatible API marks each streamed chunk of thoughts in the delta's `extra_content.google.thought`, and such a chunk is read as thinking. ## Verification ```sh -go test -tags nogui ./internal/gateway -run 'TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused' +go test -tags nogui ./internal/gateway -run 'TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused|TestGeminiThinkingOffAtItsLowest|TestGeminiThoughtsMarkedInExtraContent' go test -tags nogui ./internal/provider -run 'TestGroup|TestCodexCreditsSwitch|TestRenewedAccountForgetsItsAllowance|TestNative|TestKeyAllowance|TestPlanKeyAllowance|TestSkipsRedaction' ``` diff --git a/internal/gateway/aistudio_thinking_test.go b/internal/gateway/aistudio_thinking_test.go index e3886ba2..8626844c 100644 --- a/internal/gateway/aistudio_thinking_test.go +++ b/internal/gateway/aistudio_thinking_test.go @@ -5,6 +5,8 @@ import ( "io" "net/http" "net/http/httptest" + "os" + "slices" "strconv" "strings" "testing" @@ -33,6 +35,7 @@ func TestAIStudioThinking(t *testing.T) { {"2.5 budget", aiStudioHost, "gemini-2.5-flash", Request{GeminiCompat: true, Thinking: true, Effort: "medium"}, want{"", `{"include_thoughts":true,"thinking_budget":8192}`}}, {"no effort", aiStudioHost, "gemini-3.8-flash", Request{GeminiCompat: true, Thinking: true}, want{"", `{"include_thoughts":true}`}}, {"off", aiStudioHost, "gemini-3.8-flash", Request{GeminiCompat: true, ThinkOff: true}, want{"minimal", ""}}, + {"off at its lowest", aiStudioHost, "gemini-3.8-flash", Request{GeminiCompat: true, ThinkOff: true, OffLevel: "low"}, want{"low", ""}}, {"effort unasked to show", aiStudioHost, "gemini-3.8-flash", Request{GeminiCompat: true, Effort: "medium"}, want{"medium", ""}}, {"other host", "api.deepseek.com", "deepseek-chat", Request{Thinking: true, Effort: "high"}, want{"high", ""}}, {"other host off", "api.deepseek.com", "deepseek-chat", Request{ThinkOff: true}, want{"", ""}}, @@ -246,3 +249,165 @@ func TestGeminiThoughtsThroughLocalProxy(t *testing.T) { }) } } + +// Gemini 3 can't stop thinking, and one without minimal turns it away: +// gemini-3.8-flash, asked with reasoning off as AI Studio's is, answered +// Vertex AI's OpenAI-compatible API with this 400. It names thinking, and +// was taken for thinking_config turned away, so the provider wasn't asked +// for Gemini's thoughts again, for any model, until magpie restarted. +// Reasoning off goes at the model's lowest level instead: at once where its +// levels are known, as models.dev's list for AI Studio gives them, and after +// the refusal where they aren't; the thoughts are still asked for. +func TestGeminiThinkingOffAtItsLowest(t *testing.T) { + const refusal = `[{ + "error": { + "code": 400, + "message": "Thinking level is unsupported: THINKING_LEVEL_MINIMAL", + "status": "INVALID_ARGUMENT" + } +} +] +` + thinkingConfig := func(b map[string]any) map[string]any { + extra, _ := b["extra_body"].(map[string]any) + google, _ := extra["google"].(map[string]any) + tc, _ := google["thinking_config"].(map[string]any) + return tc + } + for _, known := range []bool{false, true} { + name := "levels not known" + if known { + name = "levels known" + } + t.Run(name, func(t *testing.T) { + fresh(t) + var bodies []map[string]any + up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var b map[string]any + json.NewDecoder(r.Body).Decode(&b) + bodies = append(bodies, b) + tc := thinkingConfig(b) + if b["reasoning_effort"] == "minimal" || tc["thinking_level"] == "minimal" { + w.Header().Set("Content-Type", "application/json; charset=UTF-8") + w.WriteHeader(400) + io.WriteString(w, refusal) + return + } + w.Header().Set("Content-Type", "text/event-stream") + content := "你好！" + if tc["include_thoughts"] == true { + content = "The user says hi.你好！" + } + for _, c := range []string{ + `{"id":"x","model":"gemini-3.8-flash","choices":[{"index":0,"delta":{"role":"assistant","content":` + strconv.Quote(content) + `}}]}`, + `{"id":"x","model":"gemini-3.8-flash","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":5,"completion_tokens":3}}`, + } { + io.WriteString(w, "data: "+c+"\n\n") + } + io.WriteString(w, "data: [DONE]\n\n") + })) + t.Cleanup(up.Close) + p := provider.Provider{ID: "ai-studio", Name: "AI Studio", Key: "k", Models: []string{"gemini-3.8-flash"}, Chat: up.URL} + if err := provider.Save(p); err != nil { + t.Fatal(err) + } + if known { + if err := provider.SetModelEfforts("ai-studio/gemini-3.8-flash", []string{"low", "medium", "high"}); err != nil { + t.Fatal(err) + } + } + s := New() + ask := func(thinking string) string { + rec := httptest.NewRecorder() + req := httptest.NewRequest("POST", "/v1/messages", strings.NewReader(`{"model":"ai-studio/gemini-3.8-flash","max_tokens":32000,"stream":true, + `+thinking+`,"messages":[{"role":"user","content":"你好"}]}`)) + s.Handler().ServeHTTP(rec, req) + if rec.Code != 200 || !strings.Contains(rec.Body.String(), "你好！") { + t.Fatalf("status %d: %s", rec.Code, rec.Body.String()) + } + return rec.Body.String() + } + const off, think = `"thinking":{"type":"disabled"}`, `"thinking":{"type":"adaptive"},"output_config":{"effort":"high"}` + ask(off) + var sent []any + for _, b := range bodies { + if b["extra_body"] != nil { + t.Fatalf("reasoning off asked for the thoughts: %v", b) + } + sent = append(sent, b["reasoning_effort"]) + } + if want := []any{"minimal", "low"}; known && !slices.Equal(sent, want[1:]) || !known && !slices.Equal(sent, want) { + t.Fatalf("reasoning off went at %v", sent) + } + n := len(bodies) + out := ask(think) + last := bodies[len(bodies)-1] + if tc := thinkingConfig(last); len(bodies) != n+1 || tc["include_thoughts"] != true || tc["thinking_level"] != "high" || last["reasoning_effort"] != nil { + t.Fatalf("the next turn, thinking, was sent %v (%d requests)", last, len(bodies)-n) + } + if !strings.Contains(out, `"thinking_delta"`) || !strings.Contains(out, "The user says hi.") { + t.Fatalf("no thoughts in the reply:\n%s", out) + } + ask(off) + if last := bodies[len(bodies)-1]; len(bodies) != n+2 || last["reasoning_effort"] != "low" { + t.Fatalf("reasoning off again went at %v (%d requests)", last["reasoning_effort"], len(bodies)-n-1) + } + }) + } +} + +// Asked for Gemini's thoughts, Vertex AI's OpenAI-compatible API gives each +// of them in its own chunk, marked in the delta's extra_content rather than +// with AI Studio's tags: these are its bytes, through a proxy on +// this machine. The thoughts are a thinking block, the rest the answer. +func TestGeminiThoughtsMarkedInExtraContent(t *testing.T) { + fresh(t) + sse, err := os.ReadFile("testdata/vertex_openai_thoughts.sse") + if err != nil { + t.Fatal(err) + } + up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + w.Write(sse) + })) + t.Cleanup(up.Close) + p := provider.Provider{ID: "vertex-proxy", Name: "Vertex proxy", Key: "k", Models: []string{"gemini-3.8-flash"}, Chat: up.URL} + if err := provider.Save(p); err != nil { + t.Fatal(err) + } + rec := httptest.NewRecorder() + req := httptest.NewRequest("POST", "/v1/messages", strings.NewReader(`{"model":"vertex-proxy/gemini-3.8-flash","max_tokens":32000,"stream":true, + "thinking":{"type":"adaptive"},"output_config":{"effort":"high"},"messages":[{"role":"user","content":"你好"}]}`)) + New().Handler().ServeHTTP(rec, req) + if rec.Code != 200 { + t.Fatalf("status %d: %s", rec.Code, rec.Body.String()) + } + var think, text string + var blocks []string + for _, line := range strings.Split(rec.Body.String(), "\n") { + var ev struct { + Type string `json:"type"` + ContentBlock struct { + Type string `json:"type"` + } `json:"content_block"` + Delta struct { + Thinking string `json:"thinking"` + Text string `json:"text"` + } `json:"delta"` + } + if json.Unmarshal([]byte(strings.TrimPrefix(line, "data: ")), &ev) != nil { + continue + } + if ev.Type == "content_block_start" { + blocks = append(blocks, ev.ContentBlock.Type) + } + think += ev.Delta.Thinking + text += ev.Delta.Text + } + if !strings.HasPrefix(think, "**Acknowledging Greeting**") || text != "你好！很高兴与你交流。请问有什么我可以帮你的吗？" { + t.Fatalf("thinking %q, text %q", think, text) + } + if len(blocks) != 2 || blocks[0] != "thinking" || blocks[1] != "text" { + t.Fatalf("blocks %v, want thinking then text", blocks) + } +} diff --git a/internal/gateway/chat.go b/internal/gateway/chat.go index 76667887..0c08363f 100644 --- a/internal/gateway/chat.go +++ b/internal/gateway/chat.go @@ -8,6 +8,7 @@ import ( "mime" "net" "path" + "regexp" "sort" "strings" "time" @@ -411,8 +412,9 @@ func buildChat(r *Request, model, host string, rejectTemp bool) []byte { if tc := aiStudioThinking(r, model); tc != nil { out["extra_body"] = map[string]any{"google": map[string]any{"thinking_config": tc}} } else if r.ThinkOff { - // Gemini 3 can't stop thinking; it thinks least at minimal - out["reasoning_effort"] = "minimal" + // Gemini 3 can't stop thinking; it thinks least at minimal, + // or at its lowest level where it has no minimal + out["reasoning_effort"] = cmp.Or(r.OffLevel, "minimal") } else if r.Effort != "" { out["reasoning_effort"] = r.Effort } @@ -634,6 +636,15 @@ func thinkingEffort(body []byte) []byte { // provider that refused it. const thinkingConfigField = "thinking_config" +// geminiMinimalRefused is Gemini turning minimal away for a model that +// doesn't have it, which names thinking but isn't thinking_config turned +// away (refusesThinkingConfig): gemini-3.8-flash's 400 "Thinking level is +// unsupported: THINKING_LEVEL_MINIMAL", for reasoning_effort minimal and +// thinking_level minimal alike, at Vertex AI's OpenAI-compatible API; and +// gemini-3.1-pro-preview's "thinking_level MINIMAL is not supported by this +// model" at its generateContent. +var geminiMinimalRefused = regexp.MustCompile(`(?i)thinking[ _]level.*minimal`) + // refusesThinkingConfig recognizes an upstream turning a request away for // the thinking_config it was sent, by its error naming it. func refusesThinkingConfig(status int, body []byte) bool { @@ -814,6 +825,13 @@ func (d *chatDecoder) decode(data string, emit func(Event)) error { ReasoningContent string `json:"reasoning_content"` Reasoning string `json:"reasoning"` ToolCalls []cToolCall `json:"tool_calls"` + // Vertex AI's OpenAI-compatible API marks a chunk of + // Gemini's thoughts here, the thought its content + ExtraContent struct { + Google struct { + Thought bool `json:"thought"` + } `json:"google"` + } `json:"extra_content"` } `json:"delta"` FinishReason string `json:"finish_reason"` } `json:"choices"` @@ -859,6 +877,9 @@ func (d *chatDecoder) decode(data string, emit func(Event)) error { } else { json.Unmarshal(c.Delta.Content, &content) } + if c.Delta.ExtraContent.Google.Thought { + t, content = t+content, "" + } if t != "" { emit(Event{Kind: KThink, Text: t}) } diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go index cb3691a5..cea5ff9e 100644 --- a/internal/gateway/gateway.go +++ b/internal/gateway/gateway.go @@ -3302,6 +3302,12 @@ func (s *Server) forwardTranslated(ctx context.Context, p provider.Provider, to r := *req r.GeminiCompat, req = want, &r } + if req.GeminiCompat && req.ThinkOff { + if l := geminiOffLevel(p, model, s.fits(p.ID, offRefused(model), to)); l != req.OffLevel { + r := *req + r.OffLevel, req = l, &r + } + } body, err := build(to, req, model, p.Host(), p.RejectsTemperature(model)) if err != nil { return nil, to, err @@ -3347,6 +3353,13 @@ func (s *Server) forwardTranslated(ctx context.Context, p provider.Provider, to b, _ := io.ReadAll(io.LimitReader(res.Body, 1<<20)) res.Body.Close() res.Body = io.NopCloser(bytes.NewReader(b)) + if req.GeminiCompat && res.StatusCode == http.StatusBadRequest && geminiMinimalRefused.Match(b) && s.fits(p.ID, offRefused(model), to) { + // minimal turned away by a Gemini that hasn't it, reasoning + // off or asked for at minimal: the fields were taken, and the + // model is asked again at its lowest, and so from then on + s.markUnfit(p.ID, offRefused(model), to) + continue + } if req.GeminiCompat && refusesThinkingConfig(res.StatusCode, b) { // Gemini's own fields turned away (a proxy that isn't in front // of Google after all, or Google changing them): asked as @@ -4235,6 +4248,18 @@ func onEffort(p provider.Provider, model string) string { return fitEffort("low", levels) } +// geminiOffLevel is the level Gemini's OpenAI-compatible API is asked to +// think at for model with reasoning turned off: minimal, its least, where +// its levels have it or aren't known; its lowest where they haven't, or +// once it turned minimal away (fits false). Gemini 3 turns away a level it +// doesn't have, gemini-3.8-flash minimal with a 400. +func geminiOffLevel(p provider.Provider, model string, fits bool) string { + if !fits { + return onEffort(p, model) + } + return fitEffort("minimal", p.Efforts(model)) +} + // offRefused is how unfit remembers a provider refusing reasoning turned // off for model. func offRefused(model string) string { return "reasoning off\x00" + model } diff --git a/internal/gateway/ir.go b/internal/gateway/ir.go index 4778e0ba..2563e4a9 100644 --- a/internal/gateway/ir.go +++ b/internal/gateway/ir.go @@ -137,6 +137,11 @@ type Request struct { // (AI Studio's, or a proxy in front of it on this machine or the LAN), // which gives the model's thoughts only when asked in thinking_config. GeminiCompat bool + // OffLevel is the level such an API is asked to think at when the + // client turned reasoning off: Gemini 3 can't stop thinking, and thinks + // least at minimal, or at its lowest level where it has no minimal + // (geminiOffLevel). "" is minimal. + OffLevel string // Resume is set on a request built to go on with a reply the client // already has part of (continuation.go): its last message is that // part, an assistant message the model goes on from, not a turn diff --git a/internal/gateway/testdata/vertex_openai_thoughts.sse b/internal/gateway/testdata/vertex_openai_thoughts.sse new file mode 100644 index 00000000..a5327daa --- /dev/null +++ b/internal/gateway/testdata/vertex_openai_thoughts.sse @@ -0,0 +1,10 @@ +data: {"choices":[{"delta":{"content":"**Acknowledging Greeting**\n\nI'm recognizing a simple, friendly \"hello\" in Chinese. My processing confirms it's a standard greeting, initiating interaction with politeness. I'm ready to proceed based on this acknowledgment.\n\n","extra_content":{"google":{"thought":true}},"role":"assistant"},"index":0,"logprobs":null}],"created":1791318365,"id":"XVnFapHzGtiF998Po_-IwQ4","model":"google/gemini-3.8-flash","object":"chat.completion.chunk","system_fingerprint":""} + +data: {"choices":[{"delta":{"content":"\u4f60\u597d\uff01\u5f88\u9ad8","role":"assistant"},"index":0,"logprobs":null}],"created":1791318365,"id":"XVnFapHzGtiF998Po_-IwQ4","model":"google/gemini-3.8-flash","object":"chat.completion.chunk","system_fingerprint":""} + +data: {"choices":[{"delta":{"content":"\u5174\u4e0e\u4f60\u4ea4\u6d41\u3002\u8bf7\u95ee\u6709\u4ec0\u4e48\u6211\u53ef\u4ee5\u5e2e\u4f60\u7684\u5417\uff1f","role":"assistant"},"index":0,"logprobs":null}],"created":1791318365,"id":"XVnFapHzGtiF998Po_-IwQ4","model":"google/gemini-3.8-flash","object":"chat.completion.chunk","system_fingerprint":""} + +data: {"choices":[{"delta":{"content":"","extra_content":{"google":{"thought_signature":"AY89a19hMHL5rERAubRcDfLpzjHh1if4TijaMFTbbGfO9iNlC7bdKgKR3PDfH/f5eNhNNKirqU1kgjHZh5+zSGaTCiD2Y4WPIsoiKWX11+/TSMkrnENS1haCiMCeswc+3pO9uo/pyB/jacOY5V2MCGq0U1R/dZHGGK0X4bjVs6E9vgYbiP+8wWfVeltJjQE0ndI9iA7KdZSBPEqG0ZP2p8+gos/ySRP2diiSUic51ID0PkR04+BmCc9aMHvCI8iFHTQIZmmE0TAN5C3F1s7+wE0rqWMp5Sq5O911d0UWwsvNOG6m6KmHZnB0RF1BJXwZWNXmOICdYWzXKNAjAl0TIUoxSZ7asVIogGQZRkbmVN1yaZrV/LCm3tkEXA8glMFB36e70CHBmeOvjaXoVt/Ha7o1mc0aYU62Oy9MSP+OYPpJWjynyXawWXgYwsXByeOj/0vOlg8G837kdKqh2JRk9IeBVwCFRZBak9pQ4dD+y9Ys13QY6Fil7GDUCezJL54swfKOHp1x1paeVzrGu09KPtnn7+SH47zEweLAjsptS6gzU7KKnw2i6wZvILXGjhxtv0zTiKketyOvy/g6Fg1gRcotIlIxIpRlMkO06/o01hbjlFVAT843BB2tGyl4Q+DNs8VdSP0JNDY0C/Foz9gUuqNkB4X60xGiSwuuElrEnr/iCt3B6JXG3RuONj3c1cvDo5KNXLrgaNfCZbmINP2vTTFjMj9J4wqipVWm9d2X5WJ8x7WdSs4gdvi8yE6+Qj4Yzn2nX+Sbiu9G43VIi82ZiVMTCVcQTLf45+pAKirqwi0LIRjtnerN6rC3/2vHvKvgsD6oeXclqc6ikx32MUjCRU+G6Mj0poyuIp0EuMc8xpLEDXDCG5CqKGsStcv4+ndBU11sZrNHYaiKvMIqQdQ+FX3llR2C1JQLK9XSUaK2z9Ek7VsMCQ4WPLoW2oCxYpoNnk8f4GgQubKInXqQaRhq6TSN0ZZtt059Xr8lzOrLTUbUUawTv2slf2V0qKR/dxBdQk9551ZzmSb27DhFzETSj92YrPdVHEuAIb1bNwsELm5qIAGq/P/g41Np9Hw49ihPsoWu5qDGmhJca8ZCm6giNgkjIxT/A8poZMKs0G7r9Lp5n2MCL0K4ZueS95RF+PZgU+eOobrZ//XBLNcrgjXbB5iYoTFRTpoFhjnN5n8qwo7Cgygyr6fbt9L4PEtDNZ2RKXHR+gpA/1CO8ceOJu+DAVKOeViNRGiQgRxtJqWGcQhU0lz7d7C+ykxICaXBfCiSLS5ZHYBU3WlCuxrVoOY7jbdpJS27KIv3H00J0kXnRaItJn357R8SV+fkeyLKyabggse88zELZga69jElnqL22bQD1lntQp4Gcq6XX9qfD2DBLkCC7m+MUrKo0m7ycik7GoeRcEUDFvijCM4NON5BIJ8="}},"role":"assistant"},"finish_reason":"stop","index":0,"logprobs":null}],"created":1791318365,"id":"XVnFapHzGtiF998Po_-IwQ4","model":"google/gemini-3.8-flash","object":"chat.completion.chunk","system_fingerprint":"","usage":{"completion_tokens":16,"completion_tokens_details":{"reasoning_tokens":219},"extra_properties":{"google":{"traffic_type":"ON_DEMAND"}},"prompt_tokens":1,"total_tokens":236}} + +data: [DONE] +