diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md index 83eb2cd5..a4559250 100644 --- a/docs/subsystems/gateway-routing.md +++ b/docs/subsystems/gateway-routing.md @@ -35,7 +35,7 @@ served by a plugin are in [Provider and plugin ownership](provider-plugins.md). - Sealed tasks (#619) are checked here. A Codex subagent's task its lead's `spawn_agent` sealed goes only to a ChatGPT account, or back to the Responses provider that answered the lead (`leadProvider`, a relay of the ChatGPT backend such as Sub2API, #1109), relayed as it came even when web search is offered. A lead's turn carrying its subagent's sealed reply (`send_message`) goes back to the Responses provider that answered the lead's own conversation (`leadProvider` of the request's `conversationID`, #1237); otherwise it is refused before anyone is asked (`sealedReader`, `sealedTaskError`). - Rules and image inputs are handled here too. 3. Candidates come from `planGroup` for a group or `plan` for a provider. They are filtered by `sealedReaders` and `allowedCandidates`. An account past its share is held out of the candidates (`capHeld`): a usage cap's share, or 100% of a window for a Codex account set not to spend its credits (`provider.HoldShare`, `CodexNoCredits`). When every candidate is held and one held for its credits spends its resets by itself, `autoReset` spends a reset on it and it is tried (`planned.held`); otherwise a capped or barred set returns `cappedError` or `barredError`. Then `affine` moves the conversation's last answerer first, unless it is resting or nearly used up. An `X-Magpie-Account` header (`AccountHeader`) pins the request to one account: only that account is tried, and when it can't answer the caller is told why. -4. The candidate loop calls `attempt` on each candidate in turn, through a `holdWriter`. The writer holds the reply until it is clearly passing. A candidate that fails before anything is sent rests (`failure`): out of credit for 30 minutes, out of quota until reset, rate limited until retry-after, otherwise a minute and longer on repeats. A subscription account that fails any other way with a window full, by magpie's latest reading of its windows, rests until that window renews. Both quota rests and window rests are held to `longestQuota` (8 days), so a reset time read wrong (a year off) doesn't bench the account for that long. The next candidate is then tried. A failed provider waits `fallbackCooldown` (1 minute) at the back. The vendor's safety filter refusing a reply rests nobody, whether it refuses the turn before any of the reply is said or once some of it has been said (`refused`, `refusedAfter`). The ChatGPT backend's 502 "response protection is unavailable" is about the request's content, not the account: the same history fails the same on each of its accounts and models. It rests nobody, isn't tried again after a moment, and isn't asked of the provider's other accounts or models; another provider left in a group is asked once (`protectionRefused`, `elsewhere`). +4. The candidate loop calls `attempt` on each candidate in turn, through a `holdWriter`. The writer holds the reply until it is clearly passing. A candidate that fails before anything is sent rests (`failure`): out of credit for 30 minutes, out of quota until reset, rate limited until retry-after, otherwise a minute and longer on repeats. A 400, 403, 404 or 422 that says the model isn't served (`unservedWords`) and says nothing of quota, credit or a rate limit rests only that key's or account's model (`modelRefused`, keyed by `restID`, its backoff counted apart): its other models are still served first in order (#1235, SenseNova's 403 "model is not available in the current token plan"). A subscription account that fails any other way with a window full, by magpie's latest reading of its windows, rests until that window renews. Both quota rests and window rests are held to `longestQuota` (8 days), so a reset time read wrong (a year off) doesn't bench the account for that long. The next candidate is then tried. A failed provider waits `fallbackCooldown` (1 minute) at the back. The vendor's safety filter refusing a reply rests nobody, whether it refuses the turn before any of the reply is said or once some of it has been said (`refused`, `refusedAfter`). The ChatGPT backend's 502 "response protection is unavailable" is about the request's content, not the account: the same history fails the same on each of its accounts and models. It rests nobody, isn't tried again after a moment, and isn't asked of the provider's other accounts or models; another provider left in a group is asked once (`protectionRefused`, `elsewhere`). 5. The last candidate's error reaches the agent. A try that answered is recorded (`served`, `answered`) for routing, affinity and usage. ## Constraints and failure behavior @@ -64,7 +64,7 @@ served by a plugin are in [Provider and plugin ownership](provider-plugins.md). ## Verification ```sh -go test -tags nogui ./internal/gateway -run 'TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestProtection|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestClaudeAccountBackOnceItsWindowRenews|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused|TestGeminiThinkingOffAtItsLowest|TestGeminiThoughtsMarkedInExtraContent|TestClaudeSignInNotToldItsSignInFailed|TestSignInWriterReleasesA403|TestSignInBetaStaysWithMagpie|TestStructuredOutput|TestResponsesTextKeepsVerbosityBesideFormat|TestRefusedFormatToldInWords' +go test -tags nogui ./internal/gateway -run 'TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestModelNotInPlanRestsTheModelNotTheKey|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestProtection|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestClaudeAccountBackOnceItsWindowRenews|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused|TestGeminiThinkingOffAtItsLowest|TestGeminiThoughtsMarkedInExtraContent|TestClaudeSignInNotToldItsSignInFailed|TestSignInWriterReleasesA403|TestSignInBetaStaysWithMagpie|TestStructuredOutput|TestResponsesTextKeepsVerbosityBesideFormat|TestRefusedFormatToldInWords' go test -tags nogui ./internal/provider -run 'TestGroup|TestCodexCreditsSwitch|TestRenewedAccountForgetsItsAllowance|TestNative|TestKeyAllowance|TestPlanKeyAllowance|TestClaudeAccountToldRenewed|TestPluginAccountToldRenewed|TestSkipsRedaction' ``` diff --git a/internal/gateway/fallback.go b/internal/gateway/fallback.go index 64ef4534..fdbcc238 100644 --- a/internal/gateway/fallback.go +++ b/internal/gateway/fallback.go @@ -98,14 +98,12 @@ func (c candidate) isOpenRouterFree() bool { } // restID is the candidate's model's own rest key: a free OpenRouter -// model's, whose rate limit is its free-tier limit, and a subscription's, -// out of a pool of its allowance that counts some models only (pooled). -// An account-level rest stays on restKey. +// model's, whose rate limit is its free-tier limit, a subscription's, out +// of a pool of its allowance that counts some models only (pooled), and +// any key's or account's model its vendor said it doesn't serve it +// (modelRefused). An account-level rest stays on restKey. func (c candidate) restID() string { - if c.isOpenRouterFree() || c.p.Account != nil { - return c.restKey() + "/" + c.model - } - return c.restKey() + return c.restKey() + "/" + c.model } // pooled says whether the candidate's subscription, refused for its @@ -710,6 +708,22 @@ var quotaWords = regexp.MustCompile(`(?i)quota|insufficient|balance|credit|billi // key, or this way — words another provider, or key, may not answer with. var unservedWords = regexp.MustCompile(`(?i)model.{0,80}(not (supported|accessible|available|found|enabled|allowed)|unsupported|does ?n[o']t exist|unknown|invalid)|(no such|unknown|invalid|unsupported) model|model_not_found|模型.{0,12}(不存在|不支持|无权|未开通)`) +// modelRefused says the vendor turned the request away over its model +// alone — not one this key's plan, or this key, may use: SenseNova's 403 +// "model is not available in the current token plan" for +// deepseek-v4.1-flash, while the same key serves deepseek-v4-flash +// (#1235). The key or account isn't at fault, so only its model rests, +// and its other models are asked as before. A refusal that also says +// quota, credit or a rate limit is about the account, and rests it. +func modelRefused(status int, body []byte) bool { + switch status { + case 400, 403, 404, 422: + default: + return false + } + return unservedWords.Match(body) && !quotaWords.Match(body) && !refusedWords.Match(body) +} + // refusedWords are how a vendor says it won't take requests from this // client at all — WorkBuddy's "Illegal API invocation from an unapproved // channel" to a chat opening with another agent's own system prompt — a diff --git a/internal/gateway/fallback_test.go b/internal/gateway/fallback_test.go index 7e8b6b71..f18cddce 100644 --- a/internal/gateway/fallback_test.go +++ b/internal/gateway/fallback_test.go @@ -166,6 +166,83 @@ func TestSeveralKeysOnTakeOverFromEachOther(t *testing.T) { } } +// #1235: two SenseNova keys, whose plan doesn't have deepseek-v4.1-flash. +// An agent kept asking for it, every try refused with SenseNova's 403 "model +// is not available in the current token plan", and each refusal rested the +// whole key, longer each time, till both keys sat out for every model. Only +// the model rests: the keys serve deepseek-v4-flash first in their order, +// and no key's rest grows. +func TestModelNotInPlanRestsTheModelNotTheKey(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + t.Setenv("XDG_CACHE_HOME", t.TempDir()) + restingUntil.Lock() + restingUntil.m, restingUntil.note = map[string]time.Time{}, map[string]Rest{} + restingUntil.Unlock() + routed.Lock() + routed.failures = map[string]int{} + routed.Unlock() + var seen []string + up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var req struct { + Model string `json:"model"` + } + json.NewDecoder(r.Body).Decode(&req) + key := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ") + seen = append(seen, key+":"+req.Model) + w.Header().Set("Content-Type", "application/json") + if req.Model == "deepseek-v4.1-flash" { + w.WriteHeader(403) + io.WriteString(w, `{"error": {"code": 7,"message": "model is not available in the current token plan"}}`) + return + } + io.WriteString(w, `{"id":"from-`+key+`","choices":[]}`) + })) + defer up.Close() + p := provider.Provider{ID: "sensenova", Name: "sensenova", Chat: up.URL + "/v1", + Models: []string{"deepseek-v4-flash", "deepseek-v4.1-flash"}, + Key: "k-one", KeyName: "One", Keys: []provider.KeyAccount{{Name: "Two", Key: "k-two"}}} + if err := provider.Save(p); err != nil { + t.Fatal(err) + } + saved, err := provider.Find("sensenova") + if err != nil { + t.Fatal(err) + } + s := New() + send := func(model string) (int, string) { + rec := httptest.NewRecorder() + body := `{"model":"sensenova/` + model + `","messages":[{"role":"user","content":"hi"}]}` + s.Handler().ServeHTTP(rec, httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(body))) + return rec.Code, rec.Body.String() + } + for range 4 { + if code, body := send("deepseek-v4.1-flash"); code != 403 || !strings.Contains(body, "not available in the current token plan") { + t.Fatalf("v4.1: %d %s", code, body) + } + } + for _, k := range saved.KeyList() { + if r, ok := RestOf(saved.RestKey(k)); ok { + t.Errorf("key %s rests for a model its plan lacks: %+v", k.Name, r) + } + } + // the model itself rests on each key, as the failure it is + for _, k := range saved.KeyList() { + r, ok := RestOf(saved.RestKey(k) + "/deepseek-v4.1-flash") + if !ok { + t.Errorf("key %s: deepseek-v4.1-flash doesn't rest", k.Name) + } else if r.Why != failOther || r.Status != 403 { + t.Errorf("key %s: rest %+v", k.Name, r) + } + } + seen = nil + if code, body := send("deepseek-v4-flash"); code != 200 || !strings.Contains(body, "from-k-one") { + t.Fatalf("v4: %d %s", code, body) + } + if strings.Join(seen, ",") != "k-one:deepseek-v4-flash" { + t.Fatalf("v4 tried %v; want the first key, straight", seen) + } +} + // Two ChatGPT accounts ticked: the one Codex is signed in to is out of // quota, so the request goes to the saved one, signed with its own tokens. func TestSubscriptionAccountsTakeOver(t *testing.T) { diff --git a/internal/gateway/routing.go b/internal/gateway/routing.go index f8c42ea4..4620b371 100644 --- a/internal/gateway/routing.go +++ b/internal/gateway/routing.go @@ -67,6 +67,9 @@ func servedCandidate(c candidate, tokens int) { served(c.restKey(), c.restKey(), tokens) provider.NoteServed(c.p, time.Now()) if id := c.restID(); id != c.restKey() { + routed.Lock() + delete(routed.failures, id) + routed.Unlock() clearRest(id) } } @@ -358,6 +361,9 @@ func (s *Server) restAfterMarked(c candidate, status int, header http.Header, bo // counts some models only, the account's others are still in theirs // (Cursor's Auto with Other Models used up) pooled := why == failQuota && c.pooled(now) + // a model the key's plan doesn't serve rests alone, counted by itself + // (#1235): the key's other models are still served + unserved := why == failOther && modelRefused(status, body) r := Rest{Why: why, Status: status, By: "cooldown"} if why == failProxy { // the account is as good as it was; the proxy is the user's to start @@ -402,9 +408,13 @@ func (s *Server) restAfterMarked(c candidate, status int, header http.Header, bo } r.Link, _ = provider.Verification([]byte(r.said)) default: + counted := c.restKey() + if unserved { + counted = c.restID() + } routed.Lock() - routed.failures[c.restKey()]++ - n := routed.failures[c.restKey()] + routed.failures[counted]++ + n := routed.failures[counted] routed.Unlock() d, r.By, r.Failures = min(fallbackCooldown<