gateway: a model a key's plan doesn't serve rests that model alone, not the key (#1235)

SenseNova answers a Token Plan key asked for deepseek-v4.1-flash with
403 "model is not available in the current token plan" while the same
key serves deepseek-v4-flash. That refusal was a plain failure: it
rested the whole key, its backoff counted per key and doubling up to
10 minutes, so an agent that kept asking for the model left every key
of the provider resting for every model (Sun1090: both keys "失败 · 9
分钟后恢复").

A 400/403/404/422 whose body says the model isn't served
(unservedWords), with no quota, credit, rate-limit or channel words,
now rests only that key's or account's model (modelRefused, keyed by
restID, which is now restKey/model for every candidate), its backoff
counted by the model and cleared when the model answers. The key's
other models stay first in order, and its row shows no rest.

TestModelNotInPlanRestsTheModelNotTheKey (two keys, four requests for
the refused model, SenseNova's error shape) fails without the change:
"key One rests for a model its plan lacks: {Why:other Status:403 ...
By:backoff Failures:3}" and "deepseek-v4.1-flash doesn't rest". With
it: go test -tags nogui ./... under a temp HOME passes; -race -count=5
over TestModelNotInPlan|TestSeveralKeys|TestFallback|TestRouting|
TestPool|TestGroup passes; go vet (darwin, GOOS=linux, GOOS=windows,
nogui) passes. Not tried against a real SenseNova key: the body's
"code" field in the fixture is a guess; the message is the reporter's.

docs/subsystems/gateway-routing.md

@@ -35,7 +35,7 @@ served by a plugin are in [Provider and plugin ownership](provider-plugins.md).
    - Sealed tasks (#619) are checked here. A Codex subagent's task its lead's `spawn_agent` sealed goes only to a ChatGPT account, or back to the Responses provider that answered the lead (`leadProvider`, a relay of the ChatGPT backend such as Sub2API, #1109), relayed as it came even when web search is offered. A lead's turn carrying its subagent's sealed reply (`send_message`) goes back to the Responses provider that answered the lead's own conversation (`leadProvider` of the request's `conversationID`, #1237); otherwise it is refused before anyone is asked (`sealedReader`, `sealedTaskError`).
    - Rules and image inputs are handled here too.
 3. Candidates come from `planGroup` for a group or `plan` for a provider. They are filtered by `sealedReaders` and `allowedCandidates`. An account past its share is held out of the candidates (`capHeld`): a usage cap's share, or 100% of a window for a Codex account set not to spend its credits (`provider.HoldShare`, `CodexNoCredits`). When every candidate is held and one held for its credits spends its resets by itself, `autoReset` spends a reset on it and it is tried (`planned.held`); otherwise a capped or barred set returns `cappedError` or `barredError`. Then `affine` moves the conversation's last answerer first, unless it is resting or nearly used up. An `X-Magpie-Account` header (`AccountHeader`) pins the request to one account: only that account is tried, and when it can't answer the caller is told why.
-4. The candidate loop calls `attempt` on each candidate in turn, through a `holdWriter`. The writer holds the reply until it is clearly passing. A candidate that fails before anything is sent rests (`failure`): out of credit for 30 minutes, out of quota until reset, rate limited until retry-after, otherwise a minute and longer on repeats. A subscription account that fails any other way with a window full, by magpie's latest reading of its windows, rests until that window renews. Both quota rests and window rests are held to `longestQuota` (8 days), so a reset time read wrong (a year off) doesn't bench the account for that long. The next candidate is then tried. A failed provider waits `fallbackCooldown` (1 minute) at the back. The vendor's safety filter refusing a reply rests nobody, whether it refuses the turn before any of the reply is said or once some of it has been said (`refused`, `refusedAfter`). The ChatGPT backend's 502 "response protection is unavailable" is about the request's content, not the account: the same history fails the same on each of its accounts and models. It rests nobody, isn't tried again after a moment, and isn't asked of the provider's other accounts or models; another provider left in a group is asked once (`protectionRefused`, `elsewhere`).
+4. The candidate loop calls `attempt` on each candidate in turn, through a `holdWriter`. The writer holds the reply until it is clearly passing. A candidate that fails before anything is sent rests (`failure`): out of credit for 30 minutes, out of quota until reset, rate limited until retry-after, otherwise a minute and longer on repeats. A 400, 403, 404 or 422 that says the model isn't served (`unservedWords`) and says nothing of quota, credit or a rate limit rests only that key's or account's model (`modelRefused`, keyed by `restID`, its backoff counted apart): its other models are still served first in order (#1235, SenseNova's 403 "model is not available in the current token plan"). A subscription account that fails any other way with a window full, by magpie's latest reading of its windows, rests until that window renews. Both quota rests and window rests are held to `longestQuota` (8 days), so a reset time read wrong (a year off) doesn't bench the account for that long. The next candidate is then tried. A failed provider waits `fallbackCooldown` (1 minute) at the back. The vendor's safety filter refusing a reply rests nobody, whether it refuses the turn before any of the reply is said or once some of it has been said (`refused`, `refusedAfter`). The ChatGPT backend's 502 "response protection is unavailable" is about the request's content, not the account: the same history fails the same on each of its accounts and models. It rests nobody, isn't tried again after a moment, and isn't asked of the provider's other accounts or models; another provider left in a group is asked once (`protectionRefused`, `elsewhere`).
 5. The last candidate's error reaches the agent. A try that answered is recorded (`served`, `answered`) for routing, affinity and usage.
 
 ## Constraints and failure behavior
@@ -64,7 +64,7 @@ served by a plugin are in [Provider and plugin ownership](provider-plugins.md).
 ## Verification
 
 ```sh
-go test -tags nogui ./internal/gateway -run 'TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestProtection|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestClaudeAccountBackOnceItsWindowRenews|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused|TestGeminiThinkingOffAtItsLowest|TestGeminiThoughtsMarkedInExtraContent|TestClaudeSignInNotToldItsSignInFailed|TestSignInWriterReleasesA403|TestSignInBetaStaysWithMagpie|TestStructuredOutput|TestResponsesTextKeepsVerbosityBesideFormat|TestRefusedFormatToldInWords'
+go test -tags nogui ./internal/gateway -run 'TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestModelNotInPlanRestsTheModelNotTheKey|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestProtection|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestClaudeAccountBackOnceItsWindowRenews|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted|TestCORS|TestForeignPagesRefused|TestGeminiThinkingOffAtItsLowest|TestGeminiThoughtsMarkedInExtraContent|TestClaudeSignInNotToldItsSignInFailed|TestSignInWriterReleasesA403|TestSignInBetaStaysWithMagpie|TestStructuredOutput|TestResponsesTextKeepsVerbosityBesideFormat|TestRefusedFormatToldInWords'
 go test -tags nogui ./internal/provider -run 'TestGroup|TestCodexCreditsSwitch|TestRenewedAccountForgetsItsAllowance|TestNative|TestKeyAllowance|TestPlanKeyAllowance|TestClaudeAccountToldRenewed|TestPluginAccountToldRenewed|TestSkipsRedaction'
 ```
 

internal/gateway/fallback.go

@@ -98,14 +98,12 @@ func (c candidate) isOpenRouterFree() bool {
 }
 
 // restID is the candidate's model's own rest key: a free OpenRouter
-// model's, whose rate limit is its free-tier limit, and a subscription's,
-// out of a pool of its allowance that counts some models only (pooled).
-// An account-level rest stays on restKey.
+// model's, whose rate limit is its free-tier limit, a subscription's, out
+// of a pool of its allowance that counts some models only (pooled), and
+// any key's or account's model its vendor said it doesn't serve it
+// (modelRefused). An account-level rest stays on restKey.
 func (c candidate) restID() string {
-	if c.isOpenRouterFree() || c.p.Account != nil {
-		return c.restKey() + "/" + c.model
-	}
-	return c.restKey()
+	return c.restKey() + "/" + c.model
 }
 
 // pooled says whether the candidate's subscription, refused for its
@@ -710,6 +708,22 @@ var quotaWords = regexp.MustCompile(`(?i)quota|insufficient|balance|credit|billi
 // key, or this way — words another provider, or key, may not answer with.
 var unservedWords = regexp.MustCompile(`(?i)model.{0,80}(not (supported|accessible|available|found|enabled|allowed)|unsupported|does ?n[o']t exist|unknown|invalid)|(no such|unknown|invalid|unsupported) model|model_not_found|模型.{0,12}(不存在|不支持|无权|未开通)`)
 
+// modelRefused says the vendor turned the request away over its model
+// alone — not one this key's plan, or this key, may use: SenseNova's 403
+// "model is not available in the current token plan" for
+// deepseek-v4.1-flash, while the same key serves deepseek-v4-flash
+// (#1235). The key or account isn't at fault, so only its model rests,
+// and its other models are asked as before. A refusal that also says
+// quota, credit or a rate limit is about the account, and rests it.
+func modelRefused(status int, body []byte) bool {
+	switch status {
+	case 400, 403, 404, 422:
+	default:
+		return false
+	}
+	return unservedWords.Match(body) && !quotaWords.Match(body) && !refusedWords.Match(body)
+}
+
 // refusedWords are how a vendor says it won't take requests from this
 // client at all — WorkBuddy's "Illegal API invocation from an unapproved
 // channel" to a chat opening with another agent's own system prompt — a

internal/gateway/fallback_test.go

@@ -166,6 +166,83 @@ func TestSeveralKeysOnTakeOverFromEachOther(t *testing.T) {
 	}
 }
 
+// #1235: two SenseNova keys, whose plan doesn't have deepseek-v4.1-flash.
+// An agent kept asking for it, every try refused with SenseNova's 403 "model
+// is not available in the current token plan", and each refusal rested the
+// whole key, longer each time, till both keys sat out for every model. Only
+// the model rests: the keys serve deepseek-v4-flash first in their order,
+// and no key's rest grows.
+func TestModelNotInPlanRestsTheModelNotTheKey(t *testing.T) {
+	t.Setenv("XDG_CONFIG_HOME", t.TempDir())
+	t.Setenv("XDG_CACHE_HOME", t.TempDir())
+	restingUntil.Lock()
+	restingUntil.m, restingUntil.note = map[string]time.Time{}, map[string]Rest{}
+	restingUntil.Unlock()
+	routed.Lock()
+	routed.failures = map[string]int{}
+	routed.Unlock()
+	var seen []string
+	up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		var req struct {
+			Model string `json:"model"`
+		}
+		json.NewDecoder(r.Body).Decode(&req)
+		key := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")
+		seen = append(seen, key+":"+req.Model)
+		w.Header().Set("Content-Type", "application/json")
+		if req.Model == "deepseek-v4.1-flash" {
+			w.WriteHeader(403)
+			io.WriteString(w, `{"error": {"code": 7,"message": "model is not available in the current token plan"}}`)
+			return
+		}
+		io.WriteString(w, `{"id":"from-`+key+`","choices":[]}`)
+	}))
+	defer up.Close()
+	p := provider.Provider{ID: "sensenova", Name: "sensenova", Chat: up.URL + "/v1",
+		Models: []string{"deepseek-v4-flash", "deepseek-v4.1-flash"},
+		Key:    "k-one", KeyName: "One", Keys: []provider.KeyAccount{{Name: "Two", Key: "k-two"}}}
+	if err := provider.Save(p); err != nil {
+		t.Fatal(err)
+	}
+	saved, err := provider.Find("sensenova")
+	if err != nil {
+		t.Fatal(err)
+	}
+	s := New()
+	send := func(model string) (int, string) {
+		rec := httptest.NewRecorder()
+		body := `{"model":"sensenova/` + model + `","messages":[{"role":"user","content":"hi"}]}`
+		s.Handler().ServeHTTP(rec, httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(body)))
+		return rec.Code, rec.Body.String()
+	}
+	for range 4 {
+		if code, body := send("deepseek-v4.1-flash"); code != 403 || !strings.Contains(body, "not available in the current token plan") {
+			t.Fatalf("v4.1: %d %s", code, body)
+		}
+	}
+	for _, k := range saved.KeyList() {
+		if r, ok := RestOf(saved.RestKey(k)); ok {
+			t.Errorf("key %s rests for a model its plan lacks: %+v", k.Name, r)
+		}
+	}
+	// the model itself rests on each key, as the failure it is
+	for _, k := range saved.KeyList() {
+		r, ok := RestOf(saved.RestKey(k) + "/deepseek-v4.1-flash")
+		if !ok {
+			t.Errorf("key %s: deepseek-v4.1-flash doesn't rest", k.Name)
+		} else if r.Why != failOther || r.Status != 403 {
+			t.Errorf("key %s: rest %+v", k.Name, r)
+		}
+	}
+	seen = nil
+	if code, body := send("deepseek-v4-flash"); code != 200 || !strings.Contains(body, "from-k-one") {
+		t.Fatalf("v4: %d %s", code, body)
+	}
+	if strings.Join(seen, ",") != "k-one:deepseek-v4-flash" {
+		t.Fatalf("v4 tried %v; want the first key, straight", seen)
+	}
+}
+
 // Two ChatGPT accounts ticked: the one Codex is signed in to is out of
 // quota, so the request goes to the saved one, signed with its own tokens.
 func TestSubscriptionAccountsTakeOver(t *testing.T) {

internal/gateway/routing.go

@@ -67,6 +67,9 @@ func servedCandidate(c candidate, tokens int) {
 	served(c.restKey(), c.restKey(), tokens)
 	provider.NoteServed(c.p, time.Now())
 	if id := c.restID(); id != c.restKey() {
+		routed.Lock()
+		delete(routed.failures, id)
+		routed.Unlock()
 		clearRest(id)
 	}
 }
@@ -358,6 +361,9 @@ func (s *Server) restAfterMarked(c candidate, status int, header http.Header, bo
 	// counts some models only, the account's others are still in theirs
 	// (Cursor's Auto with Other Models used up)
 	pooled := why == failQuota && c.pooled(now)
+	// a model the key's plan doesn't serve rests alone, counted by itself
+	// (#1235): the key's other models are still served
+	unserved := why == failOther && modelRefused(status, body)
 	r := Rest{Why: why, Status: status, By: "cooldown"}
 	if why == failProxy {
 		// the account is as good as it was; the proxy is the user's to start
@@ -402,9 +408,13 @@ func (s *Server) restAfterMarked(c candidate, status int, header http.Header, bo
 		}
 		r.Link, _ = provider.Verification([]byte(r.said))
 	default:
+		counted := c.restKey()
+		if unserved {
+			counted = c.restID()
+		}
 		routed.Lock()
-		routed.failures[c.restKey()]++
-		n := routed.failures[c.restKey()]
+		routed.failures[counted]++
+		n := routed.failures[counted]
 		routed.Unlock()
 		d, r.By, r.Failures = min(fallbackCooldown<<min(n-1, 10), longestRetry), "backoff", n
 		// a subscription that failed with a window full is out of it,
@@ -413,7 +423,7 @@ func (s *Server) restAfterMarked(c candidate, status int, header http.Header, bo
 		// Held to the longest an account out of quota sits out: the window
 		// is the vendor's reading, and one read a year off (a /usage date
 		// with no year) would bench the account until someone noticed
-		if t := c.full(now); !t.IsZero() && c.p.Account != nil {
+		if t := c.full(now); !t.IsZero() && c.p.Account != nil && !unserved {
 			d, r.By = min(t.Sub(now), longestQuota), "window"
 		}
 	}
@@ -439,7 +449,7 @@ func (s *Server) restAfterMarked(c candidate, status int, header http.Header, bo
 	if why == failRate && c.isOpenRouterFree() && sharedPool {
 		id = c.restID()
 	}
-	if pooled {
+	if pooled || unserved {
 		id = c.restID()
 	}
 	r.Key = id