gateway, provider: a model its vendor answers is deprecated leaves that provider's lists until it is answered again, and the next member is asked (MOMO on Discord)

MOMO's /v1/models listed opencode-zen-free/exo-free, and a request for it
got 410 "Model exo-free has been deprecated.". OpenCode Zen's own
/zen/v1/models still lists exo-free (checked 2026-10-10), and models.dev
gives it no deprecated status, so the zen-free plugin's discovery keeps it
and a refresh of the list would bring it straight back. No preset
hardcodes it.

What changed:
- fallback.go: retiredWords / modelRetired: a 400, 404, 410 or 422 saying
  the model is deprecated/retired (Zen's "ModelDeprecated", OpenAI's
  "The model ... has been deprecated"), without quota words. It is
  retryable (410 wasn't), and modelRefused, so only that model rests.
- gateway.go: such an answer calls provider.Retire(provider, model);
  routing.go: servedCandidate calls Unretire.
- provider/retired.go: retired.json beside providers.json, re-read by
  mtime; Retire/Unretire touch the catalog so agents' lists are written
  again. buildEntries leaves a retired model out, for that provider only,
  for at most a week. provider/model requests are still sent.
- docs: gateway-routing.md failure handling, reference.md.

Tests: retired_test.go with Zen's literal 410 body: the model leaves
/v1/models while the provider's other model stays; a group falls over to
the next member and only the 410's provider is marked; modelRetired
cases. All three fail with the hooks reverted.

Run: gofmt -l; go vet -tags nogui on darwin, linux, windows for provider
and gateway; go test -tags nogui ./internal/provider ./internal/gateway
./internal/agent ./internal/catalog in a sandbox HOME.
This commit is contained in:
yetone committed 2026-10-10 05:40:28 +08:00
1 parent 52f6b3a3ab
commit 0311c720c1
8 files changed
+321 -2

No files matched your search

+6
View File
@@ -230,6 +230,12 @@ separate Responses endpoint, `catalog=` to borrow a models.dev list, and
`models=` to name the models to expose. Anything a preset does not know can
be overridden the same way.
A model whose vendor answers that it is deprecated or retired (OpenCode
Zen's 410 "Model exo-free has been deprecated.") leaves that provider's
models in `magpie models`, `/v1/models` and every agent's list, even while
the vendor's own list still names it. It comes back once a request for it
as `provider/model` is answered, or after a week.
`magpie provider set <id> header.<Name>=<value>` sends a header of your own
on every request to a key+URL provider (an empty value removes it; signed-in
accounts ignore them). It replaces a header of the same name magpie would
+1 -1
View File
@@ -37,7 +37,7 @@ served by a plugin are in [Provider and plugin ownership](provider-plugins.md).
- A model set for Codex's subagents (`settings.CodexSubagentModel`, the Agents page's Codex "subagent model", willz on Discord) is put on a Codex subagent's request (`x-openai-subagent` read as `kind:collab_spawn`) before this check, whatever model its lead's `spawn_agent` asked for (`codexSubagentPick` in [`codex_subagent.go`](../../internal/gateway/codex_subagent.go)): in `codexBackend` for one of Codex's own bare models, which then goes through `serveAgent`, and in `serve` for any other. Only a ChatGPT account's model can be set (`provider.SetCodexSubagentModel`), as nothing else reads the sealed task. When no ChatGPT account on in magpie serves it now, or a gateway key held to some models may not use it, the subagent goes on the model asked for. The route trace keeps both models and the reason it stayed (`Route.Subagent`), and the Routing page's `kindWhy` says it. The lead's turns, titles and every other kind are never moved.
- Rules and image inputs are handled here too.
3. Candidates come from `planGroup` for a group or `plan` for a provider. They are filtered by `sealedReaders` and `allowedCandidates`. An account past its share is held out of the candidates (`capHeld`): a usage cap's share, each window at its own where the account gives it one (`AccountWindowCaps`, willz on Discord: 50% of the five hours, 40% of the week) and at the account's cap otherwise, or 100% of a window for a Codex account set not to spend its credits (`provider.HoldCaps`, `WindowCaps.Of`, `CodexNoCredits`). The account is held while any window is past its own share, and `cappedError` and the trace name the fullest such window's share and cap. When every candidate is held and one held for its credits spends its resets by itself, `autoReset` spends a reset on it and it is tried (`planned.held`); otherwise a capped or barred set returns `cappedError` or `barredError`. Then `affine` moves the conversation's last answerer first, unless it is resting or nearly used up. An `X-Magpie-Account` header (`AccountHeader`) pins the request to one account: only that account is tried, and when it can't answer the caller is told why.
4. The candidate loop calls `attempt` on each candidate in turn, through a `holdWriter`. The writer holds the reply until it is clearly passing. A candidate that fails before anything is sent rests (`failure`): out of credit for 30 minutes, out of quota until reset, rate limited until retry-after, otherwise a minute and longer on repeats. A 400, 403, 404 or 422 that says the model isn't served (`unservedWords`) and says nothing of quota, credit or a rate limit rests only that key's or account's model (`modelRefused`, keyed by `restID`, its backoff counted apart): its other models are still served first in order (#1235, SenseNova's 403 "model is not available in the current token plan"). A subscription account that fails any other way with a window full, by magpie's latest reading of its windows, rests until that window renews. Both quota rests and window rests are held to `longestQuota` (8 days), so a reset time read wrong (a year off) doesn't bench the account for that long. The next candidate is then tried. A failed provider waits `fallbackCooldown` (1 minute) at the back. The vendor's safety filter refusing a reply rests nobody, whether it refuses the turn before any of the reply is said or once some of it has been said (`refused`, `refusedAfter`). The ChatGPT backend's 502 "response protection is unavailable" is about the request's content, not the account: the same history fails the same on each of its accounts and models. It rests nobody, isn't tried again after a moment, and isn't asked of the provider's other accounts or models; another provider left in a group is asked once (`protectionRefused`, `elsewhere`). One cause is known (#1270): a history that replays the backend's own `web_search_call` in a request that doesn't declare the `web_search` tool is refused so, whatever the account, model or turn (a plain turn, a thread's description, a compaction, Codex's own text compaction for a custom provider with `tools: []`). Every request to the ChatGPT backend with such a history therefore declares cached-only search (`{"type":"web_search","external_web_access":false}`, `provider.DeclareSearch`): an account's in `codexBody`, Codex's own sign-in relayed in `codexUpstreamOn`. It goes in the tools, or for Responses Lite (its header, `reasoning.context` `all_turns`, or an `additional_tools` item already there), which refuses a top-level `web_search`, in the `additional_tools` item. A request that offered no tools gets `tool_choice` `none` so the model can't search; one with a forced choice keeps it. A history without a search, or a request that declares search already, is relayed unchanged. siwc, API-key requests to openai.com and the legacy `/responses/compact` are left as they were.
4. The candidate loop calls `attempt` on each candidate in turn, through a `holdWriter`. The writer holds the reply until it is clearly passing. A candidate that fails before anything is sent rests (`failure`): out of credit for 30 minutes, out of quota until reset, rate limited until retry-after, otherwise a minute and longer on repeats. A 400, 403, 404 or 422 that says the model isn't served (`unservedWords`) and says nothing of quota, credit or a rate limit rests only that key's or account's model (`modelRefused`, keyed by `restID`, its backoff counted apart): its other models are still served first in order (#1235, SenseNova's 403 "model is not available in the current token plan"). A 400, 404, 410 or 422 that says the model is retired (`retiredWords`: OpenCode Zen's 410 `ModelDeprecated` "Model exo-free has been deprecated.", whose own model list still names it; MOMO on Discord) rests only that model the same way, is retryable so a group's next member or another provider of it is asked, and is remembered (`provider.Retire`, `retired.json` beside `providers.json`): the model leaves that provider's entries in the catalog (`buildEntries`), so /v1/models, every agent's model list and the found groups go without it, until a request on it is answered again (`Unretire` in `servedCandidate`) or a week has passed. A refresh of the vendor's list doesn't bring it back early, since that list is where it still is. A `provider/model` request for it is still sent. A subscription account that fails any other way with a window full, by magpie's latest reading of its windows, rests until that window renews. Both quota rests and window rests are held to `longestQuota` (8 days), so a reset time read wrong (a year off) doesn't bench the account for that long. The next candidate is then tried. A failed provider waits `fallbackCooldown` (1 minute) at the back. The vendor's safety filter refusing a reply rests nobody, whether it refuses the turn before any of the reply is said or once some of it has been said (`refused`, `refusedAfter`). The ChatGPT backend's 502 "response protection is unavailable" is about the request's content, not the account: the same history fails the same on each of its accounts and models. It rests nobody, isn't tried again after a moment, and isn't asked of the provider's other accounts or models; another provider left in a group is asked once (`protectionRefused`, `elsewhere`). One cause is known (#1270): a history that replays the backend's own `web_search_call` in a request that doesn't declare the `web_search` tool is refused so, whatever the account, model or turn (a plain turn, a thread's description, a compaction, Codex's own text compaction for a custom provider with `tools: []`). Every request to the ChatGPT backend with such a history therefore declares cached-only search (`{"type":"web_search","external_web_access":false}`, `provider.DeclareSearch`): an account's in `codexBody`, Codex's own sign-in relayed in `codexUpstreamOn`. It goes in the tools, or for Responses Lite (its header, `reasoning.context` `all_turns`, or an `additional_tools` item already there), which refuses a top-level `web_search`, in the `additional_tools` item. A request that offered no tools gets `tool_choice` `none` so the model can't search; one with a forced choice keeps it. A history without a search, or a request that declares search already, is relayed unchanged. siwc, API-key requests to openai.com and the legacy `/responses/compact` are left as they were.
5. The last candidate's error reaches the agent. A try that answered is recorded (`served`, `answered`) for routing, affinity and usage.
## Constraints and failure behavior
+26
View File
@@ -717,6 +717,9 @@ var unservedWords = regexp.MustCompile(`(?i)model.{0,80}(not (supported|accessib
// and its other models are asked as before. A refusal that also says
// quota, credit or a rate limit is about the account, and rests it.
func modelRefused(status int, body []byte) bool {
if modelRetired(status, body) {
return true
}
switch status {
case 400, 403, 404, 422:
default:
@@ -725,6 +728,26 @@ func modelRefused(status int, body []byte) bool {
return unservedWords.Match(body) && !quotaWords.Match(body) && !refusedWords.Match(body)
}
// retiredWords are how a vendor says the model is retired, though its
// list may still name it: OpenCode Zen's 410 {"type":"error","error":
// {"type":"ModelDeprecated","message":"Model exo-free has been
// deprecated."}} (MOMO on Discord), OpenAI's "The model … has been
// deprecated".
var retiredWords = regexp.MustCompile(`(?i)"ModelDeprecated"|model_deprecated|model.{0,80}(has been|was|is now|is) (deprecated|retired|discontinued|decommissioned|sunset)`)
// modelRetired says the vendor turned the request away because the model
// is retired there: it leaves that provider's place in the lists
// (provider.Retire), only it rests, and another provider of it, or the
// group's next member, is asked. Not when it also says quota or credit.
func modelRetired(status int, body []byte) bool {
switch status {
case 400, 404, 410, 422:
default:
return false
}
return retiredWords.Match(body) && !quotaWords.Match(body)
}
// refusedWords are how a vendor says it won't take requests from this
// client at all — WorkBuddy's "Illegal API invocation from an unapproved
// channel" to a chat opening with another agent's own system prompt — a
@@ -790,6 +813,9 @@ func retryable(status int, body []byte) bool {
return false
case status == 401, status == 402, status == 403, status == 404, status == 408, status == 429, status >= 500:
return true
case modelRetired(status, body):
// another provider of the model may still serve it
return true
case status >= 400 && provider.EdgeBlocked(body):
// the vendor's firewall blocked this address (Alibaba Cloud's 405
// in front of zcode.z.ai): another provider goes another way
+5
View File
@@ -2040,6 +2040,11 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
if hw.passing {
clientResponseID = call.Usage.ResponseID
}
if !hw.passing && modelRetired(hw.code(), hw.errBody()) {
// the vendor's list may still name it (OpenCode Zen's
// exo-free): it leaves the lists of this provider's models
provider.Retire(c.p.ID, c.model)
}
outgrew := false // a failure that didn't say so was the request's length
if !hw.passing && !hw.refused && hw.code() >= 400 {
if req, err := parse(from, attemptBody); err == nil {
+130
View File
@@ -0,0 +1,130 @@
package gateway
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"slices"
"strings"
"testing"
"github.com/yetone/magpie/internal/provider"
)
// zenDeprecated is OpenCode Zen's answer to a request for exo-free, as it
// was on 2026-10-10 (HTTP/2 410, content-type: application/json), while
// its /zen/v1/models still listed exo-free (MOMO on Discord).
const zenDeprecated = `{"type":"error","error":{"type":"ModelDeprecated","message":"Model exo-free has been deprecated."},"metadata":{"model":"exo-free"}}`
// zen answers exo-free as OpenCode Zen does, and every other model.
type zen struct{ tried []string }
func (z *zen) ServeHTTP(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
model := modelOf(body)
z.tried = append(z.tried, model)
w.Header().Set("Content-Type", "application/json")
if model == "exo-free" {
w.WriteHeader(http.StatusGone)
io.WriteString(w, zenDeprecated)
return
}
io.WriteString(w, `{"id":"x","choices":[{"index":0,"message":{"role":"assistant","content":"from zen `+model+`"},"finish_reason":"stop"}],"usage":{"prompt_tokens":3,"completion_tokens":1}}`)
}
func listedModels(t *testing.T, s *Server) []string {
t.Helper()
rec := httptest.NewRecorder()
s.Handler().ServeHTTP(rec, httptest.NewRequest("GET", "/v1/models", nil))
var got struct {
Data []struct {
ID string `json:"id"`
} `json:"data"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("/v1/models: %v %s", err, rec.Body.String())
}
var ids []string
for _, m := range got.Data {
ids = append(ids, m.ID)
}
return ids
}
func TestRetiredModelLeavesTheLists(t *testing.T) {
fresh(t)
z := &zen{}
serveOn(t, "opencode-zen-free", "k", []string{"exo-free", "space-bunny-free"}, z)
s := New()
if ids := listedModels(t, s); !slices.Contains(ids, "opencode-zen-free/exo-free") {
t.Fatalf("before: %v", ids)
}
code, body := postAs(t, s, "", `{"model":"opencode-zen-free/exo-free","messages":[{"role":"user","content":"hi"}]}`)
if code != http.StatusGone || !strings.Contains(body, "Model exo-free has been deprecated.") {
t.Fatalf("the vendor's answer is the agent's: %d %s", code, body)
}
if !provider.Retired("opencode-zen-free", "exo-free") {
t.Fatal("exo-free was not remembered as retired")
}
ids := listedModels(t, s)
if slices.Contains(ids, "opencode-zen-free/exo-free") {
t.Fatalf("a retired model still listed: %v", ids)
}
if !slices.Contains(ids, "opencode-zen-free/space-bunny-free") {
t.Fatalf("the provider's other models went too: %v", ids)
}
// the provider's other models are served as before
if code, body := postAs(t, s, "", `{"model":"opencode-zen-free/space-bunny-free","messages":[{"role":"user","content":"hi"}]}`); code != http.StatusOK {
t.Fatalf("space-bunny-free: %d %s", code, body)
}
// a provider/model id is still sent: the vendor says what it says, and
// an answer puts it back
provider.Unretire("opencode-zen-free", "exo-free")
if !slices.Contains(listedModels(t, s), "opencode-zen-free/exo-free") {
t.Fatal("unretired, exo-free is still out of the list")
}
}
func TestRetiredMemberFallsOver(t *testing.T) {
fresh(t)
z := &zen{}
other := &keyed{}
serveOn(t, "opencode-zen-free", "k", []string{"exo-free"}, z)
serveOn(t, "b", "kb", []string{"exo-free"}, other)
if err := provider.SaveGroup(provider.Group{
Name: "Exo", Members: []string{"opencode-zen-free/exo-free", "b/exo-free"}, Routing: provider.Ordered,
}); err != nil {
t.Fatal(err)
}
s := New()
code, body := postAs(t, s, "", `{"model":"group/exo","messages":[{"role":"user","content":"hi"}]}`)
if code != http.StatusOK || !strings.Contains(body, "from kb") {
t.Fatalf("the group's next member wasn't asked: %d %s", code, body)
}
if !provider.Retired("opencode-zen-free", "exo-free") || provider.Retired("b", "exo-free") {
t.Fatal("retired on the wrong provider")
}
ids := listedModels(t, s)
if slices.Contains(ids, "opencode-zen-free/exo-free") || !slices.Contains(ids, "b/exo-free") {
t.Fatalf("lists: %v", ids)
}
}
func TestRetiredWords(t *testing.T) {
for _, c := range []struct {
status int
body string
want bool
}{
{410, zenDeprecated, true},
{404, `{"error":{"message":"The model text-davinci-003 has been deprecated","type":"invalid_request_error","code":"model_not_found"}}`, true},
{400, `{"error":{"message":"max_tokens is deprecated, use max_completion_tokens"}}`, false},
{410, `{"error":{"message":"gone"}}`, false},
{429, zenDeprecated, false},
} {
if got := modelRetired(c.status, []byte(c.body)); got != c.want {
t.Errorf("modelRetired(%d, %s) = %v", c.status, c.body, got)
}
}
}
+1
View File
@@ -66,6 +66,7 @@ func served(rest, key string, tokens int) {
func servedCandidate(c candidate, tokens int) {
served(c.restKey(), c.restKey(), tokens)
provider.NoteServed(c.p, time.Now())
provider.Unretire(c.p.ID, c.model) // back in service, if it was said retired
if id := c.restID(); id != c.restKey() {
routed.Lock()
delete(routed.failures, id)
+2 -1
View File
@@ -1263,12 +1263,13 @@ func providerEntries() []Entry {
func buildEntries() []Entry {
var out []Entry
s := settings.Load()
gone := retiredNow() // its vendor said it is retired (retired.go)
for _, p := range All() {
if !p.On() || p.DecideOnly() { // a dedicated decision API only routes
continue
}
for _, m := range p.Exposed() {
if !p.isDecision(m) {
if !p.isDecision(m) && !gone[p.ID+"/"+m.ID] {
out = append(out, entryFor(p, m, s))
}
}
+150
View File
@@ -0,0 +1,150 @@
package provider
// A model its vendor has retired, though the vendor's own list still names
// it (MOMO on Discord: OpenCode Zen's /v1/models still lists exo-free,
// and a request for it gets 410 {"type":"error","error":{"type":
// "ModelDeprecated","message":"Model exo-free has been deprecated."}}).
// The gateway tells Retire when a provider answers so; the model is then
// left out of the catalog — /v1/models, every agent's model list, the
// routing groups magpie finds — for that provider only, until it answers
// a request again or retiredFor has passed. A request that names it as
// "provider/model" is still sent, so the vendor's answer is what the
// agent is told, and a model back in service is seen at once.
import (
"encoding/json"
"os"
"path/filepath"
"sync"
"time"
"github.com/yetone/magpie/internal/catalog"
)
// retiredFor is how long a model retired so stays out of the lists: the
// vendor's list is what brings it back, and the vendor's list is what
// still has it, so a refresh of the list can't be the end of it. A week
// later it is listed again, and the next request it gets refused so
// takes it out again.
const retiredFor = 7 * 24 * time.Hour
var retired struct {
sync.Mutex
at map[string]time.Time // "provider/model" → when the vendor said it is retired
from string // the file read, and its time then
mod time.Time
}
func retiredPath() string { return filepath.Join(filepath.Dir(Path()), "retired.json") }
// readRetired brings retired.at up to the file, when the file changed since
// it was read: after a restart, or written by another magpie. Called with
// retired held.
func readRetired() {
path := retiredPath()
fi, err := os.Stat(path)
if err != nil {
if path != retired.from {
retired.at, retired.from, retired.mod = nil, path, time.Time{}
}
return
}
if path == retired.from && fi.ModTime().Equal(retired.mod) {
return
}
retired.from, retired.mod = path, fi.ModTime()
b, err := os.ReadFile(path)
if err != nil {
return
}
var saved map[string]time.Time
if json.Unmarshal(b, &saved) != nil {
return // unreadable: what was read before stands
}
retired.at = saved
}
// writeRetired saves retired.at. Called with retired held.
func writeRetired() {
now := time.Now()
for k, at := range retired.at {
if now.Sub(at) >= retiredFor {
delete(retired.at, k)
}
}
b, err := json.MarshalIndent(retired.at, "", " ")
if err != nil {
return
}
path := retiredPath()
if os.MkdirAll(filepath.Dir(path), 0o700) != nil || writeFileAtomic(path, b) != nil {
return
}
if fi, err := os.Stat(path); err == nil {
retired.from, retired.mod = path, fi.ModTime()
}
}
// Retire notes that provider pid's vendor said model is retired: it leaves
// the lists, and the agents' lists are written again.
func Retire(pid, model string) {
if pid == "" || model == "" {
return
}
k := pid + "/" + model
retired.Lock()
readRetired()
at, was := retired.at[k]
if was && time.Since(at) < retiredFor {
retired.Unlock()
return
}
if retired.at == nil {
retired.at = map[string]time.Time{}
}
retired.at[k] = time.Now()
writeRetired()
retired.Unlock()
catalog.Touched()
}
// Unretire takes back a retirement of provider pid's model, when it has
// one: the vendor answered a request for it.
func Unretire(pid, model string) {
k := pid + "/" + model
retired.Lock()
readRetired()
if _, was := retired.at[k]; !was {
retired.Unlock()
return
}
delete(retired.at, k)
writeRetired()
retired.Unlock()
catalog.Touched()
}
// Retired says whether provider pid's vendor said model is retired, within
// retiredFor.
func Retired(pid, model string) bool {
retired.Lock()
defer retired.Unlock()
readRetired()
at, ok := retired.at[pid+"/"+model]
return ok && time.Since(at) < retiredFor
}
// retiredNow is the models retired now, by "provider/model", read once for
// a catalog built.
func retiredNow() map[string]bool {
retired.Lock()
defer retired.Unlock()
readRetired()
out := map[string]bool{}
for k, at := range retired.at {
if time.Since(at) < retiredFor {
out[k] = true
}
}
return out
}