provider, docs: a bare model id several providers serve goes by the Providers order, said and tested (MOMO on Discord)
MOMO's {"model":"glm-5.3"} was answered by opencode-go though a6api
serves glm-5.3 too, and they asked for a 409 or for the response to say
which provider was used. The pick was already deterministic, but
Resolve's comment said "exactly one provider" and no doc said what
happens.
Decision: keep serving bare ids, no 409. magpie writes only
provider/model ids into agents' configs (agent/route.go magpieModels),
so a bare id comes from a user's own config, and a 409 would break every
such client the day a second provider adds the model. The rule is now
written down and pinned:
- found groups on (the default): GroupFor takes the bare id to its found
group auto-<slug>, members in the Providers order, smart routing and
fail-over to the next provider; a conversation stays with the member
that answered it while its cache is worth it (affinity).
- found groups off: Resolve takes the first provider in the Providers
order that exposes it, alone.
- every reply already carries X-Magpie-Provider / X-Magpie-Model
(member.go), and the usage log has the provider.
What changed: Resolve's comment; gateway-routing.md model resolution
row; reference.md, the ways a model is named and the headers;
bare_slug_test.go: two providers serving glm-5.3, both modes: the first
in the Providers order answers every new conversation, reordering moves
new ones, an answered one stays in the found group, and a 503 from the
first falls over only in the found group. It passes on the old code
too: it pins behaviour that was already there, now documented.
Run: gofmt -l; go vet -tags nogui on darwin, linux, windows for provider
and gateway; go test -tags nogui ./internal/gateway ./internal/provider
in a sandbox HOME.
This commit is contained in:
1 parent
e50a958014
commit
c1c887798b
4 files changed
+84
-4
No files matched your search
+11
-2
@@ -1172,8 +1172,17 @@ request to which is translated anyway.
|
||||
Requests pass straight through when the vendor speaks the agent's API and
|
||||
are translated otherwise, streaming, tool calls and reasoning included. The
|
||||
key is `magpie` (any value works; the gateway only listens on loopback), and
|
||||
models are named `provider/model`. Anything with a base-URL setting can use
|
||||
it:
|
||||
models are named `provider/model`.
|
||||
|
||||
- A routing group is asked by its id (`group/<id>`) or its name.
|
||||
- A bare model id (`glm-5.3`) works too. When several providers serve it,
|
||||
it goes to the routing group magpie found for it, which asks them in the
|
||||
order of the Providers page and falls over to the next. With found
|
||||
groups off, it goes to the first of them in that order alone.
|
||||
- Every reply names who answered in `X-Magpie-Provider` (the provider)
|
||||
and `X-Magpie-Model` (`provider/model`), and so does the usage log.
|
||||
|
||||
Anything with a base-URL setting can use it:
|
||||
|
||||
| Tool speaks | Base URL | Environment |
|
||||
| ----------- | -------------------------- | --------------------------------------------- |
|
||||
|
||||
@@ -17,7 +17,7 @@ served by a plugin are in [Provider and plugin ownership](provider-plugins.md).
|
||||
| --- | --- | --- |
|
||||
| Address and caller | `DefaultAddr`. The local token `magpie` and the per-agent `magpie-<agent>` (`TokenFor`). `agentOf` tells which agent asked, first by token, then by User-Agent. | [`gateway.go`](../../internal/gateway/gateway.go) |
|
||||
| HTTP surface | `Handler` routes `/v1/chat/completions`, `/v1/responses`, `/v1/messages` (each also without `/v1`), `/v1/models`, Gemini's `/v1beta/models/…`, the Codex backend path, `/v1/codex/models` (Codex's catalog shape for a Codex elsewhere with a gateway key, `codexCatalog`, kept to the key's models by `keyAllowed`; #1281), `/mcp/{name}` and the `/v1/magpie/*` status endpoints. A GET on `/v1/responses` (a client opening Responses over a WebSocket) is answered 426, a plain one 405: Responses is relayed over HTTP only. Every route is wrapped in `counted`, `corsGuard`, `callerGuard`, `withCaller` and `keyLimited`; `/v1/messages`, `/messages` and `/v1/messages/count_tokens` in `keepsSignIn` too. `corsGuard` (#1051) serves CORS only to the origins listed in Settings › Network › Web pages (`settings.CORSOrigins`, cleaned to `scheme://host[:port]` by `settings.CleanOrigins`, which refuses a wildcard). For a listed origin it answers the preflight (204) and adds `Access-Control-Allow-Origin`, and every other request from it needs an enabled gateway key (`access.Authenticate`), even on loopback, or gets a 401. It takes the key from the caller `lanGuard` recorded too, since on the running server `lanGuard` has already put magpie's own token in the key's place. Any other web page is refused 403 on every route (`foreignPage`): on loopback a call needs no key and the body's Content-Type isn't checked, so a page the user visits could otherwise send a "simple" `text/plain` POST and spend their subscriptions blind. Only an `http`/`https` Origin is judged, and one on this computer (`localhost`, `*.localhost`, `127.0.0.0/8`, `[::1]`, any port) or the request's own origin (its host and port equal to `Host`, such as a page on the gateway's LAN address) passes, as does usemagpie.ai on `/v1/magpie/omarchy`. `null` and other schemes (`app://`, `file://`, `tauri://`, `vscode-webview://`, `chrome-extension://`) come from desktop apps, webviews and extensions that don't enforce CORS, and pass. A request with no Origin (every agent's) passes through with no CORS headers, as before. There is no `Host` check, so a page whose own name resolves to 127.0.0.1 (DNS rebinding) counts as same-origin. | [`gateway.go`](../../internal/gateway/gateway.go), [`cors.go`](../../internal/gateway/cors.go), [`codex_catalog.go`](../../internal/gateway/codex_catalog.go), [`internal/settings/origins.go`](../../internal/settings/origins.go) |
|
||||
| Model resolution | `serve` resolves the model the agent asked for: Claude tier stand-ins, a routing group (`provider.GroupFor`: its id, the model's id however a vendor spells it, else the one group whose name it is, unless a provider serves a model by that id), auto stand-ins, then `provider.Resolve`. A model id with `~` and no slash that resolves as nothing else is the one served id (a provider's model or a group) whose slashes it spells as `~` (`unflat`, `provider.Unflat`; MiniMax Code's spelling, #1387), taken back in `validateModel` before key limits, middleware and usage see it; two ids spelled alike are neither. An unknown model gets a 404 naming why: a switched-off provider, a disabled group or an empty group. Otherwise it names up to three ids nearest to what was asked (`provider.Closest`: words of a model's display name count before words of its id, and a provider's id or name only alongside them, so "Zen-Space-Bunny" is told `opencode-zen-free/space-bunny-free`, MOMO on Discord), then the ways a model can be named, then every id (`unknownModel`; chat and embeddings). | [`gateway.go`](../../internal/gateway/gateway.go), [`internal/provider/group.go`](../../internal/provider/group.go), [`internal/provider/flat.go`](../../internal/provider/flat.go) |
|
||||
| Model resolution | `serve` resolves the model the agent asked for: Claude tier stand-ins, a routing group (`provider.GroupFor`: its id, the model's id however a vendor spells it, else the one group whose name it is, unless a provider serves a model by that id), auto stand-ins, then `provider.Resolve`. A bare model id several providers serve (MOMO on Discord: `glm-5.3` under opencode-go and a6api) is never refused: with found groups on (the default) `GroupFor` takes it to the group magpie found for it (`auto-<slug>`), its members in the Providers order, routed and failed over as any group, a conversation staying with the member that answered it while its cache is worth it (affinity); with them off, `Resolve` takes the first provider in the Providers order that exposes it, with no fallback. magpie writes only `provider/model` ids into agents' configs (`magpieModels`), so a bare id is one the user wrote. Every reply says who answered in `X-Magpie-Provider` and `X-Magpie-Model` (`noteMember`), and the usage log and the Routing page keep it. A model id with `~` and no slash that resolves as nothing else is the one served id (a provider's model or a group) whose slashes it spells as `~` (`unflat`, `provider.Unflat`; MiniMax Code's spelling, #1387), taken back in `validateModel` before key limits, middleware and usage see it; two ids spelled alike are neither. An unknown model gets a 404 naming why: a switched-off provider, a disabled group or an empty group. Otherwise it names up to three ids nearest to what was asked (`provider.Closest`: words of a model's display name count before words of its id, and a provider's id or name only alongside them, so "Zen-Space-Bunny" is told `opencode-zen-free/space-bunny-free`, MOMO on Discord), then the ways a model can be named, then every id (`unknownModel`; chat and embeddings). | [`gateway.go`](../../internal/gateway/gateway.go), [`internal/provider/group.go`](../../internal/provider/group.go), [`internal/provider/flat.go`](../../internal/provider/flat.go) |
|
||||
| Candidates | `plan` lists one provider's candidates: one per key or account on, then each `Fallback` model, with resting ones last (`restLast`). `planGroup` and `planLevel` weigh a group's members together. A nested group plans by its own routing (#576). A member's own fallbacks are not the group's. | [`fallback.go`](../../internal/gateway/fallback.go) |
|
||||
| Order of keys and accounts | `Routing`: smart (""), `order`, `rotate`, `usage` (least used, with an `usageHalfLife` of 1h), `pace` or `weight` (`weightedFirst`). Smart, `usage` and `pace` weigh a subscription by its allowance (`allowances`), and a key by its own usage windows when magpie reads them (`keyAllowance`: a sub2api key given a 5-hour, day or 7-day limit, or a key bought as a plan, such as Kimi Code's or a GLM Coding Plan's); any other key has none. A lone key or account is weighed against nobody, but what it has left is still read into the trace (`weighRouted`), so the Routing page shows it as it does for several (#1016); all but used up, it keeps its conversation rather than being said to move (`affine`). A failure rests a candidate (`failure`, `failureOf`). | [`routing.go`](../../internal/gateway/routing.go), [`weighted.go`](../../internal/gateway/weighted.go), [`internal/provider/routing.go`](../../internal/provider/routing.go) |
|
||||
| Limits | Each key or account (a candidate's `who()`) has at most `MaxConcurrency` requests out at once (its own `AccountConcurrency` over it); one more waits in its lane's queue, bounded by `QueueLimit` and `QueueWait` and turned away with a 429 past either (`lanes.take`). `MaxRPM` (coeo91 on Discord: OpenRouter's free models take 20 a minute) bounds how many requests it sends the vendor in any rolling 60 seconds (`rpms`): one more waits for room in the minute, in the order they came, after it has its slot and right before it is sent. A wait that would be longer than 2 minutes (or the provider's `QueueWait`, when shorter: `rpmWait`) is turned away at once with a 429 whose `Retry-After` is when the minute has room; nothing rests for it. An agent gone while it waits gives its place back. Every request that reaches the vendor counts: each try of the candidate loop (a retry, the next key, a fallback) waits in `serve`, and the requests a try makes on the way and the gateway's own asks (count_tokens, System One's decisions, embeddings, drawing, Kimi's search) are counted by `rpmTransport` on `s.client` (`metered`, `paidFor`). count_tokens never waits: a candidate with no room is skipped, and the local estimate answers. A full lane or a minute with no room gives way to a ready key or account of the provider or the group (`laneMate`, `ready`); a fallback is never asked for want of either. A plugin's account sends through its own transport, so only its tries are counted, not a plugin's own extra requests. Set in the provider editor (Concurrency, Requests per minute, Queue size, Queue wait) or with `magpie provider account-concurrency`, `queue` and `rpm`; a subscription's are kept with its picks. | [`concurrency.go`](../../internal/gateway/concurrency.go), [`rpm.go`](../../internal/gateway/rpm.go), [`internal/provider/concurrency.go`](../../internal/provider/concurrency.go) |
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
package gateway
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/yetone/magpie/internal/provider"
|
||||
)
|
||||
|
||||
// bareGLM asks for glm-5.3 by its bare id, as MOMO's client did, and says
|
||||
// which provider the reply named and what it said. Each say is a
|
||||
// conversation of its own: one answered stays where it was answered.
|
||||
func bareGLM(t *testing.T, s *Server, say string) (code int, from, body string) {
|
||||
t.Helper()
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(`{"model":"glm-5.3","messages":[{"role":"user","content":"`+say+`"}]}`))
|
||||
s.Handler().ServeHTTP(rec, req)
|
||||
return rec.Code, rec.Header().Get(providerHeader), rec.Body.String()
|
||||
}
|
||||
|
||||
// A bare model id two providers serve (MOMO on Discord: glm-5.3 under
|
||||
// opencode-go and a6api) goes where the Providers order says, every
|
||||
// time, and the reply says which provider it was. With found groups on
|
||||
// (the default) the group magpie found for it is what's routed, so the
|
||||
// next provider answers when the first can't; off, it is the first
|
||||
// provider alone.
|
||||
func TestBareSlugFollowsTheProvidersOrder(t *testing.T) {
|
||||
for _, auto := range []bool{true, false} {
|
||||
t.Run(map[bool]string{true: "found groups", false: "no found groups"}[auto], func(t *testing.T) {
|
||||
fresh(t)
|
||||
provider.SetAutoGroups(auto)
|
||||
t.Cleanup(func() { provider.SetAutoGroups(true) })
|
||||
goUp, a6 := &keyed{fail: map[string]int{}}, &keyed{fail: map[string]int{}}
|
||||
serveOn(t, "opencode-go", "kg", []string{"glm-5.3"}, goUp)
|
||||
serveOn(t, "a6api", "ka", []string{"glm-5.3"}, a6)
|
||||
s := New()
|
||||
for _, say := range []string{"a", "b", "c"} {
|
||||
if code, from, body := bareGLM(t, s, say); code != http.StatusOK || from != "opencode-go" || !strings.Contains(body, "from kg") {
|
||||
t.Fatalf("%d %q %s", code, from, body)
|
||||
}
|
||||
}
|
||||
if err := provider.SetOrder([]string{"a6api", "opencode-go"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if code, from, body := bareGLM(t, s, "d"); code != http.StatusOK || from != "a6api" || !strings.Contains(body, "from ka") {
|
||||
t.Fatalf("reordered: %d %q %s", code, from, body)
|
||||
}
|
||||
// a conversation opencode-go answered, with its cache read,
|
||||
// stays there in the found group; with none, the order decides
|
||||
stays := map[bool]string{true: "opencode-go", false: "a6api"}[auto]
|
||||
if code, from, _ := bareGLM(t, s, "a"); code != http.StatusOK || from != stays {
|
||||
t.Fatalf("conversation a: %d %q, want %s", code, from, stays)
|
||||
}
|
||||
a6.fail["ka"] = http.StatusServiceUnavailable
|
||||
code, from, _ := bareGLM(t, s, "e")
|
||||
if auto && (code != http.StatusOK || from != "opencode-go") {
|
||||
t.Fatalf("the found group's next provider wasn't asked: %d %q", code, from)
|
||||
}
|
||||
if !auto && code == http.StatusOK {
|
||||
t.Fatalf("with no found group, %q answered", from)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1364,7 +1364,12 @@ func entryIn(entries []Entry, id string) (Entry, bool) {
|
||||
|
||||
// Resolve maps an id an agent sent to a provider and the vendor's model id.
|
||||
// It accepts catalog ids, "provider/model" for any model (exposed or not),
|
||||
// and the bare model id when exactly one provider serves it.
|
||||
// and a bare model id: the first provider in the Providers order (All)
|
||||
// that exposes it, else the one provider that lists it unexposed. The
|
||||
// gateway asks GroupFor first, so with found groups on a bare id several
|
||||
// providers serve is that group, its members in the same order (MOMO on
|
||||
// Discord: glm-5.3 under opencode-go and a6api). magpie writes only
|
||||
// provider/model ids into agents' configs, so a bare one is the user's.
|
||||
// A group's id resolves to its first member.
|
||||
func Resolve(id string) (Provider, string, bool) {
|
||||
// Claude Code's mark for a model with a 1M window; it drops it before
|
||||
|
||||
Reference in new issue
Block a user