diff --git a/docs/reference.md b/docs/reference.md index 8dfebd55..7aea896f 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -1172,8 +1172,17 @@ request to which is translated anyway. Requests pass straight through when the vendor speaks the agent's API and are translated otherwise, streaming, tool calls and reasoning included. The key is `magpie` (any value works; the gateway only listens on loopback), and -models are named `provider/model`. Anything with a base-URL setting can use -it: +models are named `provider/model`. + +- A routing group is asked by its id (`group/`) or its name. +- A bare model id (`glm-5.3`) works too. When several providers serve it, + it goes to the routing group magpie found for it, which asks them in the + order of the Providers page and falls over to the next. With found + groups off, it goes to the first of them in that order alone. +- Every reply names who answered in `X-Magpie-Provider` (the provider) + and `X-Magpie-Model` (`provider/model`), and so does the usage log. + +Anything with a base-URL setting can use it: | Tool speaks | Base URL | Environment | | ----------- | -------------------------- | --------------------------------------------- | diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md index 4b9ae1f8..915465d2 100644 --- a/docs/subsystems/gateway-routing.md +++ b/docs/subsystems/gateway-routing.md @@ -17,7 +17,7 @@ served by a plugin are in [Provider and plugin ownership](provider-plugins.md). | --- | --- | --- | | Address and caller | `DefaultAddr`. The local token `magpie` and the per-agent `magpie-` (`TokenFor`). `agentOf` tells which agent asked, first by token, then by User-Agent. | [`gateway.go`](../../internal/gateway/gateway.go) | | HTTP surface | `Handler` routes `/v1/chat/completions`, `/v1/responses`, `/v1/messages` (each also without `/v1`), `/v1/models`, Gemini's `/v1beta/models/…`, the Codex backend path, `/v1/codex/models` (Codex's catalog shape for a Codex elsewhere with a gateway key, `codexCatalog`, kept to the key's models by `keyAllowed`; #1281), `/mcp/{name}` and the `/v1/magpie/*` status endpoints. A GET on `/v1/responses` (a client opening Responses over a WebSocket) is answered 426, a plain one 405: Responses is relayed over HTTP only. Every route is wrapped in `counted`, `corsGuard`, `callerGuard`, `withCaller` and `keyLimited`; `/v1/messages`, `/messages` and `/v1/messages/count_tokens` in `keepsSignIn` too. `corsGuard` (#1051) serves CORS only to the origins listed in Settings › Network › Web pages (`settings.CORSOrigins`, cleaned to `scheme://host[:port]` by `settings.CleanOrigins`, which refuses a wildcard). For a listed origin it answers the preflight (204) and adds `Access-Control-Allow-Origin`, and every other request from it needs an enabled gateway key (`access.Authenticate`), even on loopback, or gets a 401. It takes the key from the caller `lanGuard` recorded too, since on the running server `lanGuard` has already put magpie's own token in the key's place. Any other web page is refused 403 on every route (`foreignPage`): on loopback a call needs no key and the body's Content-Type isn't checked, so a page the user visits could otherwise send a "simple" `text/plain` POST and spend their subscriptions blind. Only an `http`/`https` Origin is judged, and one on this computer (`localhost`, `*.localhost`, `127.0.0.0/8`, `[::1]`, any port) or the request's own origin (its host and port equal to `Host`, such as a page on the gateway's LAN address) passes, as does usemagpie.ai on `/v1/magpie/omarchy`. `null` and other schemes (`app://`, `file://`, `tauri://`, `vscode-webview://`, `chrome-extension://`) come from desktop apps, webviews and extensions that don't enforce CORS, and pass. A request with no Origin (every agent's) passes through with no CORS headers, as before. There is no `Host` check, so a page whose own name resolves to 127.0.0.1 (DNS rebinding) counts as same-origin. | [`gateway.go`](../../internal/gateway/gateway.go), [`cors.go`](../../internal/gateway/cors.go), [`codex_catalog.go`](../../internal/gateway/codex_catalog.go), [`internal/settings/origins.go`](../../internal/settings/origins.go) | -| Model resolution | `serve` resolves the model the agent asked for: Claude tier stand-ins, a routing group (`provider.GroupFor`: its id, the model's id however a vendor spells it, else the one group whose name it is, unless a provider serves a model by that id), auto stand-ins, then `provider.Resolve`. A model id with `~` and no slash that resolves as nothing else is the one served id (a provider's model or a group) whose slashes it spells as `~` (`unflat`, `provider.Unflat`; MiniMax Code's spelling, #1387), taken back in `validateModel` before key limits, middleware and usage see it; two ids spelled alike are neither. An unknown model gets a 404 naming why: a switched-off provider, a disabled group or an empty group. Otherwise it names up to three ids nearest to what was asked (`provider.Closest`: words of a model's display name count before words of its id, and a provider's id or name only alongside them, so "Zen-Space-Bunny" is told `opencode-zen-free/space-bunny-free`, MOMO on Discord), then the ways a model can be named, then every id (`unknownModel`; chat and embeddings). | [`gateway.go`](../../internal/gateway/gateway.go), [`internal/provider/group.go`](../../internal/provider/group.go), [`internal/provider/flat.go`](../../internal/provider/flat.go) | +| Model resolution | `serve` resolves the model the agent asked for: Claude tier stand-ins, a routing group (`provider.GroupFor`: its id, the model's id however a vendor spells it, else the one group whose name it is, unless a provider serves a model by that id), auto stand-ins, then `provider.Resolve`. A bare model id several providers serve (MOMO on Discord: `glm-5.3` under opencode-go and a6api) is never refused: with found groups on (the default) `GroupFor` takes it to the group magpie found for it (`auto-`), its members in the Providers order, routed and failed over as any group, a conversation staying with the member that answered it while its cache is worth it (affinity); with them off, `Resolve` takes the first provider in the Providers order that exposes it, with no fallback. magpie writes only `provider/model` ids into agents' configs (`magpieModels`), so a bare id is one the user wrote. Every reply says who answered in `X-Magpie-Provider` and `X-Magpie-Model` (`noteMember`), and the usage log and the Routing page keep it. A model id with `~` and no slash that resolves as nothing else is the one served id (a provider's model or a group) whose slashes it spells as `~` (`unflat`, `provider.Unflat`; MiniMax Code's spelling, #1387), taken back in `validateModel` before key limits, middleware and usage see it; two ids spelled alike are neither. An unknown model gets a 404 naming why: a switched-off provider, a disabled group or an empty group. Otherwise it names up to three ids nearest to what was asked (`provider.Closest`: words of a model's display name count before words of its id, and a provider's id or name only alongside them, so "Zen-Space-Bunny" is told `opencode-zen-free/space-bunny-free`, MOMO on Discord), then the ways a model can be named, then every id (`unknownModel`; chat and embeddings). | [`gateway.go`](../../internal/gateway/gateway.go), [`internal/provider/group.go`](../../internal/provider/group.go), [`internal/provider/flat.go`](../../internal/provider/flat.go) | | Candidates | `plan` lists one provider's candidates: one per key or account on, then each `Fallback` model, with resting ones last (`restLast`). `planGroup` and `planLevel` weigh a group's members together. A nested group plans by its own routing (#576). A member's own fallbacks are not the group's. | [`fallback.go`](../../internal/gateway/fallback.go) | | Order of keys and accounts | `Routing`: smart (""), `order`, `rotate`, `usage` (least used, with an `usageHalfLife` of 1h), `pace` or `weight` (`weightedFirst`). Smart, `usage` and `pace` weigh a subscription by its allowance (`allowances`), and a key by its own usage windows when magpie reads them (`keyAllowance`: a sub2api key given a 5-hour, day or 7-day limit, or a key bought as a plan, such as Kimi Code's or a GLM Coding Plan's); any other key has none. A lone key or account is weighed against nobody, but what it has left is still read into the trace (`weighRouted`), so the Routing page shows it as it does for several (#1016); all but used up, it keeps its conversation rather than being said to move (`affine`). A failure rests a candidate (`failure`, `failureOf`). | [`routing.go`](../../internal/gateway/routing.go), [`weighted.go`](../../internal/gateway/weighted.go), [`internal/provider/routing.go`](../../internal/provider/routing.go) | | Limits | Each key or account (a candidate's `who()`) has at most `MaxConcurrency` requests out at once (its own `AccountConcurrency` over it); one more waits in its lane's queue, bounded by `QueueLimit` and `QueueWait` and turned away with a 429 past either (`lanes.take`). `MaxRPM` (coeo91 on Discord: OpenRouter's free models take 20 a minute) bounds how many requests it sends the vendor in any rolling 60 seconds (`rpms`): one more waits for room in the minute, in the order they came, after it has its slot and right before it is sent. A wait that would be longer than 2 minutes (or the provider's `QueueWait`, when shorter: `rpmWait`) is turned away at once with a 429 whose `Retry-After` is when the minute has room; nothing rests for it. An agent gone while it waits gives its place back. Every request that reaches the vendor counts: each try of the candidate loop (a retry, the next key, a fallback) waits in `serve`, and the requests a try makes on the way and the gateway's own asks (count_tokens, System One's decisions, embeddings, drawing, Kimi's search) are counted by `rpmTransport` on `s.client` (`metered`, `paidFor`). count_tokens never waits: a candidate with no room is skipped, and the local estimate answers. A full lane or a minute with no room gives way to a ready key or account of the provider or the group (`laneMate`, `ready`); a fallback is never asked for want of either. A plugin's account sends through its own transport, so only its tries are counted, not a plugin's own extra requests. Set in the provider editor (Concurrency, Requests per minute, Queue size, Queue wait) or with `magpie provider account-concurrency`, `queue` and `rpm`; a subscription's are kept with its picks. | [`concurrency.go`](../../internal/gateway/concurrency.go), [`rpm.go`](../../internal/gateway/rpm.go), [`internal/provider/concurrency.go`](../../internal/provider/concurrency.go) | diff --git a/internal/gateway/bare_slug_test.go b/internal/gateway/bare_slug_test.go new file mode 100644 index 00000000..a9cd9c06 --- /dev/null +++ b/internal/gateway/bare_slug_test.go @@ -0,0 +1,66 @@ +package gateway + +import ( + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/yetone/magpie/internal/provider" +) + +// bareGLM asks for glm-5.3 by its bare id, as MOMO's client did, and says +// which provider the reply named and what it said. Each say is a +// conversation of its own: one answered stays where it was answered. +func bareGLM(t *testing.T, s *Server, say string) (code int, from, body string) { + t.Helper() + rec := httptest.NewRecorder() + req := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(`{"model":"glm-5.3","messages":[{"role":"user","content":"`+say+`"}]}`)) + s.Handler().ServeHTTP(rec, req) + return rec.Code, rec.Header().Get(providerHeader), rec.Body.String() +} + +// A bare model id two providers serve (MOMO on Discord: glm-5.3 under +// opencode-go and a6api) goes where the Providers order says, every +// time, and the reply says which provider it was. With found groups on +// (the default) the group magpie found for it is what's routed, so the +// next provider answers when the first can't; off, it is the first +// provider alone. +func TestBareSlugFollowsTheProvidersOrder(t *testing.T) { + for _, auto := range []bool{true, false} { + t.Run(map[bool]string{true: "found groups", false: "no found groups"}[auto], func(t *testing.T) { + fresh(t) + provider.SetAutoGroups(auto) + t.Cleanup(func() { provider.SetAutoGroups(true) }) + goUp, a6 := &keyed{fail: map[string]int{}}, &keyed{fail: map[string]int{}} + serveOn(t, "opencode-go", "kg", []string{"glm-5.3"}, goUp) + serveOn(t, "a6api", "ka", []string{"glm-5.3"}, a6) + s := New() + for _, say := range []string{"a", "b", "c"} { + if code, from, body := bareGLM(t, s, say); code != http.StatusOK || from != "opencode-go" || !strings.Contains(body, "from kg") { + t.Fatalf("%d %q %s", code, from, body) + } + } + if err := provider.SetOrder([]string{"a6api", "opencode-go"}); err != nil { + t.Fatal(err) + } + if code, from, body := bareGLM(t, s, "d"); code != http.StatusOK || from != "a6api" || !strings.Contains(body, "from ka") { + t.Fatalf("reordered: %d %q %s", code, from, body) + } + // a conversation opencode-go answered, with its cache read, + // stays there in the found group; with none, the order decides + stays := map[bool]string{true: "opencode-go", false: "a6api"}[auto] + if code, from, _ := bareGLM(t, s, "a"); code != http.StatusOK || from != stays { + t.Fatalf("conversation a: %d %q, want %s", code, from, stays) + } + a6.fail["ka"] = http.StatusServiceUnavailable + code, from, _ := bareGLM(t, s, "e") + if auto && (code != http.StatusOK || from != "opencode-go") { + t.Fatalf("the found group's next provider wasn't asked: %d %q", code, from) + } + if !auto && code == http.StatusOK { + t.Fatalf("with no found group, %q answered", from) + } + }) + } +} diff --git a/internal/provider/models.go b/internal/provider/models.go index 64ea3e67..bccfbbc4 100644 --- a/internal/provider/models.go +++ b/internal/provider/models.go @@ -1364,7 +1364,12 @@ func entryIn(entries []Entry, id string) (Entry, bool) { // Resolve maps an id an agent sent to a provider and the vendor's model id. // It accepts catalog ids, "provider/model" for any model (exposed or not), -// and the bare model id when exactly one provider serves it. +// and a bare model id: the first provider in the Providers order (All) +// that exposes it, else the one provider that lists it unexposed. The +// gateway asks GroupFor first, so with found groups on a bare id several +// providers serve is that group, its members in the same order (MOMO on +// Discord: glm-5.3 under opencode-go and a6api). magpie writes only +// provider/model ids into agents' configs, so a bare one is the user's. // A group's id resolves to its first member. func Resolve(id string) (Provider, string, bool) { // Claude Code's mark for a model with a 1M window; it drops it before