magpie/docs/subsystems/gateway-routing.md

Gateway routing and fallback

The gateway is magpie's local LLM endpoint (127.0.0.1:3425 by default). It serves Chat Completions, Responses, Anthropic Messages and Gemini. It finds the provider, key or account and model for each request. When the upstream doesn't speak the agent's API, the gateway translates. When an upstream fails before any of the reply is sent, the gateway moves to the next candidate.

This page covers how a request is routed. Plugins that change requests and replies are in Gateway middleware. Subscriptions served by a plugin are in Provider and plugin ownership.

Responsibilities and sources of truth

Part	Responsibility	Source
Address and caller	DefaultAddr. The local token magpie and the per-agent magpie-<agent> (TokenFor). agentOf tells which agent asked, first by token, then by User-Agent.	gateway.go
HTTP surface	Handler routes /v1/chat/completions, /v1/responses, /v1/messages (each also without /v1), /v1/models, Gemini's /v1beta/models/…, the Codex backend path, /mcp/{name} and the /v1/magpie/* status endpoints. A GET on /v1/responses (a client opening Responses over a WebSocket) is answered 426, a plain one 405: Responses is relayed over HTTP only. Every route is wrapped in counted, callerGuard, withCaller and keyLimited.	gateway.go
Model resolution	serve resolves the model the agent asked for: Claude tier stand-ins, a routing group (provider.GroupFor), auto stand-ins, then provider.Resolve. An unknown model gets a 404 naming why: a switched-off provider, a disabled group or an empty group.	gateway.go, internal/provider/group.go
Candidates	plan lists one provider's candidates: one per key or account on, then each Fallback model, with resting ones last (restLast). planGroup and planLevel weigh a group's members together. A nested group plans by its own routing (#576). A member's own fallbacks are not the group's.	fallback.go
Order of keys and accounts	Routing: smart (""), order, rotate, usage (least used, with an usageHalfLife of 1h), pace or weight (weightedFirst). Smart, usage and pace weigh a subscription by its allowance (allowances), and a key by its own usage windows when magpie reads them (keyAllowance: a sub2api key given a 5-hour, day or 7-day limit, or a key bought as a plan, such as Kimi Code's or a GLM Coding Plan's); any other key has none. A lone key or account is weighed against nobody, but what it has left is still read into the trace (weighRouted), so the Routing page shows it as it does for several (#1016); all but used up, it keeps its conversation rather than being said to move (affine). A failure rests a candidate (failure, failureOf).	routing.go, weighted.go, internal/provider/routing.go
Sink	With Sink on, a key or account that answers 429 while it still has quota goes to the back, behind every one not rate limited since. Held in memory until restart.	sink.go
Affinity	A conversation stays with the key or account that answered it, so the vendor's prompt cache is read again. Modes: auto (""), session, turn, off. Stored in affinity.json next to providers.json.	affinity.go
One try	attempt sends one candidate the request. It goes through the provider's proxy (p.Via). A subscription with its own client is served specially, such as Claude through the genuine Claude Code. The request is relayed as is (passthrough) when the model is usable on the agent's API, except that a Claude model where the provider has an Anthropic URL is translated to Messages from Chat or Responses too (OnMessages: OpenCode speaks Chat for every model, and a relay drops cache_control there), unless the user set the model's API or the vendor's list names its APIs. A request offering web search to a provider that doesn't search by itself is translated too, so magpie can search for it (search.go), but on the agent's own API where the provider serves it and no ResponsesFirst/MessagesFirst rule prefers another: Codex offers web_search on every turn, and each went out as Chat (#997). Otherwise it is translated (translate) to the first API usable gives: Chat, except Responses for OpenAI's models where their maker serves them (ResponsesFirst) and Anthropic's Messages for a Claude model where the provider has an Anthropic URL (MessagesFirst, which keeps its cache breakpoints, #997). Provider.Native, the API agents with a per-model API (Pi, omp, droid) are wired to ask on, follows the same order.	gateway.go
Trace	The Routing view's record of each decision, written where the decision is made: candidates, rests, which try answered.	trace.go

Runtime path

1. handle(proto) reads the body and the model (requestBody, requestModel), then calls serveAgent. serveAgent runs the middleware chain, then serve.
2. serve redacts secrets in the request (redacted), resolves the model to a provider or a group, and refuses what can't be served. A request whose provider and fallbacks, or whose group's models, all skip redaction (unredactedRoute, provider.SkipsRedaction: set Send requests unmasked, Unredacted, with every address on this machine or the local network) goes to them as the agent wrote it; the log and recent calls keep the masked body. count_tokens and embeddings follow the same rule.
   • DecidesModel gives a 400.
   • A gateway key held to some models (keyHolds, #882) gets a 403 for any other model.
   • Sealed tasks (#619) are checked here.
   • Rules and image inputs are handled here too.
3. Candidates come from planGroup for a group or plan for a provider. They are filtered by sealedReaders and allowedCandidates. An account past its share is held out of the candidates (capHeld): a usage cap's share, or 100% of a window for a Codex account set not to spend its credits (provider.HoldShare, CodexNoCredits). When every candidate is held and one held for its credits spends its resets by itself, autoReset spends a reset on it and it is tried (planned.held); otherwise a capped or barred set returns cappedError or barredError. Then affine moves the conversation's last answerer first, unless it is resting or nearly used up. An X-Magpie-Account header (AccountHeader) pins the request to one account: only that account is tried, and when it can't answer the caller is told why.
4. The candidate loop calls attempt on each candidate in turn, through a holdWriter. The writer holds the reply until it is clearly passing. A candidate that fails before anything is sent rests (failure): out of credit for 30 minutes, out of quota until reset, rate limited until retry-after, otherwise a minute and longer on repeats. The next candidate is then tried. A failed provider waits fallbackCooldown (1 minute) at the back. The vendor's safety filter refusing a reply rests nobody, whether it refuses the turn before any of the reply is said or once some of it has been said (refused, refusedAfter).
5. The last candidate's error reaches the agent. A try that answered is recorded (served, answered) for routing, affinity and usage.

Constraints and failure behavior

• A fallback happens only while none of the reply has been sent. An agent never gets half a reply from one upstream and the rest from another.
• A 2xx that can't be an API's answer is the 502 it stands for (notAnAPIReply, #1012): a web page (text/html, or a body that begins as one: a sign-in page, Cloudflare's challenge), a body with nothing in it, or one sent as JSON that doesn't begin as JSON. Every upstream request goes through it (forwardOnce, and the built-in clients' own: the ChatGPT backend's /responses, Kiro, Zed, Qoder, Devin, Cursor, Command Code), so it fails over, rests and is logged as a 502 does, its reason naming the page's title. Only the first bytes are peeked, which a stream waits for anyway, and a compressed body is left as it came.
• A streamed request is never left silent while its vendor says nothing. holdWriter.watch runs beside each try: past keepHeldAfter (15s) it sends a held try's agent the stream's 200 and SSE comments (keepAlive), none of the try, so another candidate may still answer in the same stream; once the reply passes, it puts a comment between two of its lines after keepaliveEvery of quiet. It stops after keepaliveLongest (5 minutes) with nothing from the vendor. A proxy in front such as Cloudflare (125s) otherwise ends the request (#947). Gemini streams get no comments (#934).
• Claude's, GPT's and Gemini's reasoning is held (refusesAfterThinking, up to holdThinking, 4 minutes) so that a refusal after it goes to the next candidate unseen (#248). It is held only while one could: on the last candidate a refusal isn't asked again, so Claude's and GPT's reasoning streams as it comes there, and a refusal after it reaches the agent as the vendor's own. Held there, a lone relay's Claude or GPT showed Claude Code its thinking all at once with the text, and Codex none until then (iTianbao on X). Gemini's stays held on the last candidate too, for a reply that only reasoned to be asked again (#667). Any other model's reasoning is never held.
• The order of the steps a request's accounts go through, how much the accounts' own order (Make first) counts under each routing, and how often an allowance is read are described for users in How magpie picks an account. Smart compares renewals window by window, biggest first, each truncated to the hour (weighRouted), so the order decides only when every window falls in the same hour. Affinity keeps a conversation on an account at 90–97%: only SpentShareOf (98%, 100% In order) stops it (affine), while the 90% tier (lowShare) orders only conversations nobody has answered.
• A resting candidate is tried last, never dropped (restLast). A lone candidate is tried even while it rests, because there is no other.
• Routing decides who goes first only when nobody has answered the conversation yet. Affinity keeps the answerer while its cache is worth it. Auto mode keeps it across turns while at least cacheWorth (1024) tokens were read from the cache and the cache isn't cold (cacheCold, 5 minutes).
• In turn (rotate), a conversation nobody has answered yet goes to the key, account or model the fewest other conversations of the provider's model or the group answered in the last heldFor (30 minutes) are on (leastHeld, #946). Ties, resting ones and a group with a group in it keep routing's order. With session affinity, two sessions of one group so stay on different models.
• A ChatGPT account that holds credits answers past its windows on them; the vendor spends them. That is the default. Set not to spend them (Usage card's Use credits, magpie quota credits <account> off), the account is held at 100% of a window till it renews, by magpie's latest reading of its windows (provider.Allowances, refreshed about every minute), so the other accounts, groups and fallbacks take the request. A cap set on the account (1–99%) takes precedence. Unlike a cap, the hold is at the vendor's own 100%, so an account allowed to spend its resets spends one there, both when every candidate is held and after the last one is refused. A reset made forgets the account's cached allowance (renewedNow), so neither hold outlives the windows it started again.
• Different failures are classified separately: out of credit, out of quota, rate limit and other errors rest for different lengths. A fix must not merge them into one bucket (see #873).
• A sub2api key out of its own limit answers 429 api key 7天限额已用完 (or 日限额, 5小时限额) as rate_limit_exceeded. It is out of quota (plannedWords), not rate limited: it rests till the window it filled renews when its windows as last read say so, else quotaRest, and its windows are read again (staleKeyAllowance). A reading that finds that window full no more (the limit raised in the relay's panel, or the key's usage reset) lifts the rest (renewed, told through provider.OnRenewed), as a Codex reset spent does an account's; the key's other rests stay. While it rests out of its windows, they are read again each time it is planned (restingOf, once a minute at most), so this holds where nothing weighs it by them: an ordered group. The relay out of accounts for everyone (503 No available accounts, or a later version's 429 All available accounts are currently rate-limited) says nothing of the key's windows: it backs off as any failure or rate limit does, never to the window. A key's windows count only its own spend, so other users draining the relay's accounts show up as those errors, not in the windows.
• Only an account is held at a usage cap (capHeld) or rested out of one model's pool (pooled); a key's windows order it and rest it, nothing more.
• Codex offers its image tool (the image_gen namespace, #870) to every magpie provider. When a provider other than a ChatGPT sign-in or openai.com answers a bare 400, attempt asks it once more without the tool (image_tool.go). This happens both passed through and translated. If that works, the provider is marked unfit for the tool (tool:image_gen) and isn't offered it again. If it doesn't, the tool is restored and the 400 stands (#949).
• A translated Responses reply gives Codex reasoning items magpie made: an rs_ id and a summary, nothing sealed. Codex hands them back on every turn. A request relayed as is to OpenAI's API or Azure OpenAI goes without them (withoutBareReasoning) unless store is true, because both look the id up among stored items and answer 400 "Item with id 'rs_…' not found" (#1008). Any other Responses upstream gets them, because a DeepSeek model wants its reasoning back (#388). When one answers 400 naming the input (refusesInput: param is input, or the error names an item by id), as a relay in front of OpenAI does with a message of its own, passthrough asks once more without them and, once that works, leaves them out for that provider and model from then on (bareReasoningRefused, #1044). Codex's compaction offers no tools, so it is relayed even when its turns were translated. When a provider answers its summary request 404, or 400 for an item it didn't find (itemNotFound, a relay in front of either), codexCompact asks once more with the conversation as text (plainCompact), then compacts locally (#866, #1008).
• Group members are provider/model[:effort]. A member with a fixed effort is asked for that effort, bounded by the group's levels (#671).
• An agent whose config can't carry an effort has one picked in magpie (provider.AgentEffort, settings' agentEfforts; Cursor Private Inference's row, #1003). It is asked for on that agent's requests to a model that reasons (Provider.Thinks), in place of the level the request sent, fitted to the model's levels and bounded by the group's. Order: a member's fixed effort, then a model's :effort suffix, then the turn's pick (a routing rule's), then the agent's pick, then the request's own. A model that doesn't reason is sent as asked; the default (none picked) leaves every request as it came. The pick beats the request's own because Cursor sends its own default as often as a real choice. Picking Default hands the choice back to Cursor.

Verification

go test -tags nogui ./internal/gateway -run 'TestQuiet|TestSilentHeld|TestRotateSpreadsSessions|TestRouting|TestSmartRouting|TestFallback|TestNoFallbackForOtherErrors|TestLastFallbackErrorReachesTheAgent|TestRateLimitedSinksToTheBack|TestGroupRateLimitedSinks|TestAffinity|TestSeveralKeysOnTakeOverFromEachOther|TestSubscriptionAccountsTakeOver|TestGroup|TestTrace|TestImageTool|TestCodexNoCredits|TestClaudeGoesOnMessagesWhereTheRelayHasThem|TestAutoRelaysOnTheClientsOwnAPI|TestSearchOfferedStaysOnTheClientsAPI|TestCodexSearchAnsweredOnResponses|TestAgentEffortOnCursorLocal|TestAzureCompactionWithoutMagpiesReasoning|TestCodexCompact|TestNotAnAPIReply|TestAPIReplyUntouched|TestCodexBackendWebPage|TestSub2APIKeyLimitIsQuota|TestKeysWeighedByTheirWindows|TestGroupWeighsKeyWindows|TestKeyOutOfItsWindowRestsUntilReset|TestKeyPoolErrorDoesNotRestForWindow|TestKeyBackOnceItsWindowIsNotFull|TestRestingKeyReadAgainInAnOrderedGroup|TestThinkingStreams|TestHeldThinkingKeepsTheAgentAlive|TestRefusalAfterThinkingFailsOver|TestGeminiEmptyRepliesAskedAgain|TestLocalUnredacted'
go test -tags nogui ./internal/provider -run 'TestGroup|TestCodexCreditsSwitch|TestRenewedAccountForgetsItsAllowance|TestNative|TestKeyAllowance|TestPlanKeyAllowance|TestSkipsRedaction'

The gateway's TestMain gives the package a home of its own, so these tests never read real providers or accounts.