yetone
7f62dbf3ed
#1438 (yhong91): GET /v1/models listed grok/grok-4.7 with context_window 256000 (the Grok backend's own) and max_output_tokens 500000 (models.dev's x-ai entry, which catalog.Decorate fills in when the backend gives no output). A client that reads both sizes its replies past what the window holds. The agents' files already kept output within the window (maxTokens, #338/#347), but the gateway's lists, the GUI and the CLI did not. What changed: - provider.OutputWithin(window, output) and Entry.PublishedOutput(): the output is kept within the window; an unknown window or output (0) is left as it is. - Every place that publishes the pair uses it: modelObject (/v1/models), cursorLocalModel (Cursor Private Inference), Muse Code's list, the Gateway page's models and groups (providerInfo, providersState), the agents' maxTokens, and `magpie model output` (which also says "500k, kept within its 200k window" when it lowers one). - Entry.Output stays the model's own limit. The request-path cap (outputLimit/withMaxOutput) and the user's Max output setting are unchanged, so no request is lowered further than before. This is why the clamp is not in replyLimit/entryFor as the issue suggests, and why the commenter's patch (modelObject only) was not taken as is: it was right but missed Cursor, Muse, the GUI and the CLI. - docs/subsystems/gateway-routing.md: the rule and its routes. Tests (each fails without the change): - gateway TestModelListsKeepOutputWithinTheWindow: models.dev 500000 over a backend 256000 window, plus output within the window, unknown window and unknown output, on /v1/models, Cursor's capabilities and Muse's limit; a group request with max_tokens 300000 still reaches upstream as 300000. - provider TestOutputWithin, gui TestProviderModelOutputWithinTheWindow, TestModelOutputCmdWithinTheWindow. Breaking it by hand: OutputWithin returning output unclamped turns all of them red, plus agent TestMaxTokensWithinContextWindow; gateway sites back on e.Output turn the gateway test red on all three lists; moving the clamp into Entry.Output turns the fixture and the group-request guard red. Real data: an offline dump of the owner's catalog (copies of ~/.config/magpie and its caches, 203 entries) has 3 models with output over their window, grok/grok-4.5, 4.6 and 4.7 (256000 vs 500000), now published as 256000; 0 are not Grok. The live gateway on 3425 was not running, so /v1/models itself was not read, and no real Grok request was made. Run: gofmt -l; go vet -tags nogui ./... on darwin, linux and windows; go test -tags nogui ./... -count=1: all green except TestLoopEndsWorkBuddysReply/plugin/anthropic ("let go only after 3207 lines", bound 2000) at load average ~35 with other suites running; it passes 5 of 5 alone and does not touch the output limit or request path.
52 lines
2.0 KiB
Go
52 lines
2.0 KiB
Go
package gateway
|
|
|
|
import (
|
|
"cmp"
|
|
"net/http"
|
|
|
|
"github.com/yetone/magpie/internal/provider"
|
|
)
|
|
|
|
// museContext and museOutput are what Muse is told of a model whose window
|
|
// or reply size magpie doesn't know.
|
|
const (
|
|
museContext = 128000
|
|
museOutput = 32000
|
|
)
|
|
|
|
// museModels serves Muse Code its model list. Muse asks for it at
|
|
// /muse-code/models on its endpoint's host, whatever path the endpoint has,
|
|
// before every session, and starts none when it isn't there, whatever model
|
|
// it is told to use; a model it lists is one whose metadata["muse-code"]
|
|
// says how Muse is to treat it (a row without it is one Muse hides, and a
|
|
// list of none it refuses). The ids are the catalog's, which Muse sends back
|
|
// as the model it asks /v1/responses for.
|
|
func (s *Server) museModels(w http.ResponseWriter, r *http.Request) {
|
|
shown := catalogFor(r)
|
|
labels := provider.Labels(shown)
|
|
data := []map[string]any{}
|
|
for i, e := range shown {
|
|
input := []string{"text"}
|
|
if e.Images {
|
|
input = append(input, "image")
|
|
}
|
|
meta := map[string]any{"name": labels[i], "family": e.Provider.ID, "is_hidden": false,
|
|
"attachment": e.Images, "reasoning": len(e.Efforts) > 0, "temperature": false, "tool_call": true,
|
|
"modalities": map[string]any{"input": input, "output": []string{"text"}},
|
|
"options": map[string]any{"include": []string{}}, "variants": map[string]any{},
|
|
"description": labels[i] + " via magpie"}
|
|
// the window and the most a reply may hold: Muse hides a model
|
|
// whose limit lacks either, and without a limit asks for replies
|
|
// of 128K tokens, more than most models give; one magpie doesn't
|
|
// know is told 128K and Muse's own 32K
|
|
ctx, out := cmp.Or(e.Context, museContext), e.PublishedOutput()
|
|
if out <= 0 {
|
|
out = min(museOutput, ctx)
|
|
}
|
|
meta["limit"] = map[string]any{"context": ctx, "output": out}
|
|
data = append(data, map[string]any{"id": e.ID, "object": "model", "created": 0, "owned_by": e.Provider.ID,
|
|
"metadata": map[string]any{"muse-code": meta}})
|
|
}
|
|
writeJSON(w, 200, map[string]any{"object": "list", "data": data})
|
|
}
|