sessions, gateway: Sessions lists gateway sessions, and can keep their text on this computer when turned on (#1355, mu-zi-lee)
What changed: the Sessions page lists sessions seen only through the gateway (from the usage ledger, read-only, a native session of the same id wins), and shows in gateway mode. A Sessions switch, off by default and confirmed before it turns on, records each gateway exchange's text under the config dir (scrubbed, at most 7 days and 256 MiB, cleaned hourly). Settings sync and restore can't turn it on; POST /api/sessions/recording turns it on or off and can clear what was kept. Why: users of magpie as a gateway (gateway mode, LAN clients) had no way to see their conversations; the owner chose this design on the PR. Squashed from PR head 6a343ba1 onto origin/main. Conflicts: gui-shell.md (kept main's placeMenu text, took the PR's Sessions in gateway mode) and settings_keep_test.go (main's fields plus GatewayConversations). Run: go build ./..., go vet on gateway, gui, sessions, settings, usage; go test -race -timeout 900s on those five packages: all ok.
This commit is contained in:
1 parent
12c0550ae7
commit
c9b8263378
39 files changed
+2762
-118
No files matched your search
@@ -8,6 +8,7 @@ These pages describe current responsibilities, runtime paths, state, and contrac
|
||||
| Plugins that run on the gateway's requests and replies | [Gateway middleware](gateway-middleware.md) |
|
||||
| Claude subscription turns across Claude Code runs | [Claude subscription bridge](claude-subscription-bridge.md) |
|
||||
| How the gateway picks a provider, key or account, and falls back | [Gateway routing and fallback](gateway-routing.md) |
|
||||
| Gateway Session Aggregation | [Gateway Session Aggregation](gateway-sessions.md) |
|
||||
| Connecting an agent's config to the gateway, drift, disconnect, WSL twins | [Agent wiring](agent-wiring.md) |
|
||||
| Providers, keys, subscription accounts, refresh and allowance | [Providers and accounts](providers-accounts.md) |
|
||||
| Instructions, MCP servers, skills and RTK written into agents | [Library](library.md) |
|
||||
|
||||
@@ -20,7 +20,7 @@ plugins.json is the source of truth for which middleware is installed and on, in
|
||||
|
||||
## Runtime path
|
||||
|
||||
1. `serveAgent` calls `middleware.Begin` with the request's protocol, model, stream flag, path and agent. With no middleware it is `serve` as before, at no cost beyond the check.
|
||||
1. `serveAgent` establishes the [gateway session identity and optional recording](gateway-sessions.md), then calls `middleware.Begin` with the request's protocol, model, stream flag, path and agent. With no middleware it calls `serve` directly.
|
||||
2. `Run.Request` passes the agent's body, in its own API, to each `onRequest` in turn. An object returned is the body the next middleware and the gateway get; `undefined` keeps the bytes as they were. `ctx.reject(status, message)` stops the chain, and the agent gets that error in its API's shape (`writeError`).
|
||||
3. `Run.Wrap` wraps the response writer when some middleware has `onEvent` or `onResponse`. An SSE reply is split into events. Each event whose data is a JSON object goes to every `onEvent` that wants its name (`export const events`): an object replaces it, `null` drops it, and `undefined` writes the original bytes. `data: [DONE]` and other non-object data pass by. A whole JSON reply is buffered for `onResponse`, with `ctx.status` set. A reply with a `Content-Encoding` passes as-is.
|
||||
4. `Run.End` returns the runtimes to their pools.
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
# Gateway Session Aggregation
|
||||
|
||||
The Sessions API combines file-backed native sessions with a read-only
|
||||
projection of gateway usage records. A gateway session identity is exactly
|
||||
`AgentOf(record.Agent) + record.Session`. Client conversation requests without a
|
||||
session header receive a unique `request-…` identity when recording is enabled.
|
||||
The response returns `X-Magpie-Session`; clients can send that value on later
|
||||
requests to continue the same recorded session. Listed browser origins can
|
||||
read this header through CORS. When recording is disabled, unidentified
|
||||
requests receive no generated identity or response header. Generated
|
||||
`request-…` identities are excluded from usage and Sessions projections.
|
||||
Without reuse, each unidentified recorded request is a separate session: prompt text,
|
||||
caller keys and network addresses are never used to guess conversation ownership.
|
||||
Older ledger records without `Session` remain excluded. The projection is
|
||||
computed from the usage ledger and is not persisted. Window projections and
|
||||
routing summaries reuse versioned log snapshots; appends and rewrites invalidate
|
||||
them, and price/provider changes also invalidate priced projections. Single-session
|
||||
lookups use the cached identity index. Native exclusions are applied per caller.
|
||||
|
||||
When the same identity is present in a native session, the native session wins
|
||||
and the gateway projection is omitted. Gateway projections have no path,
|
||||
resume command, terminal action, or native-file delete capability. Their
|
||||
models, token totals, start/last timestamps and effective prices come from the
|
||||
same usage records used by Usage and stats endpoints.
|
||||
All-time statistics include gateway history older than native files. Calendar
|
||||
activity and output totals use each session's actual request dates, including
|
||||
intermediate dates; extending the range rebases existing native calendar indices.
|
||||
|
||||
Implementation: [`internal/usage/gateway_sessions.go`](../../internal/usage/gateway_sessions.go),
|
||||
[`internal/sessions/external.go`](../../internal/sessions/external.go), and
|
||||
[`internal/gui/sessions.go`](../../internal/gui/sessions.go).
|
||||
|
||||
## Local Gateway Conversation Recording
|
||||
|
||||
The Sessions page offers an explicit, default-off recording switch. Confirmation
|
||||
discloses that prompts, replies and tool results can contain private files and
|
||||
code. `Settings.GatewayConversations` is machine-local consent:
|
||||
`settings.SetGatewayConversations` updates it under the settings file lock.
|
||||
Other Settings saves retain the latest stored consent under that lock, so an
|
||||
older snapshot cannot re-enable recording after it is stopped. `KeepOwn`
|
||||
also prevents sync/restore from enabling it.
|
||||
`POST /api/sessions/recording` changes this switch; `clear: true` disables it and
|
||||
erases only locally recorded gateway content, leaving usage and native files.
|
||||
The existing browser-mode authentication guard protects these APIs; this is an
|
||||
administrator view, not per-caller-key access control.
|
||||
|
||||
[`recordConversation`](../../internal/gateway/conversations.go) wraps
|
||||
`serveAgent` outside protocol translation and gateway middleware. It records
|
||||
one client-facing exchange, not each retry. Both explicit and generated
|
||||
identities are recorded, using the same `usage.AgentOf` identity as the usage
|
||||
projection. Codex's direct `/backend-api/codex/responses` relay is captured too.
|
||||
Internal helper calls, token counting, embeddings, model listings, and compaction
|
||||
paths that do not pass through these entry points are not recorded.
|
||||
Generated identities live in request context, preserving the original headers
|
||||
used by routing and prompt-cache affinity. Automatically assigned identities
|
||||
are neither written to the usage ledger nor forwarded to remote Magpie instances.
|
||||
Reused `request-…` headers are also excluded from the ledger and retain OTel
|
||||
agent-level deduplication when no native session header exists. The existing protocol
|
||||
readers handle Chat, Anthropic, Responses and Gemini; streaming replies use the
|
||||
existing decoders, with completed Responses items retaining custom tool calls.
|
||||
Only content actually sent through the gateway is available. Images are shown
|
||||
as placeholders; hidden reasoning, client-only operations and unsupported wire
|
||||
items cannot be reconstructed. This is not a raw request archive or remote
|
||||
session synchronization, and existing S3 archives are not imported.
|
||||
|
||||
[`SaveGatewayTurn`](../../internal/sessions/gateway.go) stores normalized,
|
||||
scrubbed exchanges in one append-only `turns.jsonl` per
|
||||
`gateway-conversations/<UTC date>/<identity hash>/` under
|
||||
the Magpie config directory, using 0700 directories and 0600 files. Each line is
|
||||
one bounded turn; legacy per-turn JSON files remain readable until they expire.
|
||||
An incomplete final append is omitted from reads (marked truncated) and removed
|
||||
before the next append; complete earlier lines remain available. No headers are
|
||||
stored. Known secrets and secret-named JSON
|
||||
fields are scrubbed, but arbitrary sensitive text is not guaranteed removable.
|
||||
Response capture uses an 8 MiB temporary spool; input is bounded to 8 MiB too.
|
||||
Normalized turns are bounded to 1 MiB on disk, individual parts to 64 KiB,
|
||||
and each daily session file to 256 MiB. A full daily session file rejects further
|
||||
recording that day. Hourly background cleanup removes expired buckets and the
|
||||
oldest files above 256 MiB; this is a soft store limit between cleanup passes.
|
||||
Saving a turn does not rescan the whole store, and background traversal and
|
||||
sorting do not hold the save lock.
|
||||
Capture, parsing or persistence failure does not change the model response;
|
||||
filesystem failures are logged without conversation bodies.
|
||||
|
||||
Content older than seven days is excluded from reads. The serving gateway
|
||||
cleans up expired UTC-day buckets at startup and hourly, including with recording
|
||||
off; the daily layout means physical cleanup can lag the read cutoff by one day.
|
||||
Disable-and-clear and writes share a lock and recheck the consent generation,
|
||||
preventing an in-flight request from refilling a cleared store even if recording
|
||||
is enabled again before that request completes.
|
||||
|
||||
`GET /api/sessions/transcript` still prefers a matching native session. Otherwise
|
||||
it reads a known gateway session's last 128 retained exchanges. Generated
|
||||
recording identities do not create rows in Sessions, including for agents whose
|
||||
native files provide their own sessions. A full exact
|
||||
previous-history prefix is omitted from the next request, while repeated new
|
||||
messages are preserved. System/tool-definition context is separately collapsed
|
||||
when unchanged. Compacted, edited or incremental contexts may repeat material;
|
||||
they are never guessed into a single canonical history. The existing transcript
|
||||
display bounds apply and truncation is marked. Empty/expired/missing content
|
||||
has an explicit empty state and all gateway transcripts identify their source.
|
||||
The page invalidates pending transcript reads on reload or clear. A completed
|
||||
read updates the current open transcript after redraw, and an older response
|
||||
cannot replace content fetched after clearing the store.
|
||||
|
||||
Verification: `TestGatewayConversation*`, `TestGatewaySessionHistoryAndCalendar`,
|
||||
`TestGatewaySessionsNativeWinsBeforeLimits`,
|
||||
`TestGatewayStatsRebaseNativeCalendar`, and `TestCORSKeyThroughTheServer`; browser
|
||||
`gateway-conversations.test.cjs`, `sessions-talk.test.cjs`, and locale checks.
|
||||
@@ -14,7 +14,7 @@ three.
|
||||
| Native host | `Run` starts the Wails v3 app: the tray icon, the panel, the main window, single-instance handover, `magpie://` import links and open at login. `host` implements `Windows` for the API: show and hide, open URLs and folders, copy to the clipboard, fit and tint the panel, and set the text size. | [`app.go`](../../internal/gui/app.go), [`mainwindow.go`](../../internal/gui/mainwindow.go), [`traymenu.go`](../../internal/gui/traymenu.go) |
|
||||
| Backend | `startBackend` starts the gateway this process serves, unless another magpie already serves it. In that case this process only shows the other one's status (`gw` is nil). `stopServing` drains requests in flight. | [`backend.go`](../../internal/gui/backend.go) |
|
||||
| API and assets | `Handler` serves `assets/` (embedded with `//go:embed assets`) and the `/api/*` routes, such as `/api/state`, `/api/set` and `/api/agents/{action}/{id}`. `held` lets one API request's reads share one catalog build (`provider.Hold`), and drops it after any write. `revalidated` and `versionedPage` name each script by its content hash, so a cache never serves an old `app.js` under a new page. | [`api.go`](../../internal/gui/api.go) |
|
||||
| Browser mode | `StartWeb` serves the page and the gateway for `magpie web`. The browser signs in with a run key in a link. Gateway mode shows only Providers, Gateway, Routing, Usage, Plugins and Settings on a machine that is only a gateway. | [`web.go`](../../internal/gui/web.go), [`gatewaymode.go`](../../internal/gui/gatewaymode.go) |
|
||||
| Browser mode | `StartWeb` serves the page and the gateway for `magpie web`. The browser signs in with a run key in a link. Gateway mode shows Providers, Gateway, Routing, Usage, Sessions, Plugins and Settings on a machine that is only a gateway. | [`web.go`](../../internal/gui/web.go), [`gatewaymode.go`](../../internal/gui/gatewaymode.go) |
|
||||
| The page | `app.js` keeps one state object per view and renders it into a list, with no framework. The views have their own files: `routing.js`, `library.js`, `plugins.js`, `sessions.js` and `context.js` (Usage's Context tab, and the context window card the Routing page draws under a request's story, fed by `/api/context` in `context.go`, which scores and tags each agent). Strings go through `t()` with translations in `i18n.js` (zh, zh-TW, ja and de; the English text is the key and the fallback). zh-TW is Traditional Chinese with Taiwan's words; a system language of zh-TW, zh-HK, zh-MO or any zh-Hant picks it, and the tray menu and notifications follow (`trayLang`). Dropdowns use `openProtoMenu`. Every app menu (`openProtoMenu`, `openRowMenu`, `openSessCombo`) is placed by `placeMenu`: under its anchor, or over it when the window runs out below; one that fits neither way takes the larger room, its height held to it, and scrolls inside itself, and a scroll inside a menu doesn't close it (#1437, `removed-groups-menu-scroll.test.cjs`). Scrolling is done only by `scrollOnPurpose(e)` with the reader's event. | [`assets/app.js`](../../internal/gui/assets/app.js), [`assets/i18n.js`](../../internal/gui/assets/i18n.js) |
|
||||
| No GUI build | `-tags nogui` builds magpie without the GUI: `hasGUI` is false and `runGUI` says to use `magpie tui`. The files of `internal/gui` that need Wails build under `!nogui`. | [`gui_off.go`](../../gui_off.go) |
|
||||
|
||||
|
||||
@@ -149,6 +149,12 @@ func (s *Server) codexBackend(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
}
|
||||
if r.Method == http.MethodPost && rest == "/responses" {
|
||||
r = withGatewaySession(w, r)
|
||||
var recorded func()
|
||||
w, recorded = recordConversation(w, r, provider.Responses, body)
|
||||
defer recorded()
|
||||
}
|
||||
s.codexUpstream(w, r, rest, body)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,266 @@
|
||||
package gateway
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/json"
|
||||
"log"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/yetone/magpie/internal/provider"
|
||||
"github.com/yetone/magpie/internal/redact"
|
||||
"github.com/yetone/magpie/internal/sessions"
|
||||
"github.com/yetone/magpie/internal/settings"
|
||||
"github.com/yetone/magpie/internal/usage"
|
||||
)
|
||||
|
||||
const conversationBodyMax = 8 << 20
|
||||
|
||||
type gatewaySessionKey struct{}
|
||||
|
||||
// withGatewaySession assigns unidentified recorded requests a content identity.
|
||||
// Keep it out of the input headers so routing still uses the client's context.
|
||||
func withGatewaySession(w http.ResponseWriter, r *http.Request) *http.Request {
|
||||
id := sessionOf(r.Header)
|
||||
if id == "" && settings.Load().GatewayConversations {
|
||||
id = "request-" + rand.Text()
|
||||
r = r.WithContext(context.WithValue(r.Context(), gatewaySessionKey{}, id))
|
||||
}
|
||||
if id != "" {
|
||||
w.Header().Set(SessionHeader, id)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func gatewaySessionOf(r *http.Request) string {
|
||||
if id := sessionOf(r.Header); id != "" {
|
||||
return id
|
||||
}
|
||||
id, _ := r.Context().Value(gatewaySessionKey{}).(string)
|
||||
return id
|
||||
}
|
||||
|
||||
func init() {
|
||||
WhileServing = append(WhileServing, func(ctx context.Context) {
|
||||
tick := time.NewTicker(time.Hour)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
if err := sessions.PruneGatewayConversations(time.Now()); err != nil {
|
||||
log.Printf("gateway conversation cleanup failed: %v", err)
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-tick.C:
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
type conversationWriter struct {
|
||||
*captureResponseWriter
|
||||
status int
|
||||
}
|
||||
|
||||
func (w *conversationWriter) WriteHeader(status int) {
|
||||
if w.status == 0 {
|
||||
w.status = status
|
||||
}
|
||||
w.ResponseWriter.WriteHeader(status)
|
||||
}
|
||||
|
||||
// recordConversation wraps the client-facing path, outside middleware and
|
||||
// translation, so retries never become duplicate assistant replies.
|
||||
func recordConversation(w http.ResponseWriter, r *http.Request, proto provider.Protocol, body []byte) (http.ResponseWriter, func()) {
|
||||
id := gatewaySessionOf(r)
|
||||
generation := sessions.GatewayRecordingGeneration()
|
||||
if !settings.Load().GatewayConversations || id == "" {
|
||||
return w, func() {}
|
||||
}
|
||||
at := time.Now()
|
||||
input := append([]byte(nil), body[:min(len(body), conversationBodyMax)]...)
|
||||
sink := &conversationWriter{captureResponseWriter: &captureResponseWriter{
|
||||
ResponseWriter: w, full: &spool{limit: conversationBodyMax, pattern: "magpie-conversation-*"},
|
||||
}}
|
||||
return sink, func() {
|
||||
defer sink.full.discard()
|
||||
cut := len(body) > conversationBodyMax || sink.full.cut()
|
||||
output, err := sink.full.read()
|
||||
if err != nil {
|
||||
log.Printf("gateway conversation capture failed: %v", err)
|
||||
return
|
||||
}
|
||||
turn := conversationTurn(proto, input, output)
|
||||
turn.Agent, turn.Session, turn.Time = usage.AgentOf(callerOf(r).agent), id, at
|
||||
turn.Status = sink.status
|
||||
if turn.Status == 0 {
|
||||
turn.Status = http.StatusOK
|
||||
}
|
||||
turn.Cut = turn.Cut || cut
|
||||
if err := sessions.SaveGatewayTurnGeneration(turn, generation); err != nil {
|
||||
log.Printf("gateway conversation save failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func conversationParts(role string, parts []Part) []sessions.Part {
|
||||
var out []sessions.Part
|
||||
for _, p := range parts {
|
||||
q := sessions.Part{Role: role, Text: p.Text, Kind: string(p.Kind), Name: p.Name}
|
||||
switch p.Kind {
|
||||
case ToolCall:
|
||||
q.Kind, q.Text = "tool_use", string(redact.ScrubJSON(p.Args))
|
||||
case ToolResult:
|
||||
q.Role = "tool"
|
||||
if len(p.Images) > 0 {
|
||||
q.Text += "\n[image]"
|
||||
}
|
||||
case Image:
|
||||
q.Text = "[image]"
|
||||
case File:
|
||||
q.Kind, q.Text = "context", attachmentText(p)
|
||||
case Search:
|
||||
q.Kind, q.Name = "tool_use", "web_search"
|
||||
for _, h := range p.Hits {
|
||||
q.Text += "\n" + h.Title + " " + h.URL
|
||||
}
|
||||
}
|
||||
if q.Text != "" {
|
||||
out = append(out, q)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func conversationTurn(proto provider.Protocol, input, output []byte) sessions.GatewayTurn {
|
||||
t := sessions.GatewayTurn{Input: []sessions.Part{}, Output: []sessions.Part{}}
|
||||
input, output = redact.ScrubJSON(input), redact.ScrubJSON(output)
|
||||
q, err := parse(proto, input)
|
||||
if err != nil {
|
||||
t.Cut = true
|
||||
} else {
|
||||
t.Model = q.Model
|
||||
if q.System != "" {
|
||||
t.Input = append(t.Input, sessions.Part{Role: "system", Kind: "context", Name: "System prompt", Text: q.System})
|
||||
}
|
||||
// Keep tool definitions as context, including native tools the IR may
|
||||
// not translate; never render embedded images or execute their content.
|
||||
var raw map[string]json.RawMessage
|
||||
if json.Unmarshal(input, &raw) == nil && len(raw["tools"]) > 0 {
|
||||
t.Input = append(t.Input, sessions.Part{Role: "system", Kind: "context", Name: "Tools", Text: string(raw["tools"])})
|
||||
}
|
||||
for _, m := range q.Messages {
|
||||
t.Input = append(t.Input, conversationParts(m.Role, m.Parts)...)
|
||||
}
|
||||
}
|
||||
trim := bytes.TrimSpace(output)
|
||||
if json.Valid(trim) {
|
||||
t.Output, err = conversationJSON(proto, trim)
|
||||
t.Cut = t.Cut || err != nil
|
||||
} else if bytes.Contains(trim, []byte("data:")) {
|
||||
var c collector
|
||||
var completedItems []json.RawMessage
|
||||
decode := decoder(proto)
|
||||
ended := false
|
||||
err = readSSE(bytes.NewReader(output), func(_ string, data string) error {
|
||||
if data == "[DONE]" {
|
||||
ended = true
|
||||
return nil
|
||||
}
|
||||
if !json.Valid([]byte(data)) {
|
||||
t.Cut = true
|
||||
return nil
|
||||
}
|
||||
if proto == provider.Responses {
|
||||
var e struct {
|
||||
Type string `json:"type"`
|
||||
Item json.RawMessage `json:"item"`
|
||||
Response struct {
|
||||
Output []json.RawMessage `json:"output"`
|
||||
} `json:"response"`
|
||||
}
|
||||
_ = json.Unmarshal([]byte(data), &e)
|
||||
if e.Type == "response.output_item.done" {
|
||||
completedItems = append(completedItems, e.Item)
|
||||
}
|
||||
if e.Type == "response.completed" && len(e.Response.Output) > 0 {
|
||||
completedItems = e.Response.Output
|
||||
}
|
||||
}
|
||||
return decode(data, func(e Event) {
|
||||
if e.Kind == KStop {
|
||||
ended = true
|
||||
}
|
||||
c.add(e)
|
||||
})
|
||||
})
|
||||
t.Output = conversationParts("assistant", c.finish().Parts)
|
||||
if proto == provider.Responses && ended && len(completedItems) > 0 {
|
||||
b, _ := json.Marshal(map[string]any{"output": completedItems})
|
||||
if parts, e := conversationJSON(proto, b); e == nil {
|
||||
t.Output = parts
|
||||
} else {
|
||||
t.Cut = true
|
||||
}
|
||||
}
|
||||
t.Cut = t.Cut || err != nil || !ended
|
||||
if c.err != "" {
|
||||
t.Output = append(t.Output, sessions.Part{Role: "assistant", Kind: "context", Text: c.err})
|
||||
}
|
||||
} else if len(trim) > 0 {
|
||||
t.Cut = true
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// Reuse the request readers for whole replies by placing the actual output in
|
||||
// their message/input shape. The wire content, not a generated summary, is read.
|
||||
func conversationJSON(proto provider.Protocol, body []byte) ([]sessions.Part, error) {
|
||||
var raw map[string]json.RawMessage
|
||||
if err := json.Unmarshal(body, &raw); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if e := raw["error"]; len(e) > 0 && string(e) != "null" {
|
||||
return []sessions.Part{{Role: "assistant", Kind: "context", Text: string(e)}}, nil
|
||||
}
|
||||
var request []byte
|
||||
switch proto {
|
||||
case provider.Chat:
|
||||
var choices []struct {
|
||||
Message json.RawMessage `json:"message"`
|
||||
}
|
||||
if err := json.Unmarshal(raw["choices"], &choices); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var messages []json.RawMessage
|
||||
for _, c := range choices {
|
||||
messages = append(messages, c.Message)
|
||||
}
|
||||
request, _ = json.Marshal(map[string]any{"messages": messages})
|
||||
case provider.Responses:
|
||||
request, _ = json.Marshal(map[string]any{"input": raw["output"]})
|
||||
case provider.Gemini:
|
||||
var c collector
|
||||
if err := decoder(proto)(string(body), c.add); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return conversationParts("assistant", c.finish().Parts), nil
|
||||
default:
|
||||
request, _ = json.Marshal(map[string]any{"messages": []any{map[string]any{"role": "assistant", "content": raw["content"]}}})
|
||||
}
|
||||
q, err := parse(proto, request)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var parts []sessions.Part
|
||||
for _, m := range q.Messages {
|
||||
parts = append(parts, conversationParts("assistant", m.Parts)...)
|
||||
}
|
||||
if strings.TrimSpace(q.System) != "" {
|
||||
parts = append(parts, sessions.Part{Role: "assistant", Kind: "context", Text: q.System})
|
||||
}
|
||||
return parts, nil
|
||||
}
|
||||
@@ -0,0 +1,196 @@
|
||||
package gateway
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/yetone/magpie/internal/provider"
|
||||
"github.com/yetone/magpie/internal/sessions"
|
||||
"github.com/yetone/magpie/internal/usage"
|
||||
)
|
||||
|
||||
func TestGatewayConversationCapture(t *testing.T) {
|
||||
fresh(t)
|
||||
serveOn(t, "fake", "k", []string{"m1"}, archiveVendor{})
|
||||
s := New()
|
||||
post := func(id string) {
|
||||
r := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(`{"model":"fake/m1","messages":[{"role":"user","content":"hello"}]}`))
|
||||
r.Header.Set("User-Agent", "claude-cli/2.1.0")
|
||||
r.Header.Set("X-Magpie-Session", id)
|
||||
w := httptest.NewRecorder()
|
||||
s.Handler().ServeHTTP(w, r)
|
||||
if w.Code != 200 || !strings.Contains(w.Body.String(), "done") {
|
||||
t.Fatalf("relay changed: %d %s", w.Code, w.Body)
|
||||
}
|
||||
}
|
||||
post("s")
|
||||
agent := usage.AgentOf("claude-cli/2.1.0")
|
||||
tr, err := sessions.GatewayTranscript(agent, "s")
|
||||
if err != nil || tr.Captured != 0 {
|
||||
t.Fatalf("default must be off: %+v %v", tr, err)
|
||||
}
|
||||
if err := sessions.SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
post("s")
|
||||
post("")
|
||||
tr, err = sessions.GatewayTranscript(agent, "s")
|
||||
if err != nil || tr.Captured != 1 || len(tr.Parts) != 2 || tr.Parts[0].Text != "hello" || tr.Parts[1].Text != "done" {
|
||||
t.Fatalf("captured: %+v %v", tr, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationCodexDirectRelay(t *testing.T) {
|
||||
fresh(t)
|
||||
chatgpt(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Header.Get(SessionHeader) != "" {
|
||||
t.Error("gateway identity leaked to vendor")
|
||||
}
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
io.WriteString(w, sse(`data: {"type":"response.completed","response":{"id":"r1","status":"completed","output":[{"type":"message","role":"assistant","content":[{"type":"output_text","text":"direct answer"}]}],"usage":{"input_tokens":9,"output_tokens":2}}}`))
|
||||
})
|
||||
if err := sessions.SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := httptest.NewRequest("POST", CodexPath+"/responses", strings.NewReader(`{"model":"gpt-5.5","stream":true,"input":"direct question"}`))
|
||||
r.Header.Set("Authorization", "Bearer chatgpt-token")
|
||||
r.Header.Set("User-Agent", "codex/1")
|
||||
w := httptest.NewRecorder()
|
||||
New().Handler().ServeHTTP(w, r)
|
||||
id := w.Header().Get(SessionHeader)
|
||||
tr, err := sessions.GatewayTranscript("codex", id)
|
||||
if w.Code != 200 || err != nil || tr.Captured != 1 || len(tr.Parts) != 2 || tr.Parts[1].Text != "direct answer" {
|
||||
t.Fatalf("direct relay missing from Sessions: status=%d id=%q transcript=%+v error=%v", w.Code, id, tr, err)
|
||||
}
|
||||
if id == "" {
|
||||
t.Fatal("recorded direct relay should expose its session identity")
|
||||
}
|
||||
if _, ok := usage.GatewaySessionByID("codex", id, nil); ok {
|
||||
t.Fatal("generated request identity must stay out of the session ledger")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationIdentityKeepsRoutingContext(t *testing.T) {
|
||||
fresh(t)
|
||||
if err := sessions.SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := httptest.NewRequest("POST", "/v1/chat/completions", nil)
|
||||
input := []byte(`{"messages":[{"role":"user","content":"same prompt"}]}`)
|
||||
before := convOf(r.Header, input)
|
||||
r = withGatewaySession(httptest.NewRecorder(), r)
|
||||
if gatewaySessionOf(r) == "" {
|
||||
t.Fatal("recording identity missing")
|
||||
}
|
||||
if r.Header.Get(SessionHeader) != "" || convOf(r.Header, input) != before {
|
||||
t.Fatal("automatic ledger identity changed routing conversation")
|
||||
}
|
||||
forward := httptest.NewRequest("POST", "/", nil)
|
||||
passOnCaller(r.Context(), forward)
|
||||
if forward.Header.Get(SessionHeader) != "" {
|
||||
t.Fatal("recording identity forwarded to remote Magpie")
|
||||
}
|
||||
r.Header.Set("session_id", "native")
|
||||
r.Header.Set(SessionHeader, "override")
|
||||
if gatewaySessionOf(r) != "override" || nativeSessionOf(r.Header) != "native" {
|
||||
t.Fatal("client identities lost precedence")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationAutomaticRequestSession(t *testing.T) {
|
||||
fresh(t)
|
||||
serveOn(t, "fake", "k", []string{"m1"}, archiveVendor{})
|
||||
_, secrets := newCaller(t, "NAS client")
|
||||
server := lanGuard(New().Handler())
|
||||
post := func(header, id string) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(`{"model":"fake/m1","messages":[{"role":"user","content":"hello"}]}`))
|
||||
r.RemoteAddr = "192.168.1.19:12345"
|
||||
r.Header.Set("Authorization", "Bearer "+secrets[0])
|
||||
r.Header.Set("User-Agent", "review-client/1")
|
||||
if header != "" {
|
||||
r.Header.Set(header, id)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
server.ServeHTTP(w, r)
|
||||
if w.Code != http.StatusOK || !strings.Contains(w.Body.String(), "done") {
|
||||
t.Fatalf("relay changed: %d %s", w.Code, w.Body)
|
||||
}
|
||||
return w
|
||||
}
|
||||
if got := post("", "").Header().Get(SessionHeader); got != "" {
|
||||
t.Fatalf("recording off generated session %q", got)
|
||||
}
|
||||
if err := sessions.SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
first := post("", "").Header().Get(SessionHeader)
|
||||
second := post("", "").Header().Get(SessionHeader)
|
||||
if first == "" || second == "" || first == second {
|
||||
t.Fatalf("recorded requests need distinct IDs: %q %q", first, second)
|
||||
}
|
||||
if _, ok := usage.GatewaySessionByID("review-client", first, nil); ok {
|
||||
t.Fatal("generated identity entered Sessions")
|
||||
}
|
||||
usage.Visit(time.Time{}, func(r usage.Record) {
|
||||
if r.Session != "" {
|
||||
t.Errorf("generated identity entered ledger: %q", r.Session)
|
||||
}
|
||||
})
|
||||
tr, err := sessions.GatewayTranscript("review-client", first)
|
||||
if err != nil || tr.Captured != 1 {
|
||||
t.Fatalf("recorded request missing: %+v %v", tr, err)
|
||||
}
|
||||
for range 2 {
|
||||
if got := post(SessionHeader, first).Header().Get(SessionHeader); got != first {
|
||||
t.Fatalf("session continuity lost: %q", got)
|
||||
}
|
||||
}
|
||||
tr, err = sessions.GatewayTranscript("review-client", first)
|
||||
if err != nil || tr.Captured != 3 || len(tr.Parts) != 6 {
|
||||
t.Fatalf("reused session content missing: %+v %v", tr, err)
|
||||
}
|
||||
if got := post("session_id", "native-session").Header().Get(SessionHeader); got != "native-session" {
|
||||
t.Fatalf("native identity replaced: %q", got)
|
||||
}
|
||||
generated := post("", "").Header().Get(SessionHeader)
|
||||
tr, err = sessions.GatewayTranscript("review-client", generated)
|
||||
if err != nil || tr.Captured != 1 || len(tr.Parts) != 2 {
|
||||
t.Fatalf("automatic session transcript missing: %+v %v", tr, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationProtocols(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
proto provider.Protocol
|
||||
input, output, want string
|
||||
}{
|
||||
{"chat", provider.Chat, `{"messages":[{"role":"system","content":"rules"},{"role":"user","content":"hello"}]}`, `{"choices":[{"message":{"role":"assistant","content":"answer"}}]}`, "answer"},
|
||||
{"anthropic", provider.Anthropic, `{"messages":[{"role":"user","content":"hello"}]}`, `{"content":[{"type":"text","text":"answer"}]}`, "answer"},
|
||||
{"responses", provider.Responses, `{"input":"hello"}`, `{"output":[{"type":"message","role":"assistant","content":[{"type":"output_text","text":"answer"}]}]}`, "answer"},
|
||||
{"chat-stream", provider.Chat, `{"messages":[]}`, "data: {\"choices\":[{\"delta\":{\"content\":\"ans\"}}]}\n\ndata: {\"choices\":[{\"delta\":{\"content\":\"wer\"},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n", "answer"},
|
||||
{"anthropic-tool", provider.Anthropic, `{"messages":[]}`, "data: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"tool_use\",\"id\":\"t\",\"name\":\"bash\",\"input\":{}}}\n\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"{\\\"command\\\":\\\"pwd\\\"}\"}}\n\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"tool_use\"}}\n\n", "pwd"},
|
||||
{"responses-custom", provider.Responses, `{"input":[]}`, "data: {\"type\":\"response.output_item.done\",\"item\":{\"type\":\"custom_tool_call\",\"call_id\":\"t\",\"name\":\"apply_patch\",\"input\":\"patch body\"}}\n\ndata: {\"type\":\"response.completed\",\"response\":{\"status\":\"completed\",\"output\":[]}}\n\n", "patch body"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
turn := conversationTurn(tc.proto, []byte(tc.input), []byte(tc.output))
|
||||
b, _ := json.Marshal(turn.Output)
|
||||
if turn.Cut || !strings.Contains(string(b), tc.want) {
|
||||
t.Fatalf("parsed: %+v", turn)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationScrubsAndMarksIncomplete(t *testing.T) {
|
||||
turn := conversationTurn(provider.Chat, []byte(`{"messages":[{"role":"assistant","tool_calls":[{"id":"t","type":"function","function":{"name":"login","arguments":"{\"password\":\"private-value\"}"}}]}]}`), []byte("data: {\"choices\":[{\"delta\":{\"content\":\"partial\"}}]}\n\n"))
|
||||
b, _ := json.Marshal(turn)
|
||||
if !turn.Cut || strings.Contains(string(b), "private-value") || !strings.Contains(string(b), "partial") {
|
||||
t.Fatalf("scrub or incomplete failed: %s", b)
|
||||
}
|
||||
}
|
||||
@@ -42,6 +42,7 @@ func corsGuard(next http.Handler) http.Handler {
|
||||
}
|
||||
h := w.Header()
|
||||
h.Set("Access-Control-Allow-Origin", origin)
|
||||
h.Add("Access-Control-Expose-Headers", SessionHeader)
|
||||
h.Add("Vary", "Origin")
|
||||
if r.Method == http.MethodOptions && r.Header.Get("Access-Control-Request-Method") != "" {
|
||||
h.Set("Access-Control-Allow-Methods", "GET, POST, OPTIONS")
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
|
||||
"github.com/yetone/magpie/internal/access"
|
||||
"github.com/yetone/magpie/internal/provider"
|
||||
"github.com/yetone/magpie/internal/sessions"
|
||||
"github.com/yetone/magpie/internal/settings"
|
||||
)
|
||||
|
||||
@@ -141,6 +142,9 @@ func TestCORSKeyThroughTheServer(t *testing.T) {
|
||||
if err := settings.Save(s); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := sessions.SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv := lanGuard(New().Handler())
|
||||
for _, key := range []string{secret, ""} {
|
||||
r := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(chatReq))
|
||||
@@ -159,6 +163,9 @@ func TestCORSKeyThroughTheServer(t *testing.T) {
|
||||
if w.Code != want || w.Header().Get("Access-Control-Allow-Origin") != "http://localhost:3000" {
|
||||
t.Errorf("key %q: %d %v %s", key, w.Code, w.Header(), w.Body)
|
||||
}
|
||||
if key != "" && (w.Header().Get(SessionHeader) == "" || w.Header().Get("Access-Control-Expose-Headers") != SessionHeader) {
|
||||
t.Error("browser client cannot reuse the gateway session identity")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -315,6 +315,12 @@ func accountOf(p provider.Provider) string {
|
||||
}
|
||||
|
||||
func appendUsage(r *http.Request, rec usage.Record) {
|
||||
if rec.Session == "" {
|
||||
rec.Session = sessionOf(r.Header)
|
||||
}
|
||||
if strings.HasPrefix(rec.Session, "request-") {
|
||||
rec.Session = ""
|
||||
}
|
||||
who := access.Caller(r.Context())
|
||||
rec.CallerKeyID, rec.CallerKeyName = who.KeyID, who.KeyName
|
||||
rec.Local = local(r)
|
||||
|
||||
@@ -13,6 +13,9 @@ import (
|
||||
// (titles, the router's classifier, search and vision stand-ins) go
|
||||
// without them, as their replies are magpie's to read.
|
||||
func (s *Server) serveAgent(w http.ResponseWriter, r *http.Request, from provider.Protocol, body []byte) {
|
||||
r = withGatewaySession(w, r)
|
||||
w, recorded := recordConversation(w, r, from, body)
|
||||
defer recorded()
|
||||
run := middleware.Begin(middleware.Info{Protocol: string(from), Model: modelOf(body), Stream: streamOf(body), Path: r.URL.Path, Agent: agentOf(r)})
|
||||
if run == nil {
|
||||
s.serve(w, r, from, body)
|
||||
|
||||
@@ -66,7 +66,11 @@ func otelSessionOf(h http.Header) string {
|
||||
if id := nativeSessionOf(h); id != "" {
|
||||
return id
|
||||
}
|
||||
return sessionOf(h)
|
||||
id := sessionOf(h)
|
||||
if strings.HasPrefix(id, "request-") {
|
||||
return ""
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
// Claude's tool-less small tasks (titles, suggestions, haiku helpers) may
|
||||
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
|
||||
"github.com/yetone/magpie/internal/access"
|
||||
"github.com/yetone/magpie/internal/provider"
|
||||
"github.com/yetone/magpie/internal/sessions"
|
||||
"github.com/yetone/magpie/internal/usage"
|
||||
)
|
||||
|
||||
@@ -100,11 +101,16 @@ func TestOTelSessionDedupOnlyUnkeyedLoopback(t *testing.T) {
|
||||
if !usage.OTelSessionAgent("pi") {
|
||||
t.Fatal("local Pi reader unavailable")
|
||||
}
|
||||
if err := sessions.SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cases := []struct {
|
||||
name, addr, key string
|
||||
session, override, kind string
|
||||
dedup bool
|
||||
}{
|
||||
{"generated recording identity", "127.0.0.1:1234", "", "", "request-recorded", "", true},
|
||||
{"native with generated override", "127.0.0.1:1234", "", "session", "request-recorded", "", true},
|
||||
{"loopback", "127.0.0.1:1234", "", "", "", "", true},
|
||||
{"IPv6 loopback", "[::1]:1234", "", "", "", "", true},
|
||||
{"LAN", "192.168.1.5:1234", "key", "", "", "", false},
|
||||
|
||||
+7
-6
@@ -1010,12 +1010,13 @@ func Handler(w Windows, gw *gateway.Server) http.Handler {
|
||||
settings.CarryPerModel(&in, &cur)
|
||||
in.LAN, in.LANKey = cur.LAN, cur.LANKey
|
||||
in.LANKeyID = cur.LANKeyID
|
||||
in.Port = cur.Port // set on its own (port below), which moves the gateway
|
||||
in.CORSOrigins = cur.CORSOrigins // set on its own (cors below)
|
||||
in.GitHubToken = cur.GitHubToken // set on its own (github-token below), never sent to the page
|
||||
in.RequestArchive = cur.RequestArchive // the Gateway page's, set on its own
|
||||
in.RequestArchiveMaxMB = cur.RequestArchiveMaxMB // in settings.json only
|
||||
in.RedactRules = cur.RedactRules // the masking rules, set on their own
|
||||
in.Port = cur.Port // set on its own (port below), which moves the gateway
|
||||
in.CORSOrigins = cur.CORSOrigins // set on its own (cors below)
|
||||
in.GitHubToken = cur.GitHubToken // set on its own (github-token below), never sent to the page
|
||||
in.RequestArchive = cur.RequestArchive // the Gateway page's, set on its own
|
||||
in.RequestArchiveMaxMB = cur.RequestArchiveMaxMB // in settings.json only
|
||||
in.GatewayConversations = cur.GatewayConversations // Sessions' explicit recording consent
|
||||
in.RedactRules = cur.RedactRules // the masking rules, set on their own
|
||||
// used or left is the Usage page's toggle as much as Settings', set on its own
|
||||
in.QuotaLeft = cur.QuotaLeft
|
||||
in.UsageOrder = cur.UsageOrder // the Usage page's, dragged there
|
||||
|
||||
@@ -14,7 +14,7 @@ const web = !!window.bootPrefs?.web;
|
||||
// pages — Agents, Sessions, Library — and their settings; Settings ›
|
||||
// General turns it off, and they are back.
|
||||
let gatewayMode = web && !!window.bootPrefs?.gateway;
|
||||
const GATEWAY_HIDES = ["agents", "sessions", "library"];
|
||||
const GATEWAY_HIDES = ["agents", "library"];
|
||||
// The window is dragged by its header, and only where the header says so
|
||||
// (--wails-draggable), so the tabs and buttons in it stay plain clicks.
|
||||
// Outside the app — a browser on the gateway's page, or on `magpie web` —
|
||||
@@ -17228,7 +17228,7 @@ function renderSessions() {
|
||||
// the rest were behind a "Show more" that isn't there (yetone, #1019).
|
||||
// Nothing left to draw, the note is just the list's length, in words a
|
||||
// reader uses: one session is a session, not "1 sessions".
|
||||
const parts = [t("Totals count every session in the agents' own files")];
|
||||
const parts = [t("Totals count native sessions and gateway requests with a session ID")];
|
||||
if (sessDrawn !== null) {
|
||||
parts.push(sessDrawn < sessTotal
|
||||
? t("showing {drawn} of {n}", { drawn: sessDrawn, n: sessTotal })
|
||||
@@ -18633,7 +18633,7 @@ function renderGatewayMode(s) {
|
||||
flag: t("Now on: magpie web was started with --gateway"),
|
||||
"no-agents": t("Now on: no agents on this computer"),
|
||||
}[s.gatewayWhy] || t("Now off: agents found on this computer");
|
||||
$("#gatewayModeSub").textContent = [t("Only what a gateway for other computers needs: no Agents, Sessions or Library, nor this computer's agents' settings"), now].filter(Boolean).join(" · ");
|
||||
$("#gatewayModeSub").textContent = [t("Only what a gateway for other computers needs: no Agents or Library, nor this computer's agents' settings"), now].filter(Boolean).join(" · ");
|
||||
}
|
||||
|
||||
function renderSessionTerminal(s, keep) {
|
||||
|
||||
@@ -4,6 +4,13 @@
|
||||
// works. {name} slots are filled by t(str, vars).
|
||||
const I18N = {
|
||||
zh: {
|
||||
"Enable": "启用",
|
||||
"Record gateway conversations": "记录网关对话",
|
||||
"Prompts, replies and tool results may contain private code and files. Store them on this server for up to 7 days (256 MiB total)? Recognized secrets are masked, but this is not a privacy guarantee.": "提示词、回复和工具结果可能包含私有代码与文件。是否在此服务器保存最多 7 天(总计 256 MiB)?已识别的密钥会被遮蔽,但不能保证所有敏感信息都被移除。",
|
||||
"Stop recording and clear saved conversations": "停止记录并清空已保存的对话",
|
||||
"Delete all locally recorded gateway conversation content? Usage totals and native session files are kept.": "删除本地记录的全部网关对话内容?用量统计与原生会话文件会保留。",
|
||||
"No saved gateway content for this session": "此会话没有已保存的网关正文",
|
||||
"Recorded gateway traffic only; earlier, expired or uncaptured content may be missing": "仅包含已记录的网关内容;更早、已过期或未捕获的内容可能缺失",
|
||||
// the context window (context.js)
|
||||
"1 request": "1 个请求",
|
||||
"1 session": "1 个会话",
|
||||
@@ -580,7 +587,7 @@ const I18N = {
|
||||
"Keeps this computer from going to sleep and its display on while agents work through magpie and for ten minutes after": "Agent 通过 magpie 工作时及之后十分钟内,阻止这台电脑睡眠,屏幕也保持常亮、不锁屏",
|
||||
"Keeps this computer from going to sleep by itself while agents work through magpie and for ten minutes after; the display may still turn off": "Agent 通过 magpie 工作时及之后十分钟内,阻止这台电脑自动睡眠;屏幕仍可能关闭",
|
||||
"Gateway mode": "网关模式",
|
||||
"Only what a gateway for other computers needs: no Agents, Sessions or Library, nor this computer's agents' settings": "只显示给其他电脑当网关所需的内容:不显示 Agent、会话和资源库,也不显示本机 Agent 的设置",
|
||||
"Only what a gateway for other computers needs: no Agents or Library, nor this computer's agents' settings": "只显示给其他电脑当网关所需的内容:不显示 Agent 和资源库,也不显示本机 Agent 的设置",
|
||||
"Now on: magpie web was started with --gateway": "当前开启:magpie web 以 --gateway 启动",
|
||||
"Now on: no agents on this computer": "当前开启:本机没有 Agent",
|
||||
"Now off: agents found on this computer": "当前关闭:本机找到了 Agent",
|
||||
@@ -2282,7 +2289,7 @@ const I18N = {
|
||||
"No sessions yet. Claude Code's, Codex's, Hermes, OpenCode's and Pi's sessions on this computer show up here, with what each cost and the command that resumes it.": "还没有会话。本机 Claude Code、Codex、Hermes、OpenCode 和 Pi 的会话会显示在这里,附费用和继续会话的命令。",
|
||||
"Reading the agents' session files…": "正在读取 Agent 的会话文件…",
|
||||
"Totals count every session in the agents' own files; the list is the latest {n} by activity · {dirs}": "合计含 Agent 文件中的全部会话;列表为最近活动的 {n} 个 · {dirs}",
|
||||
"Totals count every session in the agents' own files": "合计含 Agent 文件中的全部会话",
|
||||
"Totals count native sessions and gateway requests with a session ID": "合计含本机会话和带有会话 ID 的网关请求",
|
||||
"showing {drawn} of {n}": "显示 {n} 个中的 {drawn} 个",
|
||||
"90 days": "90 天",
|
||||
"All folders": "全部文件夹",
|
||||
@@ -3884,6 +3891,13 @@ const I18N = {
|
||||
"magpie's key": "magpie 的金鑰",
|
||||
"Claude Code sends magpie its key and is signed out of claude.ai while it runs through magpie: claude.ai's plan limits in /usage, its connectors, voice and /teleport are off": "Claude Code 向 magpie 傳送 magpie 的金鑰,經 magpie 執行期間處於 claude.ai 未登入狀態:/usage 裡的方案限額、claude.ai 連接器、語音和 /teleport 都無法使用",
|
||||
"Claude Code keeps its claude.ai sign-in (/login), so those work; it sends that sign-in to magpie, which never passes it on. Remote Control and ultrareview stay off: Claude Code has them only on Anthropic's own address": "Claude Code 保留 claude.ai 登入(/login),上述功能可用;它把這份登入憑證傳給 magpie,magpie 不會轉交到任何地方。Remote Control 和 ultrareview 仍無法使用:Claude Code 只在 Anthropic 官方位址上提供它們",
|
||||
"Enable": "啟用",
|
||||
"Record gateway conversations": "記錄閘道對話",
|
||||
"Prompts, replies and tool results may contain private code and files. Store them on this server for up to 7 days (256 MiB total)? Recognized secrets are masked, but this is not a privacy guarantee.": "提示詞、回覆和工具結果可能包含私人程式碼與檔案。是否在此伺服器儲存最多 7 天(總計 256 MiB)?已辨識的金鑰會被遮蔽,但無法保證所有敏感資訊都被移除。",
|
||||
"Stop recording and clear saved conversations": "停止記錄並清空已儲存的對話",
|
||||
"Delete all locally recorded gateway conversation content? Usage totals and native session files are kept.": "刪除本機記錄的全部閘道對話內容?用量統計與原生工作階段檔案會保留。",
|
||||
"No saved gateway content for this session": "此工作階段沒有已儲存的閘道內容",
|
||||
"Recorded gateway traffic only; earlier, expired or uncaptured content may be missing": "僅包含已記錄的閘道內容;更早、已過期或未擷取的內容可能缺失",
|
||||
// the context window (context.js)
|
||||
"1 request": "1 個請求",
|
||||
"1 session": "1 個會話",
|
||||
@@ -4456,7 +4470,7 @@ const I18N = {
|
||||
"Keeps this computer from going to sleep and its display on while agents work through magpie and for ten minutes after": "Agent 透過 magpie 工作時及之後十分鐘內,阻止這臺電腦睡眠,螢幕也保持常亮、不鎖屏",
|
||||
"Keeps this computer from going to sleep by itself while agents work through magpie and for ten minutes after; the display may still turn off": "Agent 透過 magpie 工作時及之後十分鐘內,阻止這臺電腦自動睡眠;螢幕仍可能關閉",
|
||||
"Gateway mode": "閘道模式",
|
||||
"Only what a gateway for other computers needs: no Agents, Sessions or Library, nor this computer's agents' settings": "只顯示給其他電腦當閘道所需的內容:不顯示 Agent、會話和資源庫,也不顯示本機 Agent 的設定",
|
||||
"Only what a gateway for other computers needs: no Agents or Library, nor this computer's agents' settings": "只顯示給其他電腦當閘道所需的內容:不顯示 Agent 和資源庫,也不顯示本機 Agent 的設定",
|
||||
"Now on: magpie web was started with --gateway": "目前開啟:magpie web 以 --gateway 啟動",
|
||||
"Now on: no agents on this computer": "目前開啟:本機沒有 Agent",
|
||||
"Now off: agents found on this computer": "目前關閉:本機找到了 Agent",
|
||||
@@ -6163,7 +6177,7 @@ const I18N = {
|
||||
"No sessions yet. Claude Code's, Codex's, Hermes, OpenCode's and Pi's sessions on this computer show up here, with what each cost and the command that resumes it.": "還沒有會話。本機 Claude Code、Codex、Hermes、OpenCode 和 Pi 的會話會顯示在這裡,附費用和繼續會話的命令。",
|
||||
"Reading the agents' session files…": "正在讀取 Agent 的會話檔案…",
|
||||
"Totals count every session in the agents' own files; the list is the latest {n} by activity · {dirs}": "合計含 Agent 檔案中的全部會話;列表為最近活動的 {n} 個 · {dirs}",
|
||||
"Totals count every session in the agents' own files": "合計含 Agent 檔案中的全部會話",
|
||||
"Totals count native sessions and gateway requests with a session ID": "合計含本機會話和帶有會話 ID 的閘道請求",
|
||||
"showing {drawn} of {n}": "顯示 {n} 箇中的 {drawn} 個",
|
||||
"90 days": "90 天",
|
||||
"All folders": "全部資料夾",
|
||||
@@ -7741,6 +7755,13 @@ const I18N = {
|
||||
"Video · 7 days": "影片 · 7 天",
|
||||
},
|
||||
ja: {
|
||||
"Enable": "有効にする",
|
||||
"Record gateway conversations": "ゲートウェイの会話を記録",
|
||||
"Prompts, replies and tool results may contain private code and files. Store them on this server for up to 7 days (256 MiB total)? Recognized secrets are masked, but this is not a privacy guarantee.": "プロンプト、応答、ツールの結果には非公開のコードやファイルが含まれる場合があります。このサーバーに最大7日間(合計256 MiB)保存しますか?検出された秘密情報は隠されますが、すべての機密情報の除去は保証されません。",
|
||||
"Stop recording and clear saved conversations": "記録を停止して保存済みの会話を消去",
|
||||
"Delete all locally recorded gateway conversation content? Usage totals and native session files are kept.": "ローカルに記録したゲートウェイの会話をすべて削除しますか?使用量の集計と元のセッションファイルは保持されます。",
|
||||
"No saved gateway content for this session": "このセッションに保存済みのゲートウェイの内容はありません",
|
||||
"Recorded gateway traffic only; earlier, expired or uncaptured content may be missing": "記録されたゲートウェイ通信のみです。以前の内容、期限切れ、未記録の内容は含まれない場合があります",
|
||||
// the context window (context.js)
|
||||
"1 request": "1 件のリクエスト",
|
||||
"1 session": "1 件のセッション",
|
||||
@@ -8303,7 +8324,7 @@ const I18N = {
|
||||
"Keeps this computer from going to sleep and its display on while agents work through magpie and for ten minutes after": "エージェントが magpie 経由で作業している間とその後 10 分間、このコンピューターをスリープさせず、ディスプレイもオンのままにします",
|
||||
"Keeps this computer from going to sleep by itself while agents work through magpie and for ten minutes after; the display may still turn off": "エージェントが magpie 経由で作業している間とその後 10 分間、このコンピューターが自動でスリープしないようにします。ディスプレイはオフになることがあります",
|
||||
"Gateway mode": "ゲートウェイモード",
|
||||
"Only what a gateway for other computers needs: no Agents, Sessions or Library, nor this computer's agents' settings": "他のコンピューターのゲートウェイに必要なものだけ:エージェント、セッション、ライブラリ、このコンピューターのエージェントの設定は表示しません",
|
||||
"Only what a gateway for other computers needs: no Agents or Library, nor this computer's agents' settings": "他のコンピューターのゲートウェイに必要なものだけ:エージェント、ライブラリ、このコンピューターのエージェントの設定は表示しません",
|
||||
"Now on: magpie web was started with --gateway": "現在オン:magpie web が --gateway で起動されています",
|
||||
"Now on: no agents on this computer": "現在オン:このコンピューターにエージェントがありません",
|
||||
"Now off: agents found on this computer": "現在オフ:このコンピューターにエージェントがあります",
|
||||
@@ -10008,7 +10029,7 @@ const I18N = {
|
||||
"No sessions yet. Claude Code's, Codex's, Hermes, OpenCode's and Pi's sessions on this computer show up here, with what each cost and the command that resumes it.": "セッションはまだありません。このコンピュータ上の Claude Code、Codex、Hermes、OpenCode、Pi のセッションが、それぞれの費用と再開コマンドとともにここに表示されます。",
|
||||
"Reading the agents' session files…": "エージェントのセッションファイルを読み取り中…",
|
||||
"Totals count every session in the agents' own files; the list is the latest {n} by activity · {dirs}": "合計はエージェント自身のファイルにあるすべてのセッションを集計しています。一覧は最近のアクティビティ順で {n} 件 · {dirs}",
|
||||
"Totals count every session in the agents' own files": "合計はエージェント自身のファイルにあるすべてのセッションを集計しています",
|
||||
"Totals count native sessions and gateway requests with a session ID": "合計にはローカルのセッションとセッション ID のあるゲートウェイ要求が含まれます",
|
||||
"showing {drawn} of {n}": "{n} 件中 {drawn} 件を表示",
|
||||
"90 days": "90 日",
|
||||
"All folders": "すべてのフォルダ",
|
||||
@@ -11603,6 +11624,13 @@ const I18N = {
|
||||
"Main navigation": "メインナビゲーション",
|
||||
},
|
||||
de: {
|
||||
"Enable": "Aktivieren",
|
||||
"Record gateway conversations": "Gateway-Gespräche aufzeichnen",
|
||||
"Prompts, replies and tool results may contain private code and files. Store them on this server for up to 7 days (256 MiB total)? Recognized secrets are masked, but this is not a privacy guarantee.": "Prompts, Antworten und Werkzeugergebnisse können privaten Code und Dateien enthalten. Bis zu 7 Tage auf diesem Server speichern (insgesamt 256 MiB)? Erkannte Geheimnisse werden maskiert, aber vollständiger Datenschutz ist nicht garantiert.",
|
||||
"Stop recording and clear saved conversations": "Aufzeichnung stoppen und gespeicherte Gespräche löschen",
|
||||
"Delete all locally recorded gateway conversation content? Usage totals and native session files are kept.": "Alle lokal aufgezeichneten Gateway-Gespräche löschen? Nutzungsstatistiken und native Sitzungsdateien bleiben erhalten.",
|
||||
"No saved gateway content for this session": "Keine gespeicherten Gateway-Inhalte für diese Sitzung",
|
||||
"Recorded gateway traffic only; earlier, expired or uncaptured content may be missing": "Nur aufgezeichneter Gateway-Verkehr; frühere, abgelaufene oder nicht erfasste Inhalte können fehlen",
|
||||
// the context window (context.js)
|
||||
"1 request": "1 Anfrage",
|
||||
"1 session": "1 Sitzung",
|
||||
@@ -12165,7 +12193,7 @@ const I18N = {
|
||||
"Keeps this computer from going to sleep and its display on while agents work through magpie and for ten minutes after": "Verhindert, dass dieser Computer in den Ruhezustand geht, und hält den Bildschirm an, während Agenten über magpie arbeiten, und zehn Minuten danach",
|
||||
"Keeps this computer from going to sleep by itself while agents work through magpie and for ten minutes after; the display may still turn off": "Verhindert, dass dieser Computer von selbst in den Ruhezustand geht, während Agenten über magpie arbeiten, und zehn Minuten danach; der Bildschirm kann sich trotzdem ausschalten",
|
||||
"Gateway mode": "Gateway-Modus",
|
||||
"Only what a gateway for other computers needs: no Agents, Sessions or Library, nor this computer's agents' settings": "Nur, was ein Gateway für andere Computer braucht: keine Agenten, Sitzungen oder Bibliothek, keine Einstellungen der Agenten auf diesem Computer",
|
||||
"Only what a gateway for other computers needs: no Agents or Library, nor this computer's agents' settings": "Nur, was ein Gateway für andere Computer braucht: keine Agenten oder Bibliothek, keine Einstellungen der Agenten auf diesem Computer",
|
||||
"Now on: magpie web was started with --gateway": "Jetzt an: magpie web wurde mit --gateway gestartet",
|
||||
"Now on: no agents on this computer": "Jetzt an: keine Agenten auf diesem Computer",
|
||||
"Now off: agents found on this computer": "Jetzt aus: Agenten auf diesem Computer gefunden",
|
||||
@@ -13872,7 +13900,7 @@ const I18N = {
|
||||
"No sessions yet. Claude Code's, Codex's, Hermes, OpenCode's and Pi's sessions on this computer show up here, with what each cost and the command that resumes it.": "Noch keine Sitzungen. Sitzungen von Claude Code, Codex, Hermes, OpenCode und Pi auf diesem Computer erscheinen hier, jeweils mit Kosten und dem Befehl zum Fortsetzen.",
|
||||
"Reading the agents' session files…": "Sitzungsdateien der Agenten werden gelesen…",
|
||||
"Totals count every session in the agents' own files; the list is the latest {n} by activity · {dirs}": "Summen zählen jede Sitzung in den Dateien der Agenten; die Liste zeigt die {n} zuletzt aktiven · {dirs}",
|
||||
"Totals count every session in the agents' own files": "Summen zählen jede Sitzung in den Dateien der Agenten",
|
||||
"Totals count native sessions and gateway requests with a session ID": "Summen zählen lokale Sitzungen und Gateway-Anfragen mit Sitzungs-ID",
|
||||
"showing {drawn} of {n}": "{drawn} von {n} angezeigt",
|
||||
"90 days": "90 Tage",
|
||||
"All folders": "Alle Ordner",
|
||||
|
||||
@@ -77,3 +77,8 @@
|
||||
.sm-provider-list li .name { min-width: 0; flex: 1; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.sm-provider-list li .note { flex: none; font-size: 11px; color: var(--faint); }
|
||||
.sm-provider-list { max-height: min(300px, 45vh); overflow-y: auto; }
|
||||
.sm-recording { margin: 0 0 12px; gap: 8px; min-height: 28px; }
|
||||
.sm-recording:empty { display: none; }
|
||||
.sm-record-label { display: flex; align-items: center; gap: 8px; min-width: 0; font-size: 12px; }
|
||||
.sm-record-label input { flex: 0 0 auto; accent-color: var(--fg); }
|
||||
.sm-record-clear { flex: 0 0 28px; width: 28px; height: 28px; }
|
||||
@@ -27,6 +27,7 @@
|
||||
let loading = 0;
|
||||
let fitObserver = null;
|
||||
let focus = null; // { agent, id, until }: a session to bring into sight once drawn
|
||||
let recordingBusy = false;
|
||||
|
||||
const TRASH = "M3 4.5h10M6.5 4.5V3h3v1.5M4.5 4.5l.6 8.5h5.8l.6-8.5M7 7v4M9 7v4";
|
||||
const TERM = "M3 4.5 6 7.5 3 10.5M7.5 11.5h5.5";
|
||||
@@ -98,7 +99,7 @@
|
||||
fitObserver?.disconnect();
|
||||
if (page.querySelector(".sm-agent-pick.open")) closeProtoMenu();
|
||||
page.classList.toggle("loading", !data?.sessions && !failed);
|
||||
page.replaceChildren(head(), body());
|
||||
page.replaceChildren(head(), recordingControls(), body());
|
||||
const wrap = page.querySelector(".sm-switch");
|
||||
const tabs = wrap.querySelector(".sm-agents");
|
||||
const pick = wrap.querySelector(".sm-agent-pick");
|
||||
@@ -134,6 +135,47 @@
|
||||
load(name);
|
||||
}
|
||||
|
||||
function recordingControls() {
|
||||
const bar = el("div", "bar sm-recording");
|
||||
if (typeof data?.recording !== "boolean") return bar;
|
||||
const label = el("label", "sm-record-label");
|
||||
const toggle = el("input");
|
||||
toggle.type = "checkbox";
|
||||
toggle.checked = data.recording;
|
||||
toggle.disabled = recordingBusy;
|
||||
toggle.setAttribute("role", "switch");
|
||||
toggle.setAttribute("aria-label", t("Record gateway conversations"));
|
||||
toggle.onchange = async () => {
|
||||
const on = toggle.checked;
|
||||
toggle.checked = data.recording;
|
||||
if (on && !await confirmAction(t("Record gateway conversations"), t("Prompts, replies and tool results may contain private code and files. Store them on this server for up to 7 days (256 MiB total)? Recognized secrets are masked, but this is not a privacy guarantee."), t("Enable"))) return;
|
||||
recordingBusy = true;
|
||||
toggle.disabled = true;
|
||||
try {
|
||||
const result = await api("sessions/recording", { on });
|
||||
data.recording = result.recording;
|
||||
} catch (err) { status(err.message, "err"); }
|
||||
finally { recordingBusy = false; draw(); }
|
||||
};
|
||||
label.append(toggle, el("span", "", t("Record gateway conversations")));
|
||||
const clear = el("button", "copy sm-record-clear");
|
||||
clear.type = "button";
|
||||
clear.title = t("Stop recording and clear saved conversations");
|
||||
clear.setAttribute("aria-label", clear.title);
|
||||
clear.disabled = recordingBusy;
|
||||
clear.append(svg(TRASH, 13, 1.4));
|
||||
clear.onclick = async () => {
|
||||
if (!await confirmAction(t("Stop recording and clear saved conversations"), t("Delete all locally recorded gateway conversation content? Usage totals and native session files are kept."), t("Delete"))) return;
|
||||
recordingBusy = true;
|
||||
clear.disabled = true;
|
||||
try { await api("sessions/recording", { clear: true }); talks.clear(); await load(); }
|
||||
catch (err) { status(err.message, "err"); }
|
||||
finally { recordingBusy = false; draw(); }
|
||||
};
|
||||
bar.append(label, el("span", "grow"), clear);
|
||||
return bar;
|
||||
}
|
||||
|
||||
function head() {
|
||||
const h = el("div", "usage-head sm-head");
|
||||
const wrap = el("div", "sm-switch");
|
||||
@@ -527,6 +569,7 @@
|
||||
if (s.path) line(t("File"), s.path + (s.files > 1 ? " " + t("+{n} more", { n: s.files - 1 }) : ""));
|
||||
if (s.transcript) {
|
||||
const box = el("div", "sess-talk");
|
||||
box.dataset.talk = s.agent + "/" + s.id;
|
||||
const b = el("button", "text sess-talk-btn");
|
||||
b.type = "button";
|
||||
const show = () => {
|
||||
@@ -624,20 +667,29 @@
|
||||
const got = talks.get(k);
|
||||
if (!got) {
|
||||
box.replaceChildren(el("p", "cx-none", t("Reading…")));
|
||||
talks.set(k, { busy: true });
|
||||
const pending = { busy: true };
|
||||
talks.set(k, pending);
|
||||
const finish = (value) => {
|
||||
// A refresh or clear invalidates older responses, even for the same ID.
|
||||
if (talks.get(k) !== pending) return;
|
||||
talks.set(k, value);
|
||||
if (!talkOpen.has(k)) return;
|
||||
for (const currentBox of page.querySelectorAll(".sess-talk")) {
|
||||
if (currentBox.dataset.talk === k) drawTalk(currentBox, s);
|
||||
}
|
||||
};
|
||||
api("sessions/transcript?agent=" + encodeURIComponent(s.agent) + "&id=" + encodeURIComponent(s.id))
|
||||
.then((tr) => talks.set(k, { tr }), (err) => talks.set(k, { err: err.message }))
|
||||
.then(() => { if (box.isConnected && talkOpen.has(k)) drawTalk(box, s); });
|
||||
.then((tr) => finish({ tr }), (err) => finish({ err: err.message }));
|
||||
return;
|
||||
}
|
||||
if (got.busy) return;
|
||||
if (got.busy) { box.replaceChildren(el("p", "cx-none", t("Reading…"))); return; }
|
||||
if (got.err) { box.replaceChildren(el("p", "cx-none", got.err)); talks.delete(k); return; }
|
||||
const parts = got.tr?.parts || [];
|
||||
const out = [];
|
||||
if (!parts.length) out.push(el("p", "cx-none", t("Nothing was said here")));
|
||||
if (!parts.length) out.push(el("p", "cx-none", t(got.tr?.source === "gateway" ? "No saved gateway content for this session" : "Nothing was said here")));
|
||||
for (const p of parts) out.push(ledSaid(p));
|
||||
if (got.tr?.cut) out.push(el("p", "cx-none", t("There was more than is shown here")));
|
||||
out.push(el("p", "cx-src", t("Read from the agent's session file; magpie keeps no copy")));
|
||||
out.push(el("p", "cx-src", t(got.tr?.source === "gateway" ? "Recorded gateway traffic only; earlier, expired or uncaptured content may be missing" : "Read from the agent's session file; magpie keeps no copy")));
|
||||
box.replaceChildren(...out);
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
package gui
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/yetone/magpie/internal/sessions"
|
||||
"github.com/yetone/magpie/internal/settings"
|
||||
"github.com/yetone/magpie/internal/usage"
|
||||
)
|
||||
|
||||
func TestGatewayConversationAPI(t *testing.T) {
|
||||
sandboxHome(t)
|
||||
sessions.Reset()
|
||||
t.Cleanup(sessions.Reset)
|
||||
mux := http.NewServeMux()
|
||||
sessionRoutes(mux, folderOnly{})
|
||||
sessionManageRoutes(mux, folderOnly{})
|
||||
request := func(method, path, body string) *httptest.ResponseRecorder {
|
||||
w := httptest.NewRecorder()
|
||||
mux.ServeHTTP(w, httptest.NewRequest(method, path, strings.NewReader(body)))
|
||||
return w
|
||||
}
|
||||
if w := request("POST", "/api/sessions/recording", `{"on":true}`); w.Code != 200 || !settings.Load().GatewayConversations {
|
||||
t.Fatalf("consent: %d %s", w.Code, w.Body)
|
||||
}
|
||||
usage.Append(usage.Record{Time: time.Now(), Agent: "opencode", Session: "remote-only", Provider: "fake", Model: "m", Status: 200})
|
||||
if err := sessions.SaveGatewayTurn(sessions.GatewayTurn{Agent: "opencode", Session: "remote-only", Time: time.Now(), Input: []sessions.Part{{Role: "user", Kind: "text", Text: "remote question"}}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
w := request("GET", "/api/sessions/manage?agent=opencode", "")
|
||||
var list struct {
|
||||
Sessions []sessions.Session `json:"sessions"`
|
||||
Recording bool `json:"recording"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &list); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !list.Recording || len(list.Sessions) != 1 || !list.Sessions[0].Transcript || !list.Sessions[0].ReadOnly || list.Sessions[0].Resume != "" {
|
||||
t.Fatalf("listing: %s", w.Body)
|
||||
}
|
||||
w = request("GET", "/api/sessions/transcript?agent=opencode&id=remote-only", "")
|
||||
var tr sessions.Transcript
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &tr); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if tr.Source != "gateway" || len(tr.Parts) != 1 || tr.Parts[0].Text != "remote question" {
|
||||
t.Fatalf("transcript: %s", w.Body)
|
||||
}
|
||||
if w := request("GET", "/api/sessions/transcript?agent=codex&id=remote-only", ""); strings.Contains(w.Body.String(), "remote question") {
|
||||
t.Fatal("cross-agent disclosure")
|
||||
}
|
||||
if w := request("POST", "/api/sessions/recording", `{"clear":true}`); w.Code != 200 {
|
||||
t.Fatalf("clear: %s", w.Body)
|
||||
}
|
||||
w = request("GET", "/api/sessions/transcript?agent=opencode&id=remote-only", "")
|
||||
if strings.Contains(w.Body.String(), "remote question") || settings.Load().GatewayConversations {
|
||||
t.Fatal("clear retained content or consent")
|
||||
}
|
||||
if _, ok := usage.GatewaySessionByID("opencode", "remote-only", nil); !ok {
|
||||
t.Fatal("clear removed usage")
|
||||
}
|
||||
guard := webGuard("test-session", "secret", 0, mux)
|
||||
w = httptest.NewRecorder()
|
||||
guard.ServeHTTP(w, httptest.NewRequest("GET", "/api/sessions/transcript?agent=opencode&id=remote-only", nil))
|
||||
if w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("unguarded transcript: %d", w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewaySessionHistoryAndCalendar(t *testing.T) {
|
||||
sandboxHome(t)
|
||||
sessions.Reset()
|
||||
t.Cleanup(sessions.Reset)
|
||||
now := time.Now()
|
||||
today := time.Date(now.Year(), now.Month(), now.Day(), 12, 0, 0, 0, time.Local)
|
||||
for _, offset := range []int{-3, -2, -1} {
|
||||
usage.Append(usage.Record{Time: today.AddDate(0, 0, offset), Agent: "opencode", Session: "multi-day", Provider: "fake", Model: "m", Input: 10, Output: 2, Status: 200})
|
||||
}
|
||||
all := combinedSessionStats(0)
|
||||
if all.From != today.AddDate(0, 0, -3).Format(time.DateOnly) || len(all.Sessions) != 1 || all.Sessions[0].Input != 30 {
|
||||
t.Fatalf("all-time gateway history missing: %+v", all)
|
||||
}
|
||||
view := all.Overview("", "", "", 8)
|
||||
if len(view.Days) != 4 || view.Days[0] != 1 || view.Days[1] != 1 || view.Days[2] != 1 || view.Days[3] != 0 {
|
||||
t.Fatalf("gateway calendar lost active dates: %v", view.Days)
|
||||
}
|
||||
if len(view.Output) != 4 || view.Output[0] != 2 || view.Output[1] != 2 || view.Output[2] != 2 {
|
||||
t.Fatalf("gateway daily output missing: %v", view.Output)
|
||||
}
|
||||
window := combinedSessionStats(2)
|
||||
if len(window.Sessions) != 1 || window.Sessions[0].Input != 10 {
|
||||
t.Fatalf("two-day window includes older requests: %+v", window)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewaySessionsNativeWinsBeforeLimits(t *testing.T) {
|
||||
h := sandboxHome(t)
|
||||
claude := filepath.Join(h, ".claude")
|
||||
t.Setenv("CLAUDE_CONFIG_DIR", claude)
|
||||
proj := filepath.Join(claude, "projects", "-work")
|
||||
if err := os.MkdirAll(proj, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
now := time.Now()
|
||||
for id, offset := range map[string]int{"old-native": -4, "new-native": -1} {
|
||||
at := now.AddDate(0, 0, offset).Format(time.RFC3339)
|
||||
line := fmt.Sprintf(`{"type":"user","message":{"role":"user","content":"hello"},"uuid":"u","timestamp":%q,"cwd":"/work","sessionId":%q}`+"\n"+
|
||||
`{"type":"assistant","message":{"model":"claude-sonnet-5","id":"msg_1","role":"assistant","content":[{"type":"text","text":"done"}],"usage":{"input_tokens":10,"output_tokens":2}},"uuid":"a","timestamp":%q,"cwd":"/work","sessionId":%q}`+"\n", at, id, at, id)
|
||||
if err := os.WriteFile(filepath.Join(proj, id+".jsonl"), []byte(line), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
sessions.Reset()
|
||||
t.Cleanup(sessions.Reset)
|
||||
usage.Append(usage.Record{Time: now, Agent: "claude", Session: "old-native", Provider: "fake", Model: "m", Input: 50, Status: 200})
|
||||
mux := http.NewServeMux()
|
||||
sessionRoutes(mux, folderOnly{})
|
||||
w := httptest.NewRecorder()
|
||||
mux.ServeHTTP(w, httptest.NewRequest("GET", "/api/sessions?limit=1", nil))
|
||||
var out sessionsJSON
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(out.Sessions) != 1 || out.Sessions[0].ID != "new-native" || out.Sessions[0].ReadOnly {
|
||||
t.Fatalf("limit exposed a duplicate gateway projection: %s", w.Body)
|
||||
}
|
||||
st := combinedSessionStats(2)
|
||||
if len(st.Sessions) != 1 || st.Sessions[0].ID != "new-native" || st.Sessions[0].Input != 10 {
|
||||
t.Fatalf("date window exposed a duplicate gateway projection: %+v", st.Sessions)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewaySessionsIgnoreLargeGeneratedHistory(t *testing.T) {
|
||||
sandboxHome(t)
|
||||
sessions.Reset()
|
||||
t.Cleanup(sessions.Reset)
|
||||
if err := os.MkdirAll(filepath.Dir(usage.Path()), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f, err := os.Create(usage.Path())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
enc := json.NewEncoder(f)
|
||||
for i := range 150000 {
|
||||
if err := enc.Encode(usage.Record{Time: time.Now(), Agent: "claude", Session: fmt.Sprintf("request-%d", i), Model: "m", Input: 1, Status: 200}); err != nil {
|
||||
f.Close()
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
sessionRoutes(mux, folderOnly{})
|
||||
sessionManageRoutes(mux, folderOnly{})
|
||||
for _, path := range []string{"/api/sessions/manage?agent=claude", "/api/sessions"} {
|
||||
w := httptest.NewRecorder()
|
||||
mux.ServeHTTP(w, httptest.NewRequest("GET", path, nil))
|
||||
var out struct {
|
||||
Sessions []json.RawMessage `json:"sessions"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if w.Code != 200 || len(out.Sessions) != 0 {
|
||||
t.Fatalf("generated history leaked into %s: status=%d rows=%d", path, w.Code, len(out.Sessions))
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -12,8 +12,8 @@ import (
|
||||
|
||||
// Gateway mode (Player on Discord): `magpie web` on a server that is only
|
||||
// the gateway for other machines' agents shows what such a gateway needs —
|
||||
// Providers, Gateway, Routing, Usage, Plugins and Settings — and leaves out
|
||||
// what is this machine's agents' (Agents, Sessions, Library, the settings
|
||||
// Providers, Gateway, Routing, Usage, Sessions, Plugins and Settings — and leaves out
|
||||
// what is this machine's agents' (Agents, Library, the settings
|
||||
// written into agents' files) and its desktop's (the tray, the Dock, open
|
||||
// at login, notifications). It is the browser page's alone: the app's
|
||||
// windows show everything whatever it says.
|
||||
|
||||
@@ -40,6 +40,21 @@ type sessionsJSON struct {
|
||||
Dirs []string `json:"dirs"` // where they were read from
|
||||
}
|
||||
|
||||
func gatewayExternal(g usage.GatewaySession) sessions.ExternalSession {
|
||||
e := sessions.ExternalSession{Agent: g.Agent, ID: g.ID, Start: g.Start, Last: g.Last, Tokens: g.Tokens, Cost: g.Cost, Unpriced: g.Unpriced}
|
||||
for _, m := range g.Models {
|
||||
e.Models = append(e.Models, sessions.ExternalModel{Model: m.Model, Tokens: m.Tokens, Cost: m.Cost, Priced: m.Priced})
|
||||
}
|
||||
for _, d := range g.Daily {
|
||||
e.Daily = append(e.Daily, sessions.ExternalDayUsage{Date: d.Date, Agent: d.Agent, Model: d.Model, Tokens: d.Tokens, Cost: d.Cost, Priced: d.Priced})
|
||||
}
|
||||
return e
|
||||
}
|
||||
|
||||
func gatewaySession(g usage.GatewaySession) sessions.Session {
|
||||
return sessions.MergeExternal(nil, []sessions.ExternalSession{gatewayExternal(g)})[0]
|
||||
}
|
||||
|
||||
func sessionRoutes(mux *http.ServeMux, w Windows) {
|
||||
warmSessions()
|
||||
mux.HandleFunc("GET /api/sessions", func(rw http.ResponseWriter, r *http.Request) {
|
||||
@@ -63,7 +78,20 @@ func sessionRoutes(mux *http.ServeMux, w Windows) {
|
||||
for _, d := range sessions.Dirs() {
|
||||
out.Dirs = append(out.Dirs, tilde(d))
|
||||
}
|
||||
list := sessions.List(n)
|
||||
list := sessions.List(sessions.All)
|
||||
nativeKeys := map[string]bool{}
|
||||
for _, s := range list {
|
||||
nativeKeys[s.Agent+"|"+s.ID] = true
|
||||
}
|
||||
gateway := usage.GatewaySessions(time.Time{}, nativeKeys)
|
||||
ext := make([]sessions.ExternalSession, 0, len(gateway))
|
||||
for _, g := range gateway {
|
||||
ext = append(ext, gatewayExternal(g))
|
||||
}
|
||||
list = sessions.MergeExternal(list, ext)
|
||||
if len(list) > n {
|
||||
list = list[:n]
|
||||
}
|
||||
since := time.Now()
|
||||
for _, s := range list {
|
||||
if !s.Start.IsZero() && s.Start.Before(since) {
|
||||
@@ -122,7 +150,16 @@ func sessionRoutes(mux *http.ServeMux, w Windows) {
|
||||
// one is a session by its stats key, read however long ago it was at
|
||||
// work: the page's top sessions reach past the latest List reads.
|
||||
mux.HandleFunc("GET /api/sessions/one", func(rw http.ResponseWriter, r *http.Request) {
|
||||
s, ok := sessions.Get(r.URL.Query().Get("key"))
|
||||
key := r.URL.Query().Get("key")
|
||||
s, ok := sessions.Get(key)
|
||||
if !ok {
|
||||
if a, id, found := strings.Cut(key, ":"); found {
|
||||
if g, gok := usage.GatewaySessionByID(a, id, nil); gok {
|
||||
s = gatewaySession(g)
|
||||
ok = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if !ok {
|
||||
rw.Header().Set("Content-Type", "application/json")
|
||||
rw.WriteHeader(http.StatusNotFound)
|
||||
@@ -151,6 +188,15 @@ func sessionRoutes(mux *http.ServeMux, w Windows) {
|
||||
q := r.URL.Query()
|
||||
s, ok := sessions.Find(q.Get("agent"), q.Get("id"))
|
||||
if !ok {
|
||||
if _, found := usage.GatewaySessionByID(q.Get("agent"), q.Get("id"), nil); found {
|
||||
t, err := sessions.GatewayTranscript(q.Get("agent"), q.Get("id"))
|
||||
if err != nil {
|
||||
fail(rw, err)
|
||||
return
|
||||
}
|
||||
writeJSON(rw, t)
|
||||
return
|
||||
}
|
||||
fail(rw, errors.New("no such session"))
|
||||
return
|
||||
}
|
||||
@@ -288,7 +334,7 @@ func statsFor(days int) sessions.Stats {
|
||||
if !k.busy && time.Since(k.at) > 10*time.Second {
|
||||
k.busy = true
|
||||
go func() {
|
||||
st := sessions.StatsFor(days)
|
||||
st := combinedSessionStats(days)
|
||||
statsMemo.Lock()
|
||||
k.at, k.st, k.busy = time.Now(), st, false
|
||||
statsMemo.Unlock()
|
||||
@@ -299,7 +345,7 @@ func statsFor(days int) sessions.Stats {
|
||||
return st
|
||||
}
|
||||
statsMemo.Unlock()
|
||||
st := sessions.StatsFor(days)
|
||||
st := combinedSessionStats(days)
|
||||
statsMemo.Lock()
|
||||
if statsMemo.m == nil {
|
||||
statsMemo.m = map[int]*statsKept{}
|
||||
@@ -311,6 +357,28 @@ func statsFor(days int) sessions.Stats {
|
||||
return st
|
||||
}
|
||||
|
||||
func combinedSessionStats(days int) sessions.Stats {
|
||||
st := sessions.StatsFor(days)
|
||||
nativeKeys := map[string]bool{}
|
||||
for _, s := range sessions.List(sessions.All) {
|
||||
nativeKeys[s.Agent+"|"+s.ID] = true
|
||||
}
|
||||
var since time.Time
|
||||
if days > 0 && st.From != "" {
|
||||
since, _ = time.ParseInLocation(time.DateOnly, st.From, time.Local)
|
||||
}
|
||||
gs, gd := usage.GatewaySessionWindowExcept(since, nativeKeys)
|
||||
ext := make([]sessions.ExternalSession, 0, len(gs))
|
||||
for _, g := range gs {
|
||||
ext = append(ext, gatewayExternal(g))
|
||||
}
|
||||
daily := make([]sessions.ExternalDayUsage, 0, len(gd))
|
||||
for _, d := range gd {
|
||||
daily = append(daily, sessions.ExternalDayUsage{Date: d.Date, Agent: d.Agent, Model: d.Model, Tokens: d.Tokens, Cost: d.Cost, Priced: d.Priced})
|
||||
}
|
||||
return sessions.MergeExternalStats(st, daily, ext)
|
||||
}
|
||||
|
||||
// warmSessions reads every session file once magpie is up, so the Sessions
|
||||
// page opens on the kept index and not on a first read of them all. Usage ›
|
||||
// Requests' whole history is read after it, not beside it on the disk: a
|
||||
|
||||
@@ -6,10 +6,12 @@ import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"runtime"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/yetone/magpie/internal/agent"
|
||||
"github.com/yetone/magpie/internal/sessions"
|
||||
"github.com/yetone/magpie/internal/settings"
|
||||
"github.com/yetone/magpie/internal/usage"
|
||||
)
|
||||
|
||||
@@ -63,46 +65,113 @@ func trashJSON(looks map[string]*agent.Agent) []trashedJSON {
|
||||
}
|
||||
|
||||
func sessionManageRoutes(mux *http.ServeMux, w Windows) {
|
||||
mux.HandleFunc("POST /api/sessions/recording", func(rw http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
On bool `json:"on"`
|
||||
Clear bool `json:"clear"`
|
||||
}
|
||||
if err := json.NewDecoder(http.MaxBytesReader(rw, r.Body, 1024)).Decode(&in); err != nil {
|
||||
fail(rw, err)
|
||||
return
|
||||
}
|
||||
if err := sessions.SetGatewayRecording(in.On, in.Clear); err != nil {
|
||||
fail(rw, err)
|
||||
return
|
||||
}
|
||||
writeJSON(rw, map[string]bool{"recording": settings.Load().GatewayConversations})
|
||||
})
|
||||
// manage is the agents with sessions, and every session of ?agent=
|
||||
// (the one with the most when none is named), and the trash.
|
||||
mux.HandleFunc("GET /api/sessions/manage", func(rw http.ResponseWriter, r *http.Request) {
|
||||
looks := agentLooks()
|
||||
out := struct {
|
||||
Agents []manageAgentJSON `json:"agents"`
|
||||
Agent string `json:"agent"`
|
||||
Sessions []managedJSON `json:"sessions"`
|
||||
Terminal bool `json:"terminal"`
|
||||
Trash []trashedJSON `json:"trash"`
|
||||
TrashDir string `json:"trashDir"`
|
||||
Agents []manageAgentJSON `json:"agents"`
|
||||
Agent string `json:"agent"`
|
||||
Sessions []managedJSON `json:"sessions"`
|
||||
Terminal bool `json:"terminal"`
|
||||
Trash []trashedJSON `json:"trash"`
|
||||
TrashDir string `json:"trashDir"`
|
||||
Recording bool `json:"recording"`
|
||||
}{Agents: []manageAgentJSON{}, Sessions: []managedJSON{}, Terminal: runtime.GOOS == "darwin" && !isWeb(w),
|
||||
Trash: trashJSON(looks), TrashDir: tilde(sessions.TrashDir())}
|
||||
Trash: trashJSON(looks), TrashDir: tilde(sessions.TrashDir()), Recording: settings.Load().GatewayConversations}
|
||||
want := r.URL.Query().Get("agent")
|
||||
counts := map[string]*manageAgentJSON{}
|
||||
for _, a := range sessions.Agents() {
|
||||
j := manageAgentJSON{AgentCount: a, Name: a.Agent, Icon: "generic"}
|
||||
if l := looks[a.Agent]; l != nil {
|
||||
j.Name, j.Icon = l.Name, l.Icon
|
||||
}
|
||||
out.Agents = append(out.Agents, j)
|
||||
if a.Agent == want {
|
||||
out.Agent = want
|
||||
counts[a.Agent] = &j
|
||||
}
|
||||
gateway := usage.GatewaySessions(time.Time{}, nil)
|
||||
nativeCounts := map[string]map[string]bool{}
|
||||
for _, s := range sessions.List(sessions.All) {
|
||||
if nativeCounts[s.Agent] == nil {
|
||||
nativeCounts[s.Agent] = map[string]bool{}
|
||||
}
|
||||
nativeCounts[s.Agent][s.ID] = true
|
||||
}
|
||||
for _, s := range gateway {
|
||||
if nativeCounts[s.Agent][s.ID] {
|
||||
continue
|
||||
}
|
||||
j := counts[s.Agent]
|
||||
if j == nil {
|
||||
j = &manageAgentJSON{AgentCount: sessions.AgentCount{Agent: s.Agent}, Name: s.Agent, Icon: "generic"}
|
||||
if l := looks[s.Agent]; l != nil {
|
||||
j.Name, j.Icon = l.Name, l.Icon
|
||||
}
|
||||
counts[s.Agent] = j
|
||||
}
|
||||
j.Count++
|
||||
}
|
||||
for _, j := range counts {
|
||||
out.Agents = append(out.Agents, *j)
|
||||
}
|
||||
sort.Slice(out.Agents, func(i, j int) bool {
|
||||
if out.Agents[i].Count != out.Agents[j].Count {
|
||||
return out.Agents[i].Count > out.Agents[j].Count
|
||||
}
|
||||
return out.Agents[i].Agent < out.Agents[j].Agent
|
||||
})
|
||||
if _, ok := counts[want]; ok {
|
||||
out.Agent = want
|
||||
}
|
||||
if out.Agent == "" && len(out.Agents) > 0 {
|
||||
out.Agent = out.Agents[0].Agent
|
||||
}
|
||||
if out.Agent != "" {
|
||||
list := sessions.ListAgent(out.Agent)
|
||||
native := map[string]bool{}
|
||||
for _, s := range list {
|
||||
native[s.ID] = true
|
||||
}
|
||||
since := time.Now()
|
||||
for _, s := range list {
|
||||
if !s.Start.IsZero() && s.Start.Before(since) {
|
||||
since = s.Start
|
||||
}
|
||||
}
|
||||
for _, g := range gateway {
|
||||
if g.Agent == out.Agent && !g.Start.IsZero() && g.Start.Before(since) {
|
||||
since = g.Start
|
||||
}
|
||||
}
|
||||
vias := usage.Vias(since.Add(-time.Minute))
|
||||
for _, s := range list {
|
||||
s.Path = tilde(s.Path)
|
||||
out.Sessions = append(out.Sessions, managedJSON{Managed: s, Via: vias[s.Agent+"|"+s.ID]})
|
||||
}
|
||||
for _, g := range gateway {
|
||||
if g.Agent != out.Agent || native[g.ID] {
|
||||
continue
|
||||
}
|
||||
s := gatewaySession(g)
|
||||
out.Sessions = append(out.Sessions, managedJSON{Managed: sessions.Managed{Session: s}, Via: vias[s.Agent+"|"+s.ID]})
|
||||
}
|
||||
sort.SliceStable(out.Sessions, func(i, j int) bool {
|
||||
return out.Sessions[i].Last.After(out.Sessions[j].Last)
|
||||
})
|
||||
}
|
||||
writeJSON(rw, out)
|
||||
})
|
||||
|
||||
@@ -107,60 +107,64 @@ func TestSettingsSaveKeepsWhatItDoesNotSend(t *testing.T) {
|
||||
}
|
||||
one, two := 1.0, 2.0
|
||||
was := settings.Settings{
|
||||
AgentOrder: []string{"codex"},
|
||||
AgentsHidden: []string{"goose"},
|
||||
AgentsShown: []string{"pi"},
|
||||
Visible: map[string][]string{"claude": {"anthropic"}},
|
||||
HiddenModels: map[string][]string{"claude": {"p/m"}, "codex": {"p/n", "group/g"}},
|
||||
PickedModels: map[string][]string{"opencode": {"p/m"}},
|
||||
OrderedModels: map[string][]string{"codex": {"group/g", "p/n"}},
|
||||
FastPicks: map[string][]string{"codex": {"p/n"}},
|
||||
AgentEfforts: map[string]string{"cursor-local": "high"},
|
||||
PluginCheckins: map[string]bool{"fakeco": true},
|
||||
ModelNames: map[string]string{"p/m": "Mine"},
|
||||
ModelEfforts: map[string][]string{"p/m": {"low"}},
|
||||
ModelImages: map[string]bool{"p/m": true},
|
||||
ModelOutputs: map[string]int{"p/m": 131072},
|
||||
ModelCompacts: map[string]int{"p/*": 500000},
|
||||
CompactAt: 400000,
|
||||
ModelPrices: map[string]settings.ModelPrice{"p/m": {Input: &one, Output: &two}},
|
||||
ModelWires: map[string]string{"p/m": "vendor-c/m"},
|
||||
ModelAPIs: map[string]string{"p/m": "anthropic"},
|
||||
ModelSameAs: map[string]string{"p/m": "deepseek-v4.1-flash"},
|
||||
RedactRules: []redact.Rule{{Kind: "prefix", Prefix: "oc_sk_"}},
|
||||
LAN: true,
|
||||
LANKey: "sk-lan",
|
||||
LANKeyID: "lan-key-id",
|
||||
Port: 3591,
|
||||
CORSOrigins: []string{"http://localhost:3000"},
|
||||
GitHubToken: "ghp_kept",
|
||||
RequestArchive: true,
|
||||
RequestArchiveMaxMB: 64,
|
||||
QuotaLeft: true,
|
||||
UsageOrder: []string{"kimi", "codex"},
|
||||
PanelUsageHidden: []string{"deepseek"},
|
||||
PlainNames: true,
|
||||
PlainOwnNames: true,
|
||||
CodexAgentsV1: true,
|
||||
CodexTitles: "off",
|
||||
CodexAutoReview: "fake/m1",
|
||||
CodexSubagentModel: "codex/gpt-6-astra",
|
||||
ChinaMirror: true,
|
||||
CodexAutoReset: []string{"me@example.com"},
|
||||
CodexNoCredits: []string{"me@example.com"},
|
||||
CodexWarmAtOf: map[string]string{"me@example.com": "09:00"},
|
||||
TextSize: 125,
|
||||
UpdateSkip: "0.1.500",
|
||||
UpdateMirror: "https://mirror.example/",
|
||||
Window: []int{900, 700},
|
||||
WindowMaximised: true,
|
||||
FullContext: true,
|
||||
DesktopLongest: true,
|
||||
GatewayMode: "off",
|
||||
AgentOrder: []string{"codex"},
|
||||
AgentsHidden: []string{"goose"},
|
||||
AgentsShown: []string{"pi"},
|
||||
Visible: map[string][]string{"claude": {"anthropic"}},
|
||||
HiddenModels: map[string][]string{"claude": {"p/m"}, "codex": {"p/n", "group/g"}},
|
||||
PickedModels: map[string][]string{"opencode": {"p/m"}},
|
||||
OrderedModels: map[string][]string{"codex": {"group/g", "p/n"}},
|
||||
FastPicks: map[string][]string{"codex": {"p/n"}},
|
||||
AgentEfforts: map[string]string{"cursor-local": "high"},
|
||||
PluginCheckins: map[string]bool{"fakeco": true},
|
||||
ModelNames: map[string]string{"p/m": "Mine"},
|
||||
ModelEfforts: map[string][]string{"p/m": {"low"}},
|
||||
ModelImages: map[string]bool{"p/m": true},
|
||||
ModelOutputs: map[string]int{"p/m": 131072},
|
||||
ModelCompacts: map[string]int{"p/*": 500000},
|
||||
CompactAt: 400000,
|
||||
ModelPrices: map[string]settings.ModelPrice{"p/m": {Input: &one, Output: &two}},
|
||||
ModelWires: map[string]string{"p/m": "vendor-c/m"},
|
||||
ModelAPIs: map[string]string{"p/m": "anthropic"},
|
||||
ModelSameAs: map[string]string{"p/m": "deepseek-v4.1-flash"},
|
||||
RedactRules: []redact.Rule{{Kind: "prefix", Prefix: "oc_sk_"}},
|
||||
LAN: true,
|
||||
LANKey: "sk-lan",
|
||||
LANKeyID: "lan-key-id",
|
||||
Port: 3591,
|
||||
CORSOrigins: []string{"http://localhost:3000"},
|
||||
GitHubToken: "ghp_kept",
|
||||
RequestArchive: true,
|
||||
RequestArchiveMaxMB: 64,
|
||||
GatewayConversations: true,
|
||||
QuotaLeft: true,
|
||||
UsageOrder: []string{"kimi", "codex"},
|
||||
PanelUsageHidden: []string{"deepseek"},
|
||||
PlainNames: true,
|
||||
PlainOwnNames: true,
|
||||
CodexAgentsV1: true,
|
||||
CodexTitles: "off",
|
||||
CodexAutoReview: "fake/m1",
|
||||
CodexSubagentModel: "codex/gpt-6-astra",
|
||||
ChinaMirror: true,
|
||||
CodexAutoReset: []string{"me@example.com"},
|
||||
CodexNoCredits: []string{"me@example.com"},
|
||||
CodexWarmAtOf: map[string]string{"me@example.com": "09:00"},
|
||||
TextSize: 125,
|
||||
UpdateSkip: "0.1.500",
|
||||
UpdateMirror: "https://mirror.example/",
|
||||
Window: []int{900, 700},
|
||||
WindowMaximised: true,
|
||||
FullContext: true,
|
||||
DesktopLongest: true,
|
||||
GatewayMode: "off",
|
||||
}
|
||||
if err := settings.Save(was); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := settings.SetGatewayConversations(true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
was = settings.Load()
|
||||
rec := httptest.NewRecorder()
|
||||
Handler(nil, nil).ServeHTTP(rec, httptest.NewRequest("POST", "/api/settings", strings.NewReader(`{"theme":"dark"}`)))
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
const assert = require("node:assert/strict");
|
||||
const fs = require("node:fs/promises");
|
||||
const path = require("node:path");
|
||||
const { test } = require("node:test");
|
||||
const { chromium, webkit } = require("playwright");
|
||||
|
||||
const assets = path.resolve(__dirname, "../assets");
|
||||
for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium", "webkit"])) {
|
||||
for (const lang of ["en", "zh", "zh-TW", "ja", "de"]) {
|
||||
test(`${engine} ${lang}: gateway recording consent, content, clear and narrow layout`, async (t) => {
|
||||
const browser = await (engine === "webkit" ? webkit : chromium).launch();
|
||||
t.after(() => browser.close());
|
||||
const page = await browser.newPage({ viewport: { width: 1000, height: 850 }, reducedMotion: "reduce" });
|
||||
page.setDefaultTimeout(5000);
|
||||
let recording = false, cleared = false;
|
||||
let holdNextTranscript = false, releaseTranscript;
|
||||
const writes = [], errors = [];
|
||||
page.on("pageerror", (e) => errors.push(e.stack || e.message));
|
||||
await page.route("**/*", async (route) => {
|
||||
const u = new URL(route.request().url());
|
||||
const json = (value) => route.fulfill({ json: value });
|
||||
if (u.pathname === "/boot.js") return route.fulfill({ contentType: "text/javascript", body: `window.bootPrefs=${JSON.stringify({ web: true, gateway: true, lang, theme: "dark" })}` });
|
||||
if (u.pathname === "/api/state") return json({ agents: [], profiles: [], settings: { lang, theme: "dark", gatewayMode: "on" } });
|
||||
if (u.pathname === "/api/providers") return json({ providers: [], presets: [], excluded: [], gateway: { running: true } });
|
||||
if (u.pathname === "/api/plugins") return json({ plugins: [] });
|
||||
if (u.pathname === "/api/groups") return json({ groups: [], models: [] });
|
||||
if (u.pathname === "/api/sessions/manage") return json({
|
||||
agents: [{ agent: "claude", name: "Claude Code", icon: "claudecode", count: 1, deletable: false }],
|
||||
agent: "claude", terminal: false, trash: [], recording,
|
||||
sessions: [{ agent: "claude", id: "gateway-session", title: "Gateway conversation", start: new Date().toISOString(), last: new Date().toISOString(), input: 1200, output: 180, cache_read: 800, cache_write: 0, size: 0, models: [], read_only: true, transcript: true }],
|
||||
});
|
||||
if (u.pathname === "/api/sessions/transcript") {
|
||||
const transcript = { source: "gateway", cut: !cleared, parts: cleared ? [] : [
|
||||
{ role: "system", kind: "context", name: "System prompt", text: "Project rules" },
|
||||
{ role: "user", kind: "text", text: "Find the project version" },
|
||||
{ role: "assistant", kind: "tool_use", name: "bash", text: '{"command":"cat package.json"}' },
|
||||
{ role: "tool", kind: "tool_result", text: '{"version":"1.2.3"}' },
|
||||
{ role: "assistant", kind: "text", text: "The project version is 1.2.3." },
|
||||
] };
|
||||
if (holdNextTranscript) {
|
||||
holdNextTranscript = false;
|
||||
await new Promise((resolve) => { releaseTranscript = resolve; });
|
||||
}
|
||||
return json(transcript);
|
||||
}
|
||||
if (u.pathname === "/api/sessions/recording") {
|
||||
const body = route.request().postDataJSON();
|
||||
writes.push(body);
|
||||
recording = !!body.on && !body.clear;
|
||||
cleared ||= !!body.clear;
|
||||
return json({ recording });
|
||||
}
|
||||
if (u.pathname.startsWith("/api/")) return json({});
|
||||
const file = path.join(assets, u.pathname === "/" ? "index.html" : u.pathname);
|
||||
try { return route.fulfill({ body: await fs.readFile(file), contentType: { ".html": "text/html", ".js": "text/javascript", ".css": "text/css", ".svg": "image/svg+xml" }[path.extname(file)] }); }
|
||||
catch { return route.fulfill({ status: 404, body: "" }); }
|
||||
});
|
||||
await page.goto("http://magpie.test/?view=providers");
|
||||
await page.locator('#nav button[data-view="sessions"]').click();
|
||||
const sw = page.locator(".sm-record-label input");
|
||||
await sw.waitFor();
|
||||
assert.equal(await sw.isChecked(), false);
|
||||
await sw.click();
|
||||
await page.locator("dialog.action-confirm").waitFor();
|
||||
if (process.env.ARTIFACT_DIR) {
|
||||
await fs.mkdir(process.env.ARTIFACT_DIR, { recursive: true });
|
||||
await page.screenshot({ path: path.join(process.env.ARTIFACT_DIR, `${engine}-${lang}-recording-consent.png`), fullPage: true });
|
||||
}
|
||||
assert.equal(writes.length, 0, "no consent yet");
|
||||
await page.keyboard.press("Escape");
|
||||
assert.equal(writes.length, 0, "cancel must not record");
|
||||
await sw.click();
|
||||
await page.locator("dialog.action-confirm .primary").click();
|
||||
await page.waitForFunction(() => document.querySelector(".sm-record-label input")?.checked);
|
||||
assert.deepEqual(writes, [{ on: true }]);
|
||||
const row = page.locator('.sm-sess[data-id="gateway-session"]');
|
||||
assert.equal(await row.locator(".sm-del, .sess-carry").count(), 0);
|
||||
await row.locator(".who").click();
|
||||
const top = await row.evaluate((e) => e.getBoundingClientRect().top);
|
||||
await page.locator(".sess-talk-btn").click();
|
||||
await page.locator(".sess-talk .cx-part").first().waitFor();
|
||||
assert.equal(await page.locator(".sess-talk .cx-part").count(), 5);
|
||||
assert.equal(await page.locator(".sess-talk details").getAttribute("open"), null);
|
||||
assert.equal(await row.evaluate((e) => e.getBoundingClientRect().top), top, "no click scroll");
|
||||
for (const width of [1000, 390]) {
|
||||
await page.setViewportSize({ width, height: 850 });
|
||||
await page.waitForTimeout(100);
|
||||
assert(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth), "no page overflow");
|
||||
if (process.env.ARTIFACT_DIR) {
|
||||
await fs.mkdir(process.env.ARTIFACT_DIR, { recursive: true });
|
||||
await page.screenshot({ path: path.join(process.env.ARTIFACT_DIR, `${engine}-${lang}-gateway-conversation-${width}.png`), fullPage: true });
|
||||
}
|
||||
}
|
||||
// A transcript read started before clearing must not restore old content.
|
||||
holdNextTranscript = true;
|
||||
const readStarted = page.waitForRequest((r) => new URL(r.url()).pathname === "/api/sessions/transcript");
|
||||
await page.evaluate(() => window.loadSessionsPage());
|
||||
await readStarted;
|
||||
await page.locator(".sm-record-clear").click();
|
||||
await page.locator("dialog.action-confirm .primary").click();
|
||||
await page.waitForFunction(() => !document.querySelector(".sm-record-label input")?.checked);
|
||||
await page.waitForFunction(() => document.querySelector(".sess-talk .cx-none") && !document.querySelector(".sess-talk .cx-part"));
|
||||
const lateResponse = page.waitForResponse((r) => new URL(r.url()).pathname === "/api/sessions/transcript");
|
||||
releaseTranscript();
|
||||
await lateResponse;
|
||||
await page.evaluate(() => new Promise((resolve) => requestAnimationFrame(() => requestAnimationFrame(resolve))));
|
||||
assert.equal(await page.locator(".sess-talk .cx-part").count(), 0, "late transcript cannot restore cleared content");
|
||||
assert.deepEqual(writes, [{ on: true }, { clear: true }]);
|
||||
if (process.env.ARTIFACT_DIR) await page.screenshot({ path: path.join(process.env.ARTIFACT_DIR, `${engine}-${lang}-recording-cleared.png`), fullPage: true });
|
||||
assert.equal(await row.count(), 1, "usage session survives clear");
|
||||
assert.deepEqual(errors, []);
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
// Run with Node's test runner and Playwright on the module path; see README.md.
|
||||
// Gateway mode (Player on Discord): magpie web on a server that is only
|
||||
// the gateway for other computers' agents. boot.js says so and the page
|
||||
// opens on Providers with no Agents, Sessions or Library tab, an address
|
||||
// opens on Providers with no Agents or Library tab, an address
|
||||
// kept for one of them opening Providers; Settings leaves out what is
|
||||
// written into this computer's agents' files (provider in model names,
|
||||
// Codex subagents, long conversations, Codex thread titles) and the
|
||||
@@ -70,8 +70,8 @@ const L = {
|
||||
en: { mode: "Gateway mode", noAgents: "Now on: no agents on this computer", off: "Now off: agents found on this computer" },
|
||||
zh: { mode: "网关模式", noAgents: "当前开启:本机没有 Agent", off: "当前关闭:本机找到了 Agent" },
|
||||
};
|
||||
const HIDDEN = ["agents", "sessions", "library"];
|
||||
const KEPT = ["providers", "gateway", "routing", "usage", "plugins"];
|
||||
const HIDDEN = ["agents", "library"];
|
||||
const KEPT = ["providers", "gateway", "routing", "usage", "sessions", "plugins"];
|
||||
|
||||
// the tabs shown, the page shown
|
||||
const tabs = (page) => page.evaluate(() => [...document.querySelectorAll("#nav button")].filter((b) => !b.hidden && b.offsetParent).map((b) => b.dataset.view));
|
||||
|
||||
@@ -71,7 +71,7 @@ func keep(p Part) (Part, bool) {
|
||||
}
|
||||
if n := utf8.RuneCountInString(p.Text); n > partMax {
|
||||
r := []rune(p.Text)
|
||||
p.Text, p.Cut = string(r[:partMax]), n-partMax
|
||||
p.Text, p.Cut = string(r[:partMax]), p.Cut+n-partMax
|
||||
}
|
||||
return p, true
|
||||
}
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
package sessions
|
||||
|
||||
import (
|
||||
"math"
|
||||
"slices"
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ExternalModel and ExternalSession are read-only session projections from a
|
||||
// request ledger such as the gateway. They intentionally carry no file or
|
||||
// resume capability.
|
||||
type ExternalModel struct {
|
||||
Model string
|
||||
Tokens
|
||||
Cost float64
|
||||
Priced bool
|
||||
}
|
||||
type ExternalSession struct {
|
||||
Agent, ID string
|
||||
Start, Last time.Time
|
||||
Models []ExternalModel
|
||||
Daily []ExternalDayUsage
|
||||
Tokens
|
||||
Cost float64
|
||||
Unpriced int
|
||||
}
|
||||
|
||||
// MergeExternal adds external sessions to a file-backed listing. Native
|
||||
// sessions win on the exact Agent+ID identity and external entries are kept
|
||||
// read-only with a gateway transcript reader, but no resume, terminal or delete capability.
|
||||
func MergeExternal(native []Session, external []ExternalSession) []Session {
|
||||
seen := map[string]bool{}
|
||||
for _, s := range native {
|
||||
seen[s.Agent+"|"+s.ID] = true
|
||||
}
|
||||
out := append([]Session(nil), native...)
|
||||
for _, e := range external {
|
||||
if e.Agent == "" || e.ID == "" || seen[e.Agent+"|"+e.ID] {
|
||||
continue
|
||||
}
|
||||
s := Session{ReadOnly: true, Transcript: true, Agent: e.Agent, ID: e.ID, Title: e.ID, Start: e.Start, Last: e.Last, Models: []Model{}, Unpriced: e.Unpriced}
|
||||
for _, m := range e.Models {
|
||||
s.Models = append(s.Models, Model{Model: m.Model, Tokens: m.Tokens, Cost: m.Cost, Priced: m.Priced})
|
||||
s.Tokens.add(m.Tokens)
|
||||
}
|
||||
s.Cost = e.Cost
|
||||
out = append(out, s)
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].Last.After(out[j].Last) })
|
||||
return out
|
||||
}
|
||||
|
||||
// MergeExternalStats appends external daily usage and session summaries to a
|
||||
// Stats snapshot. It is deliberately data-only so callers can keep their own
|
||||
// source and pricing policy.
|
||||
type ExternalDayUsage struct {
|
||||
Date, Agent, Cwd, Model string
|
||||
Tokens
|
||||
Cost float64
|
||||
Priced bool
|
||||
}
|
||||
|
||||
func MergeExternalStats(st Stats, days []ExternalDayUsage, external []ExternalSession) Stats {
|
||||
oldFirst, _ := time.ParseInLocation(time.DateOnly, st.From, time.Local)
|
||||
for _, d := range days {
|
||||
if d.Date <= st.To && (st.From == "" || d.Date < st.From) {
|
||||
st.From = d.Date
|
||||
}
|
||||
}
|
||||
first, _ := time.ParseInLocation(time.DateOnly, st.From, time.Local)
|
||||
if !oldFirst.IsZero() && !first.Equal(oldFirst) {
|
||||
shift := int(math.Round(oldFirst.Sub(first).Hours() / 24))
|
||||
st.Sessions = slices.Clone(st.Sessions)
|
||||
for i := range st.Sessions {
|
||||
s := &st.Sessions[i]
|
||||
s.Days = slices.Clone(s.Days)
|
||||
for j := range s.Days {
|
||||
s.Days[j] += shift
|
||||
}
|
||||
pd := map[int]*summaryDay{}
|
||||
for d, v := range s.perDay {
|
||||
pd[d+shift] = v
|
||||
}
|
||||
s.perDay = pd
|
||||
}
|
||||
}
|
||||
byDate := map[string]int{}
|
||||
for i := range st.Days {
|
||||
byDate[st.Days[i].Date] = i
|
||||
}
|
||||
for _, u := range days {
|
||||
if u.Date < st.From || u.Date > st.To {
|
||||
continue
|
||||
}
|
||||
i, ok := byDate[u.Date]
|
||||
if !ok {
|
||||
st.Days = append(st.Days, Day{Date: u.Date, Usage: []Usage{}, Active: []Active{}})
|
||||
i = len(st.Days) - 1
|
||||
byDate[u.Date] = i
|
||||
}
|
||||
d := &st.Days[i]
|
||||
found := false
|
||||
for i := range d.Usage {
|
||||
if d.Usage[i].Agent == u.Agent && d.Usage[i].Cwd == u.Cwd && d.Usage[i].Model == u.Model {
|
||||
d.Usage[i].Tokens.add(u.Tokens)
|
||||
d.Usage[i].Cost += u.Cost
|
||||
d.Usage[i].Priced = d.Usage[i].Priced && u.Priced
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
d.Usage = append(d.Usage, Usage{Agent: u.Agent, Cwd: u.Cwd, Model: u.Model, Tokens: u.Tokens, Cost: u.Cost, Priced: u.Priced})
|
||||
}
|
||||
}
|
||||
for _, e := range external {
|
||||
if e.ID == "" || e.Agent == "" {
|
||||
continue
|
||||
}
|
||||
s := Summary{Key: e.Agent + ":" + e.ID, Agent: e.Agent, ID: e.ID, Last: e.Last, Tokens: e.Tokens, Cost: e.Cost, Priced: true, Models: []string{}, Days: []int{}, perDay: map[int]*summaryDay{}}
|
||||
for _, u := range e.Daily {
|
||||
day, err := time.ParseInLocation(time.DateOnly, u.Date, time.Local)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
d := int(math.Round(day.Sub(first).Hours() / 24))
|
||||
if first.IsZero() || u.Date < st.From || u.Date > st.To {
|
||||
continue
|
||||
}
|
||||
pd := s.perDay[d]
|
||||
if pd == nil {
|
||||
pd = &summaryDay{}
|
||||
s.perDay[d] = pd
|
||||
s.Days = append(s.Days, d)
|
||||
}
|
||||
pd.output += u.Output
|
||||
}
|
||||
sort.Ints(s.Days)
|
||||
for _, m := range e.Models {
|
||||
s.Models = append(s.Models, m.Model)
|
||||
s.Priced = s.Priced && m.Priced
|
||||
}
|
||||
st.Sessions = append(st.Sessions, s)
|
||||
}
|
||||
sort.Slice(st.Days, func(i, j int) bool { return st.Days[i].Date < st.Days[j].Date })
|
||||
sort.SliceStable(st.Sessions, func(i, j int) bool { return st.Sessions[i].Last.After(st.Sessions[j].Last) })
|
||||
return st
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
package sessions
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGatewayStatsRebaseNativeCalendar(t *testing.T) {
|
||||
native := Stats{From: "2026-10-06", To: "2026-10-07", Sessions: []Summary{{Agent: "claude", ID: "native", Days: []int{0}, perDay: map[int]*summaryDay{0: {output: 4}}}}}
|
||||
merged := MergeExternalStats(native, []ExternalDayUsage{{Date: "2026-10-04", Agent: "opencode"}}, nil)
|
||||
view := merged.Overview("claude", "", "", 8)
|
||||
if !reflect.DeepEqual(view.Days, []int{0, 0, 1, 0}) || !reflect.DeepEqual(view.Output, []int{0, 0, 4, 0}) {
|
||||
t.Fatalf("native calendar shifted when gateway history extended range: days=%v output=%v", view.Days, view.Output)
|
||||
}
|
||||
if native.Sessions[0].Days[0] != 0 || native.Sessions[0].perDay[0].output != 4 {
|
||||
t.Fatal("merge mutated the original native snapshot")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,411 @@
|
||||
package sessions
|
||||
|
||||
// Gateway conversations are a separate, opt-in local store, not native agent
|
||||
// files or the usage ledger. IDs are hashed before they become path components.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/yetone/magpie/internal/redact"
|
||||
"github.com/yetone/magpie/internal/settings"
|
||||
)
|
||||
|
||||
const GatewayRetention = 7 * 24 * time.Hour
|
||||
const gatewayTurnMax = 1 << 20
|
||||
const gatewayReadMax = 128
|
||||
const gatewayStoreMax = 256 << 20
|
||||
const gatewaySessionMax = gatewayStoreMax
|
||||
|
||||
var gatewayMu sync.Mutex
|
||||
var gatewayGeneration uint64 = 1
|
||||
|
||||
// GatewayTurn is one captured client-facing exchange. Input may include replayed
|
||||
// history; Output is only this request's reply. It is not a native session.
|
||||
type GatewayTurn struct {
|
||||
Agent string `json:"agent"`
|
||||
Session string `json:"session"`
|
||||
Time time.Time `json:"time"`
|
||||
Model string `json:"model"`
|
||||
Status int `json:"status"`
|
||||
Input []Part `json:"input"`
|
||||
Output []Part `json:"output"`
|
||||
Cut bool `json:"cut,omitempty"`
|
||||
}
|
||||
|
||||
func gatewayDir() string { return filepath.Join(settings.Dir(), "gateway-conversations") }
|
||||
|
||||
func gatewayIdentity(agent, id string) string {
|
||||
b, _ := json.Marshal([]string{agent, id})
|
||||
h := sha256.Sum256(b)
|
||||
return hex.EncodeToString(h[:])
|
||||
}
|
||||
|
||||
// SaveGatewayTurn keeps a bounded, scrubbed exchange. Recording is checked again
|
||||
// under the same lock as disable-and-clear, so in-flight requests cannot refill it.
|
||||
func SaveGatewayTurn(turn GatewayTurn) error {
|
||||
return saveGatewayTurn(turn, GatewayRecordingGeneration())
|
||||
}
|
||||
|
||||
// GatewayRecordingGeneration returns the current consent generation. A
|
||||
// request captures it when recording starts and must present the same value
|
||||
// when saving, so clearing and re-enabling recording cannot resurrect old data.
|
||||
func GatewayRecordingGeneration() uint64 {
|
||||
gatewayMu.Lock()
|
||||
defer gatewayMu.Unlock()
|
||||
return gatewayGeneration
|
||||
}
|
||||
|
||||
func SaveGatewayTurnGeneration(turn GatewayTurn, generation uint64) error {
|
||||
return saveGatewayTurn(turn, generation)
|
||||
}
|
||||
|
||||
func saveGatewayTurn(turn GatewayTurn, generation uint64) error {
|
||||
gatewayMu.Lock()
|
||||
allowed := settings.Load().GatewayConversations && generation == gatewayGeneration
|
||||
gatewayMu.Unlock()
|
||||
if !allowed || turn.Agent == "" || turn.Session == "" {
|
||||
return nil
|
||||
}
|
||||
if turn.Time.IsZero() {
|
||||
turn.Time = time.Now()
|
||||
}
|
||||
if turn.Time.Before(time.Now().Add(-GatewayRetention)) {
|
||||
return nil
|
||||
}
|
||||
bound := func(parts []Part) []Part {
|
||||
left := 64 << 10 // Each direction gets its own budget; history cannot crowd out the reply.
|
||||
out := []Part{}
|
||||
for _, p := range parts {
|
||||
if left <= 0 {
|
||||
turn.Cut = true
|
||||
break
|
||||
}
|
||||
p.Text = string(redact.ScrubJSON([]byte(p.Text)))
|
||||
p.Name = redact.Scrub(p.Name)
|
||||
if len(p.Text) > min(left, 64<<10) {
|
||||
n := min(left, 64<<10)
|
||||
for n > 0 && !utf8.ValidString(p.Text[:n]) {
|
||||
n--
|
||||
}
|
||||
p.Cut += utf8.RuneCountInString(p.Text[n:])
|
||||
p.Text = p.Text[:n]
|
||||
turn.Cut = true
|
||||
}
|
||||
left -= len(p.Text) + len(p.Name) + 128
|
||||
out = append(out, p)
|
||||
}
|
||||
return out
|
||||
}
|
||||
turn.Input = bound(turn.Input)
|
||||
turn.Output = bound(turn.Output)
|
||||
b, err := json.Marshal(turn)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(b) > gatewayTurnMax {
|
||||
return errors.New("gateway conversation exceeds storage limit")
|
||||
}
|
||||
gatewayMu.Lock()
|
||||
defer gatewayMu.Unlock()
|
||||
if generation != gatewayGeneration || !settings.Load().GatewayConversations {
|
||||
return nil
|
||||
}
|
||||
dir := filepath.Join(gatewayDir(), turn.Time.UTC().Format(time.DateOnly), gatewayIdentity(turn.Agent, turn.Session))
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
// Append one bounded line; the cost is independent of other sessions and
|
||||
// of this session's earlier turns. Cleanup runs outside the request path.
|
||||
path := filepath.Join(dir, "turns.jsonl")
|
||||
f, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR|os.O_APPEND, 0o600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
info, err := f.Stat()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
end, err := gatewayCompleteSize(f, info.Size())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if end+int64(len(b))+1 > gatewaySessionMax {
|
||||
return errors.New("gateway session exceeds storage limit")
|
||||
}
|
||||
// A process interrupted mid-append must not poison all later turns.
|
||||
if end != info.Size() {
|
||||
if err := f.Truncate(end); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
_, err = f.Write(append(b, '\n'))
|
||||
return errors.Join(err, f.Close())
|
||||
}
|
||||
|
||||
// Ignore at most one unfinished turn; earlier complete lines remain intact.
|
||||
func gatewayCompleteSize(f *os.File, size int64) (int64, error) {
|
||||
if size == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
var last [1]byte
|
||||
if _, err := f.ReadAt(last[:], size-1); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if last[0] == '\n' {
|
||||
return size, nil
|
||||
}
|
||||
n := min(size, int64(gatewayTurnMax+1))
|
||||
tail := make([]byte, int(n))
|
||||
if _, err := f.ReadAt(tail, size-n); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if i := bytes.LastIndexByte(tail, '\n'); i >= 0 {
|
||||
return size - n + int64(i) + 1, nil
|
||||
}
|
||||
if n == size {
|
||||
return 0, nil
|
||||
}
|
||||
return 0, errors.New("unreadable gateway conversation")
|
||||
}
|
||||
|
||||
// SetGatewayRecording changes only this machine's consent. Clear disables first.
|
||||
func SetGatewayRecording(on, clear bool) error {
|
||||
gatewayMu.Lock()
|
||||
defer gatewayMu.Unlock()
|
||||
if clear || !on {
|
||||
gatewayGeneration++
|
||||
}
|
||||
if err := settings.SetGatewayConversations(on && !clear); err != nil {
|
||||
return err
|
||||
}
|
||||
if clear {
|
||||
return os.RemoveAll(gatewayDir())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// PruneGatewayConversations removes expired date buckets even while recording is
|
||||
// off. Reads enforce the exact cutoff too, between periodic cleanup passes.
|
||||
func PruneGatewayConversations(now time.Time) error {
|
||||
gatewayMu.Lock()
|
||||
root, generation := gatewayDir(), gatewayGeneration
|
||||
gatewayMu.Unlock()
|
||||
type saved struct {
|
||||
path string
|
||||
info os.FileInfo
|
||||
expired bool
|
||||
}
|
||||
var files []saved
|
||||
var total int64
|
||||
// Walking and sorting never hold the lock needed by a response's save.
|
||||
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !d.Type().IsRegular() {
|
||||
return nil
|
||||
}
|
||||
info, err := d.Info()
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
rel, err := filepath.Rel(root, path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
date, err := time.Parse(time.DateOnly, strings.Split(rel, string(filepath.Separator))[0])
|
||||
expired := err == nil && !date.Add(24*time.Hour).After(now.Add(-GatewayRetention))
|
||||
files = append(files, saved{path, info, expired})
|
||||
total += info.Size()
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sort.Slice(files, func(i, j int) bool {
|
||||
if files[i].expired != files[j].expired {
|
||||
return files[i].expired
|
||||
}
|
||||
return files[i].info.ModTime().Before(files[j].info.ModTime())
|
||||
})
|
||||
for _, f := range files {
|
||||
if !f.expired && total <= gatewayStoreMax {
|
||||
continue
|
||||
}
|
||||
gatewayMu.Lock()
|
||||
if generation != gatewayGeneration || root != gatewayDir() {
|
||||
gatewayMu.Unlock()
|
||||
return nil
|
||||
}
|
||||
current, err := os.Stat(f.path)
|
||||
if err == nil && os.SameFile(current, f.info) && current.Size() == f.info.Size() && current.ModTime().Equal(f.info.ModTime()) {
|
||||
err = os.Remove(f.path)
|
||||
if err == nil {
|
||||
total -= f.info.Size()
|
||||
_ = os.Remove(filepath.Dir(f.path))
|
||||
_ = os.Remove(filepath.Dir(filepath.Dir(f.path)))
|
||||
}
|
||||
}
|
||||
gatewayMu.Unlock()
|
||||
if err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GatewayTranscript reads recent captured exchanges. Only a full matching
|
||||
// previous history prefix is removed, never equal text found elsewhere in a chat.
|
||||
func GatewayTranscript(agent, id string) (Transcript, error) {
|
||||
out := Transcript{Parts: []Part{}, Source: "gateway"}
|
||||
if agent == "" || id == "" {
|
||||
return out, errors.New("no such session")
|
||||
}
|
||||
gatewayMu.Lock()
|
||||
defer gatewayMu.Unlock()
|
||||
now := time.Now()
|
||||
var paths []string
|
||||
for d := 0; d <= 7; d++ {
|
||||
dir := filepath.Join(gatewayDir(), now.UTC().AddDate(0, 0, -d).Format(time.DateOnly), gatewayIdentity(agent, id))
|
||||
entries, err := os.ReadDir(dir)
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
for _, e := range entries {
|
||||
if e.Type().IsRegular() && (strings.HasSuffix(e.Name(), ".json") || e.Name() == "turns.jsonl") {
|
||||
paths = append(paths, filepath.Join(dir, e.Name()))
|
||||
}
|
||||
}
|
||||
}
|
||||
var turns []GatewayTurn
|
||||
keep := func(turn GatewayTurn) {
|
||||
if turn.Agent != agent || turn.Session != id || turn.Time.Before(now.Add(-GatewayRetention)) {
|
||||
return
|
||||
}
|
||||
turns = append(turns, turn)
|
||||
if len(turns) > gatewayReadMax*2 {
|
||||
sort.SliceStable(turns, func(i, j int) bool { return turns[i].Time.Before(turns[j].Time) })
|
||||
turns = slices.Clone(turns[len(turns)-gatewayReadMax:])
|
||||
out.Cut = true
|
||||
}
|
||||
}
|
||||
for _, path := range paths {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if strings.HasSuffix(path, ".jsonl") {
|
||||
var info os.FileInfo
|
||||
info, err = f.Stat()
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return out, err
|
||||
}
|
||||
if info.Size() > gatewaySessionMax {
|
||||
f.Close()
|
||||
return out, errors.New("gateway session exceeds storage limit")
|
||||
}
|
||||
var end int64
|
||||
end, err = gatewayCompleteSize(f, info.Size())
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return out, err
|
||||
}
|
||||
out.Cut = out.Cut || end != info.Size()
|
||||
scan := bufio.NewScanner(io.LimitReader(f, end))
|
||||
scan.Buffer(make([]byte, 64<<10), gatewayTurnMax+1)
|
||||
for scan.Scan() {
|
||||
var turn GatewayTurn
|
||||
if json.Unmarshal(scan.Bytes(), &turn) != nil {
|
||||
f.Close()
|
||||
return out, errors.New("unreadable gateway conversation")
|
||||
}
|
||||
keep(turn)
|
||||
}
|
||||
err = scan.Err()
|
||||
} else {
|
||||
// Read the original per-turn files until their retention expires.
|
||||
var data []byte
|
||||
data, err = io.ReadAll(io.LimitReader(f, gatewayTurnMax+1))
|
||||
var turn GatewayTurn
|
||||
if err == nil && (len(data) > gatewayTurnMax || json.Unmarshal(data, &turn) != nil) {
|
||||
err = errors.New("unreadable gateway conversation")
|
||||
}
|
||||
if err == nil {
|
||||
keep(turn)
|
||||
}
|
||||
}
|
||||
f.Close()
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
}
|
||||
sort.SliceStable(turns, func(i, j int) bool { return turns[i].Time.Before(turns[j].Time) })
|
||||
if len(turns) > gatewayReadMax {
|
||||
turns = turns[len(turns)-gatewayReadMax:]
|
||||
out.Cut = true
|
||||
}
|
||||
var previous, system []Part
|
||||
for _, turn := range turns {
|
||||
out.Captured++
|
||||
out.Cut = out.Cut || turn.Cut
|
||||
// Context is kept independently because it is repeated before every
|
||||
// request's conversation, rather than being a new conversational turn.
|
||||
var context, input []Part
|
||||
for _, p := range turn.Input {
|
||||
if p.Role == "system" {
|
||||
context = append(context, p)
|
||||
} else {
|
||||
input = append(input, p)
|
||||
}
|
||||
}
|
||||
if !slices.Equal(context, system) {
|
||||
for _, p := range context {
|
||||
if !out.add(true, p) {
|
||||
return out, nil
|
||||
}
|
||||
}
|
||||
system = context
|
||||
}
|
||||
fresh := input
|
||||
if len(previous) > 0 && len(input) >= len(previous) && slices.Equal(input[:len(previous)], previous) {
|
||||
fresh = input[len(previous):]
|
||||
}
|
||||
for _, p := range fresh {
|
||||
if !out.add(true, p) {
|
||||
return out, nil
|
||||
}
|
||||
}
|
||||
for _, p := range turn.Output {
|
||||
if !out.add(false, p) {
|
||||
return out, nil
|
||||
}
|
||||
}
|
||||
previous = append(slices.Clone(input), turn.Output...)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,374 @@
|
||||
package sessions
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/yetone/magpie/internal/settings"
|
||||
)
|
||||
|
||||
func gatewayHome(t *testing.T) {
|
||||
t.Helper()
|
||||
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
|
||||
if err := SetGatewayRecording(false, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationManyTurnsShareFile(t *testing.T) {
|
||||
gatewayHome(t)
|
||||
if err := SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
at := time.Now().UTC().Truncate(time.Hour)
|
||||
for i := range 4200 {
|
||||
turn := GatewayTurn{Agent: "pi", Session: "s", Time: at.Add(time.Duration(i) * time.Millisecond), Output: []Part{{Role: "assistant", Kind: "text", Text: fmt.Sprint(i)}}}
|
||||
if err := SaveGatewayTurn(turn); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
dir := filepath.Join(gatewayDir(), at.UTC().Format(time.DateOnly), gatewayIdentity("pi", "s"))
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil || len(entries) != 1 {
|
||||
t.Fatalf("one session should use one file: %d %v", len(entries), err)
|
||||
}
|
||||
tr, err := GatewayTranscript("pi", "s")
|
||||
if err != nil || tr.Captured != gatewayReadMax || !tr.Cut || tr.Parts[len(tr.Parts)-1].Text != "4199" {
|
||||
t.Fatalf("recent turns missing: %+v %v", tr, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationClearAndReenableRejectsOldRequest(t *testing.T) {
|
||||
gatewayHome(t)
|
||||
if err := SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
generation := GatewayRecordingGeneration()
|
||||
if err := SetGatewayRecording(false, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := SaveGatewayTurnGeneration(GatewayTurn{Agent: "pi", Session: "s", Time: time.Now()}, generation); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(gatewayDir()); !os.IsNotExist(err) {
|
||||
t.Fatal("old request resurrected cleared content")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationLegacyAndNewTurns(t *testing.T) {
|
||||
gatewayHome(t)
|
||||
if err := SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
at := time.Now()
|
||||
dir := filepath.Join(gatewayDir(), at.UTC().Format(time.DateOnly), gatewayIdentity("pi", "s"))
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := GatewayTurn{Agent: "pi", Session: "s", Time: at, Output: []Part{{Role: "assistant", Kind: "text", Text: "old"}}}
|
||||
b, _ := json.Marshal(old)
|
||||
if err := os.WriteFile(filepath.Join(dir, "old.json"), b, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old.Time = at.Add(time.Second)
|
||||
old.Output[0].Text = "new"
|
||||
if err := SaveGatewayTurn(old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr, err := GatewayTranscript("pi", "s")
|
||||
if err != nil || tr.Captured != 2 || tr.Parts[0].Text != "old" || tr.Parts[1].Text != "new" {
|
||||
t.Fatalf("legacy content lost: %+v %v", tr, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationSaveDoesNotScanOtherSessions(t *testing.T) {
|
||||
gatewayHome(t)
|
||||
if err := SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
other := filepath.Join(gatewayDir(), "unreadable")
|
||||
if err := os.MkdirAll(other, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Chmod(other, 0); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer os.Chmod(other, 0o700)
|
||||
if err := SaveGatewayTurn(GatewayTurn{Agent: "pi", Session: "s", Time: time.Now()}); err != nil {
|
||||
t.Fatalf("save scanned unrelated storage: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationConsentAndIsolation(t *testing.T) {
|
||||
gatewayHome(t)
|
||||
now := time.Now()
|
||||
u := Part{Role: "user", Kind: "text", Text: "hello"}
|
||||
a := Part{Role: "assistant", Kind: "text", Text: "world"}
|
||||
turn := GatewayTurn{Agent: "claude", Session: "../../outside", Time: now, Input: []Part{u}, Output: []Part{a}}
|
||||
if err := SaveGatewayTurn(turn); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(gatewayDir()); !os.IsNotExist(err) {
|
||||
t.Fatal("recorded without consent")
|
||||
}
|
||||
if err := SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := SaveGatewayTurn(turn); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
turn.Time = now.Add(time.Millisecond)
|
||||
turn.Input = []Part{u, a, {Role: "user", Kind: "text", Text: "hello again"}}
|
||||
turn.Output = []Part{{Role: "assistant", Kind: "text", Text: "second reply"}}
|
||||
if err := SaveGatewayTurn(turn); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr, err := GatewayTranscript("claude", "../../outside")
|
||||
if err != nil || len(tr.Parts) != 4 || tr.Captured != 2 {
|
||||
t.Fatalf("replayed history: %+v %v", tr, err)
|
||||
}
|
||||
other, err := GatewayTranscript("codex", "../../outside")
|
||||
if err != nil || len(other.Parts) != 0 {
|
||||
t.Fatalf("identity leak: %+v %v", other, err)
|
||||
}
|
||||
if err := SetGatewayRecording(false, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := SaveGatewayTurn(turn); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr, _ = GatewayTranscript("claude", "../../outside")
|
||||
if tr.Captured != 2 {
|
||||
t.Fatal("off deleted or added content")
|
||||
}
|
||||
if err := SetGatewayRecording(false, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := SaveGatewayTurn(turn); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr, _ = GatewayTranscript("claude", "../../outside")
|
||||
if tr.Captured != 0 || settings.Load().GatewayConversations {
|
||||
t.Fatal("clear refilled by in-flight turn")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationLimitsAndExpiry(t *testing.T) {
|
||||
gatewayHome(t)
|
||||
if err := SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
turn := GatewayTurn{Agent: "pi", Session: "s", Time: time.Now(), Input: []Part{{Role: "user", Kind: "text", Text: strings.Repeat("a", 100_000)}}}
|
||||
if err := SaveGatewayTurn(turn); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr, err := GatewayTranscript("pi", "s")
|
||||
if err != nil || !tr.Cut || len(tr.Parts) != 1 || tr.Parts[0].Cut != 92_000 {
|
||||
t.Fatalf("limit: %+v %v", tr, err)
|
||||
}
|
||||
old := filepath.Join(gatewayDir(), time.Now().Add(-9*24*time.Hour).UTC().Format(time.DateOnly))
|
||||
if err := os.MkdirAll(old, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(old, "expired"), []byte("private"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := PruneGatewayConversations(time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(old); !os.IsNotExist(err) {
|
||||
t.Fatal("expired content retained")
|
||||
}
|
||||
turn.Time = time.Now().Add(-8 * 24 * time.Hour)
|
||||
if err := SaveGatewayTurn(turn); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr, _ = GatewayTranscript("pi", "s")
|
||||
if tr.Captured != 1 {
|
||||
t.Fatal("expired turn accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationKeepsRepeatedNewTurns(t *testing.T) {
|
||||
gatewayHome(t)
|
||||
if err := SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
u := Part{Role: "user", Kind: "text", Text: "again"}
|
||||
a := Part{Role: "assistant", Kind: "text", Text: "done"}
|
||||
for i := range 2 {
|
||||
if err := SaveGatewayTurn(GatewayTurn{Agent: "codex", Session: "s", Time: time.Now().Add(time.Duration(i) * time.Millisecond), Input: []Part{u}, Output: []Part{a}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
tr, err := GatewayTranscript("codex", "s")
|
||||
if err != nil || len(tr.Parts) != 4 {
|
||||
t.Fatalf("incremental repeated turns erased: %+v %v", tr, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationInterruptedAppend(t *testing.T) {
|
||||
gatewayHome(t)
|
||||
if err := SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
at := time.Now()
|
||||
turn := GatewayTurn{Agent: "pi", Session: "s", Time: at, Output: []Part{{Role: "assistant", Kind: "text", Text: "first"}}}
|
||||
if err := SaveGatewayTurn(turn); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
path := filepath.Join(gatewayDir(), at.UTC().Format(time.DateOnly), gatewayIdentity("pi", "s"), "turns.jsonl")
|
||||
f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0600)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.WriteString(`{"agent":"pi","session":`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr, err := GatewayTranscript("pi", "s")
|
||||
if err != nil || tr.Captured != 1 || !tr.Cut {
|
||||
t.Fatalf("partial append hid complete turns: %+v %v", tr, err)
|
||||
}
|
||||
turn.Time = at.Add(time.Second)
|
||||
turn.Output[0].Text = "second"
|
||||
if err := SaveGatewayTurn(turn); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr, err = GatewayTranscript("pi", "s")
|
||||
if err != nil || tr.Captured != 2 || tr.Parts[1].Text != "second" {
|
||||
t.Fatalf("partial append poisoned later turns: %+v %v", tr, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationConcurrentSaveAndPrune(t *testing.T) {
|
||||
gatewayHome(t)
|
||||
if err := SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
at := time.Now()
|
||||
var wg sync.WaitGroup
|
||||
for i := range 20 {
|
||||
wg.Go(func() {
|
||||
if err := SaveGatewayTurn(GatewayTurn{Agent: "pi", Session: "s", Time: at.Add(time.Duration(i) * time.Millisecond), Output: []Part{{Role: "assistant", Kind: "text", Text: fmt.Sprint(i)}}}); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
})
|
||||
}
|
||||
wg.Go(func() {
|
||||
if err := PruneGatewayConversations(at); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
})
|
||||
wg.Wait()
|
||||
tr, err := GatewayTranscript("pi", "s")
|
||||
if err != nil || tr.Captured != 20 {
|
||||
t.Fatalf("concurrent saves lost turns: %+v %v", tr, err)
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkGatewayConversationSaveWithHistory(b *testing.B) {
|
||||
for _, files := range []int{0, 4000} {
|
||||
b.Run(fmt.Sprint(files), func(b *testing.B) {
|
||||
b.Setenv("XDG_CONFIG_HOME", b.TempDir())
|
||||
if err := SetGatewayRecording(true, false); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
dir := filepath.Join(gatewayDir(), "history")
|
||||
if err := os.MkdirAll(dir, 0700); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
data := make([]byte, 32<<10)
|
||||
for i := range files {
|
||||
if err := os.WriteFile(filepath.Join(dir, fmt.Sprintf("%d.json", i)), data, 0600); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
turn := GatewayTurn{Agent: "pi", Session: "benchmark", Time: time.Now(), Output: []Part{{Role: "assistant", Kind: "text", Text: "reply"}}}
|
||||
b.ResetTimer()
|
||||
for b.Loop() {
|
||||
if err := SaveGatewayTurn(turn); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationOversizedLine(t *testing.T) {
|
||||
gatewayHome(t)
|
||||
if err := SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
dir := filepath.Join(gatewayDir(), time.Now().UTC().Format(time.DateOnly), gatewayIdentity("pi", "s"))
|
||||
if err := os.MkdirAll(dir, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "turns.jsonl"), []byte(strings.Repeat(" ", gatewayTurnMax+1)+"\n"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := GatewayTranscript("pi", "s"); err == nil {
|
||||
t.Fatal("oversized corrupt line reported as an empty transcript")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationQuotaRunsInBackground(t *testing.T) {
|
||||
gatewayHome(t)
|
||||
if err := SetGatewayRecording(true, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
at := time.Now()
|
||||
paths := []string{}
|
||||
for i := range 2 {
|
||||
dir := filepath.Join(gatewayDir(), at.UTC().Format(time.DateOnly), fmt.Sprint(i))
|
||||
if err := os.MkdirAll(dir, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
path := filepath.Join(dir, "turns.jsonl")
|
||||
f, err := os.Create(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
err = f.Truncate(gatewayStoreMax/2 + 1)
|
||||
f.Close()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stamp := at.Add(time.Duration(i-2) * time.Hour)
|
||||
if err := os.Chtimes(path, stamp, stamp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
paths = append(paths, path)
|
||||
}
|
||||
if err := SaveGatewayTurn(GatewayTurn{Agent: "pi", Session: "s", Time: at}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, path := range paths {
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
t.Fatalf("save pruned unrelated session: %v", err)
|
||||
}
|
||||
}
|
||||
if err := PruneGatewayConversations(at); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(paths[0]); !os.IsNotExist(err) {
|
||||
t.Fatal("background cleanup retained oldest over-quota file")
|
||||
}
|
||||
if _, err := os.Stat(paths[1]); err != nil {
|
||||
t.Fatalf("background cleanup removed newer file: %v", err)
|
||||
}
|
||||
if tr, err := GatewayTranscript("pi", "s"); err != nil || tr.Captured != 1 {
|
||||
t.Fatalf("cleanup removed current turn: %+v %v", tr, err)
|
||||
}
|
||||
}
|
||||
@@ -13,8 +13,10 @@ import (
|
||||
|
||||
// Transcript is what was said in a session, in the order it was said.
|
||||
type Transcript struct {
|
||||
Parts []Part `json:"parts"`
|
||||
Cut bool `json:"cut,omitempty"` // there was more than is kept here
|
||||
Parts []Part `json:"parts"`
|
||||
Cut bool `json:"cut,omitempty"` // there was more than is kept here
|
||||
Source string `json:"source,omitempty"`
|
||||
Captured int `json:"captured,omitempty"`
|
||||
}
|
||||
|
||||
// transcriptMax is the characters of all of a transcript's parts.
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
package settings
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestGatewayConversationConsentIsMachineLocal(t *testing.T) {
|
||||
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
|
||||
if Load().GatewayConversations {
|
||||
t.Fatal("recording on by default")
|
||||
}
|
||||
from := Settings{GatewayConversations: true}
|
||||
from.KeepOwn(Settings{})
|
||||
if from.GatewayConversations {
|
||||
t.Fatal("sync enabled recording")
|
||||
}
|
||||
from.KeepOwn(Settings{GatewayConversations: true})
|
||||
if !from.GatewayConversations {
|
||||
t.Fatal("sync removed local consent")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationGenericSaveCannotEnable(t *testing.T) {
|
||||
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
|
||||
if err := Save(Settings{GatewayConversations: true}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if Load().GatewayConversations {
|
||||
t.Fatal("generic settings save enabled recording")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewayConversationConsentSurvivesStaleSettings(t *testing.T) {
|
||||
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
|
||||
if err := SetGatewayConversations(true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
staleOn := Load()
|
||||
if !staleOn.GatewayConversations {
|
||||
t.Fatal("explicit consent was not saved")
|
||||
}
|
||||
if err := SetGatewayConversations(false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
staleOn.Theme = "dark"
|
||||
if err := Save(staleOn); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := Load(); got.GatewayConversations || got.Theme != "dark" {
|
||||
t.Fatal("stale settings save enabled recording or lost its theme change")
|
||||
}
|
||||
staleOff := Load()
|
||||
if err := SetGatewayConversations(true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
staleOff.Theme = "light"
|
||||
if err := Save(staleOff); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := Load(); !got.GatewayConversations || got.Theme != "light" {
|
||||
t.Fatal("stale settings save removed consent or lost its theme change")
|
||||
}
|
||||
if err := SetGatewayConversations(false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := Load(); got.GatewayConversations || got.Theme != "light" {
|
||||
t.Fatal("explicit consent change overwrote other settings")
|
||||
}
|
||||
}
|
||||
@@ -103,6 +103,9 @@ type Settings struct {
|
||||
// RequestArchiveMaxMB is how much of each body the archive keeps, in
|
||||
// MiB: 0 for 32, at most 1024 (#447)
|
||||
RequestArchiveMaxMB int `json:"requestArchiveMaxMB,omitempty"`
|
||||
// GatewayConversations is local, opt-in recording of session traffic.
|
||||
// Only SetGatewayConversations changes it; other saves keep local consent.
|
||||
GatewayConversations bool `json:"gatewayConversations,omitempty"`
|
||||
// CodexWarmup starts a ChatGPT account's next window as soon as the
|
||||
// last one resets, with one tiny request, so it counts from then (a
|
||||
// Codex window starts at its first use): "" off, "week" the weekly
|
||||
@@ -747,6 +750,7 @@ func (s Settings) Compact() int {
|
||||
// icon (yoooo on Discord: usage turned off on a Mac came back from a
|
||||
// Windows box that shows it).
|
||||
func (s *Settings) KeepOwn(cur Settings) {
|
||||
s.GatewayConversations = cur.GatewayConversations
|
||||
s.UIFont, s.CodeFont = cur.UIFont, cur.CodeFont
|
||||
s.Window, s.Proxy, s.Port, s.Dock, s.DockWindow, s.Lightweight = cur.Window, cur.Proxy, cur.Port, cur.Dock, cur.DockWindow, cur.Lightweight
|
||||
s.WindowMaximised, s.KeepAwake, s.KeepAwakeDisplay = cur.WindowMaximised, cur.KeepAwake, cur.KeepAwakeDisplay
|
||||
@@ -811,6 +815,11 @@ var fileMu sync.RWMutex
|
||||
func Load() Settings {
|
||||
fileMu.RLock()
|
||||
defer fileMu.RUnlock()
|
||||
return load()
|
||||
}
|
||||
|
||||
// load is called with fileMu held for reading or writing.
|
||||
func load() Settings {
|
||||
var s Settings
|
||||
// read again only once the file changed: a look at the agents asks for
|
||||
// the settings for every model of every agent (hundreds of reads, a
|
||||
@@ -876,10 +885,25 @@ func SavedAddr() string {
|
||||
return fmt.Sprintf("127.0.0.1:%d", p)
|
||||
}
|
||||
|
||||
// Save validates and writes the settings.
|
||||
// Save validates and writes the settings, preserving current recording consent.
|
||||
func Save(s Settings) error {
|
||||
fileMu.Lock()
|
||||
defer fileMu.Unlock()
|
||||
return save(s, false)
|
||||
}
|
||||
|
||||
// SetGatewayConversations changes local recording consent without overwriting
|
||||
// other settings from an earlier snapshot.
|
||||
func SetGatewayConversations(on bool) error {
|
||||
fileMu.Lock()
|
||||
defer fileMu.Unlock()
|
||||
s := load()
|
||||
s.GatewayConversations = on
|
||||
return save(s, true)
|
||||
}
|
||||
|
||||
// save is called under fileMu. Only the consent setter may replace recording.
|
||||
func save(s Settings, recording bool) error {
|
||||
defer filememo.Forget() // read again, where a request holds it
|
||||
s = s.normal()
|
||||
if !slices.Contains(Themes, s.Theme) {
|
||||
@@ -1002,6 +1026,7 @@ func Save(s Settings) error {
|
||||
}
|
||||
// Load may have returned defaults or only part of an unreadable file.
|
||||
// Do not replace it, including its permissions, with those values.
|
||||
consent := false
|
||||
if b, err := steady.ReadFile(Path()); err == nil {
|
||||
b = bytes.TrimPrefix(b, []byte("\xef\xbb\xbf"))
|
||||
if len(bytes.TrimSpace(b)) != 0 {
|
||||
@@ -1009,10 +1034,14 @@ func Save(s Settings) error {
|
||||
if err := json.Unmarshal(b, &stored); err != nil {
|
||||
return fmt.Errorf("could not read settings at %s; repair or move that file aside before saving: %w", Path(), err)
|
||||
}
|
||||
consent = stored.GatewayConversations
|
||||
}
|
||||
} else if !os.IsNotExist(err) {
|
||||
return fmt.Errorf("could not read settings at %s: %w", Path(), err)
|
||||
}
|
||||
if !recording {
|
||||
s.GatewayConversations = consent
|
||||
}
|
||||
if err := os.MkdirAll(Dir(), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ package usage
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -49,7 +50,7 @@ func direct(p Period, now time.Time, rows []Row) Summary {
|
||||
if who := r.Account(); who != "" {
|
||||
put(accounts, r.Provider+"@"+who, Group{ID: r.Provider + "@" + who, Provider: r.Provider, Account: who}, r)
|
||||
}
|
||||
if r.Session != "" {
|
||||
if r.Session != "" && !strings.HasPrefix(r.Session, "request-") {
|
||||
put(sessions, id+"|"+r.Session, Group{ID: r.Session, Agent: id}, r)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,309 @@
|
||||
package usage
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/yetone/magpie/internal/catalog"
|
||||
"github.com/yetone/magpie/internal/sessions"
|
||||
)
|
||||
|
||||
// GatewaySessionWindow returns gateway sessions and their daily usage for a
|
||||
// local-date range. It is kept separate from the GUI conversion so callers
|
||||
// can use the same single usage-log snapshot for list and stats.
|
||||
func GatewaySessionWindow(since time.Time) ([]GatewaySession, []GatewayDayUsage) {
|
||||
return gatewaySessionSnapshot(since, nil)
|
||||
}
|
||||
|
||||
func GatewaySessionWindowExcept(since time.Time, nativeKeys map[string]bool) ([]GatewaySession, []GatewayDayUsage) {
|
||||
return gatewaySessionSnapshot(since, nativeKeys)
|
||||
}
|
||||
|
||||
type GatewayDayUsage struct {
|
||||
Date, Agent, Model string
|
||||
sessions.Tokens
|
||||
Cost float64
|
||||
Priced bool
|
||||
}
|
||||
|
||||
// GatewaySession is a read-only projection of requests served by this
|
||||
// gateway. Its identity is the agent name and the session id supplied by the
|
||||
// client. Unidentified requests and generated recording identities are left out.
|
||||
type GatewaySession struct {
|
||||
Agent string `json:"agent"`
|
||||
ID string `json:"id"`
|
||||
Start time.Time `json:"start"`
|
||||
Last time.Time `json:"last"`
|
||||
Models []GatewayModel `json:"models"`
|
||||
Daily []GatewayDayUsage `json:"-"`
|
||||
sessions.Tokens
|
||||
Cost float64 `json:"cost"`
|
||||
Unpriced int `json:"unpriced"`
|
||||
}
|
||||
|
||||
type GatewayModel struct {
|
||||
Model string `json:"model"`
|
||||
sessions.Tokens
|
||||
Cost float64 `json:"cost"`
|
||||
Priced bool `json:"priced"`
|
||||
}
|
||||
|
||||
// GatewaySessions aggregates one usage snapshot without reading routing
|
||||
// history or agent files. NativeKeys contains identities already represented
|
||||
// by local session files and is used to suppress the duplicate projection.
|
||||
func GatewaySessions(since time.Time, nativeKeys map[string]bool) []GatewaySession {
|
||||
e := gatewayProjection(since)
|
||||
out := make([]GatewaySession, 0, len(e.sessions))
|
||||
for _, s := range e.sessions {
|
||||
if !nativeKeys[s.Agent+"|"+s.ID] {
|
||||
out = append(out, cloneGatewaySession(s))
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func GatewaySessionByID(agent, id string, nativeKeys map[string]bool) (GatewaySession, bool) {
|
||||
if id == "" || strings.HasPrefix(id, "request-") {
|
||||
return GatewaySession{}, false
|
||||
}
|
||||
agent = AgentOf(agent)
|
||||
if nativeKeys[agent+"|"+id] {
|
||||
return GatewaySession{}, false
|
||||
}
|
||||
entry := gatewayProjection(time.Time{})
|
||||
i, ok := entry.byID[agent+"|"+id]
|
||||
if !ok {
|
||||
return GatewaySession{}, false
|
||||
}
|
||||
return cloneGatewaySession(entry.sessions[i]), true
|
||||
}
|
||||
|
||||
type gatewayProjectionEntry struct {
|
||||
version uint64
|
||||
meta string
|
||||
since time.Time
|
||||
sessions []GatewaySession
|
||||
byID map[string]int
|
||||
}
|
||||
|
||||
var gatewayCache struct {
|
||||
sync.Mutex
|
||||
entries []gatewayProjectionEntry
|
||||
}
|
||||
|
||||
// Retain a few window projections, not one copy for every native exclusion set.
|
||||
// The log snapshot version also detects rewrites and changes of config directory.
|
||||
func gatewayProjection(since time.Time) gatewayProjectionEntry {
|
||||
snapshot := logSnapshotFor(true)
|
||||
meta := usagePriceCacheKey(time.Now())
|
||||
gatewayCache.Lock()
|
||||
defer gatewayCache.Unlock()
|
||||
for _, e := range gatewayCache.entries {
|
||||
if e.version == snapshot.version && e.meta == meta && e.since.Equal(since) {
|
||||
return e
|
||||
}
|
||||
}
|
||||
if snapshot.uncached && len(snapshot.blocks) == 0 {
|
||||
snapshot = readLogSnapshot()
|
||||
}
|
||||
ss, _ := buildGatewaySessions(snapshot, since, nil)
|
||||
e := gatewayProjectionEntry{version: snapshot.version, meta: meta, since: since, sessions: ss, byID: make(map[string]int, len(ss))}
|
||||
for i, s := range ss {
|
||||
e.byID[s.Agent+"|"+s.ID] = i
|
||||
}
|
||||
// A failed or partial read must be retried, not published as current.
|
||||
if snapshot.info != nil && snapshot.off == snapshot.info.Size() {
|
||||
gatewayCache.entries = slices.DeleteFunc(gatewayCache.entries, func(old gatewayProjectionEntry) bool {
|
||||
return old.version != e.version || old.meta != e.meta || old.since.Equal(since)
|
||||
})
|
||||
if len(gatewayCache.entries) >= 4 {
|
||||
gatewayCache.entries = gatewayCache.entries[1:]
|
||||
}
|
||||
gatewayCache.entries = append(gatewayCache.entries, e)
|
||||
}
|
||||
return e
|
||||
}
|
||||
|
||||
func cloneGatewaySession(s GatewaySession) GatewaySession {
|
||||
s.Models, s.Daily = slices.Clone(s.Models), slices.Clone(s.Daily)
|
||||
return s
|
||||
}
|
||||
|
||||
func gatewaySessionSnapshot(since time.Time, nativeKeys map[string]bool) ([]GatewaySession, []GatewayDayUsage) {
|
||||
out := GatewaySessions(since, nativeKeys)
|
||||
return out, gatewayDays(out)
|
||||
}
|
||||
|
||||
func gatewayDays(ss []GatewaySession) []GatewayDayUsage {
|
||||
type key struct{ date, agent, model string }
|
||||
groups := map[key]GatewayDayUsage{}
|
||||
for _, s := range ss {
|
||||
for _, d := range s.Daily {
|
||||
k := key{d.Date, d.Agent, d.Model}
|
||||
v, ok := groups[k]
|
||||
if !ok {
|
||||
v.Date, v.Agent, v.Model, v.Priced = d.Date, d.Agent, d.Model, true
|
||||
}
|
||||
v.Input += d.Input
|
||||
v.Output += d.Output
|
||||
v.CacheRead += d.CacheRead
|
||||
v.CacheWrite += d.CacheWrite
|
||||
v.CacheWrite1h += d.CacheWrite1h
|
||||
v.Cost += d.Cost
|
||||
v.Priced = v.Priced && d.Priced
|
||||
groups[k] = v
|
||||
}
|
||||
}
|
||||
out := make([]GatewayDayUsage, 0, len(groups))
|
||||
for _, d := range groups {
|
||||
out = append(out, d)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
a, b := out[i], out[j]
|
||||
if a.Date != b.Date {
|
||||
return a.Date < b.Date
|
||||
}
|
||||
if a.Agent != b.Agent {
|
||||
return a.Agent < b.Agent
|
||||
}
|
||||
return a.Model < b.Model
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
func buildGatewaySessions(snapshot *logSnapshot, since time.Time, nativeKeys map[string]bool) ([]GatewaySession, []GatewayDayUsage) {
|
||||
priceOf := pricer()
|
||||
type key struct{ agent, id string }
|
||||
groups := map[key]*GatewaySession{}
|
||||
models := map[key]map[string]*GatewayModel{}
|
||||
type dayKey struct{ date, agent, model string }
|
||||
daily := map[dayKey]*GatewayDayUsage{}
|
||||
perSession := map[key]map[dayKey]*GatewayDayUsage{}
|
||||
snapshot.visit(since, func(r Record) {
|
||||
if r.IsRejected() || r.Session == "" || strings.HasPrefix(r.Session, "request-") {
|
||||
return
|
||||
}
|
||||
a := AgentOf(r.Agent)
|
||||
k := key{a, r.Session}
|
||||
if nativeKeys != nil && nativeKeys[a+"|"+r.Session] {
|
||||
return
|
||||
}
|
||||
s := groups[k]
|
||||
if s == nil {
|
||||
s = &GatewaySession{Agent: a, ID: r.Session, Models: []GatewayModel{}}
|
||||
groups[k] = s
|
||||
models[k] = map[string]*GatewayModel{}
|
||||
perSession[k] = map[dayKey]*GatewayDayUsage{}
|
||||
}
|
||||
if s.Start.IsZero() || r.Time.Before(s.Start) {
|
||||
s.Start = r.Time
|
||||
}
|
||||
if end := r.Time.Add(time.Duration(max(0, r.Millis)) * time.Millisecond); end.After(s.Last) {
|
||||
s.Last = end
|
||||
}
|
||||
p := priceOf(r)
|
||||
cost, priced := 0.0, true
|
||||
if r.Input+r.Output > 0 {
|
||||
if p == nil {
|
||||
priced = false
|
||||
} else {
|
||||
cost = r.CostAt(*p)
|
||||
}
|
||||
}
|
||||
tokens := sessions.Tokens{Input: r.Input, Output: r.Output, CacheRead: r.CacheRead, CacheWrite: r.CacheWrite, CacheWrite1h: r.CacheWrite1h}
|
||||
dk := dayKey{r.Time.In(time.Local).Format(time.DateOnly), a, r.Model}
|
||||
for _, kept := range []map[dayKey]*GatewayDayUsage{daily, perSession[k]} {
|
||||
d := kept[dk]
|
||||
if d == nil {
|
||||
d = &GatewayDayUsage{Date: dk.date, Agent: a, Model: r.Model, Priced: true}
|
||||
kept[dk] = d
|
||||
}
|
||||
d.Input += tokens.Input
|
||||
d.Output += tokens.Output
|
||||
d.CacheRead += tokens.CacheRead
|
||||
d.CacheWrite += tokens.CacheWrite
|
||||
d.CacheWrite1h += tokens.CacheWrite1h
|
||||
d.Cost += cost
|
||||
d.Priced = d.Priced && priced
|
||||
}
|
||||
s.Input += r.Input
|
||||
s.Output += r.Output
|
||||
s.CacheRead += r.CacheRead
|
||||
s.CacheWrite += r.CacheWrite
|
||||
s.CacheWrite1h += r.CacheWrite1h
|
||||
m := models[k][r.Model]
|
||||
if m == nil {
|
||||
m = &GatewayModel{Model: r.Model, Priced: true}
|
||||
models[k][r.Model] = m
|
||||
}
|
||||
m.addRecord(r, p)
|
||||
})
|
||||
for k, s := range groups {
|
||||
for _, d := range perSession[k] {
|
||||
s.Daily = append(s.Daily, *d)
|
||||
}
|
||||
sort.Slice(s.Daily, func(i, j int) bool {
|
||||
if s.Daily[i].Date != s.Daily[j].Date {
|
||||
return s.Daily[i].Date < s.Daily[j].Date
|
||||
}
|
||||
return s.Daily[i].Model < s.Daily[j].Model
|
||||
})
|
||||
for _, m := range models[k] {
|
||||
s.Models = append(s.Models, *m)
|
||||
s.Cost += m.Cost
|
||||
if !m.Priced && m.Input+m.Output > 0 {
|
||||
s.Unpriced++
|
||||
}
|
||||
}
|
||||
sort.Slice(s.Models, func(i, j int) bool {
|
||||
a, b := s.Models[i], s.Models[j]
|
||||
if a.Input+a.Output != b.Input+b.Output {
|
||||
return a.Input+a.Output > b.Input+b.Output
|
||||
}
|
||||
return a.Model < b.Model
|
||||
})
|
||||
}
|
||||
out := make([]GatewaySession, 0, len(groups))
|
||||
for _, s := range groups {
|
||||
out = append(out, *s)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if !out[i].Last.Equal(out[j].Last) {
|
||||
return out[i].Last.After(out[j].Last)
|
||||
}
|
||||
return out[i].Agent+"|"+out[i].ID < out[j].Agent+"|"+out[j].ID
|
||||
})
|
||||
days := make([]GatewayDayUsage, 0, len(daily))
|
||||
for _, d := range daily {
|
||||
days = append(days, *d)
|
||||
}
|
||||
sort.Slice(days, func(i, j int) bool {
|
||||
if days[i].Date != days[j].Date {
|
||||
return days[i].Date < days[j].Date
|
||||
}
|
||||
if days[i].Agent != days[j].Agent {
|
||||
return days[i].Agent < days[j].Agent
|
||||
}
|
||||
return days[i].Model < days[j].Model
|
||||
})
|
||||
return out, days
|
||||
}
|
||||
|
||||
func (m *GatewayModel) addRecord(r Record, p *catalog.Price) {
|
||||
m.Input += r.Input
|
||||
m.Output += r.Output
|
||||
m.CacheRead += r.CacheRead
|
||||
m.CacheWrite += r.CacheWrite
|
||||
m.CacheWrite1h += r.CacheWrite1h
|
||||
if r.Input+r.Output == 0 {
|
||||
return
|
||||
}
|
||||
if p == nil {
|
||||
m.Priced = false
|
||||
return
|
||||
}
|
||||
m.Cost += r.CostAt(*p)
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
package usage
|
||||
|
||||
import (
|
||||
"github.com/yetone/magpie/internal/settings"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/yetone/magpie/internal/provider"
|
||||
)
|
||||
|
||||
func resetUsageTest(t *testing.T) {
|
||||
t.Helper()
|
||||
h := t.TempDir()
|
||||
t.Setenv("HOME", h)
|
||||
t.Setenv("USERPROFILE", h)
|
||||
t.Setenv("XDG_CONFIG_HOME", filepath.Join(h, ".config"))
|
||||
_ = provider.Path()
|
||||
}
|
||||
|
||||
func TestGatewaySessionsRequiresExplicitIdentityAndKeepsModels(t *testing.T) {
|
||||
resetUsageTest(t)
|
||||
at := time.Date(2026, 10, 7, 12, 0, 0, 0, time.Local)
|
||||
Append(Record{Time: at, Agent: "codex", Session: "s-1", Provider: "p", Model: "m1", Input: 10, Output: 2, Status: 200})
|
||||
Append(Record{Time: at.Add(time.Minute), Agent: "codex", Session: "s-1", Provider: "p", Model: "m2", Input: 3, Output: 4, Status: 200})
|
||||
Append(Record{Time: at, Agent: "codex", Provider: "p", Model: "m1", Input: 999, Output: 999, Status: 200})
|
||||
Append(Record{Time: at, Agent: "codex", Session: "request-old-generated", Provider: "p", Model: "m1", Input: 999, Status: 200})
|
||||
got := GatewaySessions(time.Time{}, nil)
|
||||
if len(got) != 1 || got[0].ID != "s-1" || len(got[0].Models) != 2 || got[0].Input != 13 || got[0].Output != 6 {
|
||||
t.Fatalf("sessions=%+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewaySessionsCacheRefreshAndOwnership(t *testing.T) {
|
||||
resetUsageTest(t)
|
||||
at := time.Now()
|
||||
Append(Record{Time: at, Agent: "pi", Session: "native", Provider: "p", Model: "m", Input: 10, Status: 200})
|
||||
first := GatewaySessions(time.Time{}, nil)
|
||||
if len(first) != 1 {
|
||||
t.Fatalf("first: %+v", first)
|
||||
}
|
||||
first[0].Models[0].Model = "changed by caller"
|
||||
first[0].Daily[0].Input = 999
|
||||
again := GatewaySessions(time.Time{}, nil)
|
||||
if again[0].Models[0].Model != "m" || again[0].Daily[0].Input != 10 {
|
||||
t.Fatalf("caller changed cache: %+v", again)
|
||||
}
|
||||
if got := GatewaySessions(time.Time{}, map[string]bool{"pi|native": true}); len(got) != 0 {
|
||||
t.Fatalf("native exclusion lost: %+v", got)
|
||||
}
|
||||
Append(Record{Time: at.Add(time.Second), Agent: "pi", Session: "native", Provider: "p", Model: "m", Input: 5, Status: 200})
|
||||
if got, ok := GatewaySessionByID("pi", "native", nil); !ok || got.Input != 15 {
|
||||
t.Fatalf("append not reflected: %+v %v", got, ok)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGeneratedSessionsExcludedFromUsage(t *testing.T) {
|
||||
resetUsageTest(t)
|
||||
Append(Record{Time: Clock(), Agent: "pi", Session: "request-old", Provider: "p", Model: "m", Input: 10, Status: 200})
|
||||
summary := Summarize(All)
|
||||
if len(summary.Sessions) != 0 || summary.Totals.Input != 10 {
|
||||
t.Fatalf("generated session polluted usage: %+v", summary)
|
||||
}
|
||||
if got := Vias(time.Time{}); len(got) != 0 {
|
||||
t.Fatalf("generated session polluted routing: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewaySessionsNativeIdentityWins(t *testing.T) {
|
||||
resetUsageTest(t)
|
||||
Append(Record{Time: time.Now(), Agent: "pi", Session: "native", Provider: "p", Model: "m", Input: 1, Status: 200})
|
||||
if got := GatewaySessions(time.Time{}, map[string]bool{"pi|native": true}); len(got) != 0 {
|
||||
t.Fatalf("got duplicate gateway session: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatewaySessionsCachePricesWindowsAndRewrite(t *testing.T) {
|
||||
resetUsageTest(t)
|
||||
at := time.Now()
|
||||
Append(Record{Time: at.Add(-time.Hour), Agent: "pi", Session: "native", Provider: "p", Model: "m", Input: 1000000, Status: 200})
|
||||
Append(Record{Time: at, Agent: "pi", Session: "native", Provider: "p", Model: "m", Input: 1000000, Status: 200})
|
||||
for _, price := range []float64{2, 4} {
|
||||
cfg := settings.Load()
|
||||
cfg.ModelPrices = map[string]settings.ModelPrice{"*/m": {Input: &price, Output: new(0.0), CacheRead: new(0.0), CacheWrite: new(0.0)}}
|
||||
if err := settings.Save(cfg); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := range 2 {
|
||||
all := GatewaySessions(time.Time{}, nil)
|
||||
window, days := GatewaySessionWindow(at.Add(-time.Minute))
|
||||
if len(all) != 1 || all[0].Cost != 2*price || len(window) != 1 || window[0].Cost != price || len(days) != 1 || days[0].Input != 1000000 {
|
||||
t.Fatalf("price=%v read=%d all=%+v window=%+v days=%+v", price, i, all, window, days)
|
||||
}
|
||||
}
|
||||
}
|
||||
vias := Vias(time.Time{})
|
||||
vias["pi|native"][0].Calls = 999
|
||||
if got := Vias(time.Time{})["pi|native"][0].Calls; got != 2 {
|
||||
t.Fatalf("caller mutated routing cache: %d", got)
|
||||
}
|
||||
if got := Vias(at.Add(-time.Minute))["pi|native"][0].Calls; got != 1 {
|
||||
t.Fatalf("routing window: %d", got)
|
||||
}
|
||||
if err := os.WriteFile(Path(), nil, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := GatewaySessions(time.Time{}, nil); len(got) != 0 {
|
||||
t.Fatalf("rewrite retained sessions: %+v", got)
|
||||
}
|
||||
if got := Vias(time.Time{}); len(got) != 0 {
|
||||
t.Fatalf("rewrite retained routing: %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -31,19 +31,7 @@ func indexedSummary(p Period, now time.Time) Summary {
|
||||
p = All
|
||||
}
|
||||
snapshot := logSnapshotFor(true)
|
||||
_, offset := now.Zone()
|
||||
meta := fmt.Sprintf("%s|%s|%s|%s|%s|%s|%d", statKey(settings.Path()), statKey(provider.Path()), statKey(catalog.CachePath()), statKey(catalog.LivePath("antigravity")), now.Format("2006-01-02"), now.Location(), offset)
|
||||
// Sign-ins and plugin lists can change independently of providers.json.
|
||||
// Their identity and price feeds affect both grouping and historical costs.
|
||||
h := sha256.New()
|
||||
prices, _ := json.Marshal(settings.Load().ModelPrices)
|
||||
aliases, _ := json.Marshal(provider.Renamed())
|
||||
fmt.Fprint(h, string(prices), string(aliases))
|
||||
for _, current := range provider.All() {
|
||||
fmt.Fprintf(h, "%q:%q:%v:%t:%t:%t;", current.ID, current.Where(), current.Catalogs(), current.IsPlugin(), current.Account == nil, current.Key != "")
|
||||
fmt.Fprint(h, statKey(catalog.LivePath(current.ID)))
|
||||
}
|
||||
meta += fmt.Sprintf("|%x", h.Sum(nil))
|
||||
meta := usagePriceCacheKey(now)
|
||||
summaries.Lock()
|
||||
old, ok := summaries.entries[p]
|
||||
summaries.Unlock()
|
||||
@@ -85,3 +73,21 @@ func cloneSummary(s Summary) Summary {
|
||||
s.Sessions = slices.Clone(s.Sessions)
|
||||
return s
|
||||
}
|
||||
|
||||
// usagePriceCacheKey invalidates priced views when local prices or provider feeds change.
|
||||
func usagePriceCacheKey(now time.Time) string {
|
||||
_, offset := now.Zone()
|
||||
meta := fmt.Sprintf("%s|%s|%s|%s|%s|%s|%d", statKey(settings.Path()), statKey(provider.Path()), statKey(catalog.CachePath()), statKey(catalog.LivePath("antigravity")), now.Format("2006-01-02"), now.Location(), offset)
|
||||
// Sign-ins and plugin lists can change independently of providers.json.
|
||||
// Their identity and price feeds affect both grouping and historical costs.
|
||||
h := sha256.New()
|
||||
prices, _ := json.Marshal(settings.Load().ModelPrices)
|
||||
aliases, _ := json.Marshal(provider.Renamed())
|
||||
fmt.Fprint(h, string(prices), string(aliases))
|
||||
for _, current := range provider.All() {
|
||||
fmt.Fprintf(h, "%q:%q:%v:%t:%t:%t;", current.ID, current.Where(), current.Catalogs(), current.IsPlugin(), current.Account == nil, current.Key != "")
|
||||
fmt.Fprint(h, statKey(catalog.LivePath(current.ID)))
|
||||
}
|
||||
meta += fmt.Sprintf("|%x", h.Sum(nil))
|
||||
return meta
|
||||
}
|
||||
@@ -109,7 +109,8 @@ type Record struct {
|
||||
Stop string `json:"stop,omitempty"`
|
||||
// Session is the conversation the call was part of, as its agent names
|
||||
// it (X-Magpie-Session, or the session header Claude Code, Codex or
|
||||
// OpenCode sends): several sessions on one model told apart
|
||||
// OpenCode sends), or a gateway-generated identity for an unidentified
|
||||
// conversation request: several sessions on one model told apart.
|
||||
Session string `json:"session,omitempty"`
|
||||
// NativeSession retains the client header when X-Magpie-Session overrides it.
|
||||
NativeSession string `json:"native_session,omitempty"`
|
||||
@@ -679,7 +680,7 @@ func summarizeFrom(p Period, now time.Time, first time.Time, historicalKeys map[
|
||||
}
|
||||
g.add(r, pr)
|
||||
}
|
||||
if r.Session != "" {
|
||||
if r.Session != "" && !strings.HasPrefix(r.Session, "request-") {
|
||||
g := sessions[id+"|"+r.Session]
|
||||
if g == nil {
|
||||
g = &Group{ID: r.Session, Agent: id}
|
||||
@@ -746,9 +747,13 @@ type Via struct {
|
||||
// Vias is what the gateway sent each session's calls to since a time, by
|
||||
// agent id and the session's id ("codex|<id>"), the most calls first.
|
||||
func Vias(since time.Time) map[string][]Via {
|
||||
return cachedVias(since)
|
||||
}
|
||||
|
||||
func buildVias(snapshot *logSnapshot, since time.Time) map[string][]Via {
|
||||
out := map[string][]Via{}
|
||||
readLogSnapshot().visit(since, func(r Record) {
|
||||
if r.IsRejected() || r.Session == "" || r.Model == "" {
|
||||
snapshot.visit(since, func(r Record) {
|
||||
if r.IsRejected() || r.Session == "" || strings.HasPrefix(r.Session, "request-") || r.Model == "" {
|
||||
return
|
||||
}
|
||||
k := AgentOf(r.Agent) + "|" + r.Session
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
package usage
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
var viasCache struct {
|
||||
sync.Mutex
|
||||
version uint64
|
||||
since time.Time
|
||||
value map[string][]Via
|
||||
}
|
||||
|
||||
func cachedVias(since time.Time) map[string][]Via {
|
||||
snapshot := logSnapshotFor(true)
|
||||
viasCache.Lock()
|
||||
defer viasCache.Unlock()
|
||||
value := viasCache.value
|
||||
if value == nil || viasCache.version != snapshot.version || !viasCache.since.Equal(since) {
|
||||
if snapshot.uncached && len(snapshot.blocks) == 0 {
|
||||
snapshot = readLogSnapshot()
|
||||
}
|
||||
value = buildVias(snapshot, since)
|
||||
if snapshot.info != nil && snapshot.off == snapshot.info.Size() {
|
||||
viasCache.version, viasCache.since, viasCache.value = snapshot.version, since, value
|
||||
}
|
||||
}
|
||||
out := make(map[string][]Via, len(value))
|
||||
for k, vs := range value {
|
||||
out[k] = slices.Clone(vs)
|
||||
}
|
||||
return out
|
||||
}
|
||||
Reference in new issue
Block a user