diff --git a/README.md b/README.md index 063c8370..44c46612 100644 --- a/README.md +++ b/README.md @@ -156,6 +156,54 @@ Its list is the models the shared magpie's agents are shown, each named with its provider there (`Claude Sonnet 5 · Relay A · office`), and its image models are listed under Settings → Images and draw through it. +The gateway issues **gateway keys** for clients, separate from a provider's +upstream API keys. Turn on **Settings → Share on local network**, then open +**Gateway → Gateway keys → Add gateway key**. This block appears only while +sharing is on. Create a named key for each client and copy it from its row. +Rename, disable, rotate or remove keys independently; rotation and removal +ask for confirmation. Rotation keeps the name, enabled state and usage +history; other keys are unchanged. While sharing is on, **Gateway → Connect** +offers loopback and shared addresses, plus enabled gateway keys, for all +connection examples. With sharing off, Connect keeps the original **API key** +field and local `magpie` token, without a gateway-key picker. Its arbitrary +local option is **This computer**, distinct from the named **Magpie** key. + +For a headless gateway, use the CLI before exposing the port: + +```sh +magpie gateway-key add "Remote laptop" # prints the new credential once +magpie gateway-key list # ids, names, enabled state and masked keys +magpie gateway-key rotate # prints the replacement; identity stays the same +magpie gateway-key remove # revokes remote access +``` + +While LAN sharing is enabled, remote requests require an enabled gateway key +sent as Bearer, `x-api-key`, `x-goog-api-key` or `?key=`. Loopback remains +permissive: any token works, including a stale or disabled gateway key. +Only a valid, enabled key is attributed to its named identity. +Without sharing, an explicitly exposed `MAGPIE_ADDR` keeps its original open +access, including old `sk-magpie-…` tokens, without key authentication. + +**Usage → Overview → Gateway keys** groups calls by the client's key, never +the provider's credential. **Usage → Requests** offers the same filter; +CSV includes `caller_key_id` and `caller_key_name`. Deleted keys keep their +history. The usual local `magpie` token and older records stay unattributed. +Caller attribution includes chat, images, video creation and System One. +Provider attribution stays independent; CSV puts `provider_key_*` before +`caller_key_*`. +An existing LAN key becomes **Magpie** without changing the credential. +`lanKey` remains in settings for older Magpie versions. Disabling or removing +the default key replaces that mirror with a random non-empty revoked value; +rotation does not re-enable it. The key store records migration completion, +even if `lanKeyId` cannot be saved, so reads do not keep retrying that write. +A migration write failure is logged without preventing gateway startup, +CLI key management, or the Settings and key-list pages from opening. +If settings are read-only, changing the default key fails without changing +it: make `settings.json` writable and retry so older versions cannot keep +accepting its old credential. Independent named keys remain manageable. +Gateway credentials stay in `~/.config/magpie/caller-keys.json` (XDG-aware, +mode `0600`), never in usage records or list responses. + Baidu Qianfan's [Token Plans](https://cloud.baidu.com/doc/qianfan/s/Dmrabu8b6) are available as `baidu-qianfan`: a personal (个人版) and an enterprise (企业版) plan and pay as you go, each with its own Chat Completions, Responses and @@ -631,9 +679,12 @@ docker run -d --name magpie -p 127.0.0.1:3425:3425 -p 127.0.0.1:3430:3430 -v mag published on the host's loopback only; Docker's `-p 3425:3425` would put it on every interface of the host, past its firewall. To reach it from other machines, turn on Settings → Share on local network in the browser UI (or -put `"lan": true, "lanKey": "sk-magpie-…"` in `/config/magpie/settings.json`): -from then on a request from outside the container must carry that key as its -API key, and only then publish the port beyond 127.0.0.1. Inside the container +put `"lan": true` in `/config/magpie/settings.json`). Turning it on in Settings +creates a named **Magpie** key. With settings edited by hand, run +`magpie gateway-key add "Docker client"` in the container to create a key +without the browser UI. A request from outside the container must carry one of +those keys as its API key. Only then publish the port beyond 127.0.0.1. +Inside the container magpie only sees the container's own address (Docker's 172.17.x), so set `-e MAGPIE_PUBLIC_URL=http://:3425` (the port published on the host) for the address it shows and prints to be the one @@ -755,6 +806,10 @@ pictures picked for them, the settings, the profiles, every agent's model and the library (unless `--no-library`): the instruction sets, the MCP servers and the skills with their files (a file over 2 MB is left out). Without keys, a server's environment variables and headers that look like a key go empty. +Gateway credentials, their names, ids and disabled state travel encrypted +with Settings too. Restoring Settings replaces the gateway-key store with +the backed-up one. `--no-keys` leaves gateway credentials and their legacy +mirror out; restoring it preserves the destination's existing keys instead. Restoring the library replaces the one there — what it replaces is kept with the library's backups — and writes it into the agents on that machine. It is encrypted on your machine (AES-256-GCM, the key derived from the diff --git a/gateway_keys_cli.go b/gateway_keys_cli.go new file mode 100644 index 00000000..54bc2df2 --- /dev/null +++ b/gateway_keys_cli.go @@ -0,0 +1,65 @@ +package main + +import ( + "fmt" + "io" + "os" + "strings" + "text/tabwriter" + + "github.com/yetone/magpie/internal/access" +) + +func gatewayKeys(args []string) error { return gatewayKeysTo(os.Stdout, args) } + +func gatewayKeysTo(out io.Writer, args []string) error { + args = args[1:] + if len(args) == 0 { + args = []string{"list"} + } + action := args[0] + if (action == "list" && len(args) != 1) || (action != "list" && len(args) != 2) { + return fmt.Errorf("usage: magpie gateway-key list | add | rotate | remove ") + } + switch action { + case "list", "add", "rotate", "remove": + default: + return fmt.Errorf("unknown gateway-key command %q", action) + } + access.MigrateLegacyLANKeyBestEffort() + if action == "list" { + keys, err := access.List() + if err != nil { + return err + } + w := tabwriter.NewWriter(out, 0, 4, 2, ' ', 0) + fmt.Fprintln(w, "ID\tNAME\tSTATE\tGATEWAY KEY") + for _, k := range keys { + state := "enabled" + if k.Off { + state = "disabled" + } + fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", k.ID, strings.Map(func(r rune) rune { + if r < 32 || r == 127 { + return ' ' + } + return r + }, k.Name), state, k.Masked) + } + return w.Flush() + } + in := access.Change{Key: args[1]} + if action == "add" { + in = access.Change{Name: args[1]} + } + secret, err := access.Update(action+"-key", in) + if err != nil { + return err + } + if secret != "" { + _, err = fmt.Fprintln(out, secret) + } else { + _, err = fmt.Fprintln(out, "Gateway key removed") + } + return err +} diff --git a/gateway_keys_cli_test.go b/gateway_keys_cli_test.go new file mode 100644 index 00000000..4e8e8b2b --- /dev/null +++ b/gateway_keys_cli_test.go @@ -0,0 +1,80 @@ +package main + +import ( + "bytes" + "os" + "strings" + "testing" + + "github.com/yetone/magpie/internal/access" + "github.com/yetone/magpie/internal/settings" +) + +func TestGatewayKeyCLI(t *testing.T) { + t.Setenv("HOME", t.TempDir()) + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + if err := settings.Save(settings.Settings{LAN: true}); err != nil { + t.Fatal(err) + } + call := func(args ...string) string { + t.Helper() + var out bytes.Buffer + if err := gatewayKeysTo(&out, append([]string{"gateway-key"}, args...)); err != nil { + t.Fatal(err) + } + return strings.TrimSpace(out.String()) + } + if got := call("list"); !strings.Contains(got, "GATEWAY KEY") { + t.Fatal(got) + } + secret := call("add", "Remote laptop") + keys, _ := access.List() + if len(keys) != 1 { + t.Fatal(keys) + } + id := keys[0].ID + if who, ok := access.Authenticate(secret); !ok || who.KeyID != id || who.KeyName != "Remote laptop" { + t.Fatal(who, ok) + } + if got := call(); strings.Contains(got, secret) || !strings.Contains(got, id) || !strings.Contains(got, "Remote laptop") { + t.Fatal("list must mask credentials", got) + } + next := call("rotate", id) + if next == secret { + t.Fatal("rotation reused the secret") + } + if _, ok := access.Authenticate(secret); ok { + t.Fatal("old credential works after rotation") + } + if who, ok := access.Authenticate(next); !ok || who.KeyID != id { + t.Fatal("rotation changed identity", who, ok) + } + call("remove", id) + if _, ok := access.Authenticate(next); ok { + t.Fatal("removed credential still works") + } + for _, args := range [][]string{{"add"}, {"add", ""}, {"list", "extra"}, {"rotate", "missing"}, {"remove", "missing"}, {"unknown", "x"}} { + if err := gatewayKeysTo(&bytes.Buffer{}, append([]string{"gateway-key"}, args...)); err == nil { + t.Error("accepted", args) + } + } +} + +func TestGatewayKeyListWithReadOnlyMigration(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + s := settings.Settings{LAN: true, LANKey: "sk-magpie-fixture-cli"} + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + if err := os.Chmod(settings.Path(), 0o400); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { os.Chmod(settings.Path(), 0o600) }) + if err := settings.Save(s); err == nil { + t.Skip("settings.json remains writable") + } + var out bytes.Buffer + if err := gatewayKeysTo(&out, []string{"gateway-key", "list"}); err != nil || !strings.Contains(out.String(), "Magpie") { + t.Fatal("read-only migration blocked CLI list", out.String(), err) + } +} diff --git a/internal/access/access.go b/internal/access/access.go new file mode 100644 index 00000000..4233d13b --- /dev/null +++ b/internal/access/access.go @@ -0,0 +1,345 @@ +// Package access keeps the named API keys issued by the gateway. +package access + +import ( + "context" + "crypto/rand" + "crypto/sha256" + "crypto/subtle" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "log" + "os" + "path/filepath" + "slices" + "strings" + "sync" + "unicode/utf8" + + "github.com/yetone/magpie/internal/edit" + "github.com/yetone/magpie/internal/settings" +) + +const Prefix = "sk-magpie-key-" +const legacyLANPrefix = "sk-magpie-" +const revokedLANPrefix = "sk-magpie-revoked-" + +func Managed(secret string) bool { + return strings.HasPrefix(secret, legacyLANPrefix) +} + +type Key struct { + ID string `json:"id"` + Name string `json:"name"` + Off bool `json:"off,omitempty"` + LAN bool `json:"lan,omitempty"` // default key and durable migration marker + Secret string `json:"secret,omitempty"` + Masked string `json:"masked,omitempty"` +} + +type Identity struct{ KeyID, KeyName string } +type contextKey struct{} + +func WithIdentity(ctx context.Context, who Identity) context.Context { + return context.WithValue(ctx, contextKey{}, who) +} +func Caller(ctx context.Context) Identity { who, _ := ctx.Value(contextKey{}).(Identity); return who } + +var mu sync.Mutex + +func Path() string { return filepath.Join(settings.Dir(), "caller-keys.json") } + +func load() ([]Key, error) { + b, err := os.ReadFile(Path()) + if errors.Is(err, os.ErrNotExist) { + return []Key{}, nil + } + if err != nil { + return nil, err + } + var keys []Key + err = json.Unmarshal(b, &keys) + return keys, err +} + +// List never returns credentials; only the administrator's copy action does. +func List() ([]Key, error) { + mu.Lock() + defer mu.Unlock() + keys, err := load() + if err != nil { + return nil, err + } + for i := range keys { + k := &keys[i] + if len(k.Secret) > 8 { + prefix := legacyLANPrefix + if strings.HasPrefix(k.Secret, Prefix) { + prefix = Prefix + } + k.Masked = prefix + "…" + k.Secret[len(k.Secret)-6:] + } + k.Secret = "" + } + return keys, nil +} + +// Export returns credentials only for the encrypted backup bundle. +func Export() ([]Key, error) { + mu.Lock() + defer mu.Unlock() + return load() +} + +// Restore replaces the gateway credentials when restoring their settings. +func Restore(keys []Key) error { + mu.Lock() + defer mu.Unlock() + return save(slices.Clone(keys)) +} + +func random(n int) (string, error) { + b := make([]byte, n) + if _, err := rand.Read(b); err != nil { + return "", err + } + return hex.EncodeToString(b), nil +} + +type Change struct { + Key string `json:"key"` + Name string `json:"name"` +} + +// Update writes the named key store atomically. +func Update(action string, in Change) (string, error) { + mu.Lock() + defer mu.Unlock() + keys, err := load() + if err != nil { + return "", err + } + name := strings.TrimSpace(in.Name) + if action == "add-key" || action == "rename-key" { + if name == "" || utf8.RuneCountInString(name) > 120 { + return "", errors.New("Use a name between 1 and 120 characters") + } + } + var secret string + var mirror *Key + if action == "add-key" { + id, err := random(12) + if err != nil { + return "", err + } + token, err := random(24) + if err != nil { + return "", err + } + secret = Prefix + token + keys = append(keys, Key{ID: id, Name: name, Secret: secret}) + } else { + i := slices.IndexFunc(keys, func(k Key) bool { return k.ID == in.Key }) + if i < 0 { + return "", errors.New("Key not found") + } + defaultKey := keys[i].LAN || keys[i].ID == settings.Load().LANKeyID + if defaultKey { + keys[i].LAN = true + } + switch action { + case "rotate-key": + token, err := random(24) + if err != nil { + return "", err + } + secret = Prefix + token + keys[i].Secret = secret + case "rename-key": + keys[i].Name = name + case "on-key", "off-key": + keys[i].Off = action == "off-key" + case "remove-key": + if defaultKey { + k := keys[i] + k.Off = true + mirror = &k + } + keys = slices.Delete(keys, i, i+1) + case "copy-key": + return keys[i].Secret, nil + default: + return "", fmt.Errorf("unknown key action %q", action) + } + if defaultKey && action != "remove-key" && action != "rename-key" { + mirror = &keys[i] + } + } + if mirror != nil { + s := settings.Load() + if err := setLegacyMirror(&s, *mirror); err != nil { + return "", err + } + // Revoke the old-version mirror before changing the active key store. + if err := settings.Save(s); err != nil { + return "", fmt.Errorf("Cannot change the default gateway key without updating settings.json: older Magpie versions may still accept its old credential. Make the settings file writable and retry; the key is unchanged: %w", err) + } + } + if err := save(keys); err != nil { + return "", err + } + return secret, nil +} + +func setLegacyMirror(s *settings.Settings, k Key) error { + s.LANKeyID = k.ID + if !k.Off { + s.LANKey = k.Secret + return nil + } + token, err := random(24) + if err != nil { + return err + } + s.LANKey = revokedLANPrefix + token + return nil +} + +// Authenticate reloads the store so revocation takes effect in running gateways. +func Authenticate(secret string) (Identity, bool) { + mu.Lock() + defer mu.Unlock() + keys, err := load() + if err != nil || secret == "" { + return Identity{}, false + } + for _, k := range keys { + if subtle.ConstantTimeCompare([]byte(secret), []byte(k.Secret)) == 1 { + if k.Off { + return Identity{}, false + } + return Identity{k.ID, k.Name}, true + } + } + s := settings.Load() + // A read-only config directory may prevent the first store write. Never + // fall back after either durable marker exists, or to a revoked mirror. + if s.LAN && s.LANKeyID == "" && s.LANKey != "" && !strings.HasPrefix(s.LANKey, revokedLANPrefix) && !slices.ContainsFunc(keys, func(k Key) bool { return k.LAN }) && subtle.ConstantTimeCompare([]byte(secret), []byte(s.LANKey)) == 1 { + k := legacyLANKey(s.LANKey) + return Identity{k.ID, k.Name}, true + } + return Identity{}, false +} + +func legacyLANKey(secret string) Key { + // The fallback and eventual persisted key must share their usage identity. + sum := sha256.Sum256([]byte(secret)) + return Key{ID: "lan-" + hex.EncodeToString(sum[:12]), Name: "Magpie", LAN: true, Secret: secret} +} + +// MigrateLegacyLANKey makes the old Settings key a normal, revocable caller +// key. Keep LANKey for older Magpie versions; LANKeyID marks a completed +// migration so the retained credential cannot resurrect a removed key. +func MigrateLegacyLANKey() error { + mu.Lock() + defer mu.Unlock() + return migrateLegacyLANKey() +} + +func MigrateLegacyLANKeyBestEffort() { + if err := MigrateLegacyLANKey(); err != nil { + log.Printf("magpie: could not migrate LAN key: %v", err) + } +} + +func migrateLegacyLANKey() error { + s := settings.Load() + if s.LANKey == "" || s.LANKeyID != "" || strings.HasPrefix(s.LANKey, revokedLANPrefix) { + return nil + } + keys, err := load() + if err != nil { + return err + } + if slices.ContainsFunc(keys, func(k Key) bool { return k.LAN }) { + return nil // the key-store write succeeded, even if settings were read-only + } + i := slices.IndexFunc(keys, func(k Key) bool { + return subtle.ConstantTimeCompare([]byte(k.Secret), []byte(s.LANKey)) == 1 + }) + if i < 0 { + keys = append(keys, legacyLANKey(s.LANKey)) + i = len(keys) - 1 + } + keys[i].LAN = true + if err := save(keys); err != nil { + return err + } + s.LANKeyID = keys[i].ID + return settings.Save(s) +} + +// ConfigureLAN starts sharing with a named key. Rotation preserves the key's +// identity, name and enabled state. +func ConfigureLAN(on, rotate bool) error { + mu.Lock() + defer mu.Unlock() + if err := migrateLegacyLANKey(); err != nil { + return err + } + s := settings.Load() + if on { + keys, err := load() + if err != nil { + return err + } + i := slices.IndexFunc(keys, func(k Key) bool { return k.ID == s.LANKeyID || k.LAN }) + if i < 0 || rotate { + token, err := random(24) + if err != nil { + return err + } + if i < 0 { + id, err := random(12) + if err != nil { + return err + } + keys = append(keys, Key{ID: id, Name: "Magpie", LAN: true}) + i = len(keys) - 1 + } + keys[i].Secret = Prefix + token + if err := save(keys); err != nil { + return err + } + s.LANKeyID = keys[i].ID + } + if err := setLegacyMirror(&s, keys[i]); err != nil { + return err + } + } + s.LAN = on + return settings.Save(s) +} + +func save(keys []Key) error { + b, err := json.MarshalIndent(keys, "", " ") + if err != nil { + return err + } + if err := os.MkdirAll(filepath.Dir(Path()), 0o755); err != nil { + return err + } + f, err := os.OpenFile(Path(), os.O_CREATE|os.O_WRONLY, 0o600) + if err != nil { + return err + } + if err := f.Close(); err != nil { + return err + } + if err := os.Chmod(Path(), 0o600); err != nil { + return err + } + return edit.WriteAtomic(Path(), append(b, '\n')) +} diff --git a/internal/access/access_test.go b/internal/access/access_test.go new file mode 100644 index 00000000..4debd46b --- /dev/null +++ b/internal/access/access_test.go @@ -0,0 +1,302 @@ +package access + +import ( + "encoding/json" + "os" + "strings" + "sync" + "testing" + + "github.com/yetone/magpie/internal/settings" +) + +func TestNamedKeys(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + if _, err := Update("add-key", Change{Name: " "}); err == nil { + t.Fatal("accepted empty name") + } + keys, err := List() + if err != nil || len(keys) != 0 { + t.Fatal(keys, err) + } + secret, err := Update("add-key", Change{Name: "Laptop"}) + if err != nil { + t.Fatal(err) + } + keys, err = List() + if err != nil || len(keys) != 1 || keys[0].Name != "Laptop" || keys[0].Secret != "" || !strings.HasPrefix(keys[0].Masked, Prefix+"…") { + t.Fatal(keys, err) + } + id := keys[0].ID + if who, ok := Authenticate(secret); !ok || who.KeyID != id || who.KeyName != "Laptop" { + t.Fatal(who, ok) + } + if _, ok := Authenticate(secret + "x"); ok { + t.Fatal("accepted invalid key") + } + for _, step := range []struct { + action, name string + valid bool + }{ + {"rename-key", "Travel", true}, {"off-key", "", false}, {"on-key", "", true}, + } { + if _, err := Update(step.action, Change{Key: id, Name: step.name}); err != nil { + t.Fatal(err) + } + if who, ok := Authenticate(secret); ok != step.valid || (ok && who.KeyName != "Travel") { + t.Fatal(step.action, who, ok) + } + } + if got, err := Update("copy-key", Change{Key: id}); err != nil || got != secret { + t.Fatal("copy", err) + } + st, err := os.Stat(Path()) + if err != nil || st.Mode().Perm() != 0o600 { + t.Fatal("permissions", err) + } + var persisted []Key + b, _ := os.ReadFile(Path()) + if err := json.Unmarshal(b, &persisted); err != nil || len(persisted) != 1 || persisted[0].ID != id || persisted[0].Secret != secret { + t.Fatal("key was not persisted", err) + } + if _, err := Update("remove-key", Change{Key: id}); err != nil { + t.Fatal(err) + } + if _, ok := Authenticate(secret); ok { + t.Fatal("removed key works") + } +} + +func TestConcurrentKeyCreationAndCorruptStore(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + var wg sync.WaitGroup + for i := 0; i < 12; i++ { + wg.Add(1) + go func() { + defer wg.Done() + if _, err := Update("add-key", Change{Name: "Key"}); err != nil { + t.Error(err) + } + }() + } + wg.Wait() + keys, _ := List() + if len(keys) != 12 { + t.Fatal("lost concurrent changes") + } + secret, _ := Update("copy-key", Change{Key: keys[0].ID}) + os.WriteFile(Path(), []byte("broken"), 0o600) + if _, ok := Authenticate(secret); ok { + t.Fatal("corrupt store accepted a key") + } + if _, err := Update("add-key", Change{Name: "New"}); err == nil { + t.Fatal("overwrote corrupt store") + } +} + +func TestLegacyLANKeyMigrationIsIdempotentAndRevocable(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + s := settings.Load() + s.LAN, s.LANKey = true, "sk-magpie-legacy-lan-secret" + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + if err := MigrateLegacyLANKey(); err != nil { + t.Fatal(err) + } + keys, err := List() + if err != nil || len(keys) != 1 || keys[0].Name != "Magpie" || !strings.HasPrefix(keys[0].Masked, "sk-magpie-…") { + t.Fatal(keys, err) + } + if settings.Load().LANKey != s.LANKey || !settings.Load().LAN { + t.Fatal("migration broke older Magpie's sharing settings") + } + if who, ok := Authenticate(s.LANKey); !ok || who.KeyID != keys[0].ID { + t.Fatal(who, ok) + } + if err := MigrateLegacyLANKey(); err != nil { + t.Fatal(err) + } + keys, _ = List() + if len(keys) != 1 { + t.Fatal("duplicate migration", keys) + } + if _, err := Update("off-key", Change{Key: keys[0].ID}); err != nil { + t.Fatal(err) + } + if _, ok := Authenticate(s.LANKey); ok { + t.Fatal("disabled LAN key still works") + } + if _, err := Update("remove-key", Change{Key: keys[0].ID}); err != nil { + t.Fatal(err) + } + if _, ok := Authenticate(s.LANKey); ok { + t.Fatal("removed LAN key still works") + } + if err := MigrateLegacyLANKey(); err != nil { + t.Fatal(err) + } + if keys, _ := List(); len(keys) != 0 { + t.Fatal("retained legacy credential resurrected a removed key", keys) + } +} + +func TestLegacyLANKeyMigrationPreservesNamedKeys(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + secret, err := Update("add-key", Change{Name: "Client"}) + if err != nil { + t.Fatal(err) + } + keys, _ := List() + id := keys[0].ID + s := settings.Load() + s.LANKey = "sk-magpie-legacy" + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + if err := MigrateLegacyLANKey(); err != nil { + t.Fatal(err) + } + keys, _ = List() + if len(keys) != 2 || keys[0].ID != id || keys[1].Name != "Magpie" { + t.Fatal(keys) + } + if _, ok := Authenticate(secret); !ok { + t.Fatal("lost named key") + } + if _, ok := Authenticate(s.LANKey); !ok { + t.Fatal("lost LAN key") + } +} + +func TestLegacyLANKeyMigrationRetryAndFailure(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + s := settings.Load() + s.LANKey = "sk-magpie-test-legacy-lan" + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + // The key write succeeded but saving the migration marker was interrupted. + mu.Lock() + err := save([]Key{{ID: "migrated", Name: "Remote", Off: true, Secret: s.LANKey}}) + mu.Unlock() + if err != nil { + t.Fatal(err) + } + if err := MigrateLegacyLANKey(); err != nil { + t.Fatal(err) + } + keys, err := List() + if err != nil || len(keys) != 1 || keys[0].ID != "migrated" || !keys[0].Off || settings.Load().LANKeyID != "migrated" { + t.Fatal("retry duplicated or re-enabled a key", keys, err) + } + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(Path(), []byte("broken"), 0o600); err != nil { + t.Fatal(err) + } + if err := MigrateLegacyLANKey(); err == nil { + t.Fatal("migration accepted a corrupt store") + } + if settings.Load().LANKey != s.LANKey { + t.Fatal("failed migration cleared the old credential") + } +} + +func TestNamedLANKeyKeepsItsName(t *testing.T) { + for _, name := range []string{"Local network", "Magpie", "Desk"} { + t.Run(name, func(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + s := settings.Load() + s.LANKeyID = "lan" + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + original := []Key{{ID: "lan", Name: name, Off: true, Secret: Prefix + "original"}, + {ID: "other", Name: "Other", Secret: Prefix + "other"}, + {ID: "named", Name: "Local network", Secret: Prefix + "named"}} + if err := save(original); err != nil { + t.Fatal(err) + } + for range 2 { + if err := MigrateLegacyLANKey(); err != nil { + t.Fatal(err) + } + } + keys, err := load() + if err != nil || len(keys) != 3 || keys[0] != original[0] || keys[1] != original[1] || keys[2] != original[2] { + t.Fatal("migration changed an existing named key", keys, err) + } + }) + } +} + +func TestCallerKeyRotationPreservesIdentityAndState(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + old, err := Update("add-key", Change{Name: "Desk"}) + if err != nil { + t.Fatal(err) + } + other, err := Update("add-key", Change{Name: "Other"}) + if err != nil { + t.Fatal(err) + } + keys, _ := List() + id := keys[0].ID + for _, off := range []bool{false, true} { + if off { + if _, err := Update("off-key", Change{Key: id}); err != nil { + t.Fatal(err) + } + } + secret, err := Update("rotate-key", Change{Key: id}) + if err != nil || secret == old || !strings.HasPrefix(secret, Prefix) { + t.Fatal("key was not rotated", err) + } + if _, ok := Authenticate(old); ok { + t.Fatal("old key still authenticates") + } + if who, ok := Authenticate(secret); ok == off || (!off && (who.KeyID != id || who.KeyName != "Desk")) { + t.Fatal("rotation changed identity or enabled state", who, ok) + } + keys, _ = List() + if keys[0].ID != id || keys[0].Name != "Desk" || keys[0].Off != off || keys[0].Secret != "" { + t.Fatal(keys) + } + if _, ok := Authenticate(other); !ok { + t.Fatal("rotation revoked another key") + } + old = secret + } + if _, err := Update("rotate-key", Change{Key: "missing"}); err == nil { + t.Fatal("rotated a missing key") + } +} + +func TestLANRotationKeepsOlderMagpieWorking(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + if err := ConfigureLAN(true, false); err != nil { + t.Fatal(err) + } + before := settings.Load() + if before.LANKey == "" { + t.Fatal("older Magpie has no sharing key") + } + next, err := Update("rotate-key", Change{Key: before.LANKeyID}) + if err != nil { + t.Fatal(err) + } + if settings.Load().LANKey != next { + t.Fatal("rotation left the old version with a stale key") + } + if err := MigrateLegacyLANKey(); err != nil { + t.Fatal(err) + } + if keys, _ := List(); len(keys) != 1 || keys[0].ID != before.LANKeyID { + t.Fatal("rotation duplicated the default key", keys) + } + if _, ok := Authenticate(before.LANKey); ok { + t.Fatal("old key resurrected") + } +} diff --git a/internal/access/legacy_fallback_test.go b/internal/access/legacy_fallback_test.go new file mode 100644 index 00000000..533a13a6 --- /dev/null +++ b/internal/access/legacy_fallback_test.go @@ -0,0 +1,101 @@ +package access + +import ( + "errors" + "os" + "testing" + + "github.com/yetone/magpie/internal/settings" +) + +func TestLegacyLANFallbackBeforeMigration(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + s := settings.Settings{LAN: true, LANKey: "sk-magpie-old-client"} + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + before, ok := Authenticate(s.LANKey) + if !ok || before.KeyID == "" || before.KeyName != "Magpie" { + t.Fatal("unmigrated legacy key lost access", before, ok) + } + if _, err := os.Stat(Path()); !errors.Is(err, os.ErrNotExist) { + t.Fatal("authentication must not create a key store", err) + } + for _, secret := range []string{"", "wrong", "magpie", s.LANKey + "x"} { + if _, ok := Authenticate(secret); ok { + t.Fatal("legacy fallback accepted another token") + } + } + if err := MigrateLegacyLANKey(); err != nil { + t.Fatal(err) + } + if after, ok := Authenticate(s.LANKey); !ok || after != before { + t.Fatal("migration split the legacy caller's usage identity", before, after, ok) + } +} + +func TestLegacyLANFallbackDoesNotBypassRevocation(t *testing.T) { + for _, tc := range []struct { + name string + cfg settings.Settings + keys []Key + }{ + {"not-shared", settings.Settings{LANKey: "legacy"}, nil}, + {"completed", settings.Settings{LAN: true, LANKey: "legacy", LANKeyID: "removed"}, nil}, + {"revoked-mirror", settings.Settings{LAN: true, LANKey: revokedLANPrefix + "old"}, nil}, + {"disabled", settings.Settings{LAN: true, LANKey: "legacy"}, []Key{{ID: "default", LAN: true, Secret: "legacy", Off: true}}}, + {"rotated", settings.Settings{LAN: true, LANKey: "legacy"}, []Key{{ID: "default", LAN: true, Secret: "new"}}}, + {"disabled-matching-key", settings.Settings{LAN: true, LANKey: "legacy"}, []Key{{ID: "other", Secret: "legacy", Off: true}}}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + if err := settings.Save(tc.cfg); err != nil { + t.Fatal(err) + } + if tc.keys != nil { + if err := Restore(tc.keys); err != nil { + t.Fatal(err) + } + } + if _, ok := Authenticate(tc.cfg.LANKey); ok { + t.Fatal("fallback accepted a revoked or inactive legacy credential") + } + }) + } +} + +func TestLegacyLANFallbackWithReadOnlyDirectory(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + s := settings.Settings{LAN: true, LANKey: "sk-magpie-readonly"} + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + if err := os.Chmod(settings.Dir(), 0o500); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { os.Chmod(settings.Dir(), 0o700) }) + if err := MigrateLegacyLANKey(); err == nil { + t.Skip("configuration directory remains writable") + } else if !errors.Is(err, os.ErrPermission) { + t.Fatal(err) + } + for range 3 { + if who, ok := Authenticate(s.LANKey); !ok || who.KeyID == "" || who.KeyName != "Magpie" { + t.Fatal("failed key-store creation broke legacy access", who, ok) + } + } +} + +func TestLegacyLANFallbackRefusesCorruptStore(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + s := settings.Settings{LAN: true, LANKey: "legacy"} + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(Path(), []byte("broken"), 0o600); err != nil { + t.Fatal(err) + } + if _, ok := Authenticate(s.LANKey); ok { + t.Fatal("an unreadable revocation state must not enable legacy fallback") + } +} diff --git a/internal/access/legacy_revocation_test.go b/internal/access/legacy_revocation_test.go new file mode 100644 index 00000000..0f264875 --- /dev/null +++ b/internal/access/legacy_revocation_test.go @@ -0,0 +1,133 @@ +package access + +import ( + "errors" + "os" + "strings" + "testing" + + "github.com/yetone/magpie/internal/settings" +) + +func TestDefaultKeyRevocationSurvivesDowngrade(t *testing.T) { + for _, action := range []string{"off-key", "remove-key"} { + t.Run(action, func(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + if err := ConfigureLAN(true, false); err != nil { + t.Fatal(err) + } + before := settings.Load() + if _, err := Update(action, Change{Key: before.LANKeyID}); err != nil { + t.Fatal(err) + } + after := settings.Load() + if after.LANKey == "" || after.LANKey == before.LANKey { + t.Fatal("older Magpie would accept the revoked credential") + } + if _, ok := Authenticate(after.LANKey); ok { + t.Fatal("legacy replacement is an active gateway key") + } + // An older binary saves settings without the new migration marker. + after.LANKeyID = "" + if err := settings.Save(after); err != nil { + t.Fatal(err) + } + if err := MigrateLegacyLANKey(); err != nil { + t.Fatal(err) + } + keys, err := List() + if err != nil || (action == "remove-key" && len(keys) != 0) || (action == "off-key" && (len(keys) != 1 || !keys[0].Off)) { + t.Fatal("downgrade resurrected a key", keys, err) + } + if action == "off-key" { + if _, err := Update("rotate-key", Change{Key: before.LANKeyID}); err != nil { + t.Fatal(err) + } + current, _ := Update("copy-key", Change{Key: before.LANKeyID}) + if settings.Load().LANKey == current { + t.Fatal("rotation re-enabled the disabled legacy credential") + } + if _, err := Update("on-key", Change{Key: before.LANKeyID}); err != nil { + t.Fatal(err) + } + if settings.Load().LANKey != current { + t.Fatal("re-enabling did not restore the legacy mirror") + } + } + }) + } +} + +func TestReadOnlyMigrationDoesNotRetryCompletedKeyWrite(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + s := settings.Settings{LAN: true, LANKey: "sk-magpie-fixture-legacy"} + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + if err := os.Chmod(settings.Path(), 0o400); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { os.Chmod(settings.Path(), 0o600) }) + if err := settings.Save(s); err == nil { + t.Skip("settings.json remains writable") + } + if err := MigrateLegacyLANKey(); err == nil { + t.Fatal("initial marker write unexpectedly succeeded") + } + for range 3 { + if err := MigrateLegacyLANKey(); err != nil { + t.Fatal("completed key migration retried read-only settings", err) + } + } + if who, ok := Authenticate(s.LANKey); !ok || who.KeyName != "Magpie" { + t.Fatal("legacy key was not usable after partial migration", who, ok) + } +} + +func TestReadOnlyDefaultKeyMutationExplainsLegacyMirror(t *testing.T) { + for _, action := range []string{"rotate-key", "off-key", "remove-key"} { + t.Run(action, func(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + if err := ConfigureLAN(true, false); err != nil { + t.Fatal(err) + } + s := settings.Load() + independent, err := Update("add-key", Change{Name: "Laptop"}) + if err != nil { + t.Fatal(err) + } + if err := os.Chmod(settings.Path(), 0o400); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { os.Chmod(settings.Path(), 0o600) }) + if err := settings.Save(s); err == nil { + t.Skip("settings.json remains writable") + } + before, err := os.ReadFile(Path()) + if err != nil { + t.Fatal(err) + } + _, err = Update(action, Change{Key: s.LANKeyID}) + if !errors.Is(err, os.ErrPermission) || !strings.Contains(err.Error(), "older Magpie") || !strings.Contains(err.Error(), "writable") || !strings.Contains(err.Error(), "unchanged") { + t.Fatal("default-key error does not explain the recovery or downgrade risk", err) + } + after, err := os.ReadFile(Path()) + if err != nil || string(after) != string(before) { + t.Fatal("failed legacy mirror write changed active keys", err) + } + if _, ok := Authenticate(s.LANKey); !ok { + t.Fatal("operation reported failure but revoked the default key") + } + who, ok := Authenticate(independent) + if !ok { + t.Fatal("independent key is missing") + } + if _, err := Update(action, Change{Key: who.KeyID}); err != nil { + t.Fatal("read-only settings blocked an independent key", err) + } + if _, ok := Authenticate(independent); ok { + t.Fatal("independent key kept its old credential") + } + }) + } +} diff --git a/internal/backup/backup.go b/internal/backup/backup.go index f6704535..0794f4cc 100644 --- a/internal/backup/backup.go +++ b/internal/backup/backup.go @@ -28,6 +28,7 @@ import ( "strings" "time" + "github.com/yetone/magpie/internal/access" "github.com/yetone/magpie/internal/agent" "github.com/yetone/magpie/internal/edit" "github.com/yetone/magpie/internal/library" @@ -46,17 +47,18 @@ const ( // Bundle is what a backup holds. type Bundle struct { - Version int `json:"version"` - Created time.Time `json:"created"` - App string `json:"app,omitempty"` // the magpie that made it - Keys bool `json:"keys"` // whether the providers carry their keys - Providers []provider.Provider `json:"providers"` - Icons map[string][]byte `json:"icons,omitempty"` // pictures picked for providers, by file name - Groups []provider.Group `json:"groups,omitempty"` // the user's model groups - Settings *settings.Settings `json:"settings,omitempty"` - Profiles map[string]profile.Profile `json:"profiles,omitempty"` - Agents map[string]string `json:"agents,omitempty"` // every agent's fields as they are now - Library *library.Bundle `json:"library,omitempty"` // nil from a magpie before it, or with none + Version int `json:"version"` + Created time.Time `json:"created"` + App string `json:"app,omitempty"` // the magpie that made it + Keys bool `json:"keys"` // whether credentials are included + Providers []provider.Provider `json:"providers"` + Icons map[string][]byte `json:"icons,omitempty"` // pictures picked for providers, by file name + Groups []provider.Group `json:"groups,omitempty"` // the user's model groups + Settings *settings.Settings `json:"settings,omitempty"` + GatewayKeys *[]access.Key `json:"gatewayKeys,omitempty"` + Profiles map[string]profile.Profile `json:"profiles,omitempty"` + Agents map[string]string `json:"agents,omitempty"` // every agent's fields as they are now + Library *library.Bundle `json:"library,omitempty"` // nil from a magpie before it, or with none // Searches are the web search APIs (#419), their keys with the // providers'; nil from a magpie before them. Searches *[]provider.SearchAPI `json:"searches,omitempty"` @@ -90,6 +92,18 @@ func Collect(keys bool, app string) (Bundle, error) { if b.Groups, err = provider.StoredGroups(); err != nil { return b, err } + if keys { + access.MigrateLegacyLANKeyBestEffort() + if _, err := os.Stat(access.Path()); err == nil { + gatewayKeys, err := access.Export() + if err != nil { + return b, err + } + b.GatewayKeys = &gatewayKeys + } else if !errors.Is(err, os.ErrNotExist) { + return b, err + } + } for _, p := range stored { if !keys { p = withoutKeys(p) @@ -116,6 +130,9 @@ func Collect(keys bool, app string) (Bundle, error) { b.Searches = &searches if _, err := os.Stat(settings.Path()); err == nil { s := settings.Load() + if !keys { + s.LANKey, s.LANKeyID = "", "" + } b.Settings = &s } ps, err := profile.Load() @@ -285,13 +302,44 @@ func Restore(b Bundle, parts Parts) (Result, error) { } } } - if parts.Settings && b.Settings != nil { + if parts.Settings && (b.Settings != nil || b.GatewayKeys != nil) { // the window's size and the proxy are this machine's own - s, cur := *b.Settings, settings.Load() + cur := settings.Load() + s := cur + if b.Settings != nil { + s = *b.Settings + } s.Window, s.Proxy, s.Dock, s.DockWindow = cur.Window, cur.Proxy, cur.Dock, cur.DockWindow + if !b.Keys { + s.LANKey, s.LANKeyID = cur.LANKey, cur.LANKeyID + } else if b.GatewayKeys == nil { + // An older backup may carry a marker without its named-key store. + s.LANKeyID = "" + keys, err := access.List() + if err != nil { + return r, err + } + if slices.ContainsFunc(keys, func(k access.Key) bool { return k.LAN }) { + // Its missing store must not detach this machine's default key. + s.LANKey, s.LANKeyID = cur.LANKey, cur.LANKeyID + } + } if err := settings.Save(s); err != nil { return r, err } + if b.Keys && b.GatewayKeys != nil { + if err := access.Restore(*b.GatewayKeys); err != nil { + // Settings must be writable before replacing credentials. If the + // store refuses the write, restore their original association. + if rollback := settings.Save(cur); rollback != nil { + return r, errors.Join(err, fmt.Errorf("could not restore original settings after gateway-key restore failed: %w", rollback)) + } + return r, err + } + } + if b.Keys && b.GatewayKeys == nil { + access.MigrateLegacyLANKeyBestEffort() + } r.Settings = true } if parts.Profiles && len(b.Profiles) > 0 { diff --git a/internal/backup/caller_keys_test.go b/internal/backup/caller_keys_test.go new file mode 100644 index 00000000..bd0f17fc --- /dev/null +++ b/internal/backup/caller_keys_test.go @@ -0,0 +1,212 @@ +package backup + +import ( + "encoding/json" + "errors" + "os" + "reflect" + "strings" + "testing" + + "github.com/yetone/magpie/internal/access" + "github.com/yetone/magpie/internal/settings" +) + +func TestGatewayKeysBackupRoundTrip(t *testing.T) { + home(t) + if err := access.ConfigureLAN(true, false); err != nil { + t.Fatal(err) + } + defaultKey := settings.Load() + client, err := access.Update("add-key", access.Change{Name: "Laptop"}) + if err != nil { + t.Fatal(err) + } + keys, _ := access.List() + if _, err := access.Update("off-key", access.Change{Key: defaultKey.LANKeyID}); err != nil { + t.Fatal(err) + } + b, err := Collect(true, "test") + if err != nil { + t.Fatal(err) + } + data, err := Seal(b, "gateway passphrase") + if err != nil || strings.Contains(string(data), client) || strings.Contains(string(data), defaultKey.LANKey) { + t.Fatal("backup did not seal gateway credentials", err) + } + home(t) + b, err = Open(data, "gateway passphrase") + if err != nil { + t.Fatal(err) + } + if _, err := Restore(b, Parts{Settings: true}); err != nil { + t.Fatal(err) + } + got, err := access.List() + if err != nil || len(got) != len(keys) || got[0].ID != defaultKey.LANKeyID || !got[0].Off || got[1].ID != keys[1].ID || got[1].Name != "Laptop" { + t.Fatal("named key identity/state lost during restore", got, err) + } + if who, ok := access.Authenticate(client); !ok || who.KeyID != keys[1].ID { + t.Fatal("restored client cannot authenticate", who, ok) + } + if _, ok := access.Authenticate(defaultKey.LANKey); ok { + t.Fatal("restore resurrected the disabled default key") + } + if err := access.MigrateLegacyLANKey(); err != nil { + t.Fatal(err) + } + if got, _ := access.List(); len(got) != len(keys) { + t.Fatal("restore imported a duplicate default key", got) + } + if _, err := access.Update("on-key", access.Change{Key: defaultKey.LANKeyID}); err != nil { + t.Fatal(err) + } + if settings.Load().LANKey != defaultKey.LANKey { + t.Fatal("restored default key lost its legacy link") + } + info, err := os.Stat(access.Path()) + if err != nil || info.Mode().Perm() != 0o600 { + t.Fatal("restored key-store permissions", info, err) + } +} + +func TestGatewayKeysWithoutSecretsOrSettingsRestore(t *testing.T) { + home(t) + if err := access.ConfigureLAN(true, false); err != nil { + t.Fatal(err) + } + secret := settings.Load().LANKey + full, err := Collect(true, "test") + if err != nil { + t.Fatal(err) + } + without, err := Collect(false, "test") + if err != nil { + t.Fatal(err) + } + plain, _ := json.Marshal(without) + if strings.Contains(string(plain), secret) || without.Settings.LANKey != "" || without.Settings.LANKeyID != "" { + t.Fatal("--no-keys backup contains a gateway credential") + } + home(t) + if err := access.ConfigureLAN(true, false); err != nil { + t.Fatal(err) + } + current := settings.Load() + if _, err := Restore(full, Parts{}); err != nil { + t.Fatal(err) + } + if _, err := Restore(without, Parts{Settings: true}); err != nil { + t.Fatal(err) + } + if got := settings.Load(); got.LANKey != current.LANKey || got.LANKeyID != current.LANKeyID { + t.Fatal("keyless restore replaced this machine's LAN credential") + } + if _, ok := access.Authenticate(current.LANKey); !ok { + t.Fatal("restore without gateway keys revoked this machine's key") + } +} + +func TestOlderBackupRecoversLegacyGatewayCredential(t *testing.T) { + home(t) + b := Bundle{Version: 1, Keys: true, Settings: &settings.Settings{LAN: true, LANKey: "{{API_KEY_88kqppzx}}", LANKeyID: "id-from-incomplete-backup"}} + if _, err := Restore(b, Parts{Settings: true}); err != nil { + t.Fatal(err) + } + if who, ok := access.Authenticate(b.Settings.LANKey); !ok || who.KeyName != "Magpie" { + t.Fatal("old backup's migration marker lost its credential", who, ok) + } +} + +func TestOlderBackupKeepsExistingGatewayKeyStore(t *testing.T) { + home(t) + if err := access.ConfigureLAN(true, false); err != nil { + t.Fatal(err) + } + cur := settings.Load() + b := Bundle{Version: 1, Keys: true, Settings: &settings.Settings{LAN: true, LANKey: "fixture-older-backup-key", LANKeyID: "old-marker"}} + if _, err := Restore(b, Parts{Settings: true}); err != nil { + t.Fatal(err) + } + if got := settings.Load(); got.LANKey != cur.LANKey || got.LANKeyID != cur.LANKeyID { + t.Fatal("backup without a key store detached the existing default key") + } + if _, ok := access.Authenticate(cur.LANKey); !ok { + t.Fatal("backup without a key store revoked the existing gateway key") + } +} + +func TestGatewayKeysBackupWithoutSettingsFile(t *testing.T) { + home(t) + secret, err := access.Update("add-key", access.Change{Name: "CLI client"}) + if err != nil { + t.Fatal(err) + } + b, err := Collect(true, "test") + if err != nil || b.Settings != nil { + t.Fatal("CLI-only key fixture unexpectedly has settings", err) + } + home(t) + if _, err := Restore(b, Parts{Settings: true}); err != nil { + t.Fatal(err) + } + if who, ok := access.Authenticate(secret); !ok || who.KeyName != "CLI client" { + t.Fatal("standalone CLI key was not restored", who, ok) + } +} + +func TestGatewayKeyRestoreFailureKeepsSettingsAndKeys(t *testing.T) { + for _, failure := range []string{"invalid-settings", "readonly-settings", "readonly-store"} { + t.Run(failure, func(t *testing.T) { + home(t) + if err := access.ConfigureLAN(true, false); err != nil { + t.Fatal(err) + } + cur := settings.Load() + before, err := os.ReadFile(access.Path()) + if err != nil { + t.Fatal(err) + } + incoming := cur + incoming.Theme, incoming.LANKeyID, incoming.LANKey = "dark", "restored", "sk-magpie-restored" + keys := []access.Key{{ID: incoming.LANKeyID, Name: "Restored", Secret: incoming.LANKey, LAN: true}} + if failure == "invalid-settings" { + incoming.Theme = "invalid-theme" + } else { + path := settings.Path() + if failure == "readonly-store" { + path = access.Path() + } + if err := os.Chmod(path, 0o400); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { os.Chmod(path, 0o600) }) + f, err := os.OpenFile(path, os.O_WRONLY, 0) + if err == nil { + f.Close() + t.Skip("file remains writable") + } + if !errors.Is(err, os.ErrPermission) { + t.Fatal(err) + } + } + r, err := Restore(Bundle{Version: 1, Keys: true, Settings: &incoming, GatewayKeys: &keys}, Parts{Settings: true}) + if err == nil || r.Settings { + t.Fatal("failed restore reported success", r, err) + } + if failure != "invalid-settings" && !errors.Is(err, os.ErrPermission) { + t.Fatal("restore lost permission error", err) + } + after, err := os.ReadFile(access.Path()) + if err != nil || string(after) != string(before) || !reflect.DeepEqual(settings.Load(), cur) { + t.Fatal("failed restore left settings and keys inconsistent", err) + } + if _, ok := access.Authenticate(cur.LANKey); !ok { + t.Fatal("failed restore revoked the current key") + } + if _, ok := access.Authenticate(incoming.LANKey); ok { + t.Fatal("failed restore activated the incoming key") + } + }) + } +} diff --git a/internal/gateway/caller_regression_test.go b/internal/gateway/caller_regression_test.go new file mode 100644 index 00000000..fcd16aff --- /dev/null +++ b/internal/gateway/caller_regression_test.go @@ -0,0 +1,306 @@ +package gateway + +import ( + "bytes" + "context" + "errors" + "io" + "log" + "net/http" + "net/http/httptest" + "os" + "strings" + "sync" + "testing" + "time" + + "github.com/yetone/magpie/internal/access" + "github.com/yetone/magpie/internal/provider" + "github.com/yetone/magpie/internal/settings" + "github.com/yetone/magpie/internal/usage" +) + +func TestLoopbackRemainsPermissive(t *testing.T) { + fresh(t) + keys, secrets := newCaller(t, "Desk") + access.Update("off-key", access.Change{Key: keys[0].ID}) + for _, shared := range []bool{false, true} { + if err := settings.Save(settings.Settings{LAN: shared}); err != nil { + t.Fatal(err) + } + for _, token := range []string{"", "anything", Token, TokenFor("alma"), "sk-magpie-stale", secrets[0]} { + for _, addr := range []string{"127.0.0.1:5000", "[::1]:5000"} { + r := httptest.NewRequest("GET", "/v1/models", nil) + r.RemoteAddr = addr + r.Header.Set("Authorization", "Bearer "+token) + w := httptest.NewRecorder() + lanGuard(New().Handler()).ServeHTTP(w, r) + if w.Code != 200 { + t.Fatalf("shared=%v %s %q: %d", shared, addr, token, w.Code) + } + } + } + } +} + +func TestExplicitOpenGatewayAcceptsAllRemoteTokens(t *testing.T) { + fresh(t) + t.Setenv("MAGPIE_ADDR", "0.0.0.0:3425") + if _, err := access.Update("add-key", access.Change{Name: "Unused"}); err != nil { + t.Fatal(err) + } + keys, _ := access.List() + secret, _ := access.Update("copy-key", access.Change{Key: keys[0].ID}) + for _, token := range []string{"", "anything", "sk-magpie-stale", secret} { + for _, header := range []string{"Authorization", "x-api-key", "query"} { + r := httptest.NewRequest("GET", "/v1/models", nil) + r.RemoteAddr = "192.168.1.9:5000" + if header == "query" { + r.URL.RawQuery = "key=" + token + } else if header == "Authorization" { + r.Header.Set(header, "Bearer "+token) + } else { + r.Header.Set(header, token) + } + w := httptest.NewRecorder() + lanGuard(New().Handler()).ServeHTTP(w, r) + if w.Code != 200 { + t.Fatalf("explicit open gateway rejected %s %q: %d", header, token, w.Code) + } + } + } +} + +func TestMigrationFailureDoesNotStopGateway(t *testing.T) { + fresh(t) + t.Setenv("MAGPIE_ADDR", "127.0.0.1:0") + if err := settings.Save(settings.Settings{LAN: true, LANKey: "sk-magpie-old"}); err != nil { + t.Fatal(err) + } + if err := os.Chmod(settings.Path(), 0o400); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { os.Chmod(settings.Path(), 0o600) }) + if err := settings.Save(settings.Load()); err == nil { + t.Skip("settings.json remains writable on this platform") + } + var logs bytes.Buffer + old := log.Writer() + log.SetOutput(&logs) + defer log.SetOutput(old) + s := New() + if err := s.Relisten(); err != nil { + t.Fatal("migration stopped Relisten", err) + } + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + done := make(chan error, 1) + go func() { done <- s.ListenAndServe(ctx) }() + defer func() { + cancel() + if err := <-done; err != nil { + t.Error(err) + } + if !strings.Contains(logs.String(), "migrat") { + t.Error("migration failure was not logged") + } + }() + deadline := time.Now().Add(3 * time.Second) + for { + s.lnMu.Lock() + ln := s.ln + s.lnMu.Unlock() + if ln != nil { + break + } + if time.Now().After(deadline) { + t.Fatal("gateway did not bind after failed migration") + } + time.Sleep(10 * time.Millisecond) + } +} + +func TestLegacyCallerWithReadOnlyConfiguration(t *testing.T) { + fresh(t) + up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer upstream" || r.URL.Query().Get("key") == "sk-magpie-readonly" { + t.Error("legacy caller credential leaked upstream") + } + w.Header().Set("Content-Type", "application/json") + io.WriteString(w, `{"choices":[{"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}],"usage":{"prompt_tokens":30,"completion_tokens":5}}`) + })) + t.Cleanup(up.Close) + if err := provider.Save(provider.Provider{ID: "plan", Name: "Plan", Key: "upstream", Chat: up.URL + "/v1", Models: []string{"m1"}}); err != nil { + t.Fatal(err) + } + s := settings.Settings{LAN: true, LANKey: "sk-magpie-readonly"} + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + // Existing logs remain appendable even if no new store can be created. + if err := os.WriteFile(usage.Path(), nil, 0o600); err != nil { + t.Fatal(err) + } + if err := os.Chmod(settings.Dir(), 0o500); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { os.Chmod(settings.Dir(), 0o700) }) + if err := migrateLANKey(); err == nil { + t.Skip("configuration directory remains writable") + } else if !errors.Is(err, os.ErrPermission) { + t.Fatal(err) + } + h := lanGuard(New().Handler()) + call := func(secret, header string) int { + r := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(chatReq)) + r.RemoteAddr = "192.168.1.9:5000" + switch header { + case "Authorization": + r.Header.Set(header, "Bearer "+secret) + case "query": + r.URL.RawQuery = "key=" + secret + default: + r.Header.Set(header, secret) + } + w := httptest.NewRecorder() + h.ServeHTTP(w, r) + return w.Code + } + for _, header := range []string{"Authorization", "x-api-key", "query"} { + if code := call(s.LANKey, header); code != http.StatusOK { + t.Fatal("unmigrated legacy client lost access", header, code) + } + for _, secret := range []string{"", "wrong", Token} { + if code := call(secret, header); code != http.StatusUnauthorized { + t.Fatal("fallback accepted another credential", header, code) + } + } + } + recs := usage.Load(time.Time{}) + if len(recs) != 3 || recs[0].CallerKeyID == "" { + t.Fatal("fallback attribution missing", recs) + } + for _, rec := range recs { + if rec.CallerKeyID != recs[0].CallerKeyID || rec.CallerKeyName != "Magpie" || rec.ProviderKeyID != provider.KeyID("upstream") { + t.Fatal("fallback attribution changed", rec) + } + } + if _, err := os.Stat(access.Path()); !errors.Is(err, os.ErrNotExist) { + t.Fatal("fallback wrote a key store", err) + } + log, err := os.ReadFile(usage.Path()) + if err != nil || strings.Contains(string(log), s.LANKey) { + t.Fatal("legacy credential in usage log", err) + } + if err := os.Chmod(settings.Dir(), 0o700); err != nil { + t.Fatal(err) + } + if err := migrateLANKey(); err != nil { + t.Fatal(err) + } + if code := call(s.LANKey, "Authorization"); code != 200 || lastUsage(t).CallerKeyID != recs[0].CallerKeyID { + t.Fatal("migration changed the fallback identity", code) + } + if _, err := access.Update("remove-key", access.Change{Key: recs[0].CallerKeyID}); err != nil { + t.Fatal(err) + } + if code := call(s.LANKey, "Authorization"); code != 401 { + t.Fatal("fallback resurrected a removed legacy key", code) + } +} + +func TestCallerUsageSystemOne(t *testing.T) { + fresh(t) + up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer upstream" { + t.Error("caller key reached System One upstream") + } + w.Header().Set("Content-Type", "application/json") + io.WriteString(w, `{"model":"jev","usage":{"input_tokens":120,"output_tokens":2}}`) + })) + defer up.Close() + if err := provider.Save(provider.Provider{ID: "jev", Name: "Jev", Key: "upstream", Decide: up.URL + "/v1"}); err != nil { + t.Fatal(err) + } + keys, secrets := newCaller(t, "System One client") + r := httptest.NewRequest("POST", "/v1/systemone", strings.NewReader(`{"model":"jev/jev-latest","state":{"message":"hi"},"questions":{}}`)) + r.Header.Set("Authorization", "Bearer "+secrets[0]) + w := httptest.NewRecorder() + s := New() + s.Handler().ServeHTTP(w, r) + if w.Code != 200 { + t.Fatal(w.Code, w.Body) + } + if rec := lastUsage(t); rec.RouteID == 0 || rec.RouteID != lastRoute(s).ID || rec.CallerKeyID != keys[0].ID || rec.CallerKeyName != "System One client" || rec.ProviderKeyID != provider.KeyID("upstream") || rec.Input != 120 { + t.Fatal(rec) + } +} + +func TestCallerUsageCodexOwnModel(t *testing.T) { + setup(t, provider.Chat, &fake{t: t}) + chatgpt(t, func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer chatgpt-token" { + t.Error("Codex's own sign-in was changed") + } + w.Header().Set("Content-Type", "text/event-stream") + io.WriteString(w, sse(`data: {"type":"response.completed","response":{"id":"r1","usage":{"input_tokens":9,"output_tokens":2}}}`)) + }) + r := httptest.NewRequest("POST", CodexPath+"/responses", strings.NewReader(`{"model":"gpt-5.5","stream":true,"input":"hi"}`)) + r.Header.Set("Authorization", "Bearer chatgpt-token") + r = r.WithContext(access.WithIdentity(r.Context(), access.Identity{KeyID: "desk", KeyName: "Desk"})) + w := httptest.NewRecorder() + s := New() + s.Handler().ServeHTTP(w, r) + if w.Code != 200 { + t.Fatal(w.Code, w.Body) + } + if rec := lastUsage(t); rec.RouteID == 0 || rec.RouteID != lastRoute(s).ID || rec.CallerKeyID != "desk" || rec.CallerKeyName != "Desk" || rec.Input != 9 || rec.Output != 2 { + t.Fatal(rec) + } +} + +func TestConcurrentCallerAttribution(t *testing.T) { + fresh(t) + up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer upstream" { + t.Error("caller credential leaked upstream") + } + w.Header().Set("Content-Type", "application/json") + io.WriteString(w, `{"choices":[{"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}],"usage":{"prompt_tokens":30,"completion_tokens":5}}`) + })) + defer up.Close() + if err := provider.Save(provider.Provider{ID: "plan", Name: "Plan", Key: "upstream", Chat: up.URL + "/v1", Models: []string{"m1"}}); err != nil { + t.Fatal(err) + } + keys, secrets := newCaller(t, "Desk", "Server") + if err := settings.Save(settings.Settings{LAN: true}); err != nil { + t.Fatal(err) + } + h := lanGuard(New().Handler()) + var wg sync.WaitGroup + for i := range 120 { + wg.Add(1) + go func() { + defer wg.Done() + r := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(chatReq)) + r.RemoteAddr = "192.168.1.9:5000" + r.Header.Set("Authorization", "Bearer "+secrets[i%len(secrets)]) + w := httptest.NewRecorder() + h.ServeHTTP(w, r) + if w.Code != 200 { + t.Error(w.Code, w.Body) + } + }() + } + wg.Wait() + counts := map[string]int{} + for _, rec := range usage.Load(time.Time{}) { + counts[rec.CallerKeyID]++ + if rec.CallerKeyName != keys[0].Name && rec.CallerKeyName != keys[1].Name { + t.Error("lost caller name", rec) + } + } + if len(counts) != 2 || counts[keys[0].ID] != 60 || counts[keys[1].ID] != 60 { + t.Fatal("concurrent calls mixed identities", counts) + } +} diff --git a/internal/gateway/caller_usage_test.go b/internal/gateway/caller_usage_test.go new file mode 100644 index 00000000..1160e5ee --- /dev/null +++ b/internal/gateway/caller_usage_test.go @@ -0,0 +1,323 @@ +package gateway + +import ( + "io" + "net/http" + "net/http/httptest" + "os" + "strings" + "testing" + "time" + + "github.com/yetone/magpie/internal/access" + "github.com/yetone/magpie/internal/provider" + "github.com/yetone/magpie/internal/settings" + "github.com/yetone/magpie/internal/usage" +) + +func newCaller(t *testing.T, names ...string) ([]access.Key, []string) { + t.Helper() + s := settings.Load() + s.LAN = true + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + var secrets []string + for _, name := range names { + secret, err := access.Update("add-key", access.Change{Name: name}) + if err != nil { + t.Fatal(err) + } + secrets = append(secrets, secret) + } + keys, err := access.List() + if err != nil { + t.Fatal(err) + } + return keys[len(keys)-len(names):], secrets +} + +func TestCallerUsageAcrossKeys(t *testing.T) { + fresh(t) + up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer upstream-secret" { + t.Error("caller credential reached provider") + } + w.Header().Set("Content-Type", "application/json") + w.Header().Set("X-Request-Id", "req-caller") + io.WriteString(w, `{"choices":[{"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}],"usage":{"prompt_tokens":30,"completion_tokens":5}}`) + })) + defer up.Close() + if err := provider.Save(provider.Provider{ID: "plan", Name: "Plan", Key: "upstream-secret", Chat: up.URL + "/v1", Models: []string{"m1"}}); err != nil { + t.Fatal(err) + } + keys, secrets := newCaller(t, "Laptop", "Server", "Work") + h := New().Handler() + call := func(secret, header string) int { + r := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(chatReq)) + if header == "Authorization" { + secret = "Bearer " + secret + } + r.Header.Set(header, secret) + r.Header.Set(SessionHeader, "override-session") + r.Header.Set("X-Session-Id", "native-session") + w := httptest.NewRecorder() + h.ServeHTTP(w, r) + return w.Code + } + for i, secret := range secrets { + header := []string{"Authorization", "x-api-key", "x-goog-api-key"}[i] + if code := call(secret, header); code != 200 { + t.Fatal("request", code) + } + } + recs := usage.Load(time.Time{}) + if len(recs) != 3 || recs[0].CallerKeyID != keys[0].ID || recs[1].CallerKeyID != keys[1].ID || recs[2].CallerKeyName != "Work" { + t.Fatalf("attribution: %+v", recs) + } + for _, rec := range recs { + if rec.Input != 30 || rec.Output != 5 || rec.RequestID != "req-caller" || rec.Endpoint != "/v1/chat/completions" || rec.Session != "override-session" || rec.NativeSession != "native-session" { + t.Fatal(rec) + } + } + s := usage.Summarize(usage.All) + if len(s.CallerKeys) != 3 || s.Calls != 3 { + t.Fatalf("summary: %+v", s) + } + rows, totals, _ := usage.Ledger(usage.All, usage.Filter{CallerKey: keys[1].ID}) + if len(rows) != 1 || totals.Input != 30 || rows[0].CallerKeyName != "Server" { + t.Fatal(rows, totals) + } + access.Update("off-key", access.Change{Key: keys[0].ID}) + if code := call(secrets[0], "Authorization"); code != 401 { + t.Fatal("disabled key", code) + } + if code := call(secrets[2], "Authorization"); code != 200 { + t.Fatal("other key", code) + } + access.Update("remove-key", access.Change{Key: keys[2].ID}) + if code := call(secrets[2], "Authorization"); code != 401 { + t.Fatal("deleted key", code) + } + log, _ := os.ReadFile(usage.Path()) + for _, secret := range secrets { + if strings.Contains(string(log), secret) { + t.Fatal("caller credential in ledger") + } + } +} + +func TestRejectedRequestRetainsCallerIdentity(t *testing.T) { + fresh(t) + keys, secrets := newCaller(t, "Rejected client") + r := httptest.NewRequest("POST", "/v1/messages", strings.NewReader(`{"model":"nothing/here","max_tokens":10,"messages":[{"role":"user","content":"hi"}]}`)) + r.Header.Set("Authorization", "Bearer "+secrets[0]) + w := httptest.NewRecorder() + New().Handler().ServeHTTP(w, r) + if w.Code != http.StatusNotFound { + t.Fatal(w.Code, w.Body.String()) + } + rec := lastUsage(t) + if !rec.Rejected || rec.CallerKeyID != keys[0].ID || rec.CallerKeyName != "Rejected client" || rec.Endpoint != "/v1/messages" || rec.Error != "unknown model" || rec.ProviderKeyID != "" { + t.Fatalf("rejected request lost caller or failure metadata: %+v", rec) + } + page := usage.QueryPage(usage.All, usage.Filter{CallerKey: keys[0].ID}, 0, 100) + if page.Total != 1 || page.Sum.Calls != 0 || len(page.CallerKeys) != 0 || usage.Summarize(usage.All).Calls != 0 { + t.Fatalf("local rejection must stay visible without inflating usage: %+v", page) + } + log, err := os.ReadFile(usage.Path()) + if err != nil || strings.Contains(string(log), secrets[0]) { + t.Fatal("caller credential in rejected request log", err) + } +} + +func TestManagedLANGuard(t *testing.T) { + fresh(t) + keys, secrets := newCaller(t, "Remote") + shared := settings.Load() + shared.LAN = true + if err := settings.Save(shared); err != nil { + t.Fatal(err) + } + h := lanGuard(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + who := access.Caller(r.Context()) + if who.KeyID != keys[0].ID || r.URL.Query().Get("key") != Token { + t.Fatal("query key or user not normalized", who) + } + })) + r := httptest.NewRequest("POST", "/v1beta/models/m:generateContent?key="+secrets[0], nil) + r.RemoteAddr = "192.168.1.9:5000" + w := httptest.NewRecorder() + h.ServeHTTP(w, r) + if w.Code != 200 { + t.Fatal(w.Code) + } + access.Update("off-key", access.Change{Key: keys[0].ID}) + w = httptest.NewRecorder() + h.ServeHTTP(w, r) + if w.Code != 401 { + t.Fatal("disabled remote key", w.Code) + } +} + +func TestMigratedLANKeyUsageAndRevocation(t *testing.T) { + fresh(t) + up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer upstream-secret" { + t.Error("caller credential reached provider") + } + w.Header().Set("Content-Type", "application/json") + io.WriteString(w, `{"choices":[{"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}],"usage":{"prompt_tokens":30,"completion_tokens":5}}`) + })) + defer up.Close() + if err := provider.Save(provider.Provider{ID: "plan", Name: "Plan", Key: "upstream-secret", Chat: up.URL + "/v1", Models: []string{"m1"}}); err != nil { + t.Fatal(err) + } + s := settings.Load() + s.LAN, s.LANKey = true, "sk-magpie-test-legacy-lan" + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + if err := access.MigrateLegacyLANKey(); err != nil { + t.Fatal(err) + } + keys, err := access.List() + if err != nil || len(keys) != 1 { + t.Fatal(keys, err) + } + h := lanGuard(New().Handler()) + call := func() int { + r := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(chatReq)) + r.RemoteAddr = "192.168.1.9:5000" + r.Header.Set("Authorization", "Bearer "+s.LANKey) + w := httptest.NewRecorder() + h.ServeHTTP(w, r) + return w.Code + } + if code := call(); code != http.StatusOK { + t.Fatal("migrated remote caller", code) + } + rec := lastUsage(t) + if rec.CallerKeyID != keys[0].ID || rec.CallerKeyName != keys[0].Name || rec.Input != 30 || rec.Output != 5 { + t.Fatal("remote usage attribution", rec) + } + if err := access.ConfigureLAN(true, true); err != nil { + t.Fatal(err) + } + if code := call(); code != http.StatusUnauthorized { + t.Fatal("rotated LAN credential still works", code) + } + s.LANKey, err = access.Update("copy-key", access.Change{Key: keys[0].ID}) + if err != nil { + t.Fatal(err) + } + if code := call(); code != http.StatusOK || lastUsage(t).CallerKeyID != keys[0].ID { + t.Fatal("rotation changed remote usage identity", code) + } + if _, err := access.Update("remove-key", access.Change{Key: keys[0].ID}); err != nil { + t.Fatal(err) + } + if code := call(); code != http.StatusUnauthorized { + t.Fatal("revoked old LAN credential still works", code) + } + if recs := usage.Load(time.Time{}); len(recs) != 2 { + t.Fatal("unauthorized request counted as usage", recs) + } +} + +func TestImageUsageIncludesCaller(t *testing.T) { + s, _ := easeled(t) + keys, secrets := newCaller(t, "Drawing") + r := httptest.NewRequest("POST", "/v1/images/generations", strings.NewReader(`{"model":"art/gpt-image-1","prompt":"a bird"}`)) + r.Header.Set("Authorization", "Bearer "+secrets[0]) + w := httptest.NewRecorder() + s.Handler().ServeHTTP(w, r) + if w.Code != 200 { + t.Fatal(w.Code, w.Body.String()) + } + rec := lastUsage(t) + if rec.CallerKeyID != keys[0].ID || rec.ProviderKeyID != provider.KeyID("key") || rec.Input != 7 { + t.Fatal(rec) + } +} + +func TestCallerIdentitySurvivesStreamingFailover(t *testing.T) { + fresh(t) + up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") == "Bearer first" { + w.WriteHeader(429) + io.WriteString(w, `{"error":{"message":"rate limited"}}`) + return + } + w.Header().Set("Content-Type", "text/event-stream") + io.WriteString(w, "data: "+`{"choices":[{"index":0,"delta":{"content":"ok"},"finish_reason":null}]}`+"\n\n") + io.WriteString(w, "data: "+`{"choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":100,"completion_tokens":20}}`+"\n\ndata: [DONE]\n\n") + })) + defer up.Close() + p := provider.Provider{ID: "plan", Name: "Plan", Key: "first", Chat: up.URL + "/v1", Models: []string{"m1"}, + Keys: []provider.KeyAccount{{Key: "backup", Name: "Backup"}}} + if err := provider.Save(p); err != nil { + t.Fatal(err) + } + keys, secrets := newCaller(t, "Stream") + r := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(`{"model":"plan/m1","messages":[{"role":"user","content":"hi"}],"stream":true}`)) + r.Header.Set("Authorization", "Bearer "+secrets[0]) + w := httptest.NewRecorder() + s := New() + s.Handler().ServeHTTP(w, r) + if w.Code != 200 || !strings.Contains(w.Body.String(), "[DONE]") { + t.Fatal(w.Code, w.Body.String()) + } + rec := lastUsage(t) + if rec.RouteID == 0 || rec.RouteID != lastRoute(s).ID || rec.CallerKeyID != keys[0].ID || rec.CallerKeyName != "Stream" || rec.ProviderKeyID != provider.KeyID("backup") || rec.ProviderKeyName != "Backup" || rec.Input != 100 || rec.Output != 20 { + t.Fatal(rec) + } +} + +func TestCallerUsageWithRequestArchive(t *testing.T) { + for _, header := range []string{"Authorization", "x-api-key", "query"} { + t.Run(header, func(t *testing.T) { + fresh(t) + serveOn(t, "fake", "provider-secret", []string{"m1"}, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer provider-secret" { + t.Error("caller credential reached archived request upstream") + } + archiveVendor{}.ServeHTTP(w, r) + })) + keys, secrets := newCaller(t, "Archived client") + shared := settings.Load() + shared.RequestArchive = true + if err := settings.Save(shared); err != nil { + t.Fatal(err) + } + bucket := &memBucket{objs: map[string][]byte{}} + archiveTo(t, bucket) + r := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(`{"model":"fake/m1","messages":[{"role":"user","content":"hi"}]}`)) + r.RemoteAddr = "192.168.1.9:5000" + switch header { + case "Authorization": + r.Header.Set(header, "Bearer "+secrets[0]) + case "query": + r.URL.RawQuery = "key=" + secrets[0] + default: + r.Header.Set(header, secrets[0]) + } + s := New() + w := httptest.NewRecorder() + lanGuard(s.Handler()).ServeHTTP(w, r) + archivePending.Wait() + if w.Code != 200 || w.Header().Get(ArchiveHeader) == "" || len(bucket.objs) != 1 { + t.Fatal("named caller request was not archived", w.Code, w.Body) + } + for _, data := range bucket.objs { + if strings.Contains(string(data), secrets[0]) || strings.Contains(string(data), "provider-secret") { + t.Fatal("credential leaked into request archive") + } + } + rec := lastUsage(t) + if rec.RouteID == 0 || rec.RouteID != lastRoute(s).ID || rec.CallerKeyID != keys[0].ID || rec.CallerKeyName != "Archived client" || rec.ProviderKeyID != provider.KeyID("provider-secret") || rec.Input != 3 { + t.Fatal("archive lost usage attribution", rec) + } + }) + } +} diff --git a/internal/gateway/caller_video_usage_test.go b/internal/gateway/caller_video_usage_test.go new file mode 100644 index 00000000..9b93a5fe --- /dev/null +++ b/internal/gateway/caller_video_usage_test.go @@ -0,0 +1,57 @@ +package gateway + +import ( + "net/http/httptest" + "os" + "strings" + "testing" + + "github.com/yetone/magpie/internal/usage" +) + +func TestCallerUsageVideos(t *testing.T) { + for _, tc := range []struct { + prompt string + status int + }{ + {"a fox", 200}, + {"TOOLONG", 400}, + {"NOID", 502}, + } { + for _, header := range []string{"Authorization", "x-api-key", "query"} { + t.Run(tc.prompt+"/"+header, func(t *testing.T) { + grokSignedIn(t) + newGrokMedia(t) + keys, secrets := newCaller(t, "Video client") + r := httptest.NewRequest("POST", "/v1/videos", strings.NewReader(`{"model":"grok/grok-imagine-video","prompt":"`+tc.prompt+`"}`)) + r.RemoteAddr = "192.168.1.9:5000" + r.Header.Set("Content-Type", "application/json") + r.Header.Set("X-Magpie-Session", "video-session") + if header == "query" { + r.URL.RawQuery = "key=" + secrets[0] + } else if header == "Authorization" { + r.Header.Set(header, "Bearer "+secrets[0]) + } else { + r.Header.Set(header, secrets[0]) + } + w := httptest.NewRecorder() + lanGuard(New().Handler()).ServeHTTP(w, r) + if w.Code != tc.status { + t.Fatal(w.Code, w.Body) + } + rec := lastUsage(t) + if rec.CallerKeyID != keys[0].ID || rec.CallerKeyName != "Video client" || rec.Provider != "grok" || rec.Model != "grok-imagine-video" || rec.Status != tc.status || rec.Session != "video-session" { + t.Fatalf("video caller attribution: %+v", rec) + } + rows, _, _ := usage.Ledger(usage.All, usage.Filter{CallerKey: keys[0].ID}) + if len(rows) != 1 { + t.Fatal("video missing from caller-key ledger", rows) + } + log, err := os.ReadFile(usage.Path()) + if err != nil || strings.Contains(string(log), secrets[0]) { + t.Fatal("video usage leaked the caller credential", err) + } + }) + } + } +} diff --git a/internal/gateway/codex_backend.go b/internal/gateway/codex_backend.go index 431ebdee..cb166d53 100644 --- a/internal/gateway/codex_backend.go +++ b/internal/gateway/codex_backend.go @@ -403,7 +403,7 @@ func (s *Server) codexUpstream(w http.ResponseWriter, r *http.Request, rest stri Reasoning: uu.Reasoning, Millis: call.Millis, TTFT: call.TTFT, FirstText: call.FirstText, Status: call.Status, Session: sessionOf(r.Header), NativeSession: nativeSessionOf(r.Header), Kind: call.Kind, RequestID: requestID(res.Header), Endpoint: r.URL.Path} failedWith(&rec, call.Status, call.Error, errType) - usage.Append(rec) + appendUsage(r, rec) } // unreadableItem is the item OpenAI's refusal names: sealed content it diff --git a/internal/gateway/decide.go b/internal/gateway/decide.go index 54702ee1..cf149d32 100644 --- a/internal/gateway/decide.go +++ b/internal/gateway/decide.go @@ -465,7 +465,7 @@ func (s *Server) serveSystemOne(w http.ResponseWriter, r *http.Request) { if p.Account == nil && p.Key != "" { keyID, keyName = provider.KeyID(p.Key), p.KeyName } - usage.Append(usage.Record{RouteID: tr.ID, Time: start, Agent: agentOf(r), Provider: p.ID, Host: p.Where(), Model: model, Requested: asked, Served: use.Model, + appendUsage(r, usage.Record{RouteID: tr.ID, Time: start, Agent: agentOf(r), Provider: p.ID, Host: p.Where(), Model: model, Requested: asked, Served: use.Model, ProviderKeyID: keyID, ProviderKeyName: keyName, Input: use.Usage.Input, Output: use.Usage.Output, Millis: time.Since(start).Milliseconds(), Status: status}) end(status, errMsg, tokens) diff --git a/internal/gateway/draw.go b/internal/gateway/draw.go index 0659cd6b..aeab6e86 100644 --- a/internal/gateway/draw.go +++ b/internal/gateway/draw.go @@ -243,7 +243,7 @@ func (s *Server) images(edit bool) http.HandlerFunc { if p.Account == nil && p.Key != "" { providerKeyID, providerKeyName = provider.KeyID(p.Key), p.KeyName } - usage.Append(usage.Record{Time: start, Agent: call.Agent, Via: call.Via, Provider: p.ID, Host: p.Where(), Model: model, Requested: call.Model, ProviderKeyID: providerKeyID, ProviderKeyName: providerKeyName, + appendUsage(r, usage.Record{Time: start, Agent: call.Agent, Via: call.Via, Provider: p.ID, Host: p.Where(), Model: model, Requested: call.Model, ProviderKeyID: providerKeyID, ProviderKeyName: providerKeyName, Input: out.Input, Output: out.Output, Millis: call.Millis, Status: call.Status, Session: sessionOf(r.Header)}) if err != nil { call.Error = err.Error() diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go index 5a25c773..54adfe46 100644 --- a/internal/gateway/gateway.go +++ b/internal/gateway/gateway.go @@ -294,7 +294,7 @@ var WhileServing []func(context.Context) // long as a stream takes. A bind error means another magpie is already // serving, which is fine for the caller to ignore. func (s *Server) ListenAndServe(ctx context.Context) error { - loadLANKey() + migrateLANKeyBestEffort() ln, err := Listen(listenAddr()) if err != nil { return err @@ -392,7 +392,7 @@ func (s *Server) Handler() http.Handler { mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { writeError(w, provider.Chat, http.StatusNotFound, "magpie serves /v1/chat/completions, /v1/responses, /v1/messages, /v1/systemone, /v1/images/generations, /v1/images/edits, /v1/videos and /v1beta/models/*") }) - return withCaller(mux) + return callerGuard(withCaller(mux)) } func (s *Server) info(w http.ResponseWriter, r *http.Request) { @@ -818,7 +818,7 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro rec := usage.Record{Time: start, Agent: call.Agent, Via: call.Via, Provider: call.Provider, Model: call.Model, Requested: call.Model, Millis: call.Millis, Status: call.Status, Rejected: true, Session: sessionOf(r.Header), NativeSession: nativeSessionOf(r.Header), Kind: call.Kind, Endpoint: endpointOf(r, from, "")} failedWith(&rec, call.Status, call.Error, "") - usage.Append(rec) + appendUsage(r, rec) } // a model's id without a provider in it that names a routing group is // the group's, as "group/" is, rather than one provider's that @@ -1225,7 +1225,7 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro TTFT: try.TTFT, FirstText: try.FirstText, Session: sessionOf(r.Header), NativeSession: nativeSessionOf(r.Header), Kind: call.Kind, RequestID: call.Usage.RequestID, Endpoint: endpointOf(r, from, call.To)} failedWith(&rec, call.Status, call.Error, call.Usage.ErrType) - usage.Append(rec) + appendUsage(r, rec) } continue } @@ -1383,7 +1383,7 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro TTFT: call.TTFT, FirstText: call.FirstText, Session: sessionOf(r.Header), NativeSession: nativeSessionOf(r.Header), Kind: call.Kind, RequestID: call.Usage.RequestID, Endpoint: endpointOf(r, from, call.To)} failedWith(&rec, call.Status, call.Error, call.Usage.ErrType) - usage.Append(rec) + appendUsage(r, rec) } } diff --git a/internal/gateway/lan.go b/internal/gateway/lan.go index 4d9fecdb..aa27df6f 100644 --- a/internal/gateway/lan.go +++ b/internal/gateway/lan.go @@ -2,34 +2,26 @@ package gateway import ( "context" - "crypto/rand" - "crypto/subtle" - "encoding/hex" "net" "net/http" "net/url" "os" "path/filepath" "strings" - "sync/atomic" + "github.com/yetone/magpie/internal/access" + "github.com/yetone/magpie/internal/provider" "github.com/yetone/magpie/internal/settings" + "github.com/yetone/magpie/internal/usage" ) -// The gateway listens on loopback and takes any token, which is safe only -// because nothing but this computer reaches it. Shared on the local network -// (settings' LAN), it listens on every interface, and a request from -// another machine must carry the key magpie made for it, as its API key; -// this computer's agents go on as before. - -// lanKey is the key a request from another machine must carry, "" while -// the gateway isn't shared. -var lanKey atomic.Pointer[string] +// The gateway accepts named caller keys locally and, while shared, remotely. +// Loopback clients may still use any token, including a stale named key. // listenAddr is where the gateway listens: every interface while it is // shared, on its port, else its address. func listenAddr() string { - if s := settings.Load(); s.LAN && s.LANKey != "" { + if s := settings.Load(); s.LAN { return "0.0.0.0:" + Port() } return Addr() @@ -44,19 +36,10 @@ func Port() string { return p } -func loadLANKey() { - k := "" - if s := settings.Load(); s.LAN { - k = s.LANKey - } - lanKey.Store(&k) -} +func migrateLANKey() error { return access.MigrateLegacyLANKey() } -// NewLANKey makes a key for sharing the gateway. -func NewLANKey() string { - b := make([]byte, 20) - rand.Read(b) - return "sk-magpie-" + hex.EncodeToString(b) +func migrateLANKeyBestEffort() { + access.MigrateLegacyLANKeyBestEffort() } // publicURL is MAGPIE_PUBLIC_URL, the base URL other machines are told to @@ -130,10 +113,9 @@ func LANURLs() []string { } // Relisten moves the gateway to where settings now say it listens — onto -// the network or back to loopback — and takes up the key; requests in -// flight finish. +// the network or back to loopback. Requests in flight finish. func (s *Server) Relisten() error { - loadLANKey() + migrateLANKeyBestEffort() s.lnMu.Lock() defer s.lnMu.Unlock() if s.ln == nil { @@ -158,50 +140,75 @@ func (s *Server) Relisten() error { return nil } -// lanGuard lets a request from another machine through only with the key, -// and hands it on as though it came from here, with the gateway's token. -// A gateway put on the network with MAGPIE_ADDR and not shared from the -// Settings page is open, as it has always been. +// lanGuard requires a named caller key on the local network. An explicit +// MAGPIE_ADDR without LAN sharing retains its existing open-gateway behavior. func lanGuard(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if local(r) { - next.ServeHTTP(w, r) - return - } - key := "" - if k := lanKey.Load(); k != nil { - key = *k - } - if key == "" { - if os.Getenv("MAGPIE_ADDR") != "" { - next.ServeHTTP(w, r) - return - } + remote := !local(r) + shared := settings.Load().LAN + if remote && !shared && os.Getenv("MAGPIE_ADDR") == "" { http.Error(w, "magpie isn't shared on the local network", http.StatusForbidden) return } - if subtle.ConstantTimeCompare([]byte(callerKey(r)), []byte(key)) != 1 { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusUnauthorized) - w.Write([]byte(`{"error":{"type":"authentication_error","message":"use the API key shown in magpie's Settings, under Share on local network"}}`)) - return - } - r = r.WithContext(context.WithValue(r.Context(), lanKeyed{}, true)) - r.Header.Set("Authorization", "Bearer "+Token) - if r.Header.Get("x-api-key") != "" { - r.Header.Set("x-api-key", Token) - } - if r.Header.Get("x-goog-api-key") != "" { - r.Header.Set("x-goog-api-key", Token) - } - if q := r.URL.Query(); q.Get("key") != "" { - q.Set("key", Token) - r.URL.RawQuery = q.Encode() + if (remote && shared) || (!remote && access.Managed(callerKey(r))) { + var ok bool + r, ok = identifyCaller(w, r) + if !ok { + return + } + if remote && shared { + r = r.WithContext(context.WithValue(r.Context(), lanKeyed{}, true)) + } } next.ServeHTTP(w, r) }) } +// callerGuard also covers embedded handlers used by the web app and tests. +func callerGuard(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if access.Caller(r.Context()).KeyID == "" && access.Managed(callerKey(r)) && (local(r) || settings.Load().LAN) { + var ok bool + r, ok = identifyCaller(w, r) + if !ok { + return + } + } + next.ServeHTTP(w, r) + }) +} + +func identifyCaller(w http.ResponseWriter, r *http.Request) (*http.Request, bool) { + who, ok := access.Authenticate(callerKey(r)) + if !ok && !local(r) { + writeError(w, provider.Chat, http.StatusUnauthorized, "API key is disabled, removed or invalid") + return r, false + } + if ok { + r = r.WithContext(access.WithIdentity(r.Context(), who)) + } + r.Header = r.Header.Clone() + r.Header.Set("Authorization", "Bearer "+Token) + for _, h := range []string{"x-api-key", "x-goog-api-key"} { + if r.Header.Get(h) != "" { + r.Header.Set(h, Token) + } + } + if q := r.URL.Query(); q.Get("key") != "" { + q.Set("key", Token) + u := *r.URL + u.RawQuery = q.Encode() + r.URL = &u + } + return r, true +} + +func appendUsage(r *http.Request, rec usage.Record) { + who := access.Caller(r.Context()) + rec.CallerKeyID, rec.CallerKeyName = who.KeyID, who.KeyName + usage.Append(rec) +} + // lanKeyed marks a request from another machine that carried the key. type lanKeyed struct{} diff --git a/internal/gateway/lan_test.go b/internal/gateway/lan_test.go index e7a76870..dbdcf0a3 100644 --- a/internal/gateway/lan_test.go +++ b/internal/gateway/lan_test.go @@ -6,17 +6,22 @@ import ( "os" "path/filepath" "testing" + + "github.com/yetone/magpie/internal/access" + "github.com/yetone/magpie/internal/settings" ) -// Shared on the network, a request from another machine needs the key, and -// goes on with the gateway's own token; this computer's need none. +// Sharing gates remote access; each authorized remote call carries the same +// named identity as its local counterpart, so usage is attributed by key. func TestLANGuard(t *testing.T) { - var got string + fresh(t) + var got, id string h := lanGuard(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { got = r.Header.Get("Authorization") + "|" + r.Header.Get("x-api-key") + id = access.Caller(r.Context()).KeyID })) call := func(from string, hdr ...string) int { - got = "" + got, id = "", "" r := httptest.NewRequest("POST", "/v1/messages", nil) r.RemoteAddr = from for i := 0; i+1 < len(hdr); i += 2 { @@ -26,35 +31,52 @@ func TestLANGuard(t *testing.T) { h.ServeHTTP(w, r) return w.Code } - key := "sk-magpie-k" - none := "" - t.Cleanup(func() { lanKey.Store(&none) }) - - lanKey.Store(&none) + secret, err := access.Update("add-key", access.Change{Name: "Laptop"}) + if err != nil { + t.Fatal(err) + } + keys, _ := access.List() + keyID := keys[0].ID t.Setenv("MAGPIE_ADDR", "") - if c := call("192.168.1.9:5000"); c != http.StatusForbidden { - t.Fatal("unshared, another machine got", c) + if c := call("192.168.1.9:5000", "x-api-key", secret); c != http.StatusForbidden { + t.Fatal("unshared remote", c) } - if c := call("127.0.0.1:5000"); c != 200 { - t.Fatal("loopback got", c) + if c := call("127.0.0.1:5000", "Authorization", "Bearer anything"); c != 200 { + t.Fatal("loopback", c) } - t.Setenv("MAGPIE_ADDR", "0.0.0.0:3425") + s := settings.Load() + s.LAN = true + if err := settings.Save(s); err != nil { + t.Fatal(err) + } + if addr := listenAddr(); addr != "0.0.0.0:"+Port() { + t.Fatal(addr) + } + for _, hdr := range [][]string{nil, {"Authorization", "Bearer magpie"}, {"x-api-key", "wrong"}} { + if c := call("192.168.1.9:5000", hdr...); c != http.StatusUnauthorized { + t.Fatal("unauthorized remote", hdr, c) + } + } + if c := call("192.168.1.9:5000", "x-api-key", secret); c != 200 || id != keyID || got != "Bearer magpie|magpie" { + t.Fatal(c, id, got) + } + if c := call("127.0.0.1:5000", "Authorization", "Bearer "+secret); c != 200 || id != keyID { + t.Fatal("local named key", c, id) + } + access.Update("off-key", access.Change{Key: keyID}) + if c := call("192.168.1.9:5000", "x-api-key", secret); c != http.StatusUnauthorized { + t.Fatal("disabled key", c) + } + access.Update("on-key", access.Change{Key: keyID}) + access.Update("remove-key", access.Change{Key: keyID}) + if c := call("192.168.1.9:5000", "x-api-key", secret); c != http.StatusUnauthorized { + t.Fatal("removed key", c) + } + s.LAN = false + settings.Save(s) + t.Setenv("MAGPIE_ADDR", "0.0.0.0:3499") if c := call("192.168.1.9:5000"); c != 200 { - t.Fatal("MAGPIE_ADDR's open gateway got", c) - } - - lanKey.Store(&key) - if c := call("192.168.1.9:5000"); c != http.StatusUnauthorized { - t.Fatal("no key got", c) - } - if c := call("192.168.1.9:5000", "Authorization", "Bearer wrong"); c != http.StatusUnauthorized { - t.Fatal("a wrong key got", c) - } - if c := call("192.168.1.9:5000", "x-api-key", key); c != 200 || got != "Bearer magpie|magpie" { - t.Fatal(c, got) - } - if c := call("[::1]:5000", "Authorization", "Bearer anything"); c != 200 || got != "Bearer anything|" { - t.Fatal("loopback", c, got) + t.Fatal("explicit open gateway", c) } } @@ -70,8 +92,8 @@ func TestURLOfWildcard(t *testing.T) { } } -// An agent whose User-Agent is only the AI SDK's (Alma) is known by the -// token it was given; the others still by their User-Agent. +// An agent whose User-Agent is only the AI SDK's (Alma) is known by +// the token it was given; the others still by their User-Agent. func TestAgentOf(t *testing.T) { for _, c := range []struct{ auth, key, ua, want string }{ {"Bearer " + TokenFor("alma"), "", "ai-sdk/openai/2.0.52 ai-sdk/provider-utils/3.0.12 runtime/node.js/v22", "alma"}, diff --git a/internal/gateway/quotas_lan_test.go b/internal/gateway/quotas_lan_test.go index 22dcf7e1..c7f44485 100644 --- a/internal/gateway/quotas_lan_test.go +++ b/internal/gateway/quotas_lan_test.go @@ -5,6 +5,9 @@ import ( "net/http/httptest" "strings" "testing" + + "github.com/yetone/magpie/internal/access" + "github.com/yetone/magpie/internal/settings" ) // GET /v1/magpie/quotas answers another machine with the key of the @@ -12,7 +15,7 @@ import ( // a wrong key, none, or a gateway not shared still get nothing, and the // refusal says how. func TestQuotasOverLAN(t *testing.T) { - setHome(t, t.TempDir()) + fresh(t) t.Setenv("MAGPIE_ADDR", "") h := lanGuard(New().Handler()) call := func(from string, hdr ...string) (int, string) { @@ -25,11 +28,8 @@ func TestQuotasOverLAN(t *testing.T) { h.ServeHTTP(w, r) return w.Code, w.Body.String() } - key := "sk-magpie-k" - none := "" - t.Cleanup(func() { lanKey.Store(&none) }) - - lanKey.Store(&key) + keys, secrets := newCaller(t, "Quota client") + key := secrets[0] if c, b := call("100.64.0.7:5000", "Authorization", "Bearer "+key); c != 200 || !strings.Contains(b, `"object":"list"`) { t.Fatal("shared, with the key:", c, b) } @@ -46,7 +46,17 @@ func TestQuotasOverLAN(t *testing.T) { t.Fatal("loopback got", c) } - lanKey.Store(&none) + if _, err := access.Update("off-key", access.Change{Key: keys[0].ID}); err != nil { + t.Fatal(err) + } + if c, _ := call("192.168.1.9:5000", "Authorization", "Bearer "+key); c != http.StatusUnauthorized { + t.Fatal("disabled key got", c) + } + shared := settings.Load() + shared.LAN = false + if err := settings.Save(shared); err != nil { + t.Fatal(err) + } if c, _ := call("192.168.1.9:5000", "Authorization", "Bearer "+key); c != http.StatusForbidden { t.Fatal("not shared got", c) } diff --git a/internal/gateway/session_route_test.go b/internal/gateway/session_route_test.go index a26923c2..58525f8b 100644 --- a/internal/gateway/session_route_test.go +++ b/internal/gateway/session_route_test.go @@ -11,7 +11,9 @@ import ( "testing" "time" + "github.com/yetone/magpie/internal/access" "github.com/yetone/magpie/internal/provider" + "github.com/yetone/magpie/internal/settings" ) // GET /v1/magpie/route tells an agent's UI where its session's turn went @@ -132,16 +134,34 @@ func TestSessionRoute(t *testing.T) { t.Fatal("no session got", code) } // another machine needs the shared gateway's key, as the quotas do - key, none := "sk-magpie-k", "" - t.Cleanup(func() { lanKey.Store(&none) }) - lanKey.Store(&key) + callerKeys, secrets := newCaller(t, "Session status") + key := secrets[0] if code, _ := get("192.168.1.9:5000", "/v1/magpie/route?session=pi-1"); code != http.StatusUnauthorized { t.Fatal("no key got", code) } if code, out := get("192.168.1.9:5000", "/v1/magpie/route?session=pi-1", "Authorization", "Bearer "+key); code != 200 || out["route"] == nil { t.Fatalf("with the key: %d %v", code, out) } - lanKey.Store(&none) + if code, out := get("192.168.1.9:5000", "/v1/magpie/route?session=pi-1", "x-api-key", key); code != 200 || out["route"] == nil { + t.Fatalf("with x-api-key: %d %v", code, out) + } + if code, out := get("192.168.1.9:5000", "/v1/magpie/route?session=pi-1&key="+key); code != 200 || out["route"] == nil { + t.Fatalf("with query key: %d %v", code, out) + } + if code, _ := get("192.168.1.9:5000", "/v1/magpie/route?session=pi-1", "Authorization", "Bearer wrong"); code != http.StatusUnauthorized { + t.Fatal("wrong key got", code) + } + if _, err := access.Update("off-key", access.Change{Key: callerKeys[0].ID}); err != nil { + t.Fatal(err) + } + if code, _ := get("192.168.1.9:5000", "/v1/magpie/route?session=pi-1", "Authorization", "Bearer "+key); code != http.StatusUnauthorized { + t.Fatal("disabled key got", code) + } + shared := settings.Load() + shared.LAN = false + if err := settings.Save(shared); err != nil { + t.Fatal(err) + } t.Setenv("MAGPIE_ADDR", "0.0.0.0:3425") if code, _ := get("192.168.1.9:5000", "/v1/magpie/route?session=pi-1"); code != http.StatusForbidden { t.Fatal("MAGPIE_ADDR, not shared, got", code) diff --git a/internal/gateway/video.go b/internal/gateway/video.go index 4948e926..0d808cbe 100644 --- a/internal/gateway/video.go +++ b/internal/gateway/video.go @@ -461,7 +461,7 @@ func (s *Server) videosCreate(w http.ResponseWriter, r *http.Request) { call.Status = code } } - usage.Append(usage.Record{Time: start, Agent: call.Agent, Via: call.Via, Provider: p.ID, Host: p.Where(), Model: model, Requested: call.Model, + appendUsage(r, usage.Record{Time: start, Agent: call.Agent, Via: call.Via, Provider: p.ID, Host: p.Where(), Model: model, Requested: call.Model, Millis: call.Millis, Status: call.Status, Session: sessionOf(r.Header)}) if err != nil { call.Error = err.Error() diff --git a/internal/gui/api.go b/internal/gui/api.go index fefe79a2..2b831146 100644 --- a/internal/gui/api.go +++ b/internal/gui/api.go @@ -20,6 +20,7 @@ import ( "strings" "time" + "github.com/yetone/magpie/internal/access" "github.com/yetone/magpie/internal/agent" "github.com/yetone/magpie/internal/autostart" "github.com/yetone/magpie/internal/catalog" @@ -269,6 +270,7 @@ func searchState(s *settingsJSON) { func settingsState() settingsJSON { s := settingsJSON{Settings: settings.Load(), Version: Version, Dir: tilde(settings.Dir()), Gateway: gateway.URL()} + s.LANKey = "" // the retained credential belongs on disk, not in UI state if found, err := discoverTerminals(); err == nil { for _, app := range found.Apps { s.TerminalApps = append(s.TerminalApps, terminalChoice{ID: app.ID, Name: app.Name}) @@ -546,12 +548,14 @@ func Handler(w Windows, gw *gateway.Server) http.Handler { providerRoutes(mux, w) importRoutes(mux) usageRoutes(mux, w) + callerKeyRoutes(mux) sessionRoutes(mux, w) backupRoutes(mux, w) archiveRoutes(mux) libraryRoutes(mux, w) updateRoutes(mux, w) mux.HandleFunc("GET /api/settings", func(rw http.ResponseWriter, r *http.Request) { + access.MigrateLegacyLANKeyBestEffort() writeJSON(rw, settingsState()) }) mux.HandleFunc("POST /api/settings", func(rw http.ResponseWriter, r *http.Request) { @@ -576,6 +580,7 @@ func Handler(w Windows, gw *gateway.Server) http.Handler { in.Visible, in.HiddenModels = cur.Visible, cur.HiddenModels settings.CarryPerModel(&in, &cur) in.LAN, in.LANKey = cur.LAN, cur.LANKey + in.LANKeyID = cur.LANKeyID in.RequestArchive = cur.RequestArchive // the Gateway page's, set on its own in.RedactRules = cur.RedactRules // the masking rules, set on their own // used or left is the Usage page's toggle as much as Settings', set on its own @@ -750,20 +755,15 @@ func Handler(w Windows, gw *gateway.Server) http.Handler { } writeJSON(rw, settingsState()) }) - // sharing the gateway on the local network: on makes its key the first - // time, new asks for another (the old one stops working) + // Sharing controls exposure; the gateway's named caller keys authenticate + // remote clients just as they do local ones. mux.HandleFunc("POST /api/settings/lan", func(rw http.ResponseWriter, r *http.Request) { var in struct{ On, NewKey bool } if err := json.NewDecoder(r.Body).Decode(&in); err != nil { fail(rw, err) return } - s := settings.Load() - s.LAN = in.On - if s.LANKey == "" || in.NewKey { - s.LANKey = gateway.NewLANKey() - } - if err := settings.Save(s); err != nil { + if err := access.ConfigureLAN(in.On, in.NewKey); err != nil { fail(rw, err) return } diff --git a/internal/gui/assets/app.css b/internal/gui/assets/app.css index ebca2a39..2758347d 100644 --- a/internal/gui/assets/app.css +++ b/internal/gui/assets/app.css @@ -1334,8 +1334,12 @@ body.window .ag-sk-icon { width: 22px; height: 22px; border-radius: 6px; } .connect .hint { font-size: 11px; color: var(--faint); margin-top: 4px; line-height: 1.4; } .connect .val { display: inline-flex; align-items: center; gap: 4px; height: 26px; padding: 0 4px 0 9px; border-radius: 7px; background: var(--pill); box-shadow: inset 0 0 0 1px var(--line-2); max-width: 100%; } .connect .val code { font-size: 12px; color: var(--fg); user-select: text; -webkit-user-select: text; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } +.connect .val > .val { min-width: 0; height: auto; padding: 0; background: none; box-shadow: none; } .connect .segs { margin-top: -1px; } .connect .stack { display: flex; flex-direction: column; gap: 8px; min-width: 0; } +@media (max-width: 520px) { + .connect { grid-template-columns: 48px minmax(0, 1fr); column-gap: 8px; padding-left: 10px; padding-right: 10px; } +} .copy { width: 22px; height: 22px; border-radius: 5px; display: inline-flex; align-items: center; justify-content: center; color: var(--muted); flex: none; transition: background .12s, color .12s; } .copy:hover { background: var(--pill-hover); color: var(--fg); } .copy.done { color: var(--green); background: var(--green-soft); animation: copy-pop .36s var(--ease); } @@ -1494,6 +1498,8 @@ body.window .ag-sk-icon { width: 22px; height: 22px; border-radius: 6px; } .acc .proto:hover, .acc .proto.open { color: var(--fg); background: var(--pill-hover); } .acc .n, .acc .rename { flex: 0 1 auto; } .acc .plan { flex: none; } +#gatewayKeys .acc { min-width: 0; } +#gatewayKeys .acc .plan { flex: 0 1 auto; min-width: 0; overflow: hidden; text-overflow: ellipsis; } /* the Settings row's picker: the key row's badge, always shown */ .row.pref .proto.pick { flex: none; margin-left: auto; display: inline-flex; align-items: center; gap: 3px; height: 24px; padding: 0 6px 0 9px; font: inherit; font-size: 12px; color: var(--fg-2); border: 0; border-radius: 7px; background: var(--pill); white-space: nowrap; cursor: default; transition: background .12s, color .12s; } .row.pref .proto.pick svg { opacity: .7; transition: transform .15s var(--ease); } @@ -1656,14 +1662,14 @@ body.window .ag-sk-icon { width: 22px; height: 22px; border-radius: 6px; } .stepplan .steps { margin: 0; padding-left: 18px; display: flex; flex-direction: column; gap: 8px; font-size: 12px; color: var(--muted); line-height: 1.5; } .stepplan .steps .pair { margin-top: 2px; } .stepplan .bookmarklet { display: inline-block; padding: 1px 8px; border: 1px solid var(--line); border-radius: 6px; color: var(--fg); background: var(--pill); text-decoration: none; cursor: grab; white-space: nowrap; } -.editor input[type="text"], .editor input[type="password"], .editor input[type="url"] { +.editor input[type="text"], .editor input[type="password"], .editor input[type="url"], .acc .kf input, .acc input.rename-in { height: 27px; padding: 0 9px; border-radius: 7px; border: 1px solid var(--line); background: var(--card); font-size: 12.5px; outline: none; width: 100%; min-width: 0; transition: border-color .12s, box-shadow .12s; } -.editor input:focus, .editor .headers textarea.json:focus { border-color: var(--accent); box-shadow: 0 0 0 3px color-mix(in srgb, var(--accent) 22%, transparent); outline: none; } -.editor input::placeholder { color: var(--faint); } +.editor input:focus, .editor .headers textarea.json:focus, .acc input:focus { border-color: var(--accent); box-shadow: 0 0 0 3px color-mix(in srgb, var(--accent) 22%, transparent); outline: none; } +.editor input::placeholder, .acc input::placeholder { color: var(--faint); } .editor .pair { display: flex; gap: 8px; align-items: center; } .editor .pair input:first-child { flex: 1; } .editor .side { display: flex; gap: 4px; align-items: center; flex: none; min-width: 52px; justify-content: flex-end; } diff --git a/internal/gui/assets/app.js b/internal/gui/assets/app.js index e12e1d89..cec722f7 100644 --- a/internal/gui/assets/app.js +++ b/internal/gui/assets/app.js @@ -2493,13 +2493,13 @@ function askForgetSaved(x) { return; } } - closeResetAsk(); + closeConfirmAsk(); }; const cancel = el("button", "text", t("Cancel")); - cancel.onclick = (e) => { e.stopPropagation(); closeResetAsk(); }; + cancel.onclick = (e) => { e.stopPropagation(); closeConfirmAsk(); }; bar.append(el("span", "grow"), cancel, go); ed.append(bar); - resetAsk = ed; + confirmAsk = ed; openModal(ed); $("#modal").classList.add("lib"); cancel.focus(); @@ -2628,6 +2628,9 @@ function renderGatewayView() { page.classList.remove("loading"); page.removeAttribute("aria-busy"); renderGateway(); + $("#gatewayKeysBlock").hidden = !providers.gateway.lan; + if (!providers.gateway.lan) gatewayKeyDraft = null; + if (gatewayKeyDraft === null && !$("#gatewayKeys .rename-in")) renderGatewayKeys(); renderConnect(); renderGatewayModels(); renderArchive(); @@ -2662,19 +2665,19 @@ const FLAVORS = { openai: { name: "OpenAI", base: (u) => u + "/v1", baseEnv: "OPENAI_BASE_URL", keyEnv: "OPENAI_API_KEY", note: "Chat Completions, the API most tools speak. Anything with an OpenAI base-URL setting works.", - curl: (b, m) => ({ url: `${b}/chat/completions`, headers: ["Authorization: Bearer magpie"], + curl: (b, m, k = "magpie") => ({ url: `${b}/chat/completions`, headers: [`Authorization: Bearer ${k}`], body: `{"model": "${m}",\n "messages": [{"role": "user", "content": "hi"}]}` }), - python: (b, m) => `from openai import OpenAI + python: (b, m, k = "magpie") => `from openai import OpenAI -client = OpenAI(base_url="${b}", api_key="magpie") +client = OpenAI(base_url="${b}", api_key="${k}") r = client.chat.completions.create( model="${m}", messages=[{"role": "user", "content": "hi"}], ) print(r.choices[0].message.content)`, - node: (b, m) => `import OpenAI from "openai"; + node: (b, m, k = "magpie") => `import OpenAI from "openai"; -const client = new OpenAI({ baseURL: "${b}", apiKey: "magpie" }); +const client = new OpenAI({ baseURL: "${b}", apiKey: "${k}" }); const r = await client.chat.completions.create({ model: "${m}", messages: [{ role: "user", content: "hi" }], @@ -2684,28 +2687,28 @@ console.log(r.choices[0].message.content);`, responses: { name: "Responses", base: (u) => u + "/v1", baseEnv: "OPENAI_BASE_URL", keyEnv: "OPENAI_API_KEY", note: "OpenAI's newer API: reasoning, built-in tool items, encrypted reasoning. Codex speaks this.", - curl: (b, m) => ({ url: `${b}/responses`, headers: ["Authorization: Bearer magpie"], + curl: (b, m, k = "magpie") => ({ url: `${b}/responses`, headers: [`Authorization: Bearer ${k}`], body: `{"model": "${m}", "input": "hi"}` }), - python: (b, m) => `from openai import OpenAI + python: (b, m, k = "magpie") => `from openai import OpenAI -client = OpenAI(base_url="${b}", api_key="magpie") +client = OpenAI(base_url="${b}", api_key="${k}") r = client.responses.create(model="${m}", input="hi") print(r.output_text)`, - node: (b, m) => `import OpenAI from "openai"; + node: (b, m, k = "magpie") => `import OpenAI from "openai"; -const client = new OpenAI({ baseURL: "${b}", apiKey: "magpie" }); +const client = new OpenAI({ baseURL: "${b}", apiKey: "${k}" }); const r = await client.responses.create({ model: "${m}", input: "hi" }); console.log(r.output_text);`, }, anthropic: { name: "Anthropic", base: (u) => u, baseEnv: "ANTHROPIC_BASE_URL", keyEnv: "ANTHROPIC_API_KEY", note: "Messages API. Claude Code reads ANTHROPIC_AUTH_TOKEN instead of the key; the Agents tab sets that for you.", - curl: (b, m) => ({ url: `${b}/v1/messages`, headers: ["x-api-key: magpie", "anthropic-version: 2023-06-01"], + curl: (b, m, k = "magpie") => ({ url: `${b}/v1/messages`, headers: [`x-api-key: ${k}`, "anthropic-version: 2023-06-01"], body: `{"model": "${m}", "max_tokens": 1024,\n "messages": [{"role": "user", "content": "hi"}]}` }), - python: (b, m) => `import anthropic + python: (b, m, k = "magpie") => `import anthropic client = anthropic.Anthropic( - base_url="${b}", api_key="magpie", + base_url="${b}", api_key="${k}", ) m = client.messages.create( model="${m}", @@ -2713,9 +2716,9 @@ m = client.messages.create( messages=[{"role": "user", "content": "hi"}], ) print(m.content[0].text)`, - node: (b, m) => `import Anthropic from "@anthropic-ai/sdk"; + node: (b, m, k = "magpie") => `import Anthropic from "@anthropic-ai/sdk"; -const client = new Anthropic({ baseURL: "${b}", apiKey: "magpie" }); +const client = new Anthropic({ baseURL: "${b}", apiKey: "${k}" }); const m = await client.messages.create({ model: "${m}", max_tokens: 1024, @@ -2726,17 +2729,17 @@ console.log(m.content[0].text);`, gemini: { name: "Gemini", base: (u) => u, baseEnv: "GOOGLE_GEMINI_BASE_URL", keyEnv: "GEMINI_API_KEY", note: "Google's generateContent API, v1beta. Gemini CLI and the google-genai SDKs speak this.", - curl: (b, m) => ({ url: `${b}/v1beta/models/${m}:generateContent`, headers: ["x-goog-api-key: magpie"], + curl: (b, m, k = "magpie") => ({ url: `${b}/v1beta/models/${m}:generateContent`, headers: [`x-goog-api-key: ${k}`], body: `{"contents": [{"parts": [{"text": "hi"}]}]}` }), - python: (b, m) => `from google import genai + python: (b, m, k = "magpie") => `from google import genai -client = genai.Client(api_key="magpie", http_options={"base_url": "${b}"}) +client = genai.Client(api_key="${k}", http_options={"base_url": "${b}"}) r = client.models.generate_content(model="${m}", contents="hi") print(r.text)`, - node: (b, m) => `import { GoogleGenAI } from "@google/genai"; + node: (b, m, k = "magpie") => `import { GoogleGenAI } from "@google/genai"; const ai = new GoogleGenAI({ - apiKey: "magpie", + apiKey: "${k}", httpOptions: { baseUrl: "${b}" }, }); const r = await ai.models.generateContent({ model: "${m}", contents: "hi" }); @@ -2811,7 +2814,20 @@ function renderConnect() { if (!models.some((m) => m.id === exampleModel)) exampleModel = models[0]?.id || ""; const model = exampleModel || "provider/model"; const f = FLAVORS[flavor] || FLAVORS.openai; - const base = f.base(g.url); + const urls = [g.url, ...(g.lanURLs || [])]; + if (!urls.includes(connectURL)) connectURL = g.url; + const remote = connectURL !== g.url; + const keys = g.lan ? (gatewayKeys || []).filter((k) => !k.off) : []; + if (!keys.some((k) => k.id === connectKeyID)) { + connectKeyID = ""; + connectSecret = null; + } + if (remote && !connectKeyID && keys.length) { + queueMicrotask(() => selectConnectKey(keys[0].id)); + } + const key = keys.find((k) => k.id === connectKeyID); + const secret = key ? (connectSecret?.id === key.id ? connectSecret.secret : "") : remote ? "" : "magpie"; + const base = f.base(connectURL); const fold = $("#foldConnect"); fold.setAttribute("aria-expanded", String(!connectFolded)); fold.title = t(connectFolded ? "Show how to connect" : "Fold Connect away"); @@ -2821,17 +2837,31 @@ function renderConnect() { note.replaceChildren(); note.classList.toggle("brief", connectFolded); if (connectFolded) note.append(el("code", "", base), copyBtn(base, "Base URL")); - else note.textContent = t("Loopback only · the key can be anything"); + else note.textContent = t(remote ? "Local network · an enabled gateway key is required" : g.lan && gatewayKeys?.length ? "Use a gateway key to track usage" : "Loopback only · the key can be anything"); box.append(...field("API", segs(Object.entries(FLAVORS).map(([k, v]) => [k, v.name]), flavor, (id) => { flavor = id; localStorage.setItem("magpie.flavor", id); renderConnect(); }), t(f.note))); const b = el("div", "val"); - b.append(el("code", "", base), copyBtn(base, "Base URL")); + if (urls.length > 1) b.append(connectPick("connectAddress", "Address", base, + urls.map((u) => ({ v: u, name: f.base(u), note: t(u === g.url ? "This computer" : "Local network") })), connectURL, + (u) => { connectURL = u; renderConnect(); })); + else b.append(el("code", "", base)); + b.append(copyBtn(base, "Base URL")); box.append(...field("Base URL", b, t("What {env} takes.", { env: f.baseEnv }))); const k = el("div", "val"); - k.append(el("code", "", "magpie"), copyBtn("magpie", t("Key"))); - box.append(...field(t("API key"), k, t("{env}=magpie. The gateway trusts everything on loopback, so any value works.", { env: f.keyEnv }))); + const options = keys.map((k) => ({ v: k.id, name: k.name, literalName: true, note: k.masked })); + if (!remote) options.unshift({ v: "", name: "This computer", note: t("Any key · no key attribution") }); + const keyLabel = remote || key ? "Gateway key" : "API key"; + if (!g.lan) k.append(el("code", "", "magpie")); + else if (options.length) k.append(connectPick("connectKey", keyLabel, key ? key.name : remote ? t("Choose a gateway key") : t("This computer"), + options, connectKeyID, selectConnectKey)); + else k.append(el("code", "", t("Create a gateway key above to connect"))); + if (key) k.append(copyCallerKeyBtn(key)); + else if (!remote) k.append(copyBtn("magpie", t("Key"))); + box.append(...field(t(keyLabel), k, t(g.lan && (remote || gatewayKeys?.length) + ? "Choose a gateway key to use as {env}; usage is tracked by key." + : "{env}=magpie. The gateway trusts everything on loopback, so any value works.", { env: f.keyEnv }))); const m = el("div", "val"); m.append(el("code", "", model), copyBtn(model, t("Model id"))); @@ -2839,16 +2869,18 @@ function renderConnect() { const ex = el("div", "stack"); ex.append(segs(LANGS, lang, (id) => { lang = id; localStorage.setItem("magpie.lang", id); renderConnect(); })); - const code = lang === "shell" ? envSnippet([[f.baseEnv, base], [f.keyEnv, "magpie"]]) - : lang === "curl" ? curlSnippet(f.curl(base, model)) - : f[lang](base, model); + const code = !secret ? t(key ? "Loading gateway key…" : "Create a gateway key above to connect") + : lang === "shell" ? envSnippet([[f.baseEnv, base], [f.keyEnv, secret]]) + : lang === "curl" ? curlSnippet(f.curl(base, model, secret)) + : f[lang](base, model, secret); const pre = el("pre", "snip"); const c = el("code"); c.append(highlight(code, lang)); pre.append(c); // the button sits outside the scrolling box, so a long line doesn't carry it off const wrap = el("div", "snip-wrap"); - wrap.append(pre, copyBtn(code, t("Snippet"))); + wrap.append(pre); + if (secret) wrap.append(copyBtn(code, t("Snippet"))); ex.append(wrap); box.append(...field(t("Example"), ex, lang === "shell" ? t("Put these in the shell (or the tool's settings) and the tool talks to magpie instead of the vendor.") : "")); } @@ -6515,7 +6547,7 @@ function openProtoMenu(anchor, opts, value, choose, head = "Protocol this key sp b.type = "button"; b.setAttribute("role", multi && o.v && o.v !== "\x00" ? "menuitemcheckbox" : "menuitemradio"); const words = el("span", "pm-words"); - words.append(el("span", "pm-name", t(o.name)), el("span", "pm-note", t(o.note))); + words.append(el("span", "pm-name", o.literalName ? o.name : t(o.name)), el("span", "pm-note", t(o.note))); b.append(el("span", "pm-tick"), words); tick(b, o); b.onclick = (e) => { @@ -6914,7 +6946,7 @@ function renderUsageLoading() { stats.append(tile); } $("#chart").hidden = true; - for (const id of ["usageAgents", "usageModels"]) $("#" + id).hidden = true; + for (const id of ["usageAgents", "usageModels", "usageKeys"]) $("#" + id).hidden = true; for (const h of $$("#view-usage .row-head")) h.hidden = true; $("#usageNote").textContent = ""; } @@ -7509,7 +7541,7 @@ function autoResetButton(q, cls) { // askReset: spending a Codex account's reset can't be taken // back, so it asks first; then it says what came of it and reads the // usage again. -let resetAsk = null; +let confirmAsk = null; function askReset(q) { const ed = el("div", "editor reset-ask"); const head = el("div", "ehead"); @@ -7530,7 +7562,7 @@ function askReset(q) { go.classList.add("busy"); try { const out = await api("usage/codex-reset", { user: q.user || "" }); - closeResetAsk(); + closeConfirmAsk(); status(who + ": " + resetOutcome(out), out.code === "reset" ? "ok" : "err"); loadQuotas(); } catch (err) { @@ -7540,26 +7572,26 @@ function askReset(q) { } }; const cancel = el("button", "text", t("Cancel")); - cancel.onclick = (e) => { e.stopPropagation(); closeResetAsk(); }; + cancel.onclick = (e) => { e.stopPropagation(); closeConfirmAsk(); }; bar.append(el("span", "grow"), cancel, go); ed.append(bar); - resetAsk = ed; + confirmAsk = ed; openModal(ed); $("#modal").classList.add("lib"); go.focus(); } -function closeResetAsk() { - if (!resetAsk) return; - resetAsk = null; - closeModal().then(() => { if (!resetAsk) $("#modal").classList.remove("lib"); }); +function closeConfirmAsk() { + if (!confirmAsk) return; + confirmAsk = null; + closeModal().then(() => { if (!confirmAsk) $("#modal").classList.remove("lib"); }); } // the dialog is the providers page's: while this asks, its backdrop and -// Escape close only this (in the panel, Escape would hide the window) +// Escape closes only the confirmation (in the panel it would hide the window) $("#modal").addEventListener("click", (e) => { - if (resetAsk && e.target === e.currentTarget) { e.stopImmediatePropagation(); closeResetAsk(); } + if (confirmAsk && e.target === e.currentTarget) { e.stopImmediatePropagation(); closeConfirmAsk(); } }, true); document.addEventListener("keydown", (e) => { - if (resetAsk && e.key === "Escape") { e.preventDefault(); e.stopImmediatePropagation(); closeResetAsk(); } + if (confirmAsk && e.key === "Escape") { e.preventDefault(); e.stopImmediatePropagation(); closeConfirmAsk(); } }, true); function resetOutcome(out) { switch (out.code) { @@ -7744,6 +7776,7 @@ function renderUsage() { $("#chart").hidden = empty; for (const id of ["usageAgents", "usageModels"]) $("#" + id).hidden = empty; for (const h of $$("#view-usage .row-head")) h.hidden = empty; + $("#usageKeysHead").hidden = $("#usageKeys").hidden = empty || !u.callerKeys?.length; if (empty) { stats.classList.add("empty"); const none = { today: "No calls today.", "7d": "No calls in the last 7 days.", "30d": "No calls in the last 30 days.", all: "No calls yet." }[period]; @@ -7829,6 +7862,7 @@ function renderUsage() { }; list("usageAgents", u.agents); list("usageModels", u.models); + list("usageKeys", u.callerKeys || []); $("#usageNote").textContent = t("Counted from the providers' own usage reports on every call through the gateway · {path}", { path: u.path }); } @@ -7841,7 +7875,7 @@ function renderUsage() { // server pages it (/api/usage/requests) and saves it whole as CSV. let ledger = null; // the page shown: { rows, offset, total, agents, …totals } -let ledOffset = 0, ledAgent = "", ledProvider = "", ledFailed = false, ledQuery = "", ledModel = "", ledRoute = 0; +let ledOffset = 0, ledAgent = "", ledProvider = "", ledCallerKey = "", ledFailed = false, ledQuery = "", ledModel = "", ledRoute = 0; const LED_PAGE = 100; // Remember the chart metric; start each app load split by model. let ledMetric = "tokens", ledSplit = "model"; @@ -7852,10 +7886,10 @@ try { let ledRouteInfo = null, ledBeforeRoute = null; window.openUsageRoute = (route) => { - if (!ledBeforeRoute) ledBeforeRoute = { period, ledOffset, ledAgent, ledProvider, ledFailed, ledQuery, ledModel }; + if (!ledBeforeRoute) ledBeforeRoute = { period, ledOffset, ledAgent, ledProvider, ledCallerKey, ledFailed, ledQuery, ledModel }; ledRoute = route.id; ledRouteInfo = route; - ledOffset = 0; ledAgent = ""; ledProvider = ""; ledFailed = false; ledQuery = ""; ledModel = ""; + ledOffset = 0; ledAgent = ""; ledProvider = ""; ledCallerKey = ""; ledFailed = false; ledQuery = ""; ledModel = ""; $("#ledQ").value = ""; period = "all"; usageTab = "requests"; @@ -7867,6 +7901,7 @@ function ledParams(extra) { const q = new URLSearchParams({ period }); if (ledAgent) q.set("agent", ledAgent); if (ledProvider) q.set("provider", ledProvider); + if (ledCallerKey) q.set("callerKey", ledCallerKey); if (ledRoute) q.set("route", ledRoute); if (ledFailed) q.set("failed", "1"); if (ledModel) q.set("model", ledModel); @@ -8393,6 +8428,14 @@ const LED_COLS = [ function renderLedger() { const l = ledger; + const callers = l.callerKeys || []; + if (ledCallerKey && !callers.some((k) => k.id === ledCallerKey)) { + ledCallerKey = ""; + ledOffset = 0; + // Reload the rows too: this response still belongs to the missing key. + loadLedger().catch((e) => status(e.message, "err")); + return; + } const view = $("#view-usage"); view.classList.remove("loading"); view.removeAttribute("aria-busy"); @@ -8413,6 +8456,9 @@ function renderLedger() { const providers = l.providers || []; if (ledProvider && !providers.some((p) => p.id === ledProvider)) ledProvider = ""; sessPick($("#ledProvider"), "All providers", ledProvider, providers.map((p) => ({ v: p.id, name: t(p.name), note: "" })), "Provider", (v) => { ledProvider = v; ledOffset = 0; loadLedger().catch((e) => status(e.message, "err")); }); + sessPick($("#ledKey"), "All gateway keys", ledCallerKey, callers.map((k) => ({ + v: k.id, name: k.name, note: "", + })), "Gateway keys", (v) => { ledCallerKey = v; ledOffset = 0; loadLedger().catch((e) => status(e.message, "err")); }); const seg = $("#ledStatus"); seg.replaceChildren(); for (const [on, name] of [[false, "All"], [true, "Failed"]]) { @@ -8442,7 +8488,7 @@ function renderLedger() { $("#ledRouteClear").setAttribute("aria-label", t("Clear filter")); $("#ledRouteClear").onclick = () => { ledRoute = 0; ledRouteInfo = null; - if (ledBeforeRoute) ({ period, ledOffset, ledAgent, ledProvider, ledFailed, ledQuery, ledModel } = ledBeforeRoute); + if (ledBeforeRoute) ({ period, ledOffset, ledAgent, ledProvider, ledCallerKey, ledFailed, ledQuery, ledModel } = ledBeforeRoute); ledBeforeRoute = null; $("#ledQ").value = ledQuery; loadLedger().catch((e) => status(e.message, "err")); @@ -8453,7 +8499,7 @@ function renderLedger() { const pager = $("#ledPager"); if (!l.total) { wrap.classList.add("none"); - const filtered = ledRoute || ledAgent || ledProvider || ledFailed || ledQuery.trim(); + const filtered = ledRoute || ledAgent || ledProvider || ledCallerKey || ledModel || ledFailed || ledQuery.trim(); const none = { today: "No calls today.", "7d": "No calls in the last 7 days.", "30d": "No calls in the last 30 days.", all: "No calls yet." }[period]; wrap.replaceChildren(el("div", "led-none", filtered ? t("No requests match these filters.") : t(none))); pager.hidden = true; @@ -8489,7 +8535,7 @@ function renderLedger() { const who = el("span", "who"); // an agent on another computer, whose magpie passed the request on const name = r.agentName || r.agent; - who.append(icon(r.icon || "generic"), el("span", "", r.via ? t("{agent} · via {host}", { agent: name, host: r.via }) : name)); + who.append(icon(r.icon || "generic"), el("span", "", [r.via ? t("{agent} · via {host}", { agent: name, host: r.via }) : name, r.callerKeyLabel || r.callerKeyName].filter(Boolean).join(" · "))); td(who, "", [r.kind, r.session && t("session {id}", { id: r.session })].filter(Boolean).join(" · ")); td(r.req || "—", "model" + (r.req ? "" : " faint"), r.req || t("Not kept for requests before this version")); const local = r.source === "log"; @@ -10914,7 +10960,7 @@ function renderRedactRules(s, row) { } // renderLAN: the gateway shared on the local network, for agents on other -// machines — the addresses they use it at and the key they must send. +// machines. Gateway keys and connection examples live together in Gateway. let lanSelectedURL = "", lanProtocol = "openai"; function renderLAN(s) { const box = $("#lanList"); @@ -10933,7 +10979,7 @@ function renderLAN(s) { }; const set = (body) => writingPrefs(api("settings/lan", body)).then((ns) => { prefs = ns; renderSettings(); }) .catch((e) => { status(t(e.message), "err"); renderSettings(); }); - row(t("Share on local network"), t("Agents on other computers on this network can use magpie’s models, with the API key below"), "", + row(t("Share on local network"), t("Agents on other computers on this network can use magpie’s models with a gateway key from Gateway"), "", segs([["off", t("Off")], ["on", t("On")]], s.lan ? "on" : "off", (v) => set({ on: v === "on" }))); if (!s.lan) return; let urls = s.lanURLs || [], sub = ""; @@ -10981,10 +11027,6 @@ function renderLAN(s) { update(); } if (sub) row(t("In a container"), sub, "").classList.add("lan-container"); - const again = el("button", "text", t("New key")); - again.onclick = () => set({ on: true, newKey: true }); - row(t("API key"), t("Other computers send it as their API key; a new one stops the old from working"), - s.lanKey.slice(0, 14) + "…", copyBtn(s.lanKey, t("API key")), again); } // renderUpdate fills in the version row: whether a newer magpie is out. @@ -11355,6 +11397,7 @@ function show(v) { closePicker(); closeAgentModels(); if (v !== "providers" && editing !== null) cancelEdit(); + if (v === "gateway") loadGatewayKeys(); if (v === "providers" || v === "gateway" || v === "routing") loadProviders().then(back, (e) => status(e.message, "err")); if (v === "usage") loadUsage(true).then(back, (e) => status(e.message, "err")); if (v === "settings") loadSettings().then(back, (e) => status(e.message, "err")); diff --git a/internal/gui/assets/caller-keys.js b/internal/gui/assets/caller-keys.js new file mode 100644 index 00000000..787b9c47 --- /dev/null +++ b/internal/gui/assets/caller-keys.js @@ -0,0 +1,187 @@ +// Caller keys use the same account list and edit controls as provider keys. +let gatewayKeys = null, gatewayKeyDraft = null, gatewayKeysBusy = false; +let gatewayKeysRead = 0; +let connectURL = "", connectKeyID = "", connectSecret = null; + +function connectPick(id, label, text, options, value, choose) { + const b = el("button", "val"); + b.id = id; + b.type = "button"; + b.setAttribute("aria-label", t(label)); + b.append(el("code", "", text), svg(CHEV, 11, 1.6)); + b.onclick = () => { + if (b.classList.contains("open")) return closeProtoMenu(); + openProtoMenu(b, options, value, choose, label); + }; + return b; +} + +async function selectConnectKey(id) { + connectKeyID = id; + connectSecret = null; + renderConnect(); + if (!id || !providers?.gateway.lan) return; + const key = gatewayKeys?.find((k) => k.id === id && !k.off); + if (!key) return; + try { + const out = await api("caller-keys/copy-key", { key: id }); + if (connectKeyID !== id || gatewayKeys?.find((k) => k.id === id) !== key) return; + connectSecret = { id, secret: out.secret }; + if (view === "gateway") renderConnect(); + } catch (e) { status(t(e.message), "err"); } +} +async function loadGatewayKeys() { + const read = ++gatewayKeysRead; + try { + const out = await api("caller-keys"); + if (read !== gatewayKeysRead) return; + // A fresh list also refreshes a selected credential, which may have rotated. + connectSecret = null; + gatewayKeys = out.keys || []; + renderGatewayKeys(); + if (view === "gateway" && providers) selectConnectKey(connectKeyID); + } catch (e) { + if (read !== gatewayKeysRead) return; + $("#gatewayKeys").replaceChildren(el("div", "none", t(e.message))); + } +} + +async function gatewayKeyAction(action, body) { + if (gatewayKeysBusy) return null; + gatewayKeysBusy = true; + gatewayKeysRead++; + try { + const out = await api("caller-keys/" + action, body); + gatewayKeysRead++; + connectSecret = null; + gatewayKeys = out.keys || []; + gatewayKeyDraft = null; + renderGatewayKeys(); + if (view === "gateway" && providers) selectConnectKey(connectKeyID); + return out; + } catch (e) { + status(t(e.message), "err"); + return null; + } finally { + gatewayKeysBusy = false; + } +} + +function gatewayRename(k) { + const b = el("button", "n rename", k.name); + b.title = t("Rename"); + b.onclick = () => { + const i = input(k.name, t("Name")); + i.className = "rename-in"; + i.setAttribute("aria-label", t("Name")); + let finished = false; + const done = async (save) => { + if (finished) return; + finished = true; + // Keep the other controls in place when blur precedes their click. + i.replaceWith(b); + if (save && i.value.trim() !== k.name) await gatewayKeyAction("rename-key", { key: k.id, name: i.value }); + }; + i.onkeydown = (e) => { e.stopPropagation(); if (e.key === "Enter") done(true); else if (e.key === "Escape") done(false); }; + i.onblur = () => done(true); + b.replaceWith(i); + i.focus({ preventScroll: true }); + }; + return b; +} + +function gatewayKeyForm() { + const box = el("div", "acc adding"); + const name = input(gatewayKeyDraft, t("Key name, e.g. Laptop")); + name.setAttribute("aria-label", t("Gateway key name")); + name.oninput = () => { gatewayKeyDraft = name.value; }; + const add = el("button", "text primary", t("Create")); + const go = async () => { + add.disabled = true; + const out = await gatewayKeyAction("add-key", { name: name.value }); + if (!out) { add.disabled = false; return; } + status(t("Gateway key created. Use Copy on its row to connect a client."), "ok"); + }; + add.onclick = go; + name.onkeydown = (e) => { + e.stopPropagation(); + if (e.key === "Enter" && !add.disabled) go(); + else if (e.key === "Escape") { gatewayKeyDraft = null; renderGatewayKeys(); } + }; + const cancel = el("button", "text", t("Cancel")); + cancel.onclick = () => { gatewayKeyDraft = null; renderGatewayKeys(); }; + const fields = el("div", "kf"); + fields.append(name); + const bar = el("div", "kb"); + bar.append(el("span", "grow"), cancel, add); + box.append(fields, bar); + queueMicrotask(() => name.focus({ preventScroll: true })); + return box; +} + +function copyCallerKeyBtn(k) { + const cp = copyBtn("", t("Gateway key")); + cp.setAttribute("aria-label", t("Copy gateway key")); + cp.onclick = async () => { + try { + const out = await api("caller-keys/copy-key", { key: k.id }); + await copy(out.secret, t("Gateway key"), cp); + } catch (e) { status(t(e.message), "err"); } + }; + return cp; +} + +function renderGatewayKeys() { + const box = $("#gatewayKeys"); + $("#gatewayKeysBlock").hidden = !providers?.gateway.lan; + box.replaceChildren(); + $("#addGatewayKey").onclick = () => { gatewayKeyDraft = ""; renderGatewayKeys(); }; + if (gatewayKeys === null) return; + const list = el("div", "accts"); + for (const k of gatewayKeys) { + const row = el("div", "acc" + (k.off ? " off" : " in-use")); + row.dataset.key = k.id; + const tick = el("button", "dot tick"); + tick.title = t(k.off ? "Enable key" : "Disable key"); + tick.setAttribute("aria-label", tick.title); + if (!k.off) tick.append(svg(CHECK, 10, 2.2)); + tick.onclick = () => gatewayKeyAction(k.off ? "on-key" : "off-key", { key: k.id }); + const rm = el("button", "text quiet", t("Remove")); + rm.onclick = () => askGatewayKey(k, false); + const rotate = el("button", "text quiet", t("Rotate key")); + rotate.onclick = () => askGatewayKey(k, true); + row.append(tick, gatewayRename(k), el("span", "plan mono", k.masked), el("span", "grow"), copyCallerKeyBtn(k), rotate, rm); + list.append(row); + } + if (gatewayKeyDraft !== null) list.append(gatewayKeyForm()); + else if (!gatewayKeys.length) list.append(el("div", "none", t("No gateway keys yet"))); + box.append(list); +} + +function askGatewayKey(k, rotate) { + const ed = el("div", "editor"); + const head = el("div", "ehead"); + head.append(el("b", "", t(rotate ? "Rotate gateway key?" : "Remove gateway key?"))); + ed.append(head, el("p", "lib-confirm", t(rotate + ? "Clients using {name} will need the new key. Its name and usage history stay the same." + : "Clients using {name} will lose access from other computers. Its usage history is kept.", { name: k.name }))); + const bar = el("div", "bar"); + const cancel = el("button", "text", t("Cancel")); + cancel.onclick = (e) => { e.stopPropagation(); closeConfirmAsk(); }; + const go = el("button", "text primary", t(rotate ? "Rotate key" : "Remove")); + go.onclick = async (e) => { + e.stopPropagation(); + if (go.disabled) return; + go.disabled = true; + const out = await gatewayKeyAction(rotate ? "rotate-key" : "remove-key", { key: k.id }); + if (!out) { go.disabled = false; return; } + closeConfirmAsk(); + if (rotate) status(t("Gateway key rotated. Update clients on other computers."), "ok"); + }; + bar.append(el("span", "grow"), cancel, go); + ed.append(bar); + confirmAsk = ed; + openModal(ed); + $("#modal").classList.add("lib"); + cancel.focus({ preventScroll: true }); +} diff --git a/internal/gui/assets/i18n.js b/internal/gui/assets/i18n.js index 19431d65..49ac90a2 100644 --- a/internal/gui/assets/i18n.js +++ b/internal/gui/assets/i18n.js @@ -4,6 +4,35 @@ // works. {name} slots are filled by t(str, vars). const I18N = { zh: { + "Gateway keys": "网关密钥", + "Gateway key": "网关密钥", + "All gateway keys": "全部网关密钥", + "Use a gateway key to track usage": "使用网关密钥统计用量", + "Choose a gateway key to use as {env}; usage is tracked by key.": "选择填入 {env} 的网关密钥，用量按密钥统计。", + "This computer": "本机", + "Any key · no key attribution": "任意密钥 · 不归属到具体密钥", + "Choose a gateway key": "选择网关密钥", + "Create a gateway key above to connect": "请先在上方创建网关密钥", + "Loading gateway key…": "正在读取网关密钥…", + "Local network · an enabled gateway key is required": "局域网接入需要已启用的网关密钥", + "Gateway key rotated. Update clients on other computers.": "网关密钥已轮换，请更新其他电脑上的客户端。", + "Add gateway key": "添加网关密钥", + "Create a gateway key for each client. Usage is counted separately for each key.": "为每个客户端创建一个网关密钥，分别统计用量。", + "Gateway key name": "网关密钥名称", + "Key name, e.g. Laptop": "密钥名称，例如笔记本", + "Create": "创建", + "Gateway key created. Use Copy on its row to connect a client.": "网关密钥已创建。点击该行的复制按钮即可配置客户端。", + "Copy gateway key": "复制网关密钥", + "Rotate key": "轮换密钥", + "No gateway keys yet": "尚无网关密钥", + "Rotate gateway key?": "轮换网关密钥？", + "Remove gateway key?": "移除网关密钥？", + "Clients using {name} will need the new key. Its name and usage history stay the same.": "使用 {name} 的客户端需要更新密钥，名称和用量历史保持不变。", + "Clients using {name} will lose access from other computers. Its usage history is kept.": "使用 {name} 的客户端将无法从其他电脑接入，用量历史会保留。", + "Enable key": "启用密钥", + "Disable key": "停用密钥", + "Use a name between 1 and 120 characters": "请输入 1 到 120 个字符的名称", + "Key not found": "密钥不存在", // header, nav, footer "Agents": "Agent", "Providers": "供应商", @@ -138,13 +167,12 @@ const I18N = { "Share on local network": "局域网共享", "{label} address": "{label} 地址", "Copied {label} address": "已复制 {label} 地址", - "Agents on other computers on this network can use magpie’s models, with the API key below": "同一网络里其他电脑上的 Agent 可以用下面的 API Key 使用 magpie 的模型", + "Agents on other computers on this network can use magpie’s models with a gateway key from Gateway": "同一网络里其他电脑上的 Agent 可以使用 magpie 的模型，网关密钥在网关中管理", "This computer has no local network address right now": "这台电脑现在没有局域网地址", "In a container": "在容器中运行", "Where this page was opened, on the gateway’s port; MAGPIE_PUBLIC_URL sets another": "即打开本页所用的地址，端口为网关的端口；可用 MAGPIE_PUBLIC_URL 改成别的", "The container’s own addresses, which other devices can’t reach: set MAGPIE_PUBLIC_URL to the host’s": "这是容器自己的地址，其他设备连不上：请把 MAGPIE_PUBLIC_URL 设为宿主机的地址", "New key": "换新 Key", - "Other computers send it as their API key; a new one stops the old from working": "其他电脑把它填作 API Key；换新后旧的立即失效", "API keys, private keys, tokens and passwords go to vendors as placeholders, and come back as they were": "API Key、私钥、Token 和密码以占位符发给模型厂商，回复里再还原", "Mask personal data": "脱敏个人信息", "Count me as a user": "计入用户数", diff --git a/internal/gui/assets/index.html b/internal/gui/assets/index.html index 5ec394e3..3c388d91 100644 --- a/internal/gui/assets/index.html +++ b/internal/gui/assets/index.html @@ -89,6 +89,11 @@
+
@@ -133,12 +138,15 @@
Models
+ +