providers: the add sheet lists Partners first, above Subscriptions, under a heading that says they sponsor magpie; who they are comes from usemagpie.ai/api/partners (site/partners.js), so a paid listing changes with a deploy, not a release (yetone)

The app fetches the list behind the call every 6 hours (10 minutes after a
failure), never waiting, and keeps it in the cache folder. Each entry is
checked: its own id, never a built-in preset's; a name and an endpoint;
https only; at most 12; no decision API, no-key or user endpoint. Its logo
goes through FetchIcon. A partner is a preset to the rest of magpie, and
one that has left still is to the providers added from it, unsponsored.
The window lists a partner in the languages it names only, with its tagline
in the page's language; `magpie presets` and the TUI list them first too.
MAGPIE_PARTNERS moves the list, off lists none.

Run: go test -tags nogui -race ./internal/provider ./internal/gui
./internal/tui . (ok); TestPartners* -race -count=3; node --test
internal/gui/tests/partners.test.cjs add-sheet gui-ja gui-de i18n-keys in
Chromium and WebKit (en/zh/ja/de, 900 and 440px); node --test
site/worker.test.mjs site/install.test.mjs; the built binary in a sandbox
HOME against a local feed: listed, an http entry left out, listed from the
cache with the site down, `provider add` from it.
This commit is contained in:
yetone committed 2026-10-08 22:52:30 +08:00
1 parent 5ba0601fda
commit 10578ce683
13 files changed
+857 -5

No files matched your search

+3
View File
@@ -30,6 +30,7 @@ keys and accounts per request is in [Gateway routing and fallback](gateway-routi
| Allowance | `Quotas` asks every vendor at once: subscription windows (`SubscriptionUsage`), key-bought plans (`PlanQuotas`) and key balances (`KeyBalances`). The Usage page, `magpie quota --json` and `GET /v1/magpie/quotas` read it. A single card can be read again on its own (#840). A remote magpie's cards come from what it has kept (`CachedCards`, `GET /v1/magpie/quotas/cards`) and are never read there for this computer except by a card's refresh (`POST /v1/magpie/quotas/refresh`, throttled per card). What it has kept is the Usage page's cards with the readings made behind the requests since laid over them, a later one over an earlier by `readSeq` (`cards_kept.go`): each account's `LoginUsage` reading (the used-up switch, the Codex warm-up, routing's `Allowances`), and each key's `KeyAllowance` windows on its plan or balance card. An account or key read that way and never on the page gets a card from that reading, and after a restart, before either is read again, the cards come from `quotas.json`, dated (`AsOf`). A magpie serving only other magpies has nobody on its page (#1313). Cursor, Grok and a Kiro key keep the page's cards only. | [`cards_kept.go`](../../internal/provider/cards_kept.go), [`quotas.go`](../../internal/provider/quotas.go), [`subscription_usage.go`](../../internal/provider/subscription_usage.go), [`planquota.go`](../../internal/provider/planquota.go), [`balance.go`](../../internal/provider/balance.go), [`quota_refresh.go`](../../internal/provider/quota_refresh.go), [`remote_quotas.go`](../../internal/provider/remote_quotas.go) |
| A key's own windows | A sub2api key given a spending limit per 5 hours, day or 7 days answers its `/v1/usage` with `rate_limits` and no `remaining` (`readSub2APIKeyLimits`). With the Balance URL set to it, the key's card shows them as the windows *5 hours*, *1 day*, *7 days*, in USD, beside a balance field the user wrote; *Check balance* and `magpie provider show` give what its fullest window has left (`Balance`). `KeyAllowance` gives routing the same windows: never waited for, read in the background at most once a minute, a failed read keeping the last, and seeded from the card's last reading (`quotas.json`) after a restart. A key bought as a plan (Kimi Code, GLM Coding Plan and Z.ai, MiniMax Coding Plan, OpenCode Go, Command Code: `planQuotaSourceOf`) gives routing its card's windows the same way (`planKeyWindows`, Zhipu's team quota behind a key with none of its own); the Usage page's read of its card is taken at once (`noteKeyAllowance`), and a key there that turns out pay-as-you-go, with no windows, is asked again only every 30 minutes (`keyNoWindowsAge`). StepFun's Step Plan is read through a sign-in, not the key, and isn't one of them. | [`sub2api_usage.go`](../../internal/provider/sub2api_usage.go), [`key_allowance.go`](../../internal/provider/key_allowance.go) |
| Last served | `served.json` records when each account, plan or key last answered, shown as `lastServedAt` (#570). | [`served.go`](../../internal/provider/served.go) |
| Partners | Services that pay to be listed first in the add sheet, under a heading of their own ("Partners · magpie's sponsors"). The list is `site/partners.js`, served at `usemagpie.ai/api/partners`: a change reaches users with a deploy, not a release. `Partners` never waits: it returns what was fetched last and fetches again behind the call every 6 hours (10 minutes after a failure), keeping the list in `partners.json` in the cache folder. Each entry is checked again on every read (`validPartners`): its own id, never a built-in preset's, a name, an endpoint, https only, at most 12; what only magpie's presets may set (a decision API, no key, a user endpoint) is cleared. Its logo is fetched through `FetchIcon` and kept as an uploaded icon. A partner is a preset to the rest of magpie (`Preset`, `FromPreset`, the editor), and one no longer listed still is to the providers added from it (`Seen`), without the sponsored mark. The window lists a partner in the languages it names (`langs`) only and shows its tagline in the page's language (`notes`, else `en`). `MAGPIE_PARTNERS` points it elsewhere, `off` lists none; tests list none unless they name a list. | [`partners.go`](../../internal/provider/partners.go), [`site/partners.js`](../../site/partners.js) |
| Protocol detection | Works out which APIs a base URL speaks when a provider is added. Detection, endpoint tests and model tests share `probe`, which adds `anthropic-version` only for Anthropic Messages; OpenAI probes omit that default so a relay that refuses it can answer. User-supplied headers still apply through `Sign`. | [`detect.go`](../../internal/provider/detect.go), [`test.go`](../../internal/provider/test.go) |
## Runtime path
@@ -62,6 +63,8 @@ keys and accounts per request is in [Gateway routing and fallback](gateway-routi
go test -tags nogui ./internal/provider -run 'TestAccount|TestCodex|TestGroup|TestQuota|TestLogin|TestSignIn|TestSeveralKeys|TestKeyProtocol|TestSetKeyWeight|TestReadSub2APIKeyLimits|TestSub2APIKeyLimitsOnItsCard|TestKeyAllowance|TestPlanKeyAllowance|TestPayAsYouGoKeyAskedSeldom|TestClaudeAccountToldRenewed|TestPluginAccountToldRenewed'
go test -tags nogui ./internal/provider -run 'TestVolcengine|TestQianfan|TestBedrock'
go test -tags nogui ./internal/provider -run '^TestProviderProbesUseProtocolHeaders$'
go test -tags nogui ./internal/provider -run '^TestPartners'
node --test internal/gui/tests/partners.test.cjs site/worker.test.mjs
go test -tags nogui ./internal/provider
```
+36 -1
View File
@@ -6479,6 +6479,15 @@ function renderAdd() {
const named = (x) => !f || x.name.toLowerCase().includes(f) || x.agent.includes(f) || "subscription".includes(f);
const subs = SUBS.filter((x) => !(replacedSub(x.agent) || unusedSub(x.agent)) || signing?.agent === x.agent).map((x) => subOf(x.agent)).filter(named);
const inPlugins = f && SUBS.some((x) => unusedSub(x.agent) && signing?.agent !== x.agent && named(subOf(x.agent)));
// partners pay to be listed first, under a heading that says so; who
// they are comes from usemagpie.ai (internal/provider/partners.go), and
// each may name the languages it is shown in
const partners = providers.presets.filter((p) => p.kind === "partner" && partnerShown(p) && (hit(p) || partnerNote(p).toLowerCase().includes(f)));
if (partners.length) {
any = true;
const grid = section("Partners", "magpie's sponsors");
for (const pr of partners) grid.append(partnerTile(pr));
}
if (subs.length || inPlugins) {
any = true;
const grid = section("Subscriptions", "sign in, no key");
@@ -6688,6 +6697,32 @@ function tile(pr) {
return b;
}
// the page's language and its base ("zh-TW", "zh")
function pageLangs() {
const l = document.documentElement.lang || "en";
return [l, l.split("-")[0]];
}
// a partner listed in no language of its own is listed in every one
function partnerShown(pr) {
return !pr.langs?.length || pageLangs().some((l) => pr.langs.includes(l));
}
// the partner's tagline in the page's language, else its English one; the
// partner writes it, so it isn't translated here
function partnerNote(pr) {
const n = pr.notes || {};
for (const l of pageLangs()) if (n[l]) return n[l];
return n.en || "";
}
// a partner's row: its own section says it is sponsored, so no badge
function partnerTile(pr) {
const b = tile({ ...pr, sponsored: false, note: "" });
if (!pr.added) b.title = pr.name + (partnerNote(pr) ? " · " + partnerNote(pr) : "") + "\n" + hostOf(pr.chat || pr.responses || pr.anthropic);
return b;
}
let fieldIDs = 0;
function field(label, control, hint) {
const l = el("label", "", label);
@@ -7642,7 +7677,7 @@ function drawEditor(p, presetID) {
const h = el("div", "ehead");
const copyOf = draft.copyOf && providers.providers.find((x) => x.id === draft.copyOf);
h.append(icon(p?.icon || (copyOf && draft.icon) || pr?.icon || "generic"), el("b", "", p ? p.name : copyOf ? t("Copy of {name}", { name: copyOf.name }) : pr ? pr.name : t("Custom provider")));
if (pr?.note) h.append(el("span", "note", t(pr.note)));
if (pr?.kind === "partner") { if (partnerNote(pr)) h.append(el("span", "note", partnerNote(pr))); } else if (pr?.note) h.append(el("span", "note", t(pr.note)));
h.append(el("span", "grow"));
// a plugin's provider has plugin://<id> for its base, and its id is no
// address to open: only a host with a dot or a port makes a link
+8
View File
@@ -1002,6 +1002,8 @@ const I18N = {
"the makers' own APIs": "模型厂商的 API",
"Subscriptions": "订阅",
"sign in, no key": "登录即可,无需密钥",
"Partners": "合作伙伴",
"magpie's sponsors": "赞助 magpie 的服务",
"Relays": "中转",
"one key, many vendors": "一个密钥,多家模型",
"Removed from magpie": "已从 magpie 移除",
@@ -4773,6 +4775,8 @@ const I18N = {
"the makers' own APIs": "模型廠商的 API",
"Subscriptions": "訂閱",
"sign in, no key": "登入即可,無需金鑰",
"Partners": "合作夥伴",
"magpie's sponsors": "贊助 magpie 的服務",
"Relays": "中轉",
"one key, many vendors": "一個金鑰,多家模型",
"Removed from magpie": "已從 magpie 移除",
@@ -8518,6 +8522,8 @@ const I18N = {
"the makers' own APIs": "メーカー自身の API",
"Subscriptions": "サブスクリプション",
"sign in, no key": "サインインのみ、キー不要",
"Partners": "パートナー",
"magpie's sponsors": "magpie のスポンサー",
"Relays": "中継サービス",
"one key, many vendors": "1 つのキーで複数のベンダー",
"Removed from magpie": "magpie から削除済み",
@@ -12282,6 +12288,8 @@ const I18N = {
"the makers' own APIs": "die eigenen APIs der Hersteller",
"Subscriptions": "Abos",
"sign in, no key": "anmelden, kein Schlüssel",
"Partners": "Partner",
"magpie's sponsors": "Sponsoren von magpie",
"Relays": "Relays",
"one key, many vendors": "ein Schlüssel, viele Anbieter",
"Removed from magpie": "Aus magpie entfernt",
+7
View File
@@ -285,6 +285,9 @@ type presetJSON struct {
// ZhipuTeam: a key of it may be on a team's GLM Coding Plan, whose
// organization and project the editor offers to take
ZhipuTeam bool `json:"zhipuTeam,omitempty"`
// a partner's tagline by language, and the languages it is listed in
Notes map[string]string `json:"notes,omitempty"`
Langs []string `json:"langs,omitempty"`
}
type gatewayJSON struct {
@@ -639,6 +642,10 @@ func providersState() providersJSON {
have[p.ID], have[p.Preset] = true, true
s.Providers = append(s.Providers, providerInfo(p, uses))
}
// partners first, as the add sheet lists them
for _, pa := range provider.Partners() {
s.Presets = append(s.Presets, presetJSON{PresetDef: pa.PresetDef, Added: have[pa.ID], Notes: pa.Notes, Langs: pa.Langs})
}
for _, pr := range provider.Presets() {
team := provider.TakesZhipuTeam(provider.Provider{Chat: pr.Chat, Responses: pr.Responses, Anthropic: pr.Anthropic})
s.Presets = append(s.Presets, presetJSON{PresetDef: pr, Added: have[pr.ID], ZhipuTeam: team})
+125
View File
@@ -0,0 +1,125 @@
// Run with Node's test runner and Playwright on the module path; see README.md.
// The add sheet's Partners (yetone: 在订阅上面加上一栏是「合作伙伴」): the
// services that pay to be listed, first and under a heading that says so,
// each one quiet row with no badge of its own. Who they are comes from
// usemagpie.ai through /api/providers (internal/provider/partners.go); one
// that names its languages is listed in those only, and its tagline, in
// the row's title and the editor, is its own in the page's language, else
// its English one. Searching finds one by its tagline. With no partner
// there is no heading. English, Chinese, Japanese and German, at 900px and
// 440px; no backend, the API is faked here.
const assert = require("node:assert/strict");
const fs = require("node:fs/promises");
const path = require("node:path");
const { test } = require("node:test");
const { chromium, webkit } = require("playwright");
const assets = path.resolve(__dirname, "../assets");
const partner = (id, name, extra = {}) => ({ id, name, icon: "generic", kind: "partner", sponsored: true, chat: `https://api.${id}.example.com/v1`, ...extra });
const partners = [
partner("acme", "Acme AI", { notes: { en: "Fast relay", zh: "快速中转", ja: "高速な中継" } }),
partner("zhonly", "China Only", { langs: ["zh"], notes: { en: "Mainland relay", zh: "国内中转" } }),
partner("taken", "Taken", { added: true }),
];
const presets = [
{ id: "openai", name: "OpenAI", icon: "openai", kind: "vendor", chat: "https://api.openai.example.com/v1" },
{ id: "openrouter", name: "OpenRouter", icon: "openai", kind: "relay", chat: "https://openrouter.example.com/v1", sponsored: true },
];
const providers = [{ id: "taken", name: "Taken", icon: "generic", preset: "taken", models: [], agents: [], key: { set: true, masked: "sk-…ab12" } }];
function server(lang, list) {
return async (route) => {
const url = new URL(route.request().url());
const json = (data) => route.fulfill({ json: data });
if (url.pathname === "/boot.js") return route.fulfill({ contentType: "text/javascript", body: `window.bootPrefs = {lang:"${lang}",theme:"light",web:false};` });
if (url.pathname === "/wails/runtime.js") return route.fulfill({ contentType: "text/javascript", body: "export const Window = {};" });
if (url.pathname === "/api/state") return json({ agents: [], profiles: [], settings: { lang, theme: "light" } });
if (url.pathname === "/api/providers") return json({ providers, presets: [...list, ...presets], excluded: [], gateway: { running: true, window: true } });
if (url.pathname === "/api/groups") return json({ groups: [], pools: [] });
if (url.pathname === "/api/gateway/trace") return json({ routes: [] });
if (url.pathname.startsWith("/api/")) return json({});
const file = path.join(assets, url.pathname === "/" ? "index.html" : url.pathname);
const contentType = { ".html": "text/html", ".js": "text/javascript", ".css": "text/css", ".svg": "image/svg+xml", ".png": "image/png" }[path.extname(file)];
try { await route.fulfill({ body: await fs.readFile(file), contentType }); } catch { await route.fulfill({ status: 404, body: "" }); }
};
}
const L = {
en: { head: ["Partners", "magpie's sponsors"], subs: "Subscriptions", rows: ["Acme AI", "Taken"], note: "Fast relay" },
zh: { head: ["合作伙伴", "赞助 magpie 的服务"], subs: "订阅", rows: ["Acme AI", "China Only", "Taken"], note: "快速中转" },
ja: { head: ["パートナー", "magpie のスポンサー"], subs: null, rows: ["Acme AI", "Taken"], note: "高速な中継" },
de: { head: ["Partner", "Sponsoren von magpie"], subs: null, rows: ["Acme AI", "Taken"], note: "Fast relay" },
};
async function openSheet(page) {
await page.goto("http://magpie.test/?view=providers");
await page.locator("#addProvider").click();
const sheet = page.locator("#addSheet");
await sheet.locator(".tile").first().waitFor();
await sheet.evaluate((el) => Promise.all(el.getAnimations().map((a) => a.finished)));
return sheet;
}
for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium", "webkit"])) {
test(engine + ": partners come first in the add sheet", async (t) => {
assert(["chromium", "webkit"].includes(engine), "BROWSER must be chromium or webkit");
const browser = await (engine === "webkit" ? webkit.launch() : chromium.launch({ channel: "chromium" }));
t.after(() => browser.close());
for (const lang of Object.keys(L)) for (const width of [900, 440]) {
await t.test(lang + " " + width, async () => {
const w = L[lang];
const page = await (await browser.newContext({ viewport: { width, height: 800 } })).newPage();
const errors = [];
page.on("pageerror", (e) => errors.push(e.message));
await page.route("**/*", server(lang, partners));
const sheet = await openSheet(page);
// first, under a heading that says they are sponsors
const kinds = await sheet.locator(".kind").evaluateAll((ks) => ks.map((k) => [k.querySelector("b").textContent, k.querySelector("span")?.textContent || ""]));
assert.deepEqual(kinds[0], w.head);
if (w.subs) assert.equal(kinds[1][0], w.subs);
// its rows: listed in the page's language only, one quiet line each
// with no badge, the tagline in the title
const grid = sheet.locator(".kind").first().locator("xpath=following-sibling::div[1]");
const rows = grid.locator(".tile");
assert.deepEqual(await rows.locator(".n").allTextContents(), w.rows);
const look = await rows.evaluateAll((rs) => rs.map((r) => ({ badge: r.querySelectorAll(".badge").length, over: r.scrollWidth > r.clientWidth + 1, h: r.getBoundingClientRect().height })));
for (const l of look) {
assert.equal(l.badge, 0);
assert.equal(l.over, false);
assert(l.h <= 40, "row " + l.h + "px tall");
}
const acme = rows.filter({ hasText: "Acme AI" });
assert.match(await acme.getAttribute("title"), new RegExp("Acme AI · " + w.note));
assert.match(await acme.getAttribute("title"), /api\.acme\.example\.com/);
assert.equal(await rows.filter({ hasText: "Taken" }).locator(".have").count(), 1);
// a sponsored built-in preset keeps its badge where it is
assert.equal(await sheet.locator(".tile", { hasText: "OpenRouter" }).locator(".badge").count(), 1);
// the tagline finds it
await page.locator("#addSheet input").first().fill(w.note.toLowerCase());
await sheet.locator(".kind").first().waitFor();
assert.deepEqual(await sheet.locator(".tile .n").allTextContents(), ["Acme AI"]);
await page.locator("#addSheet input").first().fill("");
// its editor shows the tagline
await sheet.locator(".tile", { hasText: "Acme AI" }).click();
await page.locator("#modal .editor .ehead .note").waitFor();
assert.equal(await page.locator("#modal .editor .ehead .note").textContent(), w.note);
assert.deepEqual(errors, []);
await page.context().close();
});
}
await t.test("no partners, no heading", async () => {
const page = await (await browser.newContext({ viewport: { width: 900, height: 800 } })).newPage();
await page.route("**/*", server("en", []));
const sheet = await openSheet(page);
const heads = await sheet.locator(".kind b").allTextContents();
assert(!heads.includes("Partners"), heads.join(","));
assert.equal(heads[0], "Subscriptions");
await page.context().close();
});
});
}
+357
View File
@@ -0,0 +1,357 @@
package provider
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"path/filepath"
"regexp"
"strings"
"sync"
"testing"
"time"
"github.com/yetone/magpie/internal/appdir"
"github.com/yetone/magpie/internal/source"
)
// Partners are services that pay to be listed first in the add sheet,
// under a heading of their own that says so. Who they are changes without
// a release: the list is fetched from PartnersURL, as the plugin market's
// is, and kept on disk. A partner is a preset in all but where it comes
// from, so adding one, its editor and its key page work as a preset's do.
// PartnersURL is where the list is kept; MAGPIE_PARTNERS moves it, and
// MAGPIE_PARTNERS=off keeps to none.
const PartnersURL = "https://usemagpie.ai/api/partners"
// KindPartner is a partner's Kind: the add sheet lists them apart.
const KindPartner Kind = "partner"
// MaxPartners is the most partners listed; the rest of a longer list are
// left out.
const MaxPartners = 12
// Partner is one entry of the list: a preset, its tagline in each
// language, the languages it is shown in, and when it is shown.
type Partner struct {
PresetDef
// IconURL is the partner's logo, kept beside the user's own pictures
// and shown as Icon.
IconURL string `json:"iconUrl,omitempty"`
// Notes is the tagline by language ("en", "zh", "zh-TW", "ja", ...);
// "en" stands in for a language without one.
Notes map[string]string `json:"notes,omitempty"`
// Langs, when set, are the only languages the partner is listed in.
Langs []string `json:"langs,omitempty"`
// From and Until bound when it is listed; zero is unbounded.
From time.Time `json:"from,omitzero"`
Until time.Time `json:"until,omitzero"`
}
// Live reports whether the partner is listed at now.
func (p Partner) Live(now time.Time) bool {
return (p.From.IsZero() || !now.Before(p.From)) && (p.Until.IsZero() || now.Before(p.Until))
}
type partnerFeed struct {
Partners []Partner `json:"partners"`
}
// partnerCache is what is kept on disk: the list fetched last, and every
// partner ever listed, so a provider added from one that has since left
// still finds its preset (its endpoints, regions and key page).
type partnerCache struct {
At time.Time `json:"at"`
Partners []Partner `json:"partners"`
Seen map[string]Partner `json:"seen,omitempty"`
}
var (
partnerMu sync.Mutex
partnerState *partnerCache // nil until read from disk
partnerNext time.Time // the next fetch, when none is running
partnerFetches bool
)
const (
partnerEvery = 6 * time.Hour
partnerRetry = 10 * time.Minute
)
var partnerID = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{1,40}$`)
func partnerCacheFile() string { return filepath.Join(appdir.Cache(), "partners.json") }
func partnersSource() string {
src := os.Getenv("MAGPIE_PARTNERS")
if src == "" {
if testing.Testing() {
// a test never asks the real site; one that wants a list
// names its own
return "off"
}
src = PartnersURL
}
return src
}
// Partners are the partners listed now, in the list's order: what was
// fetched last, never waited for. A list older than six hours is fetched
// again behind the call, and the page asked again shows it.
func Partners() []Partner {
if partnersSource() == "off" {
return nil
}
partnerMu.Lock()
defer partnerMu.Unlock()
loadPartners()
if !partnerFetches && time.Now().After(partnerNext) {
partnerFetches = true
go refreshPartners()
}
now := time.Now()
var out []Partner
for _, p := range partnerState.Partners {
if p.Live(now) {
p.Sponsored = true
out = append(out, p)
}
}
return out
}
// PartnersNow is Partners for a command that ends at once, before a fetch
// behind it could: a list due is fetched first, waited for at most wait.
func PartnersNow(wait time.Duration) []Partner {
if partnersSource() == "off" {
return nil
}
partnerMu.Lock()
loadPartners()
due := !partnerFetches && time.Now().After(partnerNext)
if due {
partnerFetches = true
}
partnerMu.Unlock()
if due {
done := make(chan struct{})
go func() { refreshPartners(); close(done) }()
select {
case <-done:
case <-time.After(wait):
}
}
return Partners()
}
// loadPartners reads the list kept on disk the first time it's asked for;
// partnerMu held.
func loadPartners() {
if partnerState != nil {
return
}
partnerState = &partnerCache{Seen: map[string]Partner{}}
b, err := os.ReadFile(partnerCacheFile())
if err != nil {
return
}
var c partnerCache
if json.Unmarshal(b, &c) != nil {
return
}
if c.Seen == nil {
c.Seen = map[string]Partner{}
}
// what was written is checked again: the rules may be stricter now
c.Partners = validPartners(c.Partners)
for id, p := range c.Seen {
if v := validPartners([]Partner{p}); len(v) == 1 && v[0].ID == id {
c.Seen[id] = v[0]
} else {
delete(c.Seen, id)
}
}
partnerState = &c
partnerNext = c.At.Add(partnerEvery)
}
// refreshPartners fetches the list and keeps it; on a failure the list
// held stays, and it is tried again in ten minutes.
func refreshPartners() {
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
list, err := fetchPartners(ctx, partnersSource())
if err == nil {
for i := range list {
list[i].Icon = partnerIcon(ctx, list[i])
}
}
partnerMu.Lock()
defer partnerMu.Unlock()
partnerFetches = false
loadPartners()
if err != nil {
partnerNext = time.Now().Add(partnerRetry)
return
}
now := time.Now()
partnerState.At, partnerState.Partners = now, list
for _, p := range list {
partnerState.Seen[p.ID] = p
}
partnerNext = now.Add(partnerEvery)
if b, err := json.Marshal(partnerState); err == nil {
_ = os.MkdirAll(filepath.Dir(partnerCacheFile()), 0o755)
tmp := partnerCacheFile() + ".tmp"
if os.WriteFile(tmp, b, 0o644) == nil {
_ = os.Rename(tmp, partnerCacheFile())
}
}
}
func fetchPartners(ctx context.Context, src string) ([]Partner, error) {
req, err := http.NewRequestWithContext(ctx, "GET", src, nil)
if err != nil {
return nil, err
}
req.Header.Set("User-Agent", "magpie")
req.Header.Set("Accept", "application/json")
res, err := source.Do(http.DefaultClient, req)
if err != nil {
return nil, err
}
defer res.Body.Close()
if res.StatusCode != http.StatusOK {
return nil, fmt.Errorf("partners: %s", res.Status)
}
var f partnerFeed
if err := json.NewDecoder(io.LimitReader(res.Body, 1<<20)).Decode(&f); err != nil {
return nil, fmt.Errorf("partners: %w", err)
}
return validPartners(f.Partners), nil
}
// validPartners keeps the entries magpie can list safely, at most
// MaxPartners: an id of its own that no built-in preset has, a name, and
// https addresses only. What a partner may not set (a decision API, no
// key, an endpoint of the user's own) is cleared.
func validPartners(in []Partner) []Partner {
var out []Partner
ids := map[string]bool{}
for _, p := range in {
if len(out) == MaxPartners {
break
}
if err := checkPartner(p); err != nil || ids[p.ID] {
continue
}
ids[p.ID] = true
p.Kind, p.Sponsored = KindPartner, false
p.Decide, p.NoKey, p.Hosts = "", false, false
p.Endpoint, p.EndpointHint, p.EndpointNeeded = "", "", ""
for i := range p.Regions {
p.Regions[i].Decide = ""
}
if p.Chat == "" && p.Responses == "" && p.Anthropic == "" {
// the first region is the default, as a built-in preset's is
r := p.Regions[0]
p.Chat, p.Responses, p.Anthropic = r.Chat, r.Responses, r.Anthropic
}
p.Note = p.Notes["en"]
if !strings.HasPrefix(p.Icon, "file:") || IconFile(strings.TrimPrefix(p.Icon, "file:")) == "" {
p.Icon = "generic"
}
out = append(out, p)
}
return out
}
func checkPartner(p Partner) error {
if !partnerID.MatchString(p.ID) {
return errors.New("bad id")
}
if builtinPreset(p.ID) != nil {
return errors.New("a built-in preset's id")
}
if strings.TrimSpace(p.Name) == "" {
return errors.New("no name")
}
if p.Chat == "" && p.Responses == "" && p.Anthropic == "" && len(p.Regions) == 0 {
return errors.New("no endpoint")
}
urls := []string{p.Chat, p.Responses, p.Anthropic, p.Website, p.KeysURL, p.IconURL}
for _, r := range p.Regions {
if r.Chat == "" && r.Responses == "" && r.Anthropic == "" {
return errors.New("a region with no endpoint")
}
urls = append(urls, r.Chat, r.Responses, r.Anthropic, r.KeysURL, r.Website)
}
for _, u := range urls {
if u != "" && !httpsURL(u) {
return fmt.Errorf("not https: %s", u)
}
}
return nil
}
func httpsURL(s string) bool {
u, err := url.Parse(s)
return err == nil && u.Scheme == "https" && u.Host != "" && u.User == nil
}
// partnerIcon keeps the partner's logo, through FetchIcon's checks, and
// returns the Icon that shows it: the one kept before when it can't be
// fetched, else the generic one.
func partnerIcon(ctx context.Context, p Partner) string {
if p.IconURL == "" {
return "generic"
}
if icon, err := fetchPartnerIcon(ctx, p.IconURL); err == nil {
return icon
}
partnerMu.Lock()
defer partnerMu.Unlock()
loadPartners()
if s, ok := partnerState.Seen[p.ID]; ok && s.IconURL == p.IconURL {
return s.Icon
}
return "generic"
}
var fetchPartnerIcon = FetchIcon
// partnerPreset is the preset of a partner listed now or before, for a
// provider added from it: one no longer listed isn't Sponsored.
func partnerPreset(id string) *PresetDef {
if partnersSource() == "off" {
return nil
}
partnerMu.Lock()
defer partnerMu.Unlock()
loadPartners()
p, ok := partnerState.Seen[id]
if !ok {
return nil
}
d := p.PresetDef
d.Sponsored = false
for _, l := range partnerState.Partners {
if l.ID == id && l.Live(time.Now()) {
d.Sponsored = true
}
}
return &d
}
// resetPartners forgets the list held in memory, for tests.
func resetPartners() {
partnerMu.Lock()
partnerState, partnerNext, partnerFetches = nil, time.Time{}, false
partnerMu.Unlock()
}
+243
View File
@@ -0,0 +1,243 @@
package provider
import (
"context"
"net/http"
"net/http/httptest"
"os"
"strings"
"sync/atomic"
"testing"
"time"
)
// fetchPartnersNow fetches the list as the background refresh does, and
// waits for it.
func fetchPartnersNow() {
partnerMu.Lock()
loadPartners()
partnerMu.Unlock()
refreshPartners()
}
func partnerFeedServer(t *testing.T, body *atomic.Value) *httptest.Server {
srv := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
if r.Header.Get("User-Agent") != "magpie" {
t.Errorf("User-Agent %q", r.Header.Get("User-Agent"))
}
_, _ = rw.Write([]byte(body.Load().(string)))
}))
t.Cleanup(srv.Close)
return srv
}
func partnerIDs(ps []Partner) string {
var ids []string
for _, p := range ps {
ids = append(ids, p.ID)
}
return strings.Join(ids, ",")
}
// The add sheet's Partners are what usemagpie.ai lists, checked: an entry
// that would send a key over http, take a built-in preset's id or set
// what only magpie's own presets may is left out, and one outside its
// window isn't listed.
func TestPartnersFromTheFeed(t *testing.T) {
t.Setenv("XDG_CACHE_HOME", t.TempDir())
resetPartners()
t.Cleanup(resetPartners)
was := fetchPartnerIcon
t.Cleanup(func() { fetchPartnerIcon = was })
fetchPartnerIcon = func(_ context.Context, u string) (string, error) {
if u != "https://acme.example/logo.png" {
t.Errorf("icon fetched from %q", u)
}
return StoreIcon([]byte("\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR"))
}
past := time.Now().Add(-time.Hour).UTC().Format(time.RFC3339)
future := time.Now().Add(time.Hour).UTC().Format(time.RFC3339)
var body atomic.Value
body.Store(`{"partners":[
{"id":"acme","name":"Acme AI","chat":"https://api.acme.example/v1","keysUrl":"https://acme.example/keys?ref=magpie",
"iconUrl":"https://acme.example/logo.png","notes":{"en":"Fast relay","zh":"快速中转"},"langs":["zh"],
"decide":"https://api.acme.example/decide","noKey":true},
{"id":"plain","name":"Plain","chat":"http://api.plain.example/v1"},
{"id":"openrouter","name":"Not OpenRouter","chat":"https://evil.example/v1"},
{"id":"Bad_Id","name":"Bad","chat":"https://bad.example/v1"},
{"id":"noname","chat":"https://x.example/v1"},
{"id":"later","name":"Later","chat":"https://later.example/v1","from":"` + future + `"},
{"id":"gone","name":"Gone","chat":"https://gone.example/v1","until":"` + past + `"},
{"id":"regioned","name":"Regioned","regions":[{"id":"cn","name":"China","chat":"https://cn.regioned.example/v1"}]},
{"id":"acme","name":"Acme twice","chat":"https://twice.example/v1"}
]}`)
t.Setenv("MAGPIE_PARTNERS", partnerFeedServer(t, &body).URL)
if got := Partners(); len(got) != 0 {
t.Fatalf("listed before any list was fetched: %v", partnerIDs(got))
}
fetchPartnersNow()
got := Partners()
if ids := partnerIDs(got); ids != "acme,regioned" {
t.Fatalf("partners = %s, want acme,regioned", ids)
}
a := got[0]
if a.Kind != KindPartner || !a.Sponsored || a.Decide != "" || a.NoKey {
t.Errorf("acme = kind %q sponsored %v decide %q noKey %v", a.Kind, a.Sponsored, a.Decide, a.NoKey)
}
if !strings.HasPrefix(a.Icon, "file:") || a.Note != "Fast relay" || a.Notes["zh"] != "快速中转" || len(a.Langs) != 1 {
t.Errorf("acme = icon %q note %q notes %v langs %v", a.Icon, a.Note, a.Notes, a.Langs)
}
if got[1].Chat != "https://cn.regioned.example/v1" {
t.Errorf("a partner with regions only is at %q, not its first region", got[1].Chat)
}
// a provider is added from it as from a preset
p, err := FromPreset("acme")
if err != nil || p.Chat != "https://api.acme.example/v1" || p.KeysURL != "https://acme.example/keys?ref=magpie" || p.Decide != "" {
t.Fatalf("FromPreset(acme) = %+v, %v", p, err)
}
if pr := Preset("acme"); pr == nil || !pr.Sponsored {
t.Fatalf("Preset(acme) = %+v", pr)
}
// a built-in preset is never a partner's
if pr := Preset("openrouter"); pr == nil || pr.Kind == KindPartner || strings.Contains(pr.Chat, "evil") {
t.Fatalf("Preset(openrouter) = %+v", pr)
}
// it leaves the list: no longer listed nor sponsored, but a provider
// added from it still finds its preset, after a restart too
body.Store(`{"partners":[]}`)
fetchPartnersNow()
if got := Partners(); len(got) != 0 {
t.Fatalf("partners after the list emptied = %s", partnerIDs(got))
}
resetPartners()
if pr := Preset("acme"); pr == nil || pr.Sponsored || pr.Chat != "https://api.acme.example/v1" {
t.Fatalf("Preset(acme) after it left = %+v", pr)
}
// a failed fetch keeps the list held, from disk after a restart
body.Store(`{"partners":[{"id":"acme","name":"Acme AI","chat":"https://api.acme.example/v1"}]}`)
fetchPartnersNow()
body.Store(`not json`)
resetPartners()
fetchPartnersNow()
if ids := partnerIDs(Partners()); ids != "acme" {
t.Fatalf("partners after a failed fetch = %q, want acme", ids)
}
}
// MAGPIE_PARTNERS=off lists none and asks nobody, whatever was kept.
func TestPartnersOff(t *testing.T) {
t.Setenv("XDG_CACHE_HOME", t.TempDir())
resetPartners()
t.Cleanup(resetPartners)
var asked atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
asked.Add(1)
_, _ = rw.Write([]byte(`{"partners":[{"id":"acme","name":"Acme","chat":"https://a.example/v1"}]}`))
}))
defer srv.Close()
t.Setenv("MAGPIE_PARTNERS", srv.URL)
fetchPartnersNow()
if _, err := os.Stat(partnerCacheFile()); err != nil {
t.Fatal(err)
}
t.Setenv("MAGPIE_PARTNERS", "off")
resetPartners()
if got := Partners(); len(got) != 0 || Preset("acme") != nil {
t.Fatalf("off listed %s", partnerIDs(got))
}
if asked.Load() != 1 {
t.Fatalf("asked %d times", asked.Load())
}
}
// Partners never blocks on the site, and asks it at most once at a time.
func TestPartnersDoesNotWait(t *testing.T) {
t.Setenv("XDG_CACHE_HOME", t.TempDir())
resetPartners()
t.Cleanup(resetPartners)
release := make(chan struct{})
var asked atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
asked.Add(1)
<-release
_, _ = rw.Write([]byte(`{"partners":[{"id":"acme","name":"Acme","chat":"https://a.example/v1"}]}`))
}))
t.Cleanup(srv.Close)
t.Cleanup(func() {
// the fetch ends with the test
close(release)
for {
partnerMu.Lock()
busy := partnerFetches
partnerMu.Unlock()
if !busy {
return
}
time.Sleep(5 * time.Millisecond)
}
})
t.Setenv("MAGPIE_PARTNERS", srv.URL)
start := time.Now()
for range 5 {
Partners()
}
if d := time.Since(start); d > time.Second {
t.Fatalf("Partners waited %v", d)
}
deadline := time.Now().Add(5 * time.Second)
for asked.Load() == 0 && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
time.Sleep(50 * time.Millisecond)
if n := asked.Load(); n != 1 {
t.Fatalf("asked %d times, want 1", n)
}
}
// PartnersNow, for a command that ends at once, has the list fetched before
// it answers, and doesn't wait past its bound.
func TestPartnersNow(t *testing.T) {
t.Setenv("XDG_CACHE_HOME", t.TempDir())
resetPartners()
t.Cleanup(resetPartners)
var body atomic.Value
body.Store(`{"partners":[{"id":"acme","name":"Acme","chat":"https://a.example/v1"}]}`)
t.Setenv("MAGPIE_PARTNERS", partnerFeedServer(t, &body).URL)
if ids := partnerIDs(PartnersNow(5 * time.Second)); ids != "acme" {
t.Fatalf("PartnersNow = %q, want acme", ids)
}
resetPartners()
release := make(chan struct{})
srv := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) { <-release }))
t.Cleanup(srv.Close)
t.Cleanup(func() {
close(release)
for {
partnerMu.Lock()
busy := partnerFetches
partnerMu.Unlock()
if !busy {
return
}
time.Sleep(5 * time.Millisecond)
}
})
t.Setenv("MAGPIE_PARTNERS", srv.URL)
// the list kept is due again
partnerMu.Lock()
loadPartners()
partnerNext = time.Time{}
partnerMu.Unlock()
start := time.Now()
if ids := partnerIDs(PartnersNow(100 * time.Millisecond)); ids != "acme" {
t.Fatalf("PartnersNow while the site hangs = %q, want the kept acme", ids)
}
if d := time.Since(start); d < 100*time.Millisecond || d > 2*time.Second {
t.Fatalf("PartnersNow waited %v, want its bound", d)
}
}
+10 -1
View File
@@ -543,8 +543,17 @@ var presetAliases = map[string]struct{ preset, region string }{
}
// Preset finds a preset by id, or by an id it carried before
// (presetAliases).
// (presetAliases), or a partner listed now or before (partners.go).
func Preset(id string) *PresetDef {
if pr := builtinPreset(id); pr != nil {
return pr
}
return partnerPreset(id)
}
// builtinPreset is the preset of that id built into magpie, partners'
// aside.
func builtinPreset(id string) *PresetDef {
if a, ok := presetAliases[id]; ok {
id = a.preset
}
+3
View File
@@ -402,6 +402,9 @@ func (m *model) openProviderModels(id string) {
// openPresets picks a vendor magpie knows, then asks for its key.
func (m *model) openPresets() {
var items []agent.Option
for _, d := range provider.Partners() {
items = append(items, agent.Option{Value: d.ID, Note: d.Name + " · partner (sponsor)"})
}
for _, d := range provider.Presets() {
items = append(items, agent.Option{Value: d.ID, Note: d.Name + " · " + string(d.Kind)})
}
+8 -3
View File
@@ -190,13 +190,18 @@ func presets() error {
have[p.ID], have[p.Preset] = true, true
}
kind := provider.Kind("")
for _, pr := range provider.Presets() {
// partners first, as the app lists them: their heading says they pay
all := []provider.PresetDef{}
for _, pa := range provider.PartnersNow(3 * time.Second) {
all = append(all, pa.PresetDef)
}
for _, pr := range append(all, provider.Presets()...) {
if pr.Kind != kind {
kind = pr.Kind
fmt.Println(faint.Render(" " + map[provider.Kind]string{provider.KindVendor: "vendors", provider.KindRelay: "relays", provider.KindLocal: "local"}[kind]))
fmt.Println(faint.Render(" " + map[provider.Kind]string{provider.KindPartner: "partners (sponsors)", provider.KindVendor: "vendors", provider.KindRelay: "relays", provider.KindLocal: "local"}[kind]))
}
name := bold.Render(pr.Name)
if pr.Sponsored {
if pr.Sponsored && pr.Kind != provider.KindPartner {
name += " " + faint.Render("sponsored")
}
state := muted.Render("magpie provider add " + pr.ID + " <key>")
+29
View File
@@ -0,0 +1,29 @@
// The add sheet's Partners: the services that pay to be listed first in
// magpie's "Add provider" sheet, under a heading that says they sponsor
// magpie. The app fetches this list from /api/partners every six hours and
// keeps it (internal/provider/partners.go), so a change here reaches users
// with `npx wrangler deploy`, no release. At most 12 are listed, in this
// order.
//
// An entry is a preset (internal/provider/presets.go) and a few fields more:
//
// id a-z, 0-9 and "-", 2-41 characters; never a built-in preset's
// id. It is kept in the providers users add from it: don't
// change it, and don't give it to someone else later.
// name the row's name; short a shorter one, when name is long
// chat OpenAI chat completions base URL (…/v1)
// responses OpenAI Responses base URL; anthropic Anthropic messages base URL
// (at least one of the three, or regions)
// regions [{id, name, chat, responses, anthropic, keysUrl, website}]: the
// first is the default
// keysUrl where the user makes a key (a referral link is fine)
// website the service's home page
// iconUrl its logo (png, svg, webp, jpeg; square), fetched and kept by
// each app
// notes the tagline by language: {en, zh, "zh-TW", ja, de}; en stands
// in for a language without one
// langs the app languages it is listed in (["zh"]); none is all
// from, until RFC 3339 times bounding when it is listed (the paid term)
//
// Every URL is https. An entry that breaks a rule is left out by the app.
export const PARTNERS = [];
+5
View File
@@ -8,6 +8,7 @@
// Both take ?lang=zh for the notes in Chinese, where a
// release has them (below its <!-- lang:zh --> marker);
// any other lang, or none, is the English alone.
// /api/partners {partners: [...]}: the add sheet's Partners (partners.js)
// /download the Apple Silicon dmg
// /download/mac-arm64 the same; /download/mac-intel the Intel dmg
// /download/windows the Windows app (x64); /download/windows-arm64
@@ -22,6 +23,7 @@
// Everything else is the static site in public/.
import { LANGS } from "./i18n.js";
import { PARTNERS } from "./partners.js";
const REPO = "yetone/magpie-releases";
const TTL = 300; // seconds the newest release is remembered
@@ -56,6 +58,9 @@ export default {
const lang = url.searchParams.get("lang");
return json({ ...got.rel, notes: inLang(got.rel.notes, lang) }, 200, cacheFor(got));
}
if (url.pathname === "/api/partners") {
return json({ partners: PARTNERS }, 200, { "Cache-Control": "public, max-age=600" });
}
if (url.pathname === "/api/notes") {
const got = await releases(ctx, env);
if (!got) return json({ error: "no releases" }, 503, NO_STORE);
+23
View File
@@ -270,3 +270,26 @@ test("docs: an English docs page sends a browser that prefers Chinese or Japanes
const res = await worker.fetch(new Request("https://usemagpie.ai/docs/zh/start", { headers: { Cookie: "lang=en" } }), env, ctx);
assert.equal(res.status, 200);
});
// /api/partners is partners.js, cached ten minutes; every entry keeps the
// rules the app checks (internal/provider/partners.go), so none is dropped
// there unseen.
test("partners", async () => {
const { PARTNERS } = await import("./partners.js");
const res = await ask("/api/partners");
assert.equal(res.status, 200);
assert.equal(res.headers.get("Cache-Control"), "public, max-age=600");
assert.deepEqual(await res.json(), { partners: PARTNERS });
assert(PARTNERS.length <= 12);
const ids = new Set();
for (const p of PARTNERS) {
assert.match(p.id, /^[a-z0-9][a-z0-9-]{1,40}$/, p.id);
assert(!ids.has(p.id), p.id + " twice");
ids.add(p.id);
assert(p.name?.trim(), p.id + " has no name");
assert(p.chat || p.responses || p.anthropic || p.regions?.length, p.id + " has no endpoint");
const urls = [p.chat, p.responses, p.anthropic, p.website, p.keysUrl, p.iconUrl, ...(p.regions || []).flatMap((r) => [r.chat, r.responses, r.anthropic, r.keysUrl, r.website])];
for (const u of urls.filter(Boolean)) assert.equal(new URL(u).protocol, "https:", p.id + ": " + u);
for (const t of [p.from, p.until].filter(Boolean)) assert(!isNaN(Date.parse(t)), p.id + ": " + t);
}
});