wizzy-yangandwizzy-yang
e95852e301
usage, gui: 用量页补上各 agent 会话文件里的用量，请求页按「经过 / 不经过 magpie」拆开 (#948)
* sessions: dsh、ZCode、WorkBuddy 的会话文件里有各自花掉的 token，用量台账从来不读
Sessions 页面一直都在读这些 agent 自己的会话文件，用量台账没有：用量页只显示网关本
身应答过的调用——也就是装上 magpie 之前的什么都没有，而一个直接指向厂商的 dsh 更
是什么都没有。本机上装 magpie 之前就已经在用这些 agent（用它们自带的模型），那段
用量一直对不上账。
* 新增三类"整文件读取"的调用来源，走 wholeCallReaders（它们的调用不是一行一条，
或者文件不能从中间接着读）：dsh 的会话是 zstd 分帧追加的；ZCode 用的是 OpenCode
那几张表；WorkBuddy 的同一句回复会在多行里重复同一个 messageId。三者的 token
语义各自换算成 Call 的 Input/Output/CacheRead/CacheWrite，调用缓存跟着记一份（v9）。
* 每个来源都跳过经由 magpie 网关发出的调用：网关自己记过那一本账，还带着真正应答
的供应商和账号；dsh 发往网关的请求不带会话头，两边事后对不上（token 数每次不同、
又没有可配对的 id），硬读就是同一通调用记两次。dsh 认它被配置去的那个 provider
（dshGatewayProvider：magpie / dial），ZCode 和 OpenCode 认路由里写的 provider
（ocGatewayProvider）。WorkBuddy 那部分见下一个提交。
* WorkBuddy 的调用里，同一句回复的多行只记最后一次（它带这个回复的总量）；而它从
旧历史搬过来的回复一条 messageId 都没有，每一条都是独立的一通调用——会话汇总的
读法（workbuddyLine）就是这么数的，这里照它来，否则台账会比 Sessions 页面少算，
正是本次要消灭的那类"两页对不上"，只是方向相反。
* dsh 的一通调用保留它文件里记的两样东西：被配置去的供应商（Upstream，作为历史，
从来不当成路由的证据），以及厂商真正应答的模型——回复的 replayState 里那个和
请求模型不一致时，Model 记应答的、Requested 记被请求的，和 Claude Code 的文件
一样的读法，好让请求表的若干列有东西可填。
* 会话的 fork：dsh 的会话从另一个会话续起来时会带上对方已写入的事件（isSeeded /
seedLength，session/end-seed 标记之后的不算），parser 跳过的这里同样跳过，否则
同一通调用会在每个 fork 上再花一次。
* WorkBuddy 的 subagents/ 子目录是父会话自己的调用来源（它带自己的消息 id，会话
归到父会话名下）；调用这条路不读 file.main，那个标记是 Sessions 页面给"会话自己的
文件、不是子代理的"用的。
夹具：dsh 的三个会话（普通、带供应商与应答模型、以及一通经网关的），ZCode 的两条
（其中 ses_zcA 那条的 providerID 是 magpie，应被跳过），WorkBuddy 的既有会话（m1 的
usage 同时落在 assistant message 和它最后一个 function_call 上，只能记一次；旧历史
三条无 messageId 的回复各记一次）和一通经网关、一通走用户自建端点的会话。
调用缓存升到 v9：v8 里既没有 dsh 的供应商和应答模型，也没有"跳过网关调用"这件事，
旧缓存会继续把旧账 Serve 出来。
TestDshCalls、TestZCodeCalls、TestWorkBuddyCalls 是新的；把跳过网关调用那几行、以及
WorkBuddy 旧历史那条判据去掉，它们必挂、加上必过（都实际跑过反向验证）。
验证：go vet；go test ./internal/sessions（只余 TestCursor，Windows 路径前缀问题，
改动前同样失败）；go build（windows）。
* usage: 请求页把"magpie 转的"和"它只读到的"分成两个数，GUI 跟着拆成两栏
用量页一直只显示网关自己应答过的调用，而现在它还要读 agent 自己会话文件里的调用：
两本账混在一行里，读者分不清哪一段是 magpie 转的、哪一段只是它从文件里读到的，
而总数事实上几乎全部来自后者（本机 30 天 847M token 里，网关只转了 2.4%）。Summary
里的两个来源卡片也加不出这个数来：它们只算 input 和 output，上面的 KPI 把缓存的
token 也算在内，于是"全部请求"读作 13.6M，而"Tokens"是 847M。
* usage.RequestPage 和 usage.Filter 各多一样东西：前者在 Sum 之外带上 Through 和
Direct（网关应答的，和从会话文件读到的），后者多一个 Through 维度。两种建页路径
（流式的 queryPage 和治疗桌用的 pageFromLedger）都做同一个拆分，两边一致——
TestCompactPageMatchesLedger 现在也覆盖两个来源筛选。
* 每一行本来就是按 Source 区分这两类的（"log" 是会话文件里的调用，其余走的网关），
拆分就落在它上面；网关自己拒掉的调用两边都不算（页面在总数下面也这么说），所以
SourceOf 对它返回 ""，点哪一栏都同样把它排除，两栏自己的计数本来也不含它。
* 两栏自己的数字是在"来源"这个挑选被清空的情况下数的，和各维度分组一样：三栏是读者
来回切换的地方，各自得一直报着自己的数，否则点了其中一栏另一栏读作 0，且没有回
到全部的路。
* /api/usage/requests 支持 via=through|direct；GUI 那一侧的三个格子、计数单位、
方块样式在下一个提交里。
* Filter.Via 这个名字和 Record.Via（转发调用的那台电脑，#542）重了，改叫 Filter.Through。
Sessions 页列表长度那件事（sessionListLimit）拆到单独的 PR 里，不在这里。
验证：go vet；go test ./internal/usage（只余 TestOTelRetryAfterBound，计时的 ±30ms，
origin/main 同样失败）；go build（windows）。
* gui: 请求页把来源拆成三栏，汇总那一排也改成方块
用途页的请求栏一直只有一本账（Sum），而现在它既显示网关应答的调用，也显示从各 agent
自己会话文件里读到的调用。三个格子画在上方汇总那一排方块里，样式一致：三张各自带
边框、彼此留缝的卡片，窄屏（≤620px）下竖排。
* 每栏是一个 button：点一下就只列这一来源的请求，再点一下、或点"全部请求"回到
全部；被选中的那一栏底下有一道本主题色的线。aria-pressed 标出当前选中的是哪一栏。
* 计数后面跟单位（"147 请求"），免得一个光秃秃的数字被读成下面的 token；token 是
四项合计，和上面 KPI 的算法一致。
* 三个格子在重绘之间是同一批元素、原地改（class / aria / 文本 / onclick），不是每次
replaceChildren 重建——同一份 app.js 里 5810 行的规定就是这么写的（"The switch is
the same element across redraws, changed in place, so one being pressed or focused is
never swapped out"），正在点的那个不该被换掉。
* 汇总那一排本身（五张 KPI 卡）改成同样的方块：各自一张卡、有缝，边框写成
var(--line) 而不是 --card-edge（那是面板用的变量，这些卡在独立窗口里也有）。
≤860px 两列排布时卡片之间的那几条线照旧：奇数位去掉左边框、第三张起加上边框，
是 e0eaf5b4 带 #799 的老规则，这里只是跟着搬了位置，没有删。三张来源格是 button，
不参与这套分隔线。
* 供应商·账号 一栏：来自会话文件的行显示它文件里记的那个供应商 id（Codex 一向
如此），有账号时仍是账号优先，什么都没有才说"本地会话"。"本地会话"角标的条件
跟着放宽到账号或供应商任一存在，提示分两种说法：文件记了供应商的，说明这是它
被配置去调用的那个记录、不是 magpie 看到的路由（不等于官方供应商）；只认得出
账号的，沿用原来那句"仅当本地元数据能明确识别时显示账号"。
* 窗口参数支持 via，用后从地址栏清掉，和 tab/provider/agent 等一致。
* 新增的串子在 en/zh/ja/de 四种语言里都有（"{n} tokens" 上游已有，不重复加，
i18n-keys 绿）。
验证：node --check（app.js、i18n.js）；go vet；go build（windows）。
界面效果在本机用改动后的源码构建的二进制上过目：三栏的切换与计数、方块样式、
窄屏竖排、请求表新增的会话文件行（含供应商 id）。
浏览器套件（chromium 与 webkit、en/zh）：session-identity、usage-ledger、
ledger-protocol、number-units、usage-ledger-chart、usage-ttft-rounding、
usage-chart-axis、usage-fold、web-page、usage-accounts、i18n-keys 全绿；
ledger-hscroll、usage-speed、click-scroll 全绿；gui-ja、gui-de 各一条失败、
usage-ledger-detail 六条失败，均在本机 origin/main 上同样失败，与本次改动无关。
usage-ledger-chart 新钉了 ≤620px 下 KPI 卡与来源格的分隔线（把 app.css 换回上游版本
它必挂，实测过）。
---------
Co-authored-by: wizzy-yang <wizzy-yang@users.noreply.github.com>
2026-10-07 19:43:01 +08:00
..
usage: each subscription account's tokens and cost, by the account that answered (#557: 希望用量账本能按订阅账号统计，每个 Codex / Claude 账号各用了多少 token、多少钱) — a call's record named the upstream key it went out on (ProviderKeyID/Name, #500) but never the subscription account, so with two Codex accounts signed in, or a Claude account that took over after a failover, the ledger could say what Codex cost and not what each account did. A record now carries providerAccount, the account that answered as the Routing trace names it (Account.User: its email or login, never a token), set by the gateway at the try that went out (the candidate loop's last try, so a failover's answer is the account that took over and the refused try stays the first account's; an X-Magpie-Account pin is the pinned one), and on the decide, draw and video paths too. Record.Account() gives it, or for a record written before the field the account its host named (Where() has long written "chatgpt.com as dee@example.com" for an account), and otherwise "" — shown as "account not recorded", never inferred from today's sign-in. The summary groups calls by provider and account (Summary.Accounts, the same email on Codex and on Claude being two accounts), priced as every other group; the ledger's Filter has Account, the search finds a call by its account, the CSV has a provider_account column beside provider_key_name (the provider's identity columns together, the caller's still last), and the Requests page's packed rows keep it and offer the period's accounts. magpie usage prints an accounts table after the provider keys, and magpie usage --csv --account <name> keeps one account's calls. The app's Usage page lists Accounts on the Overview (tokens and cost per account, the provider under its name, an unrecorded one in the faint colour, no stripe down a row's side) and has an Account filter on Requests, each account noting the providers it answered for, which asks for and exports that account's calls without moving the page. OTLP export is an allowlist and does not carry the account. TestUsageNamesPinnedAccount (both pinned accounts on magpie's endpoint and on Codex's own, the trace's Who and the record agreeing, no token in usage.jsonl) and TestUsageNamesAccountThatTookOver (Ordered routing, the first account refusing: the answer recorded as the spare, the refusal as the first) fail without the gateway change; TestSummaryByAccount, TestLedgerByAccount and TestRequestPageByAccount cover the grouping, the host fallback, "not recorded", keyed providers kept out, the filter, the search, the CSV column and the packed rows; TestLedger and TestLedgerWithSessionLogCalls take the new column. usage-accounts.test.cjs (WebKit and Chromium, en and zh: the Overview's accounts and costs, the Account picker's notes, its request and export, All accounts clearing it, scrollTop unchanged) fails without the asset change; api-key-usage, click-scroll, panel-usage, request-archive-usage, tray-usages, claude-usage-asked and every usage-* test pass in WebKit. go vet ./internal/..., go build -tags nogui (darwin, GOOS=linux), GOOS=windows go build, go test -tags nogui -count=1 ./... under a temp HOME pass (TestFXInState failed once on its TempDir cleanup, passes alone).
2026-10-02 20:15:33 +08:00
archive: a request the archive kept is opened and downloaded from the Usage page, with its bodies whole (jorben, #447: 请求存档 reachable from usage history, full bodies, download) — the archive's name was only on the Gateway page's recent calls, gone once a call left the list or the gateway restarted, and the archive kept the first 256 KB of each body, the recent list's cut. The usage ledger's row now names the call's archive (archive in usage.jsonl, kept through the Requests page's packed rows), and its details on Usage → Requests show the Request archive panel, Fetch from archive and Download; with the archive on, a request it has no copy of says "Not archived". The bodies are kept whole up to requestArchiveMaxMB in settings.json (32 MB unless set, at most 1024): the response is teed to a temp file as it is sent, the request body written to one when the call ends, so the upload queue holds file handles, not bodies, and one upload at a time reads them; a body past the limit is kept to it and marked cut, with its whole size. What goes upstream is unchanged, the archive stays off unless turned on, headers and bodies are scrubbed as before, and the temp files are removed after the upload or when the queue is full. The page is sent a body to show only when it is 256 KB or less (a longer one is said by its size), an archive over 80 MB is only said to be; Download streams the object whole, to Downloads in the app (archive/export, -2, -3… for a name taken) and as the browser's download in magpie web (archive/file). TestArchiveWholeBodies, TestArchiveLimit, TestRowKeepsArchive and TestArchivePreviewAndDownload fail without the change; TestArchiveOffUsage, TestArchiveTooLargeToPreview, TestArchiveOldPreview keep the rest. go vet, go build -tags nogui (darwin, GOOS=linux), GOOS=windows go build, go test -tags nogui ./... pass; request-archive-usage (new), request-archive, click-scroll, old-webkit, usage-ledger, usage-ledger-detail, usage-ledger-content, web-page, sse-body pass.
2026-10-02 00:57:51 +08:00
2026-10-07 02:14:29 +08:00
2026-10-07 02:14:29 +08:00
2026-10-07 02:14:29 +08:00
2026-10-07 02:14:29 +08:00
2026-10-07 02:14:29 +08:00
2026-10-07 02:14:29 +08:00
2026-10-02 19:15:56 +08:00
2026-10-07 02:14:29 +08:00
2026-10-07 02:14:29 +08:00
2026-10-01 16:18:02 +08:00
2026-10-07 02:14:29 +08:00
2026-10-07 19:43:01 +08:00
2026-10-02 22:00:10 +08:00
2026-10-02 22:00:10 +08:00
2026-10-03 01:15:29 +08:00
2026-10-03 02:19:11 +08:00
2026-10-03 02:19:11 +08:00
2026-10-03 01:15:29 +08:00
2026-10-03 01:15:29 +08:00
2026-10-03 01:15:29 +08:00
2026-10-03 01:15:29 +08:00
2026-10-04 23:30:56 +08:00
2026-10-03 21:31:04 +08:00
2026-10-04 23:30:56 +08:00
2026-10-05 21:56:59 +08:00
2026-10-06 13:32:03 +08:00
2026-10-05 21:56:59 +08:00
2026-10-03 11:21:10 +08:00
usage, gui: Usage › Requests' All opens at once after magpie has been up a few seconds, and in half the time before (the owner) — Requests on All stayed a skeleton for 39 s on the owner's magpie right after it started: the first All parses every session file (12.3k here: 11k Codex sessions, 1.2k Claude Code projects' files), one after the other, each a wait on the disk. Those a query has to parse are now read 8 at a time ahead of the loop that takes them in order (prefetchSources), at most 32 waiting, so the history is never all held at once; and magpie reads the whole history in the background once it is up, right after the Sessions page's own warm-up and not beside it on the disk, so the page opens on the kept index. Tested in a sandbox HOME on a copy of the owner's real ~/.codex and ~/.claude history (magpie app, dev UI): the first All right at startup took 10.6 s before and 4.6–4.9 s now; opened 20 s after startup it took 10.0 s before and 0.01 s now; later Alls 0.3–0.4 s as before. TestPrefetchSources (each source handed back as its own, read in parallel, no more than the window ahead, every one read once, an unwanted one read when asked) is new. go vet, go test -race ./internal/usage, go build (darwin, GOOS=linux nogui, GOOS=windows) and go test -tags nogui ./... under a temp HOME pass.
2026-10-05 20:18:46 +08:00
2026-10-07 02:14:29 +08:00
2026-10-07 02:14:29 +08:00
2026-10-05 14:26:23 +08:00
2026-10-07 06:17:56 +08:00
2026-10-07 02:14:29 +08:00
2026-10-01 17:42:10 +08:00
2026-10-01 17:42:10 +08:00
2026-10-07 19:43:01 +08:00
2026-10-07 19:43:01 +08:00
2026-10-07 19:43:01 +08:00
routing: a group can let a member go that hasn't begun answering after a set time and ask the next, and a request's detail splits its first token into magpie's time and the vendor's (Chen on X: answers through magpie felt slow, with no way to tell whether magpie or the vendor was the wait) — a group's editor has a Slow to start row (Wait, 30 s, 1 min, 2 min; Wait, the default, keeps today's behaviour). With a time set, a streamed request to a member that has sent no text, reasoning or tool call by then is cancelled before any of its bytes reach the agent (the held start is dropped; keepalives go on meanwhile) and the next member is asked; the try is marked slow ("slow to start", narrated in the route), and the member doesn't rest, since nothing is wrong with it. The last member left is always waited for. The card says "Next after 1 min without a first token". Each request now records when it went to the vendor (httptrace WroteRequest, sent_ms), and its detail shows In magpie and Vendor's first token beside First token; after a failover In magpie includes the time spent on the members let go. With the owner's real DeepSeek key in a sandbox, a group [a local member that sends its role chunk then stalls 60 s, deepseek-flash] with 3 s: the answer came from DeepSeek in 4.4 s, none of the stalled member's stream reached the client, its try read slow / "no first token in 3s" with no rest, and the ledger had ttft 3622, sent 3003. TestGroupFirstTokenFailsOver (failover, the last one waited for, one in time kept; passes -race) and group-first-token.test.cjs and usage-ledger-detail.test.cjs (WebKit, en and zh) fail on the old code. group-from-model-add's narrow-window case fails the same on origin/main; Chromium doesn't launch in this sandbox. go vet, builds (darwin, GOOS=linux nogui, GOOS=windows) and go test -tags nogui -count=1 ./... under a temp HOME pass (TestProvidersAnswerWhileListsComeIn flaked once, passes alone).
2026-10-04 19:33:55 +08:00
2026-10-03 22:45:42 +08:00
2026-10-07 02:14:29 +08:00
usage, gateway: a model the vendor spells with other separators isn't marked as another one (vincentzhang on Discord: 这个说明有点奇怪。看不明白。) — Volcengine Ark answers deepseek-v4.1-flash as deepseek-v4-1-flash, the dots written as hyphens, and the Usage page's Requests and the Routing page marked every such row amber 「实际 deepseek-v4-1-flash」, saying another model had answered, not the one asked for under a dated name. Swapped now compares the bare names without their separators (. - _ and spaces) and case, so deepseek-v4-1-flash, deepseek_v4_1_flash, GLM-4-6 for glm-4.6 and gpt-4-1-2025-04-14 for gpt-4.1 are the model asked for, while deepseek-v4-flash or deepseek-v4.1-pro for deepseek-v4.1-flash still are another. Rows and routes kept marked swapped before are read back as not swapped when the names differ only so (usage.SameSpelled, in rowChunk.row and Route.routedAgain). The tag's explanation says it in plainer words: asked for {sent}, the reply says {served} answered, likely another model; the same model under a dated name or spelled otherwise isn't marked (zh, ja, de). TestSwappedIgnoresSeparators fails on the old code; TestKeptRowSameSpelledNotSwapped, TestRouteKeptSpelledOtherwiseNotSwapped. usage-ledger (its expected title updated), remote-group-served, panel-routing, gui-ja and gui-de pass on WebKit. go vet, builds (darwin, GOOS=linux nogui, GOOS=windows) and go test -tags nogui -count=1 ./... under a temp HOME pass (TestCodexVersion timed out once under load and passes alone).
2026-10-05 12:49:59 +08:00
usage, gateway: a model the vendor spells with other separators isn't marked as another one (vincentzhang on Discord: 这个说明有点奇怪。看不明白。) — Volcengine Ark answers deepseek-v4.1-flash as deepseek-v4-1-flash, the dots written as hyphens, and the Usage page's Requests and the Routing page marked every such row amber 「实际 deepseek-v4-1-flash」, saying another model had answered, not the one asked for under a dated name. Swapped now compares the bare names without their separators (. - _ and spaces) and case, so deepseek-v4-1-flash, deepseek_v4_1_flash, GLM-4-6 for glm-4.6 and gpt-4-1-2025-04-14 for gpt-4.1 are the model asked for, while deepseek-v4-flash or deepseek-v4.1-pro for deepseek-v4.1-flash still are another. Rows and routes kept marked swapped before are read back as not swapped when the names differ only so (usage.SameSpelled, in rowChunk.row and Route.routedAgain). The tag's explanation says it in plainer words: asked for {sent}, the reply says {served} answered, likely another model; the same model under a dated name or spelled otherwise isn't marked (zh, ja, de). TestSwappedIgnoresSeparators fails on the old code; TestKeptRowSameSpelledNotSwapped, TestRouteKeptSpelledOtherwiseNotSwapped. usage-ledger (its expected title updated), remote-group-served, panel-routing, gui-ja and gui-de pass on WebKit. go vet, builds (darwin, GOOS=linux nogui, GOOS=windows) and go test -tags nogui -count=1 ./... under a temp HOME pass (TestCodexVersion timed out once under load and passes alone).
2026-10-05 12:49:59 +08:00
2026-10-01 17:42:10 +08:00
2026-10-01 17:42:10 +08:00
2026-10-03 23:44:06 +08:00
2026-10-01 17:42:10 +08:00
usage: with WebDAV or S3 sync, each computer shares its usage and sees the others', the total and each computer's part (#542) — sync carried keys, agents and the library, and each computer's Usage page counted only the calls made on it. Sync now has a Usage tick (off unless picked, usage=yes in magpie webdav/magpie s3): a computer with it on puts its own calls up as one file a day, magpie/usage/<id>-<day>.magpie-usage beside the bundle (under the prefix's usage/ on S3), each a gzipped list of times, agents, providers, models, tokens, costs and statuses, sealed with the sync passphrase (backup.SealData, the key cached per passphrase so a sync derives it once); no request body, session, archive name or key goes, a call another computer passed on over the LAN (Via) isn't put up again, and an error is cut to 300 characters. The id is a random one kept in usage-computer.json, which no backup carries, with the host name beside it. A computer only ever writes and deletes its own names, so none overwrites another's; the files are not in the bundle and its hash, so sharing usage never makes a sync or a conflict of the bundle; the first sync puts up the last 90 days, after that only yesterday and today, at most every 15 minutes (a Save or Sync now at once), and a day past 90 is deleted by the computer it is of. The others' files, listed (PROPFIND Depth 1; ListObjectsV2) and fetched only when changed, are kept in usage-others and read with the ledger, as Record.Computer; a failure is said in the sync row ("Couldn't share usage") and never fails the sync. The Requests page and the panel's Usage tab get a Computer pick (this computer, other computers when more than one, each by name) sending computer=this|others|<id>; a row from another computer says "· on <name>" beside its agent, and its details name the computer and say what was said stays there, nothing asked of this computer's session files for it. With no other computer's usage there is no pick and nothing new on either page, and an older magpie never lists or reads usage/. TestUsageSharedBetweenComputers (computers a and b over a fake WebDAV under httptest: each sees 4 calls, the totals and its own and the other's part, nothing counted twice over repeated syncs, the bundle not put again for usage, a new call reaching the other, a day past the window deleted), TestUsageSharedThroughS3 (the same through a fake S3, signed list and delete, a prefix with a space and a plus) and TestUsageOffAsksNothing are new and don't build without the change; usage-computers.test.cjs (WebKit, en and zh: no pick without other computers, the pick and computer=this, another computer's row and details with no /content asked, the panel's pick, the sync form's tick posting usage: true, no click moving the page) fails without it on all but the no-other-computers case. s3-sync, sync-other-kind, panel-usage, request-archive-usage, click-scroll and usage-ledger-chart pass in WebKit. go vet ./internal/..., go build -tags nogui (darwin, GOOS=linux), GOOS=windows go build, go test -tags nogui -count=1 ./... under a temp HOME pass (TestFXInState failed once on its TempDir cleanup, passes alone). Turning the Usage tick off, or sync off, lets the other computers' usage kept here go (DropAllShared), so the Usage page counts this computer's alone again: TestUsageSharedBetweenComputers checks both, and fails without it.
2026-10-02 17:47:36 +08:00
2026-10-07 02:14:29 +08:00
2026-10-07 02:14:29 +08:00
2026-10-07 03:03:06 +08:00
2026-10-07 02:14:29 +08:00