Commit Graph
6 Commits
Author SHA1 Message Date
yetone 76f3b97c09 sync: it can sync by itself less often or only when asked, and the server's setup can be restored over this computer's, undoably (Sun1090, #847: 同步太快了,需要设置个开关控制自动同步,还有恢复按钮) — the gateway's magpie synced every 3 minutes with no way to change it, and a free 坚果云 account is allowed only so many WebDAV requests (600 per 30 minutes, 1 GB up and 3 GB down a month); and there was no way to take the server's setup back over a computer's own, which the next sync would otherwise push up. Settings › Sync now has Sync by itself, the app's segmented control (no <select>): Off, 3 min (the default, as before), 15 min, 30 min, 1 hour; Off syncs only on Sync now, and the sync row says only when asked. The loop reads the setting each minute (no request), so a change takes effect without a restart, and a server limiting requests is still waited for as backoff says. Restore from the server asks first in the app's own dialog, naming the parts replaced; it reads the server's file as it is now, seals this computer's setup (keys and all) into the sync folder as <time>-before-restore (newest 5 kept), brings in each part that differs as a sync brings one changed only there, and leaves the state as such a sync would, so the next sync pushes nothing back. Its notice says what was restored, with Undo, which brings the kept setup back (the next sync then takes it to the others); OK, or a later sync's own notice, drops the undo. The form's Save keeps the setting (SetAuto alone sets it). CLI: magpie webdav|s3 auto off|3|15|30|60, restore, undo; the status says how often it syncs by itself. zh, ja, de strings. TestAutoSync (off: no request by itself; back to 15 min: one sync, then none; Save keeps it; 7 refused) and TestRestore (b's own changes replaced by the server's with no push, the kept copy holds b's provider and key, Undo puts it back and a gets it, nothing differing restores nothing) pass -race; sync-auto-restore.test.cjs (WebKit, en and zh: Off posts -1 and the rows say so, Cancel posts nothing, Restore and Undo post, no click moves the page) fails on the old code. sync-other-kind, sync-spin, s3-sync, webdav-settings-refresh, usage-computers, click-scroll, gui-ja, gui-de and settings-sections pass on WebKit. go vet, builds (darwin, GOOS=linux nogui, GOOS=windows) and go test -tags nogui -count=1 ./... under a temp HOME pass. 2026-10-05 10:36:46 +08:00
yetone f1224a055e sync: a backup on a slow line is no longer cut off at 2 minutes; a server that stops moving is (#657, irx999: a 43 MB backup with many skills to a WebDAV server on a 2.4 Mbps VPS, 305 KB/s, needs 2.4 minutes and every sync failed) — a sync had a fixed deadline in all (2 minutes from the gateway's loop and magpie webdav/s3 sync, 90 s from Settings › Sync now), so a large backup on a slow line never got through, however steadily it moved. WebDAV and S3 requests now go through a watchdog instead (stall.go): the backup's GET and PUT fail once nothing has moved either way for 60 s, each byte of the body sent or read putting it back, and after a PUT's body is all handed over the answer gets 60 s on top of as long as the write took (at least 4 minutes), for what still sits in the system's buffers to leave; the other requests (PROPFIND, MKCOL, HEAD, MOVE, DELETE) have 30 s in all. A stall says so ("the server sent and took nothing for 1m0s (PUT)"). Sync now and the CLI have no deadline; the gateway's loop keeps an hour as a backstop; a wait for another magpie's sync stays 30 s. TestSlowTransferIsNotCutShort (20 MB up and down at a steady trickle, over 5 times the watchdog each way), TestStalledTransferFailsPromptly (a server stopping mid-PUT and mid-GET fails within a second) and TestMetadataRequestHasALimit fail on the old code (no watchdog: the stalled ones hang) and pass. go vet, builds (darwin, GOOS=linux, GOOS=windows) and go test -tags nogui -count=1 ./... under a temp HOME pass. Not tried against a real slow WebDAV server. 2026-10-03 16:10:02 +08:00
yetone da834411b5 usage: with WebDAV or S3 sync, each computer shares its usage and sees the others', the total and each computer's part (#542) — sync carried keys, agents and the library, and each computer's Usage page counted only the calls made on it. Sync now has a Usage tick (off unless picked, usage=yes in magpie webdav/magpie s3): a computer with it on puts its own calls up as one file a day, magpie/usage/<id>-<day>.magpie-usage beside the bundle (under the prefix's usage/ on S3), each a gzipped list of times, agents, providers, models, tokens, costs and statuses, sealed with the sync passphrase (backup.SealData, the key cached per passphrase so a sync derives it once); no request body, session, archive name or key goes, a call another computer passed on over the LAN (Via) isn't put up again, and an error is cut to 300 characters. The id is a random one kept in usage-computer.json, which no backup carries, with the host name beside it. A computer only ever writes and deletes its own names, so none overwrites another's; the files are not in the bundle and its hash, so sharing usage never makes a sync or a conflict of the bundle; the first sync puts up the last 90 days, after that only yesterday and today, at most every 15 minutes (a Save or Sync now at once), and a day past 90 is deleted by the computer it is of. The others' files, listed (PROPFIND Depth 1; ListObjectsV2) and fetched only when changed, are kept in usage-others and read with the ledger, as Record.Computer; a failure is said in the sync row ("Couldn't share usage") and never fails the sync. The Requests page and the panel's Usage tab get a Computer pick (this computer, other computers when more than one, each by name) sending computer=this|others|<id>; a row from another computer says "· on <name>" beside its agent, and its details name the computer and say what was said stays there, nothing asked of this computer's session files for it. With no other computer's usage there is no pick and nothing new on either page, and an older magpie never lists or reads usage/. TestUsageSharedBetweenComputers (computers a and b over a fake WebDAV under httptest: each sees 4 calls, the totals and its own and the other's part, nothing counted twice over repeated syncs, the bundle not put again for usage, a new call reaching the other, a day past the window deleted), TestUsageSharedThroughS3 (the same through a fake S3, signed list and delete, a prefix with a space and a plus) and TestUsageOffAsksNothing are new and don't build without the change; usage-computers.test.cjs (WebKit, en and zh: no pick without other computers, the pick and computer=this, another computer's row and details with no /content asked, the panel's pick, the sync form's tick posting usage: true, no click moving the page) fails without it on all but the no-other-computers case. s3-sync, sync-other-kind, panel-usage, request-archive-usage, click-scroll and usage-ledger-chart pass in WebKit. go vet ./internal/..., go build -tags nogui (darwin, GOOS=linux), GOOS=windows go build, go test -tags nogui -count=1 ./... under a temp HOME pass (TestFXInState failed once on its TempDir cleanup, passes alone). Turning the Usage tick off, or sync off, lets the other computers' usage kept here go (DropAllShared), so the Usage page counts this computer's alone again: TestUsageSharedBetweenComputers checks both, and fails without it. 2026-10-02 17:47:36 +08:00
yetone 2e94643c7b sync: moving between WebDAV and S3 keeps the other's server, and moving back finds it (ARNO on Discord: 我体验了一下s3同步,结果它把我之前配置好的webdav同步设置给删了) — sync.json held one setup, and saving S3 in Settings › Sync and backup (or magpie s3 on) wrote the bucket over it, so the WebDAV address, user and password were gone. sync.json now keeps the other kind's server beside the one synced to (other: its address, user, password or secret, and S3's endpoint, region and path-style), in the same file, readable by the user alone, and never in a backup, as before. Only one is synced to. Moving back with the password left empty uses the one kept, for the same server and user alone; magpie webdav on / magpie s3 on with nothing else given moves back to the kept one, and magpie webdav / s3 say which is kept. In Settings the row says "WebDAV settings kept" (or S3's), the form marks the kind synced to "· on", says whether saving keeps it or moves sync, its button reads "Move sync to WebDAV" when it does, and the other kind's fields show the kept server. A sync.json from before reads as it did; turning sync off still removes it all. TestConfigureKeepsTheOtherKind, TestS3Cmd and sync-other-kind.test.cjs fail without the change. go vet, go build -tags nogui (darwin, GOOS=linux), GOOS=windows go build, go test -tags nogui ./... pass; sync-other-kind, s3-sync, webdav-settings-refresh, click-scroll, old-webkit pass. 2026-10-01 19:44:29 +08:00
yetone 8931a4b3ac davsync: sync can keep its file in an S3-compatible bucket as well as a WebDAV folder (#296) — AWS S3, Cloudflare R2, Backblaze B2, MinIO, Garage or a NAS. Settings › Sync and backup picks WebDAV or S3: endpoint (empty is AWS), bucket, prefix, region (empty is us-east-1, auto for R2), access key ID and secret, path-style; CLI magpie s3 on s3://bucket/prefix access-key-id=… endpoint=… region=… path-style. The bucket holds the same passphrase-sealed file at <prefix>/magpie/magpie.magpie-backup, signed with SigV4 (no SDK). It is read with its ETag and written with If-Match (If-None-Match: * for the first), so two computers syncing at once don't overwrite each other; a server that refuses conditional writes (501, or 400 naming the header) is remembered and gets a HEAD-then-PUT instead. An IP, localhost or a dotted bucket on https goes path-style on its own. Errors say what to do: wrong key/secret, clock skew, bucket in another region (names it), no such bucket, 403 on the first read on AWS without s3:ListBucket. The secret is kept as the WebDAV password is, sent only to the endpoint and key it was typed for, and may not be the passphrase. sigv4_test (AWS's published vectors), s3_test (fake server: conditions, conflict, fallback, regions, errors), s3_cli_test and s3-sync.test.cjs (Chromium, WebKit, en/zh) fail without the change; tested against MinIO. go vet, go build -tags nogui (darwin, GOOS=linux), GOOS=windows go build, go test -tags nogui ./... pass (TestGroupAddSetRemove, a TempDir-cleanup flake under load, passes alone); click-scroll, old-webkit pass. 2026-09-30 17:07:47 +08:00
AronnaxandClaude Opus 5.5 7931723d79 cli: 新增 magpie webdav,在终端里开启、修改和触发 WebDAV 同步 / add magpie webdav to turn on, change and run WebDAV sync from the terminal (#127)
* cli: magpie webdav — WebDAV sync from the terminal, what Settings › WebDAV sync does, for a machine with no window (WSL, a server over SSH)

magpie webdav shows it; on <address> [user=…] [password=…] [keys=no] [agents=no] [library=no] turns it on and syncs at once, asking for the passphrase (and the password, when a user has none); set k=v… changes what is given and keeps the rest; now, dismiss and off are the page's Sync now, OK and Turn off. Through the same davsync.Configure, Now, Dismiss and Off.

A sync, Off and Dismiss hold a lock the system keeps on sync.lock (flock; LockFileEx on Windows): the gateway's sync every 3 minutes and one from a terminal — or from the app's window while magpie serve has the gateway — would each save sync-state.json over the other's. The system lets it go when the magpie holding it ends, however it ends.

Configure keeps the saved password only for the same server (scheme and host) and user: an address moved to another server, or another user, no longer takes it there — in the app too, where a new address with the password left empty was sent the old one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* davsync, cli: from the review of #127 — the setup read again under the lock, Configure locked too, the password never an argument, one left behind asked for, a wait that runs out said in the status

Now reads the setup again once it has the lock: one that waited while another magpie turned sync off went on with the setup it had read, and saved sync-state.json back after the Off. Configure takes mu and the lock too, after a sync in progress.

magpie webdav takes neither the password nor the passphrase as an argument: password= asks for it, as passphrase= does.

Configure with the saved password left behind — another server or user, the password empty — says to type it and saves nothing, where the Settings page dropped it and the sync failed with 401 under a field that still said saved.

A wait for the lock that runs out is davsync.ErrBusy. Dismiss returns it, and the davsync routes put ErrBusy from a sync in the status they answer with, where the page shows a sync's failures; Save's own errors stay in its form, what was typed in it kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* davsync, cli: the lock before mu, a wait bounded under the page's, one called off not called busy, the password rule in one place, a bare address with = in it

The lock comes before mu everywhere, so a wait for another magpie never holds mu, and Configure, Off and Dismiss wait for it 30 seconds at most, less than the Settings page gives a request: before, a request could hang on mu and then on the lock for minutes.

A wait for the lock called off, as when the page is closed or the gateway stops, is its context's error, not ErrBusy: nothing said another magpie was syncing.

What becomes of the saved password is decided in one place, davsync.SavedPassword, which Configure and magpie webdav both go by, where the two had rules of their own that had come apart: an account with a password and no user name, moved to another server, had its password dropped with nothing asked, and the sync failed with 401. It is asked for now, as for an account with a user.

magpie webdav takes a bare argument with :// in it as the address, = in it or not: https://host/dav?token=… was read as a field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 14:54:04 +08:00