Commit Graph
712 Commits
Author SHA1 Message Date
Wesley Liddick d1d3400b69 Merge pull request #3645 from bestony/worktree/lucky-harbor-dbe4
feat(keys): add api key concurrency stats
2026-07-06 16:45:15 +08:00
linshuboy f26ca5661e feat: add OpenAI advanced scheduler controls
Related: #1089, #408, #123
2026-07-05 17:24:38 +08:00
Wesley Liddick 0c53d4f334 Merge pull request #3661 from jianjianai/fix/group-capacity-batch-hotpath
fix: 优化用户组容量汇总批量统计性能
2026-07-04 09:20:02 +08:00
Wesley Liddick 8e07ec55db Merge pull request #3685 from jianjianai/fix/optimize-slot-cleanup
fix: 优化并发槽位定时清理性能
2026-07-04 09:19:34 +08:00
DaydreamCoding ebbdc70311 feat(usage): 错误请求对齐用量明细(UI/排序/筛选/列设置)
错误请求列表(/admin/usage 错误 tab、Ops 弹窗、用户端 /usage 错误 tab)
对齐用量明细的交互与信息密度。Squash of:

- feat(usage): 错误请求全面对齐用量明细(UI/列序/排序/筛选/列设置/新列)
- refactor(usage): 错误表提取共享徽章工具与 IP 批量工具条,后端排序解析归并 SetSort
- feat(usage): /admin/usage 错误请求新增分类过滤
- fix(ops): 错误列表 phase=upstream 过滤生效,守卫豁免改为显式 opt-in
- fix(ops): 错误列表用户列回退显示已删除 KEY 所有者
- fix(usage): 错误请求状态码排序对齐展示/过滤,筛选项改固定常用码
2026-07-03 23:02:27 +08:00
shaw a1b2b32e08 fix: prevent silent usage_logs drops under queue overflow (#3656)
高并发下扣费成功但 usage_logs 被双层异步队列静默丢弃(单用户实测丢失 49%),
导致消费排行与对账缺口。根因是 6a685727 为避免溢出时单行 INSERT 踩踏数据库,
把三处"队列满"改成了立即终态丢弃。

改为有界阻塞背压,兼顾数据不丢与批量写入模式不变:

- CreateBestEffort/createBatched 入队移除 default 立即丢弃分支,
  队列满时阻塞等待(受 detached 15s ctx 期限约束)
- writeUsageLogBestEffort 收到 dropped 不再直接放弃,统一走
  repo.Create 同步兜底(仍经批处理器;ctx 耗尽时换新 detached 窗口),
  重复写入由 ON CONFLICT DO NOTHING 幂等防护
- worker 池默认溢出策略 sample→sync(viper 默认 + 池常量),
  溢出时提交方内联执行(提交点在响应写出后,不阻塞客户端),
  显式配置 sample/drop 的部署不受影响
- 附带修复 ensure*Batcher 的存量数据竞争:channel nil 检查移入 sync.Once

正常负载零行为变化;突发时短暂等待替代永久丢失;数据库只见批量 INSERT。
2026-07-03 21:02:32 +08:00
jjaw f3a3a0869d 优化并发槽位清理 2026-07-03 21:02:31 +08:00
jianjian c620ad6a36 fix: align group capacity SQL with target schema 2026-07-03 08:44:39 +00:00
jianjian 72ccd1b111 fix: batch group capacity summaries 2026-07-02 20:22:17 +00:00
jianjian 3f2ef60468 fix: optimize ops realtime account stats 2026-07-02 20:04:47 +00:00
Wesley Liddick 821399ade1 Merge pull request #3622 from deqiying/feat/subscription-revoke-restore
支持恢复已撤销订阅
2026-07-02 17:34:20 +08:00
Wesley Liddick 51b5142af7 Merge pull request #3612 from StarryKira/codex/fix-user-model-stats-requested-model
[codex] fix user model stats requested aggregation
2026-07-02 17:32:19 +08:00
Wesley Liddick 634b0eee48 Merge pull request #3430 from chaochaoweb3/codex/remove-claude-oauth-expires-in
fix: omit expires_in from Claude OAuth token exchange
2026-07-02 17:32:09 +08:00
Bestony@Homelab 089a7b7fae feat(keys): add api key concurrency stats 2026-07-02 15:58:29 +08:00
deqiying b26dcc3da2 feat(subscription): 支持恢复已撤销订阅 2026-07-01 22:19:21 +08:00
Haruka e236bff1ed fix: aggregate user model stats by requested model 2026-07-01 19:01:30 +08:00
xueshiji 5e99561d6e Merge branch 'Wei-Shaw:main' into main 2026-07-01 17:48:42 +08:00
Wesley Liddick 0a9146c3d1 Merge pull request #3586 from deqiying/codex/fix-subscription-revoke-soft-delete
修复订阅撤销操作实际上是软删除的bug
2026-07-01 15:38:45 +08:00
shaw fd004bdd88 fix(account-repo): Clone query before Count to prevent state pollution
ListWithFilters reused the Ent query builder across Count() and All(),
so interceptor-appended predicates (SoftDeleteMixin's deleted_at IS NULL)
accumulated on the shared builder — the same pattern already fixed in
group_repo/promo_code_repo/user_repo by commit 2588fa6a (P1-03 audit).

The integration test now asserts pagination.Total matches len(items) on a
single page, guarding the total-vs-items invariant reported in #3601.

Fixes #3601
2026-07-01 14:35:49 +08:00
xueshiji 8b46994dc2 Merge branch 'Wei-Shaw:main' into main 2026-07-01 14:01:03 +08:00
xueshiji 5e9529bb88 Merge branch 'Wei-Shaw:main' into main 2026-07-01 14:00:51 +08:00
DaydreamCodingandClaude Sonnet 5 bdf7ead157 feat(spark-shadow): OpenAI Spark 链接型影子账号
背景:gpt-5.3-codex-spark 使用独立于 codex 全局(5h/7d)的配额窗口(数据源是
/wham/usage 响应体的 codex_bengalfox,而非 codex 全局用的 x-codex-* 响应头),且
只能挂在已完成 OAuth 授权的 OpenAI 账号下复用其登录态,不能作为独立账号单独接入。
为此新增“链接型影子账号”(spark shadow account):影子账号本身不持有任何凭据,
通过 parent_account_id 指向母账号,凭据/token/代理透传自母账号并共享母账号的刷新
周期,仅在配额维度(quota_dimension=spark)和用量窗口上与母账号完全独立调度、互不
连坐。

实现:
- 数据模型:migration 154(+154a)给 accounts 表加 parent_account_id /
  quota_dimension 列 + 4 条约束(维度合法 / parent⟺非 global 维度一致 / 禁自指 /
  FK)+ 2 个 CONCURRENTLY 索引(母账号索引 + 每母账号至多一个影子的唯一索引)。
- 创建:POST /api/v1/admin/accounts/:id/shadow(CreateShadow)—— 一母一影(唯一
  索引兜底并发竞态),继承母账号 proxy/分组/并发/优先级(显式传参可覆盖),默认
  model_mapping 恒等映射到 spark(拒绝非 spark 模型),母账号必须是真实的 OpenAI
  OAuth 账号(非影子)。
- 凭据透传:resolveCredentialAccount 把影子解析回母账号,GetAccessToken / 请求头
  / WS 三条路径统一走此函数;影子自身 Credentials 恒为空(仅允许写 model_mapping),
  凭据写入的汇聚点 persistAccountCredentials 对影子早返 no-op,防止误写。
- 调度:parentHealthyForShadow 只看母账号是否仍是 OpenAI OAuth + 凭据/传输是否
  可用(active、token 未过期、未处于 401/刷新失败/传输故障导致的临时不可调度冷却),
  刻意不看母账号的 global 限流窗口——两条 429 道互不连坐。
- 用量:影子的 codex_5h/7d 走 OpenAIQuotaService.QueryUsage(/wham/usage 的
  codex_bengalfox),与母账号走的 WSv2 探测(/responses 头)完全独立的数据源、
  刷新节流与 staleness 判定。
- 备份:ExportData 显式排除影子账号(影子不持凭据,通用凭据型导入强制
  credentials 非空、无法表达父子链接),按 skipped_shadows 计数提示前端。
- 前端:账号操作菜单新增“创建 Spark 影子”入口,影子行展示回填的母账号信息
  (邮箱 / plan / 隐私模式 / 订阅到期 / chatgpt_account_id),批量操作自动跳过
  影子账号。

说明:migrations 目录用完整文件名(而非纯数字前缀)标识迁移,故本次新增的
154_account_spark_shadow.sql / 154a_..._notx.sql 与已有的
154_add_ops_system_logs_api_key_id.sql 按序号共存,与目录里 145/151 已有的
先例一致。

测试:新增约 20 个测试文件,覆盖 handler(CreateShadow 校验 / 母账号信息回填)、
repository(影子 round-trip / 一母一影唯一索引 / 迁移 schema)、service(凭据
透传三路径 / 调度母健康门 / 用量窗口来源与刷新节流 / CRS 母账号不变量 / 各类
早返与 fail-closed 场景)及前端组件(账号列表 / 操作菜单 / 用量重置)。

验证(镜像 CI;golangci-lint 首次全量分析耗时过长被跳过,其余全部实测):
- gofmt -l:干净
- go build ./... / go vet ./...:通过
- go test ./... -count=1:全绿(全部包 ok,含 internal/service、
  internal/repository、migrations)
- go test -tags integration ./internal/repository/... ./internal/service/...
  (真实 Postgres,testcontainers):全绿,含迁移幂等性
  (TestMigrationsRunner_IsIdempotent_AndSchemaIsUpToDate)与影子相关全部用例
- pnpm lint:check / pnpm typecheck / pnpm build(真实 vite 构建)/
  pnpm vitest run:全绿(124 文件 760 用例)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:21:45 +08:00
deqiying 03727ac363 fix(subscription): 修复订阅撤销软删除失效
总: 增加明确的订阅撤销接口,修复撤销后的缓存失效和管理端 revoked 展示。

分: 同步失效订阅 L1 与 billing cache,补跨实例失效通知、soft-delete-aware 列表查询、revoked_at DTO 字段和回归测试。
2026-07-01 00:37:51 +08:00
PMExtra cafc95c3e2 feat: align user usage analytics with admin 2026-06-30 15:31:28 +08:00
xueshiji 915c60b150 feat(group): 订阅分组新增可选的高峰时段倍率,以支持智谱等coding plan的高峰时段 2026-06-30 14:17:05 +08:00
DaydreamCodingandClaude Opus 4.8 185f9c9920 fix(auth-signup): 平台配额快照脱离注册事务 + grok 补入 CHECK 约束
自助注册(含钉钉/OAuth)报 500→404 的根因:grok 自 2026-06 进入默认平台配额
(default_platform_quotas / auth_source_*),但 user_platform_quotas 的 CHECK
约束(迁移 142)仅允许 anthropic/openai/gemini/antigravity。注册时
snapshotPlatformQuotaDefaults 写 grok 行违反约束 → 整个注册事务被 Postgres 标记
aborted → consumePendingOAuthBrowserSessionTx 撞 "transaction aborted" → 500 →
clearCookies → 用户重试拿到 404(PENDING_AUTH_SESSION_NOT_FOUND)。
影响面:所有新自助注册(不限钉钉)。

修复(两层):
- 事务隔离(fix①):snapshotPlatformQuotaDefaults 用 ent.WithoutTx 剥离调用方事务,
  在基础连接 autocommit 执行。best-effort 快照失败永不毒化注册主事务,从根上消除
  "事务内 fail-open 形同虚设"陷阱——今后任何平台/约束漂移都不会再连累注册。
- 迁移 157:把 grok 加入 user_platform_quotas.platform 的 CHECK 约束,与代码平台
  列表(domain/constants.go PlatformGrok)对齐(DROP IF EXISTS + ADD,可重入)。

新增 ent.WithoutTx(ctx) helper(手写文件,不动生成代码)。

测试:
- 单测 TestSnapshotPlatformQuotaDefaults_DetachesCallerTransaction(RED→GREEN):
  快照即便在事务 ctx 中也必须用脱离事务的 ctx 调 repo。
- 集成测试 TestUserPlatformQuotaRepository_BulkInsertInitial_GrokAllowed:
  迁移 157 后 grok 可写入(真实 postgres 容器验证)。

验证:go build ./... / go vet -tags unit ./... / 全量单测(-tags unit,45 包) /
平台配额+迁移集成测试 全绿。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 13:22:27 +08:00
Wesley Liddick e70e36e4d2 Merge pull request #3484 from bwliangc/feat/risk-control-matched-keyword
feat(risk-control): record matched keyword in keyword-block logs
2026-06-30 10:40:30 +08:00
Bestony bad87ff533 feat(ops): add api key filter to system logs 2026-06-27 14:35:19 +08:00
bwlcandClaude Opus 4.8 815bc6c9b5 feat(risk-control): record matched keyword in keyword-block logs
The risk control center's moderation log records had no field for the
keyword that triggered a keyword block, so the admin UI couldn't show
which keyword was hit (only the application slog logged it).

- migration 156: add matched_keyword column to content_moderation_logs
- ContentModerationLog gains MatchedKeyword; set it on keyword block
- repo CreateLog/ListLogs persist and read the column
- frontend: show "命中关键词" inline in the log table and detail modal
- i18n: add matchedKeyword (zh/en)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 10:43:05 +08:00
Heatherm Huang e7a4f3f465 test: fix grok oauth validation suite 2026-06-26 10:36:09 +08:00
Heatherm Huang b2e2c7e69c fix: harden grok oauth gateway paths 2026-06-26 10:36:09 +08:00
Heatherm Huang b3a07aeae7 fix: align grok oauth exchange with xai 2026-06-26 10:36:09 +08:00
Heatherm Huang 39be1ec97f feat: add grok subscription support 2026-06-26 10:36:09 +08:00
Wesley Liddick 1721b2b2d1 Merge pull request #3404 from wucm667/fix/balance-preflight-prevent-overdraft
fix(billing): 余额扣减增加充足性校验,防止余额透支为负
2026-06-26 09:45:46 +08:00
chaochaoweb3 5bd9368ab4 fix claude oauth token exchange payload 2026-06-25 23:41:00 +08:00
feitianbubu 4567f6582b test(admin/usage): update sqlmock rows for cache breakdown columns 2026-06-24 23:59:22 +08:00
feitianbubu 063454ae96 fix(admin/usage): populate cache creation/read token breakdown in stats 2026-06-24 23:43:03 +08:00
wucm667 9f5b57fc96 fix(billing): 防止余额计费持续透支 2026-06-22 10:30:27 +08:00
Wesley Liddick 2e0ff1cfd5 Merge pull request #3258 from bwliangc/feat/channel-monitor-jitter
feat(渠道监控): 检测间隔支持正负随机抖动配置
2026-06-16 16:58:23 +08:00
Wesley Liddick 16765bde69 Merge pull request #3230 from DaydreamCoding/feat/openai-cyber-policy-passthrough
feat(openai): cyber_policy 硬阻断全链路透传、审计与计费
2026-06-16 16:55:51 +08:00
shaw f069c9ae00 fix(outbox-dedup): buildSchedulerGroupPayload typed-nil broke dedup_key consistency
#3255 introduced payload-aware dedup_key (sha256 over event_type, account_id,
group_id, payload_json). enqueueSchedulerOutbox checks "if payload != nil"
before json.Marshal — but Go interface containing a typed-nil map (returned
by buildSchedulerGroupPayload(empty)) is NOT == nil at the interface level.

So an ungrouped account's account_changed event went through this path:
  payload := buildSchedulerGroupPayload(account.GroupIDs)  // typed-nil map
  enqueueSchedulerOutbox(..., payload)                     // interface != nil
    → json.Marshal(typedNilMap) = "null"
    → dedup_key hash = sha256(... + "null")

While other call sites pass literal nil:
  enqueueSchedulerOutbox(..., nil)                         // interface == nil
    → payloadJSON stays empty
    → dedup_key hash = sha256(... + "")

The two dedup_keys differ for what should be the same logical event,
silently degrading dedup effectiveness in bursts on ungrouped accounts.

Fix: change buildSchedulerGroupPayload return type from map[string]any to
any so empty input returns true untyped-nil. All call sites pass the
result straight to enqueueSchedulerOutbox(payload any) — no inspection,
no breakage.

Adds regression test TestEnqueueSchedulerOutbox_UngroupedAccountDedupesWithLiteralNilPayload
asserting (1) typed-nil regression doesn't sneak back, (2) dedup_key for
empty-groups payload matches the literal-nil-payload key.
2026-06-16 14:21:09 +08:00
shaw acaffe29ec fix(account-repo): refresh candidates SQL excluded healthy accounts; fix CI build
Post-merge audit of #3272 found two regressions:

1. ListOAuthRefreshCandidates used "AND NOT (a AND b)" which, under PG
   3-valued logic, evaluates to NULL when both temp_unschedulable_until
   and temp_unschedulable_reason are NULL — i.e., the common healthy
   account state. Such rows were silently excluded from the background
   token refresh worker, so their OAuth access tokens would never get
   refreshed and eventually start returning 401.

   Verified empirically against PostgreSQL: only 3 of 5 test rows
   matched before the fix; after switching to "(a AND b) IS NOT TRUE"
   the expected 4 rows match.

2. The new ListOAuthRefreshCandidates method on AccountRepository was
   not implemented on stubAccountRepo in api_contract_test.go (build
   tag "unit"), breaking "make test-unit" which CI runs in
   .github/workflows/backend-ci.yml.

Tests:
- Added IS NOT TRUE and "AND NOT (" assertions to the SQL-shape unit
  test so the predicate can't regress to the broken form again.
- "go test -tags=unit ./internal/..." now passes cleanly.
2026-06-16 14:08:50 +08:00
shaw 31dc8913ac chore(outbox-cleanup): add 10s grace to defend against id-vs-commit race
PG sequences advance outside transactions, so a slow committer can hold
an id that gets surpassed by the watermark before its commit becomes
visible. Without the grace period, cleanup would delete such rows before
the snapshot poller ever sees them. 10s is a comfortable upper bound
on realistic enqueueSchedulerOutbox commit latency.
2026-06-16 11:57:32 +08:00
jjaw cb14935e9a fix: cleanup consumed scheduler outbox rows 2026-06-16 11:56:40 +08:00
shaw 1fdbe52f96 chore(migrations): renumber scheduler outbox dedup migrations 151/152 -> 152/153
#3232 (already merged) occupies 151. Renumber #3255's dedup migrations to
avoid collision and keep linear ordering. Updated runner constant + tests.
2026-06-16 11:50:25 +08:00
jjaw b3ec6288ad fix: release scheduler outbox dedup on claim 2026-06-16 11:48:13 +08:00
jjaw 60cf89ae26 fix: recover scheduler outbox invalid dedup index 2026-06-16 11:48:13 +08:00
jjaw 3ef70b045d fix: safely coalesce scheduler outbox events 2026-06-16 11:48:12 +08:00
jjaw 34e66ec0a5 fix: outbox scheduler snapshot coalesce 2026-06-16 11:48:12 +08:00
jjaw 74199b6a6e fix: reduce token refresh retry amplification 2026-06-16 11:44:39 +08:00