Merge pull request #1 from jinyiwei2012/feat/pools-and-sentinel
feat: 代理池 + Cookie 池预热 + 日志轮转 + 配额哨兵 + 求解器修复
This commit is contained in:
28 files changed
+4638
-68
No files matched your search
@@ -35,10 +35,11 @@ Go 实现,编译为**单个静态二进制**,无运行时依赖;内置 **W
|
||||
- **三站点聚合**:单站点每日额度耗尽自动切换到其他站点的同款模型,额度耗尽的站点临时降级 10 分钟
|
||||
- **站点额度管理**:额度按访客身份按日计算,控制台站点卡片可一键查询当前额度、重置访客身份(等同新开隐私窗口、额度即刻回满);请求撞额度时也会自动轮换访客身份恢复,无需重新求解
|
||||
- **代理线路**:主线路 + 多条备用线路,支持 `http://` `https://` `socks5://`;slow-start 检测 + 失败降级 + 流中失败不重放
|
||||
- **代理池**:Xray 核心(分享链接 / 订阅 / `ip:port` 端点;核心可自动下载)+ 粘性选路 + 后台健康探测;每个站点可绑定独立出口,Turnstile 求解与请求共用同一出口
|
||||
- **Turnstile 自动突破**:三种 Token 获取方式 — 求解服务 API(`provider="api"`)、本地浏览器自动求解(`provider="browser"`,CDP 直连 Chrome/Edge,无需 Playwright/Docker)、手动导入 Cookie(`provider="manual"`);结果按「代理线路 × 站点」缓存,TTL 过期自动重解
|
||||
- **会话自愈**:nonce / data-config 缓存 + 配额耗尽 / 验证失败时自动刷新重试
|
||||
- **Web 管理台**:仪表盘、模型与站点、API Key、代理线路、Turnstile、对话调试、实时日志、设置
|
||||
- **可观测**:按模型 / 站点 / 按日统计、P50/P95 延迟、最近请求、实时日志(SSE),数据持久化到 `data/`
|
||||
- **可观测**:按模型 / 站点 / 按日统计、P50/P95 延迟、最近请求、实时日志(SSE)、文件日志按日压缩轮转,数据持久化到 `data/`
|
||||
- **安全**:下游 Bearer / `X-Api-Key` 鉴权、每 Key 每分钟限流、每站点并发闸门、管理台独立口令
|
||||
|
||||
---
|
||||
@@ -134,14 +135,16 @@ dsfree2api -config config.toml -check
|
||||
|
||||
| 字段 | 说明 |
|
||||
|------|------|
|
||||
| `[server]` | API 监听 `host` / `port` / `log_level` |
|
||||
| `[server]` | API 监听 `host` / `port` / `log_level` / `log_file`(留空写数据目录,按日压缩轮转;`-` = 仅终端) |
|
||||
| `[security].api_keys` | 下游鉴权,空数组 = 不鉴权 |
|
||||
| `[admin]` | 管理台 `enabled` / `host` / `port` / `password`(留空启动时生成随机密码) |
|
||||
| `[limits]` | `max_concurrent_per_site` 每站点并发、`rate_per_minute` 每 Key 每分钟限流(0 = 不限) |
|
||||
| `[proxy]` | `url` 主线路、`fallback_urls` 备用线路、`slow_start_seconds` 首事件超时 |
|
||||
| `[proxypool]` | 代理池总开关、健康检查间隔 / 超时 / URL、裸 `ip:port` 默认协议、Xray 路径 / 版本 / 自动下载;`[proxypool.entries.*]` 手动节点(分享链接或端点)、`[proxypool.subscriptions.*]` 订阅自动拉取 |
|
||||
| `[quota]` | 配额哨兵:`enabled` / `check_seconds` / `warn_ratio`,定期用池内会话读取站点免费额度,低额告警与重置留档 |
|
||||
| `[upstream]` | 超时、配置缓存 TTL、会话自动刷新、`cross_site_failover` 跨站切换、`continue_rounds` 工具调用截断续写轮数(0 = 不续写、直接截断补全) |
|
||||
| `[turnstile]` | **默认 `enabled = false` 且不内置任何求解服务**;`provider` 三选一:`api`(填 `api_url` / `api_key`)、`browser`(填 `browser_path` 指向本机 Chrome/Edge)、`manual`(只用控制台导入的 Cookie),另有 Cookie TTL、重试次数 |
|
||||
| `[sites.*]` | 三个站点的 `base_url` / `ajax_url` / `sitekey` / `language` |
|
||||
| `[turnstile]` | **默认 `enabled = false` 且不内置任何求解服务**;`provider` 三选一:`api`(填 `api_url` / `api_key`)、`browser`(填 `browser_path` 指向本机 Chrome/Edge)、`manual`(只用控制台导入的 Cookie),另有 Cookie TTL、重试次数、单次求解超时(`timeout_seconds`)、求解服务风格(`api_style`)与 Cookie 池(`warm_enabled` / `warm_ratio` / `warm_check_seconds`,后台预热,请求零等待) |
|
||||
| `[sites.*]` | 三个站点的 `base_url` / `ajax_url` / `sitekey` / `language` / `proxies`(站点级出口绑定,按顺序粘性选路)/ `warm_pool_only`(只预热池绑定线路) |
|
||||
| `[models.*]` | 模型到 `site` + `upstream_id` + `bot_id` / `post_id` 的映射 |
|
||||
|
||||
### Turnstile 说明
|
||||
@@ -154,6 +157,7 @@ dsfree2api -config config.toml -check
|
||||
| `browser` | 2 · 本地浏览器自动获取 | `browser_path` | CDP 直连你配置的 Chrome / Edge:打开站点 → 注入 Turnstile widget → 可信点击 → 轮询拿到 Token;等价于 heartmore/cloudflare-solver 的做法,但**无需 Playwright / FlareSolverr / Docker**,单二进制即可。可选 `browser_user_data_dir`(真实 profile)、`browser_timezone` / `browser_locale`(对齐代理出口) |
|
||||
| `manual` | 3 · 手动导入 Cookie | — | 不自动求解,只使用控制台导入的 Cookie |
|
||||
|
||||
- **求解服务风格**:`api_style = "ezsolver"` 时兼容本地 EzSolver 形态的求解服务(`{sitekey,siteurl,timeout}` → `{token}`),`api_key` 可留空;环回 / 内网求解地址直连(不随 `[proxy]` 线路,避免 `127.0.0.1` 被解析到隧道另一侧);
|
||||
- **手动导入 Cookie**:在走同一条出口线路的浏览器里过一次验证,把 `document.cookie` 粘到控制台 **Turnstile → 手动导入 Cookie**,在 `cookie_ttl_seconds` 内免求解(Cookie 与出口 IP / UA 绑定,换线路需重新导入);
|
||||
- **浏览器模式提示**:求解期间会短暂弹出浏览器窗口(`browser_headless = true` 可关闭,但过验证率更低);带账号密码的代理暂不支持 Chrome,请用 IP 白名单代理;浏览器被系统策略锁定时会报 `browser launch failed` 并附 stderr。
|
||||
|
||||
@@ -265,8 +269,8 @@ curl http://127.0.0.1:8000/v1/messages \
|
||||
| 仪表盘 | 请求数 / 错误 / Token / P50·P95、模型用量、站点状态(含 Cookie TTL 进度条)、最近请求 |
|
||||
| 模型与站点 | 编辑标签、路径、bot_id/post_id,启停模型与站点,修改站点 URL / 语言 |
|
||||
| API Key | 生成 / 删除 / 复制下游 key,立即写回配置 |
|
||||
| 代理线路 | 主线路、备用线路增删、连通性测试、slow-start / 并发 / 限流 / 跨站切换 |
|
||||
| Turnstile | 三种 Token 获取方式的模式面板(求解服务 API / 本地浏览器自动获取 / 手动导入 Cookie)、各站 Cookie 状态与 TTL、强制刷新、求解历史 |
|
||||
| 代理线路 | 主线路、备用线路增删、连通性测试、slow-start / 并发 / 限流 / 跨站切换、代理池(节点 / 订阅 / 每站点绑定 / Xray 核心状态) |
|
||||
| Turnstile | 三种 Token 获取方式的模式面板(求解服务 API / 本地浏览器自动获取 / 手动导入 Cookie)、求解器风格(`api_style`)、各站 Cookie 状态与 TTL、Cookie 池预热状态、强制刷新、求解历史 |
|
||||
| 对话调试 | 选模型 / 系统提示 / tools JSON,流式或非流式,带连接计时与取消 |
|
||||
| 实时日志 | 级别过滤 + SSE 实时推送 |
|
||||
| 设置 | 服务与上游参数、数据目录、危险操作 |
|
||||
@@ -283,6 +287,7 @@ curl http://127.0.0.1:8000/v1/messages \
|
||||
|------|------|------|
|
||||
| `HOST` / `PORT` | API 监听地址 | `0.0.0.0` / `8000` |
|
||||
| `LOG_LEVEL` | `DEBUG` `INFO` `WARN` `ERROR` | `INFO` |
|
||||
| `LOG_FILE` | 日志文件路径;留空 = `<data_dir>/logs/dsfree2api.log`,`-` = 仅终端 | —(文件日志) |
|
||||
| `API_KEYS` | 下游 key,逗号分隔 | —(不鉴权) |
|
||||
| `PROXY_URL` | 主代理 | —(直连) |
|
||||
| `PROXY_FALLBACK_URLS` | 备用代理,逗号分隔 | — |
|
||||
@@ -291,6 +296,10 @@ curl http://127.0.0.1:8000/v1/messages \
|
||||
| `TURNSTILE_API_KEY` | Turnstile 求解服务 key(仅在 `provider=api` 且开启时校验) | 配置文件 |
|
||||
| `TURNSTILE_PROVIDER` | Token 获取方式:`api` / `browser` / `manual` | `api` |
|
||||
| `TURNSTILE_BROWSER_PATH` | 浏览器模式的 Chrome / Edge 可执行文件路径 | — |
|
||||
| `TURNSTILE_TIMEOUT_SECONDS` | 单次求解超时秒数(5–600) | `90` |
|
||||
| `TURNSTILE_WARM_ENABLED` | Cookie 池后台预热开关 | `false` |
|
||||
| `TURNSTILE_WARM_RATIO` | 剩余 TTL 低于该比例时自动续期(0–1) | `0.3` |
|
||||
| `TURNSTILE_WARM_CHECK_SECONDS` | 预热巡检间隔秒数(5–3600) | `60` |
|
||||
| `ADMIN_ENABLED` / `ADMIN_HOST` / `ADMIN_PORT` / `ADMIN_PASSWORD` | 管理台 | `true` / `127.0.0.1` / `8001` |
|
||||
| `DATA_DIR` | 统计等运行时数据目录 | `./data` |
|
||||
|
||||
@@ -302,15 +311,18 @@ curl http://127.0.0.1:8000/v1/messages \
|
||||
cmd/dsfree2api 入口:配置加载、日志、双 HTTP server、优雅退出
|
||||
internal/config TOML 结构 + 默认值 + env 覆盖 + 校验 + 写回
|
||||
internal/httpx TLS 指纹会话封装(bogdanfinn/tls-client, Chrome_120)
|
||||
internal/proxypool 代理池:Xray 核心管理 / 订阅拉取 / 分享链接解析 / 粘性选路与健康探测
|
||||
internal/quota 配额哨兵:站点余额轮询 / 低额告警 / 重置留档
|
||||
internal/openai OpenAI 协议类型、prompt 拼接、tool_calls 解析
|
||||
internal/anthropic Anthropic Messages 协议类型 + 与 OpenAI 消息的互转
|
||||
internal/turnstile Turnstile 三种求解方式(API / 浏览器 CDP / 手动)+ 按线路/站点的 Cookie 缓存
|
||||
internal/turnstile Turnstile 三种求解方式(API / 浏览器 CDP / 手动)+ 按线路/站点的 Cookie 缓存与后台预热(warmer)
|
||||
internal/upstream 上游逆向核心:data-config/nonce → SSE → 事件翻译,
|
||||
线路降级、慢启动、会话刷新、跨站切换
|
||||
internal/api OpenAI / Anthropic 路由、鉴权、限流、流式写出
|
||||
internal/admin 管理台后端 + go:embed 前端
|
||||
internal/metrics 计数/分模型/分站点/按日/延迟分位数,JSON 持久化
|
||||
internal/logbuf 环形日志 + 订阅广播(供 SSE 日志页)
|
||||
internal/logfile 文件日志:按日轮转 + gzip 归档(`[server].log_file`)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
+114
-10
@@ -5,10 +5,12 @@ import (
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
@@ -17,7 +19,10 @@ import (
|
||||
"github.com/nyoungo/dsfree2api/internal/api"
|
||||
"github.com/nyoungo/dsfree2api/internal/config"
|
||||
"github.com/nyoungo/dsfree2api/internal/logbuf"
|
||||
"github.com/nyoungo/dsfree2api/internal/logfile"
|
||||
"github.com/nyoungo/dsfree2api/internal/metrics"
|
||||
"github.com/nyoungo/dsfree2api/internal/proxypool"
|
||||
"github.com/nyoungo/dsfree2api/internal/quota"
|
||||
"github.com/nyoungo/dsfree2api/internal/turnstile"
|
||||
"github.com/nyoungo/dsfree2api/internal/upstream"
|
||||
)
|
||||
@@ -57,27 +62,65 @@ func main() {
|
||||
return
|
||||
}
|
||||
|
||||
level := parseLevel(cfg.Server.LogLevel)
|
||||
logs := logbuf.New(500)
|
||||
logger := slog.New(logbuf.NewHandler(
|
||||
slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: level}),
|
||||
logs,
|
||||
))
|
||||
|
||||
dataDir, err := cfg.DataDir()
|
||||
if err != nil {
|
||||
logger.Error("cannot create data dir", "error", err)
|
||||
fmt.Fprintf(os.Stderr, "cannot create data dir: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
level := parseLevel(cfg.Server.LogLevel)
|
||||
logs := logbuf.New(500)
|
||||
sinks, closeLog := logSink(cfg, dataDir)
|
||||
defer closeLog()
|
||||
logger := slog.New(logbuf.NewHandler(
|
||||
slog.NewTextHandler(sinks, &slog.HandlerOptions{Level: level}),
|
||||
logs,
|
||||
))
|
||||
|
||||
met := metrics.New(dataDir)
|
||||
met.Start()
|
||||
defer met.Close()
|
||||
|
||||
ts := turnstile.New(cfg.Turnstile, cfg.Upstream.UserAgent)
|
||||
up := upstream.New(cfg, ts, logger)
|
||||
|
||||
// Proxy pool: per-site egress endpoints (Xray share links, subscriptions,
|
||||
// plain http/socks5) with sticky selection. No-op unless enabled.
|
||||
pool := proxypool.New(cfg, dataDir, logger)
|
||||
poolCtx, poolCancel := context.WithCancel(context.Background())
|
||||
defer poolCancel()
|
||||
pool.Start(poolCtx)
|
||||
|
||||
up := upstream.New(cfg, ts, logger, pool)
|
||||
|
||||
// Quota watchdog: every site-side quota exhaustion is logged together with
|
||||
// the tokens spent on that site so far — including ones swallowed by
|
||||
// cross-site failover, so the cutoff is recorded rather than hidden.
|
||||
up.SetQuotaObserver(func(site, model, route string, err error) {
|
||||
met.IncSiteQuota(site)
|
||||
t := met.SiteTotals(site)
|
||||
logger.Warn("site quota exhausted",
|
||||
"site", site, "model", model, "route", route, "error", err.Error(),
|
||||
"site_requests", t.Requests,
|
||||
"site_prompt_tokens", t.PromptTokens,
|
||||
"site_completion_tokens", t.CompletionTokens,
|
||||
"site_quota_events", t.Quota)
|
||||
})
|
||||
|
||||
// Cookie pool: keep one verified session per site × route ahead of time,
|
||||
// so API requests never wait for a solve. No-op unless warm_enabled.
|
||||
warmCtx, warmCancel := context.WithCancel(context.Background())
|
||||
defer warmCancel()
|
||||
ts.StartWarmer(warmCtx, func() []turnstile.WarmTarget { return warmTargets(cfg, up, pool) })
|
||||
|
||||
// Quota sentinel: poll the sites' guest-token balances for the console
|
||||
// and the "quota low" alarms. No-op unless [quota].enabled.
|
||||
quotaCtx, quotaCancel := context.WithCancel(context.Background())
|
||||
defer quotaCancel()
|
||||
quotaWatch := quota.New(cfg, logger, ts, pool)
|
||||
quotaWatch.Start(quotaCtx)
|
||||
|
||||
apiSrv := api.New(cfg, up, met, logs, ts, logger)
|
||||
adminSrv := admin.New(cfg, up, met, logs, ts, logger)
|
||||
adminSrv := admin.New(cfg, up, met, logs, ts, logger, pool, quotaWatch)
|
||||
|
||||
mainAddr := fmt.Sprintf("%s:%d", cfg.Server.Host, cfg.Server.Port)
|
||||
mainHTTP := &http.Server{
|
||||
@@ -133,6 +176,67 @@ func main() {
|
||||
logger.Info("bye")
|
||||
}
|
||||
|
||||
// logSink opens the daily file log (default <data_dir>/logs/dsfree2api.log)
|
||||
// and returns the combined writer plus its close function. `log_file = "-"`
|
||||
// disables file logging; a setup failure degrades to stderr only.
|
||||
func logSink(cfg *config.Config, dataDir string) (io.Writer, func()) {
|
||||
path := strings.TrimSpace(cfg.Server.LogFile)
|
||||
if path == "-" {
|
||||
return os.Stderr, func() {}
|
||||
}
|
||||
if path == "" {
|
||||
path = filepath.Join(dataDir, "logs", "dsfree2api.log")
|
||||
}
|
||||
w, err := logfile.New(path)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "log file disabled: %v\n", err)
|
||||
return os.Stderr, func() {}
|
||||
}
|
||||
return io.MultiWriter(os.Stderr, w), func() { _ = w.Close() }
|
||||
}
|
||||
|
||||
// warmTargets builds the site × route matrix the cookie pool keeps fresh:
|
||||
// the sticky proxy-pool pick first, then the global primary/fallback lines.
|
||||
// A site with warm_pool_only skips the global lanes and warms only its bound
|
||||
// pool routes (request-time failover to the global lanes is unaffected).
|
||||
func warmTargets(cfg *config.Config, up *upstream.Client, pool *proxypool.Manager) []turnstile.WarmTarget {
|
||||
routes := up.Routes()
|
||||
cfg.RLock()
|
||||
sites := make([]config.Site, 0, len(cfg.Sites))
|
||||
for _, st := range cfg.Sites {
|
||||
if st != nil && st.Enabled {
|
||||
sites = append(sites, *st)
|
||||
}
|
||||
}
|
||||
cfg.RUnlock()
|
||||
out := make([]turnstile.WarmTarget, 0, len(routes)*len(sites))
|
||||
seen := map[string]bool{}
|
||||
add := func(site config.Site, proxy string) {
|
||||
key := site.Code + "|" + proxy
|
||||
if seen[key] {
|
||||
return
|
||||
}
|
||||
seen[key] = true
|
||||
out = append(out, turnstile.WarmTarget{Site: site, Proxy: proxy})
|
||||
}
|
||||
for _, site := range sites {
|
||||
if pool != nil {
|
||||
if cands := pool.Candidates(site.Code); len(cands) > 0 {
|
||||
add(site, cands[0].Proxy)
|
||||
}
|
||||
}
|
||||
// warm_pool_only sites keep their cookie pool on the bound pool
|
||||
// routes; the global lanes stay unwarmed.
|
||||
if site.WarmPoolOnly {
|
||||
continue
|
||||
}
|
||||
for _, r := range routes {
|
||||
add(site, r.Proxy)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func countModels(cfg *config.Config) int {
|
||||
cfg.RLock()
|
||||
defer cfg.RUnlock()
|
||||
|
||||
+61
-3
@@ -1,13 +1,18 @@
|
||||
# dsfree2api 示例配置。
|
||||
# 复制为 config.toml 后按需修改;运行 `dsfree2api -check -config config.toml` 可校验。
|
||||
# 环境变量优先级高于本文件:HOST PORT LOG_LEVEL API_KEYS PROXY_URL PROXY_FALLBACK_URLS
|
||||
# 环境变量优先级高于本文件:HOST PORT LOG_LEVEL LOG_FILE API_KEYS PROXY_URL PROXY_FALLBACK_URLS
|
||||
# PROXY_SLOW_START_SECONDS TURNSTILE_ENABLED TURNSTILE_API_KEY TURNSTILE_PROVIDER
|
||||
# TURNSTILE_BROWSER_PATH ADMIN_ENABLED ADMIN_HOST ADMIN_PORT ADMIN_PASSWORD DATA_DIR
|
||||
# TURNSTILE_BROWSER_PATH TURNSTILE_TIMEOUT_SECONDS TURNSTILE_WARM_ENABLED
|
||||
# TURNSTILE_WARM_RATIO TURNSTILE_WARM_CHECK_SECONDS
|
||||
# ADMIN_ENABLED ADMIN_HOST ADMIN_PORT ADMIN_PASSWORD DATA_DIR
|
||||
|
||||
[server]
|
||||
host = "0.0.0.0"
|
||||
port = 8000
|
||||
log_level = "INFO"
|
||||
# 日志文件:留空写入 <data_dir>/logs/dsfree2api.log;填路径则写到该文件;
|
||||
# "-" = 仅终端。日志按天轮转,旧文件压缩为 <名称>-YYYY-MM-DD.log.gz。
|
||||
log_file = ""
|
||||
|
||||
[security]
|
||||
# 下游鉴权。空数组 = 不鉴权;建议部署时换成自己的 key。
|
||||
@@ -33,6 +38,37 @@ fallback_urls = []
|
||||
# 主线路允许的首事件等待秒数(>0 且存在备用线路时生效)。
|
||||
slow_start_seconds = 8
|
||||
|
||||
[proxypool]
|
||||
# 代理池:为每个站点绑定独立出口(Xray 分享链接 / 订阅 / ip:port 端点)。
|
||||
# 选路为粘性:优先使用绑定列表中第一个健康节点,失败才切换;未绑定或
|
||||
# 全部不健康时回退到上面的 [proxy] 主/备线路。Turnstile 求解与请求共用
|
||||
# 同一出口(Cookie 与出口 IP 绑定,换出口后会自动重新求解)。
|
||||
enabled = false
|
||||
check_interval_seconds = 120 # 健康检查间隔
|
||||
check_timeout_seconds = 10 # 单次探测超时
|
||||
check_url = "https://www.gstatic.com/generate_204"
|
||||
default_scheme = "http" # 裸 ip:port 的默认协议(http / socks5)
|
||||
# Xray 核心:留空 xray_path 且开启自动下载时,首次使用会下载到数据目录
|
||||
# (data/xray.exe;核心配置写到 data/config.json)。分享链接(vless / vmess
|
||||
# / trojan / ss)由它承载;纯 ip:port / socks5:// 端点无需核心。
|
||||
xray_path = ""
|
||||
xray_version = "" # 留空 = 最新版
|
||||
xray_auto_download = true
|
||||
|
||||
# 手动节点(名称限字母 / 数字 / - / _):
|
||||
# [proxypool.entries.node-a]
|
||||
# link = "vless://uuid@host:443?type=ws&security=tls&sni=example.com&path=/ws#node-a"
|
||||
# link = "socks5://user:pass@1.2.3.4:1080"
|
||||
# link = "1.2.3.4:8080" # 使用 default_scheme
|
||||
# link = "1.2.3.4:8080:user:pass"
|
||||
# enabled = true # 缺省即启用
|
||||
|
||||
# 订阅(按 interval_minutes 自动刷新;节点名形如 <订阅名>#xxxxxxxx):
|
||||
# [proxypool.subscriptions.home]
|
||||
# url = "https://example.com/api/v1/client/subscribe?token=…"
|
||||
# interval_minutes = 60
|
||||
# enabled = true
|
||||
|
||||
[upstream]
|
||||
timeout = 60.0
|
||||
stream_timeout = 300.0
|
||||
@@ -56,7 +92,8 @@ continue_rounds = 20
|
||||
# manual —— 只用控制台手动导入的 Cookie(方式 3)
|
||||
enabled = false
|
||||
provider = "api"
|
||||
# 方式 1:求解服务 API
|
||||
# 方式 1:求解服务 API(api_style = "sync" 为默认 CapSolver 形状;
|
||||
# api_style = "ezsolver" 适配本地 EzSolver 服务,api_key 可留空)
|
||||
# api_url = "https://your-solver.example.com/turnstile/sync"
|
||||
# api_key = "REPLACE_ME"
|
||||
# 方式 2:本地浏览器自动获取(CDP 直连,无需 Playwright / Docker)
|
||||
@@ -67,10 +104,26 @@ provider = "api"
|
||||
# browser_locale = "" # 可选:如 "ja-JP",留空取站点语言
|
||||
sitekey = "0x4AAAAAADlLZ3ljqZP6cQwq"
|
||||
action = "chat"
|
||||
# 求解超时:单次求解(浏览器 / 求解服务 API)的最长等待秒数,5–600。
|
||||
timeout_seconds = 90
|
||||
cookie_ttl_seconds = 10800
|
||||
retries = 5
|
||||
retry_backoff_seconds = 1.5
|
||||
# ── Cookie 池(后台预热)─────────────────────────────────
|
||||
# 开启后,后台按 warm_check_seconds 巡检:Cookie 缺失 / 过期 /
|
||||
# 剩余 TTL 低于 warm_ratio 时主动重解并写回缓存,请求路径不再等待求解。
|
||||
# provider = "manual" 时不预热(手动导入的 Cookie 无法自动续期)。
|
||||
warm_enabled = false
|
||||
warm_ratio = 0.3 # 剩余 TTL 低于该比例(0.05–0.95)时触发预热
|
||||
warm_check_seconds = 60 # 后台巡检间隔秒数(5–3600)
|
||||
|
||||
# ── 配额哨兵(站点免费额度实时监控)──────────────
|
||||
# 用与聊天相同的出口 Cookie 定期查询站点余额接口(每日免费额度),结果
|
||||
# 写入日志与后台 /api/overview 的 quota 区域;剩余低于 warn_ratio 时告警。
|
||||
[quota]
|
||||
enabled = false
|
||||
check_seconds = 300
|
||||
warn_ratio = 0.2
|
||||
|
||||
[runtime]
|
||||
data_dir = "./data"
|
||||
@@ -80,6 +133,11 @@ base_url = "https://deepseek.de"
|
||||
ajax_url = "https://deepseek.de/wp-admin/admin-ajax.php"
|
||||
sitekey = "0x4AAAAAADlLZ3ljqZP6cQwq"
|
||||
language = "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"
|
||||
# 站点级出口绑定(代理池条目名或 "sub:订阅名",按顺序粘性选路);
|
||||
# 留空 = 使用 [proxy] 主/备线路。三个站点可各自独立配置。
|
||||
# proxies = ["node-a", "sub:home"]
|
||||
# 只预热代理池线路(不预热 [proxy] 主/备;请求时仍可回落):
|
||||
# warm_pool_only = true
|
||||
|
||||
[sites.es]
|
||||
base_url = "https://deepseek.es"
|
||||
|
||||
@@ -9,12 +9,14 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/nyoungo/dsfree2api/internal/config"
|
||||
"github.com/nyoungo/dsfree2api/internal/httpx"
|
||||
"github.com/nyoungo/dsfree2api/internal/openai"
|
||||
"github.com/nyoungo/dsfree2api/internal/proxypool"
|
||||
"github.com/nyoungo/dsfree2api/internal/upstream"
|
||||
)
|
||||
|
||||
@@ -233,12 +235,20 @@ func (s *Server) handleActions(w http.ResponseWriter, r *http.Request) {
|
||||
Challenge string `json:"challenge_action"`
|
||||
Legacy string `json:"ts_action"`
|
||||
CookieTT int `json:"cookie_ttl_seconds"`
|
||||
APIStyle *string `json:"api_style"`
|
||||
|
||||
BrowserPath *string `json:"browser_path"`
|
||||
BrowserHeadless *bool `json:"browser_headless"`
|
||||
BrowserUserDataDir *string `json:"browser_user_data_dir"`
|
||||
BrowserTimezone *string `json:"browser_timezone"`
|
||||
BrowserLocale *string `json:"browser_locale"`
|
||||
|
||||
TimeoutSeconds *int `json:"timeout_seconds"`
|
||||
Retries *int `json:"retries"`
|
||||
RetryBackoffSeconds *float64 `json:"retry_backoff_seconds"`
|
||||
WarmEnabled *bool `json:"warm_enabled"`
|
||||
WarmRatio *float64 `json:"warm_ratio"`
|
||||
WarmCheckSeconds *int `json:"warm_check_seconds"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &p)
|
||||
// Build the candidate config first: a rejected save must not leave
|
||||
@@ -275,9 +285,30 @@ func (s *Server) handleActions(w http.ResponseWriter, r *http.Request) {
|
||||
if act := firstNonEmpty(p.Challenge, p.Legacy); act != "" {
|
||||
tc.Action = act
|
||||
}
|
||||
if p.APIStyle != nil {
|
||||
tc.APIStyle = strings.ToLower(strings.TrimSpace(*p.APIStyle))
|
||||
}
|
||||
if p.CookieTT > 0 {
|
||||
tc.CookieTTLSeconds = p.CookieTT
|
||||
}
|
||||
if p.TimeoutSeconds != nil {
|
||||
tc.TimeoutSeconds = clampInt(*p.TimeoutSeconds, 5, 600)
|
||||
}
|
||||
if p.Retries != nil {
|
||||
tc.Retries = clampInt(*p.Retries, 1, 20)
|
||||
}
|
||||
if p.RetryBackoffSeconds != nil {
|
||||
tc.RetryBackoffSeconds = clampFloat(*p.RetryBackoffSeconds, 0, 60)
|
||||
}
|
||||
if p.WarmEnabled != nil {
|
||||
tc.WarmEnabled = *p.WarmEnabled
|
||||
}
|
||||
if p.WarmRatio != nil {
|
||||
tc.WarmRatio = clampFloat(*p.WarmRatio, 0.05, 0.95)
|
||||
}
|
||||
if p.WarmCheckSeconds != nil {
|
||||
tc.WarmCheckSeconds = clampInt(*p.WarmCheckSeconds, 5, 3600)
|
||||
}
|
||||
tc.Enabled = p.Enabled
|
||||
if tc.Enabled {
|
||||
// provider-specific requirements mirror config.Validate()
|
||||
@@ -396,6 +427,218 @@ func (s *Server) handleActions(w http.ResponseWriter, r *http.Request) {
|
||||
_ = json.Unmarshal(body, &p)
|
||||
writeJSON(w, http.StatusOK, s.rotateGuest(p.Site))
|
||||
|
||||
case "pool_set_config":
|
||||
var p struct {
|
||||
Enabled *bool `json:"enabled"`
|
||||
CheckIntervalSeconds *int `json:"check_interval_seconds"`
|
||||
CheckTimeoutSeconds *int `json:"check_timeout_seconds"`
|
||||
CheckURL *string `json:"check_url"`
|
||||
DefaultScheme *string `json:"default_scheme"`
|
||||
XrayPath *string `json:"xray_path"`
|
||||
XrayVersion *string `json:"xray_version"`
|
||||
XrayAutoDownload *bool `json:"xray_auto_download"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &p)
|
||||
s.cfg.Lock()
|
||||
pc := s.cfg.ProxyPool
|
||||
if p.Enabled != nil {
|
||||
pc.Enabled = *p.Enabled
|
||||
}
|
||||
if p.CheckIntervalSeconds != nil {
|
||||
pc.CheckIntervalSeconds = clampInt(*p.CheckIntervalSeconds, 10, 3600)
|
||||
}
|
||||
if p.CheckTimeoutSeconds != nil {
|
||||
pc.CheckTimeoutSeconds = clampInt(*p.CheckTimeoutSeconds, 2, 120)
|
||||
}
|
||||
if p.CheckURL != nil {
|
||||
pc.CheckURL = strings.TrimSpace(*p.CheckURL)
|
||||
}
|
||||
if p.DefaultScheme != nil {
|
||||
pc.DefaultScheme = strings.ToLower(strings.TrimSpace(*p.DefaultScheme))
|
||||
}
|
||||
if p.XrayPath != nil {
|
||||
pc.XrayPath = strings.TrimSpace(*p.XrayPath)
|
||||
}
|
||||
if p.XrayVersion != nil {
|
||||
pc.XrayVersion = strings.TrimSpace(*p.XrayVersion)
|
||||
}
|
||||
if p.XrayAutoDownload != nil {
|
||||
pc.XrayAutoDownload = *p.XrayAutoDownload
|
||||
}
|
||||
s.cfg.ProxyPool = pc
|
||||
s.cfg.Unlock()
|
||||
s.poolReload()
|
||||
s.save(w)
|
||||
|
||||
case "pool_add_entry":
|
||||
var p struct {
|
||||
Name string `json:"name"`
|
||||
Link string `json:"link"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &p)
|
||||
p.Name, p.Link = strings.TrimSpace(p.Name), strings.TrimSpace(p.Link)
|
||||
if !validPoolName(p.Name) {
|
||||
fail(w, "name must match [A-Za-z0-9_-]{1,64}")
|
||||
return
|
||||
}
|
||||
if p.Link == "" {
|
||||
fail(w, "link is required")
|
||||
return
|
||||
}
|
||||
s.cfg.RLock()
|
||||
scheme := s.cfg.ProxyPool.DefaultScheme
|
||||
_, exists := s.cfg.ProxyPool.Entries[p.Name]
|
||||
s.cfg.RUnlock()
|
||||
if exists {
|
||||
fail(w, "entry already exists: "+p.Name)
|
||||
return
|
||||
}
|
||||
if _, err := proxypool.ParseNode(p.Link, scheme); err != nil {
|
||||
fail(w, "node parse: "+err.Error())
|
||||
return
|
||||
}
|
||||
s.cfg.Lock()
|
||||
if s.cfg.ProxyPool.Entries == nil {
|
||||
s.cfg.ProxyPool.Entries = map[string]*config.ProxyEntry{}
|
||||
}
|
||||
s.cfg.ProxyPool.Entries[p.Name] = &config.ProxyEntry{Link: p.Link, Enabled: true}
|
||||
s.cfg.Unlock()
|
||||
s.poolReload()
|
||||
s.save(w)
|
||||
|
||||
case "pool_toggle_entry", "pool_toggle_sub":
|
||||
var p struct {
|
||||
Name string `json:"name"`
|
||||
Enabled bool `json:"enabled"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &p)
|
||||
s.cfg.Lock()
|
||||
ok := false
|
||||
if req.Action == "pool_toggle_entry" {
|
||||
if e := s.cfg.ProxyPool.Entries[p.Name]; e != nil {
|
||||
e.Enabled = p.Enabled
|
||||
ok = true
|
||||
}
|
||||
} else {
|
||||
if sc := s.cfg.ProxyPool.Subscriptions[p.Name]; sc != nil {
|
||||
sc.Enabled = p.Enabled
|
||||
ok = true
|
||||
}
|
||||
}
|
||||
s.cfg.Unlock()
|
||||
if !ok {
|
||||
fail(w, "not found: "+p.Name)
|
||||
return
|
||||
}
|
||||
s.poolReload()
|
||||
s.save(w)
|
||||
|
||||
case "pool_delete_entry", "pool_delete_sub":
|
||||
var p struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &p)
|
||||
ref := p.Name
|
||||
if req.Action == "pool_delete_sub" {
|
||||
ref = "sub:" + p.Name
|
||||
}
|
||||
s.cfg.Lock()
|
||||
if req.Action == "pool_delete_entry" {
|
||||
delete(s.cfg.ProxyPool.Entries, p.Name)
|
||||
} else {
|
||||
delete(s.cfg.ProxyPool.Subscriptions, p.Name)
|
||||
}
|
||||
for _, st := range s.cfg.Sites {
|
||||
if st != nil {
|
||||
st.Proxies = removeString(st.Proxies, ref)
|
||||
}
|
||||
}
|
||||
s.cfg.Unlock()
|
||||
s.poolReload()
|
||||
s.save(w)
|
||||
|
||||
case "pool_add_sub":
|
||||
var p struct {
|
||||
Name string `json:"name"`
|
||||
URL string `json:"url"`
|
||||
IntervalMinutes int `json:"interval_minutes"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &p)
|
||||
p.Name, p.URL = strings.TrimSpace(p.Name), strings.TrimSpace(p.URL)
|
||||
if !validPoolName(p.Name) {
|
||||
fail(w, "name must match [A-Za-z0-9_-]{1,64}")
|
||||
return
|
||||
}
|
||||
if p.URL == "" {
|
||||
fail(w, "url is required")
|
||||
return
|
||||
}
|
||||
s.cfg.Lock()
|
||||
if s.cfg.ProxyPool.Subscriptions == nil {
|
||||
s.cfg.ProxyPool.Subscriptions = map[string]*config.ProxySubscription{}
|
||||
}
|
||||
if _, exists := s.cfg.ProxyPool.Subscriptions[p.Name]; exists {
|
||||
s.cfg.Unlock()
|
||||
fail(w, "subscription already exists: "+p.Name)
|
||||
return
|
||||
}
|
||||
if p.IntervalMinutes <= 0 {
|
||||
p.IntervalMinutes = 60
|
||||
}
|
||||
s.cfg.ProxyPool.Subscriptions[p.Name] = &config.ProxySubscription{
|
||||
URL: p.URL, Enabled: true,
|
||||
IntervalMinutes: clampInt(p.IntervalMinutes, 5, 1440),
|
||||
}
|
||||
s.cfg.Unlock()
|
||||
s.poolReload()
|
||||
s.save(w)
|
||||
|
||||
case "pool_set_site":
|
||||
var p struct {
|
||||
Site string `json:"site"`
|
||||
Proxies []string `json:"proxies"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &p)
|
||||
clean := make([]string, 0, len(p.Proxies))
|
||||
for _, item := range p.Proxies {
|
||||
if item = strings.TrimSpace(item); item != "" {
|
||||
clean = append(clean, item)
|
||||
}
|
||||
}
|
||||
s.cfg.Lock()
|
||||
st, ok := s.cfg.Sites[p.Site]
|
||||
if ok && st != nil {
|
||||
for _, item := range clean {
|
||||
if name, isSub := strings.CutPrefix(item, "sub:"); isSub {
|
||||
if s.cfg.ProxyPool.Subscriptions[name] == nil {
|
||||
s.cfg.Unlock()
|
||||
fail(w, "unknown subscription: "+name)
|
||||
return
|
||||
}
|
||||
} else if s.cfg.ProxyPool.Entries[item] == nil {
|
||||
s.cfg.Unlock()
|
||||
fail(w, "unknown entry: "+item)
|
||||
return
|
||||
}
|
||||
}
|
||||
st.Proxies = clean
|
||||
}
|
||||
s.cfg.Unlock()
|
||||
if !ok || st == nil {
|
||||
fail(w, "unknown site: "+p.Site)
|
||||
return
|
||||
}
|
||||
s.poolReload()
|
||||
s.save(w)
|
||||
|
||||
case "pool_refresh":
|
||||
if s.pool == nil {
|
||||
fail(w, "proxy pool is not available")
|
||||
return
|
||||
}
|
||||
s.pool.RefreshNow(context.Background())
|
||||
writeJSON(w, http.StatusOK, map[string]any{"ok": true})
|
||||
|
||||
default:
|
||||
fail(w, "unknown action: "+req.Action)
|
||||
}
|
||||
@@ -673,3 +916,45 @@ func randomToken(n int) string {
|
||||
}
|
||||
return hex.EncodeToString(b)
|
||||
}
|
||||
|
||||
func clampInt(v, lo, hi int) int {
|
||||
if v < lo {
|
||||
return lo
|
||||
}
|
||||
if v > hi {
|
||||
return hi
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func clampFloat(v, lo, hi float64) float64 {
|
||||
if v < lo {
|
||||
return lo
|
||||
}
|
||||
if v > hi {
|
||||
return hi
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
var poolNameRe = regexp.MustCompile(`^[A-Za-z0-9_-]{1,64}$`)
|
||||
|
||||
func validPoolName(name string) bool { return poolNameRe.MatchString(name) }
|
||||
|
||||
// removeString drops every occurrence of want from list (in place copy).
|
||||
func removeString(list []string, want string) []string {
|
||||
out := list[:0:0]
|
||||
for _, v := range list {
|
||||
if v != want {
|
||||
out = append(out, v)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// poolReload pushes pool-relevant config changes into the live manager.
|
||||
func (s *Server) poolReload() {
|
||||
if s.pool != nil {
|
||||
s.pool.Reload()
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,8 @@ import (
|
||||
"github.com/nyoungo/dsfree2api/internal/config"
|
||||
"github.com/nyoungo/dsfree2api/internal/logbuf"
|
||||
"github.com/nyoungo/dsfree2api/internal/metrics"
|
||||
"github.com/nyoungo/dsfree2api/internal/proxypool"
|
||||
"github.com/nyoungo/dsfree2api/internal/quota"
|
||||
"github.com/nyoungo/dsfree2api/internal/turnstile"
|
||||
"github.com/nyoungo/dsfree2api/internal/upstream"
|
||||
)
|
||||
@@ -31,6 +33,8 @@ type Server struct {
|
||||
met *metrics.Recorder
|
||||
logs *logbuf.Buffer
|
||||
ts *turnstile.Solver
|
||||
pool *proxypool.Manager
|
||||
quota *quota.Watcher
|
||||
log *slog.Logger
|
||||
start time.Time
|
||||
|
||||
@@ -39,7 +43,7 @@ type Server struct {
|
||||
password string
|
||||
}
|
||||
|
||||
func New(cfg *config.Config, up *upstream.Client, met *metrics.Recorder, logs *logbuf.Buffer, ts *turnstile.Solver, log *slog.Logger) *Server {
|
||||
func New(cfg *config.Config, up *upstream.Client, met *metrics.Recorder, logs *logbuf.Buffer, ts *turnstile.Solver, log *slog.Logger, pool *proxypool.Manager, qw *quota.Watcher) *Server {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
@@ -50,7 +54,7 @@ func New(cfg *config.Config, up *upstream.Client, met *metrics.Recorder, logs *l
|
||||
generated = true
|
||||
}
|
||||
s := &Server{
|
||||
cfg: cfg, up: up, met: met, logs: logs, ts: ts, log: log,
|
||||
cfg: cfg, up: up, met: met, logs: logs, ts: ts, pool: pool, quota: qw, log: log,
|
||||
start: time.Now(), tokens: map[string]time.Time{}, password: pw,
|
||||
}
|
||||
if generated {
|
||||
@@ -185,7 +189,7 @@ func (s *Server) handleOverview(w http.ResponseWriter, _ *http.Request) {
|
||||
sites = append(sites, map[string]any{
|
||||
"code": code, "base_url": st.BaseURL, "ajax_url": st.AJAXURL,
|
||||
"language": st.Language, "sitekey": st.SiteKey, "enabled": st.Enabled,
|
||||
"verify_action": st.VerifyAction,
|
||||
"verify_action": st.VerifyAction, "proxies": st.Proxies,
|
||||
})
|
||||
}
|
||||
proxyCfg := struct {
|
||||
@@ -201,12 +205,23 @@ func (s *Server) handleOverview(w http.ResponseWriter, _ *http.Request) {
|
||||
keys := append([]string(nil), s.cfg.Security.APIKeys...)
|
||||
s.cfg.RUnlock()
|
||||
|
||||
var poolStatus any
|
||||
if s.pool != nil {
|
||||
poolStatus = s.pool.Status()
|
||||
}
|
||||
var quotaStatus any
|
||||
if s.quota != nil {
|
||||
quotaStatus = s.quota.Snapshot()
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, map[string]any{
|
||||
"metrics": s.met.Snapshot(),
|
||||
"models": models,
|
||||
"sites": sites,
|
||||
"proxy": proxyCfg,
|
||||
"turnstile": map[string]any{"config": turnstileCfg, "status": s.ts.Status(siteCodes), "history": s.ts.History()},
|
||||
"turnstile": map[string]any{"config": turnstileCfg, "status": s.ts.Status(siteCodes), "history": s.ts.History(), "pool": s.ts.PoolStatus()},
|
||||
"proxypool": poolStatus,
|
||||
"quota": quotaStatus,
|
||||
"keys": keys,
|
||||
"cache": map[string]any{"chat_config_entries": s.up.ConfigCacheSize(), "routes": routeList(s.up)},
|
||||
"system": map[string]any{
|
||||
@@ -269,6 +284,9 @@ func (s *Server) handlePutConfig(w http.ResponseWriter, r *http.Request) {
|
||||
// applyRuntime pushes a freshly edited config into the live subsystems.
|
||||
func (s *Server) applyRuntime() {
|
||||
s.ts.SetConfig(s.currentTurnstile())
|
||||
if s.pool != nil {
|
||||
s.pool.Reload()
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) currentTurnstile() config.Turnstile {
|
||||
|
||||
+194
-2
@@ -328,6 +328,7 @@ const mask = k => k.length > 16 ? k.slice(0, 7) + "…" + k.slice(-6) : k;
|
||||
PAGES.proxy = async () => {
|
||||
const ov = await fetchOverview();
|
||||
const p = ov.proxy;
|
||||
const pool = ov.proxypool || { xray: {}, entries: [], subscriptions: [], sticky: {} };
|
||||
view.innerHTML = `
|
||||
<div class="grid g2">
|
||||
<div class="card"><h3>主线路</h3>
|
||||
@@ -355,6 +356,49 @@ PAGES.proxy = async () => {
|
||||
<div class="row"><input id="px-fb" placeholder="socks5://..." style="width:280px"><button class="btn" id="px-add">添加</button></div></div>
|
||||
<div class="table-wrap mt"><table><thead><tr><th>#</th><th>地址</th><th></th></tr></thead><tbody id="tb-fb"></tbody></table></div>
|
||||
<div class="dim mt">当前生效顺序:${ov.cache.routes.map(r => `<span class="tag">${esc(r.name)} → ${esc(r.proxy)}</span>`).join(" ")}</div>
|
||||
</div>
|
||||
<div class="grid g2 mt">
|
||||
<div class="card"><h3>代理池 · Xray 核心</h3>
|
||||
<div class="row">
|
||||
<div class="field"><label>池开关</label>${switchHTML("pool-en", pool.enabled)}</div>
|
||||
<div class="field"><label>检查间隔(秒)</label><input id="pool-check-int" type="number" min="10" value="${pool.check_interval_seconds || 120}"></div>
|
||||
<div class="field"><label>检查超时(秒)</label><input id="pool-check-to" type="number" min="2" value="${pool.check_timeout_seconds || 10}"></div>
|
||||
</div>
|
||||
<div class="row">
|
||||
<div class="field"><label>健康检查 URL</label><input id="pool-check-url" value="${esc(pool.check_url || "")}" placeholder="https://www.gstatic.com/generate_204"></div>
|
||||
<div class="field"><label>裸 ip:port 默认协议</label><input id="pool-scheme" value="${esc(pool.default_scheme || "http")}" placeholder="http / socks5"></div>
|
||||
</div>
|
||||
<div class="row">
|
||||
<div class="field"><label>Xray 路径(留空 = 自动)</label><input id="pool-xray-path" value="${esc(pool.xray_path || "")}" placeholder="留空自动下载到数据目录"></div>
|
||||
<div class="field"><label>版本(留空 = latest)</label><input id="pool-xray-ver" value="${esc(pool.xray_version || "")}"></div>
|
||||
<div class="field"><label>自动下载</label>${switchHTML("pool-xray-auto", pool.xray_auto_download)}</div>
|
||||
</div>
|
||||
<div class="row mt">
|
||||
<button class="btn primary" id="pool-save">保存池配置</button>
|
||||
<button class="btn" id="pool-refresh">刷新节点与健康检查</button>
|
||||
</div>
|
||||
<div class="dim mt">Xray:<span class="pill ${pool.xray.running ? "on" : "off"}">${pool.xray.running ? "运行中 · pid " + pool.xray.pid : "未运行"}</span>
|
||||
${pool.xray.error ? `<span style="color:var(--err)"> ${esc(pool.xray.error)}</span>` : ""}
|
||||
${pool.last_error ? `<br><span style="color:var(--err)">${esc(pool.last_error)}</span>` : ""}</div>
|
||||
</div>
|
||||
<div class="card"><h3>节点与端点</h3>
|
||||
<div class="row"><input id="pool-name" placeholder="名称(字母数字-_)" style="width:180px">
|
||||
<input id="pool-link" placeholder="vless://… 或 ip:port 或 socks5://user:pass@host:port" style="flex:1;min-width:200px">
|
||||
<button class="btn" id="pool-add">添加</button></div>
|
||||
<div class="table-wrap mt"><table><thead><tr><th>名称</th><th>类型</th><th>目标</th><th>健康</th><th class="num">延迟</th><th></th></tr></thead><tbody id="tb-pool"></tbody></table></div>
|
||||
<div class="dim mt">分享链接(vless / vmess / trojan / ss)由内置 Xray 核心承载;ip:port / socks5:// 等端点直接使用。</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="card mt"><div class="toolbar"><h3 style="margin:0">订阅</h3>
|
||||
<div class="row"><input id="sub-name" placeholder="名称" style="width:150px">
|
||||
<input id="sub-url" placeholder="https://…/sub?token=…" style="width:360px">
|
||||
<input id="sub-int" type="number" min="5" value="60" title="刷新间隔(分钟)" style="width:90px">
|
||||
<button class="btn" id="sub-add">添加订阅</button></div></div>
|
||||
<div class="table-wrap mt"><table><thead><tr><th>名称</th><th>地址</th><th class="num">间隔</th><th class="num">节点</th><th>上次拉取</th><th></th></tr></thead><tbody id="tb-subs"></tbody></table></div>
|
||||
</div>
|
||||
<div class="card mt"><h3>站点绑定(按顺序,粘性选路)</h3>
|
||||
<div class="table-wrap"><table><thead><tr><th>站点</th><th>绑定(逗号分隔:名称 或 sub:订阅名)</th><th>当前出口</th><th></th></tr></thead><tbody id="tb-bind"></tbody></table></div>
|
||||
<div class="dim mt">未绑定、或绑定节点全部不健康时,回退到主线路 / 备用线路。Cookie 与出口 IP 绑定:切换出口后该站点会在下一轮巡检重新求解 Turnstile。</div>
|
||||
</div>`;
|
||||
|
||||
const renderFb = list => {
|
||||
@@ -369,6 +413,98 @@ PAGES.proxy = async () => {
|
||||
};
|
||||
renderFb(p.fallbacks || []);
|
||||
|
||||
const renderPool = () => {
|
||||
const entries = pool.entries || [];
|
||||
$("#tb-pool").innerHTML = entries.length ? entries.map(e => {
|
||||
const health = e.parse_error ? `<span class="pill off" title="${esc(e.parse_error)}">解析失败</span>`
|
||||
: e.checked ? (e.healthy ? '<span class="pill on">健康</span>' : `<span class="pill off" title="${esc(e.last_error || "")}">不可用</span>`)
|
||||
: '<span class="pill">未探测</span>';
|
||||
return `<tr><td><code>${esc(e.name)}</code>${e.source ? ` <span class="dim">← ${esc(e.source)}</span>` : ""}</td>
|
||||
<td>${esc(e.kind === "url" ? e.scheme : (e.scheme || "xray"))}</td>
|
||||
<td class="dim">${esc(e.target || "")}</td>
|
||||
<td>${health}</td><td class="num">${e.latency_ms ? e.latency_ms + "ms" : "–"}</td>
|
||||
<td class="right"><button class="btn small" data-ptoggle="${esc(e.name)}" data-pen="${e.enabled ? "0" : "1"}">${e.enabled ? "停用" : "启用"}</button>
|
||||
${e.source ? "" : `<button class="btn small danger" data-pdel="${esc(e.name)}">删除</button>`}</td></tr>`;
|
||||
}).join("") : '<tr><td colspan="6" class="empty">尚无节点 —— 添加分享链接或 ip:port 端点</td></tr>';
|
||||
$$("[data-ptoggle]").forEach(b => b.onclick = async () => {
|
||||
try { await api("/api/actions", { body: { action: "pool_toggle_entry", name: b.dataset.ptoggle, enabled: b.dataset.pen === "1" } }); PAGES.proxy(); }
|
||||
catch (e) { toast(e.message, "err"); }
|
||||
});
|
||||
$$("[data-pdel]").forEach(b => b.onclick = async () => {
|
||||
if (!confirm("删除节点 " + b.dataset.pdel + "?")) return;
|
||||
try { await api("/api/actions", { body: { action: "pool_delete_entry", name: b.dataset.pdel } }); PAGES.proxy(); }
|
||||
catch (e) { toast(e.message, "err"); }
|
||||
});
|
||||
|
||||
const subs = pool.subscriptions || [];
|
||||
$("#tb-subs").innerHTML = subs.length ? subs.map(s => `
|
||||
<tr><td><code>${esc(s.name)}</code></td><td class="dim">${esc(s.url)}</td>
|
||||
<td class="num">${s.interval_minutes || 60}m</td><td class="num">${s.nodes || 0}</td>
|
||||
<td class="dim">${s.fetched_at ? timeStr(s.fetched_at * 1000) : "未拉取"}${s.last_error ? ` <span style="color:var(--err)">${esc(String(s.last_error).slice(0, 60))}</span>` : ""}</td>
|
||||
<td class="right"><button class="btn small" data-stoggle="${esc(s.name)}" data-sen="${s.enabled ? "0" : "1"}">${s.enabled ? "停用" : "启用"}</button>
|
||||
<button class="btn small danger" data-sdel="${esc(s.name)}">删除</button></td></tr>`).join("")
|
||||
: '<tr><td colspan="6" class="empty">尚无订阅</td></tr>';
|
||||
$$("[data-stoggle]").forEach(b => b.onclick = async () => {
|
||||
try { await api("/api/actions", { body: { action: "pool_toggle_sub", name: b.dataset.stoggle, enabled: b.dataset.sen === "1" } }); PAGES.proxy(); }
|
||||
catch (e) { toast(e.message, "err"); }
|
||||
});
|
||||
$$("[data-sdel]").forEach(b => b.onclick = async () => {
|
||||
if (!confirm("删除订阅 " + b.dataset.sdel + "?")) return;
|
||||
try { await api("/api/actions", { body: { action: "pool_delete_sub", name: b.dataset.sdel } }); PAGES.proxy(); }
|
||||
catch (e) { toast(e.message, "err"); }
|
||||
});
|
||||
|
||||
const sticky = pool.sticky || {};
|
||||
$("#tb-bind").innerHTML = (ov.sites || []).map(s => {
|
||||
const cur = sticky[s.code];
|
||||
const curEntry = entries.find(e => e.name === cur);
|
||||
return `<tr><td><b>${esc(s.code.toUpperCase())}</b></td>
|
||||
<td><input data-bind="${esc(s.code)}" value="${esc((s.proxies || []).join(", "))}" placeholder="留空 = 全局线路"></td>
|
||||
<td class="dim">${cur ? `${esc(cur)} → ${esc(curEntry ? curEntry.target : "?")}` : "全局线路"}</td>
|
||||
<td class="right"><button class="btn small" data-bsave="${esc(s.code)}">保存</button></td></tr>`;
|
||||
}).join("");
|
||||
$$("[data-bsave]").forEach(b => b.onclick = async () => {
|
||||
const input = $(`[data-bind="${CSS.escape(b.dataset.bsave)}"]`);
|
||||
const list = input.value.split(",").map(x => x.trim()).filter(Boolean);
|
||||
try { await api("/api/actions", { body: { action: "pool_set_site", site: b.dataset.bsave, proxies: list } }); toast("已保存", "ok"); PAGES.proxy(); }
|
||||
catch (e) { toast(e.message, "err"); }
|
||||
});
|
||||
};
|
||||
renderPool();
|
||||
|
||||
$("#pool-save").onclick = async () => {
|
||||
try {
|
||||
await api("/api/actions", { body: {
|
||||
action: "pool_set_config",
|
||||
enabled: $("#pool-en").checked,
|
||||
check_interval_seconds: +$("#pool-check-int").value || 120,
|
||||
check_timeout_seconds: +$("#pool-check-to").value || 10,
|
||||
check_url: $("#pool-check-url").value.trim(),
|
||||
default_scheme: $("#pool-scheme").value.trim() || "http",
|
||||
xray_path: $("#pool-xray-path").value.trim(),
|
||||
xray_version: $("#pool-xray-ver").value.trim(),
|
||||
xray_auto_download: $("#pool-xray-auto").checked } });
|
||||
toast("池配置已保存", "ok"); PAGES.proxy();
|
||||
} catch (e) { toast(e.message, "err"); }
|
||||
};
|
||||
$("#pool-refresh").onclick = async () => {
|
||||
const b = $("#pool-refresh"); b.disabled = true; b.textContent = "刷新中…";
|
||||
try { await api("/api/actions", { body: { action: "pool_refresh" } }); toast("已在后台拉取订阅并检查健康,稍后重进本页查看", "ok"); PAGES.proxy(); }
|
||||
catch (e) { toast(e.message, "err"); b.disabled = false; b.textContent = "刷新节点与健康检查"; }
|
||||
};
|
||||
$("#pool-add").onclick = async () => {
|
||||
try {
|
||||
await api("/api/actions", { body: { action: "pool_add_entry", name: $("#pool-name").value.trim(), link: $("#pool-link").value.trim() } });
|
||||
toast("已添加", "ok"); PAGES.proxy();
|
||||
} catch (e) { toast(e.message, "err"); }
|
||||
};
|
||||
$("#sub-add").onclick = async () => {
|
||||
try {
|
||||
await api("/api/actions", { body: { action: "pool_add_sub", name: $("#sub-name").value.trim(), url: $("#sub-url").value.trim(), interval_minutes: +$("#sub-int").value || 60 } });
|
||||
toast("已添加", "ok"); PAGES.proxy();
|
||||
} catch (e) { toast(e.message, "err"); }
|
||||
};
|
||||
|
||||
$("#px-save").onclick = async () => {
|
||||
try {
|
||||
await api("/api/config", { method: "PUT", body: await merged(c => { c.proxy.url = $("#px-primary").value.trim(); }) });
|
||||
@@ -406,6 +542,7 @@ async function merged(mutate) {
|
||||
}
|
||||
|
||||
PAGES.turnstile = async () => {
|
||||
if (overviewTimer) { clearInterval(overviewTimer); overviewTimer = null; }
|
||||
const ov = await fetchOverview();
|
||||
const t = ov.turnstile.config;
|
||||
view.innerHTML = `
|
||||
@@ -426,6 +563,10 @@ PAGES.turnstile = async () => {
|
||||
<div class="row">
|
||||
<div class="field"><label>Sitekey</label><input id="ts-sitekey" value="${esc(t.sitekey)}"></div>
|
||||
<div class="field"><label>Action</label><input id="ts-action" value="${esc(t.action)}"></div>
|
||||
<div class="field"><label>API 风格</label><select id="ts-style">
|
||||
<option value="sync">sync(CapSolver 形状)</option>
|
||||
<option value="ezsolver">ezsolver(本地 EzSolver)</option>
|
||||
</select></div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="ts-browser" class="ts-panel" style="display:none">
|
||||
@@ -459,6 +600,20 @@ PAGES.turnstile = async () => {
|
||||
<div class="dim">Cookie 与出口 IP / UA 绑定:请在同一代理线路的浏览器中获取。导入后在 TTL 内免求解,状态见右侧「已验证 Cookie」。</div>
|
||||
<button class="btn primary mt" id="im-save">导入</button>
|
||||
</div>
|
||||
<div class="ts-panel">
|
||||
<div class="ts-panel-title">求解与 Cookie 池</div>
|
||||
<div class="row">
|
||||
<div class="field"><label>求解超时(秒)</label><input id="ts-timeout" type="number" min="5" max="600" value="${t.timeout_seconds || 90}"></div>
|
||||
<div class="field"><label>失败重试(次)</label><input id="ts-retries" type="number" min="1" max="20" value="${t.retries || 5}"></div>
|
||||
<div class="field"><label>重试退避(秒)</label><input id="ts-backoff" type="number" min="0" max="60" step="0.5" value="${t.retry_backoff_seconds == null ? 1.5 : t.retry_backoff_seconds}"></div>
|
||||
</div>
|
||||
<div class="row">
|
||||
<div class="field"><label>Cookie 池(后台预热)</label>${switchHTML("ts-warm", !!t.warm_enabled)}</div>
|
||||
<div class="field"><label>预热阈值(剩余 %)</label><input id="ts-ratio" type="number" min="5" max="95" value="${Math.round((t.warm_ratio == null ? 0.3 : t.warm_ratio) * 100)}"></div>
|
||||
<div class="field"><label>检查间隔(秒)</label><input id="ts-check" type="number" min="5" max="3600" value="${t.warm_check_seconds || 60}"></div>
|
||||
</div>
|
||||
<div class="dim">Cookie 池开启后:后台每隔检查间隔巡检一次,剩余 TTL 低于阈值时主动重解,请求路径不再等待求解耗时(manual 方式不预热)。</div>
|
||||
</div>
|
||||
<div class="dim" id="ts-hint" style="margin-top:10px"></div>
|
||||
<button class="btn primary mt" id="ts-save">保存配置</button>
|
||||
</div>
|
||||
@@ -468,7 +623,11 @@ PAGES.turnstile = async () => {
|
||||
<button class="btn" id="ts-refresh">强制刷新全部</button>
|
||||
<button class="btn danger" id="ts-clear">清空 Cookie</button>
|
||||
</div>
|
||||
<div class="dim mt">缓存 TTL ${t.cookie_ttl_seconds}s · 求解失败重试 ${t.retries} 次 · 退避 ${t.retry_backoff_seconds}s</div>
|
||||
<div class="dim mt">缓存 TTL ${t.cookie_ttl_seconds}s · 求解超时 ${t.timeout_seconds || 90}s · 重试 ${t.retries} 次 · 退避 ${t.retry_backoff_seconds}s</div>
|
||||
</div>
|
||||
<div class="card"><h3>Cookie 池</h3>
|
||||
<div class="dim" id="ts-pool-state"></div>
|
||||
<div id="ts-pool" class="grid mt" style="gap:8px"></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="card mt"><h3>求解历史</h3><div class="table-wrap"><table>
|
||||
@@ -489,13 +648,39 @@ PAGES.turnstile = async () => {
|
||||
<td><span class="pill ${h.ok ? "on" : "off"}">${h.ok ? "成功" : "失败"}</span>${h.error ? ` <span class="dim">${esc(h.error.slice(0, 70))}</span>` : ""}</td></tr>`).join("")
|
||||
: '<tr><td colspan="5" class="empty">尚无记录</td></tr>';
|
||||
|
||||
const renderPool = (o) => {
|
||||
const pool = (o.turnstile && o.turnstile.pool) || { entries: [] };
|
||||
const label = $("#ts-pool-state");
|
||||
if (label) {
|
||||
label.textContent = `状态:${pool.enabled ? (pool.running ? "运行中" : "已启用,等待下一轮巡检") : "未开启"} · 阈值 剩余 ${Math.round((pool.ratio || 0.3) * 100)}% · 每 ${pool.check_seconds || 60}s 巡检一次`;
|
||||
}
|
||||
const box = $("#ts-pool");
|
||||
if (!box) return;
|
||||
const entries = pool.entries || [];
|
||||
box.innerHTML = entries.length ? entries.map(e => {
|
||||
const state = e.warming ? '<span class="pill warn">预热中</span>'
|
||||
: e.valid ? '<span class="pill on">就绪</span>'
|
||||
: e.last_error ? '<span class="pill off">求解失败</span>' : '<span class="pill off">未建立</span>';
|
||||
const left = e.valid ? `剩余 ${Math.max(0, Math.round(e.remaining_s / 60))} 分钟` : "等待预热";
|
||||
const next = e.next_warm_at ? `下次 ${timeStr(e.next_warm_at * 1000)}` : "—";
|
||||
const last = e.last_at ? `上次 ${timeStr(e.last_at * 1000)} · ${fmtMs(e.last_ms)}` : "尚未求解";
|
||||
return `<div class="card" style="padding:10px">
|
||||
<div class="toolbar"><div><b>${esc(e.site.toUpperCase())}</b> <span class="dim">${esc(e.route)}</span></div>${state}</div>
|
||||
<div class="dim mt">${left} · ${next}<br>${last}${e.last_error ? `<br><span style="color:var(--err)">${esc(String(e.last_error).slice(0, 90))}</span>` : ""}</div>
|
||||
</div>`;
|
||||
}).join("") : '<div class="dim">暂无池内记录 — 开启预热并等待下一轮巡检</div>';
|
||||
};
|
||||
renderPool(ov);
|
||||
overviewTimer = setInterval(() => { fetchOverview().then(renderPool).catch(() => {}); }, 5000);
|
||||
|
||||
$$("[data-rf]").forEach(b => b.onclick = async () => {
|
||||
b.disabled = true;
|
||||
try { await api("/api/actions", { body: { action: "refresh_cookies", site: b.dataset.rf } }); toast("已作废,将在下次请求时重新求解", "ok"); PAGES.turnstile(); }
|
||||
try { await api("/api/actions", { body: { action: "refresh_cookies", site: b.dataset.rf } }); toast(t.warm_enabled ? "已作废,Cookie 池将在下一轮巡检时重新预热" : "已作废,将在下次请求时重新求解", "ok"); PAGES.turnstile(); }
|
||||
catch (e) { toast(e.message, "err"); }
|
||||
b.disabled = false;
|
||||
});
|
||||
$("#ts-provider").value = t.provider || "api";
|
||||
$("#ts-style").value = t.api_style === "ezsolver" ? "ezsolver" : "sync";
|
||||
const TS_HINTS = {
|
||||
api: ["已开启:调用求解服务 API 获取 Turnstile Token", "已关闭:跳过 Turnstile 求解,直接请求上游站点"],
|
||||
browser: ["已开启:用下方配置的本地浏览器自动求解 Token", "已关闭:跳过 Turnstile 求解,直接请求上游站点"],
|
||||
@@ -521,6 +706,13 @@ PAGES.turnstile = async () => {
|
||||
api_url: $("#ts-url").value.trim(),
|
||||
api_key: $("#ts-key").value.trim(), sitekey: $("#ts-sitekey").value.trim(),
|
||||
challenge_action: $("#ts-action").value.trim(), cookie_ttl_seconds: +$("#ts-ttl").value,
|
||||
api_style: $("#ts-style").value,
|
||||
timeout_seconds: Math.round(+$("#ts-timeout").value || 90),
|
||||
retries: Math.round(+$("#ts-retries").value || 5),
|
||||
retry_backoff_seconds: Math.max(0, +$("#ts-backoff").value || 0),
|
||||
warm_enabled: $("#ts-warm").checked,
|
||||
warm_ratio: Math.min(95, Math.max(5, Math.round(+$("#ts-ratio").value || 30))) / 100,
|
||||
warm_check_seconds: Math.round(+$("#ts-check").value || 60),
|
||||
browser_path: $("#ts-bpath").value.trim(),
|
||||
browser_headless: $("#ts-bhead").checked,
|
||||
browser_user_data_dir: $("#ts-bprof").value.trim(),
|
||||
|
||||
@@ -83,6 +83,7 @@ label{display:block;font-size:12px;color:var(--muted);margin-bottom:5px}
|
||||
.pill{display:inline-block;padding:2px 8px;border-radius:20px;font-size:11px;border:1px solid var(--line);color:var(--muted)}
|
||||
.pill.on{color:var(--ok);border-color:#1f6a34;background:#0f2318}
|
||||
.pill.off{color:var(--dim)}
|
||||
.pill.warn{color:var(--warn);border-color:#6b5417;background:#2a2210}
|
||||
|
||||
/* console / logs */
|
||||
.log-box{background:#0a0e14;border:1px solid var(--line);border-radius:10px;padding:10px;height:56vh;overflow:auto;font-family:var(--mono);font-size:12.5px;line-height:1.55}
|
||||
|
||||
+230
-2
@@ -14,6 +14,15 @@ import (
|
||||
|
||||
const DefaultUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36"
|
||||
|
||||
// Turnstile tuning defaults. DefaultSolveTimeoutSeconds is the single source
|
||||
// for how long one solve attempt may run; the warm_* values drive the cookie
|
||||
// pool (background pre-solving).
|
||||
const (
|
||||
DefaultSolveTimeoutSeconds = 90
|
||||
DefaultWarmRatio = 0.3
|
||||
DefaultWarmCheckSeconds = 60
|
||||
)
|
||||
|
||||
// Turnstile token providers: api calls a hosted solver, browser drives a
|
||||
// local Chrome/Edge over CDP, manual only ever uses imported cookies.
|
||||
const (
|
||||
@@ -29,11 +38,18 @@ type Turnstile struct {
|
||||
APIKey string `toml:"api_key" json:"api_key"`
|
||||
SiteKey string `toml:"sitekey" json:"sitekey"`
|
||||
Action string `toml:"action" json:"action"`
|
||||
APIStyle string `toml:"api_style" json:"api_style"`
|
||||
TimeoutSeconds int `toml:"timeout_seconds" json:"timeout_seconds"`
|
||||
CookieTTLSeconds int `toml:"cookie_ttl_seconds" json:"cookie_ttl_seconds"`
|
||||
Retries int `toml:"retries" json:"retries"`
|
||||
RetryBackoffSeconds float64 `toml:"retry_backoff_seconds" json:"retry_backoff_seconds"`
|
||||
|
||||
// Cookie pool: a background loop re-solves a site × route slot before its
|
||||
// cached cookie expires, so requests never pay the solve latency.
|
||||
WarmEnabled bool `toml:"warm_enabled" json:"warm_enabled"`
|
||||
WarmRatio float64 `toml:"warm_ratio" json:"warm_ratio"`
|
||||
WarmCheckSeconds int `toml:"warm_check_seconds" json:"warm_check_seconds"`
|
||||
|
||||
// provider = "browser"
|
||||
BrowserPath string `toml:"browser_path" json:"browser_path"`
|
||||
BrowserHeadless bool `toml:"browser_headless" json:"browser_headless"`
|
||||
@@ -51,6 +67,112 @@ func (t Turnstile) ProviderValue() string {
|
||||
return p
|
||||
}
|
||||
|
||||
// APIStyleValue normalizes api_style for provider "api": "sync" (CapSolver-
|
||||
// shaped single POST) or "ezsolver" (POST {sitekey,siteurl,timeout} → {token}).
|
||||
func (t Turnstile) APIStyleValue() string {
|
||||
switch strings.ToLower(strings.TrimSpace(t.APIStyle)) {
|
||||
case "ezsolver", "ez":
|
||||
return "ezsolver"
|
||||
default:
|
||||
return "sync"
|
||||
}
|
||||
}
|
||||
|
||||
// SolveTimeoutValue normalizes timeout_seconds: the longest one solve attempt
|
||||
// may run (default 90s, clamped to 5–600s).
|
||||
func (t Turnstile) SolveTimeoutValue() int {
|
||||
n := t.TimeoutSeconds
|
||||
if n <= 0 {
|
||||
return DefaultSolveTimeoutSeconds
|
||||
}
|
||||
if n < 5 {
|
||||
return 5
|
||||
}
|
||||
if n > 600 {
|
||||
return 600
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// WarmRatioValue normalizes warm_ratio: a pooled cookie is re-solved once its
|
||||
// remaining TTL drops below this fraction (default 30%, clamped 5–95%).
|
||||
func (t Turnstile) WarmRatioValue() float64 {
|
||||
r := t.WarmRatio
|
||||
if r <= 0 {
|
||||
return DefaultWarmRatio
|
||||
}
|
||||
if r < 0.05 {
|
||||
return 0.05
|
||||
}
|
||||
if r > 0.95 {
|
||||
return 0.95
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// WarmCheckValue normalizes warm_check_seconds: how often the pool is
|
||||
// inspected (default 60s, clamped 5s–1h).
|
||||
func (t Turnstile) WarmCheckValue() int {
|
||||
n := t.WarmCheckSeconds
|
||||
if n <= 0 {
|
||||
return DefaultWarmCheckSeconds
|
||||
}
|
||||
if n < 5 {
|
||||
return 5
|
||||
}
|
||||
if n > 3600 {
|
||||
return 3600
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// Quota sentinel defaults: polling interval and the remaining-quota warning
|
||||
// threshold (share of the daily free limit left).
|
||||
const (
|
||||
DefaultQuotaCheckSeconds = 300
|
||||
DefaultQuotaWarnRatio = 0.2
|
||||
)
|
||||
|
||||
// QuotaWatch configures the guest-token balance sentinel: it polls each
|
||||
// site's balance endpoint with the pooled session (same egress and cookies as
|
||||
// chat) and reports remaining daily quota to the console and logs.
|
||||
type QuotaWatch struct {
|
||||
Enabled bool `toml:"enabled" json:"enabled"`
|
||||
CheckSeconds int `toml:"check_seconds" json:"check_seconds"`
|
||||
WarnRatio float64 `toml:"warn_ratio" json:"warn_ratio"`
|
||||
}
|
||||
|
||||
// CheckSecondsValue normalizes check_seconds (default 300s, clamped 30–3600).
|
||||
func (q QuotaWatch) CheckSecondsValue() int {
|
||||
n := q.CheckSeconds
|
||||
if n <= 0 {
|
||||
return DefaultQuotaCheckSeconds
|
||||
}
|
||||
if n < 30 {
|
||||
return 30
|
||||
}
|
||||
if n > 3600 {
|
||||
return 3600
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// WarnRatioValue normalizes warn_ratio: remaining/limit below this fraction
|
||||
// raises a "quota low" warning (default 20%, clamped 5–95%).
|
||||
func (q QuotaWatch) WarnRatioValue() float64 {
|
||||
r := q.WarnRatio
|
||||
if r <= 0 {
|
||||
return DefaultQuotaWarnRatio
|
||||
}
|
||||
if r < 0.05 {
|
||||
return 0.05
|
||||
}
|
||||
if r > 0.95 {
|
||||
return 0.95
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
type Site struct {
|
||||
Code string `toml:"-"`
|
||||
BaseURL string `toml:"base_url" json:"base_url"`
|
||||
@@ -59,6 +181,41 @@ type Site struct {
|
||||
VerifyAction string `toml:"verify_action" json:"verify_action"`
|
||||
Language string `toml:"language" json:"language"`
|
||||
Enabled bool `toml:"enabled" json:"enabled"`
|
||||
// Proxies binds this site to the proxy pool: an ordered list of entry
|
||||
// names and/or "sub:<name>" references. Empty = global [proxy] routes.
|
||||
Proxies []string `toml:"proxies" json:"proxies"`
|
||||
// WarmPoolOnly keeps the cookie warmer on the pool routes bound via
|
||||
// Proxies; the global [proxy] primary/fallback lanes are not pre-solved
|
||||
// (they remain available for request-time failover). Default false.
|
||||
WarmPoolOnly bool `toml:"warm_pool_only" json:"warm_pool_only"`
|
||||
}
|
||||
|
||||
// ProxyEntry is one manually configured pool node: an Xray share link
|
||||
// (vless/vmess/trojan/ss) or a plain http/https/socks5 endpoint.
|
||||
type ProxyEntry struct {
|
||||
Link string `toml:"link" json:"link"`
|
||||
Enabled bool `toml:"enabled" json:"enabled"`
|
||||
}
|
||||
|
||||
// ProxySubscription is a remote node list refreshed on an interval.
|
||||
type ProxySubscription struct {
|
||||
URL string `toml:"url" json:"url"`
|
||||
Enabled bool `toml:"enabled" json:"enabled"`
|
||||
IntervalMinutes int `toml:"interval_minutes" json:"interval_minutes"`
|
||||
}
|
||||
|
||||
// ProxyPool configures the egress pool and its managed Xray core.
|
||||
type ProxyPool struct {
|
||||
Enabled bool `toml:"enabled" json:"enabled"`
|
||||
CheckIntervalSeconds int `toml:"check_interval_seconds" json:"check_interval_seconds"`
|
||||
CheckTimeoutSeconds int `toml:"check_timeout_seconds" json:"check_timeout_seconds"`
|
||||
CheckURL string `toml:"check_url" json:"check_url"`
|
||||
DefaultScheme string `toml:"default_scheme" json:"default_scheme"`
|
||||
XrayPath string `toml:"xray_path" json:"xray_path"`
|
||||
XrayVersion string `toml:"xray_version" json:"xray_version"`
|
||||
XrayAutoDownload bool `toml:"xray_auto_download" json:"xray_auto_download"`
|
||||
Entries map[string]*ProxyEntry `toml:"entries" json:"entries"`
|
||||
Subscriptions map[string]*ProxySubscription `toml:"subscriptions" json:"subscriptions"`
|
||||
}
|
||||
|
||||
type Model struct {
|
||||
@@ -77,6 +234,8 @@ type Config struct {
|
||||
Host string `toml:"host" json:"host"`
|
||||
Port int `toml:"port" json:"port"`
|
||||
LogLevel string `toml:"log_level" json:"log_level"`
|
||||
// LogFile: "" = <data_dir>/logs/dsfree2api.log, "-" = stderr only.
|
||||
LogFile string `toml:"log_file" json:"log_file"`
|
||||
} `toml:"server" json:"server"`
|
||||
|
||||
Security struct {
|
||||
@@ -101,6 +260,12 @@ type Config struct {
|
||||
SlowStartSeconds float64 `toml:"slow_start_seconds" json:"slow_start_seconds"`
|
||||
} `toml:"proxy" json:"proxy"`
|
||||
|
||||
ProxyPool ProxyPool `toml:"proxypool" json:"proxypool"`
|
||||
|
||||
// Quota is the balance sentinel: it polls each site's guest-token balance
|
||||
// so the console shows how much daily free quota is left.
|
||||
Quota QuotaWatch `toml:"quota" json:"quota"`
|
||||
|
||||
Upstream struct {
|
||||
Timeout float64 `toml:"timeout" json:"timeout"`
|
||||
StreamTimeout float64 `toml:"stream_timeout" json:"stream_timeout"`
|
||||
@@ -125,6 +290,20 @@ type Config struct {
|
||||
path string `toml:"-"`
|
||||
}
|
||||
|
||||
// DefaultProxyPool returns the built-in pool settings (feature off, Xray
|
||||
// auto-download on).
|
||||
func DefaultProxyPool() ProxyPool {
|
||||
return ProxyPool{
|
||||
CheckIntervalSeconds: 120,
|
||||
CheckTimeoutSeconds: 10,
|
||||
CheckURL: "https://www.gstatic.com/generate_204",
|
||||
DefaultScheme: "http",
|
||||
XrayAutoDownload: true,
|
||||
Entries: map[string]*ProxyEntry{},
|
||||
Subscriptions: map[string]*ProxySubscription{},
|
||||
}
|
||||
}
|
||||
|
||||
func DefaultTurnstile() Turnstile {
|
||||
return Turnstile{
|
||||
Enabled: false,
|
||||
@@ -133,10 +312,12 @@ func DefaultTurnstile() Turnstile {
|
||||
APIKey: "",
|
||||
SiteKey: "0x4AAAAAADlLZ3ljqZP6cQwq",
|
||||
Action: "chat",
|
||||
TimeoutSeconds: 90,
|
||||
TimeoutSeconds: DefaultSolveTimeoutSeconds,
|
||||
CookieTTLSeconds: 10800,
|
||||
Retries: 5,
|
||||
RetryBackoffSeconds: 1.5,
|
||||
WarmRatio: DefaultWarmRatio,
|
||||
WarmCheckSeconds: DefaultWarmCheckSeconds,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -201,6 +382,7 @@ func newConfig() *Config {
|
||||
c.Upstream.CrossSiteFailover = true
|
||||
c.Upstream.ContinueRounds = 20
|
||||
c.Runtime.DataDir = "./data"
|
||||
c.ProxyPool = DefaultProxyPool()
|
||||
c.Turnstile = DefaultTurnstile()
|
||||
c.Sites = DefaultSites(c.Turnstile.SiteKey)
|
||||
c.Models = DefaultModels()
|
||||
@@ -300,22 +482,36 @@ func (c *Config) applyDefaults(md *toml.MetaData) {
|
||||
c.Turnstile.Enabled = true
|
||||
c.Upstream.AutoRefresh = true
|
||||
c.Upstream.ContinueRounds = 20
|
||||
c.ProxyPool.XrayAutoDownload = true
|
||||
for _, s := range c.Sites {
|
||||
s.Enabled = true
|
||||
}
|
||||
for _, m := range c.Models {
|
||||
m.Enabled = true
|
||||
}
|
||||
for _, e := range c.ProxyPool.Entries {
|
||||
e.Enabled = true
|
||||
}
|
||||
for _, sc := range c.ProxyPool.Subscriptions {
|
||||
sc.Enabled = true
|
||||
}
|
||||
return
|
||||
}
|
||||
def(md.IsDefined("turnstile", "enabled"), &c.Turnstile.Enabled, true)
|
||||
def(md.IsDefined("upstream", "auto_refresh"), &c.Upstream.AutoRefresh, true)
|
||||
def(md.IsDefined("proxypool", "xray_auto_download"), &c.ProxyPool.XrayAutoDownload, true)
|
||||
for code, s := range c.Sites {
|
||||
def(md.IsDefined("sites", code, "enabled"), &s.Enabled, true)
|
||||
}
|
||||
for id, m := range c.Models {
|
||||
def(md.IsDefined("models", id, "enabled"), &m.Enabled, true)
|
||||
}
|
||||
for name, e := range c.ProxyPool.Entries {
|
||||
def(md.IsDefined("proxypool", "entries", name, "enabled"), &e.Enabled, true)
|
||||
}
|
||||
for name, sc := range c.ProxyPool.Subscriptions {
|
||||
def(md.IsDefined("proxypool", "subscriptions", name, "enabled"), &sc.Enabled, true)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Config) applyEnv() error {
|
||||
@@ -331,6 +527,9 @@ func (c *Config) applyEnv() error {
|
||||
if v := str("LOG_LEVEL"); v != nil {
|
||||
c.Server.LogLevel = *v
|
||||
}
|
||||
if v := str("LOG_FILE"); v != nil {
|
||||
c.Server.LogFile = strings.TrimSpace(*v)
|
||||
}
|
||||
if v := str("PORT"); v != nil {
|
||||
n, err := strconv.Atoi(strings.TrimSpace(*v))
|
||||
if err != nil || n < 1 || n > 65535 {
|
||||
@@ -363,6 +562,34 @@ func (c *Config) applyEnv() error {
|
||||
if v := str("TURNSTILE_BROWSER_PATH"); v != nil && strings.TrimSpace(*v) != "" {
|
||||
c.Turnstile.BrowserPath = strings.TrimSpace(*v)
|
||||
}
|
||||
if v := str("TURNSTILE_TIMEOUT_SECONDS"); v != nil {
|
||||
n, err := strconv.Atoi(strings.TrimSpace(*v))
|
||||
if err != nil || n < 1 {
|
||||
return fmt.Errorf("invalid TURNSTILE_TIMEOUT_SECONDS %q", *v)
|
||||
}
|
||||
c.Turnstile.TimeoutSeconds = n
|
||||
}
|
||||
if v := str("TURNSTILE_WARM_ENABLED"); v != nil {
|
||||
b, err := strconv.ParseBool(strings.TrimSpace(*v))
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid TURNSTILE_WARM_ENABLED %q", *v)
|
||||
}
|
||||
c.Turnstile.WarmEnabled = b
|
||||
}
|
||||
if v := str("TURNSTILE_WARM_RATIO"); v != nil {
|
||||
f, err := strconv.ParseFloat(strings.TrimSpace(*v), 64)
|
||||
if err != nil || f <= 0 || f > 1 {
|
||||
return fmt.Errorf("invalid TURNSTILE_WARM_RATIO %q (expected 0–1)", *v)
|
||||
}
|
||||
c.Turnstile.WarmRatio = f
|
||||
}
|
||||
if v := str("TURNSTILE_WARM_CHECK_SECONDS"); v != nil {
|
||||
n, err := strconv.Atoi(strings.TrimSpace(*v))
|
||||
if err != nil || n < 1 {
|
||||
return fmt.Errorf("invalid TURNSTILE_WARM_CHECK_SECONDS %q", *v)
|
||||
}
|
||||
c.Turnstile.WarmCheckSeconds = n
|
||||
}
|
||||
if v := str("TURNSTILE_ENABLED"); v != nil {
|
||||
b, err := strconv.ParseBool(strings.TrimSpace(*v))
|
||||
if err != nil {
|
||||
@@ -452,7 +679,8 @@ func (c *Config) validateTurnstile() error {
|
||||
if c.Turnstile.APIURL == "" {
|
||||
return errors.New("turnstile.api_url is empty (set config to your solver endpoint, or use provider = \"manual\")")
|
||||
}
|
||||
if c.Turnstile.APIKey == "" {
|
||||
// Local solve services (api_style = "ezsolver") usually need no key.
|
||||
if c.Turnstile.APIKey == "" && c.Turnstile.APIStyleValue() != "ezsolver" {
|
||||
return errors.New("turnstile.api_key is empty (set config, or TURNSTILE_API_KEY)")
|
||||
}
|
||||
case ProviderBrowser:
|
||||
|
||||
@@ -100,6 +100,11 @@ func TestValidateRejectsBadValues(t *testing.T) {
|
||||
if err := cfg.Validate(); err == nil {
|
||||
t.Error("expected error for empty turnstile api_key")
|
||||
}
|
||||
cfg.Turnstile.APIStyle = "ezsolver"
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Errorf("ezsolver style without api_key: %v", err)
|
||||
}
|
||||
cfg.Turnstile.APIStyle = ""
|
||||
cfg.Turnstile.APIKey = "k"
|
||||
cfg.Turnstile.Enabled = false
|
||||
cfg.Admin.Port = cfg.Server.Port
|
||||
@@ -204,6 +209,78 @@ func TestTurnstileProviderEnv(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestProxyPoolDefaults(t *testing.T) {
|
||||
d := DefaultProxyPool()
|
||||
if d.Enabled {
|
||||
t.Error("proxy pool must be opt-in")
|
||||
}
|
||||
if !d.XrayAutoDownload || d.CheckIntervalSeconds != 120 || d.CheckTimeoutSeconds != 10 {
|
||||
t.Errorf("pool defaults = %+v", d)
|
||||
}
|
||||
if d.DefaultScheme != "http" {
|
||||
t.Errorf("default scheme = %q", d.DefaultScheme)
|
||||
}
|
||||
cfg, err := Load(examplePath)
|
||||
if err != nil {
|
||||
t.Fatalf("load: %v", err)
|
||||
}
|
||||
if !cfg.ProxyPool.XrayAutoDownload {
|
||||
t.Error("example config should keep xray_auto_download = true")
|
||||
}
|
||||
if cfg.ProxyPool.CheckIntervalSeconds != 120 {
|
||||
t.Errorf("check interval = %d", cfg.ProxyPool.CheckIntervalSeconds)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTurnstileSolveTimeoutAndWarmDefaults(t *testing.T) {
|
||||
d := DefaultTurnstile()
|
||||
if d.WarmEnabled {
|
||||
t.Error("cookie pool must be opt-in")
|
||||
}
|
||||
if d.WarmRatio != 0.3 || d.WarmCheckSeconds != 60 {
|
||||
t.Errorf("warm defaults = %v / %v", d.WarmRatio, d.WarmCheckSeconds)
|
||||
}
|
||||
z := Turnstile{}
|
||||
if z.SolveTimeoutValue() != 90 {
|
||||
t.Errorf("zero timeout should normalize to 90, got %d", z.SolveTimeoutValue())
|
||||
}
|
||||
if z.WarmRatioValue() != 0.3 || z.WarmCheckValue() != 60 {
|
||||
t.Errorf("zero warm values should normalize to 0.3 / 60")
|
||||
}
|
||||
if got := (Turnstile{TimeoutSeconds: 1}).SolveTimeoutValue(); got != 5 {
|
||||
t.Errorf("timeout floor = %d, want 5", got)
|
||||
}
|
||||
if got := (Turnstile{TimeoutSeconds: 9999}).SolveTimeoutValue(); got != 600 {
|
||||
t.Errorf("timeout ceiling = %d, want 600", got)
|
||||
}
|
||||
if got := (Turnstile{WarmRatio: 0.99}).WarmRatioValue(); got != 0.95 {
|
||||
t.Errorf("warm ratio ceiling = %v, want 0.95", got)
|
||||
}
|
||||
if (Turnstile{}).APIStyleValue() != "sync" {
|
||||
t.Error("zero api_style should normalize to sync")
|
||||
}
|
||||
if (Turnstile{APIStyle: "EzSolver"}).APIStyleValue() != "ezsolver" {
|
||||
t.Error("api_style should normalize EzSolver")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTurnstileSolveAndWarmEnvOverrides(t *testing.T) {
|
||||
t.Setenv("TURNSTILE_TIMEOUT_SECONDS", "45")
|
||||
t.Setenv("TURNSTILE_WARM_ENABLED", "true")
|
||||
t.Setenv("TURNSTILE_WARM_RATIO", "0.5")
|
||||
t.Setenv("TURNSTILE_WARM_CHECK_SECONDS", "120")
|
||||
cfg, err := Load(examplePath)
|
||||
if err != nil {
|
||||
t.Fatalf("load: %v", err)
|
||||
}
|
||||
if cfg.Turnstile.TimeoutSeconds != 45 {
|
||||
t.Errorf("timeout = %d, want 45", cfg.Turnstile.TimeoutSeconds)
|
||||
}
|
||||
if !cfg.Turnstile.WarmEnabled || cfg.Turnstile.WarmRatio != 0.5 || cfg.Turnstile.WarmCheckSeconds != 120 {
|
||||
t.Errorf("warm env = %+v", cfg.Turnstile)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSaveRoundTrip(t *testing.T) {
|
||||
cfg, err := Load(examplePath)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
// Package logfile writes logs to a daily file and compresses the previous
|
||||
// day's file on rotation. It is an io.Writer meant to sit behind an slog
|
||||
// handler, next to the stderr writer.
|
||||
package logfile
|
||||
|
||||
import (
|
||||
"compress/gzip"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
const dayLayout = "2006-01-02"
|
||||
|
||||
// Writer appends to path and rotates once per day: the retired file becomes
|
||||
// "<base>-YYYY-MM-DD.log.gz" next to it.
|
||||
type Writer struct {
|
||||
mu sync.Mutex
|
||||
dir string
|
||||
base string
|
||||
path string
|
||||
file *os.File
|
||||
day string
|
||||
closed bool
|
||||
now func() time.Time
|
||||
wg sync.WaitGroup
|
||||
}
|
||||
|
||||
// New opens (or creates) path and immediately rotates a stale file left over
|
||||
// from an earlier day.
|
||||
func New(path string) (*Writer, error) {
|
||||
path = strings.TrimSpace(path)
|
||||
if path == "" {
|
||||
return nil, fmt.Errorf("log file path is empty")
|
||||
}
|
||||
dir := filepath.Dir(path)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
w := &Writer{dir: dir, base: baseName(path), path: path, now: time.Now}
|
||||
if st, err := os.Stat(path); err == nil {
|
||||
day := st.ModTime().Format(dayLayout)
|
||||
if day != w.now().Format(dayLayout) && st.Size() > 0 {
|
||||
w.day = day
|
||||
if err := w.rotateLocked(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return w, nil
|
||||
}
|
||||
}
|
||||
return w, w.openLocked()
|
||||
}
|
||||
|
||||
// Write appends p, rotating first when the local day changed.
|
||||
func (w *Writer) Write(p []byte) (int, error) {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
if w.closed {
|
||||
return 0, os.ErrClosed
|
||||
}
|
||||
if w.file == nil {
|
||||
if err := w.openLocked(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
}
|
||||
if day := w.now().Format(dayLayout); day != w.day {
|
||||
if err := w.rotateLocked(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
}
|
||||
return w.file.Write(p)
|
||||
}
|
||||
|
||||
// Close flushes and waits for any in-flight compression.
|
||||
func (w *Writer) Close() error {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
if w.closed {
|
||||
return nil
|
||||
}
|
||||
w.closed = true
|
||||
var err error
|
||||
if w.file != nil {
|
||||
err = w.file.Close()
|
||||
w.file = nil
|
||||
}
|
||||
w.wg.Wait()
|
||||
return err
|
||||
}
|
||||
|
||||
func (w *Writer) openLocked() error {
|
||||
f, err := os.OpenFile(w.path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
w.file = f
|
||||
w.day = w.now().Format(dayLayout)
|
||||
return nil
|
||||
}
|
||||
|
||||
// rotateLocked retires the current file — renaming it first so the async
|
||||
// compression can never read a freshly opened file — and starts a new one.
|
||||
func (w *Writer) rotateLocked() error {
|
||||
if w.file != nil {
|
||||
_ = w.file.Close()
|
||||
w.file = nil
|
||||
}
|
||||
old, day := w.path, w.day
|
||||
if st, err := os.Stat(old); err == nil && st.Size() > 0 {
|
||||
tmp := old + ".rotating"
|
||||
if err := os.Rename(old, tmp); err == nil {
|
||||
gz := filepath.Join(w.dir, fmt.Sprintf("%s-%s.log.gz", w.base, day))
|
||||
w.wg.Add(1)
|
||||
go func() {
|
||||
defer w.wg.Done()
|
||||
compressFile(tmp, gz)
|
||||
}()
|
||||
}
|
||||
} else {
|
||||
_ = os.Remove(old)
|
||||
}
|
||||
return w.openLocked()
|
||||
}
|
||||
|
||||
// compressFile gzips src into dst and removes src on success; on any failure
|
||||
// the raw .rotating file is left in place so no log data is ever lost.
|
||||
// NOTE: the source handle is closed before removal — Windows refuses to
|
||||
// delete a file that is still open.
|
||||
func compressFile(src, dst string) {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
_ = in.Close()
|
||||
return
|
||||
}
|
||||
gz := gzip.NewWriter(out)
|
||||
_, copyErr := io.Copy(gz, in)
|
||||
_ = in.Close()
|
||||
if copyErr != nil {
|
||||
gz.Close()
|
||||
out.Close()
|
||||
_ = os.Remove(dst)
|
||||
return
|
||||
}
|
||||
if err := gz.Close(); err != nil {
|
||||
out.Close()
|
||||
_ = os.Remove(dst)
|
||||
return
|
||||
}
|
||||
if err := out.Close(); err != nil {
|
||||
_ = os.Remove(dst)
|
||||
return
|
||||
}
|
||||
_ = os.Remove(src)
|
||||
}
|
||||
|
||||
func baseName(path string) string {
|
||||
base := filepath.Base(path)
|
||||
if ext := filepath.Ext(base); ext != "" {
|
||||
base = strings.TrimSuffix(base, ext)
|
||||
}
|
||||
return base
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package logfile
|
||||
|
||||
import (
|
||||
"compress/gzip"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func readGz(t *testing.T, path string) string {
|
||||
t.Helper()
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
t.Fatalf("open %s: %v", path, err)
|
||||
}
|
||||
defer f.Close()
|
||||
gz, err := gzip.NewReader(f)
|
||||
if err != nil {
|
||||
t.Fatalf("gzip %s: %v", path, err)
|
||||
}
|
||||
defer gz.Close()
|
||||
b, err := io.ReadAll(gz)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", path, err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func TestDailyRotationCompresses(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "dsfree2api.log")
|
||||
w, err := New(path)
|
||||
if err != nil {
|
||||
t.Fatalf("new: %v", err)
|
||||
}
|
||||
day1 := w.now()
|
||||
if _, err := w.Write([]byte("day-one\n")); err != nil {
|
||||
t.Fatalf("write: %v", err)
|
||||
}
|
||||
|
||||
// Cross midnight: the next write rotates and compresses day one.
|
||||
w.now = func() time.Time { return day1.Add(24 * time.Hour) }
|
||||
if _, err := w.Write([]byte("day-two\n")); err != nil {
|
||||
t.Fatalf("write: %v", err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatalf("close: %v", err)
|
||||
}
|
||||
|
||||
cur, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("read current: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(cur), "day-two") || strings.Contains(string(cur), "day-one") {
|
||||
t.Fatalf("current log = %q", cur)
|
||||
}
|
||||
gzPath := filepath.Join(dir, "dsfree2api-"+day1.Format(dayLayout)+".log.gz")
|
||||
if got := readGz(t, gzPath); !strings.Contains(got, "day-one") {
|
||||
t.Fatalf("rotated log = %q", got)
|
||||
}
|
||||
if _, err := os.Stat(path + ".rotating"); !os.IsNotExist(err) {
|
||||
t.Error(".rotating file should be gone after compression")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteAfterCloseFails(t *testing.T) {
|
||||
w, err := New(filepath.Join(t.TempDir(), "x.log"))
|
||||
if err != nil {
|
||||
t.Fatalf("new: %v", err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatalf("close: %v", err)
|
||||
}
|
||||
if _, err := w.Write([]byte("no\n")); err == nil {
|
||||
t.Fatal("write after close should fail")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStaleFileRotatedOnOpen(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "app.log")
|
||||
if err := os.WriteFile(path, []byte("old\n"), 0o644); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
old := time.Now().Add(-48 * time.Hour)
|
||||
if err := os.Chtimes(path, old, old); err != nil {
|
||||
t.Fatalf("chtimes: %v", err)
|
||||
}
|
||||
w, err := New(path)
|
||||
if err != nil {
|
||||
t.Fatalf("new: %v", err)
|
||||
}
|
||||
if _, err := w.Write([]byte("new\n")); err != nil {
|
||||
t.Fatalf("write: %v", err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatalf("close: %v", err)
|
||||
}
|
||||
gzPath := filepath.Join(dir, "app-"+old.Format(dayLayout)+".log.gz")
|
||||
if got := readGz(t, gzPath); !strings.Contains(got, "old") {
|
||||
t.Fatalf("rotated = %q", got)
|
||||
}
|
||||
cur, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("read current: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(cur), "new") {
|
||||
t.Fatalf("current = %q", cur)
|
||||
}
|
||||
}
|
||||
@@ -46,9 +46,11 @@ type modelStats struct {
|
||||
}
|
||||
|
||||
type siteStats struct {
|
||||
Requests int64 `json:"requests"`
|
||||
Errors int64 `json:"errors"`
|
||||
Quota int64 `json:"quota_exhausted"`
|
||||
Requests int64 `json:"requests"`
|
||||
Errors int64 `json:"errors"`
|
||||
Quota int64 `json:"quota_exhausted"`
|
||||
Prompt int64 `json:"prompt_tokens"`
|
||||
Completed int64 `json:"completion_tokens"`
|
||||
}
|
||||
|
||||
type dayStats struct {
|
||||
@@ -228,6 +230,8 @@ func (r *Recorder) Record(rec Record) {
|
||||
r.sites[rec.Site] = s
|
||||
}
|
||||
s.Requests++
|
||||
s.Prompt += int64(rec.PromptTokens)
|
||||
s.Completed += int64(rec.CompletionTokens)
|
||||
if rec.Status == StatusError || rec.Status == StatusEmpty {
|
||||
s.Errors++
|
||||
}
|
||||
@@ -264,6 +268,45 @@ func (r *Recorder) Record(rec Record) {
|
||||
}
|
||||
}
|
||||
|
||||
// SiteTotals is the cumulative per-site usage snapshot; it sizes a quota
|
||||
// exhaustion — how many tokens had flowed before the site cut us off.
|
||||
type SiteTotals struct {
|
||||
Requests int64 `json:"requests"`
|
||||
PromptTokens int64 `json:"prompt_tokens"`
|
||||
CompletionTokens int64 `json:"completion_tokens"`
|
||||
Quota int64 `json:"quota_exhausted"`
|
||||
}
|
||||
|
||||
// SiteTotals returns the cumulative usage attributed to one site so far.
|
||||
// Totals persist across restarts through metrics.json.
|
||||
func (r *Recorder) SiteTotals(site string) SiteTotals {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
s := r.sites[site]
|
||||
if s == nil {
|
||||
return SiteTotals{}
|
||||
}
|
||||
return SiteTotals{
|
||||
Requests: s.Requests,
|
||||
PromptTokens: s.Prompt,
|
||||
CompletionTokens: s.Completed,
|
||||
Quota: s.Quota,
|
||||
}
|
||||
}
|
||||
|
||||
// IncSiteQuota counts one site-side quota exhaustion that may never surface as
|
||||
// a failed request (e.g. swallowed by cross-site failover).
|
||||
func (r *Recorder) IncSiteQuota(site string) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
s := r.sites[site]
|
||||
if s == nil {
|
||||
s = &siteStats{}
|
||||
r.sites[site] = s
|
||||
}
|
||||
s.Quota++
|
||||
}
|
||||
|
||||
func (r *Recorder) Snapshot() any {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
package metrics
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestRecordTracksSiteTokensAndQuota(t *testing.T) {
|
||||
r := New(t.TempDir())
|
||||
r.Record(Record{Model: "deepseek-v4-flash-de", Site: "de", PromptTokens: 120, CompletionTokens: 80, Status: StatusOK})
|
||||
r.Record(Record{Model: "deepseek-v4-flash-de", Site: "de", PromptTokens: 15, Status: StatusError, Error: "sse quota exhausted"})
|
||||
|
||||
got := r.SiteTotals("de")
|
||||
if got.Requests != 2 {
|
||||
t.Fatalf("Requests = %d, want 2", got.Requests)
|
||||
}
|
||||
if got.PromptTokens != 135 || got.CompletionTokens != 80 {
|
||||
t.Errorf("tokens = %d/%d, want 135/80", got.PromptTokens, got.CompletionTokens)
|
||||
}
|
||||
|
||||
r.IncSiteQuota("de")
|
||||
if got := r.SiteTotals("de"); got.Quota != 1 {
|
||||
t.Errorf("Quota = %d, want 1", got.Quota)
|
||||
}
|
||||
|
||||
if empty := r.SiteTotals("fr"); empty.Requests != 0 || empty.Quota != 0 {
|
||||
t.Errorf("unknown site totals = %+v, want zero", empty)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,722 @@
|
||||
package proxypool
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/nyoungo/dsfree2api/internal/config"
|
||||
"github.com/nyoungo/dsfree2api/internal/upstream"
|
||||
)
|
||||
|
||||
// endpoint is one runtime pool slot.
|
||||
type endpoint struct {
|
||||
name string
|
||||
kind string // "xray" | "url" | "invalid"
|
||||
scheme string
|
||||
display string
|
||||
link string
|
||||
source string // "" = manual entry, else subscription name
|
||||
enabled bool
|
||||
|
||||
parseErr string
|
||||
outbound map[string]any
|
||||
url string
|
||||
proxyURL string // resolved egress URL (socks5://127.0.0.1:port or plain)
|
||||
|
||||
checked bool
|
||||
healthy bool
|
||||
latencyMs int64
|
||||
lastErr string
|
||||
checkedAt time.Time
|
||||
}
|
||||
|
||||
type subState struct {
|
||||
name string
|
||||
nodes []string
|
||||
lastErr string
|
||||
fetchedAt time.Time
|
||||
}
|
||||
|
||||
// Manager owns the proxy pool: endpoints, health, subscriptions and sticky
|
||||
// per-site selection. It is safe for concurrent use.
|
||||
type Manager struct {
|
||||
cfg *config.Config
|
||||
log *slog.Logger
|
||||
dir string
|
||||
xray *xrayCore
|
||||
|
||||
reloadMu sync.Mutex
|
||||
|
||||
mu sync.Mutex
|
||||
endpoints map[string]*endpoint
|
||||
subs map[string]*subState
|
||||
sticky map[string]string
|
||||
fetching map[string]bool
|
||||
started bool
|
||||
lastCheck time.Time
|
||||
lastErr string
|
||||
}
|
||||
|
||||
func New(cfg *config.Config, dataDir string, log *slog.Logger) *Manager {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
return &Manager{
|
||||
cfg: cfg,
|
||||
log: log,
|
||||
dir: dataDir,
|
||||
xray: newXrayCore(dataDir, log),
|
||||
endpoints: map[string]*endpoint{},
|
||||
subs: map[string]*subState{},
|
||||
sticky: map[string]string{},
|
||||
fetching: map[string]bool{},
|
||||
}
|
||||
}
|
||||
|
||||
// Start runs the background loop: reload from config, refresh due
|
||||
// subscriptions, probe endpoint health.
|
||||
func (m *Manager) Start(ctx context.Context) {
|
||||
m.mu.Lock()
|
||||
if m.started {
|
||||
m.mu.Unlock()
|
||||
return
|
||||
}
|
||||
m.started = true
|
||||
m.mu.Unlock()
|
||||
go m.loop(ctx)
|
||||
}
|
||||
|
||||
func (m *Manager) loop(ctx context.Context) {
|
||||
ticker := time.NewTicker(5 * time.Second)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
m.xray.stop()
|
||||
return
|
||||
case <-ticker.C:
|
||||
}
|
||||
m.cycle(ctx)
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Manager) cycle(ctx context.Context) {
|
||||
m.Reload()
|
||||
|
||||
m.cfg.RLock()
|
||||
enabled := m.cfg.ProxyPool.Enabled
|
||||
interval := time.Duration(m.cfg.ProxyPool.CheckIntervalSeconds) * time.Second
|
||||
subs := make(map[string]config.ProxySubscription, len(m.cfg.ProxyPool.Subscriptions))
|
||||
for name, sc := range m.cfg.ProxyPool.Subscriptions {
|
||||
if sc != nil {
|
||||
subs[name] = *sc
|
||||
}
|
||||
}
|
||||
m.cfg.RUnlock()
|
||||
if !enabled {
|
||||
return
|
||||
}
|
||||
if interval <= 0 {
|
||||
interval = 120 * time.Second
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
due := m.lastCheck.IsZero() || time.Since(m.lastCheck) >= interval
|
||||
m.mu.Unlock()
|
||||
if due {
|
||||
m.checkAll(ctx)
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
for name, sc := range subs {
|
||||
if !sc.Enabled || strings.TrimSpace(sc.URL) == "" {
|
||||
continue
|
||||
}
|
||||
m.mu.Lock()
|
||||
st := m.subs[name]
|
||||
busy := m.fetching[name]
|
||||
m.mu.Unlock()
|
||||
if busy {
|
||||
continue
|
||||
}
|
||||
iv := time.Duration(sc.IntervalMinutes) * time.Minute
|
||||
if iv <= 0 {
|
||||
iv = time.Hour
|
||||
}
|
||||
if st != nil && !st.fetchedAt.IsZero() && now.Sub(st.fetchedAt) < iv {
|
||||
continue
|
||||
}
|
||||
go m.fetchSub(ctx, name, sc)
|
||||
}
|
||||
}
|
||||
|
||||
// Reload rebuilds the runtime endpoint table from the live config plus the
|
||||
// cached subscription nodes, (re)configures the Xray core and prunes sticky
|
||||
// picks that no longer resolve.
|
||||
func (m *Manager) Reload() {
|
||||
m.reloadMu.Lock()
|
||||
defer m.reloadMu.Unlock()
|
||||
|
||||
m.cfg.RLock()
|
||||
poolCfg := m.cfg.ProxyPool
|
||||
entries := make(map[string]config.ProxyEntry, len(poolCfg.Entries))
|
||||
for name, e := range poolCfg.Entries {
|
||||
if e != nil {
|
||||
entries[name] = *e
|
||||
}
|
||||
}
|
||||
subsCfg := make(map[string]config.ProxySubscription, len(poolCfg.Subscriptions))
|
||||
for name, sc := range poolCfg.Subscriptions {
|
||||
if sc != nil {
|
||||
subsCfg[name] = *sc
|
||||
}
|
||||
}
|
||||
m.cfg.RUnlock()
|
||||
|
||||
m.mu.Lock()
|
||||
subNodes := make(map[string][]string, len(m.subs))
|
||||
for name, st := range m.subs {
|
||||
subNodes[name] = append([]string(nil), st.nodes...)
|
||||
}
|
||||
old := m.endpoints
|
||||
m.mu.Unlock()
|
||||
|
||||
desired := map[string]*endpoint{}
|
||||
for name, e := range entries {
|
||||
if strings.TrimSpace(e.Link) == "" {
|
||||
continue
|
||||
}
|
||||
ep := newEndpoint(name, e.Link, poolCfg.DefaultScheme, "")
|
||||
ep.enabled = e.Enabled
|
||||
desired[name] = ep
|
||||
}
|
||||
for subName, nodes := range subNodes {
|
||||
sc, ok := subsCfg[subName]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
for _, line := range nodes {
|
||||
name := subName + "#" + shortHash(line)
|
||||
if _, dup := desired[name]; dup {
|
||||
continue
|
||||
}
|
||||
ep := newEndpoint(name, line, poolCfg.DefaultScheme, subName)
|
||||
ep.enabled = sc.Enabled
|
||||
desired[name] = ep
|
||||
}
|
||||
}
|
||||
for name, ep := range desired {
|
||||
if o := old[name]; o != nil && o.link == ep.link {
|
||||
ep.checked = o.checked
|
||||
ep.healthy = o.healthy
|
||||
ep.latencyMs = o.latencyMs
|
||||
ep.lastErr = o.lastErr
|
||||
ep.checkedAt = o.checkedAt
|
||||
}
|
||||
}
|
||||
|
||||
nodes := make([]xrayNode, 0, len(desired))
|
||||
for _, ep := range desired {
|
||||
if ep.kind == "xray" && ep.enabled && ep.parseErr == "" && ep.outbound != nil {
|
||||
nodes = append(nodes, xrayNode{Name: ep.name, Outbound: ep.outbound})
|
||||
}
|
||||
}
|
||||
sort.Slice(nodes, func(i, j int) bool { return nodes[i].Name < nodes[j].Name })
|
||||
|
||||
if len(nodes) > 0 {
|
||||
if err := m.xray.ensure(poolCfg.XrayPath, poolCfg.XrayAutoDownload, poolCfg.XrayVersion); err != nil {
|
||||
m.setErr(err)
|
||||
} else if err := m.xray.configure(nodes); err != nil {
|
||||
m.setErr(err)
|
||||
} else {
|
||||
m.setErr(nil)
|
||||
}
|
||||
} else {
|
||||
_ = m.xray.configure(nil)
|
||||
m.setErr(nil)
|
||||
}
|
||||
|
||||
xrayUp := m.xray.status().Running
|
||||
for _, ep := range desired {
|
||||
switch ep.kind {
|
||||
case "url":
|
||||
ep.proxyURL = ep.url
|
||||
case "xray":
|
||||
if ep.parseErr == "" && xrayUp {
|
||||
if port := m.xray.port(ep.name); port > 0 {
|
||||
ep.proxyURL = fmt.Sprintf("socks5://127.0.0.1:%d", port)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
m.endpoints = desired
|
||||
for site, name := range m.sticky {
|
||||
ep := desired[name]
|
||||
if ep == nil || !ep.enabled || ep.proxyURL == "" {
|
||||
delete(m.sticky, site)
|
||||
}
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
// RefreshNow refreshes every subscription and probes every endpoint, then
|
||||
// reloads. Used by the console button.
|
||||
func (m *Manager) RefreshNow(ctx context.Context) {
|
||||
go func() {
|
||||
m.Reload()
|
||||
m.cfg.RLock()
|
||||
enabled := m.cfg.ProxyPool.Enabled
|
||||
subs := make(map[string]config.ProxySubscription, len(m.cfg.ProxyPool.Subscriptions))
|
||||
for name, sc := range m.cfg.ProxyPool.Subscriptions {
|
||||
if sc != nil {
|
||||
subs[name] = *sc
|
||||
}
|
||||
}
|
||||
m.cfg.RUnlock()
|
||||
for name, sc := range subs {
|
||||
if sc.Enabled && strings.TrimSpace(sc.URL) != "" {
|
||||
m.fetchSub(ctx, name, sc)
|
||||
}
|
||||
}
|
||||
if enabled {
|
||||
m.checkAll(ctx)
|
||||
}
|
||||
m.Reload()
|
||||
}()
|
||||
}
|
||||
|
||||
func (m *Manager) fetchSub(ctx context.Context, name string, sc config.ProxySubscription) {
|
||||
m.mu.Lock()
|
||||
m.fetching[name] = true
|
||||
m.mu.Unlock()
|
||||
defer func() {
|
||||
m.mu.Lock()
|
||||
delete(m.fetching, name)
|
||||
m.mu.Unlock()
|
||||
}()
|
||||
|
||||
client, err := m.fetchClient()
|
||||
if err == nil {
|
||||
var req *http.Request
|
||||
req, err = http.NewRequestWithContext(ctx, http.MethodGet, sc.URL, nil)
|
||||
if err == nil {
|
||||
var resp *http.Response
|
||||
resp, err = client.Do(req)
|
||||
if err == nil {
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
err = fmt.Errorf("subscription http %d", resp.StatusCode)
|
||||
} else {
|
||||
var raw []byte
|
||||
raw, err = io.ReadAll(io.LimitReader(resp.Body, 8<<20))
|
||||
if err == nil {
|
||||
lines := ParseSubscriptionBody(raw)
|
||||
if len(lines) == 0 {
|
||||
err = fmt.Errorf("subscription returned no nodes")
|
||||
} else {
|
||||
err = nil
|
||||
m.subResult(name, lines, nil)
|
||||
m.log.Info("subscription refreshed", "name", name, "nodes", len(lines))
|
||||
m.Reload()
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
m.subResult(name, nil, err)
|
||||
m.log.Warn("subscription refresh failed", "name", name, "error", err)
|
||||
}
|
||||
|
||||
func (m *Manager) subResult(name string, nodes []string, err error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
st := m.subs[name]
|
||||
if st == nil {
|
||||
st = &subState{name: name}
|
||||
m.subs[name] = st
|
||||
}
|
||||
if err != nil {
|
||||
st.lastErr = err.Error()
|
||||
return
|
||||
}
|
||||
st.nodes = nodes
|
||||
st.lastErr = ""
|
||||
st.fetchedAt = time.Now()
|
||||
}
|
||||
|
||||
func (m *Manager) fetchClient() (*http.Client, error) {
|
||||
m.cfg.RLock()
|
||||
proxy := strings.TrimSpace(m.cfg.Proxy.URL)
|
||||
m.cfg.RUnlock()
|
||||
tr := &http.Transport{
|
||||
TLSHandshakeTimeout: 20 * time.Second,
|
||||
ResponseHeaderTimeout: 30 * time.Second,
|
||||
}
|
||||
if proxy != "" {
|
||||
u, err := url.Parse(proxy)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("proxy url: %w", err)
|
||||
}
|
||||
tr.Proxy = http.ProxyURL(u)
|
||||
}
|
||||
return &http.Client{Transport: tr, Timeout: 45 * time.Second}, nil
|
||||
}
|
||||
|
||||
func (m *Manager) checkAll(ctx context.Context) {
|
||||
m.mu.Lock()
|
||||
list := make([]*endpoint, 0, len(m.endpoints))
|
||||
for _, ep := range m.endpoints {
|
||||
list = append(list, ep)
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
m.cfg.RLock()
|
||||
timeout := time.Duration(m.cfg.ProxyPool.CheckTimeoutSeconds) * time.Second
|
||||
checkURL := strings.TrimSpace(m.cfg.ProxyPool.CheckURL)
|
||||
m.cfg.RUnlock()
|
||||
if timeout <= 0 {
|
||||
timeout = 10 * time.Second
|
||||
}
|
||||
if checkURL == "" {
|
||||
checkURL = "https://www.gstatic.com/generate_204"
|
||||
}
|
||||
|
||||
sem := make(chan struct{}, 4)
|
||||
var wg sync.WaitGroup
|
||||
for _, ep := range list {
|
||||
if !ep.enabled || ep.proxyURL == "" {
|
||||
continue
|
||||
}
|
||||
wg.Add(1)
|
||||
sem <- struct{}{}
|
||||
go func(ep *endpoint) {
|
||||
defer wg.Done()
|
||||
defer func() { <-sem }()
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
ok, ms, errMsg := probeURL(ctx, ep.proxyURL, checkURL, timeout)
|
||||
m.mu.Lock()
|
||||
ep.checked = true
|
||||
ep.checkedAt = time.Now()
|
||||
ep.healthy = ok
|
||||
ep.latencyMs = ms
|
||||
ep.lastErr = errMsg
|
||||
m.mu.Unlock()
|
||||
}(ep)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
m.mu.Lock()
|
||||
m.lastCheck = time.Now()
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
func probeURL(ctx context.Context, proxyURL, checkURL string, timeout time.Duration) (bool, int64, string) {
|
||||
u, err := url.Parse(proxyURL)
|
||||
if err != nil {
|
||||
return false, 0, "bad proxy url"
|
||||
}
|
||||
tr := &http.Transport{
|
||||
Proxy: http.ProxyURL(u),
|
||||
TLSHandshakeTimeout: timeout,
|
||||
ResponseHeaderTimeout: timeout,
|
||||
DisableKeepAlives: true,
|
||||
}
|
||||
client := &http.Client{Transport: tr, Timeout: timeout}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, checkURL, nil)
|
||||
if err != nil {
|
||||
return false, 0, err.Error()
|
||||
}
|
||||
start := time.Now()
|
||||
resp, err := client.Do(req)
|
||||
ms := time.Since(start).Milliseconds()
|
||||
if err != nil {
|
||||
return false, ms, trimErr(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 4096))
|
||||
if resp.StatusCode >= 400 {
|
||||
return false, ms, fmt.Sprintf("http %d", resp.StatusCode)
|
||||
}
|
||||
return true, ms, ""
|
||||
}
|
||||
|
||||
// Candidates resolves the ordered route candidates for one site: its bound
|
||||
// entries (or subscriptions) filtered to healthy ones, sticky pick first.
|
||||
// It implements upstream.RouteSource.
|
||||
func (m *Manager) Candidates(site string) []upstream.RouteCandidate {
|
||||
m.cfg.RLock()
|
||||
enabled := m.cfg.ProxyPool.Enabled
|
||||
var items []string
|
||||
if st, ok := m.cfg.Sites[site]; ok && st != nil {
|
||||
items = append(items, st.Proxies...)
|
||||
}
|
||||
m.cfg.RUnlock()
|
||||
if !enabled || len(items) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
var all []*endpoint
|
||||
for _, item := range items {
|
||||
all = append(all, m.resolveItemLocked(item)...)
|
||||
}
|
||||
usable := make([]*endpoint, 0, len(all))
|
||||
for _, ep := range all {
|
||||
if ep.enabled && ep.proxyURL != "" && ep.parseErr == "" {
|
||||
usable = append(usable, ep)
|
||||
}
|
||||
}
|
||||
pick := make([]*endpoint, 0, len(usable))
|
||||
for _, ep := range usable {
|
||||
if !ep.checked || ep.healthy {
|
||||
pick = append(pick, ep)
|
||||
}
|
||||
}
|
||||
if len(pick) == 0 {
|
||||
pick = usable
|
||||
}
|
||||
if len(pick) == 0 {
|
||||
return nil
|
||||
}
|
||||
if chosen := m.sticky[site]; chosen != "" {
|
||||
for i, ep := range pick {
|
||||
if ep.name == chosen && i > 0 {
|
||||
pick = append([]*endpoint{ep}, append(pick[:i], pick[i+1:]...)...)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
m.sticky[site] = pick[0].name
|
||||
out := make([]upstream.RouteCandidate, 0, len(pick))
|
||||
for _, ep := range pick {
|
||||
out = append(out, upstream.RouteCandidate{Name: ep.name, Proxy: ep.proxyURL})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (m *Manager) resolveItemLocked(item string) []*endpoint {
|
||||
item = strings.TrimSpace(item)
|
||||
if item == "" {
|
||||
return nil
|
||||
}
|
||||
if name, ok := strings.CutPrefix(item, "sub:"); ok {
|
||||
var out []*endpoint
|
||||
for _, ep := range m.endpoints {
|
||||
if ep.source == name {
|
||||
out = append(out, ep)
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].name < out[j].name })
|
||||
return out
|
||||
}
|
||||
if ep := m.endpoints[item]; ep != nil {
|
||||
return []*endpoint{ep}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Report records how a request through `proxy` went, so a failing endpoint
|
||||
// loses its sticky pick immediately and a healthy one is trusted again.
|
||||
func (m *Manager) Report(proxy string, ok bool) {
|
||||
if m == nil || proxy == "" {
|
||||
return
|
||||
}
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
for _, ep := range m.endpoints {
|
||||
if ep.proxyURL != proxy {
|
||||
continue
|
||||
}
|
||||
ep.checked = true
|
||||
ep.checkedAt = time.Now()
|
||||
if ok {
|
||||
ep.healthy = true
|
||||
ep.lastErr = ""
|
||||
} else {
|
||||
ep.healthy = false
|
||||
ep.lastErr = "request failed"
|
||||
for site, name := range m.sticky {
|
||||
if name == ep.name {
|
||||
delete(m.sticky, site)
|
||||
}
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// Status is the console snapshot.
|
||||
type Status struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
CheckIntervalS int `json:"check_interval_seconds"`
|
||||
CheckTimeoutS int `json:"check_timeout_seconds"`
|
||||
CheckURL string `json:"check_url"`
|
||||
DefaultScheme string `json:"default_scheme"`
|
||||
XrayPath string `json:"xray_path"`
|
||||
XrayVersion string `json:"xray_version"`
|
||||
XrayAutoDownload bool `json:"xray_auto_download"`
|
||||
Xray XrayStatus `json:"xray"`
|
||||
Entries []EntryStatus `json:"entries"`
|
||||
Subscriptions []SubStatus `json:"subscriptions"`
|
||||
Sticky map[string]string `json:"sticky"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
}
|
||||
|
||||
type EntryStatus struct {
|
||||
Name string `json:"name"`
|
||||
Kind string `json:"kind"`
|
||||
Scheme string `json:"scheme"`
|
||||
Target string `json:"target"`
|
||||
Source string `json:"source,omitempty"`
|
||||
Enabled bool `json:"enabled"`
|
||||
Healthy bool `json:"healthy"`
|
||||
Checked bool `json:"checked"`
|
||||
LatencyMs int64 `json:"latency_ms"`
|
||||
LastErr string `json:"last_error,omitempty"`
|
||||
ParseError string `json:"parse_error,omitempty"`
|
||||
CheckedAt int64 `json:"checked_at"`
|
||||
}
|
||||
|
||||
type SubStatus struct {
|
||||
Name string `json:"name"`
|
||||
URL string `json:"url"`
|
||||
Enabled bool `json:"enabled"`
|
||||
IntervalMinutes int `json:"interval_minutes"`
|
||||
Nodes int `json:"nodes"`
|
||||
FetchedAt int64 `json:"fetched_at"`
|
||||
LastErr string `json:"last_error,omitempty"`
|
||||
}
|
||||
|
||||
func (m *Manager) Status() Status {
|
||||
m.cfg.RLock()
|
||||
poolCfg := m.cfg.ProxyPool
|
||||
subsCfg := make(map[string]config.ProxySubscription, len(poolCfg.Subscriptions))
|
||||
for name, sc := range poolCfg.Subscriptions {
|
||||
if sc != nil {
|
||||
subsCfg[name] = *sc
|
||||
}
|
||||
}
|
||||
m.cfg.RUnlock()
|
||||
|
||||
m.mu.Lock()
|
||||
endpoints := make([]*endpoint, 0, len(m.endpoints))
|
||||
for _, ep := range m.endpoints {
|
||||
endpoints = append(endpoints, ep)
|
||||
}
|
||||
sticky := make(map[string]string, len(m.sticky))
|
||||
for k, v := range m.sticky {
|
||||
sticky[k] = v
|
||||
}
|
||||
subs := make(map[string]*subState, len(m.subs))
|
||||
for name, st := range m.subs {
|
||||
cp := *st
|
||||
subs[name] = &cp
|
||||
}
|
||||
lastErr := m.lastErr
|
||||
m.mu.Unlock()
|
||||
|
||||
st := Status{
|
||||
Enabled: poolCfg.Enabled,
|
||||
CheckIntervalS: poolCfg.CheckIntervalSeconds,
|
||||
CheckTimeoutS: poolCfg.CheckTimeoutSeconds,
|
||||
CheckURL: poolCfg.CheckURL,
|
||||
DefaultScheme: poolCfg.DefaultScheme,
|
||||
XrayPath: poolCfg.XrayPath,
|
||||
XrayVersion: poolCfg.XrayVersion,
|
||||
XrayAutoDownload: poolCfg.XrayAutoDownload,
|
||||
Xray: m.xray.status(),
|
||||
Sticky: sticky,
|
||||
LastError: lastErr,
|
||||
}
|
||||
for _, ep := range endpoints {
|
||||
es := EntryStatus{
|
||||
Name: ep.name, Kind: ep.kind, Scheme: ep.scheme, Target: ep.display,
|
||||
Source: ep.source, Enabled: ep.enabled, Healthy: ep.healthy,
|
||||
Checked: ep.checked, LatencyMs: ep.latencyMs, LastErr: ep.lastErr,
|
||||
ParseError: ep.parseErr,
|
||||
}
|
||||
if !ep.checkedAt.IsZero() {
|
||||
es.CheckedAt = ep.checkedAt.Unix()
|
||||
}
|
||||
st.Entries = append(st.Entries, es)
|
||||
}
|
||||
sort.Slice(st.Entries, func(i, j int) bool {
|
||||
a, b := st.Entries[i], st.Entries[j]
|
||||
if (a.Source == "") != (b.Source == "") {
|
||||
return a.Source == ""
|
||||
}
|
||||
if a.Source != b.Source {
|
||||
return a.Source < b.Source
|
||||
}
|
||||
return a.Name < b.Name
|
||||
})
|
||||
for name, sc := range subsCfg {
|
||||
ss := SubStatus{Name: name, URL: sc.URL, Enabled: sc.Enabled, IntervalMinutes: sc.IntervalMinutes}
|
||||
if stt := subs[name]; stt != nil {
|
||||
ss.Nodes = len(stt.nodes)
|
||||
ss.LastErr = stt.lastErr
|
||||
if !stt.fetchedAt.IsZero() {
|
||||
ss.FetchedAt = stt.fetchedAt.Unix()
|
||||
}
|
||||
}
|
||||
st.Subscriptions = append(st.Subscriptions, ss)
|
||||
}
|
||||
sort.Slice(st.Subscriptions, func(i, j int) bool { return st.Subscriptions[i].Name < st.Subscriptions[j].Name })
|
||||
return st
|
||||
}
|
||||
|
||||
func newEndpoint(name, link, defaultScheme, source string) *endpoint {
|
||||
ep := &endpoint{name: name, source: source, link: strings.TrimSpace(link)}
|
||||
node, err := ParseNode(ep.link, defaultScheme)
|
||||
if err != nil {
|
||||
ep.kind = "invalid"
|
||||
ep.parseErr = err.Error()
|
||||
return ep
|
||||
}
|
||||
ep.kind = node.Kind
|
||||
ep.scheme = node.Scheme
|
||||
ep.display = node.Display
|
||||
ep.outbound = node.Outbound
|
||||
ep.url = node.URL
|
||||
return ep
|
||||
}
|
||||
|
||||
func (m *Manager) setErr(err error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if err == nil {
|
||||
m.lastErr = ""
|
||||
return
|
||||
}
|
||||
m.lastErr = err.Error()
|
||||
}
|
||||
|
||||
func shortHash(s string) string {
|
||||
sum := sha256.Sum256([]byte(s))
|
||||
return hex.EncodeToString(sum[:4])
|
||||
}
|
||||
|
||||
func trimErr(err error) string {
|
||||
s := err.Error()
|
||||
if len(s) > 160 {
|
||||
return s[:160] + "..."
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
package proxypool
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/nyoungo/dsfree2api/internal/config"
|
||||
)
|
||||
|
||||
const examplePath = "../../config.example.toml"
|
||||
|
||||
func TestCandidatesStickySwitchOnFailure(t *testing.T) {
|
||||
cfg, err := config.Load(examplePath)
|
||||
if err != nil {
|
||||
t.Fatalf("load config: %v", err)
|
||||
}
|
||||
cfg.ProxyPool.Enabled = true
|
||||
cfg.ProxyPool.Entries = map[string]*config.ProxyEntry{
|
||||
"a": {Link: "http://127.0.0.1:9", Enabled: true},
|
||||
"b": {Link: "socks5://127.0.0.1:10", Enabled: true},
|
||||
}
|
||||
cfg.Sites["de"].Proxies = []string{"a", "b"}
|
||||
|
||||
m := New(cfg, t.TempDir(), nil)
|
||||
m.Reload()
|
||||
|
||||
cands := m.Candidates("de")
|
||||
if len(cands) != 2 || cands[0].Name != "a" || cands[0].Proxy != "http://127.0.0.1:9" {
|
||||
t.Fatalf("candidates = %+v", cands)
|
||||
}
|
||||
|
||||
// Both healthy: the sticky pick stays a.
|
||||
m.mu.Lock()
|
||||
for _, ep := range m.endpoints {
|
||||
ep.checked = true
|
||||
ep.healthy = true
|
||||
}
|
||||
m.mu.Unlock()
|
||||
cands = m.Candidates("de")
|
||||
if cands[0].Name != "a" {
|
||||
t.Fatalf("sticky pick changed without failure: %+v", cands)
|
||||
}
|
||||
|
||||
// a fails: the request path must immediately switch to b.
|
||||
m.Report("http://127.0.0.1:9", false)
|
||||
cands = m.Candidates("de")
|
||||
if len(cands) != 1 || cands[0].Name != "b" {
|
||||
t.Fatalf("after failure = %+v", cands)
|
||||
}
|
||||
|
||||
// Everything unhealthy: fall back to the bound order so requests still try.
|
||||
m.Report("socks5://127.0.0.1:10", false)
|
||||
cands = m.Candidates("de")
|
||||
if len(cands) != 2 || cands[0].Name != "a" {
|
||||
t.Fatalf("all-unhealthy fallback = %+v", cands)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCandidatesDisabledOrUnbound(t *testing.T) {
|
||||
cfg, err := config.Load(examplePath)
|
||||
if err != nil {
|
||||
t.Fatalf("load config: %v", err)
|
||||
}
|
||||
cfg.ProxyPool.Enabled = false
|
||||
cfg.Sites["de"].Proxies = []string{"a"}
|
||||
m := New(cfg, t.TempDir(), nil)
|
||||
m.Reload()
|
||||
if got := m.Candidates("de"); got != nil {
|
||||
t.Fatalf("disabled pool returned %+v", got)
|
||||
}
|
||||
|
||||
cfg.ProxyPool.Enabled = true
|
||||
m.Reload()
|
||||
if got := m.Candidates("de"); got != nil {
|
||||
t.Fatalf("unbound/unknown entry returned %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,517 @@
|
||||
// Package proxypool manages the egress endpoints used to reach the upstream
|
||||
// sites: Xray share links driven by a managed Xray core, plus plain
|
||||
// http/https/socks5 endpoints. Selection is sticky per site and health is
|
||||
// probed in the background.
|
||||
package proxypool
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Node is a parsed proxy node.
|
||||
type Node struct {
|
||||
Kind string // "xray" (share link) or "url" (plain endpoint)
|
||||
Scheme string // vless / vmess / trojan / ss / http / https / socks5
|
||||
Display string // host:port, for the console
|
||||
Link string // original input line
|
||||
URL string // Kind == "url": ready-to-use proxy URL
|
||||
Outbound map[string]any // Kind == "xray": Xray outbound object (tag added later)
|
||||
}
|
||||
|
||||
// ParseNode parses one line: an Xray share link (vless/vmess/trojan/ss) or a
|
||||
// plain endpoint. Accepted plain forms:
|
||||
//
|
||||
// scheme://user:pass@host:port
|
||||
// user:pass@host:port (scheme = defaultScheme)
|
||||
// host:port (scheme = defaultScheme)
|
||||
// host:port:user:pass (scheme = defaultScheme)
|
||||
func ParseNode(line, defaultScheme string) (*Node, error) {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
return nil, errors.New("empty node")
|
||||
}
|
||||
switch {
|
||||
case strings.HasPrefix(line, "vless://"):
|
||||
return parseVless(line)
|
||||
case strings.HasPrefix(line, "vmess://"):
|
||||
return parseVmess(line)
|
||||
case strings.HasPrefix(line, "trojan://"):
|
||||
return parseTrojan(line)
|
||||
case strings.HasPrefix(line, "ss://"):
|
||||
return parseShadowsocks(line)
|
||||
default:
|
||||
return parsePlainEndpoint(line, defaultScheme)
|
||||
}
|
||||
}
|
||||
|
||||
func parseVless(raw string) (*Node, error) {
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("vless: %w", err)
|
||||
}
|
||||
id := u.User.Username()
|
||||
host, port := u.Hostname(), u.Port()
|
||||
if id == "" || host == "" || port == "" {
|
||||
return nil, errors.New("vless: missing uuid / host / port")
|
||||
}
|
||||
q := u.Query()
|
||||
user := map[string]any{"id": id, "encryption": "none"}
|
||||
if flow := q.Get("flow"); flow != "" {
|
||||
user["flow"] = flow
|
||||
}
|
||||
out := map[string]any{
|
||||
"protocol": "vless",
|
||||
"settings": map[string]any{
|
||||
"vnext": []any{map[string]any{
|
||||
"address": host, "port": portInt(port), "users": []any{user},
|
||||
}},
|
||||
},
|
||||
"streamSettings": streamSettings(firstNonEmptyStr(q.Get("type"), "tcp"), q.Get("security"), q),
|
||||
}
|
||||
return &Node{Kind: "xray", Scheme: "vless", Display: net.JoinHostPort(host, port), Link: raw, Outbound: out}, nil
|
||||
}
|
||||
|
||||
func parseVmess(raw string) (*Node, error) {
|
||||
body := strings.TrimPrefix(raw, "vmess://")
|
||||
if i := strings.IndexAny(body, "#"); i >= 0 {
|
||||
body = body[:i]
|
||||
}
|
||||
dec, err := b64Decode(body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("vmess: invalid base64 payload: %w", err)
|
||||
}
|
||||
var v struct {
|
||||
Add string `json:"add"`
|
||||
Port any `json:"port"`
|
||||
ID string `json:"id"`
|
||||
Aid any `json:"aid"`
|
||||
Scy string `json:"scy"`
|
||||
Net string `json:"net"`
|
||||
Type string `json:"type"`
|
||||
Host string `json:"host"`
|
||||
Path string `json:"path"`
|
||||
TLS string `json:"tls"`
|
||||
SNI string `json:"sni"`
|
||||
FP string `json:"fp"`
|
||||
ALPN string `json:"alpn"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(dec), &v); err != nil {
|
||||
return nil, fmt.Errorf("vmess: bad json: %w", err)
|
||||
}
|
||||
port := toInt(v.Port)
|
||||
if v.Add == "" || port == 0 || v.ID == "" {
|
||||
return nil, errors.New("vmess: missing add / port / id")
|
||||
}
|
||||
security := strings.TrimSpace(v.Scy)
|
||||
if security == "" {
|
||||
security = "auto"
|
||||
}
|
||||
out := map[string]any{
|
||||
"protocol": "vmess",
|
||||
"settings": map[string]any{
|
||||
"vnext": []any{map[string]any{
|
||||
"address": v.Add, "port": port,
|
||||
"users": []any{map[string]any{"id": v.ID, "alterId": toInt(v.Aid), "security": security}},
|
||||
}},
|
||||
},
|
||||
}
|
||||
network := strings.ToLower(strings.TrimSpace(v.Net))
|
||||
if network == "" {
|
||||
network = "tcp"
|
||||
}
|
||||
ss := map[string]any{"network": network}
|
||||
switch network {
|
||||
case "ws":
|
||||
ws := map[string]any{}
|
||||
if v.Path != "" {
|
||||
ws["path"] = v.Path
|
||||
}
|
||||
if v.Host != "" {
|
||||
ws["headers"] = map[string]any{"Host": v.Host}
|
||||
}
|
||||
ss["wsSettings"] = ws
|
||||
case "h2":
|
||||
h2 := map[string]any{}
|
||||
if v.Path != "" {
|
||||
h2["path"] = v.Path
|
||||
}
|
||||
if v.Host != "" {
|
||||
h2["host"] = []any{v.Host}
|
||||
}
|
||||
ss["h2Settings"] = h2
|
||||
case "grpc":
|
||||
g := map[string]any{}
|
||||
if v.Path != "" {
|
||||
g["serviceName"] = v.Path
|
||||
}
|
||||
ss["grpcSettings"] = g
|
||||
case "tcp":
|
||||
if strings.EqualFold(v.Type, "http") {
|
||||
req := map[string]any{"path": []any{firstNonEmptyStr(v.Path, "/")}}
|
||||
if v.Host != "" {
|
||||
req["headers"] = map[string]any{"Host": []any{v.Host}}
|
||||
}
|
||||
ss["tcpSettings"] = map[string]any{"header": map[string]any{"type": "http", "request": req}}
|
||||
}
|
||||
case "httpupgrade", "xhttp":
|
||||
sub := map[string]any{}
|
||||
if v.Path != "" {
|
||||
sub["path"] = v.Path
|
||||
}
|
||||
if v.Host != "" {
|
||||
sub["host"] = v.Host
|
||||
}
|
||||
ss[network+"Settings"] = sub
|
||||
}
|
||||
if strings.EqualFold(v.TLS, "tls") {
|
||||
ss["security"] = "tls"
|
||||
t := map[string]any{}
|
||||
if sni := firstNonEmptyStr(v.SNI, v.Host); sni != "" {
|
||||
t["serverName"] = sni
|
||||
}
|
||||
if v.FP != "" {
|
||||
t["fingerprint"] = v.FP
|
||||
}
|
||||
if v.ALPN != "" {
|
||||
t["alpn"] = strings.Split(v.ALPN, ",")
|
||||
}
|
||||
ss["tlsSettings"] = t
|
||||
}
|
||||
out["streamSettings"] = ss
|
||||
return &Node{Kind: "xray", Scheme: "vmess", Display: net.JoinHostPort(v.Add, strconv.Itoa(port)), Link: raw, Outbound: out}, nil
|
||||
}
|
||||
|
||||
func parseTrojan(raw string) (*Node, error) {
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trojan: %w", err)
|
||||
}
|
||||
pass := u.User.Username()
|
||||
host, port := u.Hostname(), u.Port()
|
||||
if pass == "" || host == "" || port == "" {
|
||||
return nil, errors.New("trojan: missing password / host / port")
|
||||
}
|
||||
q := u.Query()
|
||||
security := q.Get("security")
|
||||
if security == "" {
|
||||
security = "tls"
|
||||
}
|
||||
out := map[string]any{
|
||||
"protocol": "trojan",
|
||||
"settings": map[string]any{
|
||||
"servers": []any{map[string]any{"address": host, "port": portInt(port), "password": pass}},
|
||||
},
|
||||
"streamSettings": streamSettings(firstNonEmptyStr(q.Get("type"), "tcp"), security, q),
|
||||
}
|
||||
return &Node{Kind: "xray", Scheme: "trojan", Display: net.JoinHostPort(host, port), Link: raw, Outbound: out}, nil
|
||||
}
|
||||
|
||||
func parseShadowsocks(raw string) (*Node, error) {
|
||||
body := strings.TrimPrefix(raw, "ss://")
|
||||
if i := strings.Index(body, "#"); i >= 0 {
|
||||
body = body[:i]
|
||||
}
|
||||
var method, pass, hostport string
|
||||
if at := strings.LastIndex(body, "@"); at >= 0 {
|
||||
userinfo, hp := body[:at], body[at+1:]
|
||||
if i := strings.IndexAny(hp, "?"); i >= 0 {
|
||||
hp = hp[:i]
|
||||
}
|
||||
if strings.Contains(userinfo, ":") {
|
||||
// plain method:password (uncommon)
|
||||
} else {
|
||||
dec, err := b64Decode(userinfo)
|
||||
if err != nil {
|
||||
return nil, errors.New("ss: cannot decode credentials")
|
||||
}
|
||||
userinfo = dec
|
||||
}
|
||||
mp := strings.SplitN(userinfo, ":", 2)
|
||||
if len(mp) != 2 || mp[0] == "" || mp[1] == "" {
|
||||
return nil, errors.New("ss: bad credentials")
|
||||
}
|
||||
method, pass, hostport = mp[0], mp[1], hp
|
||||
} else {
|
||||
// legacy: ss://base64(method:password@host:port)
|
||||
dec, err := b64Decode(body)
|
||||
if err != nil {
|
||||
return nil, errors.New("ss: invalid base64 payload")
|
||||
}
|
||||
at := strings.LastIndex(dec, "@")
|
||||
if at < 0 {
|
||||
return nil, errors.New("ss: missing @host")
|
||||
}
|
||||
mp := strings.SplitN(dec[:at], ":", 2)
|
||||
if len(mp) != 2 {
|
||||
return nil, errors.New("ss: bad credentials")
|
||||
}
|
||||
method, pass, hostport = mp[0], mp[1], dec[at+1:]
|
||||
}
|
||||
host, port, err := splitHostPort(hostport)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("ss: %w", err)
|
||||
}
|
||||
out := map[string]any{
|
||||
"protocol": "shadowsocks",
|
||||
"settings": map[string]any{
|
||||
"servers": []any{map[string]any{"address": host, "port": port, "method": method, "password": pass}},
|
||||
},
|
||||
}
|
||||
return &Node{Kind: "xray", Scheme: "ss", Display: net.JoinHostPort(host, strconv.Itoa(port)), Link: raw, Outbound: out}, nil
|
||||
}
|
||||
|
||||
func parsePlainEndpoint(line, defaultScheme string) (*Node, error) {
|
||||
if defaultScheme = strings.ToLower(strings.TrimSpace(defaultScheme)); defaultScheme == "" {
|
||||
defaultScheme = "http"
|
||||
}
|
||||
build := func(scheme, host, port, user, pass string) (*Node, error) {
|
||||
scheme = normalizeScheme(scheme)
|
||||
if scheme == "" {
|
||||
return nil, fmt.Errorf("unsupported proxy scheme %q (http/https/socks5)", scheme)
|
||||
}
|
||||
if host == "" || port == "" {
|
||||
return nil, errors.New("missing host or port")
|
||||
}
|
||||
u := &url.URL{Scheme: scheme, Host: net.JoinHostPort(host, port)}
|
||||
if user != "" {
|
||||
u.User = url.UserPassword(user, pass)
|
||||
}
|
||||
return &Node{
|
||||
Kind: "url", Scheme: scheme,
|
||||
Display: net.JoinHostPort(host, port),
|
||||
Link: line, URL: u.String(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
switch {
|
||||
case strings.Contains(line, "://"):
|
||||
u, err := url.Parse(line)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("bad proxy url: %w", err)
|
||||
}
|
||||
user, pass := "", ""
|
||||
if u.User != nil {
|
||||
user = u.User.Username()
|
||||
pass, _ = u.User.Password()
|
||||
}
|
||||
return build(u.Scheme, u.Hostname(), u.Port(), user, pass)
|
||||
case strings.Contains(line, "@"):
|
||||
at := strings.LastIndex(line, "@")
|
||||
userinfo, hostport := line[:at], line[at+1:]
|
||||
mp := strings.SplitN(userinfo, ":", 2)
|
||||
user, pass := mp[0], ""
|
||||
if len(mp) == 2 {
|
||||
pass = mp[1]
|
||||
}
|
||||
host, port, err := splitHostPort(hostport)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return build(defaultScheme, host, strconv.Itoa(port), user, pass)
|
||||
default:
|
||||
parts := strings.Split(line, ":")
|
||||
switch len(parts) {
|
||||
case 2:
|
||||
return build(defaultScheme, parts[0], parts[1], "", "")
|
||||
case 4:
|
||||
// ip:port:user:pass
|
||||
return build(defaultScheme, parts[0], parts[1], parts[2], parts[3])
|
||||
default:
|
||||
return nil, fmt.Errorf("unrecognized endpoint %q", line)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ParseSubscriptionBody turns a subscription response into node lines. The
|
||||
// body may be plain text (one node per line) or a base64 blob of that text.
|
||||
func ParseSubscriptionBody(raw []byte) []string {
|
||||
s := strings.TrimSpace(string(raw))
|
||||
if s == "" {
|
||||
return nil
|
||||
}
|
||||
if !strings.Contains(s, "://") {
|
||||
if dec, err := b64Decode(strings.ReplaceAll(s, "\n", "")); err == nil {
|
||||
s = dec
|
||||
}
|
||||
}
|
||||
out := make([]string, 0, 16)
|
||||
for _, line := range strings.Split(s, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
out = append(out, line)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ── helpers ──────────────────────────────────────────────────────
|
||||
|
||||
// streamSettings builds Xray streamSettings from share-link query params.
|
||||
func streamSettings(network, security string, q url.Values) map[string]any {
|
||||
network = strings.ToLower(strings.TrimSpace(network))
|
||||
if network == "" {
|
||||
network = "tcp"
|
||||
}
|
||||
ss := map[string]any{"network": network}
|
||||
switch network {
|
||||
case "ws":
|
||||
ws := map[string]any{}
|
||||
if p := q.Get("path"); p != "" {
|
||||
ws["path"] = p
|
||||
}
|
||||
if h := q.Get("host"); h != "" {
|
||||
ws["headers"] = map[string]any{"Host": h}
|
||||
}
|
||||
ss["wsSettings"] = ws
|
||||
case "grpc":
|
||||
g := map[string]any{}
|
||||
if s := q.Get("serviceName"); s != "" {
|
||||
g["serviceName"] = s
|
||||
}
|
||||
ss["grpcSettings"] = g
|
||||
case "httpupgrade":
|
||||
h := map[string]any{}
|
||||
if p := q.Get("path"); p != "" {
|
||||
h["path"] = p
|
||||
}
|
||||
if hh := q.Get("host"); hh != "" {
|
||||
h["host"] = hh
|
||||
}
|
||||
ss["httpupgradeSettings"] = h
|
||||
case "xhttp":
|
||||
x := map[string]any{}
|
||||
if p := q.Get("path"); p != "" {
|
||||
x["path"] = p
|
||||
}
|
||||
if hh := q.Get("host"); hh != "" {
|
||||
x["host"] = hh
|
||||
}
|
||||
if m := q.Get("mode"); m != "" {
|
||||
x["mode"] = m
|
||||
}
|
||||
ss["xhttpSettings"] = x
|
||||
case "tcp":
|
||||
if strings.EqualFold(q.Get("headerType"), "http") {
|
||||
req := map[string]any{}
|
||||
if h := q.Get("host"); h != "" {
|
||||
req["headers"] = map[string]any{"Host": []any{h}}
|
||||
}
|
||||
if p := q.Get("path"); p != "" {
|
||||
req["path"] = []any{p}
|
||||
}
|
||||
ss["tcpSettings"] = map[string]any{"header": map[string]any{"type": "http", "request": req}}
|
||||
}
|
||||
}
|
||||
switch strings.ToLower(security) {
|
||||
case "tls":
|
||||
t := map[string]any{}
|
||||
if sni := firstNonEmptyStr(q.Get("sni"), q.Get("host")); sni != "" {
|
||||
t["serverName"] = sni
|
||||
}
|
||||
if fp := q.Get("fp"); fp != "" {
|
||||
t["fingerprint"] = fp
|
||||
}
|
||||
if alpn := q.Get("alpn"); alpn != "" {
|
||||
t["alpn"] = strings.Split(alpn, ",")
|
||||
}
|
||||
ss["security"] = "tls"
|
||||
ss["tlsSettings"] = t
|
||||
case "reality":
|
||||
r := map[string]any{}
|
||||
if sni := q.Get("sni"); sni != "" {
|
||||
r["serverName"] = sni
|
||||
}
|
||||
if fp := q.Get("fp"); fp != "" {
|
||||
r["fingerprint"] = fp
|
||||
}
|
||||
if pbk := q.Get("pbk"); pbk != "" {
|
||||
r["publicKey"] = pbk
|
||||
}
|
||||
if sid := q.Get("sid"); sid != "" {
|
||||
r["shortId"] = sid
|
||||
}
|
||||
if spx := q.Get("spx"); spx != "" {
|
||||
r["spiderX"] = spx
|
||||
}
|
||||
ss["security"] = "reality"
|
||||
ss["realitySettings"] = r
|
||||
}
|
||||
return ss
|
||||
}
|
||||
|
||||
func normalizeScheme(s string) string {
|
||||
switch strings.ToLower(strings.TrimSpace(s)) {
|
||||
case "http":
|
||||
return "http"
|
||||
case "https":
|
||||
return "https"
|
||||
case "socks", "socks5", "socks5h":
|
||||
return "socks5"
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
func b64Decode(s string) (string, error) {
|
||||
s = strings.TrimSpace(s)
|
||||
trimmed := strings.TrimRight(s, "=")
|
||||
for _, enc := range []*base64.Encoding{
|
||||
base64.RawURLEncoding, base64.URLEncoding,
|
||||
base64.RawStdEncoding, base64.StdEncoding,
|
||||
} {
|
||||
if b, err := enc.DecodeString(trimmed); err == nil {
|
||||
return string(b), nil
|
||||
}
|
||||
}
|
||||
return "", errors.New("invalid base64")
|
||||
}
|
||||
|
||||
func firstNonEmptyStr(vals ...string) string {
|
||||
for _, v := range vals {
|
||||
if strings.TrimSpace(v) != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func portInt(s string) int {
|
||||
n, _ := strconv.Atoi(strings.TrimSpace(s))
|
||||
return n
|
||||
}
|
||||
|
||||
func toInt(v any) int {
|
||||
switch x := v.(type) {
|
||||
case float64:
|
||||
return int(x)
|
||||
case int:
|
||||
return x
|
||||
case int64:
|
||||
return int(x)
|
||||
case string:
|
||||
n, _ := strconv.Atoi(strings.TrimSpace(x))
|
||||
return n
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
func splitHostPort(hostport string) (string, int, error) {
|
||||
host, port, err := net.SplitHostPort(strings.TrimSpace(hostport))
|
||||
if err != nil {
|
||||
return "", 0, fmt.Errorf("bad host:port %q", hostport)
|
||||
}
|
||||
n, err := strconv.Atoi(port)
|
||||
if err != nil || n < 1 || n > 65535 {
|
||||
return "", 0, fmt.Errorf("bad port in %q", hostport)
|
||||
}
|
||||
return host, n, nil
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
package proxypool
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestParseVless(t *testing.T) {
|
||||
line := "vless://11111111-2222-3333-4444-555555555555@example.com:443" +
|
||||
"?type=ws&security=tls&sni=sni.example.com&path=%2Fws&host=cdn.example.com&fp=chrome#node"
|
||||
n, err := ParseNode(line, "http")
|
||||
if err != nil {
|
||||
t.Fatalf("parse: %v", err)
|
||||
}
|
||||
if n.Kind != "xray" || n.Scheme != "vless" || n.Display != "example.com:443" {
|
||||
t.Fatalf("node = %+v", n)
|
||||
}
|
||||
if n.Outbound["protocol"] != "vless" {
|
||||
t.Fatalf("protocol = %v", n.Outbound["protocol"])
|
||||
}
|
||||
settings := n.Outbound["settings"].(map[string]any)
|
||||
vnext := settings["vnext"].([]any)[0].(map[string]any)
|
||||
if vnext["address"] != "example.com" || vnext["port"] != 443 {
|
||||
t.Fatalf("vnext = %+v", vnext)
|
||||
}
|
||||
users := vnext["users"].([]any)[0].(map[string]any)
|
||||
if users["id"] != "11111111-2222-3333-4444-555555555555" || users["encryption"] != "none" {
|
||||
t.Fatalf("users = %+v", users)
|
||||
}
|
||||
ss := n.Outbound["streamSettings"].(map[string]any)
|
||||
if ss["network"] != "ws" || ss["security"] != "tls" {
|
||||
t.Fatalf("stream = %+v", ss)
|
||||
}
|
||||
ws := ss["wsSettings"].(map[string]any)
|
||||
if ws["path"] != "/ws" {
|
||||
t.Fatalf("ws = %+v", ws)
|
||||
}
|
||||
tls := ss["tlsSettings"].(map[string]any)
|
||||
if tls["serverName"] != "sni.example.com" {
|
||||
t.Fatalf("tls = %+v", tls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseVmess(t *testing.T) {
|
||||
payload := `{"v":"2","ps":"n","add":"vm.example.com","port":"443","id":"uuid-1",` +
|
||||
`"aid":"0","scy":"auto","net":"ws","type":"none","host":"vm.example.com",` +
|
||||
`"path":"/vm","tls":"tls","sni":"vm.example.com"}`
|
||||
line := "vmess://" + base64.StdEncoding.EncodeToString([]byte(payload))
|
||||
n, err := ParseNode(line, "")
|
||||
if err != nil {
|
||||
t.Fatalf("parse: %v", err)
|
||||
}
|
||||
if n.Kind != "xray" || n.Scheme != "vmess" || n.Display != "vm.example.com:443" {
|
||||
t.Fatalf("node = %+v", n)
|
||||
}
|
||||
ss := n.Outbound["streamSettings"].(map[string]any)
|
||||
if ss["network"] != "ws" || ss["security"] != "tls" {
|
||||
t.Fatalf("stream = %+v", ss)
|
||||
}
|
||||
if ss["wsSettings"].(map[string]any)["path"] != "/vm" {
|
||||
t.Fatalf("ws = %+v", ss["wsSettings"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseTrojan(t *testing.T) {
|
||||
n, err := ParseNode("trojan://secret@tr.example.com:443?security=tls&sni=tr.example.com#t", "")
|
||||
if err != nil {
|
||||
t.Fatalf("parse: %v", err)
|
||||
}
|
||||
if n.Scheme != "trojan" || n.Display != "tr.example.com:443" {
|
||||
t.Fatalf("node = %+v", n)
|
||||
}
|
||||
settings := n.Outbound["settings"].(map[string]any)
|
||||
srv := settings["servers"].([]any)[0].(map[string]any)
|
||||
if srv["address"] != "tr.example.com" || srv["port"] != 443 || srv["password"] != "secret" {
|
||||
t.Fatalf("server = %+v", srv)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseShadowsocks(t *testing.T) {
|
||||
// SIP002: base64(method:password)@host:port
|
||||
ui := base64.RawURLEncoding.EncodeToString([]byte("aes-256-gcm:pass:word"))
|
||||
n, err := ParseNode("ss://"+ui+"@ss.example.com:8388#x", "")
|
||||
if err != nil {
|
||||
t.Fatalf("sip002 parse: %v", err)
|
||||
}
|
||||
srv := n.Outbound["settings"].(map[string]any)["servers"].([]any)[0].(map[string]any)
|
||||
if srv["method"] != "aes-256-gcm" || srv["password"] != "pass:word" || srv["address"] != "ss.example.com" {
|
||||
t.Fatalf("server = %+v", srv)
|
||||
}
|
||||
|
||||
// legacy: base64(method:password@host:port)
|
||||
legacy := base64.RawURLEncoding.EncodeToString([]byte("aes-256-gcm:secret@ss.example.com:8388"))
|
||||
n2, err := ParseNode("ss://"+legacy, "")
|
||||
if err != nil {
|
||||
t.Fatalf("legacy parse: %v", err)
|
||||
}
|
||||
srv2 := n2.Outbound["settings"].(map[string]any)["servers"].([]any)[0].(map[string]any)
|
||||
if srv2["password"] != "secret" || srv2["address"] != "ss.example.com" || srv2["port"] != 8388 {
|
||||
t.Fatalf("legacy server = %+v", srv2)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParsePlainEndpoints(t *testing.T) {
|
||||
cases := []struct {
|
||||
line, def, wantURL, wantDisplay string
|
||||
}{
|
||||
{"socks5://u:p@1.2.3.4:1080", "http", "socks5://u:p@1.2.3.4:1080", "1.2.3.4:1080"},
|
||||
{"http://1.2.3.4:8080", "socks5", "http://1.2.3.4:8080", "1.2.3.4:8080"},
|
||||
{"1.2.3.4:8080", "http", "http://1.2.3.4:8080", "1.2.3.4:8080"},
|
||||
{"1.2.3.4:8080", "socks5", "socks5://1.2.3.4:8080", "1.2.3.4:8080"},
|
||||
{"u:p@1.2.3.4:1080", "http", "http://u:p@1.2.3.4:1080", "1.2.3.4:1080"},
|
||||
{"1.2.3.4:8080:u:p", "http", "http://u:p@1.2.3.4:8080", "1.2.3.4:8080"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
n, err := ParseNode(c.line, c.def)
|
||||
if err != nil {
|
||||
t.Fatalf("%q: %v", c.line, err)
|
||||
}
|
||||
if n.Kind != "url" || n.URL != c.wantURL || n.Display != c.wantDisplay {
|
||||
t.Errorf("%q => kind=%s url=%q display=%q", c.line, n.Kind, n.URL, n.Display)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseSubscriptionBody(t *testing.T) {
|
||||
plain := "vless://a@b:1\n# comment\nsocks5://x:y@1.2.3.4:5\n\n"
|
||||
got := ParseSubscriptionBody([]byte(plain))
|
||||
if len(got) != 2 || got[0] != "vless://a@b:1" || got[1] != "socks5://x:y@1.2.3.4:5" {
|
||||
t.Fatalf("plain parse = %v", got)
|
||||
}
|
||||
enc := base64.StdEncoding.EncodeToString([]byte(plain))
|
||||
got2 := ParseSubscriptionBody([]byte(enc))
|
||||
if len(got2) != 2 || got2[0] != got[0] {
|
||||
t.Fatalf("base64 parse = %v", got2)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseNodeRejectsJunk(t *testing.T) {
|
||||
for _, bad := range []string{"", "hello world", "vless://no-port", "ss://@@@", "1.2.3.4", "ftp://1.2.3.4:21"} {
|
||||
if _, err := ParseNode(bad, "http"); err == nil {
|
||||
t.Errorf("%q should fail to parse", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,498 @@
|
||||
package proxypool
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// xrayNode is one Xray-backed endpoint, ready to be written into the core
|
||||
// config.
|
||||
type xrayNode struct {
|
||||
Name string
|
||||
Outbound map[string]any
|
||||
}
|
||||
|
||||
// xrayCore owns the managed Xray process: one socks inbound per pool node,
|
||||
// each routed to its own outbound.
|
||||
type xrayCore struct {
|
||||
dir string
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
path string
|
||||
proc *exec.Cmd
|
||||
exited chan struct{}
|
||||
running bool
|
||||
pid int
|
||||
cfgPath string
|
||||
cfgHash string
|
||||
ports map[string]int
|
||||
lastErr string
|
||||
tail *logTail
|
||||
}
|
||||
|
||||
// XrayStatus is the console view of the core.
|
||||
type XrayStatus struct {
|
||||
Path string `json:"path"`
|
||||
Running bool `json:"running"`
|
||||
PID int `json:"pid"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
func newXrayCore(dir string, log *slog.Logger) *xrayCore {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
return &xrayCore{dir: dir, log: log, ports: map[string]int{}}
|
||||
}
|
||||
|
||||
func (c *xrayCore) status() XrayStatus {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return XrayStatus{Path: c.path, Running: c.running, PID: c.pid, Error: c.lastErr}
|
||||
}
|
||||
|
||||
// ensure resolves the Xray binary: explicit path, then a previously downloaded
|
||||
// copy, then (optionally) a fresh download. The download runs unlocked so
|
||||
// status reads never stall behind a slow network.
|
||||
func (c *xrayCore) ensure(explicitPath string, autoDownload bool, version string) error {
|
||||
c.mu.Lock()
|
||||
if c.path != "" {
|
||||
if _, err := os.Stat(c.path); err == nil {
|
||||
c.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
c.path = ""
|
||||
}
|
||||
if p := strings.TrimSpace(explicitPath); p != "" {
|
||||
if _, err := os.Stat(p); err != nil {
|
||||
c.mu.Unlock()
|
||||
return fmt.Errorf("xray_path: %w", err)
|
||||
}
|
||||
c.path = p
|
||||
c.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
target := filepath.Join(c.dir, "xray"+exeSuffix())
|
||||
if _, err := os.Stat(target); err == nil {
|
||||
c.path = target
|
||||
c.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
if !autoDownload {
|
||||
c.mu.Unlock()
|
||||
return errors.New("xray binary not found — set [proxypool].xray_path, or enable xray_auto_download")
|
||||
}
|
||||
c.mu.Unlock()
|
||||
|
||||
if err := c.download(target, version); err != nil {
|
||||
c.mu.Lock()
|
||||
c.lastErr = err.Error()
|
||||
c.mu.Unlock()
|
||||
return err
|
||||
}
|
||||
c.mu.Lock()
|
||||
c.path = target
|
||||
c.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *xrayCore) download(target, version string) error {
|
||||
if err := os.MkdirAll(c.dir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
osName := map[string]string{"windows": "windows", "linux": "linux", "darwin": "macos"}[runtime.GOOS]
|
||||
arch := map[string]string{"amd64": "64", "arm64": "arm64-v8a", "386": "32", "arm": "arm32-v7a"}[runtime.GOARCH]
|
||||
if osName == "" || arch == "" {
|
||||
return fmt.Errorf("xray: unsupported platform %s/%s", runtime.GOOS, runtime.GOARCH)
|
||||
}
|
||||
asset := fmt.Sprintf("Xray-%s-%s.zip", osName, arch)
|
||||
downloadURL := "https://github.com/XTLS/Xray-core/releases/latest/download/" + asset
|
||||
if v := strings.TrimSpace(version); v != "" {
|
||||
if !strings.HasPrefix(v, "v") {
|
||||
v = "v" + v
|
||||
}
|
||||
downloadURL = fmt.Sprintf("https://github.com/XTLS/Xray-core/releases/download/%s/%s", v, asset)
|
||||
}
|
||||
c.log.Info("downloading xray core", "asset", asset)
|
||||
zipPath := target + ".download"
|
||||
if err := httpDownload(downloadURL, zipPath, 180*time.Second); err != nil {
|
||||
return err
|
||||
}
|
||||
defer os.Remove(zipPath)
|
||||
if want, err := fetchDigest(downloadURL + ".dgst"); err != nil {
|
||||
c.log.Warn("xray checksum unavailable, skipping verification", "error", err)
|
||||
} else if want != "" {
|
||||
got, err := fileSHA256(zipPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.EqualFold(got, want) {
|
||||
return fmt.Errorf("xray checksum mismatch: got %s want %s", got, want)
|
||||
}
|
||||
}
|
||||
if err := extractZipBinary(zipPath, target); err != nil {
|
||||
return err
|
||||
}
|
||||
_ = os.Chmod(target, 0o755)
|
||||
c.log.Info("xray core ready", "path", target)
|
||||
return nil
|
||||
}
|
||||
|
||||
// configure regenerates the core config for the given nodes and restarts the
|
||||
// process when something changed. Ports are kept stable across restarts so
|
||||
// cached cookies stay valid.
|
||||
func (c *xrayCore) configure(nodes []xrayNode) error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if len(nodes) == 0 {
|
||||
c.stopLocked()
|
||||
c.cfgHash = ""
|
||||
return nil
|
||||
}
|
||||
if c.path == "" {
|
||||
return errors.New("xray binary unavailable")
|
||||
}
|
||||
for _, n := range nodes {
|
||||
if c.ports[n.Name] == 0 {
|
||||
p, err := freePort()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
c.ports[n.Name] = p
|
||||
}
|
||||
}
|
||||
if c.configHash(nodes) == c.cfgHash && c.running {
|
||||
return nil
|
||||
}
|
||||
// A restart is due. Stop first so our own listeners do not count as
|
||||
// "port busy", then reclaim or reassign every assigned port — another
|
||||
// local core may have raced us for one, since the OS only guaranteed it
|
||||
// at allocation time.
|
||||
c.stopLocked()
|
||||
for _, n := range nodes {
|
||||
if !portAvailable(c.ports[n.Name]) {
|
||||
p, err := freePort()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
c.ports[n.Name] = p
|
||||
}
|
||||
}
|
||||
raw, err := json.Marshal(buildXrayConfig(nodes, c.ports))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sum := sha256.Sum256(raw)
|
||||
c.cfgHash = hex.EncodeToString(sum[:])
|
||||
cfgPath := filepath.Join(c.dir, "config.json")
|
||||
if err := os.MkdirAll(c.dir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(cfgPath, raw, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
c.cfgPath = cfgPath
|
||||
return c.startLocked(nodes)
|
||||
}
|
||||
|
||||
// configHash is the digest of the config that would be generated for nodes
|
||||
// with the current port assignments.
|
||||
func (c *xrayCore) configHash(nodes []xrayNode) string {
|
||||
raw, err := json.Marshal(buildXrayConfig(nodes, c.ports))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
sum := sha256.Sum256(raw)
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// portAvailable reports whether 127.0.0.1:port can still be claimed.
|
||||
func portAvailable(port int) bool {
|
||||
if port <= 0 {
|
||||
return false
|
||||
}
|
||||
l, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", port))
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
l.Close()
|
||||
return true
|
||||
}
|
||||
|
||||
func (c *xrayCore) startLocked(nodes []xrayNode) error {
|
||||
cmd := exec.Command(c.path, "run", "-c", c.cfgPath)
|
||||
c.tail = &logTail{max: 40}
|
||||
cmd.Stdout = c.tail
|
||||
cmd.Stderr = c.tail
|
||||
if err := cmd.Start(); err != nil {
|
||||
c.lastErr = "start xray: " + err.Error()
|
||||
return errors.New(c.lastErr)
|
||||
}
|
||||
c.proc = cmd
|
||||
c.pid = cmd.Process.Pid
|
||||
c.running = true
|
||||
c.lastErr = ""
|
||||
exited := make(chan struct{})
|
||||
c.exited = exited
|
||||
go func() {
|
||||
_ = cmd.Wait()
|
||||
c.mu.Lock()
|
||||
if c.proc == cmd {
|
||||
c.running = false
|
||||
c.pid = 0
|
||||
if t := c.tail.Last(); t != "" {
|
||||
c.lastErr = "xray exited: " + t
|
||||
} else {
|
||||
c.lastErr = "xray exited"
|
||||
}
|
||||
}
|
||||
c.mu.Unlock()
|
||||
close(exited)
|
||||
}()
|
||||
// Wait until the local socks inbounds answer.
|
||||
deadline := time.Now().Add(8 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
ok := true
|
||||
for _, n := range nodes {
|
||||
conn, err := net.DialTimeout("tcp", fmt.Sprintf("127.0.0.1:%d", c.ports[n.Name]), 300*time.Millisecond)
|
||||
if err != nil {
|
||||
ok = false
|
||||
break
|
||||
}
|
||||
conn.Close()
|
||||
}
|
||||
if ok {
|
||||
return nil
|
||||
}
|
||||
if !c.running {
|
||||
break
|
||||
}
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
}
|
||||
if !c.running {
|
||||
if c.lastErr == "" {
|
||||
c.lastErr = "xray failed to start"
|
||||
}
|
||||
return errors.New(c.lastErr)
|
||||
}
|
||||
c.lastErr = "xray socks ports did not come up in time"
|
||||
return errors.New(c.lastErr)
|
||||
}
|
||||
|
||||
func (c *xrayCore) stopLocked() {
|
||||
if c.proc != nil && c.proc.Process != nil {
|
||||
_ = c.proc.Process.Kill()
|
||||
}
|
||||
if c.exited != nil {
|
||||
select {
|
||||
case <-c.exited:
|
||||
case <-time.After(3 * time.Second):
|
||||
}
|
||||
}
|
||||
c.proc = nil
|
||||
c.exited = nil
|
||||
c.running = false
|
||||
c.pid = 0
|
||||
}
|
||||
|
||||
func (c *xrayCore) stop() {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.stopLocked()
|
||||
}
|
||||
|
||||
// port returns the local socks port assigned to a node name.
|
||||
func (c *xrayCore) port(name string) int {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.ports[name]
|
||||
}
|
||||
|
||||
func buildXrayConfig(nodes []xrayNode, ports map[string]int) map[string]any {
|
||||
inbounds := make([]any, 0, len(nodes))
|
||||
outbounds := make([]any, 0, len(nodes)+2)
|
||||
rules := make([]any, 0, len(nodes))
|
||||
for i, n := range nodes {
|
||||
inTag := fmt.Sprintf("in-%d", i)
|
||||
outTag := fmt.Sprintf("out-%d", i)
|
||||
inbounds = append(inbounds, map[string]any{
|
||||
"tag": inTag, "listen": "127.0.0.1", "port": ports[n.Name],
|
||||
"protocol": "socks",
|
||||
"settings": map[string]any{"udp": false, "auth": "noauth"},
|
||||
})
|
||||
ob := make(map[string]any, len(n.Outbound)+1)
|
||||
for k, v := range n.Outbound {
|
||||
ob[k] = v
|
||||
}
|
||||
ob["tag"] = outTag
|
||||
outbounds = append(outbounds, ob)
|
||||
rules = append(rules, map[string]any{"type": "field", "inboundTag": []any{inTag}, "outboundTag": outTag})
|
||||
}
|
||||
outbounds = append(outbounds,
|
||||
map[string]any{"tag": "direct", "protocol": "freedom"},
|
||||
map[string]any{"tag": "blocked", "protocol": "blackhole"},
|
||||
)
|
||||
return map[string]any{
|
||||
"log": map[string]any{"loglevel": "warning", "access": "none"},
|
||||
"inbounds": inbounds,
|
||||
"outbounds": outbounds,
|
||||
"routing": map[string]any{"domainStrategy": "AsIs", "rules": rules},
|
||||
}
|
||||
}
|
||||
|
||||
// ── platform / io helpers ────────────────────────────────────────
|
||||
|
||||
func exeSuffix() string {
|
||||
if runtime.GOOS == "windows" {
|
||||
return ".exe"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func freePort() (int, error) {
|
||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer l.Close()
|
||||
return l.Addr().(*net.TCPAddr).Port, nil
|
||||
}
|
||||
|
||||
func httpDownload(url, dst string, timeout time.Duration) error {
|
||||
client := &http.Client{Timeout: timeout}
|
||||
resp, err := client.Get(url)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("download %s: http %d", url, resp.StatusCode)
|
||||
}
|
||||
f, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
if _, err := io.Copy(f, io.LimitReader(resp.Body, 256<<20)); err != nil {
|
||||
return err
|
||||
}
|
||||
return f.Sync()
|
||||
}
|
||||
|
||||
func fetchDigest(url string) (string, error) {
|
||||
client := &http.Client{Timeout: 20 * time.Second}
|
||||
resp, err := client.Get(url)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return "", fmt.Errorf("http %d", resp.StatusCode)
|
||||
}
|
||||
raw, err := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, line := range strings.Split(string(raw), "\n") {
|
||||
key, val, ok := strings.Cut(line, ":")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if strings.Contains(strings.ToUpper(key), "SHA2-256") || strings.Contains(strings.ToUpper(key), "SHA256") {
|
||||
if v := strings.TrimSpace(strings.Fields(val)[0]); v != "" {
|
||||
return v, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
return "", errors.New("no sha256 digest found")
|
||||
}
|
||||
|
||||
func fileSHA256(path string) (string, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer f.Close()
|
||||
h := sha256.New()
|
||||
if _, err := io.Copy(h, f); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil)), nil
|
||||
}
|
||||
|
||||
func extractZipBinary(zipPath, target string) error {
|
||||
r, err := zip.OpenReader(zipPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open archive: %w", err)
|
||||
}
|
||||
defer r.Close()
|
||||
want := "xray" + exeSuffix()
|
||||
for _, f := range r.File {
|
||||
if !strings.EqualFold(filepath.Base(f.Name), want) {
|
||||
continue
|
||||
}
|
||||
rc, err := f.Open()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer rc.Close()
|
||||
out, err := os.OpenFile(target, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o755)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := io.Copy(out, rc); err != nil {
|
||||
out.Close()
|
||||
return err
|
||||
}
|
||||
return out.Close()
|
||||
}
|
||||
return errors.New("no xray binary inside the archive")
|
||||
}
|
||||
|
||||
// logTail keeps the last few process log lines for status reporting.
|
||||
type logTail struct {
|
||||
mu sync.Mutex
|
||||
lines []string
|
||||
max int
|
||||
}
|
||||
|
||||
func (t *logTail) Write(p []byte) (int, error) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
for _, line := range strings.Split(strings.TrimRight(string(p), "\r\n"), "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
t.lines = append(t.lines, line)
|
||||
}
|
||||
if len(t.lines) > t.max {
|
||||
t.lines = t.lines[len(t.lines)-t.max:]
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func (t *logTail) Last() string {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
if len(t.lines) == 0 {
|
||||
return ""
|
||||
}
|
||||
return t.lines[len(t.lines)-1]
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
package proxypool
|
||||
|
||||
import (
|
||||
"net"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/nyoungo/dsfree2api/internal/config"
|
||||
)
|
||||
|
||||
// TestXrayCoreSmoke drives the managed core against the locally placed
|
||||
// binary (data/xray.exe) with a syntactically real VLESS node. It never
|
||||
// connects upstream — it only proves that ensure() picks up the local
|
||||
// binary, configure() starts the process and the local socks inbound is
|
||||
// live. Skipped unless opted in:
|
||||
//
|
||||
// $env:DSFREE_XRAY_SMOKE=1; go test ./internal/proxypool -run Smoke -v
|
||||
func TestXrayCoreSmoke(t *testing.T) {
|
||||
if os.Getenv("DSFREE_XRAY_SMOKE") == "" {
|
||||
t.Skip("set DSFREE_XRAY_SMOKE=1 to run the local xray smoke test")
|
||||
}
|
||||
cfg, err := config.Load(examplePath)
|
||||
if err != nil {
|
||||
t.Fatalf("load config: %v", err)
|
||||
}
|
||||
cfg.ProxyPool.Enabled = true
|
||||
cfg.ProxyPool.XrayAutoDownload = false // the local binary must be used as-is
|
||||
cfg.ProxyPool.Entries = map[string]*config.ProxyEntry{
|
||||
"smoke": {
|
||||
Link: "vless://11111111-2222-3333-4444-555555555555@example.com:443" +
|
||||
"?security=tls&type=tcp&sni=example.com",
|
||||
Enabled: true,
|
||||
},
|
||||
}
|
||||
cfg.Sites["de"].Proxies = []string{"smoke"}
|
||||
|
||||
m := New(cfg, "../../data", nil)
|
||||
m.Reload()
|
||||
st := m.Status()
|
||||
if !st.Xray.Running {
|
||||
t.Fatalf("xray core did not start: %+v", st.Xray)
|
||||
}
|
||||
t.Logf("xray running: pid=%d path=%s", st.Xray.PID, st.Xray.Path)
|
||||
|
||||
cands := m.Candidates("de")
|
||||
if len(cands) != 1 || !strings.HasPrefix(cands[0].Proxy, "socks5://127.0.0.1:") {
|
||||
t.Fatalf("candidates = %+v", cands)
|
||||
}
|
||||
t.Logf("candidate proxy: %s", cands[0].Proxy)
|
||||
|
||||
m.xray.stop()
|
||||
if m.Status().Xray.Running {
|
||||
t.Fatal("xray still running after stop")
|
||||
}
|
||||
|
||||
// Port stability: a restart re-claims the same socks port, so cached
|
||||
// cookies (keyed by the local proxy URL) stay valid.
|
||||
m.Reload()
|
||||
again := m.Candidates("de")
|
||||
if len(again) != 1 || again[0].Proxy != cands[0].Proxy {
|
||||
t.Fatalf("port not stable across restart: %q -> %+v", cands[0].Proxy, again)
|
||||
}
|
||||
m.xray.stop()
|
||||
|
||||
// Race self-heal: steal the assigned port; the next start must move away.
|
||||
port := strings.TrimPrefix(cands[0].Proxy, "socks5://127.0.0.1:")
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:"+port)
|
||||
if err != nil {
|
||||
t.Fatalf("occupy port %s: %v", port, err)
|
||||
}
|
||||
defer ln.Close()
|
||||
m.Reload()
|
||||
third := m.Candidates("de")
|
||||
if len(third) != 1 || third[0].Proxy == cands[0].Proxy {
|
||||
t.Fatalf("stolen port was not reassigned: %+v", third)
|
||||
}
|
||||
t.Logf("port moved %s -> %s", cands[0].Proxy, third[0].Proxy)
|
||||
m.xray.stop()
|
||||
}
|
||||
@@ -0,0 +1,357 @@
|
||||
// Package quota watches the upstream sites' guest-token balances so the
|
||||
// console and logs show how much free daily quota is left before a site cuts
|
||||
// requests off. Readings reuse the pooled cookies and egress of the chat
|
||||
// path, so they describe the identity requests are actually debited from.
|
||||
package quota
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/nyoungo/dsfree2api/internal/config"
|
||||
"github.com/nyoungo/dsfree2api/internal/httpx"
|
||||
"github.com/nyoungo/dsfree2api/internal/proxypool"
|
||||
"github.com/nyoungo/dsfree2api/internal/turnstile"
|
||||
)
|
||||
|
||||
// Balance is one site × bot quota reading. Empty Error means the reading was
|
||||
// taken successfully; Limit/Used/Remaining come from the site's own balance
|
||||
// endpoint (the guest daily free allowance).
|
||||
type Balance struct {
|
||||
Site string `json:"site"`
|
||||
BotID int `json:"bot_id"`
|
||||
Model string `json:"model,omitempty"`
|
||||
Route string `json:"route"`
|
||||
PaidBalance int64 `json:"paid_balance"`
|
||||
Limit int64 `json:"limit"`
|
||||
Used int64 `json:"used"`
|
||||
Remaining int64 `json:"remaining"`
|
||||
Ratio float64 `json:"ratio"`
|
||||
ResetPeriod string `json:"reset_period,omitempty"`
|
||||
CheckedAt int64 `json:"checked_at"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// Snapshot is the console view of the sentinel.
|
||||
type Snapshot struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Running bool `json:"running"`
|
||||
CheckS int `json:"check_seconds"`
|
||||
WarnRatio float64 `json:"warn_ratio"`
|
||||
Entries []Balance `json:"entries"`
|
||||
}
|
||||
|
||||
type botRef struct {
|
||||
id int
|
||||
model string
|
||||
}
|
||||
|
||||
// Watcher polls each site's balance REST endpoint on an interval. One slot
|
||||
// per site × bot; slots without verified cookies are reported as such and
|
||||
// never trigger solves by themselves.
|
||||
type Watcher struct {
|
||||
cfg *config.Config
|
||||
log *slog.Logger
|
||||
ts *turnstile.Solver
|
||||
pool *proxypool.Manager
|
||||
|
||||
// newSession is a test seam; nil uses the real Chrome-impersonating one.
|
||||
newSession func(proxy string) (httpx.Session, error)
|
||||
|
||||
mu sync.Mutex
|
||||
entries map[string]*Balance // key: site|bot
|
||||
warned map[string]bool
|
||||
lastErr map[string]string
|
||||
running bool
|
||||
}
|
||||
|
||||
// New builds a watcher. Start launches the loop when [quota].enabled.
|
||||
func New(cfg *config.Config, log *slog.Logger, ts *turnstile.Solver, pool *proxypool.Manager) *Watcher {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
return &Watcher{
|
||||
cfg: cfg,
|
||||
log: log,
|
||||
ts: ts,
|
||||
pool: pool,
|
||||
entries: map[string]*Balance{},
|
||||
warned: map[string]bool{},
|
||||
lastErr: map[string]string{},
|
||||
}
|
||||
}
|
||||
|
||||
// Start launches the polling loop; no-op unless [quota].enabled.
|
||||
func (w *Watcher) Start(ctx context.Context) {
|
||||
w.cfg.RLock()
|
||||
enabled := w.cfg.Quota.Enabled
|
||||
w.cfg.RUnlock()
|
||||
if !enabled || w.ts == nil {
|
||||
return
|
||||
}
|
||||
go w.loop(ctx)
|
||||
}
|
||||
|
||||
func (w *Watcher) loop(ctx context.Context) {
|
||||
w.mu.Lock()
|
||||
w.running = true
|
||||
w.mu.Unlock()
|
||||
defer func() {
|
||||
w.mu.Lock()
|
||||
w.running = false
|
||||
w.mu.Unlock()
|
||||
}()
|
||||
// First sweep after a short settle delay (pool/cookies may still be
|
||||
// warming up), then on every check interval.
|
||||
timer := time.NewTimer(20 * time.Second)
|
||||
defer timer.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-timer.C:
|
||||
w.Sweep(ctx)
|
||||
w.cfg.RLock()
|
||||
iv := time.Duration(w.cfg.Quota.CheckSecondsValue()) * time.Second
|
||||
w.cfg.RUnlock()
|
||||
timer.Reset(iv)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Sweep queries every enabled site once. Exposed for tests and manual runs.
|
||||
func (w *Watcher) Sweep(ctx context.Context) {
|
||||
if w.ts == nil {
|
||||
return
|
||||
}
|
||||
w.cfg.RLock()
|
||||
sites := map[string]config.Site{}
|
||||
for code, st := range w.cfg.Sites {
|
||||
if st != nil && st.Enabled {
|
||||
sites[code] = *st
|
||||
}
|
||||
}
|
||||
bots := map[string][]botRef{}
|
||||
for id, m := range w.cfg.Models {
|
||||
if m == nil || !m.Enabled || m.BotID <= 0 {
|
||||
continue
|
||||
}
|
||||
bots[m.Site] = append(bots[m.Site], botRef{id: m.BotID, model: id})
|
||||
}
|
||||
warnRatio := w.cfg.Quota.WarnRatioValue()
|
||||
w.cfg.RUnlock()
|
||||
|
||||
codes := make([]string, 0, len(sites))
|
||||
for code := range sites {
|
||||
codes = append(codes, code)
|
||||
}
|
||||
sort.Strings(codes)
|
||||
for _, code := range codes {
|
||||
refs := bots[code]
|
||||
if len(refs) == 0 {
|
||||
continue
|
||||
}
|
||||
sort.Slice(refs, func(i, j int) bool { return refs[i].id < refs[j].id })
|
||||
w.checkSite(ctx, sites[code], refs, warnRatio)
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Watcher) checkSite(ctx context.Context, site config.Site, refs []botRef, warnRatio float64) {
|
||||
route := w.routeFor(site.Code)
|
||||
if !w.ts.CookiesReady(site.Code, route) {
|
||||
w.markAll(site.Code, refs, route, errors.New("no verified cookies for this route yet"))
|
||||
return
|
||||
}
|
||||
sess, err := w.session(route)
|
||||
if err != nil {
|
||||
w.markAll(site.Code, refs, route, err)
|
||||
return
|
||||
}
|
||||
defer sess.Close()
|
||||
if err := w.ts.ApplyValidCookies(ctx, sess, &site, route); err != nil {
|
||||
w.markAll(site.Code, refs, route, err)
|
||||
return
|
||||
}
|
||||
// Guests need no REST nonce: the visitor id travels as the dsgt_gid
|
||||
// cookie and the balance endpoint keys on it.
|
||||
if gid := sess.Cookies()["dsgt_gid"]; gid == "" {
|
||||
w.markAll(site.Code, refs, route, errors.New("no dsgt_gid cookie for this route yet"))
|
||||
return
|
||||
}
|
||||
endpoint := strings.TrimRight(site.BaseURL, "/") + "/wp-json/dsgt/v1/balance"
|
||||
for _, ref := range refs {
|
||||
payload, err := fetchBalance(ctx, sess, endpoint, ref.id)
|
||||
w.record(site.Code, ref, route, payload, err, warnRatio)
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Watcher) markAll(site string, refs []botRef, route string, err error) {
|
||||
for _, ref := range refs {
|
||||
w.record(site, ref, route, nil, err, 0)
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Watcher) record(site string, ref botRef, route string, payload *balancePayload, err error, warnRatio float64) {
|
||||
key := site + "|" + strconv.Itoa(ref.id)
|
||||
b := &Balance{
|
||||
Site: site, BotID: ref.id, Model: ref.model,
|
||||
Route: routeLabel(route), CheckedAt: time.Now().Unix(),
|
||||
}
|
||||
if err != nil {
|
||||
b.Error = err.Error()
|
||||
} else {
|
||||
b.PaidBalance = payload.Balance
|
||||
b.Limit = payload.Free.Limit
|
||||
b.Used = payload.Free.Used
|
||||
b.Remaining = payload.Free.Remaining
|
||||
b.ResetPeriod = payload.Free.ResetPeriod
|
||||
if b.Limit > 0 {
|
||||
b.Ratio = float64(b.Remaining) / float64(b.Limit)
|
||||
}
|
||||
}
|
||||
|
||||
var lowNow, recovered, resetSeen bool
|
||||
var prevUsed int64
|
||||
w.mu.Lock()
|
||||
if prev := w.entries[key]; prev != nil && prev.Error == "" && err == nil && b.Used < prev.Used {
|
||||
resetSeen = true
|
||||
prevUsed = prev.Used
|
||||
}
|
||||
w.entries[key] = b
|
||||
prevErr := w.lastErr[key]
|
||||
w.lastErr[key] = b.Error
|
||||
if err == nil && b.Limit > 0 {
|
||||
low := float64(b.Remaining) <= warnRatio*float64(b.Limit)
|
||||
if low && !w.warned[key] {
|
||||
w.warned[key] = true
|
||||
lowNow = true
|
||||
}
|
||||
if !low && w.warned[key] {
|
||||
w.warned[key] = false
|
||||
recovered = true
|
||||
}
|
||||
}
|
||||
w.mu.Unlock()
|
||||
|
||||
if err != nil {
|
||||
if b.Error != prevErr {
|
||||
w.log.Warn("quota check failed",
|
||||
"site", site, "bot_id", ref.id, "model", ref.model, "route", b.Route, "error", b.Error)
|
||||
}
|
||||
return
|
||||
}
|
||||
if prevErr != "" {
|
||||
w.log.Info("quota check recovered", "site", site, "bot_id", ref.id, "route", b.Route)
|
||||
}
|
||||
w.log.Debug("quota check",
|
||||
"site", site, "bot_id", ref.id, "model", ref.model, "route", b.Route,
|
||||
"remaining", b.Remaining, "limit", b.Limit, "used", b.Used)
|
||||
if resetSeen {
|
||||
w.log.Info("quota reset observed",
|
||||
"site", site, "bot_id", ref.id, "model", ref.model, "route", b.Route,
|
||||
"used_before", prevUsed, "used_after", b.Used,
|
||||
"checked_at", time.Unix(b.CheckedAt, 0).Format(time.RFC3339))
|
||||
}
|
||||
switch {
|
||||
case lowNow:
|
||||
w.log.Warn("quota low",
|
||||
"site", site, "bot_id", ref.id, "model", ref.model, "route", b.Route,
|
||||
"remaining", b.Remaining, "limit", b.Limit, "reset_period", b.ResetPeriod)
|
||||
case recovered:
|
||||
w.log.Info("quota recovered",
|
||||
"site", site, "bot_id", ref.id, "model", ref.model,
|
||||
"remaining", b.Remaining, "limit", b.Limit)
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Watcher) routeFor(site string) string {
|
||||
if w.pool != nil {
|
||||
if cands := w.pool.Candidates(site); len(cands) > 0 {
|
||||
return cands[0].Proxy
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (w *Watcher) session(proxy string) (httpx.Session, error) {
|
||||
if w.newSession != nil {
|
||||
return w.newSession(proxy)
|
||||
}
|
||||
return httpx.NewSession(httpx.Options{ProxyURL: proxy, Timeout: 45 * time.Second})
|
||||
}
|
||||
|
||||
func routeLabel(proxy string) string {
|
||||
if proxy == "" {
|
||||
return "direct"
|
||||
}
|
||||
return proxy
|
||||
}
|
||||
|
||||
// freeQuota is the daily guest allowance block of a balance response.
|
||||
type freeQuota struct {
|
||||
Limit int64 `json:"limit"`
|
||||
Used int64 `json:"used"`
|
||||
Remaining int64 `json:"remaining"`
|
||||
ResetPeriod string `json:"reset_period"`
|
||||
}
|
||||
|
||||
type balancePayload struct {
|
||||
Balance int64 `json:"balance"`
|
||||
Free *freeQuota `json:"free"`
|
||||
}
|
||||
|
||||
func fetchBalance(ctx context.Context, sess httpx.Session, endpoint string, botID int) (*balancePayload, error) {
|
||||
u := fmt.Sprintf("%s?bot_id=%d", endpoint, botID)
|
||||
resp, err := sess.Do(ctx, httpx.Request{
|
||||
Method: http.MethodGet,
|
||||
URL: u,
|
||||
Header: map[string]string{"Accept": "application/json"},
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("balance request: %w", err)
|
||||
}
|
||||
if resp.Status != http.StatusOK {
|
||||
return nil, fmt.Errorf("balance request: HTTP %d", resp.Status)
|
||||
}
|
||||
var p balancePayload
|
||||
if err := json.Unmarshal(resp.Body, &p); err != nil {
|
||||
return nil, fmt.Errorf("balance json: %w", err)
|
||||
}
|
||||
if p.Free == nil {
|
||||
return nil, errors.New("balance response has no free quota block")
|
||||
}
|
||||
return &p, nil
|
||||
}
|
||||
|
||||
// Snapshot returns the current readings for the console.
|
||||
func (w *Watcher) Snapshot() Snapshot {
|
||||
w.cfg.RLock()
|
||||
enabled := w.cfg.Quota.Enabled
|
||||
checkS := w.cfg.Quota.CheckSecondsValue()
|
||||
ratio := w.cfg.Quota.WarnRatioValue()
|
||||
w.cfg.RUnlock()
|
||||
|
||||
w.mu.Lock()
|
||||
entries := make([]Balance, 0, len(w.entries))
|
||||
for _, b := range w.entries {
|
||||
entries = append(entries, *b)
|
||||
}
|
||||
running := w.running
|
||||
w.mu.Unlock()
|
||||
sort.Slice(entries, func(i, j int) bool {
|
||||
if entries[i].Site != entries[j].Site {
|
||||
return entries[i].Site < entries[j].Site
|
||||
}
|
||||
return entries[i].BotID < entries[j].BotID
|
||||
})
|
||||
return Snapshot{Enabled: enabled, Running: running, CheckS: checkS, WarnRatio: ratio, Entries: entries}
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
package quota
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/nyoungo/dsfree2api/internal/config"
|
||||
"github.com/nyoungo/dsfree2api/internal/httpx"
|
||||
"github.com/nyoungo/dsfree2api/internal/turnstile"
|
||||
)
|
||||
|
||||
const examplePath = "../../config.example.toml"
|
||||
|
||||
func TestQuotaResetIsRecorded(t *testing.T) {
|
||||
cfg, err := config.Load(examplePath)
|
||||
if err != nil {
|
||||
t.Fatalf("load config: %v", err)
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
w := New(cfg, slog.New(slog.NewTextHandler(&buf, nil)), nil, nil)
|
||||
ref := botRef{id: 7, model: "m"}
|
||||
w.record("de", ref, "", &balancePayload{Free: &freeQuota{Limit: 30000, Used: 500, Remaining: 29500, ResetPeriod: "daily"}}, nil, 0.2)
|
||||
w.record("de", ref, "", &balancePayload{Free: &freeQuota{Limit: 30000, Used: 0, Remaining: 30000, ResetPeriod: "daily"}}, nil, 0.2)
|
||||
if !strings.Contains(buf.String(), "quota reset observed") {
|
||||
t.Fatalf("expected a reset log line, got:\n%s", buf.String())
|
||||
}
|
||||
if !strings.Contains(buf.String(), "used_before=500") {
|
||||
t.Fatalf("expected used_before=500 in log, got:\n%s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestSweepReadsBalancesWithPooledCookies(t *testing.T) {
|
||||
var seenGid string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if !strings.HasPrefix(r.URL.Path, "/wp-json/dsgt/v1/balance") {
|
||||
t.Errorf("unexpected request %s — the sentinel must not fetch the site page", r.URL.Path)
|
||||
return
|
||||
}
|
||||
if c, err := r.Cookie("dsgt_gid"); err == nil {
|
||||
seenGid = c.Value
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"balance":0,"free":{"limit":30000,"used":500,"remaining":29500,"reset_period":"daily"}}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
cfg, err := config.Load(examplePath)
|
||||
if err != nil {
|
||||
t.Fatalf("load config: %v", err)
|
||||
}
|
||||
cfg.Quota.Enabled = true
|
||||
cfg.Quota.CheckSeconds = 300
|
||||
cfg.Quota.WarnRatio = 0.2
|
||||
site := *cfg.Sites["de"]
|
||||
site.BaseURL = srv.URL
|
||||
cfg.Sites["de"] = &site
|
||||
|
||||
ts := turnstile.New(cfg.Turnstile, "")
|
||||
if _, _, err := ts.ImportCookies("de", "dsgt_gid=test-gid; cf_clearance=abc", ""); err != nil {
|
||||
t.Fatalf("import cookies: %v", err)
|
||||
}
|
||||
|
||||
w := New(cfg, nil, ts, nil)
|
||||
w.newSession = func(string) (httpx.Session, error) {
|
||||
return httpx.NewSession(httpx.Options{Timeout: 5 * time.Second})
|
||||
}
|
||||
w.Sweep(context.Background())
|
||||
|
||||
if seenGid != "test-gid" {
|
||||
t.Fatalf("balance call visitor cookie = %q, want test-gid", seenGid)
|
||||
}
|
||||
snap := w.Snapshot()
|
||||
if !snap.Enabled {
|
||||
t.Error("snapshot should report enabled")
|
||||
}
|
||||
if len(snap.Entries) != 6 {
|
||||
t.Fatalf("entries = %d, want 6 (3 sites × 2 bots)", len(snap.Entries))
|
||||
}
|
||||
var de *Balance
|
||||
for i := range snap.Entries {
|
||||
if snap.Entries[i].Site == "de" && snap.Entries[i].BotID == 27487 {
|
||||
de = &snap.Entries[i]
|
||||
}
|
||||
}
|
||||
if de == nil {
|
||||
t.Fatalf("no de/27487 entry in %+v", snap.Entries)
|
||||
}
|
||||
if de.Error != "" || de.Limit != 30000 || de.Remaining != 29500 || de.Used != 500 || de.ResetPeriod != "daily" {
|
||||
t.Fatalf("de balance = %+v", de)
|
||||
}
|
||||
if de.Route != "direct" {
|
||||
t.Errorf("route = %q, want direct", de.Route)
|
||||
}
|
||||
for _, e := range snap.Entries {
|
||||
if e.Site != "de" && e.Error == "" {
|
||||
t.Errorf("site %s should report missing cookies, got %+v", e.Site, e)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -467,10 +467,7 @@ const uaMetadataExpr = `(() => {
|
||||
// warms the page like a human, renders the widget and clicks until a token
|
||||
// lands or the deadline passes.
|
||||
func (s *Solver) pollForToken(ctx context.Context, conn *cdp, sid, sitekey string, cfg config.Turnstile) (string, error) {
|
||||
timeout := time.Duration(cfg.TimeoutSeconds) * time.Second
|
||||
if timeout <= 0 {
|
||||
timeout = 90 * time.Second
|
||||
}
|
||||
timeout := time.Duration(cfg.SolveTimeoutValue()) * time.Second
|
||||
deadline := time.Now().Add(timeout)
|
||||
|
||||
var preDone, inited bool
|
||||
@@ -941,6 +938,10 @@ func launchBrowser(ctx context.Context, exe string, cfg config.Turnstile, locale
|
||||
"--user-data-dir=" + profile,
|
||||
"--no-first-run",
|
||||
"--no-default-browser-check",
|
||||
// Chromium/Edge re-launch themselves de-elevated when the parent runs
|
||||
// as Administrator, orphaning the real browser and breaking CDP;
|
||||
// keep the browser inside our process tree instead.
|
||||
"--do-not-de-elevate",
|
||||
"--disable-blink-features=AutomationControlled",
|
||||
"--disable-background-networking",
|
||||
"--disable-sync",
|
||||
@@ -992,13 +993,12 @@ func (b *browserProc) waitDevTools(ctx context.Context) error {
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
var lastErr error
|
||||
for time.Now().Before(deadline) {
|
||||
select {
|
||||
case <-b.exited:
|
||||
return fmt.Errorf("browser exited early (%v): %s", b.waitErr, b.stderr.String())
|
||||
case <-ctx.Done():
|
||||
if ctx.Err() != nil {
|
||||
return ctx.Err()
|
||||
default:
|
||||
}
|
||||
// A Windows browser launcher may hand off to a longer-lived process
|
||||
// and exit 0; that is not a failure while the DevTools endpoint is
|
||||
// still coming up, so the exit check happens only at timeout.
|
||||
resp, err := client.Get(endpoint)
|
||||
if err == nil {
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
@@ -1018,10 +1018,17 @@ func (b *browserProc) waitDevTools(ctx context.Context) error {
|
||||
return ctx.Err()
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-b.exited:
|
||||
return fmt.Errorf("browser exited early (%v): %s", b.waitErr, b.stderr.String())
|
||||
default:
|
||||
}
|
||||
return fmt.Errorf("devtools never came up: %v; stderr: %s", lastErr, b.stderr.String())
|
||||
}
|
||||
|
||||
// Close kills the browser if it is still alive and drops the temp profile.
|
||||
// On Windows the spawned launcher may have exited while the real browser
|
||||
// lives on, so a DevTools close is attempted before giving up.
|
||||
func (b *browserProc) Close() {
|
||||
select {
|
||||
case <-b.exited:
|
||||
@@ -1034,6 +1041,7 @@ func (b *browserProc) Close() {
|
||||
case <-time.After(3 * time.Second):
|
||||
}
|
||||
}
|
||||
b.closeViaDevTools(2 * time.Second)
|
||||
if b.tempDir != "" {
|
||||
for i := 0; i < 3; i++ {
|
||||
if err := os.RemoveAll(b.tempDir); err == nil {
|
||||
@@ -1044,6 +1052,39 @@ func (b *browserProc) Close() {
|
||||
}
|
||||
}
|
||||
|
||||
// closeViaDevTools asks the browser to shut down over the DevTools endpoint.
|
||||
// It is the only handle left when the original launcher process has exited.
|
||||
func (b *browserProc) closeViaDevTools(timeout time.Duration) {
|
||||
ws := b.wsURL
|
||||
if ws == "" {
|
||||
client := &http.Client{Timeout: timeout}
|
||||
resp, err := client.Get(fmt.Sprintf("http://127.0.0.1:%d/json/version", b.port))
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return
|
||||
}
|
||||
var info struct {
|
||||
WebSocketDebuggerURL string `json:"webSocketDebuggerUrl"`
|
||||
}
|
||||
if json.Unmarshal(body, &info) != nil || info.WebSocketDebuggerURL == "" {
|
||||
return
|
||||
}
|
||||
ws = info.WebSocketDebuggerURL
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), timeout)
|
||||
defer cancel()
|
||||
conn, err := dialCDP(ctx, ws, fmt.Sprintf("http://127.0.0.1:%d", b.port))
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer conn.Close()
|
||||
_, _ = conn.call(ctx, "", "Browser.close", nil)
|
||||
}
|
||||
|
||||
func freePort() (int, error) {
|
||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
|
||||
@@ -10,7 +10,9 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -91,6 +93,11 @@ type Solver struct {
|
||||
// browserMu serializes browser solves: one Chrome window at a time.
|
||||
browserMu sync.Mutex
|
||||
|
||||
// cookie pool: background warm refresh, one slot per site × route.
|
||||
poolMu sync.Mutex
|
||||
poolOn bool
|
||||
warms map[string]*warmState
|
||||
|
||||
// refreshOverride is a test seam; nil in production.
|
||||
refreshOverride func(ctx context.Context, sess httpx.Session, site *config.Site, c *core) error
|
||||
}
|
||||
@@ -107,7 +114,7 @@ func New(cfg config.Turnstile, userAgent string) *Solver {
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Solver) Enabled() bool { return s.cfg.Enabled }
|
||||
func (s *Solver) Enabled() bool { return s.Config().Enabled }
|
||||
|
||||
func (s *Solver) Config() config.Turnstile {
|
||||
s.mu.Lock()
|
||||
@@ -133,7 +140,7 @@ func (s *Solver) coreFor(proxy string) *core {
|
||||
defer s.mu.Unlock()
|
||||
c, ok := s.cores[key]
|
||||
if !ok {
|
||||
client, _ := httpx.StdClient(proxy, time.Duration(s.cfg.TimeoutSeconds+30)*time.Second)
|
||||
client, _ := httpx.StdClient(proxy, time.Duration(s.cfg.SolveTimeoutValue()+30)*time.Second)
|
||||
c = &core{
|
||||
proxy: key,
|
||||
client: client,
|
||||
@@ -287,9 +294,17 @@ func (s *Solver) ApplyValidCookies(ctx context.Context, sess httpx.Session, site
|
||||
return s.refreshLocked(ctx, sess, site, c)
|
||||
}
|
||||
|
||||
// CookiesReady reports whether an unexpired cached cookie set exists for the
|
||||
// site × route pair. Upstream uses it to keep the slow-start timer from
|
||||
// killing a first-time Turnstile solve, which can legitimately take minutes.
|
||||
func (s *Solver) CookiesReady(siteCode, proxy string) bool {
|
||||
c := s.coreFor(proxy)
|
||||
return c.load(siteCode) != nil
|
||||
}
|
||||
|
||||
// Refresh forces a fresh solve → verify cycle.
|
||||
func (s *Solver) Refresh(ctx context.Context, sess httpx.Session, site *config.Site, proxy string) error {
|
||||
if !s.cfg.Enabled {
|
||||
if !s.Enabled() {
|
||||
return nil
|
||||
}
|
||||
c := s.coreFor(proxy)
|
||||
@@ -385,32 +400,70 @@ func (s *Solver) solveToken(ctx context.Context, site *config.Site, sitekey stri
|
||||
|
||||
func (s *Solver) solveTokenAPI(ctx context.Context, baseURL, sitekey string, c *core) (string, error) {
|
||||
cfg := s.Config()
|
||||
payload, _ := json.Marshal(map[string]any{
|
||||
"url": baseURL,
|
||||
"sitekey": sitekey,
|
||||
"action": cfg.Action,
|
||||
"cdata": "",
|
||||
"timeoutSeconds": cfg.TimeoutSeconds,
|
||||
})
|
||||
style := cfg.APIStyleValue()
|
||||
var payload []byte
|
||||
if style == "ezsolver" {
|
||||
payload, _ = json.Marshal(map[string]any{
|
||||
"sitekey": sitekey,
|
||||
"siteurl": baseURL,
|
||||
"timeout": cfg.SolveTimeoutValue(),
|
||||
})
|
||||
} else {
|
||||
payload, _ = json.Marshal(map[string]any{
|
||||
"url": baseURL,
|
||||
"sitekey": sitekey,
|
||||
"action": cfg.Action,
|
||||
"cdata": "",
|
||||
"timeoutSeconds": cfg.SolveTimeoutValue(),
|
||||
})
|
||||
}
|
||||
var last error
|
||||
// A loopback solve service must be reached directly: pushing it through
|
||||
// the route proxy would resolve 127.0.0.1 on the far side of the tunnel.
|
||||
client := c.client
|
||||
if isLocalHost(cfg.APIURL) {
|
||||
client = &http.Client{Timeout: time.Duration(cfg.SolveTimeoutValue()+30) * time.Second}
|
||||
}
|
||||
for attempt := 0; attempt < cfg.Retries; attempt++ {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, cfg.APIURL, bytes.NewReader(payload))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", "Bearer "+cfg.APIKey)
|
||||
resp, err := c.client.Do(req)
|
||||
if cfg.APIKey != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+cfg.APIKey)
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
last = &Error{Msg: "turnstile solver request failed: " + err.Error(), Retryable: true}
|
||||
} else {
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode >= 400 {
|
||||
retry := retryableStatus(resp.StatusCode)
|
||||
if style == "ezsolver" {
|
||||
// Local solve services answer 5xx for per-attempt
|
||||
// timeouts; another attempt is reasonable.
|
||||
retry = true
|
||||
}
|
||||
last = &Error{
|
||||
Msg: fmt.Sprintf("turnstile solver http %d: %s (check [turnstile].api_key / api_url)",
|
||||
resp.StatusCode, trim(body)),
|
||||
Retryable: retryableStatus(resp.StatusCode),
|
||||
Retryable: retry,
|
||||
}
|
||||
} else if style == "ezsolver" {
|
||||
var out struct {
|
||||
Token string `json:"token"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &out); err != nil {
|
||||
last = &Error{Msg: "turnstile solver bad json: " + err.Error()}
|
||||
} else if out.Error != "" {
|
||||
last = &Error{Msg: "turnstile solver error: " + trim([]byte(out.Error))}
|
||||
} else if out.Token == "" {
|
||||
last = &Error{Msg: "turnstile solver returned no token"}
|
||||
} else {
|
||||
return out.Token, nil
|
||||
}
|
||||
} else {
|
||||
var out struct {
|
||||
@@ -444,6 +497,25 @@ func (s *Solver) solveTokenAPI(ctx context.Context, baseURL, sitekey string, c *
|
||||
return "", last
|
||||
}
|
||||
|
||||
// isLocalHost reports whether rawURL points at the local machine or a
|
||||
// private-network address. Such solve services must be reached directly
|
||||
// instead of through the route proxy.
|
||||
func isLocalHost(rawURL string) bool {
|
||||
u, err := url.Parse(rawURL)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
host := strings.ToLower(u.Hostname())
|
||||
if host == "localhost" {
|
||||
return true
|
||||
}
|
||||
ip := net.ParseIP(host)
|
||||
if ip == nil {
|
||||
return false
|
||||
}
|
||||
return ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast() || ip.IsUnspecified()
|
||||
}
|
||||
|
||||
// retryableStatus decides which solver HTTP statuses are worth another attempt.
|
||||
// Solver-side rejections (bad key, exhausted quota, challenge rejected) return
|
||||
// the same answer every time, so they are not retried.
|
||||
|
||||
@@ -2,7 +2,11 @@ package turnstile
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -239,3 +243,43 @@ func TestRotateGuestSwapsVisitorID(t *testing.T) {
|
||||
t.Errorf("rotating an unknown site returned %q, want empty", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSolveTokenAPIEzSolverStyle(t *testing.T) {
|
||||
var gotBody map[string]any
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if err := json.NewDecoder(r.Body).Decode(&gotBody); err != nil {
|
||||
t.Errorf("decode request: %v", err)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"token":"tok-123","elapsed":1.5}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
cfg := config.Turnstile{
|
||||
Enabled: true, Provider: config.ProviderAPI, APIStyle: "ezsolver",
|
||||
APIURL: srv.URL, TimeoutSeconds: 30, Retries: 1,
|
||||
}
|
||||
solver := New(cfg, httpx.DefaultUserAgent)
|
||||
// A broken route-proxied client proves the loopback bypass: the call must
|
||||
// still succeed because local solve services are reached directly.
|
||||
brokenURL, _ := url.Parse("socks5://127.0.0.1:1")
|
||||
c := &core{
|
||||
proxy: "direct",
|
||||
client: &http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(brokenURL)}},
|
||||
cache: map[string]*cookieState{}, locks: map[string]*sync.Mutex{},
|
||||
}
|
||||
|
||||
token, err := solver.solveTokenAPI(context.Background(), "https://deepseek.de/", "0xabc", c)
|
||||
if err != nil {
|
||||
t.Fatalf("solve: %v", err)
|
||||
}
|
||||
if token != "tok-123" {
|
||||
t.Fatalf("token = %q", token)
|
||||
}
|
||||
if gotBody["sitekey"] != "0xabc" || gotBody["siteurl"] != "https://deepseek.de/" {
|
||||
t.Fatalf("request body = %v", gotBody)
|
||||
}
|
||||
if gotBody["timeout"] != float64(30) {
|
||||
t.Fatalf("timeout = %v", gotBody["timeout"])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,278 @@
|
||||
package turnstile
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/nyoungo/dsfree2api/internal/config"
|
||||
"github.com/nyoungo/dsfree2api/internal/httpx"
|
||||
)
|
||||
|
||||
// WarmTarget is one cookie-pool slot: a site reached through one proxy route.
|
||||
// Verified cookies are bound to the egress IP that solved them, so the pool
|
||||
// keeps one slot per site × route.
|
||||
type WarmTarget struct {
|
||||
Site config.Site
|
||||
Proxy string // "" means direct
|
||||
}
|
||||
|
||||
// WarmEntry describes the pool state of one site × route slot.
|
||||
type WarmEntry struct {
|
||||
Site string `json:"site"`
|
||||
Route string `json:"route"`
|
||||
Valid bool `json:"valid"`
|
||||
Remaining int64 `json:"remaining_s"`
|
||||
Warming bool `json:"warming"`
|
||||
LastAt int64 `json:"last_at"`
|
||||
LastMs int64 `json:"last_ms"`
|
||||
LastErr string `json:"last_error,omitempty"`
|
||||
NextWarm int64 `json:"next_warm_at"`
|
||||
}
|
||||
|
||||
// PoolStatus is the console view of the cookie pool.
|
||||
type PoolStatus struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Running bool `json:"running"`
|
||||
Ratio float64 `json:"ratio"`
|
||||
CheckS int `json:"check_seconds"`
|
||||
Entries []WarmEntry `json:"entries"`
|
||||
}
|
||||
|
||||
type warmState struct {
|
||||
site string
|
||||
route string
|
||||
warming bool
|
||||
fails int
|
||||
lastAt time.Time
|
||||
lastMs int64
|
||||
lastErr string
|
||||
}
|
||||
|
||||
// StartWarmer launches the cookie-pool loop. targets is consulted on every
|
||||
// sweep, so console edits (sites, routes, toggles) are picked up live. A slot
|
||||
// is warmed when its cookie is missing, expired, or has less than warm_ratio
|
||||
// of its TTL left — requests then never wait for a solve to finish.
|
||||
func (s *Solver) StartWarmer(ctx context.Context, targets func() []WarmTarget) {
|
||||
if targets == nil {
|
||||
return
|
||||
}
|
||||
s.poolMu.Lock()
|
||||
if s.poolOn {
|
||||
s.poolMu.Unlock()
|
||||
return
|
||||
}
|
||||
s.poolOn = true
|
||||
if s.warms == nil {
|
||||
s.warms = map[string]*warmState{}
|
||||
}
|
||||
s.poolMu.Unlock()
|
||||
go s.warmLoop(ctx, targets)
|
||||
}
|
||||
|
||||
func (s *Solver) warmLoop(ctx context.Context, targets func() []WarmTarget) {
|
||||
defer func() {
|
||||
s.poolMu.Lock()
|
||||
s.poolOn = false
|
||||
s.poolMu.Unlock()
|
||||
}()
|
||||
for {
|
||||
s.sweep(ctx, targets)
|
||||
wait := time.Duration(s.Config().WarmCheckValue()) * time.Second
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(wait):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// sweep warms every due slot. Solves run in the background so one slow
|
||||
// browser window cannot stall the inspection; a slot already being warmed is
|
||||
// skipped until it finishes, and a slot whose last attempt failed waits out an
|
||||
// exponential backoff before the next try.
|
||||
func (s *Solver) sweep(ctx context.Context, targets func() []WarmTarget) {
|
||||
cfg := s.Config()
|
||||
if !cfg.Enabled || !cfg.WarmEnabled || cfg.ProviderValue() == config.ProviderManual {
|
||||
return
|
||||
}
|
||||
ttl := time.Duration(cfg.CookieTTLSeconds) * time.Second
|
||||
if ttl <= 0 {
|
||||
ttl = 3 * time.Hour
|
||||
}
|
||||
for _, tgt := range targets() {
|
||||
key := poolKey(tgt.Proxy, tgt.Site.Code)
|
||||
if !s.warmDue(key, tgt, ttl, cfg.WarmRatioValue(), cfg.WarmCheckValue()) {
|
||||
continue
|
||||
}
|
||||
s.markWarm(key, tgt, true, time.Time{}, 0, "")
|
||||
go s.warmOne(ctx, key, tgt, cfg)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Solver) warmDue(key string, tgt WarmTarget, ttl time.Duration, ratio float64, checkS int) bool {
|
||||
s.poolMu.Lock()
|
||||
st := s.warms[key]
|
||||
if st != nil && st.warming {
|
||||
s.poolMu.Unlock()
|
||||
return false
|
||||
}
|
||||
var lastAt time.Time
|
||||
fails := 0
|
||||
if st != nil {
|
||||
lastAt, fails = st.lastAt, st.fails
|
||||
}
|
||||
s.poolMu.Unlock()
|
||||
|
||||
// A failed slot backs off exponentially (×2 per consecutive failure, up
|
||||
// to 15 minutes) so a hopeless route cannot retry-storm the browser pool.
|
||||
if fails > 0 && !lastAt.IsZero() {
|
||||
backoff := time.Duration(checkS) * time.Second
|
||||
for i := 1; i < fails && backoff < 15*time.Minute; i++ {
|
||||
backoff *= 2
|
||||
}
|
||||
if backoff > 15*time.Minute || backoff <= 0 {
|
||||
backoff = 15 * time.Minute
|
||||
}
|
||||
if time.Since(lastAt) < backoff {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
c := s.coreFor(tgt.Proxy)
|
||||
c.mu.Lock()
|
||||
cs := c.cache[tgt.Site.Code]
|
||||
c.mu.Unlock()
|
||||
if cs == nil {
|
||||
return true // cold slot — fill it
|
||||
}
|
||||
return time.Until(cs.expiresAt) <= time.Duration(ratio*float64(ttl))
|
||||
}
|
||||
|
||||
func (s *Solver) warmOne(ctx context.Context, key string, tgt WarmTarget, cfg config.Turnstile) {
|
||||
site := tgt.Site
|
||||
start := time.Now()
|
||||
sess, err := httpx.NewSession(httpx.Options{
|
||||
ProxyURL: tgt.Proxy,
|
||||
Timeout: time.Duration(cfg.SolveTimeoutValue()+30) * time.Second,
|
||||
})
|
||||
if err == nil {
|
||||
err = s.Refresh(ctx, sess, &site, tgt.Proxy)
|
||||
sess.Close()
|
||||
}
|
||||
lastErr := ""
|
||||
if err != nil {
|
||||
lastErr = err.Error()
|
||||
}
|
||||
s.markWarm(key, tgt, false, start, time.Since(start).Milliseconds(), lastErr)
|
||||
}
|
||||
|
||||
func (s *Solver) markWarm(key string, tgt WarmTarget, warming bool, at time.Time, ms int64, lastErr string) {
|
||||
s.poolMu.Lock()
|
||||
defer s.poolMu.Unlock()
|
||||
if s.warms == nil {
|
||||
s.warms = map[string]*warmState{}
|
||||
}
|
||||
st, ok := s.warms[key]
|
||||
if !ok {
|
||||
st = &warmState{site: tgt.Site.Code, route: routeLabel(tgt.Proxy)}
|
||||
s.warms[key] = st
|
||||
}
|
||||
st.warming = warming
|
||||
if !at.IsZero() {
|
||||
st.lastAt = at
|
||||
st.lastMs = ms
|
||||
st.lastErr = lastErr
|
||||
if lastErr == "" {
|
||||
st.fails = 0
|
||||
} else {
|
||||
st.fails++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// PoolStatus reports every known slot: cached lifetime, warm progress, last
|
||||
// solve duration and the next planned refresh.
|
||||
func (s *Solver) PoolStatus() PoolStatus {
|
||||
cfg := s.Config()
|
||||
pooling := cfg.Enabled && cfg.WarmEnabled && cfg.ProviderValue() != config.ProviderManual
|
||||
ps := PoolStatus{
|
||||
Enabled: pooling,
|
||||
Ratio: cfg.WarmRatioValue(),
|
||||
CheckS: cfg.WarmCheckValue(),
|
||||
Entries: make([]WarmEntry, 0, 4),
|
||||
}
|
||||
s.poolMu.Lock()
|
||||
ps.Running = s.poolOn
|
||||
states := make(map[string]*warmState, len(s.warms))
|
||||
for k, v := range s.warms {
|
||||
cp := *v
|
||||
states[k] = &cp
|
||||
}
|
||||
s.poolMu.Unlock()
|
||||
|
||||
s.mu.Lock()
|
||||
cores := make([]*core, 0, len(s.cores))
|
||||
for _, c := range s.cores {
|
||||
cores = append(cores, c)
|
||||
}
|
||||
s.mu.Unlock()
|
||||
|
||||
now := time.Now()
|
||||
ttl := time.Duration(cfg.CookieTTLSeconds) * time.Second
|
||||
if ttl <= 0 {
|
||||
ttl = 3 * time.Hour
|
||||
}
|
||||
|
||||
entries := map[string]*WarmEntry{}
|
||||
for _, c := range cores {
|
||||
c.mu.Lock()
|
||||
for code, cs := range c.cache {
|
||||
e := &WarmEntry{Site: code, Route: c.proxy, Valid: now.Before(cs.expiresAt)}
|
||||
if e.Valid {
|
||||
e.Remaining = int64(cs.expiresAt.Sub(now).Seconds())
|
||||
}
|
||||
entries[poolKey(c.proxy, code)] = e
|
||||
}
|
||||
c.mu.Unlock()
|
||||
}
|
||||
for k, st := range states {
|
||||
e, ok := entries[k]
|
||||
if !ok {
|
||||
e = &WarmEntry{Site: st.site, Route: st.route}
|
||||
entries[k] = e
|
||||
}
|
||||
e.Warming = st.warming
|
||||
if !st.lastAt.IsZero() {
|
||||
e.LastAt = st.lastAt.Unix()
|
||||
}
|
||||
e.LastMs = st.lastMs
|
||||
e.LastErr = st.lastErr
|
||||
}
|
||||
for _, e := range entries {
|
||||
if pooling && e.Valid {
|
||||
until := time.Duration(e.Remaining)*time.Second - time.Duration(ps.Ratio*float64(ttl))
|
||||
if until < 0 {
|
||||
until = 0
|
||||
}
|
||||
e.NextWarm = now.Add(until).Unix()
|
||||
}
|
||||
ps.Entries = append(ps.Entries, *e)
|
||||
}
|
||||
sort.Slice(ps.Entries, func(i, j int) bool {
|
||||
if ps.Entries[i].Site != ps.Entries[j].Site {
|
||||
return ps.Entries[i].Site < ps.Entries[j].Site
|
||||
}
|
||||
return ps.Entries[i].Route < ps.Entries[j].Route
|
||||
})
|
||||
return ps
|
||||
}
|
||||
|
||||
func poolKey(proxy, site string) string { return routeLabel(proxy) + "|" + site }
|
||||
|
||||
func routeLabel(proxy string) string {
|
||||
if proxy == "" {
|
||||
return "direct"
|
||||
}
|
||||
return proxy
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
package turnstile
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/nyoungo/dsfree2api/internal/config"
|
||||
"github.com/nyoungo/dsfree2api/internal/httpx"
|
||||
)
|
||||
|
||||
func warmerSolver(t *testing.T, mutate func(*config.Turnstile)) (*Solver, *config.Config) {
|
||||
t.Helper()
|
||||
cfg, err := config.Load(examplePath)
|
||||
if err != nil {
|
||||
t.Fatalf("load config: %v", err)
|
||||
}
|
||||
cfg.Turnstile.Enabled = true
|
||||
cfg.Turnstile.WarmEnabled = true
|
||||
cfg.Turnstile.CookieTTLSeconds = 3600
|
||||
cfg.Turnstile.WarmRatio = 0.3
|
||||
if mutate != nil {
|
||||
mutate(&cfg.Turnstile)
|
||||
}
|
||||
return New(cfg.Turnstile, httpx.DefaultUserAgent), cfg
|
||||
}
|
||||
|
||||
func waitCalls(t *testing.T, calls *int32, want int32) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(2 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
if atomic.LoadInt32(calls) >= want {
|
||||
return
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("refresh calls = %d, want >= %d", atomic.LoadInt32(calls), want)
|
||||
}
|
||||
|
||||
func waitSettled(t *testing.T, solver *Solver) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(2 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
ps := solver.PoolStatus()
|
||||
busy := false
|
||||
for _, e := range ps.Entries {
|
||||
if e.Warming {
|
||||
busy = true
|
||||
}
|
||||
}
|
||||
if !busy {
|
||||
return
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
t.Fatal("cookie pool did not settle")
|
||||
}
|
||||
|
||||
// rewindLastAt backdates the last attempt of a slot so backoff windows can be
|
||||
// tested without sleeping through them.
|
||||
func rewindLastAt(solver *Solver, key string, d time.Duration) {
|
||||
solver.poolMu.Lock()
|
||||
if st := solver.warms[key]; st != nil {
|
||||
st.lastAt = time.Now().Add(-d)
|
||||
}
|
||||
solver.poolMu.Unlock()
|
||||
}
|
||||
|
||||
func TestWarmDueBacksOffAfterFailure(t *testing.T) {
|
||||
solver, cfg := warmerSolver(t, nil)
|
||||
tgt := WarmTarget{Site: *cfg.Sites["de"], Proxy: ""}
|
||||
key := poolKey("", "de")
|
||||
ttl := time.Hour
|
||||
|
||||
if !solver.warmDue(key, tgt, ttl, 0.3, 5) {
|
||||
t.Fatal("cold slot should be due")
|
||||
}
|
||||
|
||||
// First failure: the 5s backoff must hold.
|
||||
solver.markWarm(key, tgt, false, time.Now(), 100, "boom")
|
||||
if solver.warmDue(key, tgt, ttl, 0.3, 5) {
|
||||
t.Fatal("first backoff was ignored")
|
||||
}
|
||||
|
||||
// Second failure doubles it: 6s elapsed is still inside the 10s window.
|
||||
solver.markWarm(key, tgt, false, time.Now(), 100, "boom")
|
||||
rewindLastAt(solver, key, 6*time.Second)
|
||||
if solver.warmDue(key, tgt, ttl, 0.3, 5) {
|
||||
t.Fatal("second failure did not double the backoff")
|
||||
}
|
||||
|
||||
// 11s elapsed clears the 10s window.
|
||||
rewindLastAt(solver, key, 11*time.Second)
|
||||
if !solver.warmDue(key, tgt, ttl, 0.3, 5) {
|
||||
t.Fatal("slot should be due once the backoff elapsed")
|
||||
}
|
||||
|
||||
// A success resets the failure counter.
|
||||
solver.markWarm(key, tgt, false, time.Now(), 50, "")
|
||||
if !solver.warmDue(key, tgt, ttl, 0.3, 5) {
|
||||
t.Fatal("success should clear the backoff")
|
||||
}
|
||||
|
||||
// The backoff is capped at 15 minutes.
|
||||
for i := 0; i < 12; i++ {
|
||||
solver.markWarm(key, tgt, false, time.Now(), 100, "boom")
|
||||
}
|
||||
rewindLastAt(solver, key, 11*time.Second)
|
||||
if solver.warmDue(key, tgt, ttl, 0.3, 5) {
|
||||
t.Fatal("capped backoff should still hold at 11s")
|
||||
}
|
||||
rewindLastAt(solver, key, 16*time.Minute)
|
||||
if !solver.warmDue(key, tgt, ttl, 0.3, 5) {
|
||||
t.Fatal("slot should be due after the capped backoff")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWarmSweepFillsColdSlotAndSkipsFresh(t *testing.T) {
|
||||
solver, cfg := warmerSolver(t, nil)
|
||||
var calls int32
|
||||
solver.refreshOverride = func(_ context.Context, _ httpx.Session, s *config.Site, c *core) error {
|
||||
atomic.AddInt32(&calls, 1)
|
||||
c.mu.Lock()
|
||||
c.cache[s.Code] = &cookieState{cookies: map[string]string{"dsts": "ok"}, expiresAt: time.Now().Add(time.Hour)}
|
||||
c.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
targets := func() []WarmTarget {
|
||||
return []WarmTarget{{Site: *cfg.Sites["de"], Proxy: ""}}
|
||||
}
|
||||
|
||||
// Cold slot: the first sweep must start a solve.
|
||||
solver.sweep(context.Background(), targets)
|
||||
waitCalls(t, &calls, 1)
|
||||
waitSettled(t, solver)
|
||||
|
||||
// Fresh cookie (1h left, threshold 0.3×1h): a second sweep must not solve.
|
||||
solver.sweep(context.Background(), targets)
|
||||
time.Sleep(80 * time.Millisecond)
|
||||
if got := atomic.LoadInt32(&calls); got != 1 {
|
||||
t.Fatalf("fresh slot was re-warmed: calls = %d", got)
|
||||
}
|
||||
|
||||
// Remaining TTL below the threshold: due again.
|
||||
c := solver.coreFor("")
|
||||
c.mu.Lock()
|
||||
c.cache["de"] = &cookieState{cookies: map[string]string{"dsts": "ok"}, expiresAt: time.Now().Add(30 * time.Second)}
|
||||
c.mu.Unlock()
|
||||
solver.sweep(context.Background(), targets)
|
||||
waitCalls(t, &calls, 2)
|
||||
waitSettled(t, solver)
|
||||
|
||||
ps := solver.PoolStatus()
|
||||
if !ps.Enabled {
|
||||
t.Error("pool should report enabled")
|
||||
}
|
||||
if len(ps.Entries) != 1 {
|
||||
t.Fatalf("pool entries = %d, want 1", len(ps.Entries))
|
||||
}
|
||||
e := ps.Entries[0]
|
||||
if !e.Valid || e.LastAt == 0 || e.LastErr != "" || e.Warming {
|
||||
t.Errorf("pool entry = %+v", e)
|
||||
}
|
||||
if e.NextWarm == 0 {
|
||||
t.Error("next_warm_at should be scheduled for a valid pooled cookie")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWarmSweepSkipsManualAndDisabled(t *testing.T) {
|
||||
var calls int32
|
||||
override := func(context.Context, httpx.Session, *config.Site, *core) error {
|
||||
atomic.AddInt32(&calls, 1)
|
||||
return nil
|
||||
}
|
||||
|
||||
solver, cfg := warmerSolver(t, func(tc *config.Turnstile) { tc.Provider = config.ProviderManual })
|
||||
solver.refreshOverride = override
|
||||
solver.sweep(context.Background(), func() []WarmTarget {
|
||||
return []WarmTarget{{Site: *cfg.Sites["de"]}}
|
||||
})
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
if calls != 0 {
|
||||
t.Fatalf("manual provider must not warm, calls = %d", calls)
|
||||
}
|
||||
|
||||
solver2, cfg2 := warmerSolver(t, func(tc *config.Turnstile) { tc.Enabled = false })
|
||||
solver2.refreshOverride = override
|
||||
solver2.sweep(context.Background(), func() []WarmTarget {
|
||||
return []WarmTarget{{Site: *cfg2.Sites["de"]}}
|
||||
})
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
if calls != 0 {
|
||||
t.Fatalf("disabled solver must not warm, calls = %d", calls)
|
||||
}
|
||||
|
||||
solver3, cfg3 := warmerSolver(t, func(tc *config.Turnstile) { tc.WarmEnabled = false })
|
||||
solver3.refreshOverride = override
|
||||
solver3.sweep(context.Background(), func() []WarmTarget {
|
||||
return []WarmTarget{{Site: *cfg3.Sites["de"]}}
|
||||
})
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
if calls != 0 {
|
||||
t.Fatalf("pool switch off must not warm, calls = %d", calls)
|
||||
}
|
||||
}
|
||||
@@ -35,6 +35,19 @@ func (r Route) Key() string {
|
||||
return r.Proxy
|
||||
}
|
||||
|
||||
// RouteCandidate is one egress route supplied by the proxy pool.
|
||||
type RouteCandidate struct {
|
||||
Name string
|
||||
Proxy string
|
||||
}
|
||||
|
||||
// RouteSource resolves per-site route candidates and records request
|
||||
// outcomes. Implemented by proxypool.Manager; nil when the pool is unused.
|
||||
type RouteSource interface {
|
||||
Candidates(site string) []RouteCandidate
|
||||
Report(proxy string, ok bool)
|
||||
}
|
||||
|
||||
type chatConfig struct {
|
||||
BotID int
|
||||
PostID int
|
||||
@@ -44,9 +57,10 @@ type chatConfig struct {
|
||||
}
|
||||
|
||||
type Client struct {
|
||||
cfg *config.Config
|
||||
ts *turnstile.Solver
|
||||
log *slog.Logger
|
||||
cfg *config.Config
|
||||
ts *turnstile.Solver
|
||||
log *slog.Logger
|
||||
pool RouteSource
|
||||
|
||||
stateMu sync.Mutex
|
||||
chatCfg map[string]chatConfig
|
||||
@@ -61,12 +75,17 @@ type Client struct {
|
||||
lastWarnMsg string
|
||||
lastWarnAt time.Time
|
||||
|
||||
// quotaObserver, when set, is notified on the first site-side quota
|
||||
// exhaustion of each request — including ones hidden by cross-site
|
||||
// failover.
|
||||
quotaObserver func(site, model, route string, err error)
|
||||
|
||||
// chatOnceOverride is a test seam; nil in production.
|
||||
chatOnceOverride func(ctx context.Context, site config.Site, modelID string, model config.Model,
|
||||
prompt string, route Route, info *ServeInfo, yield func(Event) error) error
|
||||
}
|
||||
|
||||
func New(cfg *config.Config, ts *turnstile.Solver, log *slog.Logger) *Client {
|
||||
func New(cfg *config.Config, ts *turnstile.Solver, log *slog.Logger, pool RouteSource) *Client {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
@@ -74,6 +93,7 @@ func New(cfg *config.Config, ts *turnstile.Solver, log *slog.Logger) *Client {
|
||||
cfg: cfg,
|
||||
ts: ts,
|
||||
log: log,
|
||||
pool: pool,
|
||||
chatCfg: map[string]chatConfig{},
|
||||
cfgLocks: map[string]*sync.Mutex{},
|
||||
gates: map[string]chan struct{}{},
|
||||
@@ -81,6 +101,13 @@ func New(cfg *config.Config, ts *turnstile.Solver, log *slog.Logger) *Client {
|
||||
}
|
||||
}
|
||||
|
||||
// SetQuotaObserver registers a callback that fires whenever a site reports
|
||||
// quota exhaustion, even when cross-site failover later succeeds and the error
|
||||
// would otherwise never surface. Set it before serving requests.
|
||||
func (c *Client) SetQuotaObserver(fn func(site, model, route string, err error)) {
|
||||
c.quotaObserver = fn
|
||||
}
|
||||
|
||||
// Routes returns the primary proxy followed by the fallbacks, deduplicated.
|
||||
func (c *Client) Routes() []Route {
|
||||
c.cfg.RLock()
|
||||
@@ -241,6 +268,40 @@ func (c *Client) logCacheReject(site, modelID, prompt string) {
|
||||
"head", strconv.Quote(head), "tail", strconv.Quote(tail))
|
||||
}
|
||||
|
||||
// routesFor returns the ordered candidate routes for one site: proxy-pool
|
||||
// candidates first (sticky, health filtered), then the global primary and
|
||||
// fallback lines as the safety net.
|
||||
func (c *Client) routesFor(siteCode string) []Route {
|
||||
out := make([]Route, 0, 4)
|
||||
seen := map[string]bool{}
|
||||
add := func(r Route) {
|
||||
key := r.Key()
|
||||
if seen[key] {
|
||||
return
|
||||
}
|
||||
seen[key] = true
|
||||
out = append(out, r)
|
||||
}
|
||||
if c.pool != nil {
|
||||
for _, cand := range c.pool.Candidates(siteCode) {
|
||||
add(Route{Name: cand.Name, Proxy: cand.Proxy})
|
||||
}
|
||||
}
|
||||
for _, r := range c.Routes() {
|
||||
add(r)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// poolReport feeds a route outcome back to the proxy pool; it is a no-op for
|
||||
// global routes and the direct connection.
|
||||
func (c *Client) poolReport(proxy string, ok bool) {
|
||||
if c.pool == nil || proxy == "" {
|
||||
return
|
||||
}
|
||||
c.pool.Report(proxy, ok)
|
||||
}
|
||||
|
||||
// Chat streams the prompt through the upstream site, honouring route
|
||||
// failover, slow-start detection, session refresh and cross-site failover.
|
||||
func (c *Client) Chat(ctx context.Context, modelID string, prompt string, info *ServeInfo, yield func(Event) error) error {
|
||||
@@ -357,23 +418,33 @@ func (c *Client) chatWithRetries(ctx context.Context, modelID string, model conf
|
||||
if !site.Enabled {
|
||||
return errf("site %s is disabled", model.Site)
|
||||
}
|
||||
routes := c.Routes()
|
||||
routes := c.routesFor(site.Code)
|
||||
quotaNotified := false
|
||||
|
||||
var lastErr error
|
||||
sawQuota := false
|
||||
for attempt := 0; ; attempt++ {
|
||||
var failed []Route
|
||||
for i, route := range routes {
|
||||
allowSlow := i == 0 && len(routes) > 1
|
||||
// A first-time Turnstile solve can far exceed the slow-start
|
||||
// budget; only arm the timer when cookies are already cached
|
||||
// (or no solver is configured at all).
|
||||
cookiesReady := c.ts == nil || c.ts.CookiesReady(site.Code, route.Proxy)
|
||||
allowSlow := i == 0 && len(routes) > 1 && cookiesReady
|
||||
err := c.chatOnceSlowStart(ctx, site, modelID, model, prompt, route, allowSlow, info, yield)
|
||||
if err == nil {
|
||||
if sawQuota {
|
||||
c.clearQuotaCool(site.Code)
|
||||
c.probeBalanceAsync(site, model, route)
|
||||
}
|
||||
c.poolReport(route.Proxy, true)
|
||||
return nil
|
||||
}
|
||||
lastErr = err
|
||||
if isQuota(err) && !quotaNotified && c.quotaObserver != nil {
|
||||
quotaNotified = true
|
||||
c.quotaObserver(site.Code, modelID, route.Name, err)
|
||||
}
|
||||
if contentStarted(err) {
|
||||
return AsUpstream(err)
|
||||
}
|
||||
@@ -400,6 +471,11 @@ func (c *Client) chatWithRetries(ctx context.Context, modelID string, model conf
|
||||
if isCacheEmptyProblem(err) {
|
||||
return AsUpstream(err)
|
||||
}
|
||||
// Quota/session failures are site-side, not proxy-side — do not
|
||||
// blame the egress endpoint for them.
|
||||
if !isQuota(err) && !isSession(err) {
|
||||
c.poolReport(route.Proxy, false)
|
||||
}
|
||||
failed = append(failed, route)
|
||||
c.logUpstreamErr(site.Code, modelID, attempt, route, err)
|
||||
if isQuota(err) {
|
||||
|
||||
@@ -32,7 +32,7 @@ func TestRoutesKeepPrimaryFirstAndDedupe(t *testing.T) {
|
||||
cfg.Proxy.URL = ""
|
||||
cfg.Proxy.FallbackURLs = []string{"socks5://fallback:1", "socks5://fallback:1"}
|
||||
|
||||
c := New(cfg, nil, nil)
|
||||
c := New(cfg, nil, nil, nil)
|
||||
got := c.Routes()
|
||||
want := []Route{{Name: "primary", Proxy: ""}, {Name: "fallback-1", Proxy: "socks5://fallback:1"}}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
@@ -47,7 +47,7 @@ func TestSlowPrimarySwitchesToFallbackBeforeContent(t *testing.T) {
|
||||
cfg.Proxy.SlowStartSeconds = 0.02
|
||||
|
||||
var calls []string
|
||||
c := New(cfg, nil, nil)
|
||||
c := New(cfg, nil, nil, nil)
|
||||
c.chatOnceOverride = func(ctx context.Context, site config.Site, modelID string, model config.Model,
|
||||
prompt string, route Route, info *ServeInfo, yield func(Event) error) error {
|
||||
calls = append(calls, route.Name)
|
||||
@@ -86,7 +86,7 @@ func TestPrimaryErrorSwitchesToFallbackBeforeContent(t *testing.T) {
|
||||
cfg.Proxy.FallbackURLs = []string{"socks5://fallback:1"}
|
||||
|
||||
var calls []string
|
||||
c := New(cfg, nil, nil)
|
||||
c := New(cfg, nil, nil, nil)
|
||||
c.chatOnceOverride = func(ctx context.Context, site config.Site, modelID string, model config.Model,
|
||||
prompt string, route Route, info *ServeInfo, yield func(Event) error) error {
|
||||
calls = append(calls, route.Name)
|
||||
@@ -119,7 +119,7 @@ func TestNoFallbackAfterContentStarted(t *testing.T) {
|
||||
cfg.Proxy.FallbackURLs = []string{"socks5://fallback:1"}
|
||||
|
||||
var calls []string
|
||||
c := New(cfg, nil, nil)
|
||||
c := New(cfg, nil, nil, nil)
|
||||
c.chatOnceOverride = func(ctx context.Context, site config.Site, modelID string, model config.Model,
|
||||
prompt string, route Route, info *ServeInfo, yield func(Event) error) error {
|
||||
calls = append(calls, route.Name)
|
||||
@@ -144,6 +144,32 @@ func TestNoFallbackAfterContentStarted(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestQuotaObserverFiresOncePerSiteBeforeFailover(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
cfg.Upstream.AutoRefresh = false
|
||||
c := New(cfg, nil, nil, nil)
|
||||
c.chatOnceOverride = func(ctx context.Context, site config.Site, modelID string, model config.Model,
|
||||
prompt string, route Route, info *ServeInfo, yield func(Event) error) error {
|
||||
if site.Code == "de" {
|
||||
return newQuota(`sse quota exhausted: {"quota_notice":{}}`)
|
||||
}
|
||||
yield(Event{Kind: KindDelta, Value: "ok"})
|
||||
return nil
|
||||
}
|
||||
var fired []string
|
||||
c.SetQuotaObserver(func(site, model, route string, err error) {
|
||||
fired = append(fired, site+"|"+model+"|"+route)
|
||||
})
|
||||
|
||||
if err := c.Chat(context.Background(), "deepseek-v4-flash-de", "hi", &ServeInfo{},
|
||||
func(ev Event) error { return nil }); err != nil {
|
||||
t.Fatalf("chat should succeed via cross-site failover: %v", err)
|
||||
}
|
||||
if len(fired) != 1 || fired[0] != "de|deepseek-v4-flash-de|primary" {
|
||||
t.Fatalf("quota observer events = %v, want [de|deepseek-v4-flash-de|primary]", fired)
|
||||
}
|
||||
}
|
||||
|
||||
func TestQuotaNoticeTriggersQuotaError(t *testing.T) {
|
||||
payload := map[string]any{
|
||||
"error": "Dein heutiges Guthaben von 10.000 Token (V4-Pro) ist aufgebraucht.",
|
||||
@@ -211,7 +237,7 @@ func TestTurnstileRequiredWhileDisabledFailsFast(t *testing.T) {
|
||||
cfg.Upstream.RefreshRetries = 4
|
||||
|
||||
var calls int
|
||||
c := New(cfg, nil, nil)
|
||||
c := New(cfg, nil, nil, nil)
|
||||
c.chatOnceOverride = func(ctx context.Context, site config.Site, modelID string, model config.Model,
|
||||
prompt string, route Route, info *ServeInfo, yield func(Event) error) error {
|
||||
calls++
|
||||
@@ -237,7 +263,7 @@ func TestNonceFetchFailureFailsFast(t *testing.T) {
|
||||
cfg.Upstream.RefreshRetries = 4
|
||||
|
||||
var calls int
|
||||
c := New(cfg, nil, nil)
|
||||
c := New(cfg, nil, nil, nil)
|
||||
c.chatOnceOverride = func(ctx context.Context, site config.Site, modelID string, model config.Model,
|
||||
prompt string, route Route, info *ServeInfo, yield func(Event) error) error {
|
||||
calls++
|
||||
@@ -273,7 +299,7 @@ func TestSSEDeltaAndDoneEvents(t *testing.T) {
|
||||
|
||||
func TestQuotaSiteCoolsDownAndPrefersSibling(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
c := New(cfg, nil, nil)
|
||||
c := New(cfg, nil, nil, nil)
|
||||
calls := map[string]int{}
|
||||
c.chatOnceOverride = func(_ context.Context, site config.Site, modelID string, model config.Model,
|
||||
_ string, _ Route, _ *ServeInfo, _ func(Event) error) error {
|
||||
@@ -318,7 +344,7 @@ func TestCacheEmptyFailsOverToMirrorWithoutRefreshCycles(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
cfg.Upstream.AutoRefresh = true
|
||||
cfg.Upstream.RefreshRetries = 3
|
||||
c := New(cfg, nil, nil)
|
||||
c := New(cfg, nil, nil, nil)
|
||||
calls := map[string]int{}
|
||||
c.chatOnceOverride = func(_ context.Context, site config.Site, modelID string, model config.Model,
|
||||
_ string, _ Route, _ *ServeInfo, _ func(Event) error) error {
|
||||
@@ -348,7 +374,7 @@ func TestQuotaRotationRecoversWithoutSolve(t *testing.T) {
|
||||
if _, _, err := solver.ImportCookies("de", "cf_clearance=abc; dsgt_gid="+oldGid, ""); err != nil {
|
||||
t.Fatalf("seed cookies: %v", err)
|
||||
}
|
||||
c := New(cfg, solver, nil)
|
||||
c := New(cfg, solver, nil, nil)
|
||||
calls := 0
|
||||
c.chatOnceOverride = func(_ context.Context, site config.Site, modelID string, model config.Model,
|
||||
_ string, _ Route, _ *ServeInfo, _ func(Event) error) error {
|
||||
|
||||
Reference in new issue
Block a user