Refocus docs around MySearch Proxy
This commit is contained in:
skernelx committed
1 parent
27fd9f6b3c
commit
638c690ae9
5 files changed
+169
-815
No files matched your search
@@ -2,27 +2,25 @@
|
||||
|
||||
[English Guide](./README_EN.md)
|
||||
|
||||
这是现在这套真正可用的多服务注册器，已经把 **Tavily** 和
|
||||
**Firecrawl** 放进同一个启动台：
|
||||
这是当前的多服务注册器与聚合 API 上游工具，聚焦两件事：
|
||||
|
||||
- **Tavily**：AI 搜索 API
|
||||
- **Firecrawl**：网页抓取 API
|
||||
- 注册 `Tavily` / `Firecrawl` key
|
||||
- 验证 key 是否真实可用，并把可用 key 提供给统一搜索层
|
||||
|
||||
和早期零散脚本不一样，这版已经统一收口到一条稳定链路：
|
||||
它不是 `MySearch` 的正式产品仓库。
|
||||
如果你要部署公开可用的统一搜索控制台、MCP、Skill 和 Social / X
|
||||
接入，请直接使用独立项目：
|
||||
|
||||
- [MySearch-Proxy](https://github.com/skernelx/MySearch-Proxy)
|
||||
|
||||
## 这个仓库负责什么
|
||||
|
||||
- 本地真实浏览器注册
|
||||
- Tavily 本地 Turnstile Solver
|
||||
- 邮箱 API 自动收验证码 / 验证链接
|
||||
- 提取到 API Key 后立刻做真实调用验证
|
||||
- 可选自动上传到多服务代理池
|
||||
|
||||
目标很直接：把 Tavily / Firecrawl 的注册、验证、落盘、上传全部放到一套真正能长期跑的工作流里。
|
||||
|
||||
Cloudflare 通用无限别名域名邮箱教程见：
|
||||
[Cloudflare 邮件设置详解](./docs/Cloudflare%E9%82%AE%E4%BB%B6%E8%AE%BE%E7%BD%AE%E8%AF%A6%E8%A7%A3.md)
|
||||
|
||||
英文版说明见：
|
||||
[Cloudflare Mail Setup Guide](./docs/Cloudflare-Mail-Setup-Guide.md)
|
||||
- 可选自动上传到统一代理池
|
||||
- 可作为 `MySearch` / `MySearch-Proxy` 的 provider 上游之一
|
||||
|
||||
## Features
|
||||
|
||||
@@ -33,27 +31,19 @@ Cloudflare 通用无限别名域名邮箱教程见：
|
||||
- 并发注册：支持批量和并发执行
|
||||
- 后台浏览器模式：默认 headless，必要时可切前台排查
|
||||
- 真实可用性验证：拿到 key 后马上调用官方接口
|
||||
- 自动上传到代理池：上传时会带上服务标记，服务器能自动识别 Tavily / Firecrawl 并写入各自池子
|
||||
- 代理控制台：支持独立 Key 池、独立 Token 池和真实额度同步
|
||||
- 自动上传到代理池：上传时会带上服务标记，服务端能自动识别 Tavily / Firecrawl 并写入各自池子
|
||||
- 适合作为统一搜索网关的上游：给 `MySearch-Proxy` 提供稳定的 key 来源
|
||||
- 跨平台启动：Windows / macOS / Linux 都能直接跑
|
||||
|
||||
## Screenshots
|
||||
## 配套工作台预览
|
||||
|
||||
### 多服务启动台
|
||||
### MySearch Proxy 首屏
|
||||
|
||||
|
||||
|
||||
|
||||
### 并发注册与真实验证
|
||||
### MySearch Proxy 工作台
|
||||
|
||||
|
||||
|
||||
### Proxy 工作台切换
|
||||
|
||||
|
||||
|
||||
### Proxy Key 池详情
|
||||
|
||||
|
||||
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -70,7 +60,7 @@ cd tavily-key-generator
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
编辑 `.env`，填好你的邮箱链路和可选上传配置。
|
||||
编辑 `.env`，填好邮箱配置和可选上传配置。
|
||||
|
||||
### 3. Run
|
||||
|
||||
@@ -80,7 +70,7 @@ macOS / Linux:
|
||||
python3 run.py
|
||||
```
|
||||
|
||||
或：
|
||||
或者：
|
||||
|
||||
```bash
|
||||
./start_auto.sh
|
||||
@@ -92,251 +82,37 @@ Windows:
|
||||
start_auto.bat
|
||||
```
|
||||
|
||||
## How It Works
|
||||
## 推荐搭配方式
|
||||
|
||||
程序启动后会自动执行：
|
||||
如果你希望把注册出来的 key 接成公开可用的统一搜索服务，推荐直接搭配：
|
||||
|
||||
1. 选择要注册的服务（Tavily / Firecrawl）
|
||||
2. 创建或复用 `venv`
|
||||
3. 安装 Python 依赖
|
||||
4. 安装 Camoufox / Playwright 浏览器依赖
|
||||
5. 读取 `.env`
|
||||
6. 检查邮箱 provider 配置
|
||||
7. 如果配置了多个域名，提示选择本轮使用的域名
|
||||
8. 输入注册数量
|
||||
9. 输入并发数
|
||||
10. 选择是否自动上传到服务器
|
||||
11. 如果是 Tavily，自动启动本地 Solver
|
||||
12. 自动处理邮箱验证码与密码设置
|
||||
13. 遇到随机 challenge 时自动恢复（Tavily）
|
||||
14. 提取 API Key
|
||||
15. 真实调用 API 验证
|
||||
16. 保存到 `accounts.txt` 或 `firecrawl_accounts.txt`
|
||||
17. 如已开启上传，则继续上传到服务器
|
||||
- [MySearch-Proxy](https://github.com/skernelx/MySearch-Proxy)
|
||||
|
||||
## Runtime Flow
|
||||
它负责：
|
||||
|
||||
```text
|
||||
run.py
|
||||
-> choose service (Tavily / Firecrawl)
|
||||
-> load .env
|
||||
-> choose domain
|
||||
-> input count / concurrency
|
||||
-> choose upload or not
|
||||
-> [Tavily only] start Turnstile Solver
|
||||
-> create mailbox
|
||||
-> open signup page
|
||||
-> [Tavily only] solve Turnstile locally
|
||||
-> receive email verification link
|
||||
-> set password
|
||||
-> [Tavily only] recover random password-page challenge
|
||||
-> enter dashboard
|
||||
-> extract API key
|
||||
-> verify API key with real API call
|
||||
-> save / upload
|
||||
```
|
||||
- 统一控制台
|
||||
- Tavily / Firecrawl / Social / X 统一接线
|
||||
- MCP + Skill 安装说明
|
||||
- 官方接口与 compatible 网关双接入
|
||||
- 面向 Codex / Claude Code 的最终接入方式
|
||||
|
||||
## Configuration
|
||||
当前仓库则继续专注于：
|
||||
|
||||
完整配置示例见 [`.env.example`](./.env.example)。
|
||||
- 注册可用 key
|
||||
- 做真实可用性验证
|
||||
- 提供聚合 API 上游能力
|
||||
|
||||
### Cloudflare Mail API
|
||||
`proxy/` 目录目前仅保留给旧部署或迁移场景，不再作为主推荐入口。
|
||||
|
||||
```env
|
||||
EMAIL_PROVIDER=cloudflare
|
||||
EMAIL_API_URL=https://your-mail-api.example.com
|
||||
EMAIL_API_TOKEN=replace-with-your-token
|
||||
EMAIL_DOMAIN=example.com
|
||||
EMAIL_DOMAINS=example.com,example.org
|
||||
```
|
||||
## MySearch 的关系
|
||||
|
||||
说明：
|
||||
这里的 `tavily-key-generator` 只负责：
|
||||
|
||||
- 单域名可只填 `EMAIL_DOMAIN`
|
||||
- 多域名可填 `EMAIL_DOMAINS`
|
||||
- 启动时会提示选择本轮使用的域名
|
||||
- 注册 Tavily / Firecrawl key
|
||||
- 验证 key 是否真实可用
|
||||
- 提供可选聚合 API，供 `MySearch` / `MySearch-Proxy` 调用
|
||||
|
||||
### DuckMail API
|
||||
`MySearch` 自己的安装、MCP 工具、统一控制台、social gateway、Codex /
|
||||
Claude Code 接入说明，统一放在：
|
||||
|
||||
```env
|
||||
EMAIL_PROVIDER=duckmail
|
||||
DUCKMAIL_API_URL=https://api.duckmail.sbs
|
||||
DUCKMAIL_API_KEY=
|
||||
DUCKMAIL_DOMAIN=
|
||||
DUCKMAIL_DOMAINS=
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- 可使用单域名或多域名配置
|
||||
- 如果你有 DuckMail 私有域名和 API Key，直接填入即可
|
||||
- 公开 DuckMail 域名可以测试收信链路，但不保证能通过 Tavily 风控
|
||||
|
||||
### Upload to Your Server
|
||||
|
||||
```env
|
||||
SERVER_URL=https://your-server.example.com
|
||||
SERVER_ADMIN_PASSWORD=replace-with-your-admin-password
|
||||
DEFAULT_UPLOAD=true
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- `DEFAULT_UPLOAD=true` 表示启动台默认开启自动上传
|
||||
- 真正是否上传，仍以本轮启动时的选择为准
|
||||
- 上传到代理服务器时会带 `service` 字段
|
||||
- Tavily 会自动进入 Tavily 池，Firecrawl 会自动进入 Firecrawl 池
|
||||
|
||||
### Runtime Options
|
||||
|
||||
```env
|
||||
DEFAULT_COUNT=1
|
||||
DEFAULT_CONCURRENCY=2
|
||||
DEFAULT_DELAY=10
|
||||
REGISTER_HEADLESS=true
|
||||
FIRECRAWL_REGISTER_HEADLESS=true
|
||||
EMAIL_CODE_TIMEOUT=90
|
||||
API_KEY_TIMEOUT=20
|
||||
EMAIL_POLL_INTERVAL=3
|
||||
SOLVER_PORT=5073
|
||||
SOLVER_THREADS=1
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- `REGISTER_HEADLESS=true` 表示浏览器后台运行
|
||||
- `FIRECRAWL_REGISTER_HEADLESS` 未单独配置时会继承 `REGISTER_HEADLESS`
|
||||
- 现在默认是后台运行；如果你遇到 `Security check failed`，再临时改成 `false` 用前台浏览器排查
|
||||
- `SOLVER_THREADS` 最终会自动取 `max(SOLVER_THREADS, 本轮并发数)`
|
||||
- 普通使用场景下不需要额外传命令参数
|
||||
|
||||
## Output
|
||||
|
||||
注册成功后，结果会写入：
|
||||
|
||||
**Tavily**:
|
||||
```text
|
||||
accounts.txt
|
||||
```
|
||||
|
||||
**Firecrawl**:
|
||||
```text
|
||||
firecrawl_accounts.txt
|
||||
```
|
||||
|
||||
格式：
|
||||
```text
|
||||
email,password,api_key
|
||||
email,password,api_key
|
||||
```
|
||||
|
||||
## Real-World Validation
|
||||
|
||||
当前主线已经做过真实回归验证：
|
||||
|
||||
**Tavily**:
|
||||
- Cloudflare 邮箱链路可跑通完整注册
|
||||
- 邮箱验证码可自动读取
|
||||
- 获取到 API Key 后会立即做真实 API 调用验证
|
||||
- 并发注册已做过真实回归
|
||||
- 密码页随机 challenge 已补恢复逻辑，并已真测通过
|
||||
|
||||
**Firecrawl**:
|
||||
- 邮箱验证链接可自动读取
|
||||
- 支持自动登录和 API Key 提取
|
||||
- API Key 验证通过真实 API 调用
|
||||
|
||||
## Known Limitations
|
||||
|
||||
### DuckMail Public Domains
|
||||
|
||||
DuckMail 公开域名当前的状态是：
|
||||
|
||||
- 创建邮箱可以
|
||||
- 接收 6 位验证码可以
|
||||
- 但 Tavily 在密码页可能直接风控拦截
|
||||
|
||||
常见页面提示：
|
||||
|
||||
```text
|
||||
Suspicious activity detected
|
||||
```
|
||||
|
||||
如果你要跑通完整注册，建议优先使用：
|
||||
|
||||
- Cloudflare 自定义域名邮箱
|
||||
- DuckMail 私有域名 + API Key
|
||||
|
||||
### First Run on New Machine
|
||||
|
||||
首次换机器运行时，建议先单账号跑通一遍，再开并发。
|
||||
|
||||
因为首次运行会自动下载浏览器依赖，且不同机器的本地网络环境、代理环境、系统依赖可能不同。
|
||||
|
||||
## Project Structure
|
||||
|
||||
```text
|
||||
.
|
||||
├── run.py # 唯一推荐入口
|
||||
├── tavily_core.py # Tavily 注册入口
|
||||
├── tavily_browser_solver.py # Tavily 浏览器注册主逻辑
|
||||
├── firecrawl_core.py # Firecrawl 注册入口
|
||||
├── firecrawl_browser_solver.py # Firecrawl 浏览器注册主逻辑
|
||||
├── api_solver.py # 本地 Turnstile Solver（Tavily 专用）
|
||||
├── mail_provider.py # 邮箱 provider 抽象
|
||||
├── config.py # .env / 环境变量读取
|
||||
├── start_auto.sh # macOS / Linux 启动脚本
|
||||
├── start_auto.bat # Windows 启动脚本
|
||||
├── proxy/ # 可选的多服务代理（Tavily / Firecrawl）
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Module Notes
|
||||
|
||||
仓库里有一些不是主入口、但仍然是运行时依赖的模块：
|
||||
|
||||
- `tavily_core.py`
|
||||
现在是兼容层，负责把统一入口转发到浏览器注册主链路。
|
||||
|
||||
- `browser_configs.py`
|
||||
`api_solver.py` 的浏览器配置辅助模块。
|
||||
|
||||
- `db_results.py`
|
||||
`api_solver.py` 的结果存储辅助模块。
|
||||
|
||||
- `proxy/`
|
||||
独立可选模块，用于把 Tavily / Firecrawl 分别做成统一代理池。
|
||||
|
||||
## Optional Proxy Service
|
||||
|
||||
如果你希望把注册出来的 key 接成统一池子，可以使用 `proxy/`。
|
||||
现在它已经支持 Tavily 和 Firecrawl 两套独立池子、独立 Token 与独立额度同步。
|
||||
|
||||
启动方式：
|
||||
|
||||
```bash
|
||||
cd proxy
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
详细说明见 [`proxy/README.md`](./proxy/README.md)。
|
||||
|
||||
## Recommended Usage
|
||||
|
||||
如果你只是想批量拿 key，最简单的使用方式就是：
|
||||
|
||||
1. 配好 `.env`
|
||||
2. 运行 `python3 run.py`
|
||||
3. 选择服务（Tavily / Firecrawl）
|
||||
4. 选择域名
|
||||
5. 输入注册数量
|
||||
6. 输入并发数
|
||||
7. 看 `accounts.txt` 或 `firecrawl_accounts.txt`
|
||||
|
||||
如果你有自己的 key 池服务器，再把自动上传或者 `proxy/` 接上即可。
|
||||
|
||||
## Disclaimer
|
||||
|
||||
本项目仅供自动化测试、研究和个人学习使用。
|
||||
|
||||
请自行评估目标站点的服务条款、风控策略和账号使用风险。
|
||||
- [MySearch-Proxy](https://github.com/skernelx/MySearch-Proxy)
|
||||
+47
-300
@@ -2,29 +2,26 @@
|
||||
|
||||
[中文说明](./README.md)
|
||||
|
||||
This is the current production-ready multi-service registration toolkit.
|
||||
It puts **Tavily** and **Firecrawl** behind the same launcher:
|
||||
This repository is the multi-service registration toolkit and aggregation API
|
||||
upstream for:
|
||||
|
||||
- **Tavily**: AI search API
|
||||
- **Firecrawl**: web scraping API
|
||||
- registering `Tavily` / `Firecrawl` keys
|
||||
- validating that the keys actually work before feeding a unified search layer
|
||||
|
||||
Unlike the older fragmented scripts, the current workflow is built around a
|
||||
single stable pipeline:
|
||||
It is not the main product repository for `MySearch`.
|
||||
If you want the public-facing unified search console, MCP, Skill, and Social / X
|
||||
integration, use the standalone project:
|
||||
|
||||
- Real local browser registration
|
||||
- Local Turnstile solver for Tavily
|
||||
- Email API based verification-link / code retrieval
|
||||
- Real API validation immediately after a key is extracted
|
||||
- Optional automatic upload to the multi-service proxy pool
|
||||
- [MySearch-Proxy](https://github.com/skernelx/MySearch-Proxy)
|
||||
|
||||
The goal is straightforward: keep Tavily / Firecrawl registration, validation,
|
||||
output, and upload inside one workflow that can actually be reused long term.
|
||||
## What this repo is for
|
||||
|
||||
For a general Cloudflare unlimited-alias domain mail guide, see:
|
||||
[Cloudflare Mail Setup Guide](./docs/Cloudflare-Mail-Setup-Guide.md)
|
||||
|
||||
For the Chinese version of that guide, see:
|
||||
[Cloudflare Mail Setup Guide (Chinese)](./docs/Cloudflare%E9%82%AE%E4%BB%B6%E8%AE%BE%E7%BD%AE%E8%AF%A6%E8%A7%A3.md)
|
||||
- real local browser-based registration
|
||||
- local Turnstile solving for Tavily
|
||||
- email API based verification link / code retrieval
|
||||
- immediate live API validation after a key is extracted
|
||||
- optional upload into unified proxy pools
|
||||
- optional use as one upstream provider source for `MySearch` / `MySearch-Proxy`
|
||||
|
||||
## Features
|
||||
|
||||
@@ -35,28 +32,19 @@ For the Chinese version of that guide, see:
|
||||
- Concurrent registration: batch and parallel runs are supported
|
||||
- Background browser mode: headless by default, visible mode when debugging
|
||||
- Real usability verification: validates the key against the official API
|
||||
- Automatic upload to proxy pools: uploads include the service field, so the
|
||||
server can place Tavily and Firecrawl keys into the correct pools
|
||||
- Proxy console: separate key pools, token pools, and real quota sync
|
||||
- Automatic upload to proxy pools
|
||||
- A clean upstream for unified search gateways such as `MySearch-Proxy`
|
||||
- Cross-platform startup: Windows, macOS, and Linux
|
||||
|
||||
## Screenshots
|
||||
## Paired Console Preview
|
||||
|
||||
### Multi-Service Launcher
|
||||
### MySearch Proxy Hero
|
||||
|
||||
|
||||
|
||||
|
||||
### Concurrent Registration and Real API Validation
|
||||
### MySearch Proxy Workspaces
|
||||
|
||||
|
||||
|
||||
### Proxy Workspace Switcher
|
||||
|
||||
|
||||
|
||||
### Proxy Key Pool Details
|
||||
|
||||
|
||||
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -73,7 +61,7 @@ cd tavily-key-generator
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Edit `.env` and fill in your mail configuration and optional upload settings.
|
||||
Edit `.env` and fill in your mail settings and optional upload settings.
|
||||
|
||||
### 3. Run
|
||||
|
||||
@@ -95,280 +83,39 @@ Windows:
|
||||
start_auto.bat
|
||||
```
|
||||
|
||||
## How It Works
|
||||
## Recommended Pairing
|
||||
|
||||
When you run the launcher, it will automatically:
|
||||
If you want to turn the generated keys into a public, reusable unified search
|
||||
service, the recommended deployment target is:
|
||||
|
||||
1. Choose the target service: Tavily or Firecrawl
|
||||
2. Create or reuse `venv`
|
||||
3. Install Python dependencies
|
||||
4. Install Camoufox / Playwright browser dependencies
|
||||
5. Load `.env`
|
||||
6. Validate the configured mail provider
|
||||
7. Let you choose the active domain if multiple domains are configured
|
||||
8. Ask for the registration count
|
||||
9. Ask for the concurrency level
|
||||
10. Ask whether to upload results automatically
|
||||
11. Start the local solver for Tavily
|
||||
12. Handle email verification and password setup
|
||||
13. Recover from random challenge cases on the Tavily password page
|
||||
14. Extract the API key
|
||||
15. Validate the key with a real API request
|
||||
16. Save results to `accounts.txt` or `firecrawl_accounts.txt`
|
||||
17. Upload results to the server when enabled
|
||||
- [MySearch-Proxy](https://github.com/skernelx/MySearch-Proxy)
|
||||
|
||||
## Runtime Flow
|
||||
That project provides:
|
||||
|
||||
```text
|
||||
run.py
|
||||
-> choose service (Tavily / Firecrawl)
|
||||
-> load .env
|
||||
-> choose domain
|
||||
-> input count / concurrency
|
||||
-> choose upload or not
|
||||
-> [Tavily only] start Turnstile solver
|
||||
-> create mailbox
|
||||
-> open signup page
|
||||
-> [Tavily only] solve Turnstile locally
|
||||
-> receive email verification link
|
||||
-> set password
|
||||
-> [Tavily only] recover random password-page challenge
|
||||
-> enter dashboard
|
||||
-> extract API key
|
||||
-> verify API key with real API call
|
||||
-> save / upload
|
||||
```
|
||||
- a unified console
|
||||
- Tavily / Firecrawl / Social / X routing
|
||||
- MCP + Skill installation docs
|
||||
- official API and compatible gateway support
|
||||
- final integration guidance for Codex / Claude Code
|
||||
|
||||
## Configuration
|
||||
This repository stays focused on:
|
||||
|
||||
See [`.env.example`](./.env.example) for the full configuration template.
|
||||
- producing working keys
|
||||
- verifying them with live requests
|
||||
- serving as the upstream aggregation source
|
||||
|
||||
### Cloudflare Mail API
|
||||
The local `proxy/` directory is now kept only for legacy deployments or
|
||||
migration scenarios, not as the primary recommended path.
|
||||
|
||||
```env
|
||||
EMAIL_PROVIDER=cloudflare
|
||||
EMAIL_API_URL=https://your-mail-api.example.com
|
||||
EMAIL_API_TOKEN=replace-with-your-token
|
||||
EMAIL_DOMAIN=example.com
|
||||
EMAIL_DOMAINS=example.com,example.org
|
||||
```
|
||||
## Relationship to MySearch
|
||||
|
||||
Notes:
|
||||
This repo only handles:
|
||||
|
||||
- Use `EMAIL_DOMAIN` for a single domain
|
||||
- Use `EMAIL_DOMAINS` for multiple domains
|
||||
- The launcher lets you choose the active domain at runtime
|
||||
- Tavily / Firecrawl key registration
|
||||
- live key validation
|
||||
- optional aggregation API serving for `MySearch` / `MySearch-Proxy`
|
||||
|
||||
### DuckMail API
|
||||
All `MySearch` installation, MCP usage, unified console, social gateway, and
|
||||
Codex / Claude Code integration docs now live in:
|
||||
|
||||
```env
|
||||
EMAIL_PROVIDER=duckmail
|
||||
DUCKMAIL_API_URL=https://api.duckmail.sbs
|
||||
DUCKMAIL_API_KEY=
|
||||
DUCKMAIL_DOMAIN=
|
||||
DUCKMAIL_DOMAINS=
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- You can configure either one domain or multiple domains
|
||||
- If you have a private DuckMail domain and API key, just put them in `.env`
|
||||
- Public DuckMail domains can be used to test the mail path, but they are not
|
||||
guaranteed to pass Tavily risk control
|
||||
|
||||
### Upload to Your Server
|
||||
|
||||
```env
|
||||
SERVER_URL=https://your-server.example.com
|
||||
SERVER_ADMIN_PASSWORD=replace-with-your-admin-password
|
||||
DEFAULT_UPLOAD=true
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `DEFAULT_UPLOAD=true` makes upload enabled by default in the launcher
|
||||
- The actual upload decision still depends on the runtime choice you make
|
||||
- Upload payloads include the `service` field
|
||||
- Tavily keys are automatically placed into the Tavily pool
|
||||
- Firecrawl keys are automatically placed into the Firecrawl pool
|
||||
|
||||
### Runtime Options
|
||||
|
||||
```env
|
||||
DEFAULT_COUNT=1
|
||||
DEFAULT_CONCURRENCY=2
|
||||
DEFAULT_DELAY=10
|
||||
REGISTER_HEADLESS=true
|
||||
FIRECRAWL_REGISTER_HEADLESS=true
|
||||
EMAIL_CODE_TIMEOUT=90
|
||||
API_KEY_TIMEOUT=20
|
||||
EMAIL_POLL_INTERVAL=3
|
||||
SOLVER_PORT=5073
|
||||
SOLVER_THREADS=1
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `REGISTER_HEADLESS=true` keeps the browser in the background
|
||||
- `FIRECRAWL_REGISTER_HEADLESS` inherits `REGISTER_HEADLESS` if it is not set
|
||||
- If you hit `Security check failed`, temporarily switch headless off and debug
|
||||
with a visible browser
|
||||
- The effective solver thread count becomes `max(SOLVER_THREADS, concurrency)`
|
||||
- In normal use, no extra command-line flags are needed
|
||||
|
||||
## Output
|
||||
|
||||
Successful registrations are written to:
|
||||
|
||||
**Tavily**:
|
||||
|
||||
```text
|
||||
accounts.txt
|
||||
```
|
||||
|
||||
**Firecrawl**:
|
||||
|
||||
```text
|
||||
firecrawl_accounts.txt
|
||||
```
|
||||
|
||||
Format:
|
||||
|
||||
```text
|
||||
email,password,api_key
|
||||
email,password,api_key
|
||||
```
|
||||
|
||||
## Real-World Validation
|
||||
|
||||
The current mainline has been validated in real runs.
|
||||
|
||||
**Tavily**:
|
||||
|
||||
- Full registration works with the Cloudflare mail flow
|
||||
- Email verification codes can be fetched automatically
|
||||
- Extracted API keys are immediately validated with real API requests
|
||||
- Concurrent registration has been regression tested
|
||||
- Random password-page challenge recovery has been added and verified
|
||||
|
||||
**Firecrawl**:
|
||||
|
||||
- Email verification links can be fetched automatically
|
||||
- Automatic sign-in and API key extraction are supported
|
||||
- API key validation is done with real Firecrawl API requests
|
||||
- Uploads to the proxy server are auto-tagged with `service=firecrawl`
|
||||
|
||||
## Known Limitations
|
||||
|
||||
### DuckMail Public Domains
|
||||
|
||||
The current status of public DuckMail domains is:
|
||||
|
||||
- Mailbox creation works
|
||||
- 6-digit verification-code retrieval works
|
||||
- Tavily may still be blocked on the password page
|
||||
|
||||
Common page message:
|
||||
|
||||
```text
|
||||
Suspicious activity detected
|
||||
```
|
||||
|
||||
If you want reliable full registration, prefer:
|
||||
|
||||
- Cloudflare custom-domain mail
|
||||
- DuckMail private domain plus API key
|
||||
|
||||
### First Run on a New Machine
|
||||
|
||||
On a new machine, it is better to run one account first before enabling
|
||||
concurrency.
|
||||
|
||||
The first run may need to download browser dependencies, and system, network,
|
||||
or proxy conditions can differ across machines.
|
||||
|
||||
### System-Level Prerequisites
|
||||
|
||||
The launcher can bootstrap project dependencies and browser dependencies
|
||||
automatically, but it does not install Python itself.
|
||||
|
||||
At minimum, the target machine should already have:
|
||||
|
||||
- Python 3
|
||||
- `venv` support
|
||||
- A usable network environment for installing dependencies
|
||||
|
||||
## Project Structure
|
||||
|
||||
```text
|
||||
.
|
||||
├── run.py # Recommended entry point
|
||||
├── tavily_core.py # Tavily registration entry
|
||||
├── tavily_browser_solver.py # Tavily browser registration flow
|
||||
├── firecrawl_core.py # Firecrawl registration entry
|
||||
├── firecrawl_browser_solver.py # Firecrawl browser registration flow
|
||||
├── api_solver.py # Local Turnstile solver (Tavily only)
|
||||
├── mail_provider.py # Mail provider abstraction
|
||||
├── config.py # .env / environment loading
|
||||
├── start_auto.sh # macOS / Linux launcher
|
||||
├── start_auto.bat # Windows launcher
|
||||
├── proxy/ # Optional multi-service proxy
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Module Notes
|
||||
|
||||
Some files are not primary entry points, but they are still part of the
|
||||
runtime:
|
||||
|
||||
- `tavily_core.py`
|
||||
Compatibility layer that forwards the unified entry to the Tavily browser
|
||||
flow.
|
||||
|
||||
- `browser_configs.py`
|
||||
Browser configuration helper used by `api_solver.py`.
|
||||
|
||||
- `db_results.py`
|
||||
Result-storage helper used by `api_solver.py`.
|
||||
|
||||
- `proxy/`
|
||||
Optional standalone module for turning Tavily / Firecrawl keys into separate
|
||||
pooled proxy services.
|
||||
|
||||
## Optional Proxy Service
|
||||
|
||||
If you want to route registered keys behind one stable endpoint, use `proxy/`.
|
||||
It now exposes independent Tavily and Firecrawl pools, tokens, and quota sync.
|
||||
|
||||
Start it with:
|
||||
|
||||
```bash
|
||||
cd proxy
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
See [`proxy/README.md`](./proxy/README.md) for details.
|
||||
|
||||
## Recommended Usage
|
||||
|
||||
If your goal is simply to batch-register and collect keys, the shortest path
|
||||
is:
|
||||
|
||||
1. Configure `.env`
|
||||
2. Run `python3 run.py`
|
||||
3. Choose the service: Tavily or Firecrawl
|
||||
4. Choose the mail domain
|
||||
5. Enter the registration count
|
||||
6. Enter the concurrency level
|
||||
7. Read the results from `accounts.txt` or `firecrawl_accounts.txt`
|
||||
|
||||
If you also need centralized distribution, enable server upload or connect the
|
||||
generated keys to `proxy/`.
|
||||
|
||||
## Disclaimer
|
||||
|
||||
This project is provided for automation testing, research, and personal
|
||||
learning purposes only.
|
||||
|
||||
You are responsible for evaluating the target service terms, risk controls,
|
||||
and account-usage implications.
|
||||
- [MySearch-Proxy](https://github.com/skernelx/MySearch-Proxy)
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 636 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 632 KiB |
+79
-248
@@ -1,63 +1,43 @@
|
||||
# Multi-Service API Proxy
|
||||
# Legacy Proxy Snapshot
|
||||
|
||||
把多个 **Tavily** 和 **Firecrawl** API Key 做成统一代理池，对外暴露固定入口、独立 Token，并提供一个更适合日常维护的可视化控制台。
|
||||
这个 `proxy/` 目录现在是保留在 `tavily-key-generator` 里的历史兼容副本。
|
||||
|
||||
当前这版 `proxy/` 已支持：
|
||||
如果你要部署正式对外可用的统一搜索控制台、MCP、Skill 和 Social / X
|
||||
路由，请直接使用独立仓库：
|
||||
|
||||
- Tavily / Firecrawl **分服务隔离**
|
||||
- 独立 Key 池、独立 Token 池
|
||||
- 顶部卡片切换 + 单服务详情面板
|
||||
- Tavily 真实额度同步：`GET /usage`
|
||||
- Firecrawl 真实额度同步：
|
||||
- `GET /v2/team/credit-usage`
|
||||
- `GET /v2/team/credit-usage/historical?byApiKey=true`
|
||||
- 本地代理调用统计、成功率、延迟统计
|
||||
- [MySearch-Proxy](https://github.com/skernelx/MySearch-Proxy)
|
||||
|
||||
## 控制台截图
|
||||
## 当前推荐的正式入口
|
||||
|
||||
### 工作台切换
|
||||
`MySearch-Proxy` 负责：
|
||||
|
||||
|
||||
- 统一控制台
|
||||
- `Tavily + Firecrawl + Social / X` 聚合路由
|
||||
- MCP Server
|
||||
- 可直接安装给智能体使用的 Skill
|
||||
- 官方接口与 compatible 网关双接入
|
||||
- 面向 `Codex` / `Claude Code` 的最终接线方式
|
||||
|
||||
### Key 池详情
|
||||
## 控制台预览
|
||||
|
||||
|
||||
### 首屏
|
||||
|
||||
## Docker Image
|
||||
|
||||
|
||||
当前兼容镜像名仍沿用旧名字：
|
||||
### 工作台
|
||||
|
||||
```text
|
||||
docker.io/skernelx/tavily-proxy:latest
|
||||
```
|
||||
|
||||
|
||||
拉取方式：
|
||||
## 和 `tavily-key-generator` 的关系
|
||||
|
||||
```bash
|
||||
docker pull skernelx/tavily-proxy:latest
|
||||
```
|
||||
当前仓库继续负责上游 provider 层：
|
||||
|
||||
## 功能概览
|
||||
- 注册 Tavily / Firecrawl key
|
||||
- 做真实可用性验证
|
||||
- 可选上传到统一代理池
|
||||
- 作为 `MySearch-Proxy` 的 Tavily / Firecrawl key 来源
|
||||
|
||||
- Tavily 独立代理入口：
|
||||
- `POST /api/search`
|
||||
- `POST /api/extract`
|
||||
- Firecrawl 独立代理入口：
|
||||
- `/firecrawl/*`
|
||||
- 示例：`POST /firecrawl/v2/scrape`
|
||||
- 独立代理 Token：
|
||||
- Tavily Token 前缀：`tvly-`
|
||||
- Firecrawl Token 前缀：`fctk-`
|
||||
- 控制台支持：
|
||||
- 按服务导入 Key
|
||||
- 按服务创建 Token
|
||||
- 按服务同步真实额度
|
||||
- 按服务查看 Key 池与用量
|
||||
- 顶部一键切换 Tavily / Firecrawl 工作台
|
||||
|
||||
## 和注册器如何联动
|
||||
|
||||
注册器上传到 proxy 时，会直接调用：
|
||||
注册器上传时仍会带上 `service` 字段，例如：
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -67,226 +47,77 @@ docker pull skernelx/tavily-proxy:latest
|
||||
}
|
||||
```
|
||||
|
||||
也就是说：
|
||||
所以：
|
||||
|
||||
- Tavily 注册结果上传时会写入 Tavily 池
|
||||
- Firecrawl 注册结果上传时会写入 Firecrawl 池
|
||||
- 服务器不需要再靠 key 前缀猜测服务
|
||||
- Tavily 上传会进入 Tavily 池
|
||||
- Firecrawl 上传会进入 Firecrawl 池
|
||||
- 服务端不需要再靠 key 前缀猜测服务
|
||||
|
||||
这条链路已经做过真实验证：Firecrawl 上传后会被 `/api/keys`
|
||||
识别成 `service=firecrawl`，不会落到 Tavily 池里。
|
||||
## 新部署建议
|
||||
|
||||
## 推荐部署方式
|
||||
|
||||
### 1. 直接使用 Docker Hub 镜像
|
||||
推荐直接部署 `MySearch-Proxy`：
|
||||
|
||||
```bash
|
||||
mkdir -p tavily-proxy-data
|
||||
docker pull skernelx/mysearch-proxy:latest
|
||||
```
|
||||
|
||||
项目地址：
|
||||
|
||||
- [skernelx/MySearch-Proxy](https://github.com/skernelx/MySearch-Proxy)
|
||||
|
||||
## 旧部署如何迁移
|
||||
|
||||
如果你已经在跑这个目录下的旧 `proxy/`，更稳的迁移方式是：
|
||||
|
||||
1. 保留原来的数据卷目录
|
||||
2. 改用 `skernelx/mysearch-proxy:latest`
|
||||
3. 继续挂载到 `/app/data`
|
||||
4. 使用新的 `proxy/.env` 或容器环境变量补齐配置
|
||||
|
||||
一个通用迁移示例：
|
||||
|
||||
```bash
|
||||
docker pull skernelx/mysearch-proxy:latest
|
||||
|
||||
docker rm -f mysearch-proxy 2>/dev/null || true
|
||||
|
||||
docker run -d \
|
||||
--name tavily-proxy \
|
||||
--name mysearch-proxy \
|
||||
--restart unless-stopped \
|
||||
-p 9874:9874 \
|
||||
-e ADMIN_PASSWORD=your-admin-password \
|
||||
-v $(pwd)/tavily-proxy-data:/app/data \
|
||||
skernelx/tavily-proxy:latest
|
||||
-v /your/data/path:/app/data \
|
||||
skernelx/mysearch-proxy:latest
|
||||
```
|
||||
|
||||
启动后访问：
|
||||
只要保留原数据卷，已有 Key、Token 和控制台里的数据就可以继续沿用。
|
||||
|
||||
```text
|
||||
http://localhost:9874
|
||||
```
|
||||
## 兼容 API 摘要
|
||||
|
||||
### 2. 使用 docker compose
|
||||
当前这套能力对外仍然兼容这些重点接口：
|
||||
|
||||
- Tavily
|
||||
- `POST /api/search`
|
||||
- `POST /api/extract`
|
||||
- Firecrawl
|
||||
- `/firecrawl/*`
|
||||
- 示例：`POST /firecrawl/v2/scrape`
|
||||
- Social / X
|
||||
- `POST /social/search`
|
||||
- `GET /social/health`
|
||||
|
||||
认证方式支持：
|
||||
|
||||
- `Authorization: Bearer YOUR_TOKEN`
|
||||
- body 里传 `api_key`
|
||||
|
||||
## 如果你只是要保留本地旧目录
|
||||
|
||||
这个目录还可以继续作为历史兼容副本使用：
|
||||
|
||||
```bash
|
||||
cd proxy
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
默认 compose 会：
|
||||
|
||||
- 暴露端口 `9874`
|
||||
- 将数据库挂载到 `./data`
|
||||
- 使用 `ADMIN_PASSWORD` 作为控制台密码
|
||||
|
||||
### 3. 本地源码运行
|
||||
|
||||
```bash
|
||||
cd proxy
|
||||
pip install -r requirements.txt
|
||||
ADMIN_PASSWORD=your-admin-password uvicorn server:app --host 0.0.0.0 --port 9874
|
||||
```
|
||||
|
||||
## 更新方式
|
||||
|
||||
如果你已经在服务器上跑了旧版本，推荐直接拉新镜像后重启容器：
|
||||
|
||||
```bash
|
||||
docker pull skernelx/tavily-proxy:latest
|
||||
|
||||
docker rm -f tavily-proxy
|
||||
|
||||
docker run -d \
|
||||
--name tavily-proxy \
|
||||
--restart unless-stopped \
|
||||
-p 9874:9874 \
|
||||
-e ADMIN_PASSWORD=your-admin-password \
|
||||
-v /your/data/path:/app/data \
|
||||
skernelx/tavily-proxy:latest
|
||||
```
|
||||
|
||||
只要保留原来的数据卷目录，Key、Token 和控制台密码都会继续保留。旧库会自动迁移出 `service` 字段，历史 Tavily 数据会被标记为 `tavily`。
|
||||
|
||||
## 控制台里能看到什么
|
||||
|
||||
控制台现在不是上下两个长栏目硬堆，而是：
|
||||
|
||||
- 顶部服务卡片切换区
|
||||
- 首屏当前工作台概览
|
||||
- 下方当前服务的完整详情面板
|
||||
|
||||
每个服务面板里都能看到：
|
||||
|
||||
### 1. Tavily 栏目
|
||||
|
||||
- Key 池
|
||||
- Token 池
|
||||
- 真实额度汇总
|
||||
- Tavily `/usage` 同步状态
|
||||
- 代理侧成功 / 失败 / 月度统计
|
||||
|
||||
### 2. Firecrawl 栏目
|
||||
|
||||
- Key 池
|
||||
- Token 池
|
||||
- Firecrawl credits 汇总
|
||||
- Firecrawl credits 同步状态
|
||||
- 代理侧成功 / 失败 / 月度统计
|
||||
|
||||
## 使用流程
|
||||
|
||||
1. 启动 proxy
|
||||
2. 打开控制台并登录
|
||||
3. 在 Tavily 或 Firecrawl 栏目导入对应 Key
|
||||
4. 在对应栏目创建 Token
|
||||
5. 把该 Token 发给你的下游程序
|
||||
6. 用对应的代理端点发起请求
|
||||
7. 在控制台查看该服务的真实额度和代理统计
|
||||
|
||||
## API 调用方式
|
||||
|
||||
认证方式支持两种：
|
||||
|
||||
- `Authorization: Bearer YOUR_TOKEN`
|
||||
- body 里传 `api_key`
|
||||
|
||||
### Tavily 示例
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:9874/api/search \
|
||||
-H "Authorization: Bearer YOUR_TAVILY_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "hello world", "max_results": 1}'
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:9874/api/extract \
|
||||
-H "Authorization: Bearer YOUR_TAVILY_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"urls": ["https://example.com"]}'
|
||||
```
|
||||
|
||||
### Firecrawl 示例
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:9874/firecrawl/v2/scrape \
|
||||
-H "Authorization: Bearer YOUR_FIRECRAWL_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"url": "https://example.com", "formats": ["markdown"]}'
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X GET http://localhost:9874/firecrawl/v2/team/credit-usage \
|
||||
-H "Authorization: Bearer YOUR_FIRECRAWL_TOKEN"
|
||||
```
|
||||
|
||||
## 管理 API
|
||||
|
||||
所有管理 API 需要：
|
||||
|
||||
- `X-Admin-Password: your-admin-password`
|
||||
|
||||
或者：
|
||||
|
||||
- `Authorization: Bearer your-admin-password`
|
||||
|
||||
### 常用管理端点
|
||||
|
||||
- `GET /api/stats`
|
||||
返回 Tavily / Firecrawl 双服务概览
|
||||
|
||||
- `GET /api/keys?service=tavily`
|
||||
- `GET /api/keys?service=firecrawl`
|
||||
返回指定服务的 Key 列表
|
||||
|
||||
- `POST /api/keys`
|
||||
body 里传 `service`
|
||||
|
||||
- `PUT /api/keys/{id}/toggle`
|
||||
启用 / 禁用某个 Key
|
||||
|
||||
- `DELETE /api/keys/{id}`
|
||||
删除 Key
|
||||
|
||||
- `GET /api/tokens?service=tavily`
|
||||
- `GET /api/tokens?service=firecrawl`
|
||||
返回指定服务的 Token 列表
|
||||
|
||||
- `POST /api/tokens`
|
||||
body 里传 `service`
|
||||
|
||||
- `DELETE /api/tokens/{id}`
|
||||
删除 Token
|
||||
|
||||
- `POST /api/usage/sync`
|
||||
body 里传 `service`
|
||||
|
||||
- `PUT /api/password`
|
||||
修改控制台密码
|
||||
|
||||
## 配置项
|
||||
|
||||
| 环境变量 | 默认值 | 说明 |
|
||||
|----------|--------|------|
|
||||
| `ADMIN_PASSWORD` | `admin` | 控制台登录密码 |
|
||||
| `USAGE_SYNC_TTL_SECONDS` | `300` | 真实额度缓存秒数 |
|
||||
| `USAGE_SYNC_CONCURRENCY` | `4` | 并发同步额度的最大 Key 数 |
|
||||
|
||||
## 数据持久化
|
||||
|
||||
SQLite 数据库默认保存在：
|
||||
|
||||
```text
|
||||
/app/data/proxy.db
|
||||
```
|
||||
|
||||
所以容器部署时一定要挂载数据卷，例如：
|
||||
|
||||
```bash
|
||||
-v /your/data/path:/app/data
|
||||
```
|
||||
|
||||
## 适合什么场景
|
||||
|
||||
- 你有多组 Tavily / Firecrawl Key，想统一出口
|
||||
- 你不想把真实 API Key 直接发给下游程序
|
||||
- 你希望 Tavily 和 Firecrawl 各自独立统计、独立授权
|
||||
- 你想让注册器自动把新拿到的 Key 上传进对应池子
|
||||
|
||||
## 注意事项
|
||||
|
||||
- Tavily 真实额度依赖官方 `/usage`
|
||||
- Firecrawl 当前主要返回账户级 credits 视图，控制台会优先展示账户额度
|
||||
- 旧数据库会自动迁移 `service` 字段，但旧数据默认视为 Tavily
|
||||
但它不再是对外发布时的主推荐入口。
|
||||
Reference in new issue
Block a user