Add Exa registration support and update README status
This commit is contained in:
1 parent
715af4bb4b
commit
efd14c6892
7 files changed
+375
-26
No files matched your search
@@ -11,6 +11,7 @@ ENV/
|
||||
# 账号信息
|
||||
accounts.txt
|
||||
firecrawl_accounts.txt
|
||||
exa_accounts.txt
|
||||
accounts.csv
|
||||
.env
|
||||
.env.local
|
||||
|
||||
@@ -4,9 +4,15 @@
|
||||
|
||||
这是当前的多服务注册器与聚合 API 上游工具,聚焦两件事:
|
||||
|
||||
- 注册 `Tavily` / `Firecrawl` key
|
||||
- 注册 `Firecrawl` / `Exa` key
|
||||
- 验证 key 是否真实可用,并把可用 key 提供给统一搜索层
|
||||
|
||||
## 当前状态
|
||||
|
||||
- `Firecrawl`:可用
|
||||
- `Exa`:可用
|
||||
- `Tavily`:按当前项目这轮本地测试,官网已关闭邮件注册入口,暂不可用
|
||||
|
||||
它不是 `MySearch` 的正式产品仓库。
|
||||
如果你要部署公开可用的统一搜索控制台、MCP、Skill 和 Social / X
|
||||
接入,请直接使用独立项目:
|
||||
@@ -24,14 +30,14 @@
|
||||
|
||||
## Features
|
||||
|
||||
- 多服务启动台:启动时直接选择 Tavily 或 Firecrawl
|
||||
- 多服务启动台:启动时可选择 Tavily / Firecrawl / Exa
|
||||
- 自动环境准备:自动检查 `venv`、依赖和浏览器
|
||||
- 邮箱链路统一:支持 Cloudflare Mail API 和 DuckMail
|
||||
- 多域名可选:启动时选择本轮注册用的域名
|
||||
- 并发注册:支持批量和并发执行
|
||||
- 后台浏览器模式:默认 headless,必要时可切前台排查
|
||||
- 真实可用性验证:拿到 key 后马上调用官方接口
|
||||
- 自动上传到代理池:上传时会带上服务标记,服务端能自动识别 Tavily / Firecrawl 并写入各自池子
|
||||
- 自动上传到代理池:上传时会带上服务标记,服务端能自动识别 Firecrawl / Exa / Tavily 并写入对应池子
|
||||
- 适合作为统一搜索网关的上游:给 `MySearch-Proxy` 提供稳定的 key 来源
|
||||
- 跨平台启动:Windows / macOS / Linux 都能直接跑
|
||||
|
||||
@@ -62,6 +68,8 @@ cp .env.example .env
|
||||
|
||||
编辑 `.env`,填好邮箱配置和可选上传配置。
|
||||
|
||||
如果已经配置了 `SERVER_URL` 和 `SERVER_ADMIN_PASSWORD`,注册成功后可以自动上传。
|
||||
|
||||
### 3. Run
|
||||
|
||||
macOS / Linux:
|
||||
@@ -82,6 +90,20 @@ Windows:
|
||||
start_auto.bat
|
||||
```
|
||||
|
||||
启动后当前可选:
|
||||
|
||||
```text
|
||||
1. Tavily
|
||||
2. Firecrawl
|
||||
3. Exa
|
||||
```
|
||||
|
||||
补充说明:
|
||||
|
||||
- `Exa` 注册成功后会保存到 `exa_accounts.txt`
|
||||
- `Exa` 的格式是 `email,EMAIL_OTP_ONLY,api_key`
|
||||
- `Tavily` 选项目前仍保留在启动台里,但当前状态暂不可用
|
||||
|
||||
## 推荐搭配方式
|
||||
|
||||
如果你希望把注册出来的 key 接成公开可用的统一搜索服务,推荐直接搭配:
|
||||
@@ -108,7 +130,7 @@ start_auto.bat
|
||||
|
||||
这里的 `tavily-key-generator` 只负责:
|
||||
|
||||
- 注册 Tavily / Firecrawl key
|
||||
- 注册 Firecrawl / Exa key
|
||||
- 验证 key 是否真实可用
|
||||
- 提供可选聚合 API,供 `MySearch` / `MySearch-Proxy` 调用
|
||||
|
||||
|
||||
+26
-4
@@ -5,9 +5,15 @@
|
||||
This repository is the multi-service registration toolkit and aggregation API
|
||||
upstream for:
|
||||
|
||||
- registering `Tavily` / `Firecrawl` keys
|
||||
- registering `Firecrawl` / `Exa` keys
|
||||
- validating that the keys actually work before feeding a unified search layer
|
||||
|
||||
## Current Status
|
||||
|
||||
- `Firecrawl`: available
|
||||
- `Exa`: available
|
||||
- `Tavily`: based on the current local project testing, the official email registration entry is closed, so it is temporarily unavailable
|
||||
|
||||
It is not the main product repository for `MySearch`.
|
||||
If you want the public-facing unified search console, MCP, Skill, and Social / X
|
||||
integration, use the standalone project:
|
||||
@@ -25,14 +31,14 @@ integration, use the standalone project:
|
||||
|
||||
## Features
|
||||
|
||||
- Multi-service launcher: choose Tavily or Firecrawl at startup
|
||||
- Multi-service launcher: choose Tavily / Firecrawl / Exa at startup
|
||||
- Automatic environment bootstrap: checks `venv`, dependencies, and browsers
|
||||
- Unified mail layer: supports Cloudflare Mail API and DuckMail
|
||||
- Multi-domain support: choose the active domain at runtime
|
||||
- Concurrent registration: batch and parallel runs are supported
|
||||
- Background browser mode: headless by default, visible mode when debugging
|
||||
- Real usability verification: validates the key against the official API
|
||||
- Automatic upload to proxy pools
|
||||
- Automatic upload to proxy pools with a per-service marker
|
||||
- A clean upstream for unified search gateways such as `MySearch-Proxy`
|
||||
- Cross-platform startup: Windows, macOS, and Linux
|
||||
|
||||
@@ -63,6 +69,8 @@ cp .env.example .env
|
||||
|
||||
Edit `.env` and fill in your mail settings and optional upload settings.
|
||||
|
||||
If `SERVER_URL` and `SERVER_ADMIN_PASSWORD` are configured, successful registrations can be uploaded automatically.
|
||||
|
||||
### 3. Run
|
||||
|
||||
macOS / Linux:
|
||||
@@ -83,6 +91,20 @@ Windows:
|
||||
start_auto.bat
|
||||
```
|
||||
|
||||
Current launcher options:
|
||||
|
||||
```text
|
||||
1. Tavily
|
||||
2. Firecrawl
|
||||
3. Exa
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `Exa` accounts are saved to `exa_accounts.txt`
|
||||
- the saved format is `email,EMAIL_OTP_ONLY,api_key`
|
||||
- the `Tavily` option is still shown in the launcher, but it is currently unavailable based on local testing
|
||||
|
||||
## Recommended Pairing
|
||||
|
||||
If you want to turn the generated keys into a public, reusable unified search
|
||||
@@ -111,7 +133,7 @@ ships the runnable implementation.
|
||||
|
||||
This repo only handles:
|
||||
|
||||
- Tavily / Firecrawl key registration
|
||||
- Firecrawl / Exa key registration
|
||||
- live key validation
|
||||
- optional aggregation API serving for `MySearch` / `MySearch-Proxy`
|
||||
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
"""
|
||||
使用 Camoufox 完成 Exa 注册
|
||||
思路:通过邮箱验证码登录,跳过 onboarding,并提取默认 API Key
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
|
||||
import requests as std_requests
|
||||
from camoufox.sync_api import Camoufox
|
||||
|
||||
from config import API_KEY_TIMEOUT, EMAIL_CODE_TIMEOUT, REGISTER_HEADLESS
|
||||
from mail_provider import get_email_code
|
||||
|
||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
_SAVE_FILE = os.path.join(_HERE, "exa_accounts.txt")
|
||||
_SAVE_LOCK = threading.Lock()
|
||||
_ACCOUNT_PASSWORD_LABEL = "EMAIL_OTP_ONLY"
|
||||
_EXA_AUTH_URL = "https://auth.exa.ai/?callbackUrl=https%3A%2F%2Fdashboard.exa.ai%2F"
|
||||
_EXA_HOME_URL = "https://dashboard.exa.ai/home"
|
||||
|
||||
|
||||
def fill_first_input(page, selectors, value):
|
||||
"""填充第一个存在的输入框"""
|
||||
for selector in selectors:
|
||||
if page.query_selector(selector):
|
||||
page.fill(selector, value)
|
||||
return selector
|
||||
return None
|
||||
|
||||
|
||||
def click_first(page, selectors):
|
||||
"""点击第一个存在的按钮/链接"""
|
||||
for selector in selectors:
|
||||
if page.query_selector(selector):
|
||||
page.click(selector, no_wait_after=True)
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def extract_api_key(page):
|
||||
"""从页面文本或 HTML 中提取 Exa API Key。"""
|
||||
patterns = []
|
||||
|
||||
try:
|
||||
patterns.extend(re.findall(r"\b[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}\b", page.locator("main").inner_text(), re.I))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
patterns.extend(re.findall(r"\b[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}\b", page.content(), re.I))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
for candidate in patterns:
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def fetch_api_key_via_dashboard_api(page):
|
||||
"""直接调用已登录 dashboard 的 get-api-keys 接口,优先拿完整 key。"""
|
||||
try:
|
||||
payload = page.evaluate(
|
||||
"""
|
||||
async () => {
|
||||
const response = await fetch('/api/get-api-keys', {
|
||||
method: 'GET',
|
||||
credentials: 'include',
|
||||
headers: {
|
||||
'accept': 'application/json',
|
||||
},
|
||||
});
|
||||
|
||||
return {
|
||||
status: response.status,
|
||||
body: await response.text(),
|
||||
};
|
||||
}
|
||||
"""
|
||||
)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
if int(payload.get("status") or 0) != 200:
|
||||
return None
|
||||
|
||||
try:
|
||||
data = json.loads(payload.get("body") or "{}")
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
for item in data.get("apiKeys", []):
|
||||
candidate = (item.get("id") or "").strip()
|
||||
if re.fullmatch(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}", candidate, re.I):
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def ensure_dashboard_ready(page):
|
||||
"""跳过 onboarding,落到可读 API Key 的 dashboard 页面。"""
|
||||
if "dashboard.exa.ai" not in page.url.lower():
|
||||
page.wait_for_url("**/dashboard.exa.ai/**", timeout=30000, wait_until="domcontentloaded")
|
||||
|
||||
if "/onboarding" in page.url.lower():
|
||||
click_first(page, ['button:text-is("Skip")'])
|
||||
time.sleep(1)
|
||||
click_first(
|
||||
page,
|
||||
[
|
||||
'button:text-is("Yes, I don\\\'t want the $10 in credits anyway!")',
|
||||
'button:text-is("Yes")',
|
||||
],
|
||||
)
|
||||
page.wait_for_url("**/dashboard.exa.ai/**", timeout=30000, wait_until="domcontentloaded")
|
||||
time.sleep(2)
|
||||
|
||||
if "/home" not in page.url.lower():
|
||||
page.goto(_EXA_HOME_URL, wait_until="networkidle", timeout=30000)
|
||||
time.sleep(2)
|
||||
|
||||
|
||||
def wait_for_api_key(page, timeout=20):
|
||||
"""等待主页 API Key 卡片渲染并显示完整 key。"""
|
||||
start_time = time.time()
|
||||
while time.time() - start_time < timeout:
|
||||
ensure_dashboard_ready(page)
|
||||
api_key = fetch_api_key_via_dashboard_api(page)
|
||||
if api_key:
|
||||
return api_key
|
||||
|
||||
click_first(page, ['button:text-is("Show")'])
|
||||
time.sleep(1)
|
||||
api_key = extract_api_key(page)
|
||||
if api_key:
|
||||
return api_key
|
||||
time.sleep(1)
|
||||
return None
|
||||
|
||||
|
||||
def save_account(email, api_key):
|
||||
"""并发注册时串行写入 exa_accounts.txt"""
|
||||
with _SAVE_LOCK:
|
||||
with open(_SAVE_FILE, "a", encoding="utf-8") as file_obj:
|
||||
file_obj.write(f"{email},{_ACCOUNT_PASSWORD_LABEL},{api_key}\n")
|
||||
|
||||
|
||||
def verify_api_key(api_key, timeout=30):
|
||||
"""真实调用 Exa API,验证新 key 可用"""
|
||||
try:
|
||||
response = std_requests.post(
|
||||
"https://api.exa.ai/search",
|
||||
json={
|
||||
"query": "api key verification",
|
||||
"numResults": 1,
|
||||
},
|
||||
headers={
|
||||
"x-api-key": api_key,
|
||||
"Content-Type": "application/json",
|
||||
"Accept": "application/json",
|
||||
},
|
||||
timeout=timeout,
|
||||
)
|
||||
except Exception as exc:
|
||||
print(f"❌ API Key 调用测试失败: {exc}")
|
||||
return False
|
||||
|
||||
if response.status_code == 200:
|
||||
print("✅ API Key 调用测试通过")
|
||||
return True
|
||||
|
||||
preview = response.text.strip().replace("\n", " ")[:160]
|
||||
print(f"❌ API Key 调用测试失败: HTTP {response.status_code}")
|
||||
if preview:
|
||||
print(f" 响应: {preview}")
|
||||
return False
|
||||
|
||||
|
||||
def register_with_browser(email, password):
|
||||
"""使用浏览器完成 Exa 邮箱验证码注册"""
|
||||
print(f"🌐 使用浏览器模式注册 Exa: {email}")
|
||||
|
||||
try:
|
||||
with Camoufox(headless=REGISTER_HEADLESS) as browser:
|
||||
page = browser.new_page()
|
||||
|
||||
page.goto(_EXA_AUTH_URL, wait_until="networkidle", timeout=30000)
|
||||
time.sleep(2)
|
||||
|
||||
email_selector = fill_first_input(
|
||||
page,
|
||||
['input[type="email"]', 'input[placeholder="Email"]', 'input[aria-label="Email"]'],
|
||||
email,
|
||||
)
|
||||
if not email_selector:
|
||||
print("❌ Exa 登录页未找到邮箱输入框")
|
||||
return None
|
||||
|
||||
if not click_first(page, ['button:text-is("Continue")']):
|
||||
print("❌ Exa 登录页未找到 Continue 按钮")
|
||||
return None
|
||||
|
||||
page.wait_for_selector('input[placeholder*="verification" i], input[aria-label*="verification" i]', timeout=30000)
|
||||
print("✅ 到达 Exa 邮箱验证码页")
|
||||
|
||||
code = get_email_code(email, timeout=EMAIL_CODE_TIMEOUT, service="exa")
|
||||
if not code:
|
||||
return None
|
||||
|
||||
code_selector = fill_first_input(
|
||||
page,
|
||||
['input[placeholder*="verification" i]', 'input[aria-label*="verification" i]'],
|
||||
code,
|
||||
)
|
||||
if not code_selector:
|
||||
print("❌ Exa 验证码页未找到输入框")
|
||||
return None
|
||||
|
||||
if not click_first(page, ['button:text-is("VERIFY CODE")', 'button:text-is("Verify Code")', 'button:text-is("Verify")']):
|
||||
page.press(code_selector, "Enter")
|
||||
|
||||
page.wait_for_url("**/dashboard.exa.ai/**", timeout=30000, wait_until="domcontentloaded")
|
||||
print("✅ Exa 登录成功")
|
||||
|
||||
api_key = wait_for_api_key(page, timeout=API_KEY_TIMEOUT)
|
||||
if not api_key:
|
||||
print("⚠️ 未找到 Exa API Key")
|
||||
return None
|
||||
|
||||
print("🧪 验证 API Key 可用性...")
|
||||
if not verify_api_key(api_key):
|
||||
return None
|
||||
|
||||
save_account(email, api_key)
|
||||
|
||||
print("🎉 Exa 注册成功")
|
||||
print(f" 邮箱: {email}")
|
||||
print(f" 密码: {_ACCOUNT_PASSWORD_LABEL}")
|
||||
print(f" Key : {api_key}")
|
||||
return api_key
|
||||
except Exception as exc:
|
||||
print(f"❌ Exa 注册失败: {exc}")
|
||||
return None
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
"""
|
||||
Exa 注册统一入口
|
||||
"""
|
||||
from mail_provider import create_email
|
||||
from exa_browser_solver import register_with_browser
|
||||
|
||||
|
||||
def register(email, password):
|
||||
"""统一注册入口"""
|
||||
return register_with_browser(email, password)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
email, password = create_email(service="exa")
|
||||
result = register(email, password)
|
||||
if result:
|
||||
print(f"✅ 注册成功: {email}")
|
||||
else:
|
||||
print(f"❌ 注册失败: {email}")
|
||||
+35
-12
@@ -32,7 +32,7 @@ _DUCKMAIL_DOMAIN_PRIORITY = (
|
||||
_DUCKMAIL_DOMAIN_CACHE = None
|
||||
_DUCKMAIL_MAILBOX_CACHE = {}
|
||||
_SELECTED_DOMAIN = ""
|
||||
_SUPPORTED_SERVICES = ("tavily", "firecrawl")
|
||||
_SUPPORTED_SERVICES = ("tavily", "firecrawl", "exa")
|
||||
|
||||
|
||||
def rand_str(n=8):
|
||||
@@ -72,7 +72,11 @@ def _normalize_service(service):
|
||||
|
||||
def _username_prefix(service):
|
||||
service = _normalize_service(service)
|
||||
return "fc" if service == "firecrawl" else "tavily"
|
||||
if service == "firecrawl":
|
||||
return "fc"
|
||||
if service == "exa":
|
||||
return "exa"
|
||||
return "tavily"
|
||||
|
||||
|
||||
def create_email(service="tavily"):
|
||||
@@ -104,13 +108,13 @@ def get_verification_link(email, timeout=120):
|
||||
)
|
||||
|
||||
|
||||
def get_email_code(email, timeout=120):
|
||||
def get_email_code(email, timeout=120, service="tavily"):
|
||||
"""等待邮箱里的 6 位验证码。"""
|
||||
print(f"📨 等待邮箱验证码(最多 {timeout} 秒)...")
|
||||
return _poll_mailbox(
|
||||
email=email,
|
||||
timeout=timeout,
|
||||
extractor=_extract_email_code,
|
||||
extractor=lambda message: _extract_email_code(message, service=service),
|
||||
found_message="✅ 收到 6 位验证码",
|
||||
timeout_message="❌ 等待邮箱验证码超时",
|
||||
error_prefix="读取邮箱验证码失败",
|
||||
@@ -175,16 +179,35 @@ def _extract_verification_link(message):
|
||||
return None
|
||||
|
||||
|
||||
def _extract_email_code(message):
|
||||
def _extract_email_code(message, service="tavily"):
|
||||
service = _normalize_service(service)
|
||||
subject = (message.get("subject") or "").lower()
|
||||
if "verify your identity" not in subject and "verify" not in subject and "tavily" not in subject:
|
||||
return None
|
||||
|
||||
text = message.get("text") or ""
|
||||
content = _message_content(message)
|
||||
match = re.search(r"\b(\d{6})\b", content)
|
||||
if not match:
|
||||
return None
|
||||
return match.group(1)
|
||||
combined = f"{subject}\n{content}".lower()
|
||||
|
||||
if service == "exa":
|
||||
if "exa" not in combined:
|
||||
return None
|
||||
if "verification code" not in combined and "sign in" not in combined:
|
||||
return None
|
||||
for source in (text, content):
|
||||
match = re.search(
|
||||
r"verification code(?:\s+for\s+exa)?(?:\s+is)?[^0-9]*(\d{6})",
|
||||
source,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
if match:
|
||||
return match.group(1)
|
||||
else:
|
||||
if "verify your identity" not in subject and "verify" not in subject and "tavily" not in combined:
|
||||
return None
|
||||
|
||||
for source in (text, content):
|
||||
match = re.search(r"\b(\d{6})\b", source)
|
||||
if match:
|
||||
return match.group(1)
|
||||
return None
|
||||
|
||||
|
||||
def _iter_messages(email):
|
||||
|
||||
@@ -125,6 +125,7 @@ from config import (
|
||||
)
|
||||
from tavily_core import create_email as create_tavily_email, register as register_tavily
|
||||
from firecrawl_core import register as register_firecrawl
|
||||
from exa_core import register as register_exa
|
||||
from mail_provider import create_email, get_active_domain, get_configured_domains, set_selected_domain
|
||||
|
||||
# ──────────────────────────────────────────────
|
||||
@@ -199,9 +200,21 @@ def validate_runtime_config(upload, show_provider_summary=True):
|
||||
return True
|
||||
|
||||
def print_runtime_summary(service="tavily"):
|
||||
service_name = "Tavily" if service == "tavily" else "Firecrawl"
|
||||
output_file = "accounts.txt" if service == "tavily" else "firecrawl_accounts.txt"
|
||||
account_prefix = "tavily-" if service == "tavily" else "fc-"
|
||||
service_name = {
|
||||
"tavily": "Tavily",
|
||||
"firecrawl": "Firecrawl",
|
||||
"exa": "Exa",
|
||||
}.get(service, "Tavily")
|
||||
output_file = {
|
||||
"tavily": "accounts.txt",
|
||||
"firecrawl": "firecrawl_accounts.txt",
|
||||
"exa": "exa_accounts.txt",
|
||||
}.get(service, "accounts.txt")
|
||||
account_prefix = {
|
||||
"tavily": "tavily-",
|
||||
"firecrawl": "fc-",
|
||||
"exa": "exa-",
|
||||
}.get(service, "tavily-")
|
||||
print(f"""
|
||||
┌──────────────────────────────────────────┐
|
||||
│ 多服务自动注册启动台 │
|
||||
@@ -395,8 +408,10 @@ def register_one(index, total, upload, service="tavily"):
|
||||
|
||||
if service == "tavily":
|
||||
result = register_tavily(email, password)
|
||||
else: # firecrawl
|
||||
elif service == "firecrawl":
|
||||
result = register_firecrawl(email, password)
|
||||
else:
|
||||
result = register_exa(email, password)
|
||||
|
||||
if result and result != "SUCCESS_NO_KEY":
|
||||
if upload:
|
||||
@@ -476,14 +491,17 @@ def prompt_service_choice():
|
||||
print("\n请选择要注册的服务:")
|
||||
print(" 1. Tavily")
|
||||
print(" 2. Firecrawl")
|
||||
print(" 3. Exa")
|
||||
|
||||
while True:
|
||||
print("请输入选项 (1-2,默认 1): ", end="")
|
||||
print("请输入选项 (1-3,默认 1): ", end="")
|
||||
raw = input().strip()
|
||||
if raw == "" or raw == "1":
|
||||
return "tavily"
|
||||
elif raw == "2":
|
||||
return "firecrawl"
|
||||
elif raw == "3":
|
||||
return "exa"
|
||||
else:
|
||||
print("❌ 请输入有效编号")
|
||||
continue
|
||||
@@ -492,7 +510,7 @@ def main():
|
||||
service = prompt_service_choice()
|
||||
print_runtime_summary(service)
|
||||
|
||||
# Firecrawl 不需要 Solver
|
||||
# 目前只有 Tavily 需要 Solver
|
||||
need_solver = (service == "tavily")
|
||||
|
||||
if not validate_runtime_config(False, show_provider_summary=True):
|
||||
|
||||
Reference in new issue
Block a user