整理代码并提交
This commit is contained in:
commit
968f37ad94
309 files changed
+46134
No files matched your search
@@ -0,0 +1,24 @@
|
||||
.git
|
||||
.github
|
||||
.codex
|
||||
.venv
|
||||
.pytest_cache
|
||||
**/__pycache__/
|
||||
|
||||
# Env / secrets
|
||||
.env
|
||||
.env.*
|
||||
!deploy/.env.example
|
||||
|
||||
data/
|
||||
issues/
|
||||
plan/
|
||||
docs/
|
||||
参考资料/
|
||||
|
||||
frontend/node_modules/
|
||||
frontend/dist/
|
||||
|
||||
backend/.pytest_cache/
|
||||
|
||||
_tmp_append_test.md
|
||||
@@ -0,0 +1,74 @@
|
||||
name: ci
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- test
|
||||
pull_request:
|
||||
|
||||
concurrency:
|
||||
group: ci-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
backend-tests:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
cache: pip
|
||||
cache-dependency-path: requirements-dev.txt
|
||||
|
||||
- name: Install dependencies
|
||||
run: python -m pip install -r requirements-dev.txt
|
||||
|
||||
- name: Compile
|
||||
run: python -m compileall -q backend
|
||||
|
||||
- name: Tests
|
||||
run: python -m pytest -q backend/tests
|
||||
|
||||
frontend-tests:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "20"
|
||||
cache: npm
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
working-directory: frontend
|
||||
|
||||
- name: Lint
|
||||
run: npm run lint
|
||||
working-directory: frontend
|
||||
|
||||
- name: Typecheck
|
||||
run: npm run typecheck
|
||||
working-directory: frontend
|
||||
|
||||
- name: Tests
|
||||
run: npm test
|
||||
working-directory: frontend
|
||||
|
||||
compose-config:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Prepare deploy env file
|
||||
run: |
|
||||
if [ ! -f deploy/.env ]; then
|
||||
cp deploy/.env.example deploy/.env
|
||||
fi
|
||||
|
||||
- name: Validate docker compose config
|
||||
run: docker compose -f deploy/docker-compose.yml config > /dev/null
|
||||
+68
@@ -0,0 +1,68 @@
|
||||
## Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
.pytest_cache/
|
||||
.mypy_cache/
|
||||
.ruff_cache/
|
||||
*.egg-info/
|
||||
.coverage
|
||||
.coverage.*
|
||||
htmlcov/
|
||||
coverage.xml
|
||||
|
||||
## Virtualenv
|
||||
.venv/
|
||||
venv/
|
||||
ENV/
|
||||
env/
|
||||
|
||||
## Node
|
||||
node_modules/
|
||||
dist/
|
||||
.vite/
|
||||
|
||||
## Runtime data / local state (never commit)
|
||||
data/
|
||||
backend/data/
|
||||
deploy/.env
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
!deploy/.env.example
|
||||
*.db
|
||||
*.sqlite
|
||||
*.sqlite3
|
||||
|
||||
## Codex / AI tooling (repo-local)
|
||||
.codex/
|
||||
plan/
|
||||
issues/
|
||||
docs/
|
||||
参考资料/
|
||||
|
||||
## Data exports / plans
|
||||
*.csv
|
||||
*.tsv
|
||||
*.plan.md
|
||||
|
||||
## OS / IDE
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
desktop.ini
|
||||
.idea/
|
||||
.vscode/
|
||||
*.iml
|
||||
|
||||
## Local temp / scratch
|
||||
*.log
|
||||
*.tmp
|
||||
要求.txt
|
||||
_tmp_easy_export.txt
|
||||
_tmp_*.md
|
||||
_tmp_*.txt
|
||||
|
||||
## Keep public-facing docs at repo root
|
||||
/*.md
|
||||
!README.md
|
||||
!LICENSE*
|
||||
@@ -0,0 +1,122 @@
|
||||
# Random Mage (new-pixiv-api)
|
||||
|
||||
一个自托管的随机图片 API:把 Pixiv 原图链接导入到本地数据库后,即可通过 `/random` 按 **标签 / 热度 / 分辨率 / R18 / AI / 作品类型** 等条件组合筛选并随机出图;同时提供管理后台(Web UI)+ 后台 Worker,用于导入、补全元数据、代理探测等任务。
|
||||
|
||||
> 免责声明:本项目为非官方实现;请遵守 Pixiv ToS/版权与当地法律法规,自行承担使用风险。不要将任何 refresh_token / 密钥 / 代理密码提交到仓库。
|
||||
|
||||
## 功能概览
|
||||
|
||||
- 公共 API
|
||||
- `GET /random`:默认直接返回图片流;也可 `format=json|simple_json` 返回 JSON;支持大量筛选参数
|
||||
- `GET /images` / `GET /images/{id}`:按条件分页查看已入库图片
|
||||
- `GET /i/{image_id}.{ext}`:本地代理/缓存路径(`/random?redirect=1` 会跳转到这里)
|
||||
- `GET /tags`、`GET /authors`、`GET /healthz`、`GET /version`、`GET /docs`(人类可读文档页)
|
||||
- Swagger / OpenAPI:`GET /api/docs`、`GET /openapi.json`
|
||||
- 管理后台(`/admin`)
|
||||
- 登录后可管理:Pixiv refresh_token、代理池、导入、任务队列、运行时开关、审计/统计等
|
||||
- Worker(后台任务)
|
||||
- 导入图片 URL、补全元数据(tags/宽高/浏览收藏等)、修复失效 URL、代理探测等
|
||||
- 可选能力
|
||||
- 公网访问保护:启用 Public API Key + 限流
|
||||
- `imgproxy`:对外提供签名处理 URL(可隐藏 origin URL)
|
||||
- 代理路由:为上游图片/接口选择代理
|
||||
|
||||
## 一键部署(Docker Compose,推荐)
|
||||
|
||||
前置:安装 Docker + Docker Compose。
|
||||
|
||||
1) 准备环境变量
|
||||
|
||||
```bash
|
||||
cp deploy/.env.example deploy/.env
|
||||
```
|
||||
|
||||
然后编辑 `deploy/.env`,至少建议修改:
|
||||
|
||||
- `SECRET_KEY`:JWT/API Key 哈希用(生产环境必须修改)
|
||||
- `ADMIN_PASSWORD`:后台密码(生产环境必须修改)
|
||||
- `FIELD_ENCRYPTION_KEY`:用于加密保存 refresh_token(生产环境必须提供;建议用 secret 挂载或文件方式)
|
||||
|
||||
2) 启动
|
||||
|
||||
```bash
|
||||
docker compose -f deploy/docker-compose.yml up -d --build
|
||||
```
|
||||
|
||||
3) 访问
|
||||
|
||||
- API 基址:`http://localhost:23222`
|
||||
- 健康检查:`GET http://localhost:23222/healthz`
|
||||
- 文档页:`GET http://localhost:23222/docs`
|
||||
- 后台 UI:`http://localhost:23222/admin`
|
||||
- Swagger:`http://localhost:23222/api/docs`
|
||||
|
||||
## 首次使用(导入数据 + 补全元数据)
|
||||
|
||||
这个项目不会“自动替你抓全站图片”,你需要先把图片记录导入数据库。
|
||||
|
||||
1) 登录后台
|
||||
|
||||
- 打开 `http://localhost:23222/admin`
|
||||
- 使用 `deploy/.env` 中的 `ADMIN_USERNAME` / `ADMIN_PASSWORD` 登录
|
||||
|
||||
2) 导入图片原图 URL
|
||||
|
||||
后台提供“导入”入口,支持:
|
||||
|
||||
- 直接粘贴多行文本(每行一个 URL)
|
||||
- 上传 `.txt` 文件(同样每行一个 URL)
|
||||
|
||||
导入内容应为 Pixiv 原图链接(例如 `i.pximg.net/.../img-original/...` 这类)。导入时会自动去重(按 illust_id + page)。
|
||||
|
||||
3) 配置 Pixiv refresh_token(可选,但强烈建议)
|
||||
|
||||
若你希望 Worker 自动补全/刷新元数据(tags、宽高、浏览/收藏/评论、R18 标记、AI 类型等),需要:
|
||||
|
||||
- 在后台添加一个或多个 refresh_token
|
||||
- 配置 `PIXIV_OAUTH_CLIENT_ID` / `PIXIV_OAUTH_CLIENT_SECRET`(以及可选的 `PIXIV_OAUTH_HASH_SECRET`)
|
||||
|
||||
> 安全提示:refresh_token 属于敏感凭据;请只放在运行环境里(`deploy/.env` / secret 管理),不要写进代码/仓库。
|
||||
|
||||
4) 确认 Worker 正常
|
||||
|
||||
`/healthz` 会同时检查 DB、队列、以及 Worker 心跳;如果 `worker_ok=false`,通常意味着 Worker 没启动或无法连接数据库卷。
|
||||
|
||||
## 配置说明(节选)
|
||||
|
||||
所有配置均通过环境变量注入(见 `deploy/.env.example`):
|
||||
|
||||
- `APP_ENV`:`dev` / `prod`(`prod` 会启用更严格的必填校验)
|
||||
- `DATABASE_URL`:默认 SQLite(容器内 `/app/data/app.db`,通过 `../data:/app/data` 挂载)
|
||||
- `PUBLIC_API_KEY_REQUIRED` / `PUBLIC_API_KEY_RPM` / `PUBLIC_API_KEY_BURST`:公网访问保护(可选)
|
||||
- `WORKER_*`、`IMPORT_*`、`SQLITE_BUSY_TIMEOUT_MS`:Worker/导入/SQLite 运行参数
|
||||
- `IMGPROXY_*`:imgproxy 集成(可选)
|
||||
|
||||
## 本地开发(可选)
|
||||
|
||||
后端(Python 3.11):
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
python -m venv .venv
|
||||
.\.venv\Scripts\activate
|
||||
python -m pip install -r ..\requirements-dev.txt
|
||||
alembic upgrade head
|
||||
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
前端管理台(Node 20):
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm ci
|
||||
npm run dev
|
||||
```
|
||||
|
||||
如需让前端直连本地后端,设置 `VITE_API_BASE_URL`(例如 `http://localhost:8000`)。
|
||||
|
||||
## 运行测试
|
||||
|
||||
- 后端:`python -m pytest -q backend/tests`
|
||||
- 前端:`cd frontend && npm test`
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
FROM node:20-slim AS frontend-build
|
||||
|
||||
WORKDIR /frontend
|
||||
|
||||
COPY frontend/package.json frontend/package-lock.json /frontend/
|
||||
RUN npm ci --no-audit --no-fund
|
||||
|
||||
COPY frontend/ /frontend/
|
||||
RUN npm run build
|
||||
|
||||
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1
|
||||
|
||||
RUN python -m pip install --no-cache-dir --upgrade pip
|
||||
|
||||
COPY requirements.txt /app/requirements.txt
|
||||
RUN python -m pip install --no-cache-dir -r /app/requirements.txt
|
||||
|
||||
RUN mkdir -p /app/data /app/web
|
||||
|
||||
COPY backend/app /app/app
|
||||
COPY backend/alembic.ini /app/alembic.ini
|
||||
COPY backend/alembic /app/alembic
|
||||
COPY --from=frontend-build /frontend/dist /app/web/dist
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
@@ -0,0 +1,38 @@
|
||||
[alembic]
|
||||
script_location = %(here)s/alembic
|
||||
prepend_sys_path = %(here)s
|
||||
sqlalchemy.url = sqlite:///./data/app.db
|
||||
|
||||
[loggers]
|
||||
keys = root,sqlalchemy,alembic
|
||||
|
||||
[handlers]
|
||||
keys = console
|
||||
|
||||
[formatters]
|
||||
keys = generic
|
||||
|
||||
[logger_root]
|
||||
level = WARN
|
||||
handlers = console
|
||||
|
||||
[logger_sqlalchemy]
|
||||
level = WARN
|
||||
handlers =
|
||||
qualname = sqlalchemy.engine
|
||||
|
||||
[logger_alembic]
|
||||
level = INFO
|
||||
handlers =
|
||||
qualname = alembic
|
||||
|
||||
[handler_console]
|
||||
class = StreamHandler
|
||||
args = (sys.stderr,)
|
||||
level = NOTSET
|
||||
formatter = generic
|
||||
|
||||
[formatter_generic]
|
||||
format = %(levelname)-5.5s [%(name)s] %(message)s
|
||||
datefmt = %H:%M:%S
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
from logging.config import fileConfig
|
||||
from pathlib import Path
|
||||
|
||||
from alembic import context
|
||||
from sqlalchemy import engine_from_config, pool
|
||||
from sqlalchemy.engine.url import make_url
|
||||
|
||||
config = context.config
|
||||
|
||||
if config.config_file_name is not None:
|
||||
fileConfig(config.config_file_name)
|
||||
|
||||
BACKEND_DIR = Path(__file__).resolve().parents[1]
|
||||
if str(BACKEND_DIR) not in sys.path:
|
||||
sys.path.insert(0, str(BACKEND_DIR))
|
||||
|
||||
target_metadata = None
|
||||
|
||||
|
||||
def _get_database_url() -> str:
|
||||
url = (os.environ.get("DATABASE_URL") or config.get_main_option("sqlalchemy.url") or "").strip()
|
||||
if not url:
|
||||
raise RuntimeError("DATABASE_URL (or sqlalchemy.url) is required")
|
||||
return url.replace("+aiosqlite", "")
|
||||
|
||||
|
||||
def _ensure_sqlite_dir(url: str) -> None:
|
||||
parsed = make_url(url)
|
||||
if parsed.get_backend_name() != "sqlite":
|
||||
return
|
||||
db_path = parsed.database
|
||||
if not db_path or db_path == ":memory:":
|
||||
return
|
||||
Path(db_path).expanduser().resolve().parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def run_migrations_offline() -> None:
|
||||
url = _get_database_url()
|
||||
_ensure_sqlite_dir(url)
|
||||
context.configure(
|
||||
url=url,
|
||||
target_metadata=target_metadata,
|
||||
literal_binds=True,
|
||||
dialect_opts={"paramstyle": "named"},
|
||||
render_as_batch=url.startswith("sqlite"),
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
def run_migrations_online() -> None:
|
||||
url = _get_database_url()
|
||||
_ensure_sqlite_dir(url)
|
||||
|
||||
connectable = engine_from_config(
|
||||
config.get_section(config.config_ini_section, {}),
|
||||
prefix="sqlalchemy.",
|
||||
url=url,
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
|
||||
with connectable.connect() as connection:
|
||||
context.configure(
|
||||
connection=connection,
|
||||
target_metadata=target_metadata,
|
||||
render_as_batch=url.startswith("sqlite"),
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
if context.is_offline_mode():
|
||||
run_migrations_offline()
|
||||
else:
|
||||
run_migrations_online()
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0001"
|
||||
down_revision = None
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"imports",
|
||||
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column("created_by", sa.Text(), nullable=True),
|
||||
sa.Column("source", sa.Text(), nullable=True),
|
||||
sa.Column("total", sa.Integer(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("accepted", sa.Integer(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("success", sa.Integer(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("failed", sa.Integer(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("detail_json", sa.Text(), nullable=True),
|
||||
)
|
||||
op.create_index("idx_imports_created_at", "imports", ["created_at"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_imports_created_at", table_name="imports")
|
||||
op.drop_table("imports")
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0002"
|
||||
down_revision = "20260210_0001"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"images",
|
||||
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||
sa.Column("illust_id", sa.Integer(), nullable=False),
|
||||
sa.Column("page_index", sa.Integer(), nullable=False),
|
||||
sa.Column("ext", sa.Text(), nullable=False),
|
||||
sa.Column("original_url", sa.Text(), nullable=False),
|
||||
sa.Column("proxy_path", sa.Text(), nullable=False),
|
||||
sa.Column("random_key", sa.Float(), nullable=False),
|
||||
sa.Column("width", sa.Integer(), nullable=True),
|
||||
sa.Column("height", sa.Integer(), nullable=True),
|
||||
sa.Column("aspect_ratio", sa.Float(), nullable=True),
|
||||
sa.Column("orientation", sa.Integer(), nullable=True),
|
||||
sa.Column("x_restrict", sa.Integer(), nullable=True),
|
||||
sa.Column("ai_type", sa.Integer(), nullable=True),
|
||||
sa.Column("user_id", sa.Integer(), nullable=True),
|
||||
sa.Column("user_name", sa.Text(), nullable=True),
|
||||
sa.Column("title", sa.Text(), nullable=True),
|
||||
sa.Column("created_at_pixiv", sa.Text(), nullable=True),
|
||||
sa.Column("status", sa.Integer(), nullable=False, server_default=sa.text("1")),
|
||||
sa.Column("fail_count", sa.Integer(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("last_fail_at", sa.Text(), nullable=True),
|
||||
sa.Column("last_ok_at", sa.Text(), nullable=True),
|
||||
sa.Column("last_error_code", sa.Text(), nullable=True),
|
||||
sa.Column("last_error_msg", sa.Text(), nullable=True),
|
||||
sa.Column(
|
||||
"created_import_id",
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("imports.id", name="fk_images_import", ondelete="SET NULL"),
|
||||
nullable=True,
|
||||
),
|
||||
sa.Column(
|
||||
"added_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.CheckConstraint("status IN (1,2,3,4)", name="ck_images_status"),
|
||||
sa.CheckConstraint("random_key >= 0.0 AND random_key < 1.0", name="ck_images_random_key"),
|
||||
sa.UniqueConstraint("illust_id", "page_index", name="uq_images_illust_page"),
|
||||
)
|
||||
|
||||
op.create_index(
|
||||
"idx_images_filter",
|
||||
"images",
|
||||
["status", "x_restrict", "orientation", "width", "height", "random_key"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index(
|
||||
"idx_images_user_random",
|
||||
"images",
|
||||
["status", "user_id", "random_key"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index("idx_images_created_at_pixiv", "images", ["created_at_pixiv"], unique=False)
|
||||
op.create_index("idx_images_created_import_id", "images", ["created_import_id"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_images_created_import_id", table_name="images")
|
||||
op.drop_index("idx_images_created_at_pixiv", table_name="images")
|
||||
op.drop_index("idx_images_user_random", table_name="images")
|
||||
op.drop_index("idx_images_filter", table_name="images")
|
||||
op.drop_table("images")
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0003"
|
||||
down_revision = "20260210_0002"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"tags",
|
||||
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||
sa.Column("name", sa.Text(), nullable=False),
|
||||
sa.Column("translated_name", sa.Text(), nullable=True),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.UniqueConstraint("name", name="uq_tags_name"),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("tags")
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0004"
|
||||
down_revision = "20260210_0003"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"image_tags",
|
||||
sa.Column(
|
||||
"image_id",
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("images.id", name="fk_image_tags_image", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column(
|
||||
"tag_id",
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("tags.id", name="fk_image_tags_tag", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.PrimaryKeyConstraint("image_id", "tag_id"),
|
||||
)
|
||||
op.create_index("idx_image_tags_tag_image", "image_tags", ["tag_id", "image_id"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_image_tags_tag_image", table_name="image_tags")
|
||||
op.drop_table("image_tags")
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0005"
|
||||
down_revision = "20260210_0004"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"pixiv_tokens",
|
||||
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column("label", sa.Text(), nullable=True),
|
||||
sa.Column("enabled", sa.Integer(), nullable=False, server_default=sa.text("1")),
|
||||
sa.Column("refresh_token_enc", sa.Text(), nullable=False),
|
||||
sa.Column("refresh_token_masked", sa.Text(), nullable=False),
|
||||
sa.Column("weight", sa.Float(), nullable=False, server_default=sa.text("1.0")),
|
||||
sa.Column("error_count", sa.Integer(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("backoff_until", sa.Text(), nullable=True),
|
||||
sa.Column("last_ok_at", sa.Text(), nullable=True),
|
||||
sa.Column("last_fail_at", sa.Text(), nullable=True),
|
||||
sa.Column("last_error_code", sa.Text(), nullable=True),
|
||||
sa.Column("last_error_msg", sa.Text(), nullable=True),
|
||||
sa.CheckConstraint("enabled IN (0,1)", name="ck_pixiv_tokens_enabled"),
|
||||
)
|
||||
op.create_index("idx_pixiv_tokens_enabled", "pixiv_tokens", ["enabled"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_pixiv_tokens_enabled", table_name="pixiv_tokens")
|
||||
op.drop_table("pixiv_tokens")
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0006"
|
||||
down_revision = "20260210_0005"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"proxy_endpoints",
|
||||
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column("scheme", sa.Text(), nullable=False),
|
||||
sa.Column("host", sa.Text(), nullable=False),
|
||||
sa.Column("port", sa.Integer(), nullable=False),
|
||||
sa.Column("username", sa.Text(), nullable=False, server_default=sa.text("''")),
|
||||
sa.Column("password_enc", sa.Text(), nullable=False, server_default=sa.text("''")),
|
||||
sa.Column("enabled", sa.Integer(), nullable=False, server_default=sa.text("1")),
|
||||
sa.Column("source", sa.Text(), nullable=False, server_default=sa.text("'manual'")),
|
||||
sa.Column("source_ref", sa.Text(), nullable=True),
|
||||
sa.Column("last_latency_ms", sa.Float(), nullable=True),
|
||||
sa.Column("last_ok_at", sa.Text(), nullable=True),
|
||||
sa.Column("last_fail_at", sa.Text(), nullable=True),
|
||||
sa.Column("success_count", sa.Integer(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("failure_count", sa.Integer(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("blacklisted_until", sa.Text(), nullable=True),
|
||||
sa.Column("last_error", sa.Text(), nullable=True),
|
||||
sa.UniqueConstraint("scheme", "host", "port", "username", name="uq_proxy_identity"),
|
||||
sa.CheckConstraint("enabled IN (0,1)", name="ck_proxy_enabled"),
|
||||
)
|
||||
|
||||
op.create_index("idx_proxy_endpoints_enabled", "proxy_endpoints", ["enabled"], unique=False)
|
||||
op.create_index("idx_proxy_endpoints_source", "proxy_endpoints", ["source"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_proxy_endpoints_source", table_name="proxy_endpoints")
|
||||
op.drop_index("idx_proxy_endpoints_enabled", table_name="proxy_endpoints")
|
||||
op.drop_table("proxy_endpoints")
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0007"
|
||||
down_revision = "20260210_0006"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"proxy_pools",
|
||||
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column("name", sa.Text(), nullable=False),
|
||||
sa.Column("description", sa.Text(), nullable=True),
|
||||
sa.Column("enabled", sa.Integer(), nullable=False, server_default=sa.text("1")),
|
||||
sa.UniqueConstraint("name", name="uq_proxy_pools_name"),
|
||||
sa.CheckConstraint("enabled IN (0,1)", name="ck_proxy_pools_enabled"),
|
||||
)
|
||||
op.create_index("idx_proxy_pools_enabled", "proxy_pools", ["enabled"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_proxy_pools_enabled", table_name="proxy_pools")
|
||||
op.drop_table("proxy_pools")
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0008"
|
||||
down_revision = "20260210_0007"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"proxy_pool_endpoints",
|
||||
sa.Column(
|
||||
"pool_id",
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("proxy_pools.id", name="fk_ppe_pool", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column(
|
||||
"endpoint_id",
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("proxy_endpoints.id", name="fk_ppe_ep", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column("enabled", sa.Integer(), nullable=False, server_default=sa.text("1")),
|
||||
sa.Column("weight", sa.Integer(), nullable=False, server_default=sa.text("1")),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.PrimaryKeyConstraint("pool_id", "endpoint_id"),
|
||||
sa.CheckConstraint("enabled IN (0,1)", name="ck_ppe_enabled"),
|
||||
)
|
||||
|
||||
op.create_index(
|
||||
"idx_ppe_pool_enabled",
|
||||
"proxy_pool_endpoints",
|
||||
["pool_id", "enabled"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index(
|
||||
"idx_ppe_endpoint_pool",
|
||||
"proxy_pool_endpoints",
|
||||
["endpoint_id", "pool_id"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_ppe_endpoint_pool", table_name="proxy_pool_endpoints")
|
||||
op.drop_index("idx_ppe_pool_enabled", table_name="proxy_pool_endpoints")
|
||||
op.drop_table("proxy_pool_endpoints")
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0009"
|
||||
down_revision = "20260210_0008"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"token_proxy_bindings",
|
||||
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column(
|
||||
"token_id",
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("pixiv_tokens.id", name="fk_tpb_token", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column(
|
||||
"pool_id",
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("proxy_pools.id", name="fk_tpb_pool", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column(
|
||||
"primary_proxy_id",
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("proxy_endpoints.id", name="fk_tpb_primary", ondelete="RESTRICT"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column(
|
||||
"override_proxy_id",
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("proxy_endpoints.id", name="fk_tpb_override", ondelete="SET NULL"),
|
||||
nullable=True,
|
||||
),
|
||||
sa.Column("override_expires_at", sa.Text(), nullable=True),
|
||||
sa.UniqueConstraint("token_id", "pool_id", name="uq_token_pool"),
|
||||
)
|
||||
|
||||
op.create_index("idx_tpb_pool", "token_proxy_bindings", ["pool_id"], unique=False)
|
||||
op.create_index("idx_tpb_primary", "token_proxy_bindings", ["primary_proxy_id"], unique=False)
|
||||
op.create_index("idx_tpb_override", "token_proxy_bindings", ["override_proxy_id"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_tpb_override", table_name="token_proxy_bindings")
|
||||
op.drop_index("idx_tpb_primary", table_name="token_proxy_bindings")
|
||||
op.drop_index("idx_tpb_pool", table_name="token_proxy_bindings")
|
||||
op.drop_table("token_proxy_bindings")
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0010"
|
||||
down_revision = "20260210_0009"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"runtime_settings",
|
||||
sa.Column("key", sa.Text(), primary_key=True),
|
||||
sa.Column("value_json", sa.Text(), nullable=False),
|
||||
sa.Column("description", sa.Text(), nullable=True),
|
||||
sa.Column(
|
||||
"updated_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column("updated_by", sa.Text(), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("runtime_settings")
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0011"
|
||||
down_revision = "20260210_0010"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"jobs",
|
||||
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column("type", sa.Text(), nullable=False),
|
||||
sa.Column("status", sa.Text(), nullable=False),
|
||||
sa.Column("priority", sa.Integer(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("run_after", sa.Text(), nullable=True),
|
||||
sa.Column("attempt", sa.Integer(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("max_attempts", sa.Integer(), nullable=False, server_default=sa.text("3")),
|
||||
sa.Column("payload_json", sa.Text(), nullable=False),
|
||||
sa.Column("last_error", sa.Text(), nullable=True),
|
||||
sa.Column("locked_by", sa.Text(), nullable=True),
|
||||
sa.Column("locked_at", sa.Text(), nullable=True),
|
||||
sa.Column("ref_type", sa.Text(), nullable=True),
|
||||
sa.Column("ref_id", sa.Text(), nullable=True),
|
||||
sa.CheckConstraint(
|
||||
"status IN ('pending','running','paused','canceled','completed','failed','dlq')",
|
||||
name="ck_jobs_status",
|
||||
),
|
||||
)
|
||||
|
||||
op.create_index(
|
||||
"idx_jobs_status_priority",
|
||||
"jobs",
|
||||
["status", "priority", "id"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index("idx_jobs_run_after", "jobs", ["run_after"], unique=False)
|
||||
op.create_index("idx_jobs_ref", "jobs", ["ref_type", "ref_id"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_jobs_ref", table_name="jobs")
|
||||
op.drop_index("idx_jobs_run_after", table_name="jobs")
|
||||
op.drop_index("idx_jobs_status_priority", table_name="jobs")
|
||||
op.drop_table("jobs")
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0012"
|
||||
down_revision = "20260210_0011"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"hydration_runs",
|
||||
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column("type", sa.Text(), nullable=False),
|
||||
sa.Column("status", sa.Text(), nullable=False),
|
||||
sa.Column("criteria_json", sa.Text(), nullable=True),
|
||||
sa.Column("cursor_json", sa.Text(), nullable=True),
|
||||
sa.Column("total", sa.Integer(), nullable=True),
|
||||
sa.Column("processed", sa.Integer(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("success", sa.Integer(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("failed", sa.Integer(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("started_at", sa.Text(), nullable=True),
|
||||
sa.Column("finished_at", sa.Text(), nullable=True),
|
||||
sa.Column("last_error", sa.Text(), nullable=True),
|
||||
sa.CheckConstraint(
|
||||
"status IN ('pending','running','paused','canceled','completed','failed')",
|
||||
name="ck_hr_status",
|
||||
),
|
||||
)
|
||||
|
||||
op.create_index(
|
||||
"idx_hr_status_updated",
|
||||
"hydration_runs",
|
||||
["status", "updated_at"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_hr_status_updated", table_name="hydration_runs")
|
||||
op.drop_table("hydration_runs")
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0013"
|
||||
down_revision = "20260210_0012"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"request_logs",
|
||||
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column("request_id", sa.Text(), nullable=True),
|
||||
sa.Column("method", sa.Text(), nullable=False),
|
||||
sa.Column("route", sa.Text(), nullable=False),
|
||||
sa.Column("status", sa.Integer(), nullable=False),
|
||||
sa.Column("duration_ms", sa.Integer(), nullable=False),
|
||||
sa.Column("ip", sa.Text(), nullable=True),
|
||||
sa.Column("user_agent", sa.Text(), nullable=True),
|
||||
sa.Column("sample_rate", sa.Float(), nullable=True),
|
||||
)
|
||||
|
||||
op.create_index("idx_request_logs_created_at", "request_logs", ["created_at"], unique=False)
|
||||
op.create_index("idx_request_logs_route", "request_logs", ["route"], unique=False)
|
||||
op.create_index("idx_request_logs_status", "request_logs", ["status"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_request_logs_status", table_name="request_logs")
|
||||
op.drop_index("idx_request_logs_route", table_name="request_logs")
|
||||
op.drop_index("idx_request_logs_created_at", table_name="request_logs")
|
||||
op.drop_table("request_logs")
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260210_0014"
|
||||
down_revision = "20260210_0013"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"admin_audit",
|
||||
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column("actor", sa.Text(), nullable=True),
|
||||
sa.Column("action", sa.Text(), nullable=False),
|
||||
sa.Column("resource", sa.Text(), nullable=False),
|
||||
sa.Column("record_id", sa.Text(), nullable=True),
|
||||
sa.Column("request_id", sa.Text(), nullable=True),
|
||||
sa.Column("ip", sa.Text(), nullable=True),
|
||||
sa.Column("user_agent", sa.Text(), nullable=True),
|
||||
sa.Column("detail_json", sa.Text(), nullable=True),
|
||||
)
|
||||
|
||||
op.create_index("idx_admin_audit_created_at", "admin_audit", ["created_at"], unique=False)
|
||||
op.create_index("idx_admin_audit_action", "admin_audit", ["action"], unique=False)
|
||||
op.create_index("idx_admin_audit_resource", "admin_audit", ["resource"], unique=False)
|
||||
op.create_index("idx_admin_audit_record_id", "admin_audit", ["record_id"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_admin_audit_record_id", table_name="admin_audit")
|
||||
op.drop_index("idx_admin_audit_resource", table_name="admin_audit")
|
||||
op.drop_index("idx_admin_audit_action", table_name="admin_audit")
|
||||
op.drop_index("idx_admin_audit_created_at", table_name="admin_audit")
|
||||
op.drop_table("admin_audit")
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260211_0015"
|
||||
down_revision = "20260210_0014"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"api_keys",
|
||||
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at",
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
),
|
||||
sa.Column("name", sa.Text(), nullable=False),
|
||||
sa.Column("description", sa.Text(), nullable=True),
|
||||
sa.Column("key_hash", sa.Text(), nullable=False),
|
||||
sa.Column("hint", sa.Text(), nullable=False, server_default=sa.text("''")),
|
||||
sa.Column("enabled", sa.Integer(), nullable=False, server_default=sa.text("1")),
|
||||
sa.Column("last_used_at", sa.Text(), nullable=True),
|
||||
sa.UniqueConstraint("name", name="uq_api_keys_name"),
|
||||
sa.CheckConstraint("enabled IN (0,1)", name="ck_api_keys_enabled"),
|
||||
)
|
||||
|
||||
op.create_index("idx_api_keys_enabled", "api_keys", ["enabled"], unique=False)
|
||||
op.create_index("idx_api_keys_created_at", "api_keys", ["created_at"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_api_keys_created_at", table_name="api_keys")
|
||||
op.drop_index("idx_api_keys_enabled", table_name="api_keys")
|
||||
op.drop_table("api_keys")
|
||||
@@ -0,0 +1,134 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260211_0016"
|
||||
down_revision = "20260211_0015"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _try_create_fts5(conn: sa.Connection, *, table_name: str, columns_sql: str) -> bool:
|
||||
candidates = [
|
||||
f"CREATE VIRTUAL TABLE {table_name} USING fts5({columns_sql}, tokenize='trigram');",
|
||||
f"CREATE VIRTUAL TABLE {table_name} USING fts5({columns_sql});",
|
||||
]
|
||||
for sql in candidates:
|
||||
try:
|
||||
conn.exec_driver_sql(sql)
|
||||
return True
|
||||
except Exception:
|
||||
continue
|
||||
return False
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
conn = op.get_bind()
|
||||
|
||||
tags_ok = _try_create_fts5(conn, table_name="tags_fts", columns_sql="name, translated_name")
|
||||
if tags_ok:
|
||||
conn.exec_driver_sql(
|
||||
"INSERT INTO tags_fts(rowid, name, translated_name) "
|
||||
"SELECT id, name, COALESCE(translated_name,'') FROM tags;"
|
||||
)
|
||||
conn.exec_driver_sql(
|
||||
"""
|
||||
CREATE TRIGGER tags_ai_fts AFTER INSERT ON tags BEGIN
|
||||
INSERT INTO tags_fts(rowid, name, translated_name)
|
||||
VALUES (new.id, new.name, COALESCE(new.translated_name,''));
|
||||
END;
|
||||
""".strip()
|
||||
)
|
||||
conn.exec_driver_sql(
|
||||
"""
|
||||
CREATE TRIGGER tags_ad_fts AFTER DELETE ON tags BEGIN
|
||||
DELETE FROM tags_fts WHERE rowid = old.id;
|
||||
END;
|
||||
""".strip()
|
||||
)
|
||||
conn.exec_driver_sql(
|
||||
"""
|
||||
CREATE TRIGGER tags_au_fts AFTER UPDATE OF name, translated_name ON tags BEGIN
|
||||
DELETE FROM tags_fts WHERE rowid = old.id;
|
||||
INSERT INTO tags_fts(rowid, name, translated_name)
|
||||
VALUES (new.id, new.name, COALESCE(new.translated_name,''));
|
||||
END;
|
||||
""".strip()
|
||||
)
|
||||
|
||||
authors_ok = _try_create_fts5(conn, table_name="authors_fts", columns_sql="user_name")
|
||||
if authors_ok:
|
||||
conn.exec_driver_sql(
|
||||
"INSERT INTO authors_fts(rowid, user_name) "
|
||||
"SELECT user_id, MAX(user_name) "
|
||||
"FROM images "
|
||||
"WHERE user_id IS NOT NULL AND user_name IS NOT NULL AND status=1 "
|
||||
"GROUP BY user_id;"
|
||||
)
|
||||
conn.exec_driver_sql(
|
||||
"""
|
||||
CREATE TRIGGER images_ai_authors_fts AFTER INSERT ON images
|
||||
WHEN new.user_id IS NOT NULL
|
||||
BEGIN
|
||||
DELETE FROM authors_fts WHERE rowid = new.user_id;
|
||||
INSERT INTO authors_fts(rowid, user_name)
|
||||
SELECT new.user_id, MAX(user_name)
|
||||
FROM images
|
||||
WHERE user_id = new.user_id AND status = 1 AND user_name IS NOT NULL
|
||||
HAVING MAX(user_name) IS NOT NULL;
|
||||
END;
|
||||
""".strip()
|
||||
)
|
||||
conn.exec_driver_sql(
|
||||
"""
|
||||
CREATE TRIGGER images_au_authors_fts AFTER UPDATE OF user_id, user_name, status ON images
|
||||
BEGIN
|
||||
DELETE FROM authors_fts WHERE rowid = old.user_id;
|
||||
INSERT INTO authors_fts(rowid, user_name)
|
||||
SELECT old.user_id, MAX(user_name)
|
||||
FROM images
|
||||
WHERE old.user_id IS NOT NULL AND user_id = old.user_id AND status = 1 AND user_name IS NOT NULL
|
||||
HAVING MAX(user_name) IS NOT NULL;
|
||||
|
||||
DELETE FROM authors_fts
|
||||
WHERE rowid = new.user_id AND new.user_id IS NOT NULL AND (old.user_id IS NULL OR new.user_id != old.user_id);
|
||||
INSERT INTO authors_fts(rowid, user_name)
|
||||
SELECT new.user_id, MAX(user_name)
|
||||
FROM images
|
||||
WHERE new.user_id IS NOT NULL
|
||||
AND (old.user_id IS NULL OR new.user_id != old.user_id)
|
||||
AND user_id = new.user_id AND status = 1 AND user_name IS NOT NULL
|
||||
HAVING MAX(user_name) IS NOT NULL;
|
||||
END;
|
||||
""".strip()
|
||||
)
|
||||
conn.exec_driver_sql(
|
||||
"""
|
||||
CREATE TRIGGER images_ad_authors_fts AFTER DELETE ON images
|
||||
WHEN old.user_id IS NOT NULL
|
||||
BEGIN
|
||||
DELETE FROM authors_fts WHERE rowid = old.user_id;
|
||||
INSERT INTO authors_fts(rowid, user_name)
|
||||
SELECT old.user_id, MAX(user_name)
|
||||
FROM images
|
||||
WHERE user_id = old.user_id AND status = 1 AND user_name IS NOT NULL
|
||||
HAVING MAX(user_name) IS NOT NULL;
|
||||
END;
|
||||
""".strip()
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
conn = op.get_bind()
|
||||
|
||||
conn.exec_driver_sql("DROP TRIGGER IF EXISTS images_ad_authors_fts;")
|
||||
conn.exec_driver_sql("DROP TRIGGER IF EXISTS images_au_authors_fts;")
|
||||
conn.exec_driver_sql("DROP TRIGGER IF EXISTS images_ai_authors_fts;")
|
||||
conn.exec_driver_sql("DROP TABLE IF EXISTS authors_fts;")
|
||||
|
||||
conn.exec_driver_sql("DROP TRIGGER IF EXISTS tags_au_fts;")
|
||||
conn.exec_driver_sql("DROP TRIGGER IF EXISTS tags_ad_fts;")
|
||||
conn.exec_driver_sql("DROP TRIGGER IF EXISTS tags_ai_fts;")
|
||||
conn.exec_driver_sql("DROP TABLE IF EXISTS tags_fts;")
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260215_0017"
|
||||
down_revision = "20260211_0016"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("images", sa.Column("bookmark_count", sa.Integer(), nullable=True))
|
||||
op.add_column("images", sa.Column("view_count", sa.Integer(), nullable=True))
|
||||
op.add_column("images", sa.Column("comment_count", sa.Integer(), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("images") as batch_op:
|
||||
batch_op.drop_column("comment_count")
|
||||
batch_op.drop_column("view_count")
|
||||
batch_op.drop_column("bookmark_count")
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "20260218_0018"
|
||||
down_revision = "20260215_0017"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("images", sa.Column("illust_type", sa.Integer(), nullable=True))
|
||||
op.create_index("idx_images_illust_type_random", "images", ["status", "illust_type", "random_key"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_images_illust_type_random", table_name="images")
|
||||
with op.batch_alter_table("images") as batch_op:
|
||||
batch_op.drop_column("illust_type")
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,202 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import sqlalchemy as sa
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.api_keys import api_key_hint, hmac_sha256_hex
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.core.time import iso_utc_ms
|
||||
from app.db.models.api_keys import ApiKey
|
||||
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _as_bool(value: Any) -> bool | None:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, int) and value in (0, 1):
|
||||
return bool(value)
|
||||
if isinstance(value, str):
|
||||
v = value.strip().lower()
|
||||
if v in {"true", "1", "yes", "y", "on"}:
|
||||
return True
|
||||
if v in {"false", "0", "no", "n", "off"}:
|
||||
return False
|
||||
return None
|
||||
|
||||
|
||||
async def _load_json_object(request: Request) -> dict[str, Any]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
return data
|
||||
|
||||
|
||||
@router.get("/api-keys")
|
||||
async def list_api_keys(
|
||||
request: Request,
|
||||
limit: int = 50,
|
||||
cursor: str | None = None,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if limit < 1 or limit > 200:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
|
||||
|
||||
cursor_i: int | None = None
|
||||
cursor_raw = (cursor or "").strip()
|
||||
if cursor_raw:
|
||||
if not cursor_raw.isdigit():
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
cursor_i = int(cursor_raw)
|
||||
if cursor_i <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
stmt = sa.select(ApiKey).order_by(ApiKey.id.desc()).limit(limit + 1)
|
||||
if cursor_i is not None:
|
||||
stmt = stmt.where(ApiKey.id < cursor_i)
|
||||
|
||||
async with Session() as session:
|
||||
rows = ((await session.execute(stmt)).scalars().all())
|
||||
|
||||
items_rows = rows[:limit]
|
||||
next_cursor = int(items_rows[-1].id) if len(rows) > limit and items_rows else None
|
||||
|
||||
items = [
|
||||
{
|
||||
"id": str(row.id),
|
||||
"name": row.name,
|
||||
"description": row.description,
|
||||
"enabled": bool(row.enabled),
|
||||
"hint": row.hint,
|
||||
"created_at": row.created_at,
|
||||
"updated_at": row.updated_at,
|
||||
"last_used_at": row.last_used_at,
|
||||
}
|
||||
for row in items_rows
|
||||
]
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"items": items,
|
||||
"next_cursor": str(next_cursor) if next_cursor is not None else "",
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/api-keys")
|
||||
async def create_api_key(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
body = await _load_json_object(request)
|
||||
name = str(body.get("name") or "").strip()
|
||||
api_key = str(body.get("api_key") or "").strip()
|
||||
description = str(body.get("description") or "").strip() or None
|
||||
enabled_v = _as_bool(body.get("enabled"))
|
||||
enabled = bool(enabled_v) if enabled_v is not None else True
|
||||
|
||||
if not name or len(name) > 100:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid name", status_code=400)
|
||||
if not api_key or len(api_key) < 20 or len(api_key) > 500:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid api_key", status_code=400)
|
||||
if description is not None and len(description) > 1000:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid description", status_code=400)
|
||||
|
||||
settings = request.app.state.settings
|
||||
try:
|
||||
key_hash = hmac_sha256_hex(secret_key=settings.secret_key, message=api_key)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="API key hashing not configured", status_code=500) from exc
|
||||
|
||||
hint = api_key_hint(api_key)
|
||||
now = iso_utc_ms()
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> int:
|
||||
async with Session() as session:
|
||||
row = ApiKey(
|
||||
name=name,
|
||||
description=description,
|
||||
key_hash=key_hash,
|
||||
hint=hint,
|
||||
enabled=1 if enabled else 0,
|
||||
updated_at=now,
|
||||
)
|
||||
session.add(row)
|
||||
try:
|
||||
await session.commit()
|
||||
except IntegrityError as exc:
|
||||
await session.rollback()
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="API key name exists", status_code=400) from exc
|
||||
await session.refresh(row)
|
||||
return int(row.id)
|
||||
|
||||
api_key_id = await with_sqlite_busy_retry(_op)
|
||||
return {"ok": True, "api_key_id": str(api_key_id), "hint": hint, "request_id": rid}
|
||||
|
||||
|
||||
@router.put("/api-keys/{api_key_id}")
|
||||
async def update_api_key(
|
||||
api_key_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if api_key_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid api_key_id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
body = await _load_json_object(request)
|
||||
|
||||
enabled_v = _as_bool(body.get("enabled")) if "enabled" in body else None
|
||||
description_raw = body.get("description") if "description" in body else None
|
||||
description = str(description_raw).strip() if isinstance(description_raw, str) else None
|
||||
if description is not None and not description:
|
||||
description = None
|
||||
if description is not None and len(description) > 1000:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid description", status_code=400)
|
||||
|
||||
if enabled_v is None and "description" not in body:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing fields", status_code=400)
|
||||
|
||||
now = iso_utc_ms()
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with Session() as session:
|
||||
row = await session.get(ApiKey, int(api_key_id))
|
||||
if row is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="API key not found", status_code=404)
|
||||
if enabled_v is not None:
|
||||
row.enabled = 1 if bool(enabled_v) else 0
|
||||
if "description" in body:
|
||||
row.description = description
|
||||
row.updated_at = now
|
||||
await session.commit()
|
||||
|
||||
return {"ok": True, "api_key_id": str(api_key_id), "request_id": rid}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import sqlalchemy as sa
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.db.models.admin_audit import AdminAudit
|
||||
from app.db.session import create_sessionmaker
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/audit")
|
||||
async def list_admin_audit(
|
||||
request: Request,
|
||||
limit: int = 50,
|
||||
cursor: str | None = None,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if limit < 1 or limit > 200:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
|
||||
|
||||
cursor_i: int | None = None
|
||||
cursor_raw = (cursor or "").strip()
|
||||
if cursor_raw:
|
||||
if not cursor_raw.isdigit():
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
cursor_i = int(cursor_raw)
|
||||
if cursor_i <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
stmt = sa.select(AdminAudit).order_by(AdminAudit.id.desc()).limit(limit + 1)
|
||||
if cursor_i is not None:
|
||||
stmt = stmt.where(AdminAudit.id < cursor_i)
|
||||
|
||||
async with Session() as session:
|
||||
rows = (await session.execute(stmt)).scalars().all()
|
||||
|
||||
items_rows = rows[:limit]
|
||||
next_cursor = int(items_rows[-1].id) if len(rows) > limit and items_rows else None
|
||||
|
||||
def _parse_detail(text: str | None) -> dict[str, Any] | None:
|
||||
raw = str(text or "").strip()
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except Exception:
|
||||
return {"raw": raw}
|
||||
return data if isinstance(data, dict) else {"value": data}
|
||||
|
||||
items = [
|
||||
{
|
||||
"id": str(row.id),
|
||||
"created_at": row.created_at,
|
||||
"actor": row.actor,
|
||||
"action": row.action,
|
||||
"resource": row.resource,
|
||||
"record_id": row.record_id,
|
||||
"request_id": row.request_id,
|
||||
"detail_json": _parse_detail(row.detail_json),
|
||||
}
|
||||
for row in items_rows
|
||||
]
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"items": items,
|
||||
"next_cursor": str(next_cursor) if next_cursor is not None else "",
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hmac
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.core.security import create_jwt
|
||||
from fastapi import Depends
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
async def _load_login_json(request: Request) -> tuple[str, str]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
|
||||
username = str(data.get("username") or "").strip()
|
||||
password = str(data.get("password") or "")
|
||||
if not username or not password:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing credentials", status_code=400)
|
||||
|
||||
return username, password
|
||||
|
||||
|
||||
@router.post("/login")
|
||||
async def login(request: Request) -> dict[str, Any]:
|
||||
username, password = await _load_login_json(request)
|
||||
settings = request.app.state.settings
|
||||
|
||||
if username != settings.admin_username or not hmac.compare_digest(password, settings.admin_password):
|
||||
raise ApiError(code=ErrorCode.UNAUTHORIZED, message="Invalid credentials", status_code=401)
|
||||
|
||||
token = create_jwt(secret_key=settings.secret_key, subject=settings.admin_username, ttl_s=3600)
|
||||
rid = get_or_create_request_id(request)
|
||||
return {"ok": True, "token": token, "request_id": rid}
|
||||
|
||||
|
||||
@router.post("/logout")
|
||||
async def logout(request: Request, _claims: dict[str, Any] = Depends(get_admin_claims)) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
return {"ok": True, "request_id": rid}
|
||||
@@ -0,0 +1,403 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any
|
||||
|
||||
import sqlalchemy as sa
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
|
||||
from sqlalchemy.orm import aliased
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.bindings_recompute import recompute_token_proxy_bindings
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.core.time import iso_utc_ms
|
||||
from app.db.models.pixiv_tokens import PixivToken
|
||||
from app.db.models.proxy_endpoints import ProxyEndpoint
|
||||
from app.db.models.proxy_pool_endpoints import ProxyPoolEndpoint
|
||||
from app.db.models.proxy_pools import ProxyPool
|
||||
from app.db.models.token_proxy_bindings import TokenProxyBinding
|
||||
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _parse_bool(value: Any, *, default: bool) -> bool:
|
||||
if value is None:
|
||||
return bool(default)
|
||||
if isinstance(value, bool):
|
||||
return bool(value)
|
||||
if isinstance(value, int) and value in (0, 1):
|
||||
return bool(value)
|
||||
if isinstance(value, str):
|
||||
v = value.strip().lower()
|
||||
if v in {"1", "true", "yes", "y", "on"}:
|
||||
return True
|
||||
if v in {"0", "false", "no", "n", "off"}:
|
||||
return False
|
||||
return bool(default)
|
||||
|
||||
|
||||
def _fnv1a64(text: str) -> int:
|
||||
h = 14695981039346656037
|
||||
prime = 1099511628211
|
||||
for b in text.encode("utf-8"):
|
||||
h ^= b
|
||||
h = (h * prime) & 0xFFFFFFFFFFFFFFFF
|
||||
return h
|
||||
|
||||
|
||||
def _rendezvous_proxy_order(*, token_id: int, proxy_ids: list[int], salt: str) -> list[int]:
|
||||
scored = [(_fnv1a64(f"{token_id}|{pid}|{salt}"), pid) for pid in proxy_ids]
|
||||
scored.sort(key=lambda x: (-x[0], x[1]))
|
||||
return [pid for _, pid in scored]
|
||||
|
||||
|
||||
def _compute_primary_assignments(
|
||||
*,
|
||||
token_ids: list[int],
|
||||
proxy_ids: list[int],
|
||||
capacity_by_proxy_id: dict[int, int],
|
||||
salt: str,
|
||||
) -> dict[int, int]:
|
||||
remaining = {pid: int(capacity_by_proxy_id.get(int(pid), 0)) for pid in proxy_ids}
|
||||
out: dict[int, int] = {}
|
||||
for token_id in token_ids:
|
||||
for pid in _rendezvous_proxy_order(token_id=token_id, proxy_ids=proxy_ids, salt=salt):
|
||||
if remaining.get(pid, 0) > 0:
|
||||
out[token_id] = pid
|
||||
remaining[pid] -= 1
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
def _compute_primary_assignments_soft(
|
||||
*,
|
||||
token_ids: list[int],
|
||||
proxy_ids: list[int],
|
||||
capacity_by_proxy_id: dict[int, int],
|
||||
salt: str,
|
||||
) -> tuple[dict[int, int], int]:
|
||||
out = _compute_primary_assignments(
|
||||
token_ids=token_ids,
|
||||
proxy_ids=proxy_ids,
|
||||
capacity_by_proxy_id=capacity_by_proxy_id,
|
||||
salt=salt,
|
||||
)
|
||||
|
||||
over_capacity = 0
|
||||
for token_id in token_ids:
|
||||
if token_id in out:
|
||||
continue
|
||||
order = _rendezvous_proxy_order(token_id=token_id, proxy_ids=proxy_ids, salt=salt)
|
||||
if not order:
|
||||
continue
|
||||
out[token_id] = int(order[0])
|
||||
over_capacity += 1
|
||||
|
||||
return out, int(over_capacity)
|
||||
|
||||
|
||||
async def _load_recompute_json(request: Request) -> dict[str, Any]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
|
||||
try:
|
||||
pool_id = int(data.get("pool_id"))
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid pool_id", status_code=400) from exc
|
||||
if pool_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid pool_id", status_code=400)
|
||||
|
||||
raw_max = data.get("max_tokens_per_proxy", 2)
|
||||
try:
|
||||
max_tokens_per_proxy = int(raw_max)
|
||||
except Exception as exc:
|
||||
raise ApiError(
|
||||
code=ErrorCode.BAD_REQUEST,
|
||||
message="Invalid max_tokens_per_proxy",
|
||||
status_code=400,
|
||||
) from exc
|
||||
|
||||
if max_tokens_per_proxy <= 0 or max_tokens_per_proxy > 1000:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid max_tokens_per_proxy", status_code=400)
|
||||
|
||||
strict = _parse_bool(data.get("strict"), default=True)
|
||||
|
||||
return {"pool_id": pool_id, "max_tokens_per_proxy": max_tokens_per_proxy, "strict": strict}
|
||||
|
||||
|
||||
async def _load_override_json(request: Request) -> dict[str, Any]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
|
||||
try:
|
||||
override_proxy_id = int(data.get("override_proxy_id"))
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid override_proxy_id", status_code=400) from exc
|
||||
if override_proxy_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid override_proxy_id", status_code=400)
|
||||
|
||||
try:
|
||||
ttl_ms = int(data.get("ttl_ms"))
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid ttl_ms", status_code=400) from exc
|
||||
if ttl_ms <= 0 or ttl_ms > 30 * 24 * 60 * 60 * 1000:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid ttl_ms", status_code=400)
|
||||
|
||||
reason = str(data.get("reason") or "").strip()
|
||||
if reason and len(reason) > 200:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid reason", status_code=400)
|
||||
|
||||
return {"override_proxy_id": override_proxy_id, "ttl_ms": ttl_ms, "reason": reason}
|
||||
|
||||
|
||||
@router.get("/bindings")
|
||||
async def list_bindings(
|
||||
request: Request,
|
||||
pool_id: int,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if int(pool_id) <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid pool_id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
now = iso_utc_ms()
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
Primary = aliased(ProxyEndpoint)
|
||||
Override = aliased(ProxyEndpoint)
|
||||
|
||||
async with Session() as session:
|
||||
rows = (
|
||||
(
|
||||
await session.execute(
|
||||
sa.select(TokenProxyBinding, PixivToken.label, ProxyPool.name, Primary, Override)
|
||||
.join(PixivToken, PixivToken.id == TokenProxyBinding.token_id)
|
||||
.join(ProxyPool, ProxyPool.id == TokenProxyBinding.pool_id)
|
||||
.join(Primary, Primary.id == TokenProxyBinding.primary_proxy_id)
|
||||
.outerjoin(Override, Override.id == TokenProxyBinding.override_proxy_id)
|
||||
.where(TokenProxyBinding.pool_id == int(pool_id))
|
||||
.order_by(TokenProxyBinding.id.asc())
|
||||
)
|
||||
)
|
||||
.all()
|
||||
)
|
||||
|
||||
counts = (
|
||||
(
|
||||
await session.execute(
|
||||
sa.select(
|
||||
sa.func.count(ProxyPoolEndpoint.endpoint_id),
|
||||
sa.func.sum(
|
||||
sa.case(
|
||||
(sa.and_(ProxyPoolEndpoint.enabled == 1, ProxyEndpoint.enabled == 1), 1),
|
||||
else_=0,
|
||||
)
|
||||
),
|
||||
)
|
||||
.select_from(ProxyPoolEndpoint)
|
||||
.join(ProxyEndpoint, ProxyEndpoint.id == ProxyPoolEndpoint.endpoint_id)
|
||||
.where(ProxyPoolEndpoint.pool_id == int(pool_id))
|
||||
)
|
||||
)
|
||||
.first()
|
||||
)
|
||||
endpoints_total = int(counts[0] or 0) if counts is not None else 0
|
||||
endpoints_enabled = int(counts[1] or 0) if counts is not None else 0
|
||||
|
||||
items: list[dict[str, Any]] = []
|
||||
for binding, token_label, pool_name, primary_proxy, override_proxy in rows:
|
||||
override_active = False
|
||||
if binding.override_proxy_id is not None and binding.override_expires_at:
|
||||
override_active = str(binding.override_expires_at) > now
|
||||
|
||||
effective_proxy_id = binding.override_proxy_id if override_active else binding.primary_proxy_id
|
||||
effective_mode = "override" if override_active else "primary"
|
||||
|
||||
items.append(
|
||||
{
|
||||
"id": str(binding.id),
|
||||
"created_at": binding.created_at,
|
||||
"updated_at": binding.updated_at,
|
||||
"token": {"id": str(binding.token_id), "label": token_label},
|
||||
"pool": {"id": str(binding.pool_id), "name": pool_name},
|
||||
"primary_proxy": {
|
||||
"id": str(primary_proxy.id),
|
||||
"scheme": primary_proxy.scheme,
|
||||
"host": primary_proxy.host,
|
||||
"port": int(primary_proxy.port),
|
||||
"username": primary_proxy.username,
|
||||
},
|
||||
"override_proxy": (
|
||||
{
|
||||
"id": str(override_proxy.id),
|
||||
"scheme": override_proxy.scheme,
|
||||
"host": override_proxy.host,
|
||||
"port": int(override_proxy.port),
|
||||
"username": override_proxy.username,
|
||||
}
|
||||
if override_proxy is not None
|
||||
else None
|
||||
),
|
||||
"override_expires_at": binding.override_expires_at,
|
||||
"effective_proxy_id": str(effective_proxy_id),
|
||||
"effective_mode": effective_mode,
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"items": items,
|
||||
"summary": {
|
||||
"pool_id": str(pool_id),
|
||||
"pool_endpoints_total": int(endpoints_total),
|
||||
"pool_endpoints_enabled": int(endpoints_enabled),
|
||||
},
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/bindings/recompute")
|
||||
async def recompute_bindings(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
now = iso_utc_ms()
|
||||
body = await _load_recompute_json(request)
|
||||
|
||||
pool_id = int(body["pool_id"])
|
||||
max_tokens_per_proxy = int(body["max_tokens_per_proxy"])
|
||||
strict = bool(body.get("strict", True))
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with Session() as session:
|
||||
pool = await session.get(ProxyPool, pool_id)
|
||||
if pool is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Proxy pool not found", status_code=404)
|
||||
|
||||
result = await recompute_token_proxy_bindings(
|
||||
session,
|
||||
pool_id=int(pool_id),
|
||||
now=str(now),
|
||||
max_tokens_per_proxy=int(max_tokens_per_proxy),
|
||||
strict=bool(strict),
|
||||
)
|
||||
await session.commit()
|
||||
|
||||
return {"ok": True, "pool_id": str(pool_id), "request_id": rid, **result}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
@router.post("/bindings/{binding_id}/override")
|
||||
async def set_binding_override(
|
||||
binding_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if binding_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid binding id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
now = iso_utc_ms()
|
||||
body = await _load_override_json(request)
|
||||
|
||||
override_proxy_id = int(body["override_proxy_id"])
|
||||
ttl_ms = int(body["ttl_ms"])
|
||||
_reason = str(body["reason"] or "")
|
||||
|
||||
expires_at = iso_utc_ms(datetime.now(timezone.utc) + timedelta(milliseconds=ttl_ms))
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with Session() as session:
|
||||
binding = await session.get(TokenProxyBinding, binding_id)
|
||||
if binding is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Binding not found", status_code=404)
|
||||
|
||||
proxy = await session.get(ProxyEndpoint, override_proxy_id)
|
||||
if proxy is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Proxy endpoint not found", status_code=404)
|
||||
if not bool(proxy.enabled):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Proxy endpoint disabled", status_code=400)
|
||||
|
||||
in_pool = (
|
||||
await session.execute(
|
||||
sa.select(ProxyPoolEndpoint)
|
||||
.where(ProxyPoolEndpoint.pool_id == int(binding.pool_id))
|
||||
.where(ProxyPoolEndpoint.endpoint_id == int(override_proxy_id))
|
||||
.where(ProxyPoolEndpoint.enabled == 1)
|
||||
)
|
||||
).scalars().first()
|
||||
if in_pool is None:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Override proxy not in pool", status_code=400)
|
||||
|
||||
binding.override_proxy_id = int(override_proxy_id)
|
||||
binding.override_expires_at = expires_at
|
||||
binding.updated_at = now
|
||||
await session.commit()
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"binding_id": str(binding_id),
|
||||
"override_proxy_id": str(override_proxy_id),
|
||||
"override_expires_at": expires_at,
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
@router.post("/bindings/{binding_id}/clear-override")
|
||||
async def clear_binding_override(
|
||||
binding_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if binding_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid binding id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
now = iso_utc_ms()
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with Session() as session:
|
||||
binding = await session.get(TokenProxyBinding, binding_id)
|
||||
if binding is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Binding not found", status_code=404)
|
||||
|
||||
binding.override_proxy_id = None
|
||||
binding.override_expires_at = None
|
||||
binding.updated_at = now
|
||||
await session.commit()
|
||||
|
||||
return {"ok": True, "binding_id": str(binding_id), "request_id": rid}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
@@ -0,0 +1,17 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from fastapi import Request
|
||||
|
||||
from app.core.security import require_admin
|
||||
|
||||
|
||||
def get_admin_claims(request: Request) -> dict[str, Any]:
|
||||
settings = request.app.state.settings
|
||||
return require_admin(
|
||||
request.headers,
|
||||
secret_key=settings.secret_key,
|
||||
admin_username=settings.admin_username,
|
||||
)
|
||||
|
||||
@@ -0,0 +1,500 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import sqlalchemy as sa
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.core.time import iso_utc_ms
|
||||
from app.db.models.hydration_runs import HydrationRun
|
||||
from app.db.models.images import Image
|
||||
from app.db.models.jobs import JobRow
|
||||
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
_ALLOWED_RUN_STATUSES = {"pending", "running", "paused", "canceled", "completed", "failed"}
|
||||
|
||||
|
||||
def _parse_json_dict(value: str | None) -> dict[str, Any]:
|
||||
raw = str(value or "").strip()
|
||||
if not raw:
|
||||
return {}
|
||||
try:
|
||||
loaded = json.loads(raw)
|
||||
except Exception:
|
||||
return {}
|
||||
if isinstance(loaded, dict):
|
||||
return loaded
|
||||
return {}
|
||||
|
||||
|
||||
def _serialize_job(job: JobRow | None) -> dict[str, Any] | None:
|
||||
if job is None:
|
||||
return None
|
||||
return {
|
||||
"id": str(job.id),
|
||||
"status": job.status,
|
||||
"attempt": int(job.attempt),
|
||||
"max_attempts": int(job.max_attempts),
|
||||
"run_after": job.run_after,
|
||||
"last_error": job.last_error,
|
||||
"locked_by": job.locked_by,
|
||||
"locked_at": job.locked_at,
|
||||
"updated_at": job.updated_at,
|
||||
}
|
||||
|
||||
|
||||
def _serialize_run(run: HydrationRun, *, latest_job: JobRow | None = None) -> dict[str, Any]:
|
||||
return {
|
||||
"id": str(run.id),
|
||||
"type": run.type,
|
||||
"status": run.status,
|
||||
"criteria": _parse_json_dict(run.criteria_json),
|
||||
"cursor": _parse_json_dict(run.cursor_json),
|
||||
"total": int(run.total) if run.total is not None else None,
|
||||
"processed": int(run.processed),
|
||||
"success": int(run.success),
|
||||
"failed": int(run.failed),
|
||||
"started_at": run.started_at,
|
||||
"finished_at": run.finished_at,
|
||||
"last_error": run.last_error,
|
||||
"created_at": run.created_at,
|
||||
"updated_at": run.updated_at,
|
||||
"latest_job": _serialize_job(latest_job),
|
||||
}
|
||||
|
||||
|
||||
async def _latest_jobs_by_run_ids(session, *, run_ids: list[str]) -> dict[str, JobRow]:
|
||||
ids = [str(v).strip() for v in run_ids if str(v).strip()]
|
||||
if not ids:
|
||||
return {}
|
||||
|
||||
latest_per_ref = (
|
||||
sa.select(JobRow.ref_id.label("ref_id"), sa.func.max(JobRow.id).label("max_job_id"))
|
||||
.where(JobRow.ref_type == "hydration_run", JobRow.ref_id.in_(ids))
|
||||
.group_by(JobRow.ref_id)
|
||||
.subquery()
|
||||
)
|
||||
|
||||
rows = (
|
||||
await session.execute(
|
||||
sa.select(JobRow, latest_per_ref.c.ref_id)
|
||||
.join(latest_per_ref, JobRow.id == latest_per_ref.c.max_job_id)
|
||||
.order_by(JobRow.id.desc())
|
||||
)
|
||||
).all()
|
||||
|
||||
out: dict[str, JobRow] = {}
|
||||
for job, ref_id in rows:
|
||||
key = str(ref_id or "").strip()
|
||||
if not key:
|
||||
continue
|
||||
out[key] = job
|
||||
return out
|
||||
|
||||
|
||||
async def _load_create_json(request: Request) -> dict[str, Any]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
|
||||
run_type = str(data.get("type") or "backfill").strip().lower() or "backfill"
|
||||
if run_type not in {"backfill", "manual"}:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid type", status_code=400)
|
||||
|
||||
criteria = data.get("criteria")
|
||||
if criteria is None:
|
||||
criteria = {}
|
||||
if not isinstance(criteria, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid criteria", status_code=400)
|
||||
|
||||
return {"type": run_type, "criteria": criteria}
|
||||
|
||||
|
||||
async def _load_manual_job_json(request: Request) -> dict[str, int | None]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
|
||||
illust_id_raw = data.get("illust_id")
|
||||
image_id_raw = data.get("image_id")
|
||||
|
||||
if illust_id_raw is None and image_id_raw is None:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Either illust_id or image_id is required", status_code=400)
|
||||
|
||||
if illust_id_raw is not None and image_id_raw is not None:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Provide only one of illust_id or image_id", status_code=400)
|
||||
|
||||
illust_id: int | None = None
|
||||
image_id: int | None = None
|
||||
if illust_id_raw is not None:
|
||||
try:
|
||||
illust_id = int(illust_id_raw)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid illust_id", status_code=400) from exc
|
||||
if illust_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid illust_id", status_code=400)
|
||||
|
||||
if image_id_raw is not None:
|
||||
try:
|
||||
image_id = int(image_id_raw)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid image_id", status_code=400) from exc
|
||||
if image_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid image_id", status_code=400)
|
||||
|
||||
return {"illust_id": illust_id, "image_id": image_id}
|
||||
|
||||
|
||||
@router.get("/hydration-runs")
|
||||
async def list_hydration_runs(
|
||||
request: Request,
|
||||
limit: int = 20,
|
||||
cursor: str | None = None,
|
||||
status: str | None = None,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if limit < 1 or limit > 200:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
|
||||
|
||||
cursor_i: int | None = None
|
||||
cursor_raw = str(cursor or "").strip()
|
||||
if cursor_raw:
|
||||
if not cursor_raw.isdigit():
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
cursor_i = int(cursor_raw)
|
||||
if cursor_i <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
|
||||
status_norm = str(status or "").strip().lower() or None
|
||||
if status_norm is not None and status_norm not in _ALLOWED_RUN_STATUSES:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported status", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async with Session() as session:
|
||||
stmt = sa.select(HydrationRun).order_by(HydrationRun.id.desc()).limit(limit + 1)
|
||||
if cursor_i is not None:
|
||||
stmt = stmt.where(HydrationRun.id < cursor_i)
|
||||
if status_norm is not None:
|
||||
stmt = stmt.where(HydrationRun.status == status_norm)
|
||||
|
||||
rows = (await session.execute(stmt)).scalars().all()
|
||||
current_rows = rows[:limit]
|
||||
run_ids = [str(int(r.id)) for r in current_rows]
|
||||
jobs_by_run_id = await _latest_jobs_by_run_ids(session, run_ids=run_ids)
|
||||
|
||||
next_cursor = int(current_rows[-1].id) if len(rows) > limit and current_rows else None
|
||||
items = [_serialize_run(row, latest_job=jobs_by_run_id.get(str(int(row.id)))) for row in current_rows]
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"items": items,
|
||||
"next_cursor": str(next_cursor) if next_cursor is not None else "",
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/hydration-runs/{run_id}")
|
||||
async def get_hydration_run(
|
||||
run_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if run_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid hydration_run id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async with Session() as session:
|
||||
run = await session.get(HydrationRun, run_id)
|
||||
if run is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Hydration run not found", status_code=404)
|
||||
|
||||
jobs = await _latest_jobs_by_run_ids(session, run_ids=[str(run_id)])
|
||||
item = _serialize_run(run, latest_job=jobs.get(str(run_id)))
|
||||
|
||||
return {"ok": True, "item": item, "request_id": rid}
|
||||
|
||||
|
||||
@router.post("/hydration-runs/manual")
|
||||
async def create_manual_hydration_job(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
now = iso_utc_ms()
|
||||
body = await _load_manual_job_json(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with Session() as session:
|
||||
illust_id = body.get("illust_id")
|
||||
image_id = body.get("image_id")
|
||||
if illust_id is None and image_id is not None:
|
||||
image = await session.get(Image, int(image_id))
|
||||
if image is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Image not found", status_code=404)
|
||||
illust_id = int(image.illust_id)
|
||||
|
||||
if illust_id is None or int(illust_id) <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid illust_id", status_code=400)
|
||||
|
||||
existing = (
|
||||
(
|
||||
await session.execute(
|
||||
sa.select(JobRow)
|
||||
.where(
|
||||
JobRow.type == "hydrate_metadata",
|
||||
JobRow.ref_type == "manual_hydrate",
|
||||
JobRow.ref_id == str(int(illust_id)),
|
||||
JobRow.status.in_(["pending", "running"]),
|
||||
)
|
||||
.order_by(JobRow.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.first()
|
||||
)
|
||||
|
||||
if existing is not None:
|
||||
return {
|
||||
"ok": True,
|
||||
"created": False,
|
||||
"job_id": str(int(existing.id)),
|
||||
"illust_id": str(int(illust_id)),
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
job = JobRow(
|
||||
type="hydrate_metadata",
|
||||
status="pending",
|
||||
payload_json=json.dumps({"illust_id": int(illust_id)}, separators=(",", ":"), ensure_ascii=False),
|
||||
last_error=None,
|
||||
priority=0,
|
||||
run_after=None,
|
||||
attempt=0,
|
||||
max_attempts=3,
|
||||
locked_by=None,
|
||||
locked_at=None,
|
||||
ref_type="manual_hydrate",
|
||||
ref_id=str(int(illust_id)),
|
||||
updated_at=now,
|
||||
)
|
||||
session.add(job)
|
||||
await session.flush()
|
||||
await session.commit()
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"created": True,
|
||||
"job_id": str(int(job.id)),
|
||||
"illust_id": str(int(illust_id)),
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
@router.post("/hydration-runs")
|
||||
async def create_hydration_run(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
now = iso_utc_ms()
|
||||
body = await _load_create_json(request)
|
||||
|
||||
run_type = str(body["type"])
|
||||
criteria = dict(body["criteria"])
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> tuple[int, int]:
|
||||
async with Session() as session:
|
||||
run = HydrationRun(
|
||||
type=run_type,
|
||||
status="pending",
|
||||
criteria_json=json.dumps(criteria, separators=(",", ":"), ensure_ascii=False),
|
||||
cursor_json=None,
|
||||
total=None,
|
||||
processed=0,
|
||||
success=0,
|
||||
failed=0,
|
||||
started_at=None,
|
||||
finished_at=None,
|
||||
last_error=None,
|
||||
updated_at=now,
|
||||
)
|
||||
session.add(run)
|
||||
await session.flush()
|
||||
|
||||
job = JobRow(
|
||||
type="hydrate_metadata",
|
||||
status="pending",
|
||||
payload_json=json.dumps(
|
||||
{"hydration_run_id": int(run.id), "criteria": criteria},
|
||||
separators=(",", ":"),
|
||||
ensure_ascii=False,
|
||||
),
|
||||
last_error=None,
|
||||
priority=0,
|
||||
run_after=None,
|
||||
attempt=0,
|
||||
max_attempts=3,
|
||||
locked_by=None,
|
||||
locked_at=None,
|
||||
ref_type="hydration_run",
|
||||
ref_id=str(int(run.id)),
|
||||
updated_at=now,
|
||||
)
|
||||
session.add(job)
|
||||
await session.flush()
|
||||
await session.commit()
|
||||
return int(run.id), int(job.id)
|
||||
|
||||
run_id, job_id = await with_sqlite_busy_retry(_op)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"hydration_run_id": str(run_id),
|
||||
"job_id": str(job_id),
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
|
||||
async def _set_run_and_job_status(
|
||||
request: Request,
|
||||
*,
|
||||
run_id: int,
|
||||
target_status: str,
|
||||
allowed_from: set[str],
|
||||
job_status: str,
|
||||
) -> dict[str, Any]:
|
||||
if run_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid hydration_run id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
now = iso_utc_ms()
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with Session() as session:
|
||||
run = await session.get(HydrationRun, run_id)
|
||||
if run is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Hydration run not found", status_code=404)
|
||||
|
||||
if str(run.status) not in allowed_from:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported status transition", status_code=400)
|
||||
|
||||
run.status = target_status
|
||||
run.updated_at = now
|
||||
if target_status in {"canceled", "completed", "failed"} and run.finished_at is None:
|
||||
run.finished_at = now
|
||||
|
||||
job = (
|
||||
(
|
||||
await session.execute(
|
||||
sa.select(JobRow)
|
||||
.where(JobRow.ref_type == "hydration_run", JobRow.ref_id == str(run_id))
|
||||
.order_by(JobRow.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.first()
|
||||
)
|
||||
if job is not None:
|
||||
job.status = job_status
|
||||
if job_status in {"pending", "canceled", "paused", "dlq"}:
|
||||
job.run_after = None
|
||||
job.locked_by = None
|
||||
job.locked_at = None
|
||||
job.updated_at = now
|
||||
|
||||
await session.commit()
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"hydration_run_id": str(run_id),
|
||||
"status": target_status,
|
||||
"job_status": job_status if job is not None else "",
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
@router.post("/hydration-runs/{run_id}/pause")
|
||||
async def pause_hydration_run(
|
||||
run_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
return await _set_run_and_job_status(
|
||||
request,
|
||||
run_id=run_id,
|
||||
target_status="paused",
|
||||
allowed_from={"pending", "running"},
|
||||
job_status="paused",
|
||||
)
|
||||
|
||||
|
||||
@router.post("/hydration-runs/{run_id}/resume")
|
||||
async def resume_hydration_run(
|
||||
run_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
return await _set_run_and_job_status(
|
||||
request,
|
||||
run_id=run_id,
|
||||
target_status="pending",
|
||||
allowed_from={"paused"},
|
||||
job_status="pending",
|
||||
)
|
||||
|
||||
|
||||
@router.post("/hydration-runs/{run_id}/cancel")
|
||||
async def cancel_hydration_run(
|
||||
run_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
return await _set_run_and_job_status(
|
||||
request,
|
||||
run_id=run_id,
|
||||
target_status="canceled",
|
||||
allowed_from={"pending", "running", "paused"},
|
||||
job_status="canceled",
|
||||
)
|
||||
@@ -0,0 +1,343 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import sqlalchemy as sa
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.db.models.image_tags import ImageTag
|
||||
from app.db.models.images import Image
|
||||
from app.db.models.tags import Tag
|
||||
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
_ALLOWED_MISSING = {"tags", "geometry", "r18", "ai", "illust_type", "user", "title", "created_at", "popularity"}
|
||||
|
||||
|
||||
def _parse_missing(values: list[str] | None) -> list[str]:
|
||||
out: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for raw in values or []:
|
||||
for part in str(raw or "").replace(",", "|").split("|"):
|
||||
key = part.strip().lower()
|
||||
if not key or key in seen:
|
||||
continue
|
||||
if key not in _ALLOWED_MISSING:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported missing", status_code=400)
|
||||
seen.add(key)
|
||||
out.append(key)
|
||||
return out
|
||||
|
||||
|
||||
@router.get("/images")
|
||||
async def list_admin_images(
|
||||
request: Request,
|
||||
limit: int = 50,
|
||||
cursor: str | None = None,
|
||||
missing: list[str] | None = Query(default=None),
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if limit < 1 or limit > 200:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
|
||||
|
||||
cursor_i: int | None = None
|
||||
cursor_raw = (cursor or "").strip()
|
||||
if cursor_raw:
|
||||
if not cursor_raw.isdigit():
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
cursor_i = int(cursor_raw)
|
||||
if cursor_i <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
|
||||
missing_keys = _parse_missing(missing)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
tag_counts = (
|
||||
sa.select(ImageTag.image_id.label("image_id"), sa.func.count().label("tag_count"))
|
||||
.group_by(ImageTag.image_id)
|
||||
.subquery()
|
||||
)
|
||||
tag_count_col = sa.func.coalesce(tag_counts.c.tag_count, 0).label("tag_count")
|
||||
|
||||
stmt = (
|
||||
sa.select(Image, tag_count_col)
|
||||
.outerjoin(tag_counts, tag_counts.c.image_id == Image.id)
|
||||
.where(Image.status == 1)
|
||||
.order_by(Image.id.desc())
|
||||
.limit(int(limit) + 1)
|
||||
)
|
||||
if cursor_i is not None:
|
||||
stmt = stmt.where(Image.id < int(cursor_i))
|
||||
|
||||
for key in missing_keys:
|
||||
if key == "tags":
|
||||
stmt = stmt.where(tag_count_col == 0)
|
||||
elif key == "geometry":
|
||||
stmt = stmt.where((Image.width.is_(None)) | (Image.height.is_(None)))
|
||||
elif key == "r18":
|
||||
stmt = stmt.where(Image.x_restrict.is_(None))
|
||||
elif key == "ai":
|
||||
stmt = stmt.where(Image.ai_type.is_(None))
|
||||
elif key == "illust_type":
|
||||
stmt = stmt.where(Image.illust_type.is_(None))
|
||||
elif key == "user":
|
||||
stmt = stmt.where(Image.user_id.is_(None))
|
||||
elif key == "title":
|
||||
stmt = stmt.where((Image.title.is_(None)) | (sa.func.trim(Image.title) == ""))
|
||||
elif key == "created_at":
|
||||
stmt = stmt.where((Image.created_at_pixiv.is_(None)) | (sa.func.trim(Image.created_at_pixiv) == ""))
|
||||
elif key == "popularity":
|
||||
stmt = stmt.where(
|
||||
(Image.bookmark_count.is_(None)) | (Image.view_count.is_(None)) | (Image.comment_count.is_(None))
|
||||
)
|
||||
|
||||
async with Session() as session:
|
||||
rows = (await session.execute(stmt)).all()
|
||||
|
||||
rows_page = rows[: int(limit)]
|
||||
next_cursor = int(rows_page[-1][0].id) if len(rows) > int(limit) and rows_page else None
|
||||
|
||||
items: list[dict[str, Any]] = []
|
||||
for img, tag_count in rows_page:
|
||||
tag_count_i = int(tag_count or 0)
|
||||
missing_list: list[str] = []
|
||||
if tag_count_i <= 0:
|
||||
missing_list.append("tags")
|
||||
if img.width is None or img.height is None:
|
||||
missing_list.append("geometry")
|
||||
if img.x_restrict is None:
|
||||
missing_list.append("r18")
|
||||
if img.ai_type is None:
|
||||
missing_list.append("ai")
|
||||
if getattr(img, "illust_type", None) is None:
|
||||
missing_list.append("illust_type")
|
||||
if img.user_id is None:
|
||||
missing_list.append("user")
|
||||
if img.title is None or not str(img.title).strip():
|
||||
missing_list.append("title")
|
||||
if img.created_at_pixiv is None or not str(img.created_at_pixiv).strip():
|
||||
missing_list.append("created_at")
|
||||
if img.bookmark_count is None or img.view_count is None or img.comment_count is None:
|
||||
missing_list.append("popularity")
|
||||
|
||||
items.append(
|
||||
{
|
||||
"id": str(img.id),
|
||||
"illust_id": str(img.illust_id),
|
||||
"page_index": int(img.page_index),
|
||||
"ext": img.ext,
|
||||
"status": int(img.status),
|
||||
"width": img.width,
|
||||
"height": img.height,
|
||||
"orientation": img.orientation,
|
||||
"x_restrict": img.x_restrict,
|
||||
"ai_type": img.ai_type,
|
||||
"illust_type": getattr(img, "illust_type", None),
|
||||
"bookmark_count": img.bookmark_count,
|
||||
"view_count": img.view_count,
|
||||
"comment_count": img.comment_count,
|
||||
"user": {
|
||||
"id": str(img.user_id) if img.user_id is not None else None,
|
||||
"name": img.user_name,
|
||||
},
|
||||
"title": img.title,
|
||||
"created_at_pixiv": img.created_at_pixiv,
|
||||
"original_url": img.original_url,
|
||||
"proxy_path": img.proxy_path,
|
||||
"tag_count": tag_count_i,
|
||||
"missing": missing_list,
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"items": items,
|
||||
"next_cursor": str(next_cursor) if next_cursor is not None else "",
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
|
||||
async def _load_bulk_delete_json(request: Request) -> dict[str, Any]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
|
||||
raw_ids = data.get("image_ids", None)
|
||||
if raw_ids is None:
|
||||
raw_ids = data.get("ids", None)
|
||||
if raw_ids is None:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing image_ids", status_code=400)
|
||||
if not isinstance(raw_ids, list):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported image_ids", status_code=400)
|
||||
|
||||
ids: list[int] = []
|
||||
seen: set[int] = set()
|
||||
for raw in raw_ids:
|
||||
try:
|
||||
i = int(raw)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported image_ids", status_code=400) from exc
|
||||
if i <= 0 or i in seen:
|
||||
continue
|
||||
seen.add(i)
|
||||
ids.append(i)
|
||||
|
||||
if not ids:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Empty image_ids", status_code=400)
|
||||
if len(ids) > 20_000:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Too many image_ids", status_code=400)
|
||||
|
||||
return {"image_ids": ids}
|
||||
|
||||
|
||||
def _chunks(values: list[int], *, chunk_size: int) -> list[list[int]]:
|
||||
if chunk_size <= 0:
|
||||
return [values]
|
||||
out: list[list[int]] = []
|
||||
for i in range(0, len(values), chunk_size):
|
||||
out.append(values[i : i + chunk_size])
|
||||
return out
|
||||
|
||||
|
||||
def _safe_rowcount(result: Any) -> int:
|
||||
try:
|
||||
rc = int(getattr(result, "rowcount", 0) or 0)
|
||||
except Exception:
|
||||
return 0
|
||||
return rc if rc > 0 else 0
|
||||
|
||||
|
||||
@router.delete("/images/{image_id}")
|
||||
async def delete_admin_image(
|
||||
image_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if int(image_id) <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid image id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with Session() as session:
|
||||
row = await session.get(Image, int(image_id))
|
||||
if row is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Image not found", status_code=404)
|
||||
await session.execute(sa.delete(ImageTag).where(ImageTag.image_id == int(image_id)))
|
||||
await session.delete(row)
|
||||
await session.commit()
|
||||
|
||||
return {"ok": True, "image_id": str(int(image_id)), "request_id": rid}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
@router.post("/images/bulk-delete")
|
||||
async def bulk_delete_admin_images(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
body = await _load_bulk_delete_json(request)
|
||||
ids = list(body["image_ids"])
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
deleted = 0
|
||||
found = 0
|
||||
async with Session() as session:
|
||||
for chunk in _chunks(ids, chunk_size=900):
|
||||
rows = (await session.execute(sa.select(Image.id).where(Image.id.in_(chunk)))).scalars().all()
|
||||
found += len(rows)
|
||||
|
||||
for chunk in _chunks(ids, chunk_size=900):
|
||||
await session.execute(sa.delete(ImageTag).where(ImageTag.image_id.in_(chunk)))
|
||||
result = await session.execute(sa.delete(Image).where(Image.id.in_(chunk)))
|
||||
deleted += _safe_rowcount(result)
|
||||
|
||||
await session.commit()
|
||||
|
||||
missing = max(0, int(len(ids)) - int(found))
|
||||
return {
|
||||
"ok": True,
|
||||
"requested": int(len(ids)),
|
||||
"deleted": int(deleted),
|
||||
"missing": int(missing),
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
async def _load_clear_images_json(request: Request) -> dict[str, Any]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
|
||||
confirm = data.get("confirm", False)
|
||||
if confirm not in {True, 1, "1", "true", "yes", "y", "on"}:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing confirm", status_code=400)
|
||||
|
||||
delete_tags = data.get("delete_tags", True)
|
||||
delete_tags_bool = (
|
||||
bool(delete_tags)
|
||||
if isinstance(delete_tags, (bool, int))
|
||||
else str(delete_tags).strip().lower() in {"1", "true", "yes", "y", "on"}
|
||||
)
|
||||
|
||||
return {"delete_tags": bool(delete_tags_bool)}
|
||||
|
||||
|
||||
@router.post("/images/clear")
|
||||
async def clear_admin_images(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
body = await _load_clear_images_json(request)
|
||||
delete_tags = bool(body["delete_tags"])
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with Session() as session:
|
||||
result_links = await session.execute(sa.delete(ImageTag))
|
||||
result_images = await session.execute(sa.delete(Image))
|
||||
result_tags = None
|
||||
if delete_tags:
|
||||
result_tags = await session.execute(sa.delete(Tag))
|
||||
|
||||
await session.commit()
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"deleted_image_tags": _safe_rowcount(result_links),
|
||||
"deleted_images": _safe_rowcount(result_images),
|
||||
"deleted_tags": _safe_rowcount(result_tags) if result_tags is not None else 0,
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
@@ -0,0 +1,435 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from dataclasses import asdict, dataclass
|
||||
from typing import Any, Literal
|
||||
from uuid import uuid4
|
||||
|
||||
import sqlalchemy as sa
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from pydantic import BaseModel, Field
|
||||
from pydantic import ValidationError
|
||||
from starlette.datastructures import UploadFile
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.data_files import get_sqlite_db_dir, make_file_ref
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.core.time import iso_utc_ms
|
||||
from app.core.pixiv_urls import parse_pixiv_original_url
|
||||
from app.db.models.images import Image
|
||||
from app.db.models.imports import Import
|
||||
from app.db.models.jobs import JobRow
|
||||
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
|
||||
from app.jobs.dispatch import JobDispatcher
|
||||
from app.jobs.executor import execute_claimed_job
|
||||
from app.jobs.handlers.import_images import build_import_images_handler
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
class ImportCreateRequest(BaseModel):
|
||||
text: str = Field(min_length=1)
|
||||
dry_run: bool = False
|
||||
hydrate_on_import: bool = False
|
||||
source: str = "manual"
|
||||
|
||||
|
||||
class ImportRollbackRequest(BaseModel):
|
||||
mode: Literal["disable", "delete"]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ImportErrorItem:
|
||||
line: int
|
||||
url: str
|
||||
code: str
|
||||
message: str
|
||||
|
||||
|
||||
def _max_import_text_bytes() -> int:
|
||||
default = 50 * 1024 * 1024
|
||||
raw = (os.environ.get("IMPORT_MAX_BYTES") or "").strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
value = int(raw)
|
||||
except Exception:
|
||||
return default
|
||||
return max(1024, min(int(value), 50 * 1024 * 1024))
|
||||
|
||||
|
||||
def _import_inline_max_accepted() -> int:
|
||||
default = 200
|
||||
raw = (os.environ.get("IMPORT_INLINE_MAX_ACCEPTED") or "").strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
value = int(raw)
|
||||
except Exception:
|
||||
return default
|
||||
return max(0, min(int(value), 10_000))
|
||||
|
||||
|
||||
async def _claim_job_by_id(engine, *, job_id: int, worker_id: str, now: str) -> dict[str, Any] | None:
|
||||
sql = """
|
||||
UPDATE jobs
|
||||
SET status='running',
|
||||
locked_by=:worker_id,
|
||||
locked_at=:now,
|
||||
updated_at=:now
|
||||
WHERE id=:id AND status='pending'
|
||||
RETURNING *;
|
||||
""".strip()
|
||||
|
||||
async def _op() -> dict[str, Any] | None:
|
||||
async with engine.begin() as conn:
|
||||
result = await conn.exec_driver_sql(sql, {"id": int(job_id), "worker_id": worker_id, "now": now})
|
||||
row = result.mappings().first()
|
||||
return dict(row) if row else None
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
def _parse_import_text(
|
||||
text: str, *, preview_limit: int = 20
|
||||
) -> tuple[int, int, int, int, list[ImportErrorItem], list[dict[str, Any]]]:
|
||||
total = 0
|
||||
accepted = 0
|
||||
deduped = 0
|
||||
error_total = 0
|
||||
errors: list[ImportErrorItem] = []
|
||||
seen: set[tuple[int, int]] = set()
|
||||
preview: list[dict[str, Any]] = []
|
||||
|
||||
for line_no, raw in enumerate(text.splitlines(), start=1):
|
||||
url = raw.strip()
|
||||
if not url:
|
||||
continue
|
||||
total += 1
|
||||
try:
|
||||
parsed = parse_pixiv_original_url(url)
|
||||
except Exception as exc:
|
||||
error_total += 1
|
||||
if len(errors) < 200:
|
||||
errors.append(
|
||||
ImportErrorItem(
|
||||
line=line_no,
|
||||
url=url,
|
||||
code=ErrorCode.UNSUPPORTED_URL.value,
|
||||
message=str(exc) or ErrorCode.UNSUPPORTED_URL.value,
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
key = (parsed.illust_id, parsed.page_index)
|
||||
if key in seen:
|
||||
deduped += 1
|
||||
continue
|
||||
seen.add(key)
|
||||
accepted += 1
|
||||
|
||||
if len(preview) < int(preview_limit):
|
||||
preview.append(
|
||||
{
|
||||
"illust_id": parsed.illust_id,
|
||||
"page_index": parsed.page_index,
|
||||
"ext": parsed.ext,
|
||||
"url": url,
|
||||
}
|
||||
)
|
||||
|
||||
return total, accepted, deduped, error_total, errors, preview
|
||||
|
||||
|
||||
def _parse_bool(value: Any, *, default: bool = False) -> bool:
|
||||
if value is None:
|
||||
return default
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, int) and value in (0, 1):
|
||||
return bool(value)
|
||||
if isinstance(value, str):
|
||||
v = value.strip().lower()
|
||||
if v in {"1", "true", "yes", "y", "on"}:
|
||||
return True
|
||||
if v in {"0", "false", "no", "n", "off"}:
|
||||
return False
|
||||
return default
|
||||
|
||||
|
||||
def _validate_import_create(data: dict[str, Any]) -> ImportCreateRequest:
|
||||
try:
|
||||
return ImportCreateRequest(**data)
|
||||
except ValidationError as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="导入请求体无效", status_code=400) from exc
|
||||
|
||||
|
||||
async def _load_import_request(request: Request) -> ImportCreateRequest:
|
||||
content_type = (request.headers.get("content-type") or "").lower()
|
||||
max_bytes = _max_import_text_bytes()
|
||||
|
||||
if content_type.startswith("application/json"):
|
||||
data = await request.json()
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
body = _validate_import_create(data)
|
||||
if len(body.text.encode("utf-8", errors="ignore")) > max_bytes:
|
||||
raise ApiError(code=ErrorCode.PAYLOAD_TOO_LARGE, message="Payload too large", status_code=413)
|
||||
return body
|
||||
|
||||
if content_type.startswith("multipart/form-data"):
|
||||
form = await request.form()
|
||||
file_obj = form.get("file")
|
||||
if not isinstance(file_obj, UploadFile):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing file", status_code=400)
|
||||
|
||||
filename = (file_obj.filename or "").strip().lower()
|
||||
if filename and not filename.endswith(".txt"):
|
||||
raise ApiError(code=ErrorCode.INVALID_UPLOAD_TYPE, message="Unsupported upload type", status_code=400)
|
||||
|
||||
raw = await file_obj.read()
|
||||
if len(raw) > max_bytes:
|
||||
raise ApiError(code=ErrorCode.PAYLOAD_TOO_LARGE, message="Payload too large", status_code=413)
|
||||
text = raw.decode("utf-8", errors="replace")
|
||||
|
||||
return _validate_import_create(
|
||||
{
|
||||
"text": text,
|
||||
"dry_run": _parse_bool(form.get("dry_run"), default=False),
|
||||
"hydrate_on_import": _parse_bool(form.get("hydrate_on_import"), default=False),
|
||||
"source": str(form.get("source") or "manual"),
|
||||
}
|
||||
)
|
||||
|
||||
raise ApiError(code=ErrorCode.INVALID_UPLOAD_TYPE, message="Unsupported content type", status_code=400)
|
||||
|
||||
|
||||
@router.post("/imports")
|
||||
async def create_import(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
body = await _load_import_request(request)
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
total, accepted, deduped, error_total, errors, preview = _parse_import_text(body.text, preview_limit=20)
|
||||
|
||||
if body.dry_run:
|
||||
return {
|
||||
"ok": True,
|
||||
"import_id": "",
|
||||
"job_id": "",
|
||||
"accepted": accepted,
|
||||
"deduped": deduped,
|
||||
"errors": [asdict(e) for e in errors[:200]],
|
||||
"preview": preview,
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
settings = request.app.state.settings
|
||||
db_dir = get_sqlite_db_dir(settings.database_url)
|
||||
payload_dir = db_dir / "imports_payloads"
|
||||
payload_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
payload_path = payload_dir / f"import_payload_{uuid4().hex}.txt"
|
||||
try:
|
||||
payload_path.write_text(body.text, encoding="utf-8")
|
||||
except OSError as exc:
|
||||
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="写入导入内容失败", status_code=500) from exc
|
||||
|
||||
file_ref = make_file_ref(payload_path, base_dir=db_dir)
|
||||
|
||||
async def _op() -> tuple[int, int]:
|
||||
async with Session() as session:
|
||||
imp = Import(created_by=str(_claims.get("sub") or ""), source=body.source)
|
||||
session.add(imp)
|
||||
await session.flush()
|
||||
|
||||
imp.total = int(total)
|
||||
imp.accepted = int(accepted)
|
||||
imp.success = 0
|
||||
imp.failed = int(error_total)
|
||||
imp.detail_json = json.dumps(
|
||||
{
|
||||
"deduped": int(deduped),
|
||||
"errors": [asdict(e) for e in errors[:200]],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
|
||||
job = JobRow(
|
||||
type="import_images",
|
||||
status="pending",
|
||||
payload_json=json.dumps(
|
||||
{
|
||||
"import_id": int(imp.id),
|
||||
"file_ref": file_ref,
|
||||
"hydrate_on_import": bool(body.hydrate_on_import),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
separators=(",", ":"),
|
||||
),
|
||||
ref_type="import",
|
||||
ref_id=str(imp.id),
|
||||
)
|
||||
session.add(job)
|
||||
await session.flush()
|
||||
|
||||
await session.commit()
|
||||
return int(imp.id), int(job.id)
|
||||
|
||||
try:
|
||||
import_id, job_id = await with_sqlite_busy_retry(_op)
|
||||
except Exception:
|
||||
try:
|
||||
payload_path.unlink(missing_ok=True)
|
||||
except Exception:
|
||||
pass
|
||||
raise
|
||||
|
||||
inline_max = _import_inline_max_accepted()
|
||||
executed_inline = False
|
||||
if accepted > 0 and accepted <= inline_max:
|
||||
now = iso_utc_ms()
|
||||
actor = str(_claims.get("sub") or "admin").strip() or "admin"
|
||||
worker_id = f"inline-import:{actor}"
|
||||
claimed = await _claim_job_by_id(engine, job_id=int(job_id), worker_id=worker_id, now=now)
|
||||
if claimed is not None:
|
||||
dispatcher = JobDispatcher()
|
||||
dispatcher.register("import_images", build_import_images_handler(engine))
|
||||
await execute_claimed_job(engine, dispatcher, job_row=claimed, worker_id=worker_id)
|
||||
executed_inline = True
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"import_id": str(import_id),
|
||||
"job_id": str(job_id),
|
||||
"executed_inline": executed_inline,
|
||||
"accepted": accepted,
|
||||
"deduped": deduped,
|
||||
"errors": [asdict(e) for e in errors[:200]],
|
||||
"preview": preview,
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/imports/{import_id}/rollback")
|
||||
async def rollback_import(
|
||||
import_id: int,
|
||||
body: ImportRollbackRequest,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if import_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid import id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
target_status = 2 if body.mode == "disable" else 4
|
||||
now_expr = sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))")
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> int:
|
||||
async with Session() as session:
|
||||
imp = await session.get(Import, import_id)
|
||||
if imp is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Import not found", status_code=404)
|
||||
|
||||
result = await session.execute(
|
||||
sa.update(Image)
|
||||
.where(Image.created_import_id == import_id)
|
||||
.values(status=target_status, updated_at=now_expr)
|
||||
)
|
||||
updated = int(result.rowcount or 0)
|
||||
await session.commit()
|
||||
return int(updated)
|
||||
|
||||
updated = await with_sqlite_busy_retry(_op)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"mode": body.mode,
|
||||
"updated": updated,
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/imports/{import_id}")
|
||||
async def get_import(
|
||||
import_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if import_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid import id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async with Session() as session:
|
||||
imp = await session.get(Import, import_id)
|
||||
if imp is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Import not found", status_code=404)
|
||||
|
||||
job = (
|
||||
(
|
||||
await session.execute(
|
||||
sa.select(JobRow)
|
||||
.where(JobRow.ref_type == "import", JobRow.ref_id == str(import_id))
|
||||
.order_by(JobRow.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.first()
|
||||
)
|
||||
|
||||
detail: dict[str, Any] = {}
|
||||
if imp.detail_json:
|
||||
try:
|
||||
parsed = json.loads(imp.detail_json)
|
||||
if isinstance(parsed, dict):
|
||||
detail = parsed
|
||||
except Exception:
|
||||
detail = {}
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"item": {
|
||||
"import": {
|
||||
"id": str(imp.id),
|
||||
"created_at": imp.created_at,
|
||||
"created_by": imp.created_by,
|
||||
"source": imp.source,
|
||||
"total": int(imp.total or 0),
|
||||
"accepted": int(imp.accepted or 0),
|
||||
"success": int(imp.success or 0),
|
||||
"failed": int(imp.failed or 0),
|
||||
},
|
||||
"job": (
|
||||
{
|
||||
"id": str(job.id),
|
||||
"type": job.type,
|
||||
"status": job.status,
|
||||
"attempt": job.attempt,
|
||||
"max_attempts": job.max_attempts,
|
||||
"last_error": job.last_error,
|
||||
}
|
||||
if job is not None
|
||||
else None
|
||||
),
|
||||
"detail": detail,
|
||||
},
|
||||
"request_id": rid,
|
||||
}
|
||||
@@ -0,0 +1,251 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import sqlalchemy as sa
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.core.time import iso_utc_ms
|
||||
from app.db.models.jobs import JobRow
|
||||
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
_ALLOWED_JOB_STATUSES = {"pending", "running", "paused", "canceled", "completed", "failed", "dlq"}
|
||||
|
||||
|
||||
@router.get("/jobs")
|
||||
async def list_jobs(
|
||||
request: Request,
|
||||
limit: int = 50,
|
||||
cursor: str | None = None,
|
||||
status: str | None = None,
|
||||
type: str | None = None,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if limit < 1 or limit > 200:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
|
||||
|
||||
cursor_i: int | None = None
|
||||
cursor_raw = (cursor or "").strip()
|
||||
if cursor_raw:
|
||||
if not cursor_raw.isdigit():
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
cursor_i = int(cursor_raw)
|
||||
if cursor_i <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
|
||||
status_norm = (status or "").strip().lower() or None
|
||||
if status_norm is not None and status_norm not in _ALLOWED_JOB_STATUSES:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported status", status_code=400)
|
||||
|
||||
type_norm = (type or "").strip() or None
|
||||
if type_norm is not None and len(type_norm) > 100:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported type", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
stmt = sa.select(JobRow).order_by(JobRow.id.desc()).limit(limit + 1)
|
||||
if cursor_i is not None:
|
||||
stmt = stmt.where(JobRow.id < cursor_i)
|
||||
if status_norm is not None:
|
||||
stmt = stmt.where(JobRow.status == status_norm)
|
||||
if type_norm is not None:
|
||||
stmt = stmt.where(JobRow.type == type_norm)
|
||||
|
||||
async with Session() as session:
|
||||
rows = ((await session.execute(stmt)).scalars().all())
|
||||
|
||||
items_rows = rows[:limit]
|
||||
next_cursor = int(items_rows[-1].id) if len(rows) > limit and items_rows else None
|
||||
|
||||
items = [
|
||||
{
|
||||
"id": str(row.id),
|
||||
"type": row.type,
|
||||
"status": row.status,
|
||||
"priority": int(row.priority),
|
||||
"run_after": row.run_after,
|
||||
"attempt": int(row.attempt),
|
||||
"max_attempts": int(row.max_attempts),
|
||||
"last_error": row.last_error,
|
||||
"locked_by": row.locked_by,
|
||||
"locked_at": row.locked_at,
|
||||
"ref_type": row.ref_type,
|
||||
"ref_id": row.ref_id,
|
||||
"created_at": row.created_at,
|
||||
"updated_at": row.updated_at,
|
||||
}
|
||||
for row in items_rows
|
||||
]
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"items": items,
|
||||
"next_cursor": str(next_cursor) if next_cursor is not None else "",
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}")
|
||||
async def get_job(
|
||||
job_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if job_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid job id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async with Session() as session:
|
||||
row = await session.get(JobRow, job_id)
|
||||
if row is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Job not found", status_code=404)
|
||||
|
||||
payload_json = str(row.payload_json or "")
|
||||
payload: Any = None
|
||||
if payload_json.strip():
|
||||
try:
|
||||
payload = json.loads(payload_json)
|
||||
except Exception:
|
||||
payload = None
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"item": {
|
||||
"id": str(row.id),
|
||||
"type": row.type,
|
||||
"status": row.status,
|
||||
"priority": int(row.priority),
|
||||
"run_after": row.run_after,
|
||||
"attempt": int(row.attempt),
|
||||
"max_attempts": int(row.max_attempts),
|
||||
"payload": payload,
|
||||
"payload_json": payload_json,
|
||||
"last_error": row.last_error,
|
||||
"locked_by": row.locked_by,
|
||||
"locked_at": row.locked_at,
|
||||
"ref_type": row.ref_type,
|
||||
"ref_id": row.ref_id,
|
||||
"created_at": row.created_at,
|
||||
"updated_at": row.updated_at,
|
||||
},
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/retry")
|
||||
async def retry_job(
|
||||
job_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if job_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid job id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
now = iso_utc_ms()
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with Session() as session:
|
||||
row = await session.get(JobRow, job_id)
|
||||
if row is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Job not found", status_code=404)
|
||||
|
||||
if row.status == "running":
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Job is running", status_code=400)
|
||||
|
||||
row.status = "pending"
|
||||
row.run_after = None
|
||||
row.locked_by = None
|
||||
row.locked_at = None
|
||||
row.updated_at = now
|
||||
await session.commit()
|
||||
|
||||
return {"ok": True, "job_id": str(job_id), "status": "pending", "request_id": rid}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/cancel")
|
||||
async def cancel_job(
|
||||
job_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if job_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid job id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
now = iso_utc_ms()
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with Session() as session:
|
||||
row = await session.get(JobRow, job_id)
|
||||
if row is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Job not found", status_code=404)
|
||||
|
||||
row.status = "canceled"
|
||||
row.locked_by = None
|
||||
row.locked_at = None
|
||||
row.updated_at = now
|
||||
await session.commit()
|
||||
|
||||
return {"ok": True, "job_id": str(job_id), "status": "canceled", "request_id": rid}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/move-to-dlq")
|
||||
async def move_job_to_dlq(
|
||||
job_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if job_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid job id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
now = iso_utc_ms()
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with Session() as session:
|
||||
row = await session.get(JobRow, job_id)
|
||||
if row is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Job not found", status_code=404)
|
||||
|
||||
row.status = "dlq"
|
||||
row.run_after = None
|
||||
row.locked_by = None
|
||||
row.locked_at = None
|
||||
row.updated_at = now
|
||||
await session.commit()
|
||||
|
||||
return {"ok": True, "job_id": str(job_id), "status": "dlq", "request_id": rid}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
@@ -0,0 +1,118 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.db.request_logs_cleanup import (
|
||||
DEFAULT_REQUEST_LOGS_CHUNK_SIZE,
|
||||
DEFAULT_REQUEST_LOGS_KEEP_DAYS,
|
||||
DEFAULT_REQUEST_LOGS_MAX_DELETE_ROWS,
|
||||
cleanup_request_logs,
|
||||
preview_request_logs_cleanup,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _parse_bool(value: Any, *, default: bool) -> bool:
|
||||
if value is None:
|
||||
return default
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, int) and value in (0, 1):
|
||||
return bool(value)
|
||||
if isinstance(value, str):
|
||||
v = value.strip().lower()
|
||||
if v in {"1", "true", "yes", "y", "on"}:
|
||||
return True
|
||||
if v in {"0", "false", "no", "n", "off"}:
|
||||
return False
|
||||
return default
|
||||
|
||||
|
||||
async def _load_cleanup_request_logs_json(request: Request) -> dict[str, Any]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
|
||||
keep_days_raw = data.get("keep_days", DEFAULT_REQUEST_LOGS_KEEP_DAYS)
|
||||
try:
|
||||
keep_days = int(keep_days_raw)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported keep_days", status_code=400) from exc
|
||||
if keep_days < 0 or keep_days > 36500:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported keep_days", status_code=400)
|
||||
|
||||
max_delete_rows_raw = data.get("max_delete_rows", DEFAULT_REQUEST_LOGS_MAX_DELETE_ROWS)
|
||||
try:
|
||||
max_delete_rows = int(max_delete_rows_raw)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported max_delete_rows", status_code=400) from exc
|
||||
if max_delete_rows < 1 or max_delete_rows > 10_000_000:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported max_delete_rows", status_code=400)
|
||||
|
||||
chunk_size_raw = data.get("chunk_size", DEFAULT_REQUEST_LOGS_CHUNK_SIZE)
|
||||
try:
|
||||
chunk_size = int(chunk_size_raw)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported chunk_size", status_code=400) from exc
|
||||
if chunk_size < 1 or chunk_size > 100_000:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported chunk_size", status_code=400)
|
||||
|
||||
dry_run = _parse_bool(data.get("dry_run"), default=False)
|
||||
|
||||
return {
|
||||
"keep_days": int(keep_days),
|
||||
"max_delete_rows": int(max_delete_rows),
|
||||
"chunk_size": int(chunk_size),
|
||||
"dry_run": bool(dry_run),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/maintenance/request-logs/cleanup")
|
||||
async def cleanup_request_logs_endpoint(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
cfg = await _load_cleanup_request_logs_json(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
if bool(cfg["dry_run"]):
|
||||
preview = await preview_request_logs_cleanup(
|
||||
engine,
|
||||
keep_days=int(cfg["keep_days"]),
|
||||
max_delete_rows=int(cfg["max_delete_rows"]),
|
||||
)
|
||||
return {
|
||||
"ok": True,
|
||||
"dry_run": True,
|
||||
"cutoff": preview.cutoff,
|
||||
"would_delete": int(preview.would_delete),
|
||||
"has_more": bool(preview.has_more),
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
result = await cleanup_request_logs(
|
||||
engine,
|
||||
keep_days=int(cfg["keep_days"]),
|
||||
max_delete_rows=int(cfg["max_delete_rows"]),
|
||||
chunk_size=int(cfg["chunk_size"]),
|
||||
)
|
||||
return {
|
||||
"ok": True,
|
||||
"dry_run": False,
|
||||
"cutoff": result.cutoff,
|
||||
"deleted": int(result.deleted),
|
||||
"has_more": bool(result.has_more),
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,318 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import sqlalchemy as sa
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.db.models.proxy_endpoints import ProxyEndpoint
|
||||
from app.db.models.proxy_pool_endpoints import ProxyPoolEndpoint
|
||||
from app.db.models.proxy_pools import ProxyPool
|
||||
from app.db.session import create_sessionmaker
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _parse_bool(value: Any, *, default: bool) -> bool:
|
||||
if value is None:
|
||||
return default
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, int) and value in (0, 1):
|
||||
return bool(value)
|
||||
if isinstance(value, str):
|
||||
v = value.strip().lower()
|
||||
if v in {"1", "true", "yes", "y", "on"}:
|
||||
return True
|
||||
if v in {"0", "false", "no", "n", "off"}:
|
||||
return False
|
||||
return default
|
||||
|
||||
|
||||
async def _load_create_json(request: Request) -> dict[str, Any]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
|
||||
name = str(data.get("name") or "").strip()
|
||||
if not name:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing name", status_code=400)
|
||||
if len(name) > 100:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported name", status_code=400)
|
||||
|
||||
desc_raw = data.get("description")
|
||||
description = str(desc_raw).strip() if desc_raw is not None else None
|
||||
description = description if description else None
|
||||
|
||||
enabled = _parse_bool(data.get("enabled"), default=True)
|
||||
return {"name": name, "description": description, "enabled": enabled}
|
||||
|
||||
|
||||
async def _load_update_json(request: Request) -> dict[str, Any]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
|
||||
out: dict[str, Any] = {}
|
||||
|
||||
if "name" in data:
|
||||
name = str(data.get("name") or "").strip()
|
||||
if not name:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported name", status_code=400)
|
||||
if len(name) > 100:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported name", status_code=400)
|
||||
out["name"] = name
|
||||
|
||||
if "description" in data:
|
||||
desc_raw = data.get("description")
|
||||
description = str(desc_raw).strip() if desc_raw is not None else None
|
||||
out["description"] = description if description else None
|
||||
|
||||
if "enabled" in data:
|
||||
out["enabled"] = _parse_bool(data.get("enabled"), default=True)
|
||||
|
||||
if not out:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing fields", status_code=400)
|
||||
|
||||
return out
|
||||
|
||||
|
||||
async def _load_set_endpoints_json(request: Request) -> list[dict[str, Any]]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
|
||||
raw_items = data.get("items")
|
||||
if raw_items is None:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing items", status_code=400)
|
||||
if not isinstance(raw_items, list):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid items", status_code=400)
|
||||
|
||||
out: list[dict[str, Any]] = []
|
||||
seen: set[int] = set()
|
||||
|
||||
for raw in raw_items:
|
||||
if not isinstance(raw, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid items", status_code=400)
|
||||
try:
|
||||
endpoint_id = int(raw.get("endpoint_id"))
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid endpoint_id", status_code=400) from exc
|
||||
if endpoint_id <= 0 or endpoint_id in seen:
|
||||
continue
|
||||
seen.add(endpoint_id)
|
||||
|
||||
enabled = _parse_bool(raw.get("enabled"), default=True)
|
||||
|
||||
weight_raw = raw.get("weight", 1)
|
||||
try:
|
||||
weight = int(weight_raw)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid weight", status_code=400) from exc
|
||||
if weight < 0 or weight > 1000:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid weight", status_code=400)
|
||||
|
||||
out.append({"endpoint_id": endpoint_id, "enabled": enabled, "weight": weight})
|
||||
|
||||
return out
|
||||
|
||||
|
||||
@router.get("/proxy-pools")
|
||||
async def list_proxy_pools(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
async with Session() as session:
|
||||
pools = (
|
||||
(await session.execute(sa.select(ProxyPool).order_by(ProxyPool.id.desc())))
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
|
||||
items = [
|
||||
{
|
||||
"id": str(p.id),
|
||||
"name": p.name,
|
||||
"description": p.description,
|
||||
"enabled": bool(p.enabled),
|
||||
}
|
||||
for p in pools
|
||||
]
|
||||
|
||||
return {"ok": True, "items": items, "request_id": rid}
|
||||
|
||||
|
||||
@router.post("/proxy-pools")
|
||||
async def create_proxy_pool(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
body = await _load_create_json(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async with Session() as session:
|
||||
row = ProxyPool(
|
||||
name=str(body["name"]),
|
||||
description=body["description"],
|
||||
enabled=1 if bool(body["enabled"]) else 0,
|
||||
)
|
||||
session.add(row)
|
||||
try:
|
||||
await session.commit()
|
||||
except IntegrityError as exc:
|
||||
await session.rollback()
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Proxy pool name exists", status_code=400) from exc
|
||||
await session.refresh(row)
|
||||
|
||||
return {"ok": True, "pool_id": str(row.id), "request_id": rid}
|
||||
|
||||
|
||||
@router.put("/proxy-pools/{pool_id}")
|
||||
async def update_proxy_pool(
|
||||
pool_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if pool_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid pool id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
body = await _load_update_json(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
async with Session() as session:
|
||||
row = await session.get(ProxyPool, pool_id)
|
||||
if row is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Proxy pool not found", status_code=404)
|
||||
|
||||
if "name" in body:
|
||||
row.name = str(body["name"])
|
||||
if "description" in body:
|
||||
row.description = body["description"]
|
||||
if "enabled" in body:
|
||||
row.enabled = 1 if bool(body["enabled"]) else 0
|
||||
|
||||
try:
|
||||
await session.commit()
|
||||
except IntegrityError as exc:
|
||||
await session.rollback()
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Proxy pool name exists", status_code=400) from exc
|
||||
|
||||
return {"ok": True, "pool_id": str(pool_id), "request_id": rid}
|
||||
|
||||
|
||||
@router.post("/proxy-pools/{pool_id}/endpoints")
|
||||
async def set_proxy_pool_endpoints(
|
||||
pool_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if pool_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid pool id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
items = await _load_set_endpoints_json(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
created = 0
|
||||
updated = 0
|
||||
removed = 0
|
||||
|
||||
async with Session() as session:
|
||||
pool = await session.get(ProxyPool, pool_id)
|
||||
if pool is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Proxy pool not found", status_code=404)
|
||||
|
||||
endpoint_ids = [int(x["endpoint_id"]) for x in items]
|
||||
if endpoint_ids:
|
||||
existing = (
|
||||
(
|
||||
await session.execute(
|
||||
sa.select(ProxyEndpoint.id).where(ProxyEndpoint.id.in_(endpoint_ids))
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
existing_set = {int(x) for x in existing}
|
||||
missing = [eid for eid in endpoint_ids if eid not in existing_set]
|
||||
if missing:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unknown endpoint_id", status_code=400)
|
||||
|
||||
current_rows = (
|
||||
(
|
||||
await session.execute(
|
||||
sa.select(ProxyPoolEndpoint).where(ProxyPoolEndpoint.pool_id == pool_id)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
current_by_eid = {int(r.endpoint_id): r for r in current_rows}
|
||||
|
||||
keep: set[int] = set()
|
||||
for item in items:
|
||||
eid = int(item["endpoint_id"])
|
||||
keep.add(eid)
|
||||
row = current_by_eid.get(eid)
|
||||
if row is None:
|
||||
session.add(
|
||||
ProxyPoolEndpoint(
|
||||
pool_id=pool_id,
|
||||
endpoint_id=eid,
|
||||
enabled=1 if bool(item["enabled"]) else 0,
|
||||
weight=int(item["weight"]),
|
||||
)
|
||||
)
|
||||
created += 1
|
||||
else:
|
||||
row.enabled = 1 if bool(item["enabled"]) else 0
|
||||
row.weight = int(item["weight"])
|
||||
updated += 1
|
||||
|
||||
for eid, row in current_by_eid.items():
|
||||
if eid in keep:
|
||||
continue
|
||||
await session.delete(row)
|
||||
removed += 1
|
||||
|
||||
await session.commit()
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"pool_id": str(pool_id),
|
||||
"created": created,
|
||||
"updated": updated,
|
||||
"removed": removed,
|
||||
"request_id": rid,
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.api.admin.auth import router as auth_router
|
||||
from app.api.admin.api_keys import router as api_keys_router
|
||||
from app.api.admin.audit import router as audit_router
|
||||
from app.api.admin.bindings import router as bindings_router
|
||||
from app.api.admin.hydration_runs import router as hydration_runs_router
|
||||
from app.api.admin.images import router as images_router
|
||||
from app.api.admin.imports import router as imports_router
|
||||
from app.api.admin.jobs import router as jobs_router
|
||||
from app.api.admin.maintenance import router as maintenance_router
|
||||
from app.api.admin.proxies import router as proxies_router
|
||||
from app.api.admin.proxy_pools import router as proxy_pools_router
|
||||
from app.api.admin.settings import router as settings_router
|
||||
from app.api.admin.stats import router as stats_router
|
||||
from app.api.admin.summary import router as summary_router
|
||||
from app.api.admin.tokens import router as tokens_router
|
||||
|
||||
router = APIRouter(prefix="/admin/api")
|
||||
router.include_router(auth_router)
|
||||
router.include_router(api_keys_router)
|
||||
router.include_router(audit_router)
|
||||
router.include_router(bindings_router)
|
||||
router.include_router(hydration_runs_router)
|
||||
router.include_router(images_router)
|
||||
router.include_router(imports_router)
|
||||
router.include_router(jobs_router)
|
||||
router.include_router(maintenance_router)
|
||||
router.include_router(proxies_router)
|
||||
router.include_router(proxy_pools_router)
|
||||
router.include_router(settings_router)
|
||||
router.include_router(stats_router)
|
||||
router.include_router(summary_router)
|
||||
router.include_router(tokens_router)
|
||||
@@ -0,0 +1,444 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.core.runtime_settings import (
|
||||
fetch_runtime_settings,
|
||||
runtime_config_from_values,
|
||||
set_runtime_setting,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
_DEFAULT_SCORE_WEIGHTS: dict[str, float] = {
|
||||
"bookmark": 4.0,
|
||||
"view": 0.5,
|
||||
"comment": 2.0,
|
||||
"pixels": 1.0,
|
||||
"bookmark_rate": 3.0,
|
||||
}
|
||||
|
||||
_DEFAULT_RECOMMENDATION: dict[str, Any] = {
|
||||
"pick_mode": "weighted",
|
||||
"temperature": 1.0,
|
||||
"score_weights": dict(_DEFAULT_SCORE_WEIGHTS),
|
||||
"multipliers": {
|
||||
"ai": 1.0,
|
||||
"non_ai": 1.0,
|
||||
"unknown_ai": 1.0,
|
||||
"illust": 1.0,
|
||||
"manga": 1.0,
|
||||
"ugoira": 1.0,
|
||||
"unknown_illust_type": 1.0,
|
||||
},
|
||||
}
|
||||
|
||||
_DEFAULT_SETTINGS = {
|
||||
"random": {
|
||||
"default_attempts": 3,
|
||||
"default_r18_strict": True,
|
||||
"fail_cooldown_ms": 600_000,
|
||||
"strategy": "quality",
|
||||
"quality_samples": 5,
|
||||
"recommendation": dict(_DEFAULT_RECOMMENDATION),
|
||||
},
|
||||
"proxy": {"allowlist_domains": []},
|
||||
}
|
||||
|
||||
|
||||
def _as_str_list(value: Any) -> list[str]:
|
||||
if not isinstance(value, list):
|
||||
return []
|
||||
out: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for item in value:
|
||||
if not isinstance(item, str):
|
||||
continue
|
||||
v = item.strip()
|
||||
if not v or v in seen:
|
||||
continue
|
||||
seen.add(v)
|
||||
out.append(v)
|
||||
return out
|
||||
|
||||
|
||||
def _as_bool(value: Any) -> bool | None:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, int) and value in (0, 1):
|
||||
return bool(value)
|
||||
if isinstance(value, str):
|
||||
v = value.strip().lower()
|
||||
if v in {"true", "1", "yes", "y", "on"}:
|
||||
return True
|
||||
if v in {"false", "0", "no", "n", "off"}:
|
||||
return False
|
||||
return None
|
||||
|
||||
|
||||
def _as_float(value: Any) -> float | None:
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, bool):
|
||||
return None
|
||||
try:
|
||||
return float(value)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _normalize_recommendation(value: Any, *, strict: bool) -> dict[str, Any]:
|
||||
if value is None:
|
||||
return dict(_DEFAULT_RECOMMENDATION)
|
||||
if not isinstance(value, dict):
|
||||
if strict:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random.recommendation", status_code=400)
|
||||
return dict(_DEFAULT_RECOMMENDATION)
|
||||
|
||||
pick_mode_default = str(_DEFAULT_RECOMMENDATION["pick_mode"])
|
||||
pick_mode_raw = value.get("pick_mode")
|
||||
pick_mode = pick_mode_default
|
||||
if pick_mode_raw is not None:
|
||||
candidate = str(pick_mode_raw or "").strip().lower()
|
||||
if candidate not in {"best", "weighted"}:
|
||||
if strict:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random.recommendation.pick_mode", status_code=400)
|
||||
else:
|
||||
pick_mode = candidate
|
||||
|
||||
temperature_default = float(_DEFAULT_RECOMMENDATION["temperature"])
|
||||
temperature = temperature_default
|
||||
if "temperature" in value:
|
||||
v = _as_float(value.get("temperature"))
|
||||
if v is None:
|
||||
if strict:
|
||||
raise ApiError(
|
||||
code=ErrorCode.BAD_REQUEST,
|
||||
message="Invalid random.recommendation.temperature",
|
||||
status_code=400,
|
||||
)
|
||||
else:
|
||||
temperature = float(max(0.05, min(float(v), 100.0)))
|
||||
|
||||
score_weights_default = _DEFAULT_RECOMMENDATION["score_weights"]
|
||||
score_weights_raw = value.get("score_weights")
|
||||
if score_weights_raw is None:
|
||||
score_weights_obj: dict[str, Any] = {}
|
||||
elif not isinstance(score_weights_raw, dict):
|
||||
if strict:
|
||||
raise ApiError(
|
||||
code=ErrorCode.BAD_REQUEST,
|
||||
message="Invalid random.recommendation.score_weights",
|
||||
status_code=400,
|
||||
)
|
||||
score_weights_obj = {}
|
||||
else:
|
||||
score_weights_obj = score_weights_raw
|
||||
if strict:
|
||||
for k in score_weights_obj.keys():
|
||||
if str(k) not in _DEFAULT_SCORE_WEIGHTS:
|
||||
raise ApiError(
|
||||
code=ErrorCode.BAD_REQUEST,
|
||||
message="Invalid random.recommendation.score_weights",
|
||||
status_code=400,
|
||||
)
|
||||
|
||||
score_weights: dict[str, float] = {}
|
||||
for key, default_value in _DEFAULT_SCORE_WEIGHTS.items():
|
||||
if key in score_weights_obj:
|
||||
v = _as_float(score_weights_obj.get(key))
|
||||
if v is None:
|
||||
if strict:
|
||||
raise ApiError(
|
||||
code=ErrorCode.BAD_REQUEST,
|
||||
message="Invalid random.recommendation.score_weights",
|
||||
status_code=400,
|
||||
)
|
||||
v = float(default_value)
|
||||
score_weights[key] = float(max(-100.0, min(float(v), 100.0)))
|
||||
else:
|
||||
score_weights[key] = float(default_value)
|
||||
|
||||
multipliers_default = _DEFAULT_RECOMMENDATION["multipliers"]
|
||||
multipliers_raw = value.get("multipliers")
|
||||
if multipliers_raw is None:
|
||||
multipliers_obj: dict[str, Any] = {}
|
||||
elif not isinstance(multipliers_raw, dict):
|
||||
if strict:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random.recommendation.multipliers", status_code=400)
|
||||
multipliers_obj = {}
|
||||
else:
|
||||
multipliers_obj = multipliers_raw
|
||||
if strict:
|
||||
for k in multipliers_obj.keys():
|
||||
if str(k) not in multipliers_default:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random.recommendation.multipliers", status_code=400)
|
||||
|
||||
multipliers: dict[str, float] = {}
|
||||
for key, default_value in multipliers_default.items():
|
||||
if key in multipliers_obj:
|
||||
v = _as_float(multipliers_obj.get(key))
|
||||
if v is None:
|
||||
if strict:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random.recommendation.multipliers", status_code=400)
|
||||
v = float(default_value)
|
||||
multipliers[key] = float(max(0.0, min(float(v), 100.0)))
|
||||
else:
|
||||
multipliers[key] = float(default_value)
|
||||
|
||||
return {
|
||||
"pick_mode": pick_mode,
|
||||
"temperature": temperature,
|
||||
"score_weights": score_weights,
|
||||
"multipliers": multipliers,
|
||||
}
|
||||
|
||||
|
||||
async def _load_settings_json(request: Request) -> dict[str, Any]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
|
||||
settings = data.get("settings") if "settings" in data else data
|
||||
if not isinstance(settings, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid settings", status_code=400)
|
||||
if not settings:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing fields", status_code=400)
|
||||
|
||||
return settings
|
||||
|
||||
|
||||
@router.get("/settings")
|
||||
async def get_settings(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
values = await fetch_runtime_settings(engine)
|
||||
runtime = runtime_config_from_values(values)
|
||||
|
||||
random_defaults = dict(_DEFAULT_SETTINGS["random"])
|
||||
if isinstance(runtime.random_defaults, dict):
|
||||
for k in list(random_defaults.keys()):
|
||||
if k == "recommendation":
|
||||
continue
|
||||
if k in runtime.random_defaults:
|
||||
random_defaults[k] = runtime.random_defaults[k]
|
||||
random_defaults["recommendation"] = _normalize_recommendation(runtime.random_defaults.get("recommendation"), strict=False)
|
||||
else:
|
||||
random_defaults["recommendation"] = _normalize_recommendation(None, strict=False)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"settings": {
|
||||
"proxy": {
|
||||
"enabled": bool(runtime.proxy_enabled),
|
||||
"fail_closed": bool(runtime.proxy_fail_closed),
|
||||
"route_mode": runtime.proxy_route_mode,
|
||||
"allowlist_domains": list(runtime.proxy_allowlist_domains),
|
||||
"default_pool_id": str(runtime.proxy_default_pool_id) if runtime.proxy_default_pool_id is not None else "",
|
||||
"route_pools": {k: str(v) for k, v in runtime.proxy_route_pools.items()},
|
||||
},
|
||||
"random": random_defaults,
|
||||
"security": {"hide_origin_url_in_public_json": bool(runtime.hide_origin_url_in_public_json)},
|
||||
"rate_limit": dict(runtime.rate_limit),
|
||||
},
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
|
||||
@router.put("/settings")
|
||||
async def update_settings(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
rid = get_or_create_request_id(request)
|
||||
body = await _load_settings_json(request)
|
||||
|
||||
actor = str(_claims.get("sub") or "admin").strip() or "admin"
|
||||
updated_by = f"admin:{actor}"
|
||||
|
||||
updates: list[tuple[str, Any]] = []
|
||||
proxy_enabled_override: bool | None = None
|
||||
proxy_fail_closed_override: bool | None = None
|
||||
|
||||
proxy = body.get("proxy")
|
||||
if proxy is not None:
|
||||
if not isinstance(proxy, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy", status_code=400)
|
||||
|
||||
if "enabled" in proxy:
|
||||
v = _as_bool(proxy.get("enabled"))
|
||||
if v is None:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.enabled", status_code=400)
|
||||
proxy_enabled_override = bool(v)
|
||||
updates.append(("proxy.enabled", bool(v)))
|
||||
|
||||
if "fail_closed" in proxy:
|
||||
v = _as_bool(proxy.get("fail_closed"))
|
||||
if v is None:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.fail_closed", status_code=400)
|
||||
proxy_fail_closed_override = bool(v)
|
||||
updates.append(("proxy.fail_closed", bool(v)))
|
||||
|
||||
if "route_mode" in proxy:
|
||||
route_mode = str(proxy.get("route_mode") or "").strip().lower()
|
||||
if route_mode not in {"pixiv_only", "all", "allowlist", "off"}:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.route_mode", status_code=400)
|
||||
updates.append(("proxy.route_mode", route_mode))
|
||||
|
||||
if "allowlist_domains" in proxy:
|
||||
domains = _as_str_list(proxy.get("allowlist_domains"))
|
||||
if len(domains) > 200 or any(len(d) > 200 for d in domains):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.allowlist_domains", status_code=400)
|
||||
updates.append(("proxy.allowlist_domains", domains))
|
||||
|
||||
if "default_pool_id" in proxy:
|
||||
raw = proxy.get("default_pool_id")
|
||||
if raw is None or raw == "":
|
||||
updates.append(("proxy.default_pool_id", None))
|
||||
else:
|
||||
try:
|
||||
pool_id = int(raw)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.default_pool_id", status_code=400) from exc
|
||||
if pool_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.default_pool_id", status_code=400)
|
||||
updates.append(("proxy.default_pool_id", int(pool_id)))
|
||||
|
||||
if "route_pools" in proxy:
|
||||
raw = proxy.get("route_pools")
|
||||
if raw is None:
|
||||
updates.append(("proxy.route_pools", {}))
|
||||
else:
|
||||
if not isinstance(raw, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.route_pools", status_code=400)
|
||||
route_pools: dict[str, int] = {}
|
||||
if len(raw) > 200:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.route_pools", status_code=400)
|
||||
for k, v in raw.items():
|
||||
key = str(k or "").strip().lower().strip(".")
|
||||
if not key or len(key) > 200:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.route_pools", status_code=400)
|
||||
try:
|
||||
pool_id = int(v)
|
||||
except Exception as exc:
|
||||
raise ApiError(
|
||||
code=ErrorCode.BAD_REQUEST,
|
||||
message="Invalid proxy.route_pools",
|
||||
status_code=400,
|
||||
) from exc
|
||||
if pool_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.route_pools", status_code=400)
|
||||
route_pools[key] = int(pool_id)
|
||||
updates.append(("proxy.route_pools", route_pools))
|
||||
|
||||
random = body.get("random")
|
||||
if random is not None:
|
||||
if not isinstance(random, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random", status_code=400)
|
||||
|
||||
defaults: dict[str, Any] = {}
|
||||
for key in ("default_attempts", "default_r18_strict", "fail_cooldown_ms", "strategy", "quality_samples"):
|
||||
if key not in random:
|
||||
continue
|
||||
if key in {"default_attempts", "fail_cooldown_ms", "quality_samples"}:
|
||||
try:
|
||||
n = int(random.get(key))
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message=f"Invalid random.{key}", status_code=400) from exc
|
||||
if key == "quality_samples":
|
||||
if n < 1 or n > 1000:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message=f"Invalid random.{key}", status_code=400)
|
||||
elif n < 0 or n > 10_000_000:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message=f"Invalid random.{key}", status_code=400)
|
||||
defaults[key] = n
|
||||
elif key == "strategy":
|
||||
s = str(random.get(key) or "").strip().lower()
|
||||
if s not in {"quality", "random"}:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random.strategy", status_code=400)
|
||||
defaults[key] = s
|
||||
else:
|
||||
v = _as_bool(random.get(key))
|
||||
if v is None:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message=f"Invalid random.{key}", status_code=400)
|
||||
defaults[key] = bool(v)
|
||||
|
||||
if "recommendation" in random:
|
||||
defaults["recommendation"] = _normalize_recommendation(random.get("recommendation"), strict=True)
|
||||
|
||||
if defaults:
|
||||
values = await fetch_runtime_settings(request.app.state.engine)
|
||||
runtime = runtime_config_from_values(values)
|
||||
existing_raw = runtime.random_defaults if isinstance(runtime.random_defaults, dict) else {}
|
||||
if existing_raw:
|
||||
merged = dict(existing_raw)
|
||||
merged.update(defaults)
|
||||
defaults = merged
|
||||
updates.append(("random.defaults", defaults))
|
||||
|
||||
security = body.get("security")
|
||||
if security is not None:
|
||||
if not isinstance(security, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid security", status_code=400)
|
||||
|
||||
if "hide_origin_url_in_public_json" in security:
|
||||
v = _as_bool(security.get("hide_origin_url_in_public_json"))
|
||||
if v is None:
|
||||
raise ApiError(
|
||||
code=ErrorCode.BAD_REQUEST,
|
||||
message="Invalid security.hide_origin_url_in_public_json",
|
||||
status_code=400,
|
||||
)
|
||||
updates.append(("security.hide_origin_url_in_public_json", bool(v)))
|
||||
|
||||
rate_limit = body.get("rate_limit")
|
||||
if rate_limit is not None:
|
||||
if not isinstance(rate_limit, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid rate_limit", status_code=400)
|
||||
|
||||
for key, value in rate_limit.items():
|
||||
k = str(key or "").strip()
|
||||
if not k or len(k) > 100:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid rate_limit key", status_code=400)
|
||||
updates.append((f"rate_limit.{k}", value))
|
||||
|
||||
if not updates:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing fields", status_code=400)
|
||||
|
||||
engine = request.app.state.engine
|
||||
|
||||
if proxy_enabled_override is not None or proxy_fail_closed_override is not None:
|
||||
values = await fetch_runtime_settings(engine)
|
||||
runtime = runtime_config_from_values(values)
|
||||
desired_enabled = proxy_enabled_override if proxy_enabled_override is not None else bool(runtime.proxy_enabled)
|
||||
desired_fail_closed = (
|
||||
proxy_fail_closed_override if proxy_fail_closed_override is not None else bool(runtime.proxy_fail_closed)
|
||||
)
|
||||
if desired_enabled and desired_fail_closed:
|
||||
async with engine.connect() as conn:
|
||||
result = await conn.exec_driver_sql("SELECT COUNT(*) FROM proxy_endpoints WHERE enabled=1;")
|
||||
enabled_proxy_count = int(result.scalar_one())
|
||||
if enabled_proxy_count <= 0:
|
||||
raise ApiError(
|
||||
code=ErrorCode.PROXY_REQUIRED,
|
||||
message="Proxy required (fail-closed) but no enabled proxies",
|
||||
status_code=400,
|
||||
)
|
||||
|
||||
for key, value in updates:
|
||||
await set_runtime_setting(engine, key=key, value=value, updated_by=updated_by)
|
||||
|
||||
return {"ok": True, "updated": len(updates), "request_id": rid}
|
||||
@@ -0,0 +1,40 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/stats/random")
|
||||
async def get_random_stats(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
stats = getattr(request.app.state, "random_request_stats", None)
|
||||
if stats is None:
|
||||
snapshot = {
|
||||
"total_requests": 0,
|
||||
"total_ok": 0,
|
||||
"total_error": 0,
|
||||
"in_flight": 0,
|
||||
"window_seconds": 60,
|
||||
"last_window_requests": 0,
|
||||
"last_window_ok": 0,
|
||||
"last_window_error": 0,
|
||||
"last_window_success_rate": 0.0,
|
||||
}
|
||||
else:
|
||||
snap = await stats.snapshot()
|
||||
snapshot = asdict(snap)
|
||||
|
||||
return {"ok": True, "stats": snapshot, "request_id": rid}
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.db.session import with_sqlite_busy_retry
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _worker_last_seen_from_value_json(value_json: str | None) -> str | None:
|
||||
if value_json is None:
|
||||
return None
|
||||
raw = str(value_json or "").strip()
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except Exception:
|
||||
return None
|
||||
if isinstance(data, dict):
|
||||
at = data.get("at")
|
||||
return str(at) if isinstance(at, str) and at.strip() else None
|
||||
if isinstance(data, str):
|
||||
return data.strip() or None
|
||||
return None
|
||||
|
||||
|
||||
@router.get("/summary")
|
||||
async def get_summary(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with engine.connect() as conn:
|
||||
images_total = int((await conn.exec_driver_sql("SELECT COUNT(*) FROM images;")).scalar_one())
|
||||
images_enabled = int(
|
||||
(await conn.exec_driver_sql("SELECT COUNT(*) FROM images WHERE status=1;")).scalar_one()
|
||||
)
|
||||
|
||||
missing_tags = int(
|
||||
(
|
||||
await conn.exec_driver_sql(
|
||||
"""
|
||||
SELECT COUNT(*)
|
||||
FROM images
|
||||
WHERE status=1
|
||||
AND id NOT IN (SELECT DISTINCT image_id FROM image_tags);
|
||||
""".strip()
|
||||
)
|
||||
).scalar_one()
|
||||
)
|
||||
missing_geometry = int(
|
||||
(
|
||||
await conn.exec_driver_sql(
|
||||
"SELECT COUNT(*) FROM images WHERE status=1 AND (width IS NULL OR height IS NULL);"
|
||||
)
|
||||
).scalar_one()
|
||||
)
|
||||
missing_r18 = int(
|
||||
(await conn.exec_driver_sql("SELECT COUNT(*) FROM images WHERE status=1 AND x_restrict IS NULL;")).scalar_one()
|
||||
)
|
||||
missing_ai = int(
|
||||
(await conn.exec_driver_sql("SELECT COUNT(*) FROM images WHERE status=1 AND ai_type IS NULL;")).scalar_one()
|
||||
)
|
||||
missing_illust_type = int(
|
||||
(await conn.exec_driver_sql("SELECT COUNT(*) FROM images WHERE status=1 AND illust_type IS NULL;")).scalar_one()
|
||||
)
|
||||
missing_user = int(
|
||||
(await conn.exec_driver_sql("SELECT COUNT(*) FROM images WHERE status=1 AND user_id IS NULL;")).scalar_one()
|
||||
)
|
||||
missing_title = int(
|
||||
(
|
||||
await conn.exec_driver_sql(
|
||||
"SELECT COUNT(*) FROM images WHERE status=1 AND (title IS NULL OR TRIM(title)='');"
|
||||
)
|
||||
).scalar_one()
|
||||
)
|
||||
missing_created_at = int(
|
||||
(
|
||||
await conn.exec_driver_sql(
|
||||
"SELECT COUNT(*) FROM images WHERE status=1 AND (created_at_pixiv IS NULL OR TRIM(created_at_pixiv)='');"
|
||||
)
|
||||
).scalar_one()
|
||||
)
|
||||
missing_popularity = int(
|
||||
(
|
||||
await conn.exec_driver_sql(
|
||||
"SELECT COUNT(*) FROM images "
|
||||
"WHERE status=1 AND (bookmark_count IS NULL OR view_count IS NULL OR comment_count IS NULL);"
|
||||
)
|
||||
).scalar_one()
|
||||
)
|
||||
|
||||
tokens_total = int((await conn.exec_driver_sql("SELECT COUNT(*) FROM pixiv_tokens;")).scalar_one())
|
||||
tokens_enabled = int(
|
||||
(await conn.exec_driver_sql("SELECT COUNT(*) FROM pixiv_tokens WHERE enabled=1;")).scalar_one()
|
||||
)
|
||||
|
||||
proxies_total = int((await conn.exec_driver_sql("SELECT COUNT(*) FROM proxy_endpoints;")).scalar_one())
|
||||
proxies_enabled = int(
|
||||
(await conn.exec_driver_sql("SELECT COUNT(*) FROM proxy_endpoints WHERE enabled=1;")).scalar_one()
|
||||
)
|
||||
|
||||
pools_total = int((await conn.exec_driver_sql("SELECT COUNT(*) FROM proxy_pools;")).scalar_one())
|
||||
pools_enabled = int(
|
||||
(await conn.exec_driver_sql("SELECT COUNT(*) FROM proxy_pools WHERE enabled=1;")).scalar_one()
|
||||
)
|
||||
|
||||
bindings_total = int((await conn.exec_driver_sql("SELECT COUNT(*) FROM token_proxy_bindings;")).scalar_one())
|
||||
|
||||
jobs_counts: dict[str, int] = {}
|
||||
rows = (await conn.exec_driver_sql("SELECT status, COUNT(*) AS c FROM jobs GROUP BY status;")).fetchall()
|
||||
for status, count in rows:
|
||||
jobs_counts[str(status)] = int(count)
|
||||
|
||||
worker_last_seen_json = (
|
||||
await conn.exec_driver_sql("SELECT value_json FROM runtime_settings WHERE key = ?;", ("worker.last_seen_at",))
|
||||
).scalar_one_or_none()
|
||||
worker_last_seen_at = _worker_last_seen_from_value_json(str(worker_last_seen_json) if worker_last_seen_json is not None else None)
|
||||
|
||||
return {
|
||||
"images": {"total": images_total, "enabled": images_enabled},
|
||||
"hydration": {
|
||||
"enabled_images_total": images_enabled,
|
||||
"missing": {
|
||||
"tags": missing_tags,
|
||||
"geometry": missing_geometry,
|
||||
"r18": missing_r18,
|
||||
"ai": missing_ai,
|
||||
"illust_type": missing_illust_type,
|
||||
"user": missing_user,
|
||||
"title": missing_title,
|
||||
"created_at": missing_created_at,
|
||||
"popularity": missing_popularity,
|
||||
},
|
||||
},
|
||||
"tokens": {"total": tokens_total, "enabled": tokens_enabled},
|
||||
"proxies": {"endpoints_total": proxies_total, "endpoints_enabled": proxies_enabled},
|
||||
"proxy_pools": {"total": pools_total, "enabled": pools_enabled},
|
||||
"bindings": {"total": bindings_total},
|
||||
"jobs": {"counts": jobs_counts},
|
||||
"worker": {"last_seen_at": worker_last_seen_at},
|
||||
}
|
||||
|
||||
counts = await with_sqlite_busy_retry(_op)
|
||||
return {"ok": True, "counts": counts, "request_id": rid}
|
||||
@@ -0,0 +1,450 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any
|
||||
|
||||
import sqlalchemy as sa
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.crypto import FieldEncryptor, mask_secret
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.proxy_routing import select_proxy_uri_for_url
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.core.runtime_settings import load_runtime_config
|
||||
from app.core.time import iso_utc_ms
|
||||
from app.db.models.pixiv_tokens import PixivToken
|
||||
from app.db.models.token_proxy_bindings import TokenProxyBinding
|
||||
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
|
||||
from app.pixiv.oauth import OAUTH_TOKEN_PATH, PixivOauthConfig, PixivOauthError, refresh_access_token
|
||||
from app.pixiv.refresh_backoff import refresh_backoff_seconds
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _parse_bool(value: Any, *, default: bool) -> bool:
|
||||
if value is None:
|
||||
return default
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, int) and value in (0, 1):
|
||||
return bool(value)
|
||||
if isinstance(value, str):
|
||||
v = value.strip().lower()
|
||||
if v in {"1", "true", "yes", "y", "on"}:
|
||||
return True
|
||||
if v in {"0", "false", "no", "n", "off"}:
|
||||
return False
|
||||
return default
|
||||
|
||||
|
||||
def _parse_bool_strict(value: Any) -> bool | None:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, int) and value in (0, 1):
|
||||
return bool(value)
|
||||
if isinstance(value, str):
|
||||
v = value.strip().lower()
|
||||
if v in {"1", "true", "yes", "y", "on"}:
|
||||
return True
|
||||
if v in {"0", "false", "no", "n", "off"}:
|
||||
return False
|
||||
return None
|
||||
|
||||
|
||||
async def _load_create_token_json(request: Request) -> dict[str, Any]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
|
||||
refresh_token = str(data.get("refresh_token") or "").strip()
|
||||
if not refresh_token:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing refresh_token", status_code=400)
|
||||
if len(refresh_token) > 2048:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported refresh_token", status_code=400)
|
||||
|
||||
label_raw = data.get("label")
|
||||
label = str(label_raw).strip() if label_raw is not None else None
|
||||
label = label if label else None
|
||||
|
||||
enabled = _parse_bool(data.get("enabled"), default=True)
|
||||
|
||||
weight_raw = data.get("weight", 1.0)
|
||||
try:
|
||||
weight = float(weight_raw)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported weight", status_code=400) from exc
|
||||
if weight < 0.0 or weight > 100.0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported weight", status_code=400)
|
||||
|
||||
return {
|
||||
"label": label,
|
||||
"enabled": enabled,
|
||||
"weight": weight,
|
||||
"refresh_token": refresh_token,
|
||||
}
|
||||
|
||||
|
||||
async def _load_update_token_json(request: Request) -> dict[str, Any]:
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
|
||||
if not data:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing fields", status_code=400)
|
||||
|
||||
out: dict[str, Any] = {}
|
||||
|
||||
if "label" in data:
|
||||
label_raw = data.get("label")
|
||||
if label_raw is None:
|
||||
label = None
|
||||
else:
|
||||
label = str(label_raw).strip() or None
|
||||
if label is not None and len(label) > 200:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported label", status_code=400)
|
||||
out["label"] = label
|
||||
|
||||
if "enabled" in data:
|
||||
enabled = _parse_bool_strict(data.get("enabled"))
|
||||
if enabled is None:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported enabled", status_code=400)
|
||||
out["enabled"] = bool(enabled)
|
||||
|
||||
if "weight" in data:
|
||||
weight_raw = data.get("weight")
|
||||
try:
|
||||
weight = float(weight_raw)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported weight", status_code=400) from exc
|
||||
if weight < 0.0 or weight > 100.0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported weight", status_code=400)
|
||||
out["weight"] = float(weight)
|
||||
|
||||
if not out:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing fields", status_code=400)
|
||||
|
||||
return out
|
||||
|
||||
|
||||
@router.get("/tokens")
|
||||
async def list_tokens(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async with Session() as session:
|
||||
tokens = (
|
||||
(
|
||||
await session.execute(
|
||||
sa.select(PixivToken).order_by(PixivToken.id.desc())
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
|
||||
items = [
|
||||
{
|
||||
"id": str(t.id),
|
||||
"label": t.label,
|
||||
"enabled": bool(t.enabled),
|
||||
"refresh_token_masked": t.refresh_token_masked,
|
||||
"weight": float(t.weight),
|
||||
"error_count": int(t.error_count or 0),
|
||||
"backoff_until": t.backoff_until,
|
||||
"last_ok_at": t.last_ok_at,
|
||||
"last_fail_at": t.last_fail_at,
|
||||
"last_error_code": t.last_error_code,
|
||||
"last_error_msg": t.last_error_msg,
|
||||
}
|
||||
for t in tokens
|
||||
]
|
||||
|
||||
return {"ok": True, "items": items, "request_id": rid}
|
||||
|
||||
|
||||
@router.post("/tokens")
|
||||
async def create_token(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
rid = get_or_create_request_id(request)
|
||||
body = await _load_create_token_json(request)
|
||||
|
||||
settings = request.app.state.settings
|
||||
try:
|
||||
encryptor = FieldEncryptor.from_key(settings.field_encryption_key)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Encryption not configured", status_code=500) from exc
|
||||
|
||||
refresh_token = str(body["refresh_token"])
|
||||
refresh_token_enc = encryptor.encrypt_text(refresh_token)
|
||||
refresh_token_masked = mask_secret(refresh_token)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
async with Session() as session:
|
||||
row = PixivToken(
|
||||
label=body["label"],
|
||||
enabled=1 if bool(body["enabled"]) else 0,
|
||||
refresh_token_enc=refresh_token_enc,
|
||||
refresh_token_masked=refresh_token_masked,
|
||||
weight=float(body["weight"]),
|
||||
)
|
||||
session.add(row)
|
||||
await session.commit()
|
||||
await session.refresh(row)
|
||||
|
||||
return {"ok": True, "token_id": str(row.id), "request_id": rid}
|
||||
|
||||
|
||||
@router.put("/tokens/{token_id}")
|
||||
async def update_token(
|
||||
token_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if token_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid token id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
body = await _load_update_token_json(request)
|
||||
|
||||
now = iso_utc_ms()
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async with Session() as session:
|
||||
row = await session.get(PixivToken, token_id)
|
||||
if row is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Token not found", status_code=404)
|
||||
|
||||
if "label" in body:
|
||||
row.label = body["label"]
|
||||
if "enabled" in body:
|
||||
row.enabled = 1 if bool(body["enabled"]) else 0
|
||||
if "weight" in body:
|
||||
row.weight = float(body["weight"])
|
||||
|
||||
row.updated_at = now
|
||||
|
||||
await session.commit()
|
||||
|
||||
return {"ok": True, "token_id": str(token_id), "request_id": rid}
|
||||
|
||||
|
||||
@router.delete("/tokens/{token_id}")
|
||||
async def delete_token(
|
||||
token_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if token_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid token id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with Session() as session:
|
||||
row = await session.get(PixivToken, token_id)
|
||||
if row is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Token not found", status_code=404)
|
||||
|
||||
await session.execute(sa.delete(TokenProxyBinding).where(TokenProxyBinding.token_id == int(token_id)))
|
||||
await session.delete(row)
|
||||
await session.commit()
|
||||
|
||||
return {"ok": True, "token_id": str(token_id), "request_id": rid}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
@router.post("/tokens/{token_id}/test-refresh")
|
||||
async def test_refresh_token(
|
||||
token_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if token_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid token id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
settings = request.app.state.settings
|
||||
try:
|
||||
encryptor = FieldEncryptor.from_key(settings.field_encryption_key)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Encryption not configured", status_code=500) from exc
|
||||
|
||||
client_id = (settings.pixiv_oauth_client_id or "").strip()
|
||||
client_secret = (settings.pixiv_oauth_client_secret or "").strip()
|
||||
if not client_id or not client_secret:
|
||||
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Pixiv OAuth not configured", status_code=500)
|
||||
|
||||
config = PixivOauthConfig(
|
||||
client_id=client_id,
|
||||
client_secret=client_secret,
|
||||
hash_secret=(settings.pixiv_oauth_hash_secret or "").strip() or None,
|
||||
)
|
||||
|
||||
transport = getattr(request.app.state, "httpx_transport", None)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async with Session() as session:
|
||||
row = await session.get(PixivToken, token_id)
|
||||
if row is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Token not found", status_code=404)
|
||||
|
||||
try:
|
||||
refresh_token = encryptor.decrypt_text(row.refresh_token_enc)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Invalid stored token", status_code=500) from exc
|
||||
|
||||
now = iso_utc_ms()
|
||||
|
||||
try:
|
||||
runtime = await load_runtime_config(engine)
|
||||
oauth_url = config.base_url.rstrip("/") + OAUTH_TOKEN_PATH
|
||||
picked_proxy = await select_proxy_uri_for_url(
|
||||
engine,
|
||||
settings,
|
||||
runtime,
|
||||
url=oauth_url,
|
||||
token_id=int(token_id),
|
||||
)
|
||||
proxy_uri = picked_proxy.uri if picked_proxy is not None else None
|
||||
|
||||
token = await refresh_access_token(
|
||||
refresh_token=refresh_token,
|
||||
config=config,
|
||||
transport=transport,
|
||||
proxy=proxy_uri,
|
||||
)
|
||||
except PixivOauthError as exc:
|
||||
new_error_count = int(row.error_count or 0) + 1
|
||||
backoff_s = refresh_backoff_seconds(attempt=new_error_count, status_code=exc.status_code)
|
||||
backoff_until = (
|
||||
iso_utc_ms(datetime.now(timezone.utc) + timedelta(seconds=backoff_s)) if backoff_s > 0 else None
|
||||
)
|
||||
|
||||
row.error_count = new_error_count
|
||||
row.backoff_until = backoff_until
|
||||
row.last_fail_at = now
|
||||
row.last_error_code = ErrorCode.TOKEN_REFRESH_FAILED.value
|
||||
row.last_error_msg = str(exc)[:500]
|
||||
row.updated_at = now
|
||||
|
||||
await session.commit()
|
||||
|
||||
raise ApiError(
|
||||
code=ErrorCode.TOKEN_REFRESH_FAILED,
|
||||
message="Token refresh failed",
|
||||
status_code=502,
|
||||
details={"upstream_status": exc.status_code or 0, "backoff_until": backoff_until or ""},
|
||||
) from exc
|
||||
except Exception as exc:
|
||||
new_error_count = int(row.error_count or 0) + 1
|
||||
backoff_s = refresh_backoff_seconds(attempt=new_error_count, status_code=None)
|
||||
backoff_until = (
|
||||
iso_utc_ms(datetime.now(timezone.utc) + timedelta(seconds=backoff_s)) if backoff_s > 0 else None
|
||||
)
|
||||
|
||||
row.error_count = new_error_count
|
||||
row.backoff_until = backoff_until
|
||||
row.last_fail_at = now
|
||||
row.last_error_code = ErrorCode.TOKEN_REFRESH_FAILED.value
|
||||
row.last_error_msg = "exception"
|
||||
row.updated_at = now
|
||||
|
||||
await session.commit()
|
||||
|
||||
raise ApiError(
|
||||
code=ErrorCode.TOKEN_REFRESH_FAILED,
|
||||
message="Token refresh failed",
|
||||
status_code=502,
|
||||
details={"backoff_until": backoff_until or ""},
|
||||
) from exc
|
||||
|
||||
row.error_count = 0
|
||||
row.backoff_until = None
|
||||
row.last_ok_at = now
|
||||
row.last_fail_at = None
|
||||
row.last_error_code = None
|
||||
row.last_error_msg = None
|
||||
row.updated_at = now
|
||||
|
||||
rotated = token.refresh_token
|
||||
if rotated:
|
||||
row.refresh_token_enc = encryptor.encrypt_text(rotated)
|
||||
row.refresh_token_masked = mask_secret(rotated)
|
||||
|
||||
await session.commit()
|
||||
|
||||
proxy_details: dict[str, Any] | None = None
|
||||
if picked_proxy is not None:
|
||||
proxy_details = {"endpoint_id": str(picked_proxy.endpoint_id), "pool_id": str(picked_proxy.pool_id)}
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"expires_in": int(token.expires_in),
|
||||
"user_id": token.user_id,
|
||||
"proxy": proxy_details,
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/tokens/{token_id}/reset-failures")
|
||||
async def reset_failures(
|
||||
token_id: int,
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> dict[str, Any]:
|
||||
_ = _claims
|
||||
if token_id <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid token id", status_code=400)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
|
||||
now = iso_utc_ms()
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async with Session() as session:
|
||||
row = await session.get(PixivToken, token_id)
|
||||
if row is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Token not found", status_code=404)
|
||||
|
||||
row.error_count = 0
|
||||
row.backoff_until = None
|
||||
row.last_fail_at = None
|
||||
row.last_error_code = None
|
||||
row.last_error_msg = None
|
||||
row.updated_at = now
|
||||
|
||||
await session.commit()
|
||||
|
||||
return {"ok": True, "token_id": str(token_id), "request_id": rid}
|
||||
@@ -0,0 +1,88 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from fastapi.responses import Response
|
||||
from prometheus_client.exposition import CONTENT_TYPE_LATEST, generate_latest
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine
|
||||
|
||||
from app.api.admin.deps import get_admin_claims
|
||||
from app.core.metrics import (
|
||||
METRICS_LAST_SCRAPE_SUCCESS,
|
||||
METRICS_SCRAPE_ERRORS_TOTAL,
|
||||
ensure_known_keys,
|
||||
set_jobs_status_counts,
|
||||
set_proxy_state_counts,
|
||||
)
|
||||
from app.core.time import iso_utc_ms
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
async def _query_job_status_counts(engine: AsyncEngine) -> dict[str, int]:
|
||||
sql = "SELECT status, COUNT(*) AS c FROM jobs GROUP BY status"
|
||||
async with engine.connect() as conn:
|
||||
result = await conn.exec_driver_sql(sql)
|
||||
counts: dict[str, int] = {}
|
||||
for row in result.fetchall():
|
||||
status = str(row[0])
|
||||
counts[status] = int(row[1])
|
||||
return counts
|
||||
|
||||
|
||||
async def _query_proxy_state_counts(engine: AsyncEngine) -> dict[str, int]:
|
||||
now = iso_utc_ms()
|
||||
sql = """
|
||||
SELECT
|
||||
COUNT(*) AS total,
|
||||
SUM(CASE WHEN enabled = 1 THEN 1 ELSE 0 END) AS enabled,
|
||||
SUM(CASE WHEN enabled = 1 AND blacklisted_until IS NOT NULL AND blacklisted_until > :now THEN 1 ELSE 0 END) AS blacklisted,
|
||||
SUM(CASE WHEN enabled = 1
|
||||
AND (blacklisted_until IS NULL OR blacklisted_until <= :now)
|
||||
AND last_ok_at IS NOT NULL
|
||||
AND (last_fail_at IS NULL OR last_ok_at >= last_fail_at)
|
||||
THEN 1 ELSE 0 END) AS healthy,
|
||||
SUM(CASE WHEN enabled = 1
|
||||
AND (blacklisted_until IS NULL OR blacklisted_until <= :now)
|
||||
AND (last_ok_at IS NULL OR (last_fail_at IS NOT NULL AND last_fail_at > last_ok_at))
|
||||
THEN 1 ELSE 0 END) AS unhealthy
|
||||
FROM proxy_endpoints;
|
||||
""".strip()
|
||||
async with engine.connect() as conn:
|
||||
result = await conn.exec_driver_sql(sql, {"now": now})
|
||||
row = result.fetchone()
|
||||
if row is None:
|
||||
return {"total": 0, "enabled": 0, "healthy": 0, "unhealthy": 0, "blacklisted": 0}
|
||||
return {
|
||||
"total": int(row[0] or 0),
|
||||
"enabled": int(row[1] or 0),
|
||||
"blacklisted": int(row[2] or 0),
|
||||
"healthy": int(row[3] or 0),
|
||||
"unhealthy": int(row[4] or 0),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/metrics")
|
||||
async def metrics(
|
||||
request: Request,
|
||||
_claims: dict[str, Any] = Depends(get_admin_claims),
|
||||
) -> Response:
|
||||
engine: AsyncEngine | None = getattr(request.app.state, "engine", None)
|
||||
if engine is not None:
|
||||
try:
|
||||
job_counts = await _query_job_status_counts(engine)
|
||||
set_jobs_status_counts(job_counts)
|
||||
|
||||
proxy_counts = await _query_proxy_state_counts(engine)
|
||||
proxy_counts = ensure_known_keys(["total", "enabled", "healthy", "unhealthy", "blacklisted"], proxy_counts)
|
||||
set_proxy_state_counts(proxy_counts)
|
||||
|
||||
METRICS_LAST_SCRAPE_SUCCESS.set(1)
|
||||
except Exception:
|
||||
METRICS_SCRAPE_ERRORS_TOTAL.inc()
|
||||
METRICS_LAST_SCRAPE_SUCCESS.set(0)
|
||||
|
||||
content = generate_latest()
|
||||
return Response(content=content, media_type=CONTENT_TYPE_LATEST)
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.db.authors_list import list_authors as db_list_authors
|
||||
from app.db.session import create_sessionmaker
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/authors")
|
||||
async def list_authors(
|
||||
request: Request,
|
||||
q: str | None = None,
|
||||
limit: int = 50,
|
||||
cursor: str | None = None,
|
||||
) -> Any:
|
||||
if limit < 1 or limit > 100:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
|
||||
|
||||
q_norm = (q or "").strip()
|
||||
if q_norm and len(q_norm) > 200:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported q", status_code=400)
|
||||
|
||||
cursor_i: int | None = None
|
||||
cursor_raw = (cursor or "").strip()
|
||||
if cursor_raw:
|
||||
if not cursor_raw.isdigit():
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
cursor_i = int(cursor_raw)
|
||||
if cursor_i <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
async with Session() as session:
|
||||
items, next_cursor = await db_list_authors(session, limit=limit, cursor=cursor_i, q=q_norm or None)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
resp = JSONResponse(
|
||||
status_code=200,
|
||||
content={
|
||||
"ok": True,
|
||||
"items": [
|
||||
{
|
||||
"user_id": str(item.user_id),
|
||||
"user_name": item.user_name,
|
||||
"count_images": item.count_images,
|
||||
}
|
||||
for item in items
|
||||
],
|
||||
"next_cursor": str(next_cursor) if next_cursor is not None else "",
|
||||
"request_id": rid,
|
||||
},
|
||||
)
|
||||
return resp
|
||||
|
||||
@@ -0,0 +1,361 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import HTMLResponse
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _build_docs_html(*, base_url: str) -> str:
|
||||
base = (base_url or "").rstrip("/")
|
||||
if not base:
|
||||
base = ""
|
||||
|
||||
def u(path: str) -> str:
|
||||
path_norm = (path or "").strip()
|
||||
if not path_norm.startswith("/"):
|
||||
path_norm = "/" + path_norm
|
||||
return f"{base}{path_norm}"
|
||||
|
||||
examples = {
|
||||
"img_default": u("/random"),
|
||||
"img_redirect": u("/random?redirect=1"),
|
||||
"json_full": u("/random?format=json"),
|
||||
"json_simple": u("/random?format=simple_json"),
|
||||
"r18_only": u("/random?r18=1"),
|
||||
"safe_only": u("/random?r18=0"),
|
||||
"any_r18": u("/random?r18=2"),
|
||||
"portrait": u("/random?orientation=portrait"),
|
||||
"landscape": u("/random?orientation=landscape"),
|
||||
"adaptive": u("/random?adaptive=1"),
|
||||
"tag_loli": u("/random?included_tags=loli"),
|
||||
"pure_random": u("/random?strategy=random"),
|
||||
"quality_strong": u(
|
||||
"/random?strategy=quality&quality_samples=200&min_pixels=2000000&min_bookmarks=100&illust_type=illust&ai_type=0"
|
||||
),
|
||||
"complex": u(
|
||||
"/random?format=json&r18=1&illust_type=illust&orientation=portrait&min_pixels=2500000&min_bookmarks=25&min_comments=50&included_tags=loli"
|
||||
),
|
||||
"tags_api": u("/tags"),
|
||||
"authors_api": u("/authors"),
|
||||
"swagger": u("/api/docs"),
|
||||
"openapi": u("/openapi.json"),
|
||||
}
|
||||
|
||||
return f"""<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Random Mage Docs</title>
|
||||
<style>
|
||||
:root {{
|
||||
--bg: #0b1220;
|
||||
--card: rgba(255, 255, 255, 0.06);
|
||||
--card-2: rgba(255, 255, 255, 0.08);
|
||||
--border: rgba(255, 255, 255, 0.12);
|
||||
--text: rgba(255, 255, 255, 0.92);
|
||||
--muted: rgba(255, 255, 255, 0.72);
|
||||
--muted2: rgba(255, 255, 255, 0.60);
|
||||
--link: #7dd3fc;
|
||||
--accent: #a78bfa;
|
||||
--ok: #34d399;
|
||||
--warn: #fbbf24;
|
||||
--mono: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace;
|
||||
--sans: ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, Helvetica, Arial, "Apple Color Emoji",
|
||||
"Segoe UI Emoji";
|
||||
}}
|
||||
|
||||
* {{ box-sizing: border-box; }}
|
||||
body {{
|
||||
margin: 0;
|
||||
font-family: var(--sans);
|
||||
color: var(--text);
|
||||
background:
|
||||
radial-gradient(1000px 600px at 15% 10%, rgba(167, 139, 250, 0.22), transparent 55%),
|
||||
radial-gradient(900px 520px at 85% 0%, rgba(125, 211, 252, 0.18), transparent 60%),
|
||||
radial-gradient(700px 700px at 60% 85%, rgba(52, 211, 153, 0.12), transparent 55%),
|
||||
var(--bg);
|
||||
}}
|
||||
|
||||
a {{ color: var(--link); text-decoration: none; }}
|
||||
a:hover {{ text-decoration: underline; }}
|
||||
|
||||
.wrap {{
|
||||
max-width: 980px;
|
||||
margin: 0 auto;
|
||||
padding: 32px 20px 64px;
|
||||
}}
|
||||
|
||||
.hero {{
|
||||
padding: 18px 18px 16px;
|
||||
border: 1px solid var(--border);
|
||||
background: linear-gradient(180deg, rgba(255,255,255,0.08), rgba(255,255,255,0.04));
|
||||
border-radius: 14px;
|
||||
backdrop-filter: blur(10px);
|
||||
}}
|
||||
|
||||
.hero h1 {{
|
||||
margin: 0;
|
||||
font-size: 22px;
|
||||
letter-spacing: 0.2px;
|
||||
}}
|
||||
.hero p {{
|
||||
margin: 10px 0 0;
|
||||
color: var(--muted);
|
||||
line-height: 1.55;
|
||||
}}
|
||||
|
||||
.grid {{
|
||||
display: grid;
|
||||
grid-template-columns: 1fr;
|
||||
gap: 14px;
|
||||
margin-top: 14px;
|
||||
}}
|
||||
@media (min-width: 860px) {{
|
||||
.grid {{ grid-template-columns: 1fr 1fr; }}
|
||||
}}
|
||||
|
||||
.card {{
|
||||
border: 1px solid var(--border);
|
||||
background: var(--card);
|
||||
border-radius: 14px;
|
||||
padding: 16px 16px 14px;
|
||||
}}
|
||||
.card h2 {{
|
||||
font-size: 16px;
|
||||
margin: 0 0 10px;
|
||||
letter-spacing: 0.2px;
|
||||
}}
|
||||
.card p {{
|
||||
margin: 0 0 10px;
|
||||
color: var(--muted);
|
||||
line-height: 1.55;
|
||||
}}
|
||||
|
||||
.kbd {{
|
||||
display: inline-block;
|
||||
font-family: var(--mono);
|
||||
font-size: 12px;
|
||||
padding: 2px 6px;
|
||||
border-radius: 8px;
|
||||
border: 1px solid var(--border);
|
||||
background: rgba(0, 0, 0, 0.18);
|
||||
color: rgba(255,255,255,0.86);
|
||||
}}
|
||||
|
||||
pre {{
|
||||
margin: 10px 0 0;
|
||||
padding: 12px 12px;
|
||||
border-radius: 12px;
|
||||
border: 1px solid var(--border);
|
||||
background: rgba(0, 0, 0, 0.22);
|
||||
overflow: auto;
|
||||
font-family: var(--mono);
|
||||
font-size: 12px;
|
||||
line-height: 1.55;
|
||||
color: rgba(255,255,255,0.88);
|
||||
}}
|
||||
|
||||
.chips {{
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 8px;
|
||||
margin-top: 10px;
|
||||
}}
|
||||
.chip {{
|
||||
padding: 6px 10px;
|
||||
border-radius: 999px;
|
||||
border: 1px solid var(--border);
|
||||
background: var(--card-2);
|
||||
font-size: 12px;
|
||||
color: rgba(255,255,255,0.85);
|
||||
white-space: nowrap;
|
||||
}}
|
||||
.chip strong {{ color: rgba(255,255,255,0.94); }}
|
||||
|
||||
table {{
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
margin-top: 10px;
|
||||
border-radius: 12px;
|
||||
overflow: hidden;
|
||||
border: 1px solid var(--border);
|
||||
}}
|
||||
th, td {{
|
||||
padding: 10px 10px;
|
||||
border-bottom: 1px solid rgba(255,255,255,0.10);
|
||||
vertical-align: top;
|
||||
font-size: 13px;
|
||||
line-height: 1.45;
|
||||
}}
|
||||
th {{
|
||||
text-align: left;
|
||||
background: rgba(255,255,255,0.06);
|
||||
color: rgba(255,255,255,0.92);
|
||||
font-weight: 600;
|
||||
}}
|
||||
td code {{
|
||||
font-family: var(--mono);
|
||||
font-size: 12px;
|
||||
color: rgba(255,255,255,0.9);
|
||||
}}
|
||||
.muted {{ color: var(--muted2); }}
|
||||
.note {{
|
||||
margin-top: 12px;
|
||||
padding: 12px 12px;
|
||||
border-radius: 12px;
|
||||
border: 1px solid var(--border);
|
||||
background: rgba(251, 191, 36, 0.10);
|
||||
color: rgba(255,255,255,0.90);
|
||||
}}
|
||||
.note strong {{ color: rgba(255,255,255,0.95); }}
|
||||
.footer {{
|
||||
margin-top: 18px;
|
||||
color: var(--muted2);
|
||||
font-size: 12px;
|
||||
line-height: 1.6;
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
<div class="hero">
|
||||
<h1>Random Mage · 随机图片 API</h1>
|
||||
<p>质量优先随机 + 强力筛选,支持标签/热度/分辨率/R18/AI/作品类型等组合查询。</p>
|
||||
<div class="chips" aria-label="quick-links">
|
||||
<a class="chip" href="{examples["img_default"]}"><strong>/random</strong> 直接出图</a>
|
||||
<a class="chip" href="{examples["json_full"]}"><strong>format=json</strong> 返回 JSON</a>
|
||||
<a class="chip" href="{examples["pure_random"]}"><strong>strategy=random</strong> 纯随机</a>
|
||||
<a class="chip" href="{examples["swagger"]}"><strong>/api/docs</strong> Swagger</a>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="grid">
|
||||
<section class="card">
|
||||
<h2>1) 最常用(直接出图)</h2>
|
||||
<p>默认返回图片(不是 JSON)。想要稳定 URL 可加 <span class="kbd">redirect=1</span> 跳转到本站缓存/代理路径。</p>
|
||||
<pre><code>{examples["img_default"]}
|
||||
{examples["img_redirect"]}</code></pre>
|
||||
</section>
|
||||
|
||||
<section class="card">
|
||||
<h2>2) JSON / 调试信息</h2>
|
||||
<p><span class="kbd">format=json</span> 返回完整数据(含 tags / debug);<span class="kbd">format=simple_json</span> 更轻量。</p>
|
||||
<pre><code>{examples["json_full"]}
|
||||
{examples["json_simple"]}</code></pre>
|
||||
</section>
|
||||
|
||||
<section class="card">
|
||||
<h2>3) R18 / 作品类型 / AI</h2>
|
||||
<p>R18 用 <span class="kbd">r18</span> 控制:0=仅全年龄(默认),1=仅R18,2=都可。</p>
|
||||
<pre><code>{examples["safe_only"]}
|
||||
{examples["r18_only"]}
|
||||
{examples["any_r18"]}</code></pre>
|
||||
<div class="note">
|
||||
<strong>提示:</strong>如果你库里有不少作品还没补全 x_restrict,且你想“更容易命中”,可用
|
||||
<span class="kbd">r18=0&r18_strict=0</span>(允许未知的 x_restrict)。
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="card">
|
||||
<h2>4) 横竖图 / 自适应</h2>
|
||||
<p><span class="kbd">orientation</span> 支持:any / portrait / landscape / square。自适应用 <span class="kbd">adaptive=1</span>:移动端默认竖图+更低像素门槛,PC 默认横图+更高门槛(不会覆盖你显式传入的过滤条件)。</p>
|
||||
<pre><code>{examples["portrait"]}
|
||||
{examples["landscape"]}
|
||||
{examples["adaptive"]}</code></pre>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<section class="card" style="margin-top: 14px;">
|
||||
<h2>筛选语法速记</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th style="width: 170px;">参数</th>
|
||||
<th>说明</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>strategy</code></td>
|
||||
<td>
|
||||
随机策略:<code>quality</code>(默认,质量优先) / <code>random</code>(纯随机)。<br/>
|
||||
<span class="muted">quality 会先抽样 N 张候选再按热度/分辨率打分选图。</span>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>quality_samples</code></td>
|
||||
<td>
|
||||
质量抽样数量(1–1000)。越大越“挑剔”,但 DB 开销也更高。<br/>
|
||||
示例:<a href="{examples["quality_strong"]}">{examples["quality_strong"]}</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>included_tags</code></td>
|
||||
<td>
|
||||
必须包含的标签(AND)。可重复传参:<code>included_tags=a&included_tags=b</code>。<br/>
|
||||
也支持用 <code>|</code> 分隔:<code>included_tags=a|b</code> 等价于同时包含 a 与 b。<br/>
|
||||
示例:<a href="{examples["tag_loli"]}">{examples["tag_loli"]}</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>excluded_tags</code></td>
|
||||
<td>必须不包含的标签(OR)。可重复传参。示例:<code>excluded_tags=AI生成</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>min_width</code><br/><code>min_height</code><br/><code>min_pixels</code></td>
|
||||
<td>分辨率门槛:宽/高/像素数(例如 2000000 ≈ 2MP)。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>min_bookmarks</code><br/><code>min_views</code><br/><code>min_comments</code></td>
|
||||
<td>热度门槛:收藏/浏览/评论(需要补全元数据后更准确)。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>ai_type</code></td>
|
||||
<td><code>any</code> / <code>0</code>(非AI) / <code>1</code>(AI)。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>illust_type</code></td>
|
||||
<td><code>any</code> / <code>illust</code> / <code>manga</code> / <code>ugoira</code>。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>user_id</code></td>
|
||||
<td>只返回某个作者的作品(Pixiv 用户 ID)。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>created_from</code><br/><code>created_to</code></td>
|
||||
<td>按发布时间筛选(ISO8601,例如 <code>2024-01-01T00:00:00Z</code>)。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>seed</code></td>
|
||||
<td>固定随机种子(同参数 + 同 seed 更容易复现)。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>attempts</code></td>
|
||||
<td>上游失败时重试次数(1–10)。</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<div class="footer">
|
||||
其它接口:<a href="{examples["tags_api"]}">/tags</a>(标签列表)、<a href="{examples["authors_api"]}">/authors</a>(作者列表)。<br/>
|
||||
OpenAPI:<a href="{examples["openapi"]}">/openapi.json</a>,Swagger:<a href="{examples["swagger"]}">/api/docs</a>。<br/>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="card" style="margin-top: 14px;">
|
||||
<h2>复杂示例(组合查询)</h2>
|
||||
<p>示例:R18 + 插画 + 竖图 + 250万像素以上 + 收藏≥25 + 评论≥50 + 标签包含 loli,并返回 JSON。</p>
|
||||
<pre><code>{examples["complex"]}</code></pre>
|
||||
</section>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
|
||||
@router.get("/docs", include_in_schema=False)
|
||||
async def docs_page(request: Request) -> HTMLResponse:
|
||||
html = _build_docs_html(base_url=str(getattr(request, "base_url", "") or "").rstrip("/"))
|
||||
return HTMLResponse(content=html, status_code=200, headers={"Cache-Control": "no-store"})
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine
|
||||
|
||||
from app.core.errors import ErrorCode, error_body
|
||||
from app.core.request_id import get_or_create_request_id, set_request_id_header, set_request_id_on_state
|
||||
from app.db.session import with_sqlite_busy_retry
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
_JOB_STATUSES: tuple[str, ...] = (
|
||||
"pending",
|
||||
"running",
|
||||
"paused",
|
||||
"canceled",
|
||||
"completed",
|
||||
"failed",
|
||||
"dlq",
|
||||
)
|
||||
|
||||
|
||||
def _ensure_sqlite_dir(engine: AsyncEngine) -> None:
|
||||
url = engine.url
|
||||
if url.get_backend_name() != "sqlite":
|
||||
return
|
||||
db_path = url.database
|
||||
if not db_path or db_path == ":memory:":
|
||||
return
|
||||
Path(db_path).expanduser().resolve().parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
async def _check_db(engine: AsyncEngine) -> bool:
|
||||
try:
|
||||
_ensure_sqlite_dir(engine)
|
||||
async with engine.connect() as conn:
|
||||
await conn.exec_driver_sql("SELECT 1")
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _parse_iso_utc(value: str) -> datetime | None:
|
||||
raw = (value or "").strip()
|
||||
if not raw:
|
||||
return None
|
||||
if raw.endswith("Z"):
|
||||
raw = raw[:-1] + "+00:00"
|
||||
try:
|
||||
dt = datetime.fromisoformat(raw)
|
||||
except Exception:
|
||||
return None
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
return dt.astimezone(timezone.utc)
|
||||
|
||||
|
||||
async def _query_worker_last_seen(engine: AsyncEngine) -> tuple[str | None, str]:
|
||||
sql = "SELECT value_json FROM runtime_settings WHERE key = ?"
|
||||
|
||||
async def _op() -> str | None:
|
||||
async with engine.connect() as conn:
|
||||
result = await conn.exec_driver_sql(sql, ("worker.last_seen_at",))
|
||||
row = result.fetchone()
|
||||
return str(row[0]) if row is not None else None
|
||||
|
||||
try:
|
||||
raw = await with_sqlite_busy_retry(_op)
|
||||
except Exception:
|
||||
return None, "runtime_settings_unavailable"
|
||||
|
||||
if raw is None:
|
||||
return None, "no_heartbeat"
|
||||
|
||||
try:
|
||||
value = json.loads(raw)
|
||||
except Exception:
|
||||
return None, "invalid_json"
|
||||
|
||||
if isinstance(value, dict):
|
||||
at = value.get("at")
|
||||
if isinstance(at, str):
|
||||
return at, "ok"
|
||||
return None, "invalid_value"
|
||||
|
||||
if isinstance(value, str):
|
||||
return value, "ok"
|
||||
|
||||
return None, "invalid_value"
|
||||
|
||||
|
||||
async def _query_queue_status_counts(engine: AsyncEngine) -> tuple[dict[str, int] | None, str]:
|
||||
sql = "SELECT status, COUNT(*) AS c FROM jobs GROUP BY status"
|
||||
|
||||
async def _op() -> dict[str, int]:
|
||||
async with engine.connect() as conn:
|
||||
result = await conn.exec_driver_sql(sql)
|
||||
counts: dict[str, int] = {}
|
||||
for row in result.fetchall():
|
||||
counts[str(row[0])] = int(row[1])
|
||||
return counts
|
||||
|
||||
try:
|
||||
counts = await with_sqlite_busy_retry(_op)
|
||||
except Exception:
|
||||
return None, "jobs_unavailable"
|
||||
|
||||
for status in _JOB_STATUSES:
|
||||
counts.setdefault(status, 0)
|
||||
return counts, "ok"
|
||||
|
||||
|
||||
@router.get("/healthz")
|
||||
async def healthz(request: Request) -> Any:
|
||||
rid = get_or_create_request_id(request)
|
||||
set_request_id_on_state(request, rid)
|
||||
|
||||
engine: AsyncEngine | None = getattr(request.app.state, "engine", None)
|
||||
db_ok = await _check_db(engine) if engine is not None else False
|
||||
|
||||
if db_ok:
|
||||
try:
|
||||
stale_after_s = int((os.environ.get("WORKER_HEARTBEAT_STALE_SECONDS") or "60").strip() or "60")
|
||||
except Exception:
|
||||
stale_after_s = 60
|
||||
stale_after_s = max(1, min(int(stale_after_s), 24 * 60 * 60))
|
||||
|
||||
worker_last_seen_at, worker_reason = await _query_worker_last_seen(engine) # type: ignore[arg-type]
|
||||
worker_ok = False
|
||||
if worker_last_seen_at is not None:
|
||||
last_seen_dt = _parse_iso_utc(worker_last_seen_at)
|
||||
if last_seen_dt is None:
|
||||
worker_reason = "invalid_timestamp"
|
||||
else:
|
||||
worker_ok = datetime.now(timezone.utc) - last_seen_dt <= timedelta(seconds=stale_after_s)
|
||||
if not worker_ok:
|
||||
worker_reason = "stale"
|
||||
|
||||
queue_counts, queue_reason = await _query_queue_status_counts(engine) # type: ignore[arg-type]
|
||||
queue_ok = queue_counts is not None
|
||||
|
||||
resp = JSONResponse(
|
||||
status_code=200,
|
||||
content={
|
||||
"ok": True,
|
||||
"db_ok": True,
|
||||
"worker_ok": worker_ok,
|
||||
"queue_ok": queue_ok,
|
||||
"worker": {
|
||||
"last_seen_at": worker_last_seen_at,
|
||||
"stale_after_s": stale_after_s,
|
||||
"reason": worker_reason,
|
||||
},
|
||||
"queue": {
|
||||
"counts": queue_counts or {},
|
||||
"reason": queue_reason,
|
||||
},
|
||||
"request_id": rid,
|
||||
},
|
||||
)
|
||||
else:
|
||||
resp = JSONResponse(
|
||||
status_code=503,
|
||||
content=error_body(
|
||||
code=ErrorCode.INTERNAL_ERROR,
|
||||
message="Database unavailable",
|
||||
request_id=rid,
|
||||
details={"db_ok": False},
|
||||
),
|
||||
)
|
||||
|
||||
set_request_id_header(resp, rid)
|
||||
return resp
|
||||
@@ -0,0 +1,333 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Query, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.http_stream import stream_url
|
||||
import sqlalchemy as sa
|
||||
|
||||
from app.core.pixiv_urls import ALLOWED_IMAGE_EXTS
|
||||
from app.core.request_id import get_or_create_request_id, set_request_id_header, set_request_id_on_state
|
||||
from app.core.runtime_settings import load_runtime_config
|
||||
from app.core.time import iso_utc_ms
|
||||
from app.core.proxy_routing import select_proxy_uri_for_url
|
||||
from app.db.images_get import get_image_by_id
|
||||
from app.db.images_list import list_images as db_list_images
|
||||
from app.db.images_mark import mark_image_failure, mark_image_ok
|
||||
from app.db.models.image_tags import ImageTag
|
||||
from app.db.session import create_sessionmaker
|
||||
from app.db.tags_get import get_tag_names_for_image
|
||||
from app.jobs.enqueue import enqueue_opportunistic_hydrate_metadata
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
_MAX_TAG_FILTERS = 50
|
||||
|
||||
|
||||
@router.get("/images")
|
||||
async def list_images(
|
||||
request: Request,
|
||||
limit: int = 50,
|
||||
cursor: str | None = None,
|
||||
r18: int = 0,
|
||||
r18_strict: int = 1,
|
||||
ai_type: str = "any",
|
||||
orientation: str = "any",
|
||||
min_width: int = 0,
|
||||
min_height: int = 0,
|
||||
min_pixels: int = 0,
|
||||
included_tags: list[str] | None = Query(default=None),
|
||||
excluded_tags: list[str] | None = Query(default=None),
|
||||
user_id: int | None = None,
|
||||
illust_id: int | None = None,
|
||||
created_from: str | None = None,
|
||||
created_to: str | None = None,
|
||||
) -> Any:
|
||||
if limit < 1 or limit > 200:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
|
||||
|
||||
cursor_i: int | None = None
|
||||
cursor_raw = (cursor or "").strip()
|
||||
if cursor_raw:
|
||||
if not cursor_raw.isdigit():
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
cursor_i = int(cursor_raw)
|
||||
if cursor_i <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
|
||||
|
||||
if r18 not in {0, 1, 2}:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported r18", status_code=400)
|
||||
if r18_strict not in {0, 1}:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported r18_strict", status_code=400)
|
||||
|
||||
ai_type_raw = (ai_type or "any").strip().lower()
|
||||
ai_type_i: int | None = None
|
||||
if ai_type_raw in {"", "any"}:
|
||||
ai_type_i = None
|
||||
elif ai_type_raw in {"0", "1"}:
|
||||
ai_type_i = int(ai_type_raw)
|
||||
else:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported ai_type", status_code=400)
|
||||
|
||||
orientation = (orientation or "").strip().lower()
|
||||
orientation_map = {"any": None, "portrait": 1, "landscape": 2, "square": 3}
|
||||
if orientation not in orientation_map:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported orientation", status_code=400)
|
||||
|
||||
if min_width < 0 or min_height < 0 or min_pixels < 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported min_*", status_code=400)
|
||||
|
||||
def _parse_tags(values: list[str] | None) -> list[str]:
|
||||
out: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for raw in values or []:
|
||||
for part in str(raw).split("|"):
|
||||
name = part.strip()
|
||||
if not name or name in seen:
|
||||
continue
|
||||
seen.add(name)
|
||||
out.append(name)
|
||||
return out
|
||||
|
||||
included = _parse_tags(included_tags)
|
||||
excluded = _parse_tags(excluded_tags)
|
||||
if len(included) > _MAX_TAG_FILTERS or len(excluded) > _MAX_TAG_FILTERS:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Too many tag filters", status_code=400)
|
||||
|
||||
if user_id is not None and int(user_id) <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported user_id", status_code=400)
|
||||
if illust_id is not None and int(illust_id) <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported illust_id", status_code=400)
|
||||
|
||||
def _normalize_iso_utc(value: str) -> str:
|
||||
raw = (value or "").strip()
|
||||
if not raw:
|
||||
raise ValueError("empty datetime")
|
||||
if raw.endswith("Z"):
|
||||
raw = raw[:-1] + "+00:00"
|
||||
dt = datetime.fromisoformat(raw)
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
dt = dt.astimezone(timezone.utc).replace(microsecond=0)
|
||||
return dt.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
created_from_norm: str | None = None
|
||||
created_to_norm: str | None = None
|
||||
try:
|
||||
if created_from is not None:
|
||||
created_from_norm = _normalize_iso_utc(created_from)
|
||||
if created_to is not None:
|
||||
created_to_norm = _normalize_iso_utc(created_to)
|
||||
except Exception:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported created_*", status_code=400)
|
||||
|
||||
if created_from_norm is not None and created_to_norm is not None:
|
||||
if created_from_norm > created_to_norm:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="created_from > created_to", status_code=400)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
async with Session() as session:
|
||||
images, next_cursor = await db_list_images(
|
||||
session,
|
||||
limit=limit,
|
||||
cursor=cursor_i,
|
||||
r18=r18,
|
||||
r18_strict=bool(r18_strict),
|
||||
orientation=orientation_map[orientation],
|
||||
ai_type=ai_type_i,
|
||||
min_width=min_width,
|
||||
min_height=min_height,
|
||||
min_pixels=min_pixels,
|
||||
included_tags=included,
|
||||
excluded_tags=excluded,
|
||||
user_id=user_id,
|
||||
illust_id=illust_id,
|
||||
created_from=created_from_norm,
|
||||
created_to=created_to_norm,
|
||||
)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
set_request_id_on_state(request, rid)
|
||||
|
||||
items = [
|
||||
{
|
||||
"id": str(img.id),
|
||||
"illust_id": str(img.illust_id),
|
||||
"page_index": img.page_index,
|
||||
"ext": img.ext,
|
||||
"width": img.width,
|
||||
"height": img.height,
|
||||
"x_restrict": img.x_restrict,
|
||||
"ai_type": img.ai_type,
|
||||
"bookmark_count": getattr(img, "bookmark_count", None),
|
||||
"view_count": getattr(img, "view_count", None),
|
||||
"comment_count": getattr(img, "comment_count", None),
|
||||
"user": {
|
||||
"id": str(img.user_id) if img.user_id is not None else None,
|
||||
"name": img.user_name,
|
||||
},
|
||||
"title": img.title,
|
||||
"created_at_pixiv": img.created_at_pixiv,
|
||||
}
|
||||
for img in images
|
||||
]
|
||||
|
||||
resp = JSONResponse(
|
||||
status_code=200,
|
||||
content={
|
||||
"ok": True,
|
||||
"items": items,
|
||||
"next_cursor": str(next_cursor) if next_cursor is not None else "",
|
||||
"request_id": rid,
|
||||
},
|
||||
)
|
||||
set_request_id_header(resp, rid)
|
||||
return resp
|
||||
|
||||
|
||||
@router.get("/images/{image_id}")
|
||||
async def get_image(
|
||||
request: Request,
|
||||
image_id: int,
|
||||
) -> Any:
|
||||
if int(image_id) <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported image_id", status_code=400)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async with Session() as session:
|
||||
image = await get_image_by_id(session, image_id=image_id)
|
||||
if image is None:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Image not found", status_code=404)
|
||||
tags = await get_tag_names_for_image(session, image_id=image.id)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
set_request_id_on_state(request, rid)
|
||||
|
||||
resp = JSONResponse(
|
||||
status_code=200,
|
||||
content={
|
||||
"ok": True,
|
||||
"item": {
|
||||
"image": {
|
||||
"id": str(image.id),
|
||||
"illust_id": str(image.illust_id),
|
||||
"page_index": image.page_index,
|
||||
"ext": image.ext,
|
||||
"width": image.width,
|
||||
"height": image.height,
|
||||
"x_restrict": image.x_restrict,
|
||||
"ai_type": image.ai_type,
|
||||
"bookmark_count": getattr(image, "bookmark_count", None),
|
||||
"view_count": getattr(image, "view_count", None),
|
||||
"comment_count": getattr(image, "comment_count", None),
|
||||
"user": {
|
||||
"id": str(image.user_id) if image.user_id is not None else None,
|
||||
"name": image.user_name,
|
||||
},
|
||||
"title": image.title,
|
||||
"created_at_pixiv": image.created_at_pixiv,
|
||||
},
|
||||
"tags": tags,
|
||||
},
|
||||
"request_id": rid,
|
||||
},
|
||||
)
|
||||
set_request_id_header(resp, rid)
|
||||
return resp
|
||||
|
||||
|
||||
@router.get("/i/{image_id}.{ext}")
|
||||
async def proxy_image(
|
||||
request: Request,
|
||||
image_id: int,
|
||||
ext: str,
|
||||
):
|
||||
ext = (ext or "").lower()
|
||||
if ext not in ALLOWED_IMAGE_EXTS:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported ext", status_code=400)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
needs_hydrate = False
|
||||
should_mark_ok = False
|
||||
async with Session() as session:
|
||||
image = await get_image_by_id(session, image_id=image_id)
|
||||
if image is None or (image.ext or "").lower() != ext:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Image not found", status_code=404)
|
||||
should_mark_ok = image.last_ok_at is None or image.last_error_code is not None
|
||||
|
||||
missing_basic_fields = (
|
||||
getattr(image, "width", None) is None
|
||||
or getattr(image, "height", None) is None
|
||||
or getattr(image, "x_restrict", None) is None
|
||||
or getattr(image, "ai_type", None) is None
|
||||
or getattr(image, "user_id", None) is None
|
||||
or not str(getattr(image, "user_name", "") or "").strip()
|
||||
or not str(getattr(image, "title", "") or "").strip()
|
||||
or not str(getattr(image, "created_at_pixiv", "") or "").strip()
|
||||
or getattr(image, "bookmark_count", None) is None
|
||||
or getattr(image, "view_count", None) is None
|
||||
or getattr(image, "comment_count", None) is None
|
||||
)
|
||||
if missing_basic_fields:
|
||||
needs_hydrate = True
|
||||
else:
|
||||
tag_row = (
|
||||
await session.execute(
|
||||
sa.select(ImageTag.image_id).where(ImageTag.image_id == int(image.id)).limit(1)
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
needs_hydrate = tag_row is None
|
||||
|
||||
runtime = await load_runtime_config(engine)
|
||||
proxy_uri = None
|
||||
picked = await select_proxy_uri_for_url(
|
||||
engine,
|
||||
request.app.state.settings,
|
||||
runtime,
|
||||
url=str(image.original_url),
|
||||
)
|
||||
if picked is not None:
|
||||
proxy_uri = picked.uri
|
||||
|
||||
transport = getattr(request.app.state, "httpx_transport", None)
|
||||
now = iso_utc_ms()
|
||||
try:
|
||||
resp = await stream_url(
|
||||
image.original_url,
|
||||
transport=transport,
|
||||
proxy=proxy_uri,
|
||||
cache_control="public, max-age=31536000, immutable",
|
||||
range_header=request.headers.get("Range"),
|
||||
)
|
||||
if should_mark_ok:
|
||||
await mark_image_ok(engine, image_id=int(image.id), now=now)
|
||||
if needs_hydrate:
|
||||
try:
|
||||
await enqueue_opportunistic_hydrate_metadata(engine, illust_id=int(image.illust_id), reason="image_proxy")
|
||||
except Exception:
|
||||
pass
|
||||
return resp
|
||||
except ApiError as exc:
|
||||
if exc.code in {
|
||||
ErrorCode.UPSTREAM_STREAM_ERROR,
|
||||
ErrorCode.UPSTREAM_403,
|
||||
ErrorCode.UPSTREAM_404,
|
||||
ErrorCode.UPSTREAM_RATE_LIMIT,
|
||||
}:
|
||||
await mark_image_failure(
|
||||
engine,
|
||||
image_id=int(image.id),
|
||||
now=now,
|
||||
error_code=exc.code.value,
|
||||
error_message=exc.message,
|
||||
)
|
||||
raise
|
||||
@@ -0,0 +1,100 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.http_stream import stream_url
|
||||
from app.core.pixiv_urls import ALLOWED_IMAGE_EXTS
|
||||
from app.core.proxy_routing import select_proxy_uri_for_url
|
||||
from app.core.runtime_settings import load_runtime_config
|
||||
from app.db.images_get_by_illust import get_image_by_illust_page
|
||||
from app.db.session import create_sessionmaker
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/{illust_id}-{page}.{ext}")
|
||||
async def legacy_multi(
|
||||
request: Request,
|
||||
illust_id: int,
|
||||
page: int,
|
||||
ext: str,
|
||||
):
|
||||
if int(illust_id) <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported illust_id", status_code=400)
|
||||
if int(page) <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported page", status_code=400)
|
||||
|
||||
ext = (ext or "").lower()
|
||||
if ext not in ALLOWED_IMAGE_EXTS:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported ext", status_code=400)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async with Session() as session:
|
||||
image = await get_image_by_illust_page(session, illust_id=illust_id, page_index=int(page) - 1)
|
||||
if image is None or (image.ext or "").lower() != ext:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Image not found", status_code=404)
|
||||
|
||||
runtime = await load_runtime_config(engine)
|
||||
proxy_uri = None
|
||||
picked = await select_proxy_uri_for_url(
|
||||
engine,
|
||||
request.app.state.settings,
|
||||
runtime,
|
||||
url=str(image.original_url),
|
||||
)
|
||||
if picked is not None:
|
||||
proxy_uri = picked.uri
|
||||
|
||||
transport = getattr(request.app.state, "httpx_transport", None)
|
||||
return await stream_url(
|
||||
image.original_url,
|
||||
transport=transport,
|
||||
proxy=proxy_uri,
|
||||
cache_control="public, max-age=31536000, immutable",
|
||||
range_header=request.headers.get("Range"),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{illust_id}.{ext}")
|
||||
async def legacy_single(
|
||||
request: Request,
|
||||
illust_id: int,
|
||||
ext: str,
|
||||
):
|
||||
if int(illust_id) <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported illust_id", status_code=400)
|
||||
|
||||
ext = (ext or "").lower()
|
||||
if ext not in ALLOWED_IMAGE_EXTS:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported ext", status_code=400)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async with Session() as session:
|
||||
image = await get_image_by_illust_page(session, illust_id=illust_id, page_index=0)
|
||||
if image is None or (image.ext or "").lower() != ext:
|
||||
raise ApiError(code=ErrorCode.NOT_FOUND, message="Image not found", status_code=404)
|
||||
|
||||
runtime = await load_runtime_config(engine)
|
||||
proxy_uri = None
|
||||
picked = await select_proxy_uri_for_url(
|
||||
engine,
|
||||
request.app.state.settings,
|
||||
runtime,
|
||||
url=str(image.original_url),
|
||||
)
|
||||
if picked is not None:
|
||||
proxy_uri = picked.uri
|
||||
|
||||
transport = getattr(request.app.state, "httpx_transport", None)
|
||||
return await stream_url(
|
||||
image.original_url,
|
||||
transport=transport,
|
||||
proxy=proxy_uri,
|
||||
cache_control="public, max-age=31536000, immutable",
|
||||
range_header=request.headers.get("Range"),
|
||||
)
|
||||
@@ -0,0 +1,861 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
import os
|
||||
import math
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Query, Request
|
||||
from fastapi.responses import RedirectResponse
|
||||
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.http_stream import stream_url
|
||||
from app.core.imgproxy import build_signed_processing_url, load_imgproxy_config_from_settings
|
||||
from app.core.proxy_routing import select_proxy_uri_for_url
|
||||
from app.core.runtime_settings import load_runtime_config
|
||||
from app.core.time import iso_utc_ms
|
||||
from app.db.images_mark import mark_image_failure, mark_image_ok
|
||||
from app.db.tags_get import get_tag_names_for_image
|
||||
from app.db.random_pick import pick_random_image, pick_random_images
|
||||
from app.db.session import create_sessionmaker
|
||||
from app.jobs.enqueue import enqueue_opportunistic_hydrate_metadata
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
_MAX_TAG_FILTERS = 50
|
||||
|
||||
|
||||
def _as_nonneg_int(value: Any) -> int:
|
||||
if value is None:
|
||||
return 0
|
||||
if isinstance(value, bool):
|
||||
return 0
|
||||
try:
|
||||
i = int(value)
|
||||
except Exception:
|
||||
return 0
|
||||
return i if i > 0 else 0
|
||||
|
||||
|
||||
_DEFAULT_SCORE_WEIGHTS: dict[str, float] = {
|
||||
"bookmark": 4.0,
|
||||
"view": 0.5,
|
||||
"comment": 2.0,
|
||||
"pixels": 1.0,
|
||||
"bookmark_rate": 3.0,
|
||||
}
|
||||
|
||||
_DEFAULT_RECOMMENDATION: dict[str, Any] = {
|
||||
"pick_mode": "weighted",
|
||||
"temperature": 1.0,
|
||||
"score_weights": dict(_DEFAULT_SCORE_WEIGHTS),
|
||||
"multipliers": {
|
||||
"ai": 1.0,
|
||||
"non_ai": 1.0,
|
||||
"unknown_ai": 1.0,
|
||||
"illust": 1.0,
|
||||
"manga": 1.0,
|
||||
"ugoira": 1.0,
|
||||
"unknown_illust_type": 1.0,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _as_float(value: Any, *, default: float) -> float:
|
||||
if value is None:
|
||||
return float(default)
|
||||
if isinstance(value, bool):
|
||||
return float(default)
|
||||
try:
|
||||
return float(value)
|
||||
except Exception:
|
||||
return float(default)
|
||||
|
||||
|
||||
def _quality_score(image: Any, *, weights: dict[str, float] | None = None) -> float:
|
||||
w = weights or _DEFAULT_SCORE_WEIGHTS
|
||||
w_bookmark = float(w.get("bookmark", _DEFAULT_SCORE_WEIGHTS["bookmark"]))
|
||||
w_view = float(w.get("view", _DEFAULT_SCORE_WEIGHTS["view"]))
|
||||
w_comment = float(w.get("comment", _DEFAULT_SCORE_WEIGHTS["comment"]))
|
||||
w_pixels = float(w.get("pixels", _DEFAULT_SCORE_WEIGHTS["pixels"]))
|
||||
w_bookmark_rate = float(w.get("bookmark_rate", _DEFAULT_SCORE_WEIGHTS["bookmark_rate"]))
|
||||
|
||||
bookmark_count = _as_nonneg_int(getattr(image, "bookmark_count", None))
|
||||
view_count = _as_nonneg_int(getattr(image, "view_count", None))
|
||||
comment_count = _as_nonneg_int(getattr(image, "comment_count", None))
|
||||
|
||||
width = _as_nonneg_int(getattr(image, "width", None))
|
||||
height = _as_nonneg_int(getattr(image, "height", None))
|
||||
pixels = width * height if width > 0 and height > 0 else 0
|
||||
|
||||
rate_term = 0.0
|
||||
if view_count > 0:
|
||||
bookmark_rate_per_mille = (float(bookmark_count) / float(view_count)) * 1000.0
|
||||
rate_term = math.log1p(max(0.0, bookmark_rate_per_mille))
|
||||
|
||||
score = (
|
||||
float(w_bookmark) * math.log1p(bookmark_count)
|
||||
+ float(w_view) * math.log1p(view_count)
|
||||
+ float(w_comment) * math.log1p(comment_count)
|
||||
+ float(w_pixels) * math.log1p(float(pixels) / 1_000_000.0)
|
||||
+ float(w_bookmark_rate) * rate_term
|
||||
)
|
||||
return float(score)
|
||||
|
||||
|
||||
@router.get("/random")
|
||||
async def random_image(
|
||||
request: Request,
|
||||
format: str = "image",
|
||||
redirect: int = 0,
|
||||
attempts: int | None = None,
|
||||
seed: str | None = None,
|
||||
strategy: str | None = None,
|
||||
quality_samples: int | None = None,
|
||||
r18: int = 0,
|
||||
r18_strict: int | None = None,
|
||||
ai_type: str = "any",
|
||||
illust_type: str = "any",
|
||||
orientation: str = "any",
|
||||
layout: str | None = None,
|
||||
adaptive: int = 0,
|
||||
min_width: int = 0,
|
||||
min_height: int = 0,
|
||||
min_pixels: int = 0,
|
||||
min_bookmarks: int = 0,
|
||||
min_views: int = 0,
|
||||
min_comments: int = 0,
|
||||
included_tags: list[str] | None = Query(default=None),
|
||||
excluded_tags: list[str] | None = Query(default=None),
|
||||
user_id: int | None = None,
|
||||
illust_id: int | None = None,
|
||||
created_from: str | None = None,
|
||||
created_to: str | None = None,
|
||||
) -> Any:
|
||||
if format not in {"image", "json", "simple_json"}:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported format", status_code=400)
|
||||
if redirect not in {0, 1}:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported redirect", status_code=400)
|
||||
seed_norm = (seed or "").strip()
|
||||
if seed is not None and not seed_norm:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported seed", status_code=400)
|
||||
if seed_norm and len(seed_norm) > 128:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported seed", status_code=400)
|
||||
|
||||
ai_type_raw = (ai_type or "any").strip().lower()
|
||||
ai_type_i: int | None = None
|
||||
if ai_type_raw in {"", "any"}:
|
||||
ai_type_i = None
|
||||
elif ai_type_raw in {"0", "1"}:
|
||||
ai_type_i = int(ai_type_raw)
|
||||
else:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported ai_type", status_code=400)
|
||||
|
||||
illust_type_raw = (illust_type or "any").strip().lower()
|
||||
illust_type_i: int | None = None
|
||||
if illust_type_raw in {"", "any"}:
|
||||
illust_type_i = None
|
||||
elif illust_type_raw in {"0", "illust", "illustration"}:
|
||||
illust_type_i = 0
|
||||
elif illust_type_raw in {"1", "manga"}:
|
||||
illust_type_i = 1
|
||||
elif illust_type_raw in {"2", "ugoira"}:
|
||||
illust_type_i = 2
|
||||
else:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported illust_type", status_code=400)
|
||||
|
||||
if r18 not in {0, 1, 2}:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported r18", status_code=400)
|
||||
|
||||
if adaptive not in {0, 1}:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported adaptive", status_code=400)
|
||||
|
||||
layout_source = "orientation"
|
||||
raw_layout = orientation
|
||||
if layout is not None:
|
||||
layout_source = "layout"
|
||||
raw_layout = layout
|
||||
|
||||
layout_norm = (raw_layout or "").strip().lower()
|
||||
alias_map = {"vertical": "portrait", "horizontal": "landscape"}
|
||||
layout_norm = alias_map.get(layout_norm, layout_norm)
|
||||
orientation_map = {"any": None, "portrait": 1, "landscape": 2, "square": 3}
|
||||
if layout_norm not in orientation_map:
|
||||
raise ApiError(
|
||||
code=ErrorCode.BAD_REQUEST,
|
||||
message="Unsupported layout" if layout_source == "layout" else "Unsupported orientation",
|
||||
status_code=400,
|
||||
)
|
||||
|
||||
min_width_i = int(min_width)
|
||||
min_height_i = int(min_height)
|
||||
min_pixels_i = int(min_pixels)
|
||||
min_bookmarks_i = int(min_bookmarks)
|
||||
min_views_i = int(min_views)
|
||||
min_comments_i = int(min_comments)
|
||||
if min_width_i < 0 or min_height_i < 0 or min_pixels_i < 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported min_*", status_code=400)
|
||||
if min_bookmarks_i < 0 or min_views_i < 0 or min_comments_i < 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported min_*", status_code=400)
|
||||
|
||||
# 自适应:在用户没有显式指定的情况下,根据设备类型设置默认的方向/分辨率门槛。
|
||||
# 注意:不会覆盖用户显式传入的 orientation/layout/min_*。
|
||||
if int(adaptive) == 1:
|
||||
qp = request.query_params
|
||||
orientation_explicit = ("layout" in qp) or ("orientation" in qp)
|
||||
min_explicit = ("min_width" in qp) or ("min_height" in qp) or ("min_pixels" in qp)
|
||||
|
||||
ch_mobile = (request.headers.get("sec-ch-ua-mobile") or request.headers.get("Sec-CH-UA-Mobile") or "").strip()
|
||||
if ch_mobile == "?1":
|
||||
is_mobile = True
|
||||
elif ch_mobile == "?0":
|
||||
is_mobile = False
|
||||
else:
|
||||
ua = (request.headers.get("user-agent") or request.headers.get("User-Agent") or "").lower()
|
||||
is_mobile = any(x in ua for x in ("mobi", "android", "iphone", "ipad", "ipod"))
|
||||
|
||||
if not orientation_explicit and layout_norm == "any":
|
||||
layout_norm = "portrait" if is_mobile else "landscape"
|
||||
|
||||
if not min_explicit and min_width_i == 0 and min_height_i == 0 and min_pixels_i == 0:
|
||||
min_pixels_i = 1_000_000 if is_mobile else 2_000_000
|
||||
|
||||
def _parse_tags(values: list[str] | None) -> list[str]:
|
||||
out: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for raw in values or []:
|
||||
for part in str(raw).split("|"):
|
||||
name = part.strip()
|
||||
if not name or name in seen:
|
||||
continue
|
||||
seen.add(name)
|
||||
out.append(name)
|
||||
return out
|
||||
|
||||
included = _parse_tags(included_tags)
|
||||
excluded = _parse_tags(excluded_tags)
|
||||
if len(included) > _MAX_TAG_FILTERS or len(excluded) > _MAX_TAG_FILTERS:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Too many tag filters", status_code=400)
|
||||
if user_id is not None and int(user_id) <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported user_id", status_code=400)
|
||||
if illust_id is not None and int(illust_id) <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported illust_id", status_code=400)
|
||||
|
||||
def _normalize_iso_utc(value: str) -> str:
|
||||
raw = (value or "").strip()
|
||||
if not raw:
|
||||
raise ValueError("empty datetime")
|
||||
if raw.endswith("Z"):
|
||||
raw = raw[:-1] + "+00:00"
|
||||
dt = datetime.fromisoformat(raw)
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
dt = dt.astimezone(timezone.utc).replace(microsecond=0)
|
||||
return dt.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
created_from_norm: str | None = None
|
||||
created_to_norm: str | None = None
|
||||
try:
|
||||
if created_from is not None:
|
||||
created_from_norm = _normalize_iso_utc(created_from)
|
||||
if created_to is not None:
|
||||
created_to_norm = _normalize_iso_utc(created_to)
|
||||
except Exception:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported created_*", status_code=400)
|
||||
|
||||
if created_from_norm is not None and created_to_norm is not None:
|
||||
if created_from_norm > created_to_norm:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="created_from > created_to", status_code=400)
|
||||
|
||||
def _no_match_error() -> ApiError:
|
||||
applied_filters: dict[str, Any] = {
|
||||
"r18": r18,
|
||||
"r18_strict": int(r18_strict),
|
||||
"ai_type": ai_type_raw,
|
||||
"illust_type": illust_type_raw,
|
||||
"adaptive": int(adaptive),
|
||||
"orientation": layout_norm,
|
||||
"min_width": int(min_width_i),
|
||||
"min_height": int(min_height_i),
|
||||
"min_pixels": int(min_pixels_i),
|
||||
"min_bookmarks": int(min_bookmarks_i),
|
||||
"min_views": int(min_views_i),
|
||||
"min_comments": int(min_comments_i),
|
||||
}
|
||||
if included:
|
||||
applied_filters["included_tags"] = included
|
||||
if excluded:
|
||||
applied_filters["excluded_tags"] = excluded
|
||||
if user_id is not None:
|
||||
applied_filters["user_id"] = int(user_id)
|
||||
if illust_id is not None:
|
||||
applied_filters["illust_id"] = int(illust_id)
|
||||
if created_from_norm is not None:
|
||||
applied_filters["created_from"] = created_from_norm
|
||||
if created_to_norm is not None:
|
||||
applied_filters["created_to"] = created_to_norm
|
||||
|
||||
suggestions: list[str] = ["运行元数据补全任务以提升元数据覆盖率"]
|
||||
if r18 == 0 and int(r18_strict) == 1:
|
||||
suggestions.append("将 r18_strict=0 以允许未知 x_restrict(冷启动阶段更容易命中)")
|
||||
if layout_norm != "any":
|
||||
suggestions.append("将 orientation=any(取消方向限制)")
|
||||
if int(min_width_i) > 0 or int(min_height_i) > 0 or int(min_pixels_i) > 0:
|
||||
suggestions.append("降低 min_width/min_height/min_pixels(放宽分辨率门槛)")
|
||||
if int(min_bookmarks_i) > 0 or int(min_views_i) > 0 or int(min_comments_i) > 0:
|
||||
suggestions.append("降低 min_bookmarks/min_views/min_comments(放宽热度门槛)")
|
||||
if included:
|
||||
suggestions.append("放宽 included_tags(减少必须包含的标签)")
|
||||
if excluded:
|
||||
suggestions.append("放宽 excluded_tags(减少必须排除的标签)")
|
||||
if user_id is not None:
|
||||
suggestions.append("移除 user_id 过滤")
|
||||
if illust_id is not None:
|
||||
suggestions.append("移除 illust_id 过滤")
|
||||
if ai_type_i is not None:
|
||||
suggestions.append("将 ai_type=any(取消 AI 限制)")
|
||||
if illust_type_i is not None:
|
||||
suggestions.append("将 illust_type=any(取消作品类型限制)")
|
||||
if created_from_norm is not None or created_to_norm is not None:
|
||||
suggestions.append("扩大 created_from/created_to 时间范围")
|
||||
if int(adaptive) == 1:
|
||||
suggestions.append("若自适应导致过滤过严,可尝试 adaptive=0 或显式设置 min_*")
|
||||
|
||||
return ApiError(
|
||||
code=ErrorCode.NO_MATCH,
|
||||
message="没有匹配的图片。",
|
||||
status_code=404,
|
||||
details={"hints": {"applied_filters": applied_filters, "suggestions": suggestions}},
|
||||
)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
runtime = await load_runtime_config(engine)
|
||||
|
||||
random_defaults = runtime.random_defaults if isinstance(runtime.random_defaults, dict) else {}
|
||||
|
||||
attempts_source = "query"
|
||||
attempts_i = 3
|
||||
if attempts is not None:
|
||||
try:
|
||||
attempts_i = int(attempts)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported attempts", status_code=400) from exc
|
||||
else:
|
||||
raw = random_defaults.get("default_attempts")
|
||||
if raw is None:
|
||||
attempts_source = "fallback"
|
||||
attempts_i = 3
|
||||
else:
|
||||
attempts_source = "runtime"
|
||||
try:
|
||||
attempts_i = int(raw)
|
||||
except Exception:
|
||||
attempts_i = 3
|
||||
if attempts_i < 1 or attempts_i > 10:
|
||||
if attempts_source == "query":
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported attempts", status_code=400)
|
||||
attempts_source = "fallback"
|
||||
attempts_i = 3
|
||||
attempts = int(attempts_i)
|
||||
|
||||
r18_strict_source = "query"
|
||||
r18_strict_i = 1
|
||||
if r18_strict is not None:
|
||||
try:
|
||||
r18_strict_i = int(r18_strict)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported r18_strict", status_code=400) from exc
|
||||
else:
|
||||
raw = random_defaults.get("default_r18_strict")
|
||||
if raw is None:
|
||||
r18_strict_source = "fallback"
|
||||
r18_strict_i = 1
|
||||
elif isinstance(raw, bool):
|
||||
r18_strict_source = "runtime"
|
||||
r18_strict_i = 1 if raw else 0
|
||||
else:
|
||||
r18_strict_source = "runtime"
|
||||
try:
|
||||
r18_strict_i = int(raw)
|
||||
except Exception:
|
||||
r18_strict_i = 1
|
||||
if r18_strict_i not in {0, 1}:
|
||||
if r18_strict_source == "query":
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported r18_strict", status_code=400)
|
||||
r18_strict_source = "fallback"
|
||||
r18_strict_i = 1
|
||||
r18_strict = int(r18_strict_i)
|
||||
|
||||
fail_cooldown_source = "runtime"
|
||||
fail_cooldown_ms = random_defaults.get("fail_cooldown_ms")
|
||||
try:
|
||||
fail_cooldown_ms_i = int(fail_cooldown_ms) if fail_cooldown_ms is not None else None
|
||||
except Exception:
|
||||
fail_cooldown_ms_i = None
|
||||
|
||||
if fail_cooldown_ms_i is None:
|
||||
fail_cooldown_source = "fallback"
|
||||
cooldown_s_raw = (os.environ.get("RANDOM_FAIL_COOLDOWN_SECONDS") or "600").strip()
|
||||
try:
|
||||
cooldown_s = int(cooldown_s_raw)
|
||||
except Exception:
|
||||
cooldown_s = 600
|
||||
cooldown_s = max(0, min(int(cooldown_s), 24 * 60 * 60))
|
||||
fail_cooldown_ms_i = int(cooldown_s) * 1000
|
||||
fail_cooldown_ms_i = max(0, min(int(fail_cooldown_ms_i), 24 * 60 * 60 * 1000))
|
||||
|
||||
request_now = datetime.now(timezone.utc)
|
||||
fail_cooldown_before = (
|
||||
iso_utc_ms(request_now - timedelta(milliseconds=int(fail_cooldown_ms_i)))
|
||||
if int(fail_cooldown_ms_i) > 0
|
||||
else None
|
||||
)
|
||||
|
||||
pick_kwargs: dict[str, Any] = {
|
||||
"r18": r18,
|
||||
"r18_strict": bool(r18_strict),
|
||||
"ai_type": ai_type_i,
|
||||
"illust_type": illust_type_i,
|
||||
"orientation": orientation_map[layout_norm],
|
||||
"min_width": int(min_width_i),
|
||||
"min_height": int(min_height_i),
|
||||
"min_pixels": int(min_pixels_i),
|
||||
"min_bookmarks": int(min_bookmarks_i),
|
||||
"min_views": int(min_views_i),
|
||||
"min_comments": int(min_comments_i),
|
||||
"included_tags": included,
|
||||
"excluded_tags": excluded,
|
||||
"user_id": user_id,
|
||||
"illust_id": illust_id,
|
||||
"created_from": created_from_norm,
|
||||
"created_to": created_to_norm,
|
||||
"fail_cooldown_before": fail_cooldown_before,
|
||||
}
|
||||
|
||||
rng = random.Random(seed_norm) if seed_norm else random
|
||||
|
||||
strategy_raw = (strategy or "").strip().lower()
|
||||
strategy_source = "query"
|
||||
if not strategy_raw:
|
||||
strategy_source = "runtime"
|
||||
strategy_raw = str(random_defaults.get("strategy") or "").strip().lower()
|
||||
if not strategy_raw:
|
||||
strategy_source = "fallback"
|
||||
strategy_raw = "quality"
|
||||
if strategy_raw not in {"quality", "random"}:
|
||||
if strategy_source == "query":
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported strategy", status_code=400)
|
||||
strategy_source = "fallback"
|
||||
strategy_raw = "quality"
|
||||
|
||||
strategy_norm = strategy_raw
|
||||
|
||||
quality_samples_i: int
|
||||
quality_samples_source = "query"
|
||||
if quality_samples is not None:
|
||||
try:
|
||||
quality_samples_i = int(quality_samples)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported quality_samples", status_code=400) from exc
|
||||
else:
|
||||
raw = random_defaults.get("quality_samples")
|
||||
if raw is None:
|
||||
quality_samples_source = "fallback"
|
||||
quality_samples_i = 5
|
||||
else:
|
||||
quality_samples_source = "runtime"
|
||||
try:
|
||||
quality_samples_i = int(raw)
|
||||
except Exception:
|
||||
quality_samples_i = 5
|
||||
if quality_samples_i < 1 or quality_samples_i > 1000:
|
||||
if quality_samples_source == "query":
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported quality_samples", status_code=400)
|
||||
quality_samples_source = "fallback"
|
||||
quality_samples_i = 5
|
||||
|
||||
recommendation_raw = random_defaults.get("recommendation")
|
||||
recommendation_source = "fallback"
|
||||
recommendation_obj: dict[str, Any] = {}
|
||||
if isinstance(recommendation_raw, dict):
|
||||
recommendation_source = "runtime"
|
||||
recommendation_obj = dict(recommendation_raw)
|
||||
|
||||
pick_mode_raw = str(recommendation_obj.get("pick_mode") or _DEFAULT_RECOMMENDATION["pick_mode"]).strip().lower()
|
||||
if pick_mode_raw not in {"best", "weighted"}:
|
||||
pick_mode_raw = str(_DEFAULT_RECOMMENDATION["pick_mode"])
|
||||
|
||||
temperature_raw = _as_float(recommendation_obj.get("temperature"), default=float(_DEFAULT_RECOMMENDATION["temperature"]))
|
||||
temperature = float(max(0.05, min(float(temperature_raw), 100.0)))
|
||||
|
||||
score_weights_raw = recommendation_obj.get("score_weights")
|
||||
score_weights_obj = score_weights_raw if isinstance(score_weights_raw, dict) else {}
|
||||
score_weights: dict[str, float] = {}
|
||||
for key, default_value in _DEFAULT_SCORE_WEIGHTS.items():
|
||||
v = _as_float(score_weights_obj.get(key), default=float(default_value))
|
||||
score_weights[key] = float(max(-100.0, min(float(v), 100.0)))
|
||||
|
||||
multipliers_default = _DEFAULT_RECOMMENDATION["multipliers"]
|
||||
multipliers_raw = recommendation_obj.get("multipliers")
|
||||
multipliers_obj = multipliers_raw if isinstance(multipliers_raw, dict) else {}
|
||||
multipliers: dict[str, float] = {}
|
||||
for key, default_value in multipliers_default.items():
|
||||
v = _as_float(multipliers_obj.get(key), default=float(default_value))
|
||||
multipliers[key] = float(max(0.0, min(float(v), 100.0)))
|
||||
|
||||
debug_base = {
|
||||
"attempts": int(attempts_i),
|
||||
"attempts_source": attempts_source,
|
||||
"r18_strict": int(r18_strict_i),
|
||||
"r18_strict_source": r18_strict_source,
|
||||
"fail_cooldown_ms": int(fail_cooldown_ms_i),
|
||||
"fail_cooldown_source": fail_cooldown_source,
|
||||
"strategy": strategy_norm,
|
||||
"strategy_source": strategy_source,
|
||||
"quality_samples": int(quality_samples_i),
|
||||
"quality_samples_source": quality_samples_source,
|
||||
"recommendation_source": recommendation_source,
|
||||
}
|
||||
|
||||
async def _pick_with_strategy(
|
||||
*,
|
||||
session: Any,
|
||||
exclude_image_ids: list[int] | None = None,
|
||||
) -> tuple[Any, dict[str, Any]] | tuple[None, dict[str, Any]]:
|
||||
if strategy_norm == "random":
|
||||
image = await pick_random_image(session, r=rng.random(), exclude_image_ids=exclude_image_ids, **pick_kwargs)
|
||||
if image is None:
|
||||
return None, {**debug_base, "attempts_used": 1, "picked_by": "random_key"}
|
||||
return image, {**debug_base, "attempts_used": 1, "picked_by": "random_key"}
|
||||
|
||||
def _multiplier_for_image(image: Any) -> float:
|
||||
m = 1.0
|
||||
|
||||
ai = getattr(image, "ai_type", None)
|
||||
if ai == 1:
|
||||
m *= float(multipliers.get("ai", 1.0))
|
||||
elif ai == 0:
|
||||
m *= float(multipliers.get("non_ai", 1.0))
|
||||
else:
|
||||
m *= float(multipliers.get("unknown_ai", 1.0))
|
||||
|
||||
it = getattr(image, "illust_type", None)
|
||||
if it == 0:
|
||||
m *= float(multipliers.get("illust", 1.0))
|
||||
elif it == 1:
|
||||
m *= float(multipliers.get("manga", 1.0))
|
||||
elif it == 2:
|
||||
m *= float(multipliers.get("ugoira", 1.0))
|
||||
else:
|
||||
m *= float(multipliers.get("unknown_illust_type", 1.0))
|
||||
|
||||
if not math.isfinite(float(m)) or float(m) <= 0.0:
|
||||
return 0.0
|
||||
return float(m)
|
||||
|
||||
exclude_set: set[int] = set(int(x) for x in exclude_image_ids or [])
|
||||
candidates: list[tuple[Any, float, float, float]] = []
|
||||
|
||||
# 批量抽样:一次性取 N 个候选(必要时 wrap-around 再取一次),避免 N 次 DB 循环查询。
|
||||
# 若用户把某些类别倍率设为 0(例如 manga=0),直接在 SQL 抽样阶段剔除,减少无效候选。
|
||||
ai_allowed: set[int | None] = set()
|
||||
if float(multipliers.get("ai", 1.0)) > 0.0:
|
||||
ai_allowed.add(1)
|
||||
if float(multipliers.get("non_ai", 1.0)) > 0.0:
|
||||
ai_allowed.add(0)
|
||||
if float(multipliers.get("unknown_ai", 1.0)) > 0.0:
|
||||
ai_allowed.add(None)
|
||||
|
||||
illust_allowed: set[int | None] = set()
|
||||
if float(multipliers.get("illust", 1.0)) > 0.0:
|
||||
illust_allowed.add(0)
|
||||
if float(multipliers.get("manga", 1.0)) > 0.0:
|
||||
illust_allowed.add(1)
|
||||
if float(multipliers.get("ugoira", 1.0)) > 0.0:
|
||||
illust_allowed.add(2)
|
||||
if float(multipliers.get("unknown_illust_type", 1.0)) > 0.0:
|
||||
illust_allowed.add(None)
|
||||
|
||||
if not ai_allowed or not illust_allowed:
|
||||
return None, {
|
||||
**debug_base,
|
||||
"attempts_used": 1,
|
||||
"picked_by": "quality_weighted" if pick_mode_raw == "weighted" else "quality_best",
|
||||
"candidates_drawn": 0,
|
||||
"candidates_accepted": 0,
|
||||
"quality_pick_mode": pick_mode_raw,
|
||||
"quality_temperature": float(temperature),
|
||||
}
|
||||
|
||||
images = await pick_random_images(
|
||||
session,
|
||||
r=rng.random(),
|
||||
limit=int(quality_samples_i),
|
||||
exclude_image_ids=list(exclude_set),
|
||||
ai_type_allowed=ai_allowed,
|
||||
illust_type_allowed=illust_allowed,
|
||||
**pick_kwargs,
|
||||
)
|
||||
|
||||
drawn = int(len(images))
|
||||
accepted = 0
|
||||
for image in images:
|
||||
score = _quality_score(image, weights=score_weights)
|
||||
multiplier = _multiplier_for_image(image)
|
||||
if multiplier <= 0.0:
|
||||
continue
|
||||
|
||||
logit = float(score) / float(temperature) + math.log(float(multiplier))
|
||||
candidates.append((image, float(score), float(multiplier), float(logit)))
|
||||
accepted += 1
|
||||
|
||||
if not candidates:
|
||||
return None, {
|
||||
**debug_base,
|
||||
"attempts_used": 1,
|
||||
"picked_by": "quality_weighted" if pick_mode_raw == "weighted" else "quality_best",
|
||||
"candidates_drawn": int(drawn),
|
||||
"candidates_accepted": int(accepted),
|
||||
"quality_pick_mode": pick_mode_raw,
|
||||
"quality_temperature": float(temperature),
|
||||
}
|
||||
|
||||
if pick_mode_raw == "best":
|
||||
picked = max(candidates, key=lambda x: x[3])
|
||||
picked_by = "quality_best"
|
||||
else:
|
||||
max_logit = max(x[3] for x in candidates)
|
||||
weights = [math.exp(float(x[3]) - float(max_logit)) for x in candidates]
|
||||
total = float(sum(weights))
|
||||
if not math.isfinite(total) or total <= 0.0:
|
||||
picked = max(candidates, key=lambda x: x[3])
|
||||
picked_by = "quality_best"
|
||||
else:
|
||||
r = float(rng.random()) * total
|
||||
idx = 0
|
||||
for i, w in enumerate(weights):
|
||||
r -= float(w)
|
||||
if r <= 0:
|
||||
idx = i
|
||||
break
|
||||
picked = candidates[int(max(0, min(idx, len(candidates) - 1)))]
|
||||
picked_by = "quality_weighted"
|
||||
|
||||
best_image, best_score, best_multiplier, _best_logit = picked
|
||||
|
||||
return (
|
||||
best_image,
|
||||
{
|
||||
**debug_base,
|
||||
"attempts_used": 1,
|
||||
"picked_by": picked_by,
|
||||
"candidates_drawn": int(drawn),
|
||||
"candidates_accepted": int(accepted),
|
||||
"quality_pick_mode": pick_mode_raw,
|
||||
"quality_temperature": float(temperature),
|
||||
"quality_score": float(best_score),
|
||||
"quality_multiplier": float(best_multiplier),
|
||||
},
|
||||
)
|
||||
|
||||
def _needs_opportunistic_hydrate(image: Any) -> bool:
|
||||
return (
|
||||
getattr(image, "width", None) is None
|
||||
or getattr(image, "height", None) is None
|
||||
or getattr(image, "x_restrict", None) is None
|
||||
or getattr(image, "ai_type", None) is None
|
||||
or getattr(image, "illust_type", None) is None
|
||||
or getattr(image, "user_id", None) is None
|
||||
or getattr(image, "bookmark_count", None) is None
|
||||
or getattr(image, "view_count", None) is None
|
||||
or getattr(image, "comment_count", None) is None
|
||||
)
|
||||
|
||||
if format in {"json", "simple_json"} or (format == "image" and redirect == 1):
|
||||
tags: list[str] = []
|
||||
debug: dict[str, Any] = {}
|
||||
async with Session() as session:
|
||||
image, debug = await _pick_with_strategy(session=session)
|
||||
if image is None:
|
||||
raise _no_match_error()
|
||||
if format == "json":
|
||||
tags = await get_tag_names_for_image(session, image_id=image.id)
|
||||
|
||||
if _needs_opportunistic_hydrate(image):
|
||||
try:
|
||||
await enqueue_opportunistic_hydrate_metadata(engine, illust_id=int(image.illust_id), reason="random")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if format == "image" and redirect == 1:
|
||||
return RedirectResponse(
|
||||
url=f"/i/{image.id}.{image.ext}",
|
||||
status_code=302,
|
||||
headers={"Cache-Control": "no-store"},
|
||||
)
|
||||
|
||||
origin_url = None if runtime.hide_origin_url_in_public_json else image.original_url
|
||||
|
||||
imgproxy_url = None
|
||||
try:
|
||||
cfg = load_imgproxy_config_from_settings(request.app.state.settings)
|
||||
except Exception:
|
||||
cfg = None
|
||||
if cfg is not None:
|
||||
try:
|
||||
if runtime.hide_origin_url_in_public_json:
|
||||
base = str(getattr(request, "base_url", "") or "").rstrip("/")
|
||||
source_url = f"{base}/i/{image.id}.{image.ext}"
|
||||
else:
|
||||
source_url = str(image.original_url)
|
||||
imgproxy_url = build_signed_processing_url(cfg, source_url=source_url, extension=str(image.ext))
|
||||
except Exception:
|
||||
imgproxy_url = None
|
||||
|
||||
if format == "simple_json":
|
||||
return {
|
||||
"ok": True,
|
||||
"code": "OK",
|
||||
"request_id": getattr(getattr(request, "state", None), "request_id", None) or "req_unknown",
|
||||
"data": {
|
||||
"image": {
|
||||
"id": str(image.id),
|
||||
"illust_id": str(image.illust_id),
|
||||
"page_index": image.page_index,
|
||||
"ext": image.ext,
|
||||
"width": image.width,
|
||||
"height": image.height,
|
||||
"x_restrict": image.x_restrict,
|
||||
"ai_type": image.ai_type,
|
||||
"illust_type": getattr(image, "illust_type", None),
|
||||
"bookmark_count": getattr(image, "bookmark_count", None),
|
||||
"view_count": getattr(image, "view_count", None),
|
||||
"comment_count": getattr(image, "comment_count", None),
|
||||
},
|
||||
"urls": {
|
||||
"proxy": f"/i/{image.id}.{image.ext}",
|
||||
"origin": origin_url,
|
||||
"imgproxy": imgproxy_url,
|
||||
},
|
||||
"debug": {
|
||||
**debug,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"code": "OK",
|
||||
"request_id": getattr(getattr(request, "state", None), "request_id", None) or "req_unknown",
|
||||
"data": {
|
||||
"image": {
|
||||
"id": str(image.id),
|
||||
"illust_id": str(image.illust_id),
|
||||
"page_index": image.page_index,
|
||||
"ext": image.ext,
|
||||
"width": image.width,
|
||||
"height": image.height,
|
||||
"x_restrict": image.x_restrict,
|
||||
"ai_type": image.ai_type,
|
||||
"illust_type": getattr(image, "illust_type", None),
|
||||
"bookmark_count": getattr(image, "bookmark_count", None),
|
||||
"view_count": getattr(image, "view_count", None),
|
||||
"comment_count": getattr(image, "comment_count", None),
|
||||
"user": {
|
||||
"id": str(image.user_id) if image.user_id is not None else None,
|
||||
"name": image.user_name,
|
||||
},
|
||||
"title": image.title,
|
||||
"created_at_pixiv": image.created_at_pixiv,
|
||||
},
|
||||
"tags": tags,
|
||||
"urls": {
|
||||
"proxy": f"/i/{image.id}.{image.ext}",
|
||||
"origin": origin_url,
|
||||
"imgproxy": imgproxy_url,
|
||||
"legacy_single": f"/{image.illust_id}.{image.ext}",
|
||||
"legacy_multi": f"/{image.illust_id}-{image.page_index + 1}.{image.ext}",
|
||||
},
|
||||
"debug": {
|
||||
**debug,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
tried_ids: set[int] = set()
|
||||
last_error: ApiError | None = None
|
||||
attempts_i = int(attempts)
|
||||
runtime_stream = runtime
|
||||
|
||||
for _ in range(attempts_i):
|
||||
async with Session() as session:
|
||||
image, _debug = await _pick_with_strategy(session=session, exclude_image_ids=list(tried_ids))
|
||||
if image is None:
|
||||
break
|
||||
image_id = int(image.id)
|
||||
origin_url = str(image.original_url)
|
||||
illust_id_for_hydrate = int(image.illust_id)
|
||||
needs_hydrate = _needs_opportunistic_hydrate(image)
|
||||
|
||||
transport = getattr(request.app.state, "httpx_transport", None)
|
||||
proxy_uri = None
|
||||
picked = await select_proxy_uri_for_url(
|
||||
engine,
|
||||
request.app.state.settings,
|
||||
runtime_stream,
|
||||
url=origin_url,
|
||||
)
|
||||
if picked is not None:
|
||||
proxy_uri = picked.uri
|
||||
try:
|
||||
resp = await stream_url(
|
||||
origin_url,
|
||||
transport=transport,
|
||||
proxy=proxy_uri,
|
||||
cache_control="no-store",
|
||||
range_header=request.headers.get("Range"),
|
||||
)
|
||||
await mark_image_ok(engine, image_id=image_id, now=iso_utc_ms())
|
||||
if needs_hydrate:
|
||||
try:
|
||||
await enqueue_opportunistic_hydrate_metadata(
|
||||
engine,
|
||||
illust_id=illust_id_for_hydrate,
|
||||
reason="random",
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
return resp
|
||||
except ApiError as exc:
|
||||
if exc.code in {
|
||||
ErrorCode.UPSTREAM_STREAM_ERROR,
|
||||
ErrorCode.UPSTREAM_403,
|
||||
ErrorCode.UPSTREAM_404,
|
||||
ErrorCode.UPSTREAM_RATE_LIMIT,
|
||||
}:
|
||||
await mark_image_failure(
|
||||
engine,
|
||||
image_id=image_id,
|
||||
now=iso_utc_ms(),
|
||||
error_code=exc.code.value,
|
||||
error_message=exc.message,
|
||||
)
|
||||
tried_ids.add(image_id)
|
||||
last_error = exc
|
||||
continue
|
||||
raise
|
||||
|
||||
if last_error is None:
|
||||
raise _no_match_error()
|
||||
|
||||
raise ApiError(
|
||||
code=ErrorCode.UPSTREAM_STREAM_ERROR,
|
||||
message="多次尝试后上游请求仍失败。",
|
||||
status_code=502,
|
||||
details={
|
||||
"attempts_used": len(tried_ids),
|
||||
"last_upstream_code": last_error.code.value,
|
||||
},
|
||||
)
|
||||
@@ -0,0 +1,54 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.request_id import get_or_create_request_id
|
||||
from app.db.session import create_sessionmaker
|
||||
from app.db.tags_list import list_tags as db_list_tags
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/tags")
|
||||
async def list_tags(
|
||||
request: Request,
|
||||
q: str | None = None,
|
||||
limit: int = 50,
|
||||
cursor: str | None = None,
|
||||
) -> Any:
|
||||
if limit < 1 or limit > 100:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
|
||||
|
||||
q_norm = (q or "").strip()
|
||||
if q_norm and len(q_norm) > 200:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported q", status_code=400)
|
||||
|
||||
engine = request.app.state.engine
|
||||
Session = create_sessionmaker(engine)
|
||||
async with Session() as session:
|
||||
items, next_cursor = await db_list_tags(session, limit=limit, cursor=cursor, q=q_norm or None)
|
||||
|
||||
rid = get_or_create_request_id(request)
|
||||
resp = JSONResponse(
|
||||
status_code=200,
|
||||
content={
|
||||
"ok": True,
|
||||
"items": [
|
||||
{
|
||||
"id": str(item.id),
|
||||
"name": item.name,
|
||||
"translated_name": item.translated_name,
|
||||
"count_images": item.count_images,
|
||||
}
|
||||
for item in items
|
||||
],
|
||||
"next_cursor": next_cursor or "",
|
||||
"request_id": rid,
|
||||
},
|
||||
)
|
||||
return resp
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from app.core.request_id import get_or_create_request_id, set_request_id_header, set_request_id_on_state
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _get_env(key: str, default: str) -> str:
|
||||
value = os.environ.get(key, default)
|
||||
return str(value).strip()
|
||||
|
||||
|
||||
@router.get("/version")
|
||||
async def version(request: Request) -> Any:
|
||||
rid = get_or_create_request_id(request)
|
||||
set_request_id_on_state(request, rid)
|
||||
|
||||
body = {
|
||||
"ok": True,
|
||||
"version": _get_env("APP_VERSION", "dev"),
|
||||
"build_time": _get_env("APP_BUILD_TIME", ""),
|
||||
"git_commit": _get_env("APP_COMMIT", ""),
|
||||
"request_id": rid,
|
||||
}
|
||||
|
||||
resp = JSONResponse(status_code=200, content=body)
|
||||
set_request_id_header(resp, rid)
|
||||
return resp
|
||||
|
||||
@@ -0,0 +1,182 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from collections.abc import Mapping
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine
|
||||
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.db.models.api_keys import ApiKey
|
||||
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
|
||||
|
||||
|
||||
def _coerce_int(value: int, *, default: int = 0) -> int:
|
||||
try:
|
||||
return int(value)
|
||||
except Exception:
|
||||
return default
|
||||
|
||||
|
||||
def _normalize_api_key(value: str) -> str:
|
||||
v = (value or "").strip()
|
||||
return v
|
||||
|
||||
|
||||
def hmac_sha256_hex(*, secret_key: str, message: str) -> str:
|
||||
secret_key = (secret_key or "").strip()
|
||||
if not secret_key:
|
||||
raise ValueError("SECRET_KEY is required")
|
||||
mac = hmac.new(secret_key.encode("utf-8"), message.encode("utf-8"), digestmod=hashlib.sha256)
|
||||
return mac.hexdigest()
|
||||
|
||||
|
||||
def api_key_hint(api_key: str) -> str:
|
||||
api_key = _normalize_api_key(api_key)
|
||||
if not api_key:
|
||||
return ""
|
||||
digest = hashlib.sha256(api_key.encode("utf-8")).hexdigest()
|
||||
return digest[:8]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ApiKeyAuthConfig:
|
||||
required: bool
|
||||
rpm: int
|
||||
burst: int
|
||||
secret_key: str
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _CacheEntry:
|
||||
api_key_id: int
|
||||
enabled: bool
|
||||
expires_at_m: float
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class ApiKeyAuthenticator:
|
||||
engine: AsyncEngine
|
||||
cfg: ApiKeyAuthConfig
|
||||
cache_ttl_s: float = 5.0
|
||||
_cache: dict[str, _CacheEntry] = field(default_factory=dict)
|
||||
|
||||
async def _lookup(self, key_hash: str, *, now_m: float) -> _CacheEntry | None:
|
||||
Session = create_sessionmaker(self.engine)
|
||||
|
||||
async def _op() -> _CacheEntry | None:
|
||||
async with Session() as session:
|
||||
row = (
|
||||
(
|
||||
await session.execute(
|
||||
sa.select(ApiKey.id, ApiKey.enabled).where(ApiKey.key_hash == key_hash).limit(1)
|
||||
)
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if row is None:
|
||||
return None
|
||||
return _CacheEntry(
|
||||
api_key_id=int(row[0]),
|
||||
enabled=bool(int(row[1] or 0)),
|
||||
expires_at_m=float(now_m) + float(self.cache_ttl_s),
|
||||
)
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
async def authenticate(self, api_key: str) -> int | None:
|
||||
api_key = _normalize_api_key(api_key)
|
||||
if not api_key:
|
||||
return None
|
||||
|
||||
key_hash = hmac_sha256_hex(secret_key=self.cfg.secret_key, message=api_key)
|
||||
now_m = time.monotonic()
|
||||
|
||||
cached = self._cache.get(key_hash)
|
||||
if cached is not None and float(cached.expires_at_m) > float(now_m):
|
||||
return int(cached.api_key_id) if bool(cached.enabled) else None
|
||||
|
||||
entry = await self._lookup(key_hash, now_m=now_m)
|
||||
if entry is None:
|
||||
self._cache[key_hash] = _CacheEntry(api_key_id=0, enabled=False, expires_at_m=float(now_m) + 2.0)
|
||||
return None
|
||||
|
||||
self._cache[key_hash] = entry
|
||||
|
||||
if len(self._cache) > 10_000:
|
||||
self._cache = {k: v for k, v in self._cache.items() if float(v.expires_at_m) > float(now_m)}
|
||||
|
||||
return int(entry.api_key_id) if bool(entry.enabled) else None
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _Bucket:
|
||||
tokens: float
|
||||
updated_at_m: float
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class ApiKeyRateLimiter:
|
||||
rpm: int
|
||||
burst: int
|
||||
_buckets: dict[int, _Bucket] = field(default_factory=dict)
|
||||
|
||||
def _params(self) -> tuple[float, float]:
|
||||
rpm = max(0, _coerce_int(self.rpm))
|
||||
if rpm <= 0:
|
||||
return 0.0, 0.0
|
||||
capacity = max(1.0, float(_coerce_int(self.burst)) or float(rpm))
|
||||
refill_per_s = float(rpm) / 60.0
|
||||
return capacity, refill_per_s
|
||||
|
||||
def allow(self, api_key_id: int) -> bool:
|
||||
api_key_id_i = _coerce_int(api_key_id)
|
||||
if api_key_id_i <= 0:
|
||||
return False
|
||||
|
||||
capacity, refill_per_s = self._params()
|
||||
if capacity <= 0.0 or refill_per_s <= 0.0:
|
||||
return True
|
||||
|
||||
now_m = time.monotonic()
|
||||
b = self._buckets.get(api_key_id_i)
|
||||
if b is None:
|
||||
self._buckets[api_key_id_i] = _Bucket(tokens=float(capacity) - 1.0, updated_at_m=now_m)
|
||||
return True
|
||||
|
||||
elapsed = max(0.0, float(now_m) - float(b.updated_at_m))
|
||||
b.tokens = min(float(capacity), float(b.tokens) + elapsed * float(refill_per_s))
|
||||
b.updated_at_m = now_m
|
||||
if float(b.tokens) >= 1.0:
|
||||
b.tokens -= 1.0
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def extract_api_key(headers: Mapping[str, str] | None) -> str | None:
|
||||
if not headers:
|
||||
return None
|
||||
return (headers.get("X-API-Key") or headers.get("x-api-key") or "").strip() or None
|
||||
|
||||
|
||||
async def require_public_api_key(
|
||||
authenticator: ApiKeyAuthenticator,
|
||||
limiter: ApiKeyRateLimiter,
|
||||
*,
|
||||
headers: Mapping[str, str] | None,
|
||||
) -> int:
|
||||
api_key = extract_api_key(headers)
|
||||
if not api_key:
|
||||
raise ApiError(code=ErrorCode.UNAUTHORIZED, message="Missing API key", status_code=401)
|
||||
|
||||
api_key_id = await authenticator.authenticate(api_key)
|
||||
if api_key_id is None:
|
||||
raise ApiError(code=ErrorCode.UNAUTHORIZED, message="Invalid API key", status_code=401)
|
||||
|
||||
if not limiter.allow(api_key_id):
|
||||
raise ApiError(code=ErrorCode.RATE_LIMITED, message="Rate limited", status_code=429)
|
||||
|
||||
return int(api_key_id)
|
||||
@@ -0,0 +1,204 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.db.models.pixiv_tokens import PixivToken
|
||||
from app.db.models.proxy_endpoints import ProxyEndpoint
|
||||
from app.db.models.proxy_pool_endpoints import ProxyPoolEndpoint
|
||||
from app.db.models.token_proxy_bindings import TokenProxyBinding
|
||||
|
||||
|
||||
def _fnv1a64(text: str) -> int:
|
||||
h = 14695981039346656037
|
||||
prime = 1099511628211
|
||||
for b in text.encode("utf-8"):
|
||||
h ^= b
|
||||
h = (h * prime) & 0xFFFFFFFFFFFFFFFF
|
||||
return h
|
||||
|
||||
|
||||
def _rendezvous_proxy_order(*, token_id: int, proxy_ids: list[int], salt: str) -> list[int]:
|
||||
scored = [(_fnv1a64(f"{token_id}|{pid}|{salt}"), pid) for pid in proxy_ids]
|
||||
scored.sort(key=lambda x: (-x[0], x[1]))
|
||||
return [pid for _, pid in scored]
|
||||
|
||||
|
||||
def _compute_primary_assignments(
|
||||
*,
|
||||
token_ids: list[int],
|
||||
proxy_ids: list[int],
|
||||
capacity_by_proxy_id: dict[int, int],
|
||||
salt: str,
|
||||
) -> dict[int, int]:
|
||||
remaining = {pid: int(capacity_by_proxy_id.get(int(pid), 0)) for pid in proxy_ids}
|
||||
out: dict[int, int] = {}
|
||||
for token_id in token_ids:
|
||||
for pid in _rendezvous_proxy_order(token_id=token_id, proxy_ids=proxy_ids, salt=salt):
|
||||
if remaining.get(pid, 0) > 0:
|
||||
out[token_id] = pid
|
||||
remaining[pid] -= 1
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
def _compute_primary_assignments_soft(
|
||||
*,
|
||||
token_ids: list[int],
|
||||
proxy_ids: list[int],
|
||||
capacity_by_proxy_id: dict[int, int],
|
||||
salt: str,
|
||||
) -> tuple[dict[int, int], int]:
|
||||
out = _compute_primary_assignments(
|
||||
token_ids=token_ids,
|
||||
proxy_ids=proxy_ids,
|
||||
capacity_by_proxy_id=capacity_by_proxy_id,
|
||||
salt=salt,
|
||||
)
|
||||
|
||||
over_capacity = 0
|
||||
for token_id in token_ids:
|
||||
if token_id in out:
|
||||
continue
|
||||
order = _rendezvous_proxy_order(token_id=token_id, proxy_ids=proxy_ids, salt=salt)
|
||||
if not order:
|
||||
continue
|
||||
out[token_id] = int(order[0])
|
||||
over_capacity += 1
|
||||
|
||||
return out, int(over_capacity)
|
||||
|
||||
|
||||
async def recompute_token_proxy_bindings(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
pool_id: int,
|
||||
now: str,
|
||||
max_tokens_per_proxy: int,
|
||||
strict: bool,
|
||||
) -> dict[str, Any]:
|
||||
if int(pool_id) <= 0:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid pool_id", status_code=400)
|
||||
if int(max_tokens_per_proxy) <= 0 or int(max_tokens_per_proxy) > 1000:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid max_tokens_per_proxy", status_code=400)
|
||||
|
||||
proxy_rows = (
|
||||
(
|
||||
await session.execute(
|
||||
sa.select(ProxyPoolEndpoint.endpoint_id, ProxyPoolEndpoint.weight)
|
||||
.join(ProxyEndpoint, ProxyEndpoint.id == ProxyPoolEndpoint.endpoint_id)
|
||||
.where(ProxyPoolEndpoint.pool_id == int(pool_id))
|
||||
.where(ProxyPoolEndpoint.enabled == 1)
|
||||
.where(ProxyEndpoint.enabled == 1)
|
||||
.order_by(ProxyPoolEndpoint.endpoint_id.asc())
|
||||
)
|
||||
)
|
||||
.all()
|
||||
)
|
||||
|
||||
proxies: list[tuple[int, int]] = []
|
||||
for endpoint_id, weight in proxy_rows:
|
||||
try:
|
||||
eid = int(endpoint_id)
|
||||
except Exception:
|
||||
continue
|
||||
try:
|
||||
w = int(weight or 0)
|
||||
except Exception:
|
||||
w = 0
|
||||
if eid <= 0:
|
||||
continue
|
||||
proxies.append((eid, w))
|
||||
|
||||
capacity_by_proxy_id = {pid: int(max_tokens_per_proxy) * max(0, int(w)) for pid, w in proxies}
|
||||
proxy_ids_norm = [pid for pid, _w in proxies if capacity_by_proxy_id.get(pid, 0) > 0]
|
||||
weight_sum = sum(max(0, int(w)) for pid, w in proxies if int(pid) in set(proxy_ids_norm))
|
||||
|
||||
if not proxy_ids_norm:
|
||||
raise ApiError(code=ErrorCode.BAD_REQUEST, message="No enabled proxies in pool", status_code=400)
|
||||
|
||||
token_ids = (
|
||||
(
|
||||
await session.execute(
|
||||
sa.select(PixivToken.id)
|
||||
.where(PixivToken.enabled == 1)
|
||||
.where(PixivToken.weight > 0)
|
||||
.order_by(PixivToken.id.asc())
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
if not token_ids:
|
||||
return {"recomputed": 0}
|
||||
|
||||
capacity = sum(int(capacity_by_proxy_id.get(int(pid), 0)) for pid in proxy_ids_norm)
|
||||
if strict and len(token_ids) > capacity:
|
||||
raise ApiError(
|
||||
code=ErrorCode.BAD_REQUEST,
|
||||
message="代理容量不足(请增加节点或调高单代理最多绑定令牌数)",
|
||||
status_code=400,
|
||||
details={
|
||||
"token_count": len(token_ids),
|
||||
"proxy_count": len(proxy_ids_norm),
|
||||
"max_tokens_per_proxy": int(max_tokens_per_proxy),
|
||||
"weight_sum": int(weight_sum),
|
||||
"capacity": int(capacity),
|
||||
},
|
||||
)
|
||||
|
||||
salt = f"pool:{pool_id}"
|
||||
over_capacity_assigned = 0
|
||||
if strict:
|
||||
assignments = _compute_primary_assignments(
|
||||
token_ids=token_ids,
|
||||
proxy_ids=proxy_ids_norm,
|
||||
capacity_by_proxy_id=capacity_by_proxy_id,
|
||||
salt=salt,
|
||||
)
|
||||
else:
|
||||
assignments, over_capacity_assigned = _compute_primary_assignments_soft(
|
||||
token_ids=token_ids,
|
||||
proxy_ids=proxy_ids_norm,
|
||||
capacity_by_proxy_id=capacity_by_proxy_id,
|
||||
salt=salt,
|
||||
)
|
||||
|
||||
for token_id in token_ids:
|
||||
primary_proxy_id = assignments.get(int(token_id))
|
||||
if primary_proxy_id is None:
|
||||
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Binding recompute failed", status_code=500)
|
||||
|
||||
stmt = sqlite_insert(TokenProxyBinding).values(
|
||||
token_id=int(token_id),
|
||||
pool_id=int(pool_id),
|
||||
primary_proxy_id=int(primary_proxy_id),
|
||||
override_proxy_id=None,
|
||||
override_expires_at=None,
|
||||
updated_at=now,
|
||||
)
|
||||
stmt = stmt.on_conflict_do_update(
|
||||
index_elements=[TokenProxyBinding.token_id, TokenProxyBinding.pool_id],
|
||||
set_={"primary_proxy_id": int(primary_proxy_id), "updated_at": now},
|
||||
)
|
||||
await session.execute(stmt)
|
||||
|
||||
resp: dict[str, Any] = {"recomputed": len(token_ids)}
|
||||
if not strict:
|
||||
resp.update(
|
||||
{
|
||||
"strict": False,
|
||||
"over_capacity_assigned": int(over_capacity_assigned),
|
||||
"capacity": int(capacity),
|
||||
"token_count": len(token_ids),
|
||||
"proxy_count": len(proxy_ids_norm),
|
||||
"max_tokens_per_proxy": int(max_tokens_per_proxy),
|
||||
"weight_sum": int(weight_sum),
|
||||
}
|
||||
)
|
||||
return resp
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Mapping
|
||||
|
||||
from cryptography.fernet import Fernet
|
||||
|
||||
from app.core.crypto import FieldEncryptor
|
||||
from app.core.logging import get_logger
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Settings:
|
||||
app_env: str
|
||||
database_url: str
|
||||
secret_key: str
|
||||
field_encryption_key: str
|
||||
admin_username: str
|
||||
admin_password: str
|
||||
pixiv_oauth_client_id: str
|
||||
pixiv_oauth_client_secret: str
|
||||
pixiv_oauth_hash_secret: str
|
||||
imgproxy_base_url: str
|
||||
imgproxy_key: str
|
||||
imgproxy_salt: str
|
||||
imgproxy_max_dim: int
|
||||
imgproxy_default_options: str
|
||||
imgproxy_url_chunk_size: int
|
||||
public_api_key_required: bool
|
||||
public_api_key_rpm: int
|
||||
public_api_key_burst: int
|
||||
|
||||
@property
|
||||
def is_prod(self) -> bool:
|
||||
return self.app_env in {"prod", "production"}
|
||||
|
||||
|
||||
def _get(env: Mapping[str, str], key: str, default: str) -> str:
|
||||
value = env.get(key, default)
|
||||
return value.strip()
|
||||
|
||||
|
||||
def _get_bool(env: Mapping[str, str], key: str, default: bool) -> bool:
|
||||
raw = _get(env, key, "1" if default else "0").lower()
|
||||
if raw in {"1", "true", "yes", "y", "on"}:
|
||||
return True
|
||||
if raw in {"0", "false", "no", "n", "off"}:
|
||||
return False
|
||||
return default
|
||||
|
||||
|
||||
def _read_key_file(path: Path) -> str | None:
|
||||
try:
|
||||
raw = path.read_text(encoding="utf-8")
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
except Exception as exc:
|
||||
log.warning("field_encryption_key_read_failed path=%s err=%s", str(path), type(exc).__name__)
|
||||
return None
|
||||
|
||||
value = raw.strip()
|
||||
return value or None
|
||||
|
||||
|
||||
def _atomic_write(path: Path, *, content: str) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_name(path.name + ".tmp")
|
||||
tmp.write_text(content, encoding="utf-8")
|
||||
try:
|
||||
os.chmod(tmp, 0o600)
|
||||
except Exception:
|
||||
pass
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
def _ensure_field_encryption_key(env: Mapping[str, str], *, app_env: str) -> str:
|
||||
key = _get(env, "FIELD_ENCRYPTION_KEY", "")
|
||||
if key:
|
||||
FieldEncryptor.from_key(key)
|
||||
return key
|
||||
|
||||
file_raw = _get(env, "FIELD_ENCRYPTION_KEY_FILE", "")
|
||||
key_file = Path(file_raw) if file_raw else Path("./data/field_encryption_key")
|
||||
|
||||
from_file = _read_key_file(key_file)
|
||||
if from_file is not None:
|
||||
FieldEncryptor.from_key(from_file)
|
||||
return from_file
|
||||
|
||||
if app_env in {"prod", "production"}:
|
||||
return ""
|
||||
|
||||
generated = Fernet.generate_key().decode("utf-8")
|
||||
try:
|
||||
_atomic_write(key_file, content=generated + "\n")
|
||||
log.info("field_encryption_key_generated path=%s", str(key_file))
|
||||
except Exception as exc:
|
||||
log.warning(
|
||||
"field_encryption_key_generated_not_persisted path=%s err=%s",
|
||||
str(key_file),
|
||||
type(exc).__name__,
|
||||
)
|
||||
return generated
|
||||
|
||||
|
||||
def _load_pixiv_oauth_config(env: Mapping[str, str]) -> tuple[str, str, str]:
|
||||
# NOTE: We intentionally do NOT ship any embedded Pixiv OAuth credentials.
|
||||
# If you want token refresh / metadata hydration jobs, configure these via env.
|
||||
client_id = _get(env, "PIXIV_OAUTH_CLIENT_ID", "")
|
||||
client_secret = _get(env, "PIXIV_OAUTH_CLIENT_SECRET", "")
|
||||
hash_secret = _get(env, "PIXIV_OAUTH_HASH_SECRET", "")
|
||||
return client_id, client_secret, hash_secret
|
||||
|
||||
|
||||
def load_settings(env: Mapping[str, str] | None = None) -> Settings:
|
||||
env = env or os.environ
|
||||
|
||||
app_env = _get(env, "APP_ENV", "dev").lower()
|
||||
database_url = _get(env, "DATABASE_URL", "sqlite+aiosqlite:///./data/app.db")
|
||||
secret_key = _get(env, "SECRET_KEY", "dev-secret-key" if app_env != "prod" else "")
|
||||
field_encryption_key = _ensure_field_encryption_key(env, app_env=app_env)
|
||||
|
||||
admin_username = _get(env, "ADMIN_USERNAME", "admin")
|
||||
admin_password = _get(env, "ADMIN_PASSWORD", "admin" if app_env != "prod" else "")
|
||||
|
||||
pixiv_oauth_client_id, pixiv_oauth_client_secret, pixiv_oauth_hash_secret = _load_pixiv_oauth_config(env)
|
||||
|
||||
imgproxy_base_url = _get(env, "IMGPROXY_BASE_URL", "")
|
||||
imgproxy_key = _get(env, "IMGPROXY_KEY", "")
|
||||
imgproxy_salt = _get(env, "IMGPROXY_SALT", "")
|
||||
try:
|
||||
imgproxy_max_dim = int(_get(env, "IMGPROXY_MAX_DIM", "2048") or "2048")
|
||||
except Exception:
|
||||
imgproxy_max_dim = 2048
|
||||
imgproxy_max_dim = max(16, min(int(imgproxy_max_dim), 20_000))
|
||||
|
||||
imgproxy_default_options = _get(env, "IMGPROXY_DEFAULT_OPTIONS", "")
|
||||
|
||||
try:
|
||||
imgproxy_url_chunk_size = int(_get(env, "IMGPROXY_URL_CHUNK_SIZE", "16") or "16")
|
||||
except Exception:
|
||||
imgproxy_url_chunk_size = 16
|
||||
imgproxy_url_chunk_size = max(0, min(int(imgproxy_url_chunk_size), 128))
|
||||
|
||||
public_api_key_required = _get_bool(env, "PUBLIC_API_KEY_REQUIRED", False)
|
||||
try:
|
||||
public_api_key_rpm = int(_get(env, "PUBLIC_API_KEY_RPM", "0") or "0")
|
||||
except Exception:
|
||||
public_api_key_rpm = 0
|
||||
public_api_key_rpm = max(0, min(int(public_api_key_rpm), 10_000_000))
|
||||
try:
|
||||
public_api_key_burst = int(_get(env, "PUBLIC_API_KEY_BURST", "0") or "0")
|
||||
except Exception:
|
||||
public_api_key_burst = 0
|
||||
public_api_key_burst = max(0, min(int(public_api_key_burst), 10_000_000))
|
||||
|
||||
settings = Settings(
|
||||
app_env=app_env,
|
||||
database_url=database_url,
|
||||
secret_key=secret_key,
|
||||
field_encryption_key=field_encryption_key,
|
||||
admin_username=admin_username,
|
||||
admin_password=admin_password,
|
||||
pixiv_oauth_client_id=pixiv_oauth_client_id,
|
||||
pixiv_oauth_client_secret=pixiv_oauth_client_secret,
|
||||
pixiv_oauth_hash_secret=pixiv_oauth_hash_secret,
|
||||
imgproxy_base_url=imgproxy_base_url,
|
||||
imgproxy_key=imgproxy_key,
|
||||
imgproxy_salt=imgproxy_salt,
|
||||
imgproxy_max_dim=imgproxy_max_dim,
|
||||
imgproxy_default_options=imgproxy_default_options,
|
||||
imgproxy_url_chunk_size=imgproxy_url_chunk_size,
|
||||
public_api_key_required=public_api_key_required,
|
||||
public_api_key_rpm=public_api_key_rpm,
|
||||
public_api_key_burst=public_api_key_burst,
|
||||
)
|
||||
|
||||
if settings.is_prod:
|
||||
missing: list[str] = []
|
||||
if not settings.secret_key:
|
||||
missing.append("SECRET_KEY")
|
||||
if not settings.field_encryption_key:
|
||||
missing.append("FIELD_ENCRYPTION_KEY")
|
||||
if not settings.admin_password:
|
||||
missing.append("ADMIN_PASSWORD")
|
||||
if settings.imgproxy_base_url and (not settings.imgproxy_key or not settings.imgproxy_salt):
|
||||
missing.append("IMGPROXY_KEY/IMGPROXY_SALT")
|
||||
if missing:
|
||||
raise ValueError(f"Missing required env vars for prod: {', '.join(missing)}")
|
||||
|
||||
return settings
|
||||
@@ -0,0 +1,44 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from cryptography.fernet import Fernet, InvalidToken
|
||||
|
||||
|
||||
MASKED = "***"
|
||||
|
||||
|
||||
def mask_secret(value: str | None) -> str:
|
||||
value = (value or "").strip()
|
||||
return MASKED if value else ""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class FieldEncryptor:
|
||||
_fernet: Fernet
|
||||
|
||||
@classmethod
|
||||
def from_key(cls, key: str) -> "FieldEncryptor":
|
||||
key = (key or "").strip()
|
||||
if not key:
|
||||
raise ValueError("FIELD_ENCRYPTION_KEY is required")
|
||||
try:
|
||||
fernet = Fernet(key.encode("utf-8"))
|
||||
except Exception as exc:
|
||||
raise ValueError("Invalid FIELD_ENCRYPTION_KEY") from exc
|
||||
return cls(_fernet=fernet)
|
||||
|
||||
def encrypt_text(self, plaintext: str) -> str:
|
||||
if not isinstance(plaintext, str):
|
||||
raise TypeError("plaintext must be str")
|
||||
token = self._fernet.encrypt(plaintext.encode("utf-8"))
|
||||
return token.decode("utf-8")
|
||||
|
||||
def decrypt_text(self, token: str) -> str:
|
||||
if not isinstance(token, str):
|
||||
raise TypeError("token must be str")
|
||||
try:
|
||||
plaintext = self._fernet.decrypt(token.encode("utf-8"))
|
||||
except InvalidToken as exc:
|
||||
raise ValueError("Invalid encrypted value") from exc
|
||||
return plaintext.decode("utf-8")
|
||||
@@ -0,0 +1,57 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy.engine.url import make_url
|
||||
|
||||
|
||||
def get_sqlite_db_dir(database_url: str) -> Path:
|
||||
"""
|
||||
Returns the directory containing the SQLite database file.
|
||||
|
||||
For non-SQLite or in-memory DBs, falls back to ./data relative to CWD.
|
||||
"""
|
||||
|
||||
fallback = Path("./data").resolve()
|
||||
|
||||
try:
|
||||
url = make_url(database_url)
|
||||
except Exception:
|
||||
return fallback
|
||||
|
||||
backend = (url.get_backend_name() or "").lower()
|
||||
if backend != "sqlite":
|
||||
return fallback
|
||||
|
||||
db = str(url.database or "").strip()
|
||||
if not db or db == ":memory:":
|
||||
return fallback
|
||||
|
||||
p = Path(db)
|
||||
if not p.is_absolute():
|
||||
p = Path.cwd() / p
|
||||
return p.resolve().parent
|
||||
|
||||
|
||||
def make_file_ref(path: Path, *, base_dir: Path) -> str:
|
||||
base = base_dir.resolve()
|
||||
target = path.resolve()
|
||||
return target.relative_to(base).as_posix()
|
||||
|
||||
|
||||
def resolve_file_ref(file_ref: str, *, base_dir: Path) -> Path:
|
||||
raw = (file_ref or "").strip()
|
||||
if not raw:
|
||||
raise ValueError("file_ref is required")
|
||||
|
||||
rel = Path(raw)
|
||||
if rel.is_absolute():
|
||||
raise ValueError("file_ref must be relative")
|
||||
if any(part == ".." for part in rel.parts):
|
||||
raise ValueError("file_ref must not contain '..'")
|
||||
|
||||
base = base_dir.resolve()
|
||||
target = (base / rel).resolve()
|
||||
target.relative_to(base) # raises if escaped
|
||||
return target
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Mapping
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine
|
||||
|
||||
from app.core.config import Settings, load_settings
|
||||
from app.core.runtime_settings import RuntimeConfig, load_runtime_config
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class EffectiveSettings:
|
||||
env: Settings
|
||||
runtime: RuntimeConfig
|
||||
|
||||
|
||||
async def load_effective_settings(engine: AsyncEngine, *, env: Mapping[str, str] | None = None) -> EffectiveSettings:
|
||||
settings = load_settings(env)
|
||||
runtime = await load_runtime_config(engine)
|
||||
return EffectiveSettings(env=settings, runtime=runtime)
|
||||
|
||||
@@ -0,0 +1,187 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
|
||||
|
||||
class ErrorCode(str, Enum):
|
||||
# 4.1 通用
|
||||
BAD_REQUEST = "BAD_REQUEST"
|
||||
UNAUTHORIZED = "UNAUTHORIZED"
|
||||
FORBIDDEN = "FORBIDDEN"
|
||||
NOT_FOUND = "NOT_FOUND"
|
||||
RATE_LIMITED = "RATE_LIMITED"
|
||||
INTERNAL_ERROR = "INTERNAL_ERROR"
|
||||
# 4.2 Random
|
||||
NO_MATCH = "NO_MATCH"
|
||||
UPSTREAM_STREAM_ERROR = "UPSTREAM_STREAM_ERROR"
|
||||
UPSTREAM_403 = "UPSTREAM_403"
|
||||
UPSTREAM_404 = "UPSTREAM_404"
|
||||
UPSTREAM_RATE_LIMIT = "UPSTREAM_RATE_LIMIT"
|
||||
# 4.3 Import
|
||||
INVALID_UPLOAD_TYPE = "INVALID_UPLOAD_TYPE"
|
||||
PAYLOAD_TOO_LARGE = "PAYLOAD_TOO_LARGE"
|
||||
UNSUPPORTED_URL = "UNSUPPORTED_URL"
|
||||
# 4.4 Pixiv/Token
|
||||
TOKEN_REFRESH_FAILED = "TOKEN_REFRESH_FAILED"
|
||||
TOKEN_BACKOFF = "TOKEN_BACKOFF"
|
||||
NO_TOKEN_AVAILABLE = "NO_TOKEN_AVAILABLE"
|
||||
# 4.5 Proxy
|
||||
PROXY_REQUIRED = "PROXY_REQUIRED"
|
||||
PROXY_AUTH_FAILED = "PROXY_AUTH_FAILED"
|
||||
PROXY_CONNECT_FAILED = "PROXY_CONNECT_FAILED"
|
||||
|
||||
|
||||
UNKNOWN_REQUEST_ID = "req_unknown"
|
||||
|
||||
_DEFAULT_ZH_MESSAGE_BY_CODE: dict[ErrorCode, str] = {
|
||||
ErrorCode.BAD_REQUEST: "请求参数错误",
|
||||
ErrorCode.UNAUTHORIZED: "未授权",
|
||||
ErrorCode.FORBIDDEN: "禁止访问",
|
||||
ErrorCode.NOT_FOUND: "资源不存在",
|
||||
ErrorCode.RATE_LIMITED: "请求过于频繁",
|
||||
ErrorCode.INTERNAL_ERROR: "服务器内部错误",
|
||||
ErrorCode.NO_MATCH: "没有匹配的图片",
|
||||
ErrorCode.UPSTREAM_STREAM_ERROR: "上游请求失败",
|
||||
ErrorCode.UPSTREAM_403: "上游拒绝访问(403)",
|
||||
ErrorCode.UPSTREAM_404: "上游资源不存在(404)",
|
||||
ErrorCode.UPSTREAM_RATE_LIMIT: "上游触发限流(429)",
|
||||
ErrorCode.INVALID_UPLOAD_TYPE: "上传类型不支持",
|
||||
ErrorCode.PAYLOAD_TOO_LARGE: "请求体过大",
|
||||
ErrorCode.UNSUPPORTED_URL: "链接格式不支持",
|
||||
ErrorCode.TOKEN_REFRESH_FAILED: "令牌刷新失败",
|
||||
ErrorCode.TOKEN_BACKOFF: "令牌已进入退避",
|
||||
ErrorCode.NO_TOKEN_AVAILABLE: "没有可用令牌",
|
||||
ErrorCode.PROXY_REQUIRED: "需要代理",
|
||||
ErrorCode.PROXY_AUTH_FAILED: "代理认证失败",
|
||||
ErrorCode.PROXY_CONNECT_FAILED: "代理连接失败",
|
||||
}
|
||||
|
||||
|
||||
def _is_ascii_only(text: str) -> bool:
|
||||
return all(ord(ch) < 128 for ch in text)
|
||||
|
||||
|
||||
_ASCII_MESSAGE_MAP: dict[str, str] = {
|
||||
"missing api key": "缺少 API Key",
|
||||
"invalid api key": "API Key 无效",
|
||||
"rate limited": "请求过于频繁",
|
||||
"database unavailable": "数据库不可用",
|
||||
"invalid json body": "JSON 请求体无效",
|
||||
"unsupported limit": "limit 参数不支持",
|
||||
"unsupported cursor": "cursor 参数不支持",
|
||||
"unsupported status": "status 参数不支持",
|
||||
"unsupported type": "type 参数不支持",
|
||||
"unsupported format": "format 参数不支持",
|
||||
"unsupported redirect": "redirect 参数不支持",
|
||||
"unsupported seed": "seed 参数不支持",
|
||||
"unsupported ai_type": "ai_type 参数不支持",
|
||||
"unsupported orientation": "orientation 参数不支持",
|
||||
"unsupported r18": "r18 参数不支持",
|
||||
"unsupported min_*": "min_* 参数不支持",
|
||||
"too many tag filters": "标签筛选条件过多",
|
||||
"pixiv oauth not configured": "未配置 Pixiv OAuth(client_id/client_secret)",
|
||||
"encryption not configured": "未配置加密密钥(FIELD_ENCRYPTION_KEY)",
|
||||
"invalid stored token": "令牌密文无效(无法解密)",
|
||||
"token not found": "令牌不存在",
|
||||
"job not found": "任务不存在",
|
||||
"proxy endpoint not found": "代理节点不存在",
|
||||
"proxy pool not found": "代理池不存在",
|
||||
"job is running": "任务正在运行中",
|
||||
"proxy authentication failed": "代理认证失败",
|
||||
"proxy connect failed": "代理连接失败",
|
||||
"upstream request failed": "上游请求失败",
|
||||
"upstream forbidden": "上游拒绝访问(403)",
|
||||
"upstream not found": "上游资源不存在(404)",
|
||||
"upstream rate limited": "上游触发限流(429)",
|
||||
"upstream error": "上游错误",
|
||||
"upstream request failed after attempts.": "多次尝试后上游请求仍失败",
|
||||
"invalid proxy endpoint": "代理节点配置无效",
|
||||
"unsupported conflict_policy": "conflict_policy 参数不支持",
|
||||
}
|
||||
|
||||
|
||||
def normalize_error_message(*, code: ErrorCode, message: str) -> str:
|
||||
msg = str(message or "").strip()
|
||||
if not msg:
|
||||
return _DEFAULT_ZH_MESSAGE_BY_CODE.get(code, "请求失败")
|
||||
|
||||
if not _is_ascii_only(msg):
|
||||
return msg
|
||||
|
||||
key = msg.lower()
|
||||
if key in _ASCII_MESSAGE_MAP:
|
||||
return _ASCII_MESSAGE_MAP[key]
|
||||
|
||||
if key.startswith("unsupported "):
|
||||
rest = msg[len("Unsupported ") :].strip()
|
||||
return f"参数不支持:{rest}" if rest else _DEFAULT_ZH_MESSAGE_BY_CODE.get(code, "请求失败")
|
||||
|
||||
if key.startswith("invalid "):
|
||||
rest = msg[len("Invalid ") :].strip()
|
||||
return f"参数无效:{rest}" if rest else _DEFAULT_ZH_MESSAGE_BY_CODE.get(code, "请求失败")
|
||||
|
||||
if key.startswith("missing "):
|
||||
rest = msg[len("Missing ") :].strip()
|
||||
return f"缺少字段:{rest}" if rest else _DEFAULT_ZH_MESSAGE_BY_CODE.get(code, "请求失败")
|
||||
|
||||
return _DEFAULT_ZH_MESSAGE_BY_CODE.get(code, "请求失败")
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ApiError(Exception):
|
||||
code: ErrorCode
|
||||
message: str
|
||||
status_code: int = 400
|
||||
details: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def _coerce_request_id(request_id: str | None) -> str:
|
||||
request_id = (request_id or "").strip()
|
||||
return request_id if request_id else UNKNOWN_REQUEST_ID
|
||||
|
||||
|
||||
def _request_id_from_request(request: Any | None) -> str | None:
|
||||
if request is None:
|
||||
return None
|
||||
request_id = getattr(getattr(request, "state", None), "request_id", None)
|
||||
if request_id:
|
||||
return str(request_id)
|
||||
header = request.headers.get("X-Request-Id")
|
||||
return header.strip() if header else None
|
||||
|
||||
|
||||
def error_body(
|
||||
*,
|
||||
code: ErrorCode,
|
||||
message: str,
|
||||
request_id: str | None = None,
|
||||
details: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"ok": False,
|
||||
"code": code.value,
|
||||
"message": normalize_error_message(code=code, message=message),
|
||||
"request_id": _coerce_request_id(request_id),
|
||||
"details": details or {},
|
||||
}
|
||||
|
||||
|
||||
def json_error_response(
|
||||
*,
|
||||
code: ErrorCode,
|
||||
message: str,
|
||||
status_code: int,
|
||||
request: Any | None = None,
|
||||
request_id: str | None = None,
|
||||
details: dict[str, Any] | None = None,
|
||||
) -> Any:
|
||||
if request_id is None:
|
||||
request_id = _request_id_from_request(request)
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
return JSONResponse(
|
||||
status_code=status_code,
|
||||
content=error_body(code=code, message=message, request_id=request_id, details=details),
|
||||
)
|
||||
@@ -0,0 +1,76 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from enum import Enum
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
class OutboundErrorKind(str, Enum):
|
||||
PROXY_CONNECT = "proxy_connect"
|
||||
PROXY_AUTH = "proxy_auth"
|
||||
PIXIV_RATE_LIMIT = "pixiv_rate_limit"
|
||||
|
||||
|
||||
def classify_httpx_exception(exc: BaseException) -> OutboundErrorKind | None:
|
||||
if isinstance(exc, httpx.ProxyError):
|
||||
msg = str(exc).lower()
|
||||
if "407" in msg or "proxy authentication" in msg:
|
||||
return OutboundErrorKind.PROXY_AUTH
|
||||
return OutboundErrorKind.PROXY_CONNECT
|
||||
return None
|
||||
|
||||
|
||||
def classify_pixiv_rate_limit(*, status_code: int, body_text: str | None) -> OutboundErrorKind | None:
|
||||
if int(status_code) != 403:
|
||||
return None
|
||||
text = (body_text or "").strip().lower()
|
||||
if "rate limit" in text:
|
||||
return OutboundErrorKind.PIXIV_RATE_LIMIT
|
||||
return None
|
||||
|
||||
|
||||
def should_override_proxy(kind: OutboundErrorKind) -> bool:
|
||||
return kind in {OutboundErrorKind.PROXY_CONNECT, OutboundErrorKind.PROXY_AUTH}
|
||||
|
||||
|
||||
def should_backoff_token(kind: OutboundErrorKind) -> bool:
|
||||
return kind == OutboundErrorKind.PIXIV_RATE_LIMIT
|
||||
|
||||
|
||||
def proxy_override_ttl_seconds(*, attempt: int) -> int:
|
||||
attempt_i = int(attempt)
|
||||
if attempt_i <= 0:
|
||||
return 0
|
||||
|
||||
schedule = {
|
||||
1: 20 * 60,
|
||||
2: 60 * 60,
|
||||
3: 6 * 60 * 60,
|
||||
}
|
||||
if attempt_i in schedule:
|
||||
return schedule[attempt_i]
|
||||
|
||||
base = 6 * 60 * 60
|
||||
seconds = base * (2 ** (attempt_i - 3))
|
||||
return min(seconds, 24 * 60 * 60)
|
||||
|
||||
|
||||
def pixiv_rate_limit_backoff_seconds(*, attempt: int) -> int:
|
||||
attempt_i = int(attempt)
|
||||
if attempt_i <= 0:
|
||||
return 0
|
||||
|
||||
schedule = {
|
||||
1: 60,
|
||||
2: 5 * 60,
|
||||
3: 15 * 60,
|
||||
4: 60 * 60,
|
||||
5: 6 * 60 * 60,
|
||||
}
|
||||
if attempt_i in schedule:
|
||||
return schedule[attempt_i]
|
||||
|
||||
base = 6 * 60 * 60
|
||||
seconds = base * (2 ** (attempt_i - 5))
|
||||
return min(seconds, 24 * 60 * 60)
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from starlette.responses import StreamingResponse
|
||||
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.metrics import UPSTREAM_STREAM_ERRORS_TOTAL
|
||||
|
||||
PIXIV_REFERER = "https://www.pixiv.net/"
|
||||
|
||||
|
||||
async def stream_url(
|
||||
url: str,
|
||||
*,
|
||||
transport: httpx.BaseTransport | None = None,
|
||||
proxy: str | None = None,
|
||||
cache_control: str,
|
||||
referer: str = PIXIV_REFERER,
|
||||
timeout_s: float = 30.0,
|
||||
range_header: str | None = None,
|
||||
) -> StreamingResponse:
|
||||
client = httpx.AsyncClient(
|
||||
transport=transport,
|
||||
proxy=proxy,
|
||||
follow_redirects=True,
|
||||
timeout=httpx.Timeout(timeout_s, connect=10.0),
|
||||
)
|
||||
|
||||
request_headers: dict[str, str] = {}
|
||||
if referer:
|
||||
request_headers["Referer"] = referer
|
||||
if range_header:
|
||||
request_headers["Range"] = range_header
|
||||
request = client.build_request("GET", url, headers=request_headers)
|
||||
|
||||
try:
|
||||
upstream = await client.send(request, stream=True)
|
||||
except httpx.ProxyError as exc:
|
||||
UPSTREAM_STREAM_ERRORS_TOTAL.inc()
|
||||
await client.aclose()
|
||||
msg = str(exc).lower()
|
||||
if "407" in msg or "proxy authentication" in msg:
|
||||
raise ApiError(
|
||||
code=ErrorCode.PROXY_AUTH_FAILED,
|
||||
message="代理认证失败",
|
||||
status_code=502,
|
||||
) from exc
|
||||
raise ApiError(
|
||||
code=ErrorCode.PROXY_CONNECT_FAILED,
|
||||
message="代理连接失败",
|
||||
status_code=502,
|
||||
) from exc
|
||||
except Exception as exc:
|
||||
UPSTREAM_STREAM_ERRORS_TOTAL.inc()
|
||||
await client.aclose()
|
||||
raise ApiError(code=ErrorCode.UPSTREAM_STREAM_ERROR, message="上游请求失败", status_code=502) from exc
|
||||
|
||||
if upstream.status_code not in {200, 206}:
|
||||
status = upstream.status_code
|
||||
UPSTREAM_STREAM_ERRORS_TOTAL.inc()
|
||||
await upstream.aclose()
|
||||
await client.aclose()
|
||||
if status == 403:
|
||||
raise ApiError(code=ErrorCode.UPSTREAM_403, message="上游拒绝访问(403)", status_code=502)
|
||||
if status == 404:
|
||||
raise ApiError(code=ErrorCode.UPSTREAM_404, message="上游资源不存在(404)", status_code=502)
|
||||
if status == 429:
|
||||
raise ApiError(code=ErrorCode.UPSTREAM_RATE_LIMIT, message="上游触发限流(429)", status_code=502)
|
||||
raise ApiError(code=ErrorCode.UPSTREAM_STREAM_ERROR, message="上游错误", status_code=502)
|
||||
|
||||
media_type = upstream.headers.get("content-type") or "application/octet-stream"
|
||||
content_length = upstream.headers.get("content-length")
|
||||
|
||||
async def _iter_bytes() -> Any:
|
||||
try:
|
||||
async for chunk in upstream.aiter_bytes():
|
||||
yield chunk
|
||||
except Exception:
|
||||
UPSTREAM_STREAM_ERRORS_TOTAL.inc()
|
||||
raise
|
||||
finally:
|
||||
await upstream.aclose()
|
||||
await client.aclose()
|
||||
|
||||
accept_ranges = upstream.headers.get("accept-ranges")
|
||||
content_range = upstream.headers.get("content-range")
|
||||
|
||||
resp = StreamingResponse(_iter_bytes(), status_code=upstream.status_code, media_type=media_type)
|
||||
resp.headers["Cache-Control"] = cache_control
|
||||
if content_length:
|
||||
resp.headers["Content-Length"] = content_length
|
||||
if accept_ranges:
|
||||
resp.headers["Accept-Ranges"] = accept_ranges
|
||||
if content_range:
|
||||
resp.headers["Content-Range"] = content_range
|
||||
return resp
|
||||
@@ -0,0 +1,148 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
from dataclasses import dataclass
|
||||
from typing import Mapping
|
||||
|
||||
from app.core.config import Settings
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ImgproxyConfig:
|
||||
base_url: str
|
||||
key: bytes
|
||||
salt: bytes
|
||||
max_dim: int
|
||||
default_options: str
|
||||
url_chunk_size: int
|
||||
|
||||
|
||||
def _decode_hex(raw: str, *, name: str) -> bytes:
|
||||
raw = (raw or "").strip()
|
||||
if not raw:
|
||||
raise ValueError(f"{name} is required")
|
||||
try:
|
||||
return bytes.fromhex(raw)
|
||||
except Exception as exc:
|
||||
raise ValueError(f"{name} must be hex") from exc
|
||||
|
||||
|
||||
def load_imgproxy_config_from_settings(settings: Settings) -> ImgproxyConfig | None:
|
||||
base_url = (settings.imgproxy_base_url or "").strip()
|
||||
if not base_url:
|
||||
return None
|
||||
|
||||
key = _decode_hex(settings.imgproxy_key, name="IMGPROXY_KEY")
|
||||
salt = _decode_hex(settings.imgproxy_salt, name="IMGPROXY_SALT")
|
||||
|
||||
default_options = (settings.imgproxy_default_options or "").strip().strip("/")
|
||||
if not default_options:
|
||||
default_options = f"rs:fit:{int(settings.imgproxy_max_dim)}:{int(settings.imgproxy_max_dim)}"
|
||||
|
||||
return ImgproxyConfig(
|
||||
base_url=base_url.rstrip("/"),
|
||||
key=key,
|
||||
salt=salt,
|
||||
max_dim=int(settings.imgproxy_max_dim),
|
||||
default_options=default_options,
|
||||
url_chunk_size=int(settings.imgproxy_url_chunk_size),
|
||||
)
|
||||
|
||||
|
||||
def urlsafe_b64_no_pad(raw: bytes) -> str:
|
||||
return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii")
|
||||
|
||||
|
||||
def encode_source_url(source_url: str, *, chunk_size: int) -> str:
|
||||
source_url = (source_url or "").strip()
|
||||
if not source_url:
|
||||
raise ValueError("source_url is required")
|
||||
|
||||
encoded = urlsafe_b64_no_pad(source_url.encode("utf-8"))
|
||||
if chunk_size <= 0 or len(encoded) <= chunk_size:
|
||||
return encoded
|
||||
return "/".join(encoded[i : i + chunk_size] for i in range(0, len(encoded), chunk_size))
|
||||
|
||||
|
||||
def sign_path(cfg: ImgproxyConfig, path_after_signature: str) -> str:
|
||||
path = (path_after_signature or "").strip()
|
||||
if not path.startswith("/"):
|
||||
raise ValueError("path_after_signature must start with '/'")
|
||||
|
||||
mac = hmac.new(cfg.key, digestmod=hashlib.sha256)
|
||||
mac.update(cfg.salt)
|
||||
mac.update(path.encode("utf-8"))
|
||||
return urlsafe_b64_no_pad(mac.digest())
|
||||
|
||||
|
||||
def build_processing_path(
|
||||
*,
|
||||
processing_options: str,
|
||||
source_url: str,
|
||||
extension: str,
|
||||
url_chunk_size: int,
|
||||
) -> str:
|
||||
processing_options = (processing_options or "").strip().strip("/")
|
||||
if not processing_options:
|
||||
raise ValueError("processing_options is required")
|
||||
|
||||
extension = (extension or "").strip().lower().lstrip(".")
|
||||
if not extension or len(extension) > 10 or any(c for c in extension if not (c.isalnum() or c == "_")):
|
||||
raise ValueError("extension is invalid")
|
||||
|
||||
encoded = encode_source_url(source_url, chunk_size=int(url_chunk_size))
|
||||
return f"/{processing_options}/{encoded}.{extension}"
|
||||
|
||||
|
||||
def build_signed_processing_url(
|
||||
cfg: ImgproxyConfig,
|
||||
*,
|
||||
source_url: str,
|
||||
extension: str,
|
||||
processing_options: str | None = None,
|
||||
) -> str:
|
||||
path = build_processing_path(
|
||||
processing_options=processing_options or cfg.default_options,
|
||||
source_url=source_url,
|
||||
extension=extension,
|
||||
url_chunk_size=int(cfg.url_chunk_size),
|
||||
)
|
||||
sig = sign_path(cfg, path)
|
||||
return f"{cfg.base_url}/{sig}{path}"
|
||||
|
||||
|
||||
def load_imgproxy_config(env: Mapping[str, str]) -> ImgproxyConfig | None:
|
||||
base_url = (env.get("IMGPROXY_BASE_URL") or "").strip()
|
||||
if not base_url:
|
||||
return None
|
||||
|
||||
key = _decode_hex(env.get("IMGPROXY_KEY") or "", name="IMGPROXY_KEY")
|
||||
salt = _decode_hex(env.get("IMGPROXY_SALT") or "", name="IMGPROXY_SALT")
|
||||
|
||||
try:
|
||||
max_dim = int((env.get("IMGPROXY_MAX_DIM") or "2048").strip() or "2048")
|
||||
except Exception:
|
||||
max_dim = 2048
|
||||
max_dim = max(16, min(int(max_dim), 20_000))
|
||||
|
||||
default_options = (env.get("IMGPROXY_DEFAULT_OPTIONS") or "").strip().strip("/")
|
||||
if not default_options:
|
||||
default_options = f"rs:fit:{max_dim}:{max_dim}"
|
||||
|
||||
try:
|
||||
url_chunk_size = int((env.get("IMGPROXY_URL_CHUNK_SIZE") or "16").strip() or "16")
|
||||
except Exception:
|
||||
url_chunk_size = 16
|
||||
url_chunk_size = max(0, min(int(url_chunk_size), 128))
|
||||
|
||||
return ImgproxyConfig(
|
||||
base_url=base_url.rstrip("/"),
|
||||
key=key,
|
||||
salt=salt,
|
||||
max_dim=max_dim,
|
||||
default_options=default_options,
|
||||
url_chunk_size=url_chunk_size,
|
||||
)
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from app.core.redact import redact_any
|
||||
|
||||
|
||||
class RedactFilter(logging.Filter):
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
try:
|
||||
message = record.getMessage()
|
||||
record.msg = redact_any(message)
|
||||
record.args = ()
|
||||
for key, value in list(record.__dict__.items()):
|
||||
if key.startswith("_"):
|
||||
continue
|
||||
record.__dict__[key] = redact_any(value)
|
||||
except Exception:
|
||||
pass
|
||||
return True
|
||||
|
||||
|
||||
def configure_logging(level: int = logging.INFO) -> None:
|
||||
logging.basicConfig(level=level, format="%(levelname)s %(name)s %(message)s")
|
||||
logging.getLogger().addFilter(RedactFilter())
|
||||
|
||||
|
||||
def get_logger(name: str) -> logging.Logger:
|
||||
return logging.getLogger(name)
|
||||
@@ -0,0 +1,170 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable
|
||||
|
||||
from prometheus_client import Counter, Gauge, Histogram
|
||||
|
||||
RANDOM_RESULTS: tuple[str, ...] = (
|
||||
"ok",
|
||||
"no_match",
|
||||
"upstream_error",
|
||||
"bad_request",
|
||||
"error",
|
||||
)
|
||||
|
||||
JOB_STATUSES: tuple[str, ...] = (
|
||||
"pending",
|
||||
"running",
|
||||
"paused",
|
||||
"canceled",
|
||||
"completed",
|
||||
"failed",
|
||||
"dlq",
|
||||
)
|
||||
|
||||
PROXY_STATES: tuple[str, ...] = (
|
||||
"total",
|
||||
"enabled",
|
||||
"healthy",
|
||||
"unhealthy",
|
||||
"blacklisted",
|
||||
)
|
||||
|
||||
RANDOM_REQUESTS_TOTAL = Counter(
|
||||
"new_pixiv_random_requests_total",
|
||||
"Total /random requests by result.",
|
||||
["result"],
|
||||
)
|
||||
|
||||
RANDOM_NO_MATCH_TOTAL = Counter(
|
||||
"new_pixiv_random_no_match_total",
|
||||
"Total /random NO_MATCH responses.",
|
||||
)
|
||||
|
||||
RANDOM_OPPORTUNISTIC_HYDRATE_ENQUEUED_TOTAL = Counter(
|
||||
"new_pixiv_random_opportunistic_hydrate_enqueued_total",
|
||||
"Total opportunistic hydrate_metadata enqueues from /random.",
|
||||
)
|
||||
|
||||
RANDOM_LATENCY_SECONDS = Histogram(
|
||||
"new_pixiv_random_latency_seconds",
|
||||
"Latency for /random endpoint (seconds).",
|
||||
buckets=(
|
||||
0.01,
|
||||
0.025,
|
||||
0.05,
|
||||
0.1,
|
||||
0.25,
|
||||
0.5,
|
||||
1.0,
|
||||
2.5,
|
||||
5.0,
|
||||
10.0,
|
||||
),
|
||||
)
|
||||
|
||||
UPSTREAM_STREAM_ERRORS_TOTAL = Counter(
|
||||
"new_pixiv_upstream_stream_errors_total",
|
||||
"Total upstream stream failures (stream_url).",
|
||||
)
|
||||
|
||||
JOBS_CLAIM_TOTAL = Counter(
|
||||
"new_pixiv_jobs_claim_total",
|
||||
"Total jobs claimed by workers.",
|
||||
)
|
||||
|
||||
JOBS_FAILED_TOTAL = Counter(
|
||||
"new_pixiv_jobs_failed_total",
|
||||
"Total jobs transitioned to failed/dlq status.",
|
||||
)
|
||||
|
||||
TOKEN_REFRESH_FAIL_TOTAL = Counter(
|
||||
"new_pixiv_token_refresh_fail_total",
|
||||
"Total token refresh failures.",
|
||||
)
|
||||
|
||||
JOBS_STATUS_COUNT = Gauge(
|
||||
"new_pixiv_jobs_status_count",
|
||||
"Current jobs count by status (from SQLite).",
|
||||
["status"],
|
||||
)
|
||||
|
||||
PROXY_ENDPOINTS_STATE_COUNT = Gauge(
|
||||
"new_pixiv_proxy_endpoints_state_count",
|
||||
"Current proxy endpoints count by state (from SQLite).",
|
||||
["state"],
|
||||
)
|
||||
|
||||
PROXY_PROBE_LATENCY_MS = Histogram(
|
||||
"new_pixiv_proxy_probe_latency_ms",
|
||||
"Proxy probe latency (ms).",
|
||||
buckets=(
|
||||
10.0,
|
||||
25.0,
|
||||
50.0,
|
||||
100.0,
|
||||
250.0,
|
||||
500.0,
|
||||
1000.0,
|
||||
2000.0,
|
||||
5000.0,
|
||||
10000.0,
|
||||
),
|
||||
)
|
||||
|
||||
METRICS_SCRAPE_ERRORS_TOTAL = Counter(
|
||||
"new_pixiv_metrics_scrape_errors_total",
|
||||
"Total /metrics scrape errors while querying backing dependencies.",
|
||||
)
|
||||
|
||||
METRICS_LAST_SCRAPE_SUCCESS = Gauge(
|
||||
"new_pixiv_metrics_last_scrape_success",
|
||||
"Last /metrics scrape success (1=ok, 0=error).",
|
||||
)
|
||||
|
||||
|
||||
def _init_labelsets() -> None:
|
||||
for result in RANDOM_RESULTS:
|
||||
RANDOM_REQUESTS_TOTAL.labels(result=result).inc(0)
|
||||
RANDOM_NO_MATCH_TOTAL.inc(0)
|
||||
RANDOM_OPPORTUNISTIC_HYDRATE_ENQUEUED_TOTAL.inc(0)
|
||||
UPSTREAM_STREAM_ERRORS_TOTAL.inc(0)
|
||||
JOBS_CLAIM_TOTAL.inc(0)
|
||||
JOBS_FAILED_TOTAL.inc(0)
|
||||
TOKEN_REFRESH_FAIL_TOTAL.inc(0)
|
||||
for status in JOB_STATUSES:
|
||||
JOBS_STATUS_COUNT.labels(status=status).set(0)
|
||||
for state in PROXY_STATES:
|
||||
PROXY_ENDPOINTS_STATE_COUNT.labels(state=state).set(0)
|
||||
METRICS_LAST_SCRAPE_SUCCESS.set(1)
|
||||
|
||||
|
||||
_init_labelsets()
|
||||
|
||||
|
||||
def observe_random_result(*, result: str, duration_s: float | None) -> None:
|
||||
result = (result or "").strip()
|
||||
if result not in RANDOM_RESULTS:
|
||||
result = "error"
|
||||
RANDOM_REQUESTS_TOTAL.labels(result=result).inc()
|
||||
if result == "no_match":
|
||||
RANDOM_NO_MATCH_TOTAL.inc()
|
||||
if duration_s is not None and duration_s >= 0:
|
||||
RANDOM_LATENCY_SECONDS.observe(duration_s)
|
||||
|
||||
|
||||
def set_jobs_status_counts(counts: dict[str, int]) -> None:
|
||||
for status in JOB_STATUSES:
|
||||
JOBS_STATUS_COUNT.labels(status=status).set(float(int(counts.get(status, 0) or 0)))
|
||||
|
||||
|
||||
def set_proxy_state_counts(counts: dict[str, int]) -> None:
|
||||
for state in PROXY_STATES:
|
||||
PROXY_ENDPOINTS_STATE_COUNT.labels(state=state).set(float(int(counts.get(state, 0) or 0)))
|
||||
|
||||
|
||||
def ensure_known_keys(keys: Iterable[str], counts: dict[str, int]) -> dict[str, int]:
|
||||
out = dict(counts)
|
||||
for k in keys:
|
||||
out.setdefault(k, 0)
|
||||
return out
|
||||
@@ -0,0 +1,44 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PixivOriginalUrl:
|
||||
illust_id: int
|
||||
page_index: int
|
||||
ext: str
|
||||
|
||||
|
||||
_PIXIV_P_RE = re.compile(r"(?P<illust_id>\d+)_p(?P<page_index>\d+)(?:_master1200)?\.(?P<ext>[A-Za-z0-9]+)$")
|
||||
_PIXIV_UGOIRA_RE = re.compile(r"(?P<illust_id>\d+)_ugoira(?P<page_index>\d+)\.(?P<ext>[A-Za-z0-9]+)$")
|
||||
|
||||
ALLOWED_IMAGE_EXTS = {"jpg", "jpeg", "png", "gif", "webp"}
|
||||
|
||||
|
||||
def parse_pixiv_original_url(url: str) -> PixivOriginalUrl:
|
||||
url = url.strip()
|
||||
if not url:
|
||||
raise ValueError("url is required")
|
||||
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme.lower() not in {"http", "https"}:
|
||||
raise ValueError("unsupported scheme")
|
||||
|
||||
host = (parsed.hostname or "").lower()
|
||||
if not host.endswith("pximg.net"):
|
||||
raise ValueError("unsupported host")
|
||||
|
||||
m = _PIXIV_P_RE.search(parsed.path) or _PIXIV_UGOIRA_RE.search(parsed.path)
|
||||
if not m:
|
||||
raise ValueError("unsupported pixiv original url")
|
||||
|
||||
illust_id = int(m.group("illust_id"))
|
||||
page_index = int(m.group("page_index"))
|
||||
ext = m.group("ext").lower()
|
||||
if ext not in ALLOWED_IMAGE_EXTS:
|
||||
raise ValueError("unsupported ext")
|
||||
|
||||
return PixivOriginalUrl(illust_id=illust_id, page_index=page_index, ext=ext)
|
||||
@@ -0,0 +1,539 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
from urllib.parse import quote
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine
|
||||
|
||||
from app.core.config import Settings
|
||||
from app.core.crypto import FieldEncryptor
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
from app.core.runtime_settings import RuntimeConfig
|
||||
from app.core.time import iso_utc_ms
|
||||
from app.db.session import with_sqlite_busy_retry
|
||||
|
||||
|
||||
_PIXIV_HOST_SUFFIXES = (
|
||||
"pixiv.net",
|
||||
"pximg.net",
|
||||
"secure.pixiv.net",
|
||||
)
|
||||
|
||||
|
||||
def _normalize_host(host: str) -> str:
|
||||
return (host or "").strip().lower().strip(".")
|
||||
|
||||
|
||||
def host_from_url(url: str) -> str | None:
|
||||
url = (url or "").strip()
|
||||
if not url:
|
||||
return None
|
||||
try:
|
||||
parsed = urlparse(url)
|
||||
except Exception:
|
||||
return None
|
||||
host = _normalize_host(parsed.hostname or "")
|
||||
return host or None
|
||||
|
||||
|
||||
def _suffix_match(*, host: str, suffix: str) -> bool:
|
||||
host = _normalize_host(host)
|
||||
suffix = _normalize_host(suffix)
|
||||
if not host or not suffix:
|
||||
return False
|
||||
return host == suffix or host.endswith("." + suffix)
|
||||
|
||||
|
||||
def should_use_proxy_for_host(runtime: RuntimeConfig, *, host: str) -> bool:
|
||||
if not bool(runtime.proxy_enabled):
|
||||
return False
|
||||
|
||||
host_n = _normalize_host(host)
|
||||
if not host_n:
|
||||
return False
|
||||
|
||||
mode = (runtime.proxy_route_mode or "").strip().lower()
|
||||
if mode in {"off"}:
|
||||
return False
|
||||
if mode == "all":
|
||||
return True
|
||||
if mode == "allowlist":
|
||||
return any(_suffix_match(host=host_n, suffix=d) for d in (runtime.proxy_allowlist_domains or []))
|
||||
if mode == "pixiv_only":
|
||||
return any(_suffix_match(host=host_n, suffix=s) for s in _PIXIV_HOST_SUFFIXES)
|
||||
return False
|
||||
|
||||
|
||||
def resolve_pool_id_for_host(runtime: RuntimeConfig, *, host: str) -> int | None:
|
||||
host_n = _normalize_host(host)
|
||||
if not host_n:
|
||||
return None
|
||||
|
||||
best_len = -1
|
||||
best: int | None = None
|
||||
for suffix, pool_id in (runtime.proxy_route_pools or {}).items():
|
||||
suf = _normalize_host(str(suffix))
|
||||
if not suf:
|
||||
continue
|
||||
if not _suffix_match(host=host_n, suffix=suf):
|
||||
continue
|
||||
if len(suf) > best_len:
|
||||
best_len = len(suf)
|
||||
best = int(pool_id)
|
||||
|
||||
if best is not None and int(best) > 0:
|
||||
return int(best)
|
||||
|
||||
if runtime.proxy_default_pool_id is not None and int(runtime.proxy_default_pool_id) > 0:
|
||||
return int(runtime.proxy_default_pool_id)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _weighted_choice(items: list[tuple[int, int]]) -> int | None:
|
||||
if not items:
|
||||
return None
|
||||
total = sum(max(0, int(w)) for _pid, w in items)
|
||||
if total <= 0:
|
||||
items2 = [pid for pid, _w in items]
|
||||
return int(random.choice(items2)) if items2 else None
|
||||
r = random.random() * total
|
||||
for pid, w in items:
|
||||
w_i = max(0, int(w))
|
||||
if w_i <= 0:
|
||||
continue
|
||||
if r < w_i:
|
||||
return int(pid)
|
||||
r -= w_i
|
||||
return int(items[-1][0])
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ProxyUri:
|
||||
uri: str
|
||||
endpoint_id: int
|
||||
pool_id: int
|
||||
|
||||
|
||||
async def _first_enabled_pool_id(engine: AsyncEngine) -> int | None:
|
||||
sql = "SELECT id FROM proxy_pools WHERE enabled=1 ORDER BY id ASC LIMIT 1;"
|
||||
|
||||
async def _op() -> int | None:
|
||||
async with engine.connect() as conn:
|
||||
result = await conn.exec_driver_sql(sql)
|
||||
value = result.scalar_one_or_none()
|
||||
return int(value) if value is not None else None
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
async def _list_enabled_pool_ids(engine: AsyncEngine) -> list[int]:
|
||||
sql = "SELECT id FROM proxy_pools WHERE enabled=1 ORDER BY id ASC;"
|
||||
|
||||
async def _op() -> list[int]:
|
||||
async with engine.connect() as conn:
|
||||
rows = (await conn.exec_driver_sql(sql)).fetchall()
|
||||
out: list[int] = []
|
||||
seen: set[int] = set()
|
||||
for (pid,) in rows:
|
||||
try:
|
||||
pool_id = int(pid)
|
||||
except Exception:
|
||||
continue
|
||||
if pool_id <= 0 or pool_id in seen:
|
||||
continue
|
||||
seen.add(pool_id)
|
||||
out.append(pool_id)
|
||||
return out
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
async def _pool_health_stats(engine: AsyncEngine, *, pool_id: int, now_iso: str) -> dict[str, Any]:
|
||||
sql = """
|
||||
SELECT
|
||||
COUNT(*) AS endpoints_total,
|
||||
COALESCE(SUM(CASE
|
||||
WHEN pe.blacklisted_until IS NULL OR pe.blacklisted_until <= :now THEN 1
|
||||
ELSE 0
|
||||
END), 0) AS endpoints_eligible,
|
||||
MIN(CASE
|
||||
WHEN pe.blacklisted_until > :now THEN pe.blacklisted_until
|
||||
ELSE NULL
|
||||
END) AS next_available_at
|
||||
FROM proxy_pools pp
|
||||
JOIN proxy_pool_endpoints ppe
|
||||
ON ppe.pool_id = pp.id AND ppe.enabled = 1
|
||||
JOIN proxy_endpoints pe
|
||||
ON pe.id = ppe.endpoint_id AND pe.enabled = 1
|
||||
WHERE pp.id = :pool_id AND pp.enabled = 1;
|
||||
""".strip()
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with engine.connect() as conn:
|
||||
row = (await conn.exec_driver_sql(sql, {"pool_id": int(pool_id), "now": str(now_iso)})).first()
|
||||
if row is None:
|
||||
return {"endpoints_total": 0, "endpoints_eligible": 0, "next_available_at": None}
|
||||
total = int(row[0] or 0)
|
||||
eligible = int(row[1] or 0)
|
||||
next_available_at = str(row[2]).strip() if row[2] is not None else None
|
||||
return {
|
||||
"endpoints_total": max(0, total),
|
||||
"endpoints_eligible": max(0, eligible),
|
||||
"next_available_at": next_available_at or None,
|
||||
}
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
async def _pick_endpoint_in_pool(engine: AsyncEngine, *, pool_id: int, now_iso: str) -> tuple[int, str, str, int, str, str] | None:
|
||||
sql = """
|
||||
SELECT pe.id, pe.scheme, pe.host, pe.port, pe.username, pe.password_enc, ppe.weight, pe.last_ok_at, pe.last_fail_at
|
||||
FROM proxy_pools pp
|
||||
JOIN proxy_pool_endpoints ppe
|
||||
ON ppe.pool_id = pp.id AND ppe.enabled = 1
|
||||
JOIN proxy_endpoints pe
|
||||
ON pe.id = ppe.endpoint_id AND pe.enabled = 1
|
||||
WHERE pp.id = :pool_id AND pp.enabled = 1
|
||||
AND (pe.blacklisted_until IS NULL OR pe.blacklisted_until <= :now)
|
||||
ORDER BY pe.id ASC;
|
||||
""".strip()
|
||||
|
||||
async def _op() -> tuple[int, str, str, int, str, str] | None:
|
||||
async with engine.connect() as conn:
|
||||
result = await conn.exec_driver_sql(sql, {"pool_id": int(pool_id), "now": now_iso})
|
||||
rows = result.fetchall()
|
||||
ok_weighted: list[tuple[int, int]] = []
|
||||
unknown_weighted: list[tuple[int, int]] = []
|
||||
fail_weighted: list[tuple[int, int]] = []
|
||||
|
||||
for r in rows:
|
||||
endpoint_id = int(r[0])
|
||||
weight = int(r[6] or 0)
|
||||
last_ok_at = str(r[7]).strip() if r[7] is not None else ""
|
||||
last_fail_at = str(r[8]).strip() if r[8] is not None else ""
|
||||
|
||||
if last_ok_at and (not last_fail_at or last_ok_at >= last_fail_at):
|
||||
ok_weighted.append((endpoint_id, weight))
|
||||
elif not last_ok_at and not last_fail_at:
|
||||
unknown_weighted.append((endpoint_id, weight))
|
||||
else:
|
||||
fail_weighted.append((endpoint_id, weight))
|
||||
|
||||
weighted = ok_weighted or unknown_weighted or fail_weighted
|
||||
chosen_id = _weighted_choice(weighted)
|
||||
if chosen_id is None:
|
||||
return None
|
||||
for r in rows:
|
||||
if int(r[0]) == int(chosen_id):
|
||||
return (
|
||||
int(r[0]),
|
||||
str(r[1]),
|
||||
str(r[2]),
|
||||
int(r[3]),
|
||||
str(r[4] or ""),
|
||||
str(r[5] or ""),
|
||||
)
|
||||
return None
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
async def _load_token_binding(
|
||||
engine: AsyncEngine,
|
||||
*,
|
||||
token_id: int,
|
||||
pool_id: int,
|
||||
) -> tuple[int, int | None, str | None] | None:
|
||||
sql = """
|
||||
SELECT primary_proxy_id, override_proxy_id, override_expires_at
|
||||
FROM token_proxy_bindings
|
||||
WHERE token_id=:token_id AND pool_id=:pool_id
|
||||
LIMIT 1;
|
||||
""".strip()
|
||||
|
||||
async def _op() -> tuple[int, int | None, str | None] | None:
|
||||
async with engine.connect() as conn:
|
||||
result = await conn.exec_driver_sql(sql, {"token_id": int(token_id), "pool_id": int(pool_id)})
|
||||
row = result.first()
|
||||
if row is None:
|
||||
return None
|
||||
primary_id = int(row[0])
|
||||
override_id = int(row[1]) if row[1] is not None else None
|
||||
override_expires_at = str(row[2]) if row[2] is not None else None
|
||||
return primary_id, override_id, override_expires_at
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
async def _load_endpoint_in_pool(
|
||||
engine: AsyncEngine,
|
||||
*,
|
||||
pool_id: int,
|
||||
endpoint_id: int,
|
||||
now_iso: str,
|
||||
) -> tuple[int, str, str, int, str, str] | None:
|
||||
sql = """
|
||||
SELECT pe.id, pe.scheme, pe.host, pe.port, pe.username, pe.password_enc
|
||||
FROM proxy_pools pp
|
||||
JOIN proxy_pool_endpoints ppe
|
||||
ON ppe.pool_id = pp.id AND ppe.enabled = 1
|
||||
JOIN proxy_endpoints pe
|
||||
ON pe.id = ppe.endpoint_id AND pe.enabled = 1
|
||||
WHERE pp.id = :pool_id AND pp.enabled = 1
|
||||
AND pe.id = :endpoint_id
|
||||
AND (pe.blacklisted_until IS NULL OR pe.blacklisted_until <= :now)
|
||||
LIMIT 1;
|
||||
""".strip()
|
||||
|
||||
async def _op() -> tuple[int, str, str, int, str, str] | None:
|
||||
async with engine.connect() as conn:
|
||||
result = await conn.exec_driver_sql(
|
||||
sql,
|
||||
{"pool_id": int(pool_id), "endpoint_id": int(endpoint_id), "now": now_iso},
|
||||
)
|
||||
row = result.first()
|
||||
if row is None:
|
||||
return None
|
||||
return (
|
||||
int(row[0]),
|
||||
str(row[1]),
|
||||
str(row[2]),
|
||||
int(row[3]),
|
||||
str(row[4] or ""),
|
||||
str(row[5] or ""),
|
||||
)
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
def _build_proxy_uri(
|
||||
encryptor: FieldEncryptor | None,
|
||||
*,
|
||||
scheme: str,
|
||||
host: str,
|
||||
port: int,
|
||||
username: str,
|
||||
password_enc: str,
|
||||
) -> str:
|
||||
scheme = (scheme or "").strip().lower()
|
||||
host = (host or "").strip()
|
||||
username = (username or "").strip()
|
||||
password_enc = (password_enc or "").strip()
|
||||
if not scheme or not host or int(port) <= 0:
|
||||
raise ValueError("invalid proxy endpoint")
|
||||
|
||||
password = ""
|
||||
if password_enc:
|
||||
if encryptor is None:
|
||||
raise ValueError("FIELD_ENCRYPTION_KEY is required to decrypt proxy password")
|
||||
password = encryptor.decrypt_text(password_enc) if password_enc else ""
|
||||
|
||||
host_part = host
|
||||
if ":" in host_part and not host_part.startswith("["):
|
||||
host_part = f"[{host_part}]"
|
||||
|
||||
auth = ""
|
||||
if username:
|
||||
user_q = quote(username, safe="")
|
||||
pass_q = quote(password or "", safe="")
|
||||
auth = f"{user_q}:{pass_q}@"
|
||||
|
||||
return f"{scheme}://{auth}{host_part}:{int(port)}"
|
||||
|
||||
|
||||
def _proxy_uri_from_endpoint_row(
|
||||
settings: Settings,
|
||||
*,
|
||||
endpoint_id: int,
|
||||
pool_id: int,
|
||||
scheme: str,
|
||||
host: str,
|
||||
port: int,
|
||||
username: str,
|
||||
password_enc: str,
|
||||
) -> ProxyUri:
|
||||
encryptor = None
|
||||
if str(password_enc or "").strip():
|
||||
try:
|
||||
encryptor = FieldEncryptor.from_key(settings.field_encryption_key)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Encryption not configured", status_code=500) from exc
|
||||
|
||||
try:
|
||||
uri = _build_proxy_uri(
|
||||
encryptor,
|
||||
scheme=scheme,
|
||||
host=host,
|
||||
port=int(port),
|
||||
username=username,
|
||||
password_enc=password_enc,
|
||||
)
|
||||
except ApiError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Invalid proxy endpoint", status_code=500) from exc
|
||||
|
||||
return ProxyUri(uri=uri, endpoint_id=int(endpoint_id), pool_id=int(pool_id))
|
||||
|
||||
|
||||
async def select_proxy_uri_for_url(
|
||||
engine: AsyncEngine,
|
||||
settings: Settings,
|
||||
runtime: RuntimeConfig,
|
||||
*,
|
||||
url: str,
|
||||
now_iso: str | None = None,
|
||||
token_id: int | None = None,
|
||||
) -> ProxyUri | None:
|
||||
host = host_from_url(url)
|
||||
if host is None:
|
||||
return None
|
||||
|
||||
if not should_use_proxy_for_host(runtime, host=host):
|
||||
return None
|
||||
|
||||
now_iso = now_iso or iso_utc_ms()
|
||||
|
||||
preferred_pool_id = resolve_pool_id_for_host(runtime, host=host)
|
||||
enabled_pools = await _list_enabled_pool_ids(engine)
|
||||
enabled_pool_set = {int(pid) for pid in enabled_pools}
|
||||
|
||||
pool_candidates: list[int] = []
|
||||
if preferred_pool_id is not None and int(preferred_pool_id) > 0:
|
||||
preferred_pool_id_i = int(preferred_pool_id)
|
||||
if preferred_pool_id_i in enabled_pool_set:
|
||||
pool_candidates.append(preferred_pool_id_i)
|
||||
|
||||
for pid in enabled_pools:
|
||||
if pid not in pool_candidates:
|
||||
pool_candidates.append(int(pid))
|
||||
|
||||
if not pool_candidates:
|
||||
if bool(runtime.proxy_fail_closed):
|
||||
raise ApiError(
|
||||
code=ErrorCode.PROXY_REQUIRED,
|
||||
message="需要代理,但未配置代理池",
|
||||
status_code=502,
|
||||
details={
|
||||
"reason": "no_proxy_pool_configured",
|
||||
"host": host,
|
||||
"url": url,
|
||||
},
|
||||
)
|
||||
return None
|
||||
|
||||
async def _select_in_pool(pool_id: int) -> ProxyUri | None:
|
||||
if int(pool_id) <= 0:
|
||||
return None
|
||||
|
||||
if token_id is not None and int(token_id) > 0:
|
||||
binding = await _load_token_binding(engine, token_id=int(token_id), pool_id=int(pool_id))
|
||||
if binding is not None:
|
||||
primary_proxy_id, override_proxy_id, override_expires_at = binding
|
||||
override_active = bool(
|
||||
override_proxy_id is not None
|
||||
and override_expires_at
|
||||
and str(override_expires_at) > str(now_iso)
|
||||
)
|
||||
|
||||
candidates: list[int] = []
|
||||
if override_active and override_proxy_id is not None:
|
||||
candidates.append(int(override_proxy_id))
|
||||
candidates.append(int(primary_proxy_id))
|
||||
|
||||
for endpoint_id in candidates:
|
||||
picked_by_binding = await _load_endpoint_in_pool(
|
||||
engine,
|
||||
pool_id=int(pool_id),
|
||||
endpoint_id=int(endpoint_id),
|
||||
now_iso=str(now_iso),
|
||||
)
|
||||
if picked_by_binding is None:
|
||||
continue
|
||||
|
||||
eid, scheme, host_v, port, username, password_enc = picked_by_binding
|
||||
return _proxy_uri_from_endpoint_row(
|
||||
settings,
|
||||
endpoint_id=int(eid),
|
||||
pool_id=int(pool_id),
|
||||
scheme=scheme,
|
||||
host=host_v,
|
||||
port=int(port),
|
||||
username=username,
|
||||
password_enc=password_enc,
|
||||
)
|
||||
|
||||
picked = await _pick_endpoint_in_pool(engine, pool_id=int(pool_id), now_iso=now_iso)
|
||||
if picked is None:
|
||||
return None
|
||||
|
||||
endpoint_id, scheme, host_v, port, username, password_enc = picked
|
||||
return _proxy_uri_from_endpoint_row(
|
||||
settings,
|
||||
endpoint_id=int(endpoint_id),
|
||||
pool_id=int(pool_id),
|
||||
scheme=scheme,
|
||||
host=host_v,
|
||||
port=int(port),
|
||||
username=username,
|
||||
password_enc=password_enc,
|
||||
)
|
||||
|
||||
for candidate_pool_id in pool_candidates:
|
||||
picked = await _select_in_pool(int(candidate_pool_id))
|
||||
if picked is not None:
|
||||
return picked
|
||||
|
||||
if bool(runtime.proxy_fail_closed):
|
||||
pool_stats: list[dict[str, Any]] = []
|
||||
next_available_at: str | None = None
|
||||
total_any = 0
|
||||
eligible_any = 0
|
||||
any_next = False
|
||||
for pid in pool_candidates:
|
||||
stats = await _pool_health_stats(engine, pool_id=int(pid), now_iso=str(now_iso))
|
||||
total = int(stats.get("endpoints_total") or 0)
|
||||
eligible = int(stats.get("endpoints_eligible") or 0)
|
||||
total_any += max(0, total)
|
||||
eligible_any += max(0, eligible)
|
||||
nxt = stats.get("next_available_at")
|
||||
if isinstance(nxt, str) and nxt.strip():
|
||||
any_next = True
|
||||
if next_available_at is None or str(nxt) < str(next_available_at):
|
||||
next_available_at = str(nxt)
|
||||
pool_stats.append({"pool_id": int(pid), **stats})
|
||||
|
||||
reason = "no_healthy_proxy_available"
|
||||
if total_any <= 0:
|
||||
reason = "pool_has_no_endpoints"
|
||||
elif eligible_any <= 0 and any_next and next_available_at:
|
||||
reason = "all_endpoints_blacklisted"
|
||||
|
||||
primary_pool_id = (
|
||||
int(preferred_pool_id)
|
||||
if preferred_pool_id is not None and int(preferred_pool_id) > 0
|
||||
else int(pool_candidates[0])
|
||||
)
|
||||
|
||||
raise ApiError(
|
||||
code=ErrorCode.PROXY_REQUIRED,
|
||||
message="需要代理,但当前没有可用代理节点",
|
||||
status_code=502,
|
||||
details={
|
||||
"reason": reason,
|
||||
"host": host,
|
||||
"url": url,
|
||||
"pool_id": primary_pool_id,
|
||||
"attempted_pool_ids": [int(pid) for pid in pool_candidates],
|
||||
"pools": pool_stats,
|
||||
"endpoints_total": int(total_any),
|
||||
"endpoints_eligible": int(eligible_any),
|
||||
"next_available_at": next_available_at,
|
||||
},
|
||||
)
|
||||
|
||||
return None
|
||||
@@ -0,0 +1,100 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from urllib.parse import unquote
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ProxyUriParts:
|
||||
scheme: str
|
||||
host: str
|
||||
port: int
|
||||
username: str | None
|
||||
password: str | None
|
||||
|
||||
|
||||
_ALLOWED_PROXY_SCHEMES = {"http", "https", "socks4", "socks5"}
|
||||
|
||||
|
||||
def _strip_authority(rest: str) -> str:
|
||||
for sep in ("/", "?", "#"):
|
||||
idx = rest.find(sep)
|
||||
if idx >= 0:
|
||||
return rest[:idx]
|
||||
return rest
|
||||
|
||||
|
||||
def _parse_hostport(hostport: str) -> tuple[str, int]:
|
||||
hostport = hostport.strip()
|
||||
if not hostport:
|
||||
raise ValueError("missing hostport")
|
||||
|
||||
if hostport.startswith("["):
|
||||
rb = hostport.find("]")
|
||||
if rb <= 0:
|
||||
raise ValueError("invalid ipv6 hostport")
|
||||
host = hostport[1:rb].strip()
|
||||
rest = hostport[rb + 1 :].strip()
|
||||
if not rest.startswith(":"):
|
||||
raise ValueError("missing port")
|
||||
port_s = rest[1:].strip()
|
||||
else:
|
||||
if ":" not in hostport:
|
||||
raise ValueError("missing port")
|
||||
host, port_s = hostport.rsplit(":", 1)
|
||||
host = host.strip()
|
||||
port_s = port_s.strip()
|
||||
|
||||
if not host:
|
||||
raise ValueError("missing host")
|
||||
|
||||
try:
|
||||
port = int(port_s)
|
||||
except Exception as exc:
|
||||
raise ValueError("invalid port") from exc
|
||||
if port <= 0 or port > 65535:
|
||||
raise ValueError("invalid port")
|
||||
|
||||
return host, port
|
||||
|
||||
|
||||
def parse_proxy_uri(uri: str) -> ProxyUriParts:
|
||||
uri = uri.strip()
|
||||
if not uri:
|
||||
raise ValueError("uri is required")
|
||||
if "://" not in uri:
|
||||
raise ValueError("invalid uri")
|
||||
|
||||
scheme_raw, rest_raw = uri.split("://", 1)
|
||||
scheme = scheme_raw.strip().lower()
|
||||
if scheme not in _ALLOWED_PROXY_SCHEMES:
|
||||
raise ValueError("unsupported scheme")
|
||||
|
||||
authority = _strip_authority(rest_raw.strip())
|
||||
if not authority:
|
||||
raise ValueError("invalid uri")
|
||||
|
||||
username: str | None = None
|
||||
password: str | None = None
|
||||
|
||||
if "@" in authority:
|
||||
userinfo, hostport = authority.rsplit("@", 1)
|
||||
if ":" not in userinfo:
|
||||
raise ValueError("invalid userinfo")
|
||||
username, password = userinfo.split(":", 1)
|
||||
username = unquote(username.strip())
|
||||
password = unquote(password)
|
||||
if not username:
|
||||
raise ValueError("invalid userinfo")
|
||||
else:
|
||||
hostport = authority
|
||||
|
||||
host, port = _parse_hostport(hostport)
|
||||
|
||||
return ProxyUriParts(
|
||||
scheme=scheme,
|
||||
host=host,
|
||||
port=port,
|
||||
username=username,
|
||||
password=password,
|
||||
)
|
||||
@@ -0,0 +1,83 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from collections import deque
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RandomRequestStatsSnapshot:
|
||||
total_requests: int
|
||||
total_ok: int
|
||||
total_error: int
|
||||
in_flight: int
|
||||
window_seconds: int
|
||||
last_window_requests: int
|
||||
last_window_ok: int
|
||||
last_window_error: int
|
||||
last_window_success_rate: float
|
||||
|
||||
|
||||
class RandomRequestStats:
|
||||
def __init__(self, *, window_seconds: int = 60) -> None:
|
||||
self._window_s = max(1, int(window_seconds))
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
self._total_requests = 0
|
||||
self._total_ok = 0
|
||||
self._total_error = 0
|
||||
self._in_flight = 0
|
||||
|
||||
# (monotonic_seconds, ok)
|
||||
self._events: deque[tuple[float, bool]] = deque()
|
||||
|
||||
def _purge(self, *, now_m: float) -> None:
|
||||
cutoff = float(now_m) - float(self._window_s)
|
||||
while self._events:
|
||||
t, _ok = self._events[0]
|
||||
if float(t) >= cutoff:
|
||||
break
|
||||
self._events.popleft()
|
||||
|
||||
async def on_begin(self) -> float:
|
||||
now_m = float(time.monotonic())
|
||||
async with self._lock:
|
||||
self._total_requests += 1
|
||||
self._in_flight += 1
|
||||
return now_m
|
||||
|
||||
async def on_end(self, *, status_code: int) -> None:
|
||||
now_m = float(time.monotonic())
|
||||
ok = 200 <= int(status_code) < 400
|
||||
|
||||
async with self._lock:
|
||||
self._in_flight = max(0, int(self._in_flight) - 1)
|
||||
if ok:
|
||||
self._total_ok += 1
|
||||
else:
|
||||
self._total_error += 1
|
||||
self._events.append((now_m, bool(ok)))
|
||||
self._purge(now_m=now_m)
|
||||
|
||||
async def snapshot(self) -> RandomRequestStatsSnapshot:
|
||||
now_m = float(time.monotonic())
|
||||
async with self._lock:
|
||||
self._purge(now_m=now_m)
|
||||
last_total = int(len(self._events))
|
||||
last_ok = int(sum(1 for _t, ok in self._events if ok))
|
||||
last_error = int(last_total - last_ok)
|
||||
rate = float(last_ok) / float(last_total) if last_total > 0 else 0.0
|
||||
|
||||
return RandomRequestStatsSnapshot(
|
||||
total_requests=int(self._total_requests),
|
||||
total_ok=int(self._total_ok),
|
||||
total_error=int(self._total_error),
|
||||
in_flight=int(self._in_flight),
|
||||
window_seconds=int(self._window_s),
|
||||
last_window_requests=int(last_total),
|
||||
last_window_ok=int(last_ok),
|
||||
last_window_error=int(last_error),
|
||||
last_window_success_rate=float(rate),
|
||||
)
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
REDACTED = "***"
|
||||
|
||||
_SENSITIVE_KEY_PARTS = (
|
||||
"refresh",
|
||||
"token",
|
||||
"api_key",
|
||||
"apikey",
|
||||
"authorization",
|
||||
"password",
|
||||
"secret",
|
||||
"cookie",
|
||||
)
|
||||
|
||||
_BEARER_RE = re.compile(r"(?i)\bBearer\s+([^\s]+)")
|
||||
_REFRESH_QUERY_RE = re.compile(r"(?i)(refresh_token=)([^&\s]+)")
|
||||
_URI_IN_TEXT_RE = re.compile(r"(?i)(?:https?|socks[45])://[^\s\"']+")
|
||||
|
||||
_TRAILING_PUNCT = ".,);:]}"
|
||||
|
||||
|
||||
def is_sensitive_key(key: str) -> bool:
|
||||
key_l = key.lower()
|
||||
return any(part in key_l for part in _SENSITIVE_KEY_PARTS)
|
||||
|
||||
|
||||
def _redact_single_proxy_uri(text: str) -> str:
|
||||
# Handles both normal and the common \"password contains @\" variant by treating the last '@' as separator.
|
||||
m = re.match(r"^(?P<scheme>https?|socks[45])://(?P<rest>.+)$", text, flags=re.IGNORECASE)
|
||||
if not m:
|
||||
return text
|
||||
|
||||
scheme = m.group("scheme")
|
||||
rest = m.group("rest")
|
||||
if "@" not in rest or ":" not in rest:
|
||||
return text
|
||||
|
||||
userinfo, hostpart = rest.rsplit("@", 1)
|
||||
if ":" not in userinfo:
|
||||
return text
|
||||
|
||||
username, _password = userinfo.split(":", 1)
|
||||
return f"{scheme}://{username}:{REDACTED}@{hostpart}"
|
||||
|
||||
|
||||
def redact_proxy_uri(text: str) -> str:
|
||||
def _strip_trailing_punct(uri: str) -> tuple[str, str]:
|
||||
suffix = ""
|
||||
while uri and uri[-1] in _TRAILING_PUNCT:
|
||||
suffix = uri[-1] + suffix
|
||||
uri = uri[:-1]
|
||||
return uri, suffix
|
||||
|
||||
def _repl(m: re.Match[str]) -> str:
|
||||
full = m.group(0)
|
||||
core, suffix = _strip_trailing_punct(full)
|
||||
return _redact_single_proxy_uri(core) + suffix
|
||||
|
||||
return _URI_IN_TEXT_RE.sub(_repl, text)
|
||||
|
||||
|
||||
def redact_text(text: str) -> str:
|
||||
text = redact_proxy_uri(text)
|
||||
text = _BEARER_RE.sub("Bearer " + REDACTED, text)
|
||||
text = _REFRESH_QUERY_RE.sub(r"\1" + REDACTED, text)
|
||||
return text
|
||||
|
||||
|
||||
def redact_any(value: Any) -> Any:
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, str):
|
||||
return redact_text(value)
|
||||
if isinstance(value, bytes):
|
||||
try:
|
||||
return redact_text(value.decode("utf-8", errors="replace")).encode("utf-8")
|
||||
except Exception:
|
||||
return value
|
||||
if isinstance(value, Mapping):
|
||||
out: dict[Any, Any] = {}
|
||||
for k, v in value.items():
|
||||
if isinstance(k, str) and is_sensitive_key(k):
|
||||
out[k] = REDACTED
|
||||
else:
|
||||
out[k] = redact_any(v)
|
||||
return out
|
||||
if isinstance(value, (list, tuple)):
|
||||
seq = [redact_any(v) for v in value]
|
||||
return type(value)(seq) if isinstance(value, tuple) else seq
|
||||
return value
|
||||
@@ -0,0 +1,67 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import secrets
|
||||
from typing import Any, Mapping
|
||||
|
||||
REQUEST_ID_HEADER = "X-Request-Id"
|
||||
|
||||
|
||||
def new_request_id() -> str:
|
||||
return "req_" + secrets.token_hex(8)
|
||||
|
||||
|
||||
def get_request_id_from_headers(headers: Mapping[str, str] | None) -> str | None:
|
||||
if not headers:
|
||||
return None
|
||||
value = headers.get(REQUEST_ID_HEADER) or headers.get(REQUEST_ID_HEADER.lower())
|
||||
if not value:
|
||||
return None
|
||||
value = value.strip()
|
||||
return value if value else None
|
||||
|
||||
|
||||
def set_request_id_on_state(request: Any, request_id: str) -> None:
|
||||
state = getattr(request, "state", None)
|
||||
if state is None:
|
||||
class _State:
|
||||
pass
|
||||
|
||||
state = _State()
|
||||
setattr(request, "state", state)
|
||||
setattr(state, "request_id", request_id)
|
||||
|
||||
|
||||
def set_request_id_header(response: Any, request_id: str) -> None:
|
||||
headers = getattr(response, "headers", None)
|
||||
if headers is None:
|
||||
setattr(response, "headers", {})
|
||||
headers = response.headers
|
||||
headers[REQUEST_ID_HEADER] = request_id
|
||||
|
||||
|
||||
def get_or_create_request_id(request: Any) -> str:
|
||||
state = getattr(request, "state", None)
|
||||
if state is not None:
|
||||
rid = getattr(state, "request_id", None)
|
||||
if rid:
|
||||
return str(rid)
|
||||
headers = getattr(request, "headers", None)
|
||||
rid = get_request_id_from_headers(headers)
|
||||
return rid or new_request_id()
|
||||
|
||||
|
||||
def build_request_id_middleware() -> Any | None:
|
||||
try:
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
class RequestIdMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request, call_next): # type: ignore[no-untyped-def]
|
||||
rid = get_or_create_request_id(request)
|
||||
set_request_id_on_state(request, rid)
|
||||
response = await call_next(request)
|
||||
set_request_id_header(response, rid)
|
||||
return response
|
||||
|
||||
return RequestIdMiddleware
|
||||
@@ -0,0 +1,205 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import iso_utc_ms
|
||||
from app.db.models.runtime_settings import RuntimeSetting
|
||||
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
def _as_bool(value: Any) -> bool | None:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, int) and value in (0, 1):
|
||||
return bool(value)
|
||||
if isinstance(value, str):
|
||||
v = value.strip().lower()
|
||||
if v in {"true", "1", "yes", "y", "on"}:
|
||||
return True
|
||||
if v in {"false", "0", "no", "n", "off"}:
|
||||
return False
|
||||
return None
|
||||
|
||||
|
||||
def _as_str_list(value: Any) -> list[str]:
|
||||
if not isinstance(value, list):
|
||||
return []
|
||||
out: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for item in value:
|
||||
if not isinstance(item, str):
|
||||
continue
|
||||
v = item.strip()
|
||||
if not v or v in seen:
|
||||
continue
|
||||
seen.add(v)
|
||||
out.append(v)
|
||||
return out
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RuntimeConfig:
|
||||
proxy_enabled: bool
|
||||
proxy_fail_closed: bool
|
||||
proxy_route_mode: str
|
||||
proxy_allowlist_domains: list[str]
|
||||
proxy_route_pools: dict[str, int]
|
||||
proxy_default_pool_id: int | None
|
||||
random_defaults: dict[str, Any]
|
||||
hide_origin_url_in_public_json: bool
|
||||
rate_limit: dict[str, Any]
|
||||
|
||||
@classmethod
|
||||
def defaults(cls) -> "RuntimeConfig":
|
||||
return cls(
|
||||
proxy_enabled=False,
|
||||
proxy_fail_closed=True,
|
||||
proxy_route_mode="pixiv_only",
|
||||
proxy_allowlist_domains=[],
|
||||
proxy_route_pools={},
|
||||
proxy_default_pool_id=None,
|
||||
random_defaults={},
|
||||
hide_origin_url_in_public_json=True,
|
||||
rate_limit={},
|
||||
)
|
||||
|
||||
|
||||
async def fetch_runtime_settings(engine: AsyncEngine) -> dict[str, Any]:
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> dict[str, Any]:
|
||||
async with Session() as session:
|
||||
result = await session.execute(select(RuntimeSetting.key, RuntimeSetting.value_json))
|
||||
values: dict[str, Any] = {}
|
||||
for key, value_json in result.all():
|
||||
try:
|
||||
values[str(key)] = json.loads(value_json)
|
||||
except Exception:
|
||||
log.warning("runtime_settings_invalid_json key=%s", key)
|
||||
return values
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
def runtime_config_from_values(values: dict[str, Any]) -> RuntimeConfig:
|
||||
defaults = RuntimeConfig.defaults()
|
||||
|
||||
proxy_enabled = _as_bool(values.get("proxy.enabled"))
|
||||
proxy_fail_closed = _as_bool(values.get("proxy.fail_closed"))
|
||||
|
||||
proxy_route_mode_raw = values.get("proxy.route_mode")
|
||||
proxy_route_mode = defaults.proxy_route_mode
|
||||
if isinstance(proxy_route_mode_raw, str):
|
||||
candidate = proxy_route_mode_raw.strip().lower()
|
||||
if candidate in {"pixiv_only", "all", "allowlist", "off"}:
|
||||
proxy_route_mode = candidate
|
||||
|
||||
proxy_allowlist_domains = _as_str_list(values.get("proxy.allowlist_domains")) or defaults.proxy_allowlist_domains
|
||||
|
||||
proxy_route_pools_raw = values.get("proxy.route_pools")
|
||||
proxy_route_pools: dict[str, int] = {}
|
||||
if isinstance(proxy_route_pools_raw, dict):
|
||||
for k, v in proxy_route_pools_raw.items():
|
||||
key = str(k or "").strip().lower().strip(".")
|
||||
if not key or len(key) > 200:
|
||||
continue
|
||||
try:
|
||||
pool_id = int(v)
|
||||
except Exception:
|
||||
continue
|
||||
if pool_id <= 0:
|
||||
continue
|
||||
proxy_route_pools[key] = pool_id
|
||||
|
||||
proxy_default_pool_id: int | None = None
|
||||
default_pool_raw = values.get("proxy.default_pool_id")
|
||||
if default_pool_raw is not None:
|
||||
try:
|
||||
candidate = int(default_pool_raw)
|
||||
except Exception:
|
||||
candidate = 0
|
||||
proxy_default_pool_id = candidate if candidate > 0 else None
|
||||
|
||||
random_defaults_raw = values.get("random.defaults")
|
||||
random_defaults = defaults.random_defaults
|
||||
if isinstance(random_defaults_raw, dict):
|
||||
random_defaults = dict(random_defaults_raw)
|
||||
|
||||
hide_origin_url_raw = values.get("security.hide_origin_url_in_public_json")
|
||||
hide_origin_url = _as_bool(hide_origin_url_raw)
|
||||
|
||||
rate_limit: dict[str, Any] = {}
|
||||
for key, value in values.items():
|
||||
if key.startswith("rate_limit."):
|
||||
rate_limit[key.removeprefix("rate_limit.")] = value
|
||||
|
||||
return RuntimeConfig(
|
||||
proxy_enabled=proxy_enabled if proxy_enabled is not None else defaults.proxy_enabled,
|
||||
proxy_fail_closed=proxy_fail_closed if proxy_fail_closed is not None else defaults.proxy_fail_closed,
|
||||
proxy_route_mode=proxy_route_mode,
|
||||
proxy_allowlist_domains=proxy_allowlist_domains,
|
||||
proxy_route_pools=proxy_route_pools,
|
||||
proxy_default_pool_id=proxy_default_pool_id,
|
||||
random_defaults=random_defaults,
|
||||
hide_origin_url_in_public_json=hide_origin_url
|
||||
if hide_origin_url is not None
|
||||
else defaults.hide_origin_url_in_public_json,
|
||||
rate_limit=rate_limit,
|
||||
)
|
||||
|
||||
|
||||
async def load_runtime_config(engine: AsyncEngine) -> RuntimeConfig:
|
||||
values = await fetch_runtime_settings(engine)
|
||||
return runtime_config_from_values(values)
|
||||
|
||||
|
||||
async def set_runtime_setting(
|
||||
engine: AsyncEngine,
|
||||
*,
|
||||
key: str,
|
||||
value: Any,
|
||||
description: str | None = None,
|
||||
updated_by: str | None = None,
|
||||
) -> None:
|
||||
key = (key or "").strip()
|
||||
if not key:
|
||||
raise ValueError("key is required")
|
||||
|
||||
value_json = json.dumps(value, separators=(",", ":"), sort_keys=True, ensure_ascii=False)
|
||||
now = iso_utc_ms()
|
||||
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> None:
|
||||
async with Session() as session:
|
||||
stmt = sqlite_insert(RuntimeSetting).values(
|
||||
key=key,
|
||||
value_json=value_json,
|
||||
description=description,
|
||||
updated_at=now,
|
||||
updated_by=updated_by,
|
||||
)
|
||||
set_values: dict[str, Any] = {
|
||||
"value_json": value_json,
|
||||
"updated_at": now,
|
||||
"updated_by": updated_by,
|
||||
}
|
||||
if description is not None:
|
||||
set_values["description"] = description
|
||||
stmt = stmt.on_conflict_do_update(
|
||||
index_elements=[RuntimeSetting.key],
|
||||
set_=set_values,
|
||||
)
|
||||
await session.execute(stmt)
|
||||
await session.commit()
|
||||
|
||||
await with_sqlite_busy_retry(_op)
|
||||
@@ -0,0 +1,147 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import time
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
from app.core.errors import ApiError, ErrorCode
|
||||
|
||||
JWT_ALG_HS256 = "HS256"
|
||||
JWT_TYP = "JWT"
|
||||
|
||||
|
||||
def _b64url_encode(raw: bytes) -> str:
|
||||
return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii")
|
||||
|
||||
|
||||
def _b64url_decode(data: str) -> bytes:
|
||||
data = data.strip()
|
||||
padding = "=" * (-len(data) % 4)
|
||||
return base64.urlsafe_b64decode(data + padding)
|
||||
|
||||
|
||||
def create_jwt(
|
||||
*,
|
||||
secret_key: str,
|
||||
subject: str,
|
||||
ttl_s: int = 3600,
|
||||
extra_claims: Mapping[str, Any] | None = None,
|
||||
now_s: int | None = None,
|
||||
) -> str:
|
||||
secret_key = secret_key.strip()
|
||||
if not secret_key:
|
||||
raise ValueError("SECRET_KEY is required")
|
||||
|
||||
now_i = int(now_s if now_s is not None else time.time())
|
||||
|
||||
header = {"alg": JWT_ALG_HS256, "typ": JWT_TYP}
|
||||
payload: dict[str, Any] = {"sub": subject, "iat": now_i, "exp": now_i + ttl_s}
|
||||
if extra_claims:
|
||||
payload.update(dict(extra_claims))
|
||||
|
||||
header_b64 = _b64url_encode(json.dumps(header, separators=(",", ":"), sort_keys=True).encode("utf-8"))
|
||||
payload_b64 = _b64url_encode(json.dumps(payload, separators=(",", ":"), sort_keys=True).encode("utf-8"))
|
||||
signing_input = f"{header_b64}.{payload_b64}".encode("ascii")
|
||||
signature = hmac.new(secret_key.encode("utf-8"), signing_input, hashlib.sha256).digest()
|
||||
sig_b64 = _b64url_encode(signature)
|
||||
return f"{header_b64}.{payload_b64}.{sig_b64}"
|
||||
|
||||
|
||||
def decode_jwt(token: str, *, secret_key: str, leeway_s: int = 0, now_s: int | None = None) -> dict[str, Any]:
|
||||
secret_key = secret_key.strip()
|
||||
if not secret_key:
|
||||
raise ValueError("SECRET_KEY is required")
|
||||
|
||||
token = token.strip()
|
||||
parts = token.split(".")
|
||||
if len(parts) != 3:
|
||||
raise ValueError("Invalid token format")
|
||||
|
||||
header_raw = _b64url_decode(parts[0])
|
||||
payload_raw = _b64url_decode(parts[1])
|
||||
signature_raw = _b64url_decode(parts[2])
|
||||
|
||||
try:
|
||||
header = json.loads(header_raw)
|
||||
payload = json.loads(payload_raw)
|
||||
except Exception as exc:
|
||||
raise ValueError("Invalid token JSON") from exc
|
||||
|
||||
if not isinstance(header, dict) or not isinstance(payload, dict):
|
||||
raise ValueError("Invalid token JSON")
|
||||
|
||||
if header.get("alg") != JWT_ALG_HS256 or header.get("typ") != JWT_TYP:
|
||||
raise ValueError("Unsupported token")
|
||||
|
||||
signing_input = f"{parts[0]}.{parts[1]}".encode("ascii")
|
||||
expected_sig = hmac.new(secret_key.encode("utf-8"), signing_input, hashlib.sha256).digest()
|
||||
if not hmac.compare_digest(signature_raw, expected_sig):
|
||||
raise ValueError("Invalid token signature")
|
||||
|
||||
now_i = int(now_s if now_s is not None else time.time())
|
||||
|
||||
exp = payload.get("exp")
|
||||
if exp is not None:
|
||||
try:
|
||||
exp_i = int(exp)
|
||||
except Exception as exc:
|
||||
raise ValueError("Invalid exp") from exc
|
||||
if now_i > exp_i + leeway_s:
|
||||
raise ValueError("Token expired")
|
||||
|
||||
iat = payload.get("iat")
|
||||
if iat is not None:
|
||||
try:
|
||||
iat_i = int(iat)
|
||||
except Exception as exc:
|
||||
raise ValueError("Invalid iat") from exc
|
||||
if iat_i > now_i + leeway_s:
|
||||
raise ValueError("Token issued in the future")
|
||||
|
||||
return payload
|
||||
|
||||
|
||||
def _authorization_from_headers(headers: Mapping[str, str] | None) -> str | None:
|
||||
if not headers:
|
||||
return None
|
||||
return headers.get("Authorization") or headers.get("authorization")
|
||||
|
||||
|
||||
def parse_bearer_token(authorization: str | None) -> str | None:
|
||||
if not authorization:
|
||||
return None
|
||||
parts = authorization.strip().split(None, 1)
|
||||
if len(parts) != 2:
|
||||
return None
|
||||
scheme, token = parts
|
||||
if scheme.lower() != "bearer":
|
||||
return None
|
||||
token = token.strip()
|
||||
return token if token else None
|
||||
|
||||
|
||||
def require_admin(
|
||||
headers: Mapping[str, str] | None,
|
||||
*,
|
||||
secret_key: str,
|
||||
admin_username: str,
|
||||
) -> dict[str, Any]:
|
||||
token = parse_bearer_token(_authorization_from_headers(headers))
|
||||
if not token:
|
||||
raise ApiError(code=ErrorCode.UNAUTHORIZED, message="Missing admin token", status_code=401)
|
||||
|
||||
try:
|
||||
claims = decode_jwt(token, secret_key=secret_key)
|
||||
except Exception as exc:
|
||||
raise ApiError(code=ErrorCode.UNAUTHORIZED, message="Invalid admin token", status_code=401) from exc
|
||||
|
||||
subject = str(claims.get("sub") or "")
|
||||
if subject != admin_username:
|
||||
raise ApiError(code=ErrorCode.FORBIDDEN, message="Forbidden", status_code=403)
|
||||
|
||||
return claims
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
def iso_utc_ms(dt: datetime | None = None) -> str:
|
||||
dt = dt or datetime.now(timezone.utc)
|
||||
dt = dt.astimezone(timezone.utc)
|
||||
ms = dt.microsecond // 1000
|
||||
return dt.strftime("%Y-%m-%dT%H:%M:%S.") + f"{ms:03d}Z"
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.exc import DBAPIError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.models.images import Image
|
||||
from app.db.sqlite_utils import sqlite_fts_phrase_query, sqlite_table_exists
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AuthorListItem:
|
||||
user_id: int
|
||||
user_name: str | None
|
||||
count_images: int
|
||||
|
||||
|
||||
async def list_authors(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
limit: int,
|
||||
cursor: int | None = None,
|
||||
q: str | None = None,
|
||||
) -> tuple[list[AuthorListItem], int | None]:
|
||||
limit_i = int(limit)
|
||||
if limit_i < 1:
|
||||
raise ValueError("limit must be >= 1")
|
||||
|
||||
cursor_i: int | None = None
|
||||
if cursor is not None:
|
||||
cursor_i = int(cursor)
|
||||
if cursor_i <= 0:
|
||||
raise ValueError("cursor must be > 0")
|
||||
|
||||
q_norm = (q or "").strip()
|
||||
use_fts = False
|
||||
fts_q = ""
|
||||
if q_norm and len(q_norm) >= 3:
|
||||
use_fts = await sqlite_table_exists(session, name="authors_fts")
|
||||
if use_fts:
|
||||
fts_q = sqlite_fts_phrase_query(q_norm)
|
||||
|
||||
def _build_stmt(*, use_fts_filter: bool) -> sa.Select:
|
||||
clauses: list[object] = [Image.status == 1, Image.user_id.is_not(None)]
|
||||
|
||||
if cursor_i is not None:
|
||||
clauses.append(Image.user_id > cursor_i)
|
||||
|
||||
if q_norm:
|
||||
if use_fts_filter:
|
||||
fts_user_ids = (
|
||||
sa.text("SELECT rowid AS user_id FROM authors_fts WHERE authors_fts MATCH :q")
|
||||
.bindparams(sa.bindparam("q", fts_q))
|
||||
.columns(user_id=sa.Integer)
|
||||
)
|
||||
fts_user_ids_sq = fts_user_ids.subquery()
|
||||
clauses.append(Image.user_id.in_(sa.select(fts_user_ids_sq.c.user_id)))
|
||||
else:
|
||||
clauses.append(Image.user_name.like(f"%{q_norm}%"))
|
||||
|
||||
return (
|
||||
select(
|
||||
Image.user_id,
|
||||
func.max(Image.user_name).label("user_name"),
|
||||
func.count(Image.id).label("count_images"),
|
||||
)
|
||||
.where(*clauses)
|
||||
.group_by(Image.user_id)
|
||||
.order_by(Image.user_id.asc())
|
||||
.limit(limit_i + 1)
|
||||
)
|
||||
|
||||
stmt = _build_stmt(use_fts_filter=use_fts)
|
||||
try:
|
||||
rows = (await session.execute(stmt)).all()
|
||||
except DBAPIError:
|
||||
if not use_fts:
|
||||
raise
|
||||
stmt = _build_stmt(use_fts_filter=False)
|
||||
rows = (await session.execute(stmt)).all()
|
||||
next_cursor: int | None = None
|
||||
|
||||
if len(rows) > limit_i:
|
||||
rows = rows[:limit_i]
|
||||
next_cursor = int(rows[-1][0] or 0)
|
||||
|
||||
items = [
|
||||
AuthorListItem(
|
||||
user_id=int(row[0]),
|
||||
user_name=str(row[1]) if row[1] is not None else None,
|
||||
count_images=int(row[2] or 0),
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
|
||||
return items, next_cursor
|
||||
@@ -0,0 +1,72 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import event
|
||||
from sqlalchemy.engine.url import make_url
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
|
||||
|
||||
SQLITE_BUSY_TIMEOUT_MS = 30_000
|
||||
SQLITE_POOL_SIZE = 5
|
||||
SQLITE_MAX_OVERFLOW = 0
|
||||
|
||||
|
||||
def apply_sqlite_pragmas(dbapi_connection: Any) -> None:
|
||||
try:
|
||||
busy_timeout_ms = int((os.environ.get("SQLITE_BUSY_TIMEOUT_MS") or str(SQLITE_BUSY_TIMEOUT_MS)).strip() or SQLITE_BUSY_TIMEOUT_MS)
|
||||
except Exception:
|
||||
busy_timeout_ms = int(SQLITE_BUSY_TIMEOUT_MS)
|
||||
busy_timeout_ms = max(1000, min(int(busy_timeout_ms), 5 * 60_000))
|
||||
|
||||
cursor = dbapi_connection.cursor()
|
||||
try:
|
||||
cursor.execute("PRAGMA foreign_keys = ON")
|
||||
cursor.execute("PRAGMA journal_mode = WAL")
|
||||
cursor.fetchone()
|
||||
cursor.execute("PRAGMA synchronous = NORMAL")
|
||||
cursor.execute("PRAGMA temp_store = MEMORY")
|
||||
cursor.execute(f"PRAGMA busy_timeout = {busy_timeout_ms}")
|
||||
finally:
|
||||
cursor.close()
|
||||
|
||||
|
||||
def _is_sqlite_file_url(database_url: str) -> bool:
|
||||
try:
|
||||
url = make_url(database_url)
|
||||
except Exception:
|
||||
return False
|
||||
if (url.get_backend_name() or "").lower() != "sqlite":
|
||||
return False
|
||||
db = str(url.database or "").strip()
|
||||
return bool(db and db != ":memory:")
|
||||
|
||||
|
||||
def create_engine(database_url: str) -> AsyncEngine:
|
||||
kwargs: dict[str, Any] = {}
|
||||
if database_url.lower().startswith("sqlite"):
|
||||
try:
|
||||
busy_timeout_ms = int((os.environ.get("SQLITE_BUSY_TIMEOUT_MS") or str(SQLITE_BUSY_TIMEOUT_MS)).strip() or SQLITE_BUSY_TIMEOUT_MS)
|
||||
except Exception:
|
||||
busy_timeout_ms = int(SQLITE_BUSY_TIMEOUT_MS)
|
||||
busy_timeout_ms = max(1000, min(int(busy_timeout_ms), 5 * 60_000))
|
||||
|
||||
kwargs["connect_args"] = {"timeout": float(busy_timeout_ms) / 1000.0}
|
||||
if _is_sqlite_file_url(database_url):
|
||||
# 限制单进程内同时打开的 SQLite 连接数,减少并发写导致的 "database is locked"。
|
||||
kwargs.update(
|
||||
{
|
||||
"pool_size": int(SQLITE_POOL_SIZE),
|
||||
"max_overflow": int(SQLITE_MAX_OVERFLOW),
|
||||
}
|
||||
)
|
||||
|
||||
engine = create_async_engine(database_url, **kwargs)
|
||||
|
||||
if database_url.lower().startswith("sqlite"):
|
||||
def _on_connect(dbapi_connection: Any, _record: Any) -> None:
|
||||
apply_sqlite_pragmas(dbapi_connection)
|
||||
|
||||
event.listen(engine.sync_engine, "connect", _on_connect)
|
||||
|
||||
return engine
|
||||
@@ -0,0 +1,12 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.models.images import Image
|
||||
|
||||
|
||||
async def get_image_by_id(session: AsyncSession, *, image_id: int) -> Image | None:
|
||||
stmt = select(Image).where(Image.id == int(image_id), Image.status == 1).limit(1)
|
||||
return (await session.execute(stmt)).scalars().first()
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.models.images import Image
|
||||
|
||||
|
||||
async def get_image_by_illust_page(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
illust_id: int,
|
||||
page_index: int,
|
||||
) -> Image | None:
|
||||
stmt = (
|
||||
select(Image)
|
||||
.where(
|
||||
Image.status == 1,
|
||||
Image.illust_id == int(illust_id),
|
||||
Image.page_index == int(page_index),
|
||||
)
|
||||
.limit(1)
|
||||
)
|
||||
return (await session.execute(stmt)).scalars().first()
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.models.images import Image
|
||||
from app.db.random_pick import (
|
||||
_excluded_tags_where_clause,
|
||||
_included_tags_where_clause,
|
||||
_orientation_where_clause,
|
||||
_r18_where_clause,
|
||||
)
|
||||
|
||||
|
||||
async def list_images(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
limit: int,
|
||||
cursor: int | None = None,
|
||||
r18: int = 0,
|
||||
r18_strict: bool = True,
|
||||
orientation: int | None = None,
|
||||
ai_type: int | None = None,
|
||||
min_width: int = 0,
|
||||
min_height: int = 0,
|
||||
min_pixels: int = 0,
|
||||
included_tags: Sequence[str] | None = None,
|
||||
excluded_tags: Sequence[str] | None = None,
|
||||
user_id: int | None = None,
|
||||
illust_id: int | None = None,
|
||||
created_from: str | None = None,
|
||||
created_to: str | None = None,
|
||||
) -> tuple[list[Image], int | None]:
|
||||
limit_i = int(limit)
|
||||
if limit_i < 1:
|
||||
raise ValueError("limit must be >= 1")
|
||||
|
||||
r18_clause = _r18_where_clause(r18=r18, r18_strict=r18_strict)
|
||||
orientation_clause = _orientation_where_clause(orientation=orientation)
|
||||
included_tags_clause = _included_tags_where_clause(tag_names=included_tags or [])
|
||||
excluded_tags_clause = _excluded_tags_where_clause(tag_names=excluded_tags or [])
|
||||
|
||||
min_width_i = int(min_width)
|
||||
min_height_i = int(min_height)
|
||||
min_pixels_i = int(min_pixels)
|
||||
if min_width_i < 0 or min_height_i < 0 or min_pixels_i < 0:
|
||||
raise ValueError("min_* must be >= 0")
|
||||
|
||||
clauses: list[object] = [Image.status == 1]
|
||||
if cursor is not None:
|
||||
clauses.append(Image.id < int(cursor))
|
||||
if r18_clause is not None:
|
||||
clauses.append(r18_clause)
|
||||
if orientation_clause is not None:
|
||||
clauses.append(orientation_clause)
|
||||
if ai_type is not None:
|
||||
ai_type_i = int(ai_type)
|
||||
if ai_type_i not in {0, 1}:
|
||||
raise ValueError("ai_type must be 0, 1, or None")
|
||||
clauses.append(Image.ai_type == ai_type_i)
|
||||
if included_tags_clause is not None:
|
||||
clauses.append(included_tags_clause)
|
||||
if excluded_tags_clause is not None:
|
||||
clauses.append(excluded_tags_clause)
|
||||
if min_width_i > 0:
|
||||
clauses.append(Image.width >= min_width_i)
|
||||
if min_height_i > 0:
|
||||
clauses.append(Image.height >= min_height_i)
|
||||
if min_pixels_i > 0:
|
||||
clauses.append((Image.width * Image.height) >= min_pixels_i)
|
||||
if user_id is not None:
|
||||
clauses.append(Image.user_id == int(user_id))
|
||||
if illust_id is not None:
|
||||
clauses.append(Image.illust_id == int(illust_id))
|
||||
if created_from is not None:
|
||||
clauses.append(Image.created_at_pixiv >= str(created_from))
|
||||
if created_to is not None:
|
||||
clauses.append(Image.created_at_pixiv <= str(created_to))
|
||||
|
||||
stmt = select(Image).where(*clauses).order_by(Image.id.desc()).limit(limit_i + 1)
|
||||
images = (await session.execute(stmt)).scalars().all()
|
||||
|
||||
next_cursor: int | None = None
|
||||
if len(images) > limit_i:
|
||||
images = images[:limit_i]
|
||||
next_cursor = int(images[-1].id)
|
||||
|
||||
return images, next_cursor
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from sqlalchemy import update
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine
|
||||
|
||||
from app.core.redact import redact_text
|
||||
from app.db.models.images import Image
|
||||
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
|
||||
|
||||
|
||||
def _truncate(text: str, *, max_len: int = 500) -> str:
|
||||
if len(text) <= max_len:
|
||||
return text
|
||||
return text[: max_len - 3] + "..."
|
||||
|
||||
|
||||
async def mark_image_failure(
|
||||
engine: AsyncEngine,
|
||||
*,
|
||||
image_id: int,
|
||||
now: str,
|
||||
error_code: str,
|
||||
error_message: str,
|
||||
) -> None:
|
||||
Session = create_sessionmaker(engine)
|
||||
msg = _truncate(redact_text(error_message or ""))
|
||||
|
||||
async def _op() -> None:
|
||||
async with Session() as session:
|
||||
await session.execute(
|
||||
update(Image)
|
||||
.where(Image.id == int(image_id))
|
||||
.values(
|
||||
fail_count=Image.fail_count + 1,
|
||||
last_fail_at=str(now),
|
||||
last_error_code=str(error_code),
|
||||
last_error_msg=str(msg),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
|
||||
await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
async def mark_image_ok(engine: AsyncEngine, *, image_id: int, now: str) -> None:
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
async def _op() -> None:
|
||||
async with Session() as session:
|
||||
await session.execute(
|
||||
update(Image)
|
||||
.where(Image.id == int(image_id))
|
||||
.values(
|
||||
last_ok_at=str(now),
|
||||
last_error_code=None,
|
||||
last_error_msg=None,
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
|
||||
await with_sqlite_busy_retry(_op)
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.models.images import Image
|
||||
|
||||
|
||||
async def upsert_image_by_illust_page(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
illust_id: int,
|
||||
page_index: int,
|
||||
ext: str,
|
||||
original_url: str,
|
||||
proxy_path: str,
|
||||
random_key: float,
|
||||
created_import_id: int | None,
|
||||
) -> int:
|
||||
stmt = sqlite_insert(Image).values(
|
||||
illust_id=illust_id,
|
||||
page_index=page_index,
|
||||
ext=ext,
|
||||
original_url=original_url,
|
||||
proxy_path=proxy_path,
|
||||
random_key=random_key,
|
||||
created_import_id=created_import_id,
|
||||
)
|
||||
|
||||
now_expr = sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))")
|
||||
stmt = stmt.on_conflict_do_update(
|
||||
index_elements=["illust_id", "page_index"],
|
||||
set_={
|
||||
"ext": stmt.excluded.ext,
|
||||
"original_url": stmt.excluded.original_url,
|
||||
"proxy_path": stmt.excluded.proxy_path,
|
||||
"created_import_id": stmt.excluded.created_import_id,
|
||||
"updated_at": now_expr,
|
||||
},
|
||||
).returning(Image.id)
|
||||
|
||||
result = await session.execute(stmt)
|
||||
return int(result.scalar_one())
|
||||
@@ -0,0 +1,32 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class AdminAudit(Base):
|
||||
__tablename__ = "admin_audit"
|
||||
__table_args__ = (
|
||||
sa.Index("idx_admin_audit_created_at", "created_at"),
|
||||
sa.Index("idx_admin_audit_action", "action"),
|
||||
sa.Index("idx_admin_audit_resource", "resource"),
|
||||
sa.Index("idx_admin_audit_record_id", "record_id"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
|
||||
created_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
actor: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
action: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
resource: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
record_id: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
request_id: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
ip: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
user_agent: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
detail_json: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class ApiKey(Base):
|
||||
__tablename__ = "api_keys"
|
||||
__table_args__ = (
|
||||
sa.UniqueConstraint("name", name="uq_api_keys_name"),
|
||||
sa.CheckConstraint("enabled IN (0,1)", name="ck_api_keys_enabled"),
|
||||
sa.Index("idx_api_keys_enabled", "enabled"),
|
||||
sa.Index("idx_api_keys_created_at", "created_at"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
|
||||
created_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
updated_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
|
||||
name: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
description: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
key_hash: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
hint: Mapped[str] = mapped_column(sa.Text(), nullable=False, server_default=sa.text("''"))
|
||||
|
||||
enabled: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
|
||||
last_used_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from sqlalchemy.orm import DeclarativeBase
|
||||
|
||||
|
||||
class Base(DeclarativeBase):
|
||||
pass
|
||||
|
||||
|
||||
# Import all model modules so Base.metadata is fully populated for create_all().
|
||||
# This keeps tests and one-off scripts consistent, regardless of import order.
|
||||
from app.db.models import admin_audit as _admin_audit # noqa: F401,E402
|
||||
from app.db.models import api_keys as _api_keys # noqa: F401,E402
|
||||
from app.db.models import hydration_runs as _hydration_runs # noqa: F401,E402
|
||||
from app.db.models import image_tags as _image_tags # noqa: F401,E402
|
||||
from app.db.models import images as _images # noqa: F401,E402
|
||||
from app.db.models import imports as _imports # noqa: F401,E402
|
||||
from app.db.models import jobs as _jobs # noqa: F401,E402
|
||||
from app.db.models import pixiv_tokens as _pixiv_tokens # noqa: F401,E402
|
||||
from app.db.models import proxy_endpoints as _proxy_endpoints # noqa: F401,E402
|
||||
from app.db.models import proxy_pool_endpoints as _proxy_pool_endpoints # noqa: F401,E402
|
||||
from app.db.models import proxy_pools as _proxy_pools # noqa: F401,E402
|
||||
from app.db.models import request_logs as _request_logs # noqa: F401,E402
|
||||
from app.db.models import runtime_settings as _runtime_settings # noqa: F401,E402
|
||||
from app.db.models import tags as _tags # noqa: F401,E402
|
||||
from app.db.models import token_proxy_bindings as _token_proxy_bindings # noqa: F401,E402
|
||||
@@ -0,0 +1,42 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class HydrationRun(Base):
|
||||
__tablename__ = "hydration_runs"
|
||||
__table_args__ = (
|
||||
sa.CheckConstraint(
|
||||
"status IN ('pending','running','paused','canceled','completed','failed')",
|
||||
name="ck_hr_status",
|
||||
),
|
||||
sa.Index("idx_hr_status_updated", "status", "updated_at"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
|
||||
created_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
updated_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
|
||||
type: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
status: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
criteria_json: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
cursor_json: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
total: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
|
||||
processed: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
|
||||
success: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
|
||||
failed: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
|
||||
started_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
finished_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
last_error: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class ImageTag(Base):
|
||||
__tablename__ = "image_tags"
|
||||
__table_args__ = (sa.Index("idx_image_tags_tag_image", "tag_id", "image_id"),)
|
||||
|
||||
image_id: Mapped[int] = mapped_column(
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("images.id", ondelete="CASCADE"),
|
||||
primary_key=True,
|
||||
)
|
||||
tag_id: Mapped[int] = mapped_column(
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("tags.id", ondelete="CASCADE"),
|
||||
primary_key=True,
|
||||
)
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class Image(Base):
|
||||
__tablename__ = "images"
|
||||
__table_args__ = (
|
||||
sa.UniqueConstraint("illust_id", "page_index", name="uq_images_illust_page"),
|
||||
sa.CheckConstraint("status IN (1,2,3,4)", name="ck_images_status"),
|
||||
sa.CheckConstraint("random_key >= 0.0 AND random_key < 1.0", name="ck_images_random_key"),
|
||||
sa.Index(
|
||||
"idx_images_filter",
|
||||
"status",
|
||||
"x_restrict",
|
||||
"orientation",
|
||||
"width",
|
||||
"height",
|
||||
"random_key",
|
||||
),
|
||||
sa.Index("idx_images_illust_type_random", "status", "illust_type", "random_key"),
|
||||
sa.Index("idx_images_user_random", "status", "user_id", "random_key"),
|
||||
sa.Index("idx_images_created_at_pixiv", "created_at_pixiv"),
|
||||
sa.Index("idx_images_created_import_id", "created_import_id"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
|
||||
|
||||
illust_id: Mapped[int] = mapped_column(sa.Integer(), nullable=False)
|
||||
page_index: Mapped[int] = mapped_column(sa.Integer(), nullable=False)
|
||||
ext: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
|
||||
original_url: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
proxy_path: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
random_key: Mapped[float] = mapped_column(sa.Float(), nullable=False)
|
||||
|
||||
width: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
|
||||
height: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
|
||||
aspect_ratio: Mapped[float | None] = mapped_column(sa.Float(), nullable=True)
|
||||
orientation: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
|
||||
x_restrict: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
|
||||
ai_type: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
|
||||
illust_type: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
|
||||
user_id: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
|
||||
user_name: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
title: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
created_at_pixiv: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
bookmark_count: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
|
||||
view_count: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
|
||||
comment_count: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
|
||||
|
||||
status: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
|
||||
fail_count: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
|
||||
last_fail_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
last_ok_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
last_error_code: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
last_error_msg: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
created_import_id: Mapped[int | None] = mapped_column(
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("imports.id", ondelete="SET NULL"),
|
||||
nullable=True,
|
||||
)
|
||||
|
||||
added_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
updated_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
@@ -0,0 +1,27 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class Import(Base):
|
||||
__tablename__ = "imports"
|
||||
|
||||
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
|
||||
created_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
created_by: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
source: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
total: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
|
||||
accepted: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
|
||||
success: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
|
||||
failed: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
|
||||
|
||||
detail_json: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class JobRow(Base):
|
||||
__tablename__ = "jobs"
|
||||
__table_args__ = (
|
||||
sa.CheckConstraint(
|
||||
"status IN ('pending','running','paused','canceled','completed','failed','dlq')",
|
||||
name="ck_jobs_status",
|
||||
),
|
||||
sa.Index("idx_jobs_status_priority", "status", "priority", "id"),
|
||||
sa.Index("idx_jobs_run_after", "run_after"),
|
||||
sa.Index("idx_jobs_ref", "ref_type", "ref_id"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
|
||||
created_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
updated_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
|
||||
type: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
status: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
priority: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
|
||||
run_after: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
attempt: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
|
||||
max_attempts: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("3"))
|
||||
|
||||
payload_json: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
last_error: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
locked_by: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
locked_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
ref_type: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
ref_id: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class PixivToken(Base):
|
||||
__tablename__ = "pixiv_tokens"
|
||||
__table_args__ = (
|
||||
sa.CheckConstraint("enabled IN (0,1)", name="ck_pixiv_tokens_enabled"),
|
||||
sa.Index("idx_pixiv_tokens_enabled", "enabled"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
|
||||
created_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
updated_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
|
||||
label: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
enabled: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
|
||||
|
||||
refresh_token_enc: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
refresh_token_masked: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
weight: Mapped[float] = mapped_column(sa.Float(), nullable=False, server_default=sa.text("1.0"))
|
||||
|
||||
error_count: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
|
||||
backoff_until: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
last_ok_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
last_fail_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
last_error_code: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
last_error_msg: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class ProxyEndpoint(Base):
|
||||
__tablename__ = "proxy_endpoints"
|
||||
__table_args__ = (
|
||||
sa.UniqueConstraint("scheme", "host", "port", "username", name="uq_proxy_identity"),
|
||||
sa.CheckConstraint("enabled IN (0,1)", name="ck_proxy_enabled"),
|
||||
sa.Index("idx_proxy_endpoints_enabled", "enabled"),
|
||||
sa.Index("idx_proxy_endpoints_source", "source"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
|
||||
created_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
updated_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
|
||||
scheme: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
host: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
port: Mapped[int] = mapped_column(sa.Integer(), nullable=False)
|
||||
username: Mapped[str] = mapped_column(sa.Text(), nullable=False, server_default=sa.text("''"))
|
||||
password_enc: Mapped[str] = mapped_column(sa.Text(), nullable=False, server_default=sa.text("''"))
|
||||
|
||||
enabled: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
|
||||
source: Mapped[str] = mapped_column(sa.Text(), nullable=False, server_default=sa.text("'manual'"))
|
||||
source_ref: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
last_latency_ms: Mapped[float | None] = mapped_column(sa.Float(), nullable=True)
|
||||
last_ok_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
last_fail_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
success_count: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
|
||||
failure_count: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
|
||||
blacklisted_until: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
last_error: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class ProxyPoolEndpoint(Base):
|
||||
__tablename__ = "proxy_pool_endpoints"
|
||||
__table_args__ = (
|
||||
sa.Index("idx_ppe_pool_enabled", "pool_id", "enabled"),
|
||||
sa.Index("idx_ppe_endpoint_pool", "endpoint_id", "pool_id"),
|
||||
sa.CheckConstraint("enabled IN (0,1)", name="ck_ppe_enabled"),
|
||||
)
|
||||
|
||||
pool_id: Mapped[int] = mapped_column(
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("proxy_pools.id", ondelete="CASCADE"),
|
||||
primary_key=True,
|
||||
)
|
||||
endpoint_id: Mapped[int] = mapped_column(
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("proxy_endpoints.id", ondelete="CASCADE"),
|
||||
primary_key=True,
|
||||
)
|
||||
|
||||
enabled: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
|
||||
weight: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
|
||||
|
||||
created_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
updated_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class ProxyPool(Base):
|
||||
__tablename__ = "proxy_pools"
|
||||
__table_args__ = (
|
||||
sa.UniqueConstraint("name", name="uq_proxy_pools_name"),
|
||||
sa.CheckConstraint("enabled IN (0,1)", name="ck_proxy_pools_enabled"),
|
||||
sa.Index("idx_proxy_pools_enabled", "enabled"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
|
||||
created_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
updated_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
|
||||
name: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
description: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
enabled: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class RequestLog(Base):
|
||||
__tablename__ = "request_logs"
|
||||
__table_args__ = (
|
||||
sa.Index("idx_request_logs_created_at", "created_at"),
|
||||
sa.Index("idx_request_logs_route", "route"),
|
||||
sa.Index("idx_request_logs_status", "status"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
|
||||
created_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
request_id: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
method: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
route: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
status: Mapped[int] = mapped_column(sa.Integer(), nullable=False)
|
||||
duration_ms: Mapped[int] = mapped_column(sa.Integer(), nullable=False)
|
||||
ip: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
user_agent: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
sample_rate: Mapped[float | None] = mapped_column(sa.Float(), nullable=True)
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class RuntimeSetting(Base):
|
||||
__tablename__ = "runtime_settings"
|
||||
|
||||
key: Mapped[str] = mapped_column(sa.Text(), primary_key=True)
|
||||
value_json: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
description: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
updated_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
updated_by: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class Tag(Base):
|
||||
__tablename__ = "tags"
|
||||
__table_args__ = (sa.UniqueConstraint("name", name="uq_tags_name"),)
|
||||
|
||||
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
|
||||
name: Mapped[str] = mapped_column(sa.Text(), nullable=False)
|
||||
translated_name: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
created_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
updated_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.models.base import Base
|
||||
|
||||
|
||||
class TokenProxyBinding(Base):
|
||||
__tablename__ = "token_proxy_bindings"
|
||||
__table_args__ = (
|
||||
sa.UniqueConstraint("token_id", "pool_id", name="uq_token_pool"),
|
||||
sa.Index("idx_tpb_pool", "pool_id"),
|
||||
sa.Index("idx_tpb_primary", "primary_proxy_id"),
|
||||
sa.Index("idx_tpb_override", "override_proxy_id"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
|
||||
created_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
updated_at: Mapped[str] = mapped_column(
|
||||
sa.Text(),
|
||||
nullable=False,
|
||||
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
|
||||
)
|
||||
|
||||
token_id: Mapped[int] = mapped_column(
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("pixiv_tokens.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
)
|
||||
pool_id: Mapped[int] = mapped_column(
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("proxy_pools.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
)
|
||||
primary_proxy_id: Mapped[int] = mapped_column(
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("proxy_endpoints.id", ondelete="RESTRICT"),
|
||||
nullable=False,
|
||||
)
|
||||
override_proxy_id: Mapped[int | None] = mapped_column(
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("proxy_endpoints.id", ondelete="SET NULL"),
|
||||
nullable=True,
|
||||
)
|
||||
override_expires_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
|
||||
|
||||
@@ -0,0 +1,366 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from sqlalchemy import distinct, func, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.models.image_tags import ImageTag
|
||||
from app.db.models.images import Image
|
||||
from app.db.models.tags import Tag
|
||||
|
||||
|
||||
def _r18_where_clause(*, r18: int, r18_strict: bool) -> object | None:
|
||||
r18_i = int(r18)
|
||||
if r18_i not in {0, 1, 2}:
|
||||
raise ValueError("r18 must be 0, 1, or 2")
|
||||
|
||||
if r18_i == 2:
|
||||
return None
|
||||
|
||||
if r18_i == 1:
|
||||
return Image.x_restrict == 1
|
||||
|
||||
if bool(r18_strict):
|
||||
return Image.x_restrict == 0
|
||||
|
||||
return (Image.x_restrict == 0) | (Image.x_restrict.is_(None))
|
||||
|
||||
|
||||
def _orientation_where_clause(*, orientation: int | None) -> object | None:
|
||||
if orientation is None:
|
||||
return None
|
||||
orientation_i = int(orientation)
|
||||
if orientation_i not in {1, 2, 3}:
|
||||
raise ValueError("orientation must be 1, 2, or 3")
|
||||
return Image.orientation == orientation_i
|
||||
|
||||
|
||||
def _clean_tag_names(values: Sequence[str] | None) -> list[str]:
|
||||
if not values:
|
||||
return []
|
||||
out: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for v in values:
|
||||
name = str(v or "").strip()
|
||||
if not name:
|
||||
continue
|
||||
if name in seen:
|
||||
continue
|
||||
seen.add(name)
|
||||
out.append(name)
|
||||
return out
|
||||
|
||||
|
||||
def _included_tags_where_clause(*, tag_names: Sequence[str]) -> object | None:
|
||||
names = _clean_tag_names(tag_names)
|
||||
if not names:
|
||||
return None
|
||||
subq = (
|
||||
select(ImageTag.image_id)
|
||||
.join(Tag, Tag.id == ImageTag.tag_id)
|
||||
.where(Tag.name.in_(names))
|
||||
.group_by(ImageTag.image_id)
|
||||
.having(func.count(distinct(Tag.name)) == len(names))
|
||||
)
|
||||
return Image.id.in_(subq)
|
||||
|
||||
|
||||
def _excluded_tags_where_clause(*, tag_names: Sequence[str]) -> object | None:
|
||||
names = _clean_tag_names(tag_names)
|
||||
if not names:
|
||||
return None
|
||||
subq = (
|
||||
select(ImageTag.image_id)
|
||||
.join(Tag, Tag.id == ImageTag.tag_id)
|
||||
.where(Tag.name.in_(names))
|
||||
)
|
||||
return Image.id.not_in(subq)
|
||||
|
||||
|
||||
def _exclude_image_ids_where_clause(*, image_ids: Sequence[int] | None) -> object | None:
|
||||
if not image_ids:
|
||||
return None
|
||||
ids: list[int] = []
|
||||
seen: set[int] = set()
|
||||
for raw in image_ids:
|
||||
try:
|
||||
value = int(raw)
|
||||
except Exception:
|
||||
continue
|
||||
if value <= 0 or value in seen:
|
||||
continue
|
||||
seen.add(value)
|
||||
ids.append(value)
|
||||
if not ids:
|
||||
return None
|
||||
return Image.id.not_in(ids)
|
||||
|
||||
|
||||
async def pick_random_image(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
r: float,
|
||||
r18: int = 0,
|
||||
r18_strict: bool = True,
|
||||
orientation: int | None = None,
|
||||
ai_type: int | None = None,
|
||||
illust_type: int | None = None,
|
||||
min_width: int = 0,
|
||||
min_height: int = 0,
|
||||
min_pixels: int = 0,
|
||||
min_bookmarks: int = 0,
|
||||
min_views: int = 0,
|
||||
min_comments: int = 0,
|
||||
included_tags: Sequence[str] | None = None,
|
||||
excluded_tags: Sequence[str] | None = None,
|
||||
user_id: int | None = None,
|
||||
illust_id: int | None = None,
|
||||
created_from: str | None = None,
|
||||
created_to: str | None = None,
|
||||
exclude_image_ids: Sequence[int] | None = None,
|
||||
fail_cooldown_before: str | None = None,
|
||||
) -> Image | None:
|
||||
r = float(r)
|
||||
if r < 0.0:
|
||||
r = 0.0
|
||||
if r >= 1.0:
|
||||
r = 0.999999999
|
||||
|
||||
r18_clause = _r18_where_clause(r18=r18, r18_strict=r18_strict)
|
||||
orientation_clause = _orientation_where_clause(orientation=orientation)
|
||||
included_tags_clause = _included_tags_where_clause(tag_names=included_tags or [])
|
||||
excluded_tags_clause = _excluded_tags_where_clause(tag_names=excluded_tags or [])
|
||||
exclude_ids_clause = _exclude_image_ids_where_clause(image_ids=exclude_image_ids)
|
||||
|
||||
min_width_i = int(min_width)
|
||||
min_height_i = int(min_height)
|
||||
min_pixels_i = int(min_pixels)
|
||||
min_bookmarks_i = int(min_bookmarks)
|
||||
min_views_i = int(min_views)
|
||||
min_comments_i = int(min_comments)
|
||||
if min_width_i < 0 or min_height_i < 0 or min_pixels_i < 0:
|
||||
raise ValueError("min_* must be >= 0")
|
||||
if min_bookmarks_i < 0 or min_views_i < 0 or min_comments_i < 0:
|
||||
raise ValueError("min_* must be >= 0")
|
||||
|
||||
clauses = [Image.status == 1]
|
||||
if r18_clause is not None:
|
||||
clauses.append(r18_clause)
|
||||
if orientation_clause is not None:
|
||||
clauses.append(orientation_clause)
|
||||
if ai_type is not None:
|
||||
ai_type_i = int(ai_type)
|
||||
if ai_type_i not in {0, 1}:
|
||||
raise ValueError("ai_type must be 0, 1, or None")
|
||||
clauses.append(Image.ai_type == ai_type_i)
|
||||
if illust_type is not None:
|
||||
illust_type_i = int(illust_type)
|
||||
if illust_type_i not in {0, 1, 2}:
|
||||
raise ValueError("illust_type must be 0, 1, 2, or None")
|
||||
clauses.append(Image.illust_type == illust_type_i)
|
||||
if included_tags_clause is not None:
|
||||
clauses.append(included_tags_clause)
|
||||
if excluded_tags_clause is not None:
|
||||
clauses.append(excluded_tags_clause)
|
||||
if exclude_ids_clause is not None:
|
||||
clauses.append(exclude_ids_clause)
|
||||
if min_width_i > 0:
|
||||
clauses.append(Image.width >= min_width_i)
|
||||
if min_height_i > 0:
|
||||
clauses.append(Image.height >= min_height_i)
|
||||
if min_pixels_i > 0:
|
||||
clauses.append((Image.width * Image.height) >= min_pixels_i)
|
||||
if min_bookmarks_i > 0:
|
||||
clauses.append(Image.bookmark_count >= min_bookmarks_i)
|
||||
if min_views_i > 0:
|
||||
clauses.append(Image.view_count >= min_views_i)
|
||||
if min_comments_i > 0:
|
||||
clauses.append(Image.comment_count >= min_comments_i)
|
||||
if user_id is not None:
|
||||
clauses.append(Image.user_id == int(user_id))
|
||||
if illust_id is not None:
|
||||
clauses.append(Image.illust_id == int(illust_id))
|
||||
if created_from is not None:
|
||||
clauses.append(Image.created_at_pixiv >= str(created_from))
|
||||
if created_to is not None:
|
||||
clauses.append(Image.created_at_pixiv <= str(created_to))
|
||||
if fail_cooldown_before is not None:
|
||||
clauses.append((Image.last_fail_at.is_(None)) | (Image.last_fail_at <= str(fail_cooldown_before)))
|
||||
|
||||
stmt = (
|
||||
select(Image)
|
||||
.where(*clauses, Image.random_key >= r)
|
||||
.order_by(Image.random_key.asc())
|
||||
.limit(1)
|
||||
)
|
||||
image = (await session.execute(stmt)).scalars().first()
|
||||
if image is not None:
|
||||
return image
|
||||
|
||||
stmt2 = (
|
||||
select(Image)
|
||||
.where(*clauses)
|
||||
.order_by(Image.random_key.asc())
|
||||
.limit(1)
|
||||
)
|
||||
return (await session.execute(stmt2)).scalars().first()
|
||||
|
||||
|
||||
def _allowed_int_or_null_clause(column: object, *, allowed: set[int | None]) -> object | None:
|
||||
allowed_norm: set[int | None] = set(allowed)
|
||||
ints = sorted({int(v) for v in allowed_norm if v is not None})
|
||||
has_null = None in allowed_norm
|
||||
|
||||
if ints and has_null and set(ints) in ({0, 1}, {0, 1, 2}):
|
||||
return None
|
||||
|
||||
if ints and has_null:
|
||||
return or_(column.in_(ints), column.is_(None)) # type: ignore[attr-defined]
|
||||
if ints:
|
||||
if len(ints) == 1:
|
||||
return column == ints[0] # type: ignore[operator]
|
||||
return column.in_(ints) # type: ignore[attr-defined]
|
||||
if has_null:
|
||||
return column.is_(None) # type: ignore[attr-defined]
|
||||
return None
|
||||
|
||||
|
||||
async def pick_random_images(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
r: float,
|
||||
limit: int,
|
||||
r18: int = 0,
|
||||
r18_strict: bool = True,
|
||||
orientation: int | None = None,
|
||||
ai_type: int | None = None,
|
||||
illust_type: int | None = None,
|
||||
ai_type_allowed: set[int | None] | None = None,
|
||||
illust_type_allowed: set[int | None] | None = None,
|
||||
min_width: int = 0,
|
||||
min_height: int = 0,
|
||||
min_pixels: int = 0,
|
||||
min_bookmarks: int = 0,
|
||||
min_views: int = 0,
|
||||
min_comments: int = 0,
|
||||
included_tags: Sequence[str] | None = None,
|
||||
excluded_tags: Sequence[str] | None = None,
|
||||
user_id: int | None = None,
|
||||
illust_id: int | None = None,
|
||||
created_from: str | None = None,
|
||||
created_to: str | None = None,
|
||||
exclude_image_ids: Sequence[int] | None = None,
|
||||
fail_cooldown_before: str | None = None,
|
||||
) -> list[Image]:
|
||||
limit_i = int(limit)
|
||||
if limit_i <= 0:
|
||||
return []
|
||||
if limit_i > 5000:
|
||||
limit_i = 5000
|
||||
|
||||
r = float(r)
|
||||
if r < 0.0:
|
||||
r = 0.0
|
||||
if r >= 1.0:
|
||||
r = 0.999999999
|
||||
|
||||
r18_clause = _r18_where_clause(r18=r18, r18_strict=r18_strict)
|
||||
orientation_clause = _orientation_where_clause(orientation=orientation)
|
||||
included_tags_clause = _included_tags_where_clause(tag_names=included_tags or [])
|
||||
excluded_tags_clause = _excluded_tags_where_clause(tag_names=excluded_tags or [])
|
||||
exclude_ids_clause = _exclude_image_ids_where_clause(image_ids=exclude_image_ids)
|
||||
|
||||
min_width_i = int(min_width)
|
||||
min_height_i = int(min_height)
|
||||
min_pixels_i = int(min_pixels)
|
||||
min_bookmarks_i = int(min_bookmarks)
|
||||
min_views_i = int(min_views)
|
||||
min_comments_i = int(min_comments)
|
||||
if min_width_i < 0 or min_height_i < 0 or min_pixels_i < 0:
|
||||
raise ValueError("min_* must be >= 0")
|
||||
if min_bookmarks_i < 0 or min_views_i < 0 or min_comments_i < 0:
|
||||
raise ValueError("min_* must be >= 0")
|
||||
|
||||
clauses = [Image.status == 1]
|
||||
if r18_clause is not None:
|
||||
clauses.append(r18_clause)
|
||||
if orientation_clause is not None:
|
||||
clauses.append(orientation_clause)
|
||||
if ai_type is not None:
|
||||
ai_type_i = int(ai_type)
|
||||
if ai_type_i not in {0, 1}:
|
||||
raise ValueError("ai_type must be 0, 1, or None")
|
||||
clauses.append(Image.ai_type == ai_type_i)
|
||||
if illust_type is not None:
|
||||
illust_type_i = int(illust_type)
|
||||
if illust_type_i not in {0, 1, 2}:
|
||||
raise ValueError("illust_type must be 0, 1, 2, or None")
|
||||
clauses.append(Image.illust_type == illust_type_i)
|
||||
if ai_type_allowed is not None:
|
||||
allowed = set(ai_type_allowed)
|
||||
if not allowed:
|
||||
return []
|
||||
clause = _allowed_int_or_null_clause(Image.ai_type, allowed=allowed)
|
||||
if clause is not None:
|
||||
clauses.append(clause)
|
||||
if illust_type_allowed is not None:
|
||||
allowed = set(illust_type_allowed)
|
||||
if not allowed:
|
||||
return []
|
||||
clause = _allowed_int_or_null_clause(Image.illust_type, allowed=allowed)
|
||||
if clause is not None:
|
||||
clauses.append(clause)
|
||||
if included_tags_clause is not None:
|
||||
clauses.append(included_tags_clause)
|
||||
if excluded_tags_clause is not None:
|
||||
clauses.append(excluded_tags_clause)
|
||||
if exclude_ids_clause is not None:
|
||||
clauses.append(exclude_ids_clause)
|
||||
if min_width_i > 0:
|
||||
clauses.append(Image.width >= min_width_i)
|
||||
if min_height_i > 0:
|
||||
clauses.append(Image.height >= min_height_i)
|
||||
if min_pixels_i > 0:
|
||||
clauses.append((Image.width * Image.height) >= min_pixels_i)
|
||||
if min_bookmarks_i > 0:
|
||||
clauses.append(Image.bookmark_count >= min_bookmarks_i)
|
||||
if min_views_i > 0:
|
||||
clauses.append(Image.view_count >= min_views_i)
|
||||
if min_comments_i > 0:
|
||||
clauses.append(Image.comment_count >= min_comments_i)
|
||||
if user_id is not None:
|
||||
clauses.append(Image.user_id == int(user_id))
|
||||
if illust_id is not None:
|
||||
clauses.append(Image.illust_id == int(illust_id))
|
||||
if created_from is not None:
|
||||
clauses.append(Image.created_at_pixiv >= str(created_from))
|
||||
if created_to is not None:
|
||||
clauses.append(Image.created_at_pixiv <= str(created_to))
|
||||
if fail_cooldown_before is not None:
|
||||
clauses.append((Image.last_fail_at.is_(None)) | (Image.last_fail_at <= str(fail_cooldown_before)))
|
||||
|
||||
stmt = (
|
||||
select(Image)
|
||||
.where(*clauses, Image.random_key >= r)
|
||||
.order_by(Image.random_key.asc())
|
||||
.limit(limit_i)
|
||||
)
|
||||
items = (await session.execute(stmt)).scalars().all()
|
||||
if len(items) >= limit_i:
|
||||
return list(items)
|
||||
|
||||
remain = int(limit_i - len(items))
|
||||
if remain <= 0:
|
||||
return list(items)
|
||||
|
||||
stmt2 = (
|
||||
select(Image)
|
||||
.where(*clauses, Image.random_key < r)
|
||||
.order_by(Image.random_key.asc())
|
||||
.limit(remain)
|
||||
)
|
||||
more = (await session.execute(stmt2)).scalars().all()
|
||||
if not more:
|
||||
return list(items)
|
||||
return list(items) + list(more)
|
||||
@@ -0,0 +1,130 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine
|
||||
|
||||
from app.core.time import iso_utc_ms
|
||||
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
|
||||
|
||||
DEFAULT_REQUEST_LOGS_KEEP_DAYS = 30
|
||||
DEFAULT_REQUEST_LOGS_MAX_DELETE_ROWS = 50_000
|
||||
DEFAULT_REQUEST_LOGS_CHUNK_SIZE = 1_000
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RequestLogsCleanupResult:
|
||||
cutoff: str
|
||||
deleted: int
|
||||
has_more: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RequestLogsCleanupPreview:
|
||||
cutoff: str
|
||||
would_delete: int
|
||||
has_more: bool
|
||||
|
||||
|
||||
def _clamp_int(value: int, *, min_v: int, max_v: int) -> int:
|
||||
return max(min_v, min(int(value), max_v))
|
||||
|
||||
|
||||
def _cutoff_iso(*, keep_days: int, now: datetime | None = None) -> str:
|
||||
keep_days_i = int(keep_days)
|
||||
if keep_days_i < 0:
|
||||
raise ValueError("keep_days must be >= 0")
|
||||
|
||||
now_dt = now or datetime.now(timezone.utc)
|
||||
cutoff_dt = now_dt - timedelta(days=keep_days_i)
|
||||
return iso_utc_ms(cutoff_dt)
|
||||
|
||||
|
||||
async def preview_request_logs_cleanup(
|
||||
engine: AsyncEngine,
|
||||
*,
|
||||
keep_days: int = DEFAULT_REQUEST_LOGS_KEEP_DAYS,
|
||||
max_delete_rows: int = DEFAULT_REQUEST_LOGS_MAX_DELETE_ROWS,
|
||||
now: datetime | None = None,
|
||||
) -> RequestLogsCleanupPreview:
|
||||
cutoff = _cutoff_iso(keep_days=int(keep_days), now=now)
|
||||
max_delete_rows_i = _clamp_int(int(max_delete_rows), min_v=1, max_v=10_000_000)
|
||||
|
||||
Session = create_sessionmaker(engine)
|
||||
sql = """
|
||||
SELECT id
|
||||
FROM request_logs
|
||||
WHERE created_at < :cutoff
|
||||
ORDER BY created_at ASC
|
||||
LIMIT :limit;
|
||||
""".strip()
|
||||
|
||||
async def _op() -> RequestLogsCleanupPreview:
|
||||
async with Session() as session:
|
||||
res = await session.execute(sa.text(sql), {"cutoff": cutoff, "limit": int(max_delete_rows_i) + 1})
|
||||
rows = res.all()
|
||||
would_delete = min(len(rows), int(max_delete_rows_i))
|
||||
has_more = len(rows) > int(max_delete_rows_i)
|
||||
return RequestLogsCleanupPreview(cutoff=cutoff, would_delete=would_delete, has_more=has_more)
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
|
||||
async def cleanup_request_logs(
|
||||
engine: AsyncEngine,
|
||||
*,
|
||||
keep_days: int = DEFAULT_REQUEST_LOGS_KEEP_DAYS,
|
||||
max_delete_rows: int = DEFAULT_REQUEST_LOGS_MAX_DELETE_ROWS,
|
||||
chunk_size: int = DEFAULT_REQUEST_LOGS_CHUNK_SIZE,
|
||||
now: datetime | None = None,
|
||||
) -> RequestLogsCleanupResult:
|
||||
cutoff = _cutoff_iso(keep_days=int(keep_days), now=now)
|
||||
max_delete_rows_i = _clamp_int(int(max_delete_rows), min_v=1, max_v=10_000_000)
|
||||
chunk_size_i = _clamp_int(int(chunk_size), min_v=1, max_v=100_000)
|
||||
chunk_size_i = min(int(chunk_size_i), int(max_delete_rows_i))
|
||||
|
||||
Session = create_sessionmaker(engine)
|
||||
|
||||
delete_sql = """
|
||||
DELETE FROM request_logs
|
||||
WHERE id IN (
|
||||
SELECT id
|
||||
FROM request_logs
|
||||
WHERE created_at < :cutoff
|
||||
ORDER BY created_at ASC
|
||||
LIMIT :chunk
|
||||
);
|
||||
""".strip()
|
||||
|
||||
has_more_sql = """
|
||||
SELECT 1
|
||||
FROM request_logs
|
||||
WHERE created_at < :cutoff
|
||||
LIMIT 1;
|
||||
""".strip()
|
||||
|
||||
async def _op() -> RequestLogsCleanupResult:
|
||||
deleted = 0
|
||||
has_more = False
|
||||
|
||||
async with Session() as session:
|
||||
while deleted < int(max_delete_rows_i):
|
||||
chunk = min(int(chunk_size_i), int(max_delete_rows_i) - int(deleted))
|
||||
result = await session.execute(sa.text(delete_sql), {"cutoff": cutoff, "chunk": int(chunk)})
|
||||
n = int(result.rowcount or 0)
|
||||
if n <= 0:
|
||||
await session.commit()
|
||||
break
|
||||
deleted += n
|
||||
await session.commit()
|
||||
|
||||
if deleted >= int(max_delete_rows_i):
|
||||
res = await session.execute(sa.text(has_more_sql), {"cutoff": cutoff})
|
||||
has_more = res.first() is not None
|
||||
|
||||
return RequestLogsCleanupResult(cutoff=cutoff, deleted=int(deleted), has_more=bool(has_more))
|
||||
|
||||
return await with_sqlite_busy_retry(_op)
|
||||
|
||||
Loaded 100 of 309 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user