整理代码并提交

This commit is contained in:
inliver233 committed 2026-02-22 18:58:55 +08:00
commit 968f37ad94
309 files changed
+46134

No files matched your search

+24
View File
@@ -0,0 +1,24 @@
.git
.github
.codex
.venv
.pytest_cache
**/__pycache__/
# Env / secrets
.env
.env.*
!deploy/.env.example
data/
issues/
plan/
docs/
参考资料/
frontend/node_modules/
frontend/dist/
backend/.pytest_cache/
_tmp_append_test.md
+74
View File
@@ -0,0 +1,74 @@
name: ci
on:
push:
branches:
- main
- test
pull_request:
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
jobs:
backend-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: pip
cache-dependency-path: requirements-dev.txt
- name: Install dependencies
run: python -m pip install -r requirements-dev.txt
- name: Compile
run: python -m compileall -q backend
- name: Tests
run: python -m pytest -q backend/tests
frontend-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
cache: npm
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
run: npm ci
working-directory: frontend
- name: Lint
run: npm run lint
working-directory: frontend
- name: Typecheck
run: npm run typecheck
working-directory: frontend
- name: Tests
run: npm test
working-directory: frontend
compose-config:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Prepare deploy env file
run: |
if [ ! -f deploy/.env ]; then
cp deploy/.env.example deploy/.env
fi
- name: Validate docker compose config
run: docker compose -f deploy/docker-compose.yml config > /dev/null
+68
View File
@@ -0,0 +1,68 @@
## Python
__pycache__/
*.py[cod]
*$py.class
.pytest_cache/
.mypy_cache/
.ruff_cache/
*.egg-info/
.coverage
.coverage.*
htmlcov/
coverage.xml
## Virtualenv
.venv/
venv/
ENV/
env/
## Node
node_modules/
dist/
.vite/
## Runtime data / local state (never commit)
data/
backend/data/
deploy/.env
.env
.env.*
!.env.example
!deploy/.env.example
*.db
*.sqlite
*.sqlite3
## Codex / AI tooling (repo-local)
.codex/
plan/
issues/
docs/
参考资料/
## Data exports / plans
*.csv
*.tsv
*.plan.md
## OS / IDE
.DS_Store
Thumbs.db
desktop.ini
.idea/
.vscode/
*.iml
## Local temp / scratch
*.log
*.tmp
要求.txt
_tmp_easy_export.txt
_tmp_*.md
_tmp_*.txt
## Keep public-facing docs at repo root
/*.md
!README.md
!LICENSE*
+122
View File
@@ -0,0 +1,122 @@
# Random Mage (new-pixiv-api)
一个自托管的随机图片 API:把 Pixiv 原图链接导入到本地数据库后,即可通过 `/random` 按 **标签 / 热度 / 分辨率 / R18 / AI / 作品类型** 等条件组合筛选并随机出图;同时提供管理后台(Web UI)+ 后台 Worker,用于导入、补全元数据、代理探测等任务。
> 免责声明:本项目为非官方实现;请遵守 Pixiv ToS/版权与当地法律法规,自行承担使用风险。不要将任何 refresh_token / 密钥 / 代理密码提交到仓库。
## 功能概览
- 公共 API
- `GET /random`:默认直接返回图片流;也可 `format=json|simple_json` 返回 JSON;支持大量筛选参数
- `GET /images` / `GET /images/{id}`:按条件分页查看已入库图片
- `GET /i/{image_id}.{ext}`:本地代理/缓存路径(`/random?redirect=1` 会跳转到这里)
- `GET /tags`、`GET /authors`、`GET /healthz`、`GET /version`、`GET /docs`(人类可读文档页)
- Swagger / OpenAPI:`GET /api/docs`、`GET /openapi.json`
- 管理后台(`/admin`)
- 登录后可管理:Pixiv refresh_token、代理池、导入、任务队列、运行时开关、审计/统计等
- Worker(后台任务)
- 导入图片 URL、补全元数据(tags/宽高/浏览收藏等)、修复失效 URL、代理探测等
- 可选能力
- 公网访问保护:启用 Public API Key + 限流
- `imgproxy`:对外提供签名处理 URL(可隐藏 origin URL)
- 代理路由:为上游图片/接口选择代理
## 一键部署(Docker Compose,推荐)
前置:安装 Docker + Docker Compose。
1) 准备环境变量
```bash
cp deploy/.env.example deploy/.env
```
然后编辑 `deploy/.env`,至少建议修改:
- `SECRET_KEY`:JWT/API Key 哈希用(生产环境必须修改)
- `ADMIN_PASSWORD`:后台密码(生产环境必须修改)
- `FIELD_ENCRYPTION_KEY`:用于加密保存 refresh_token(生产环境必须提供;建议用 secret 挂载或文件方式)
2) 启动
```bash
docker compose -f deploy/docker-compose.yml up -d --build
```
3) 访问
- API 基址:`http://localhost:23222`
- 健康检查:`GET http://localhost:23222/healthz`
- 文档页:`GET http://localhost:23222/docs`
- 后台 UI:`http://localhost:23222/admin`
- Swagger:`http://localhost:23222/api/docs`
## 首次使用(导入数据 + 补全元数据)
这个项目不会“自动替你抓全站图片”,你需要先把图片记录导入数据库。
1) 登录后台
- 打开 `http://localhost:23222/admin`
- 使用 `deploy/.env` 中的 `ADMIN_USERNAME` / `ADMIN_PASSWORD` 登录
2) 导入图片原图 URL
后台提供“导入”入口,支持:
- 直接粘贴多行文本(每行一个 URL)
- 上传 `.txt` 文件(同样每行一个 URL)
导入内容应为 Pixiv 原图链接(例如 `i.pximg.net/.../img-original/...` 这类)。导入时会自动去重(按 illust_id + page)。
3) 配置 Pixiv refresh_token(可选,但强烈建议)
若你希望 Worker 自动补全/刷新元数据(tags、宽高、浏览/收藏/评论、R18 标记、AI 类型等),需要:
- 在后台添加一个或多个 refresh_token
- 配置 `PIXIV_OAUTH_CLIENT_ID` / `PIXIV_OAUTH_CLIENT_SECRET`(以及可选的 `PIXIV_OAUTH_HASH_SECRET`)
> 安全提示:refresh_token 属于敏感凭据;请只放在运行环境里(`deploy/.env` / secret 管理),不要写进代码/仓库。
4) 确认 Worker 正常
`/healthz` 会同时检查 DB、队列、以及 Worker 心跳;如果 `worker_ok=false`,通常意味着 Worker 没启动或无法连接数据库卷。
## 配置说明(节选)
所有配置均通过环境变量注入(见 `deploy/.env.example`):
- `APP_ENV`:`dev` / `prod`(`prod` 会启用更严格的必填校验)
- `DATABASE_URL`:默认 SQLite(容器内 `/app/data/app.db`,通过 `../data:/app/data` 挂载)
- `PUBLIC_API_KEY_REQUIRED` / `PUBLIC_API_KEY_RPM` / `PUBLIC_API_KEY_BURST`:公网访问保护(可选)
- `WORKER_*`、`IMPORT_*`、`SQLITE_BUSY_TIMEOUT_MS`:Worker/导入/SQLite 运行参数
- `IMGPROXY_*`:imgproxy 集成(可选)
## 本地开发(可选)
后端(Python 3.11):
```bash
cd backend
python -m venv .venv
.\.venv\Scripts\activate
python -m pip install -r ..\requirements-dev.txt
alembic upgrade head
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
```
前端管理台(Node 20):
```bash
cd frontend
npm ci
npm run dev
```
如需让前端直连本地后端,设置 `VITE_API_BASE_URL`(例如 `http://localhost:8000`)。
## 运行测试
- 后端:`python -m pytest -q backend/tests`
- 前端:`cd frontend && npm test`
+33
View File
@@ -0,0 +1,33 @@
FROM node:20-slim AS frontend-build
WORKDIR /frontend
COPY frontend/package.json frontend/package-lock.json /frontend/
RUN npm ci --no-audit --no-fund
COPY frontend/ /frontend/
RUN npm run build
FROM python:3.11-slim
WORKDIR /app
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1
RUN python -m pip install --no-cache-dir --upgrade pip
COPY requirements.txt /app/requirements.txt
RUN python -m pip install --no-cache-dir -r /app/requirements.txt
RUN mkdir -p /app/data /app/web
COPY backend/app /app/app
COPY backend/alembic.ini /app/alembic.ini
COPY backend/alembic /app/alembic
COPY --from=frontend-build /frontend/dist /app/web/dist
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
+38
View File
@@ -0,0 +1,38 @@
[alembic]
script_location = %(here)s/alembic
prepend_sys_path = %(here)s
sqlalchemy.url = sqlite:///./data/app.db
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
+82
View File
@@ -0,0 +1,82 @@
from __future__ import annotations
import os
import sys
from logging.config import fileConfig
from pathlib import Path
from alembic import context
from sqlalchemy import engine_from_config, pool
from sqlalchemy.engine.url import make_url
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
BACKEND_DIR = Path(__file__).resolve().parents[1]
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
target_metadata = None
def _get_database_url() -> str:
url = (os.environ.get("DATABASE_URL") or config.get_main_option("sqlalchemy.url") or "").strip()
if not url:
raise RuntimeError("DATABASE_URL (or sqlalchemy.url) is required")
return url.replace("+aiosqlite", "")
def _ensure_sqlite_dir(url: str) -> None:
parsed = make_url(url)
if parsed.get_backend_name() != "sqlite":
return
db_path = parsed.database
if not db_path or db_path == ":memory:":
return
Path(db_path).expanduser().resolve().parent.mkdir(parents=True, exist_ok=True)
def run_migrations_offline() -> None:
url = _get_database_url()
_ensure_sqlite_dir(url)
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
render_as_batch=url.startswith("sqlite"),
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online() -> None:
url = _get_database_url()
_ensure_sqlite_dir(url)
connectable = engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
url=url,
poolclass=pool.NullPool,
)
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
render_as_batch=url.startswith("sqlite"),
)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
+1
View File
@@ -0,0 +1 @@
@@ -0,0 +1,36 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0001"
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"imports",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column(
"created_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column("created_by", sa.Text(), nullable=True),
sa.Column("source", sa.Text(), nullable=True),
sa.Column("total", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("accepted", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("success", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("failed", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("detail_json", sa.Text(), nullable=True),
)
op.create_index("idx_imports_created_at", "imports", ["created_at"], unique=False)
def downgrade() -> None:
op.drop_index("idx_imports_created_at", table_name="imports")
op.drop_table("imports")
@@ -0,0 +1,83 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0002"
down_revision = "20260210_0001"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"images",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column("illust_id", sa.Integer(), nullable=False),
sa.Column("page_index", sa.Integer(), nullable=False),
sa.Column("ext", sa.Text(), nullable=False),
sa.Column("original_url", sa.Text(), nullable=False),
sa.Column("proxy_path", sa.Text(), nullable=False),
sa.Column("random_key", sa.Float(), nullable=False),
sa.Column("width", sa.Integer(), nullable=True),
sa.Column("height", sa.Integer(), nullable=True),
sa.Column("aspect_ratio", sa.Float(), nullable=True),
sa.Column("orientation", sa.Integer(), nullable=True),
sa.Column("x_restrict", sa.Integer(), nullable=True),
sa.Column("ai_type", sa.Integer(), nullable=True),
sa.Column("user_id", sa.Integer(), nullable=True),
sa.Column("user_name", sa.Text(), nullable=True),
sa.Column("title", sa.Text(), nullable=True),
sa.Column("created_at_pixiv", sa.Text(), nullable=True),
sa.Column("status", sa.Integer(), nullable=False, server_default=sa.text("1")),
sa.Column("fail_count", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("last_fail_at", sa.Text(), nullable=True),
sa.Column("last_ok_at", sa.Text(), nullable=True),
sa.Column("last_error_code", sa.Text(), nullable=True),
sa.Column("last_error_msg", sa.Text(), nullable=True),
sa.Column(
"created_import_id",
sa.Integer(),
sa.ForeignKey("imports.id", name="fk_images_import", ondelete="SET NULL"),
nullable=True,
),
sa.Column(
"added_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column(
"updated_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.CheckConstraint("status IN (1,2,3,4)", name="ck_images_status"),
sa.CheckConstraint("random_key >= 0.0 AND random_key < 1.0", name="ck_images_random_key"),
sa.UniqueConstraint("illust_id", "page_index", name="uq_images_illust_page"),
)
op.create_index(
"idx_images_filter",
"images",
["status", "x_restrict", "orientation", "width", "height", "random_key"],
unique=False,
)
op.create_index(
"idx_images_user_random",
"images",
["status", "user_id", "random_key"],
unique=False,
)
op.create_index("idx_images_created_at_pixiv", "images", ["created_at_pixiv"], unique=False)
op.create_index("idx_images_created_import_id", "images", ["created_import_id"], unique=False)
def downgrade() -> None:
op.drop_index("idx_images_created_import_id", table_name="images")
op.drop_index("idx_images_created_at_pixiv", table_name="images")
op.drop_index("idx_images_user_random", table_name="images")
op.drop_index("idx_images_filter", table_name="images")
op.drop_table("images")
@@ -0,0 +1,36 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0003"
down_revision = "20260210_0002"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"tags",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column("name", sa.Text(), nullable=False),
sa.Column("translated_name", sa.Text(), nullable=True),
sa.Column(
"created_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column(
"updated_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.UniqueConstraint("name", name="uq_tags_name"),
)
def downgrade() -> None:
op.drop_table("tags")
@@ -0,0 +1,35 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0004"
down_revision = "20260210_0003"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"image_tags",
sa.Column(
"image_id",
sa.Integer(),
sa.ForeignKey("images.id", name="fk_image_tags_image", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"tag_id",
sa.Integer(),
sa.ForeignKey("tags.id", name="fk_image_tags_tag", ondelete="CASCADE"),
nullable=False,
),
sa.PrimaryKeyConstraint("image_id", "tag_id"),
)
op.create_index("idx_image_tags_tag_image", "image_tags", ["tag_id", "image_id"], unique=False)
def downgrade() -> None:
op.drop_index("idx_image_tags_tag_image", table_name="image_tags")
op.drop_table("image_tags")
@@ -0,0 +1,47 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0005"
down_revision = "20260210_0004"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"pixiv_tokens",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column(
"created_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column(
"updated_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column("label", sa.Text(), nullable=True),
sa.Column("enabled", sa.Integer(), nullable=False, server_default=sa.text("1")),
sa.Column("refresh_token_enc", sa.Text(), nullable=False),
sa.Column("refresh_token_masked", sa.Text(), nullable=False),
sa.Column("weight", sa.Float(), nullable=False, server_default=sa.text("1.0")),
sa.Column("error_count", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("backoff_until", sa.Text(), nullable=True),
sa.Column("last_ok_at", sa.Text(), nullable=True),
sa.Column("last_fail_at", sa.Text(), nullable=True),
sa.Column("last_error_code", sa.Text(), nullable=True),
sa.Column("last_error_msg", sa.Text(), nullable=True),
sa.CheckConstraint("enabled IN (0,1)", name="ck_pixiv_tokens_enabled"),
)
op.create_index("idx_pixiv_tokens_enabled", "pixiv_tokens", ["enabled"], unique=False)
def downgrade() -> None:
op.drop_index("idx_pixiv_tokens_enabled", table_name="pixiv_tokens")
op.drop_table("pixiv_tokens")
@@ -0,0 +1,55 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0006"
down_revision = "20260210_0005"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"proxy_endpoints",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column(
"created_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column(
"updated_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column("scheme", sa.Text(), nullable=False),
sa.Column("host", sa.Text(), nullable=False),
sa.Column("port", sa.Integer(), nullable=False),
sa.Column("username", sa.Text(), nullable=False, server_default=sa.text("''")),
sa.Column("password_enc", sa.Text(), nullable=False, server_default=sa.text("''")),
sa.Column("enabled", sa.Integer(), nullable=False, server_default=sa.text("1")),
sa.Column("source", sa.Text(), nullable=False, server_default=sa.text("'manual'")),
sa.Column("source_ref", sa.Text(), nullable=True),
sa.Column("last_latency_ms", sa.Float(), nullable=True),
sa.Column("last_ok_at", sa.Text(), nullable=True),
sa.Column("last_fail_at", sa.Text(), nullable=True),
sa.Column("success_count", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("failure_count", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("blacklisted_until", sa.Text(), nullable=True),
sa.Column("last_error", sa.Text(), nullable=True),
sa.UniqueConstraint("scheme", "host", "port", "username", name="uq_proxy_identity"),
sa.CheckConstraint("enabled IN (0,1)", name="ck_proxy_enabled"),
)
op.create_index("idx_proxy_endpoints_enabled", "proxy_endpoints", ["enabled"], unique=False)
op.create_index("idx_proxy_endpoints_source", "proxy_endpoints", ["source"], unique=False)
def downgrade() -> None:
op.drop_index("idx_proxy_endpoints_source", table_name="proxy_endpoints")
op.drop_index("idx_proxy_endpoints_enabled", table_name="proxy_endpoints")
op.drop_table("proxy_endpoints")
@@ -0,0 +1,40 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0007"
down_revision = "20260210_0006"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"proxy_pools",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column(
"created_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column(
"updated_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column("name", sa.Text(), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("enabled", sa.Integer(), nullable=False, server_default=sa.text("1")),
sa.UniqueConstraint("name", name="uq_proxy_pools_name"),
sa.CheckConstraint("enabled IN (0,1)", name="ck_proxy_pools_enabled"),
)
op.create_index("idx_proxy_pools_enabled", "proxy_pools", ["enabled"], unique=False)
def downgrade() -> None:
op.drop_index("idx_proxy_pools_enabled", table_name="proxy_pools")
op.drop_table("proxy_pools")
@@ -0,0 +1,63 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0008"
down_revision = "20260210_0007"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"proxy_pool_endpoints",
sa.Column(
"pool_id",
sa.Integer(),
sa.ForeignKey("proxy_pools.id", name="fk_ppe_pool", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"endpoint_id",
sa.Integer(),
sa.ForeignKey("proxy_endpoints.id", name="fk_ppe_ep", ondelete="CASCADE"),
nullable=False,
),
sa.Column("enabled", sa.Integer(), nullable=False, server_default=sa.text("1")),
sa.Column("weight", sa.Integer(), nullable=False, server_default=sa.text("1")),
sa.Column(
"created_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column(
"updated_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.PrimaryKeyConstraint("pool_id", "endpoint_id"),
sa.CheckConstraint("enabled IN (0,1)", name="ck_ppe_enabled"),
)
op.create_index(
"idx_ppe_pool_enabled",
"proxy_pool_endpoints",
["pool_id", "enabled"],
unique=False,
)
op.create_index(
"idx_ppe_endpoint_pool",
"proxy_pool_endpoints",
["endpoint_id", "pool_id"],
unique=False,
)
def downgrade() -> None:
op.drop_index("idx_ppe_endpoint_pool", table_name="proxy_pool_endpoints")
op.drop_index("idx_ppe_pool_enabled", table_name="proxy_pool_endpoints")
op.drop_table("proxy_pool_endpoints")
@@ -0,0 +1,66 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0009"
down_revision = "20260210_0008"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"token_proxy_bindings",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column(
"created_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column(
"updated_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column(
"token_id",
sa.Integer(),
sa.ForeignKey("pixiv_tokens.id", name="fk_tpb_token", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"pool_id",
sa.Integer(),
sa.ForeignKey("proxy_pools.id", name="fk_tpb_pool", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"primary_proxy_id",
sa.Integer(),
sa.ForeignKey("proxy_endpoints.id", name="fk_tpb_primary", ondelete="RESTRICT"),
nullable=False,
),
sa.Column(
"override_proxy_id",
sa.Integer(),
sa.ForeignKey("proxy_endpoints.id", name="fk_tpb_override", ondelete="SET NULL"),
nullable=True,
),
sa.Column("override_expires_at", sa.Text(), nullable=True),
sa.UniqueConstraint("token_id", "pool_id", name="uq_token_pool"),
)
op.create_index("idx_tpb_pool", "token_proxy_bindings", ["pool_id"], unique=False)
op.create_index("idx_tpb_primary", "token_proxy_bindings", ["primary_proxy_id"], unique=False)
op.create_index("idx_tpb_override", "token_proxy_bindings", ["override_proxy_id"], unique=False)
def downgrade() -> None:
op.drop_index("idx_tpb_override", table_name="token_proxy_bindings")
op.drop_index("idx_tpb_primary", table_name="token_proxy_bindings")
op.drop_index("idx_tpb_pool", table_name="token_proxy_bindings")
op.drop_table("token_proxy_bindings")
@@ -0,0 +1,30 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0010"
down_revision = "20260210_0009"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"runtime_settings",
sa.Column("key", sa.Text(), primary_key=True),
sa.Column("value_json", sa.Text(), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column(
"updated_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column("updated_by", sa.Text(), nullable=True),
)
def downgrade() -> None:
op.drop_table("runtime_settings")
@@ -0,0 +1,61 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0011"
down_revision = "20260210_0010"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"jobs",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column(
"created_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column(
"updated_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column("type", sa.Text(), nullable=False),
sa.Column("status", sa.Text(), nullable=False),
sa.Column("priority", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("run_after", sa.Text(), nullable=True),
sa.Column("attempt", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("max_attempts", sa.Integer(), nullable=False, server_default=sa.text("3")),
sa.Column("payload_json", sa.Text(), nullable=False),
sa.Column("last_error", sa.Text(), nullable=True),
sa.Column("locked_by", sa.Text(), nullable=True),
sa.Column("locked_at", sa.Text(), nullable=True),
sa.Column("ref_type", sa.Text(), nullable=True),
sa.Column("ref_id", sa.Text(), nullable=True),
sa.CheckConstraint(
"status IN ('pending','running','paused','canceled','completed','failed','dlq')",
name="ck_jobs_status",
),
)
op.create_index(
"idx_jobs_status_priority",
"jobs",
["status", "priority", "id"],
unique=False,
)
op.create_index("idx_jobs_run_after", "jobs", ["run_after"], unique=False)
op.create_index("idx_jobs_ref", "jobs", ["ref_type", "ref_id"], unique=False)
def downgrade() -> None:
op.drop_index("idx_jobs_ref", table_name="jobs")
op.drop_index("idx_jobs_run_after", table_name="jobs")
op.drop_index("idx_jobs_status_priority", table_name="jobs")
op.drop_table("jobs")
@@ -0,0 +1,56 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0012"
down_revision = "20260210_0011"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"hydration_runs",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column(
"created_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column(
"updated_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column("type", sa.Text(), nullable=False),
sa.Column("status", sa.Text(), nullable=False),
sa.Column("criteria_json", sa.Text(), nullable=True),
sa.Column("cursor_json", sa.Text(), nullable=True),
sa.Column("total", sa.Integer(), nullable=True),
sa.Column("processed", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("success", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("failed", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("started_at", sa.Text(), nullable=True),
sa.Column("finished_at", sa.Text(), nullable=True),
sa.Column("last_error", sa.Text(), nullable=True),
sa.CheckConstraint(
"status IN ('pending','running','paused','canceled','completed','failed')",
name="ck_hr_status",
),
)
op.create_index(
"idx_hr_status_updated",
"hydration_runs",
["status", "updated_at"],
unique=False,
)
def downgrade() -> None:
op.drop_index("idx_hr_status_updated", table_name="hydration_runs")
op.drop_table("hydration_runs")
@@ -0,0 +1,42 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0013"
down_revision = "20260210_0012"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"request_logs",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column(
"created_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column("request_id", sa.Text(), nullable=True),
sa.Column("method", sa.Text(), nullable=False),
sa.Column("route", sa.Text(), nullable=False),
sa.Column("status", sa.Integer(), nullable=False),
sa.Column("duration_ms", sa.Integer(), nullable=False),
sa.Column("ip", sa.Text(), nullable=True),
sa.Column("user_agent", sa.Text(), nullable=True),
sa.Column("sample_rate", sa.Float(), nullable=True),
)
op.create_index("idx_request_logs_created_at", "request_logs", ["created_at"], unique=False)
op.create_index("idx_request_logs_route", "request_logs", ["route"], unique=False)
op.create_index("idx_request_logs_status", "request_logs", ["status"], unique=False)
def downgrade() -> None:
op.drop_index("idx_request_logs_status", table_name="request_logs")
op.drop_index("idx_request_logs_route", table_name="request_logs")
op.drop_index("idx_request_logs_created_at", table_name="request_logs")
op.drop_table("request_logs")
@@ -0,0 +1,44 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260210_0014"
down_revision = "20260210_0013"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"admin_audit",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column(
"created_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column("actor", sa.Text(), nullable=True),
sa.Column("action", sa.Text(), nullable=False),
sa.Column("resource", sa.Text(), nullable=False),
sa.Column("record_id", sa.Text(), nullable=True),
sa.Column("request_id", sa.Text(), nullable=True),
sa.Column("ip", sa.Text(), nullable=True),
sa.Column("user_agent", sa.Text(), nullable=True),
sa.Column("detail_json", sa.Text(), nullable=True),
)
op.create_index("idx_admin_audit_created_at", "admin_audit", ["created_at"], unique=False)
op.create_index("idx_admin_audit_action", "admin_audit", ["action"], unique=False)
op.create_index("idx_admin_audit_resource", "admin_audit", ["resource"], unique=False)
op.create_index("idx_admin_audit_record_id", "admin_audit", ["record_id"], unique=False)
def downgrade() -> None:
op.drop_index("idx_admin_audit_record_id", table_name="admin_audit")
op.drop_index("idx_admin_audit_resource", table_name="admin_audit")
op.drop_index("idx_admin_audit_action", table_name="admin_audit")
op.drop_index("idx_admin_audit_created_at", table_name="admin_audit")
op.drop_table("admin_audit")
@@ -0,0 +1,45 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260211_0015"
down_revision = "20260210_0014"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"api_keys",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column(
"created_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column(
"updated_at",
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
),
sa.Column("name", sa.Text(), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("key_hash", sa.Text(), nullable=False),
sa.Column("hint", sa.Text(), nullable=False, server_default=sa.text("''")),
sa.Column("enabled", sa.Integer(), nullable=False, server_default=sa.text("1")),
sa.Column("last_used_at", sa.Text(), nullable=True),
sa.UniqueConstraint("name", name="uq_api_keys_name"),
sa.CheckConstraint("enabled IN (0,1)", name="ck_api_keys_enabled"),
)
op.create_index("idx_api_keys_enabled", "api_keys", ["enabled"], unique=False)
op.create_index("idx_api_keys_created_at", "api_keys", ["created_at"], unique=False)
def downgrade() -> None:
op.drop_index("idx_api_keys_created_at", table_name="api_keys")
op.drop_index("idx_api_keys_enabled", table_name="api_keys")
op.drop_table("api_keys")
@@ -0,0 +1,134 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260211_0016"
down_revision = "20260211_0015"
branch_labels = None
depends_on = None
def _try_create_fts5(conn: sa.Connection, *, table_name: str, columns_sql: str) -> bool:
candidates = [
f"CREATE VIRTUAL TABLE {table_name} USING fts5({columns_sql}, tokenize='trigram');",
f"CREATE VIRTUAL TABLE {table_name} USING fts5({columns_sql});",
]
for sql in candidates:
try:
conn.exec_driver_sql(sql)
return True
except Exception:
continue
return False
def upgrade() -> None:
conn = op.get_bind()
tags_ok = _try_create_fts5(conn, table_name="tags_fts", columns_sql="name, translated_name")
if tags_ok:
conn.exec_driver_sql(
"INSERT INTO tags_fts(rowid, name, translated_name) "
"SELECT id, name, COALESCE(translated_name,'') FROM tags;"
)
conn.exec_driver_sql(
"""
CREATE TRIGGER tags_ai_fts AFTER INSERT ON tags BEGIN
INSERT INTO tags_fts(rowid, name, translated_name)
VALUES (new.id, new.name, COALESCE(new.translated_name,''));
END;
""".strip()
)
conn.exec_driver_sql(
"""
CREATE TRIGGER tags_ad_fts AFTER DELETE ON tags BEGIN
DELETE FROM tags_fts WHERE rowid = old.id;
END;
""".strip()
)
conn.exec_driver_sql(
"""
CREATE TRIGGER tags_au_fts AFTER UPDATE OF name, translated_name ON tags BEGIN
DELETE FROM tags_fts WHERE rowid = old.id;
INSERT INTO tags_fts(rowid, name, translated_name)
VALUES (new.id, new.name, COALESCE(new.translated_name,''));
END;
""".strip()
)
authors_ok = _try_create_fts5(conn, table_name="authors_fts", columns_sql="user_name")
if authors_ok:
conn.exec_driver_sql(
"INSERT INTO authors_fts(rowid, user_name) "
"SELECT user_id, MAX(user_name) "
"FROM images "
"WHERE user_id IS NOT NULL AND user_name IS NOT NULL AND status=1 "
"GROUP BY user_id;"
)
conn.exec_driver_sql(
"""
CREATE TRIGGER images_ai_authors_fts AFTER INSERT ON images
WHEN new.user_id IS NOT NULL
BEGIN
DELETE FROM authors_fts WHERE rowid = new.user_id;
INSERT INTO authors_fts(rowid, user_name)
SELECT new.user_id, MAX(user_name)
FROM images
WHERE user_id = new.user_id AND status = 1 AND user_name IS NOT NULL
HAVING MAX(user_name) IS NOT NULL;
END;
""".strip()
)
conn.exec_driver_sql(
"""
CREATE TRIGGER images_au_authors_fts AFTER UPDATE OF user_id, user_name, status ON images
BEGIN
DELETE FROM authors_fts WHERE rowid = old.user_id;
INSERT INTO authors_fts(rowid, user_name)
SELECT old.user_id, MAX(user_name)
FROM images
WHERE old.user_id IS NOT NULL AND user_id = old.user_id AND status = 1 AND user_name IS NOT NULL
HAVING MAX(user_name) IS NOT NULL;
DELETE FROM authors_fts
WHERE rowid = new.user_id AND new.user_id IS NOT NULL AND (old.user_id IS NULL OR new.user_id != old.user_id);
INSERT INTO authors_fts(rowid, user_name)
SELECT new.user_id, MAX(user_name)
FROM images
WHERE new.user_id IS NOT NULL
AND (old.user_id IS NULL OR new.user_id != old.user_id)
AND user_id = new.user_id AND status = 1 AND user_name IS NOT NULL
HAVING MAX(user_name) IS NOT NULL;
END;
""".strip()
)
conn.exec_driver_sql(
"""
CREATE TRIGGER images_ad_authors_fts AFTER DELETE ON images
WHEN old.user_id IS NOT NULL
BEGIN
DELETE FROM authors_fts WHERE rowid = old.user_id;
INSERT INTO authors_fts(rowid, user_name)
SELECT old.user_id, MAX(user_name)
FROM images
WHERE user_id = old.user_id AND status = 1 AND user_name IS NOT NULL
HAVING MAX(user_name) IS NOT NULL;
END;
""".strip()
)
def downgrade() -> None:
conn = op.get_bind()
conn.exec_driver_sql("DROP TRIGGER IF EXISTS images_ad_authors_fts;")
conn.exec_driver_sql("DROP TRIGGER IF EXISTS images_au_authors_fts;")
conn.exec_driver_sql("DROP TRIGGER IF EXISTS images_ai_authors_fts;")
conn.exec_driver_sql("DROP TABLE IF EXISTS authors_fts;")
conn.exec_driver_sql("DROP TRIGGER IF EXISTS tags_au_fts;")
conn.exec_driver_sql("DROP TRIGGER IF EXISTS tags_ad_fts;")
conn.exec_driver_sql("DROP TRIGGER IF EXISTS tags_ai_fts;")
conn.exec_driver_sql("DROP TABLE IF EXISTS tags_fts;")
@@ -0,0 +1,23 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260215_0017"
down_revision = "20260211_0016"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column("images", sa.Column("bookmark_count", sa.Integer(), nullable=True))
op.add_column("images", sa.Column("view_count", sa.Integer(), nullable=True))
op.add_column("images", sa.Column("comment_count", sa.Integer(), nullable=True))
def downgrade() -> None:
with op.batch_alter_table("images") as batch_op:
batch_op.drop_column("comment_count")
batch_op.drop_column("view_count")
batch_op.drop_column("bookmark_count")
@@ -0,0 +1,21 @@
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260218_0018"
down_revision = "20260215_0017"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column("images", sa.Column("illust_type", sa.Integer(), nullable=True))
op.create_index("idx_images_illust_type_random", "images", ["status", "illust_type", "random_key"])
def downgrade() -> None:
op.drop_index("idx_images_illust_type_random", table_name="images")
with op.batch_alter_table("images") as batch_op:
batch_op.drop_column("illust_type")
+1
View File
@@ -0,0 +1 @@
+1
View File
@@ -0,0 +1 @@
+202
View File
@@ -0,0 +1,202 @@
from __future__ import annotations
from typing import Any
import sqlalchemy as sa
from fastapi import APIRouter, Depends, Request
from sqlalchemy.exc import IntegrityError
from app.api.admin.deps import get_admin_claims
from app.core.api_keys import api_key_hint, hmac_sha256_hex
from app.core.errors import ApiError, ErrorCode
from app.core.request_id import get_or_create_request_id
from app.core.time import iso_utc_ms
from app.db.models.api_keys import ApiKey
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
router = APIRouter()
def _as_bool(value: Any) -> bool | None:
if isinstance(value, bool):
return value
if isinstance(value, int) and value in (0, 1):
return bool(value)
if isinstance(value, str):
v = value.strip().lower()
if v in {"true", "1", "yes", "y", "on"}:
return True
if v in {"false", "0", "no", "n", "off"}:
return False
return None
async def _load_json_object(request: Request) -> dict[str, Any]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
return data
@router.get("/api-keys")
async def list_api_keys(
request: Request,
limit: int = 50,
cursor: str | None = None,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if limit < 1 or limit > 200:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
cursor_i: int | None = None
cursor_raw = (cursor or "").strip()
if cursor_raw:
if not cursor_raw.isdigit():
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
cursor_i = int(cursor_raw)
if cursor_i <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
rid = get_or_create_request_id(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
stmt = sa.select(ApiKey).order_by(ApiKey.id.desc()).limit(limit + 1)
if cursor_i is not None:
stmt = stmt.where(ApiKey.id < cursor_i)
async with Session() as session:
rows = ((await session.execute(stmt)).scalars().all())
items_rows = rows[:limit]
next_cursor = int(items_rows[-1].id) if len(rows) > limit and items_rows else None
items = [
{
"id": str(row.id),
"name": row.name,
"description": row.description,
"enabled": bool(row.enabled),
"hint": row.hint,
"created_at": row.created_at,
"updated_at": row.updated_at,
"last_used_at": row.last_used_at,
}
for row in items_rows
]
return {
"ok": True,
"items": items,
"next_cursor": str(next_cursor) if next_cursor is not None else "",
"request_id": rid,
}
@router.post("/api-keys")
async def create_api_key(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
body = await _load_json_object(request)
name = str(body.get("name") or "").strip()
api_key = str(body.get("api_key") or "").strip()
description = str(body.get("description") or "").strip() or None
enabled_v = _as_bool(body.get("enabled"))
enabled = bool(enabled_v) if enabled_v is not None else True
if not name or len(name) > 100:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid name", status_code=400)
if not api_key or len(api_key) < 20 or len(api_key) > 500:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid api_key", status_code=400)
if description is not None and len(description) > 1000:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid description", status_code=400)
settings = request.app.state.settings
try:
key_hash = hmac_sha256_hex(secret_key=settings.secret_key, message=api_key)
except Exception as exc:
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="API key hashing not configured", status_code=500) from exc
hint = api_key_hint(api_key)
now = iso_utc_ms()
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> int:
async with Session() as session:
row = ApiKey(
name=name,
description=description,
key_hash=key_hash,
hint=hint,
enabled=1 if enabled else 0,
updated_at=now,
)
session.add(row)
try:
await session.commit()
except IntegrityError as exc:
await session.rollback()
raise ApiError(code=ErrorCode.BAD_REQUEST, message="API key name exists", status_code=400) from exc
await session.refresh(row)
return int(row.id)
api_key_id = await with_sqlite_busy_retry(_op)
return {"ok": True, "api_key_id": str(api_key_id), "hint": hint, "request_id": rid}
@router.put("/api-keys/{api_key_id}")
async def update_api_key(
api_key_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if api_key_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid api_key_id", status_code=400)
rid = get_or_create_request_id(request)
body = await _load_json_object(request)
enabled_v = _as_bool(body.get("enabled")) if "enabled" in body else None
description_raw = body.get("description") if "description" in body else None
description = str(description_raw).strip() if isinstance(description_raw, str) else None
if description is not None and not description:
description = None
if description is not None and len(description) > 1000:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid description", status_code=400)
if enabled_v is None and "description" not in body:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing fields", status_code=400)
now = iso_utc_ms()
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
async with Session() as session:
row = await session.get(ApiKey, int(api_key_id))
if row is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="API key not found", status_code=404)
if enabled_v is not None:
row.enabled = 1 if bool(enabled_v) else 0
if "description" in body:
row.description = description
row.updated_at = now
await session.commit()
return {"ok": True, "api_key_id": str(api_key_id), "request_id": rid}
return await with_sqlite_busy_retry(_op)
+83
View File
@@ -0,0 +1,83 @@
from __future__ import annotations
import json
from typing import Any
import sqlalchemy as sa
from fastapi import APIRouter, Depends, Request
from app.api.admin.deps import get_admin_claims
from app.core.errors import ApiError, ErrorCode
from app.core.request_id import get_or_create_request_id
from app.db.models.admin_audit import AdminAudit
from app.db.session import create_sessionmaker
router = APIRouter()
@router.get("/audit")
async def list_admin_audit(
request: Request,
limit: int = 50,
cursor: str | None = None,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if limit < 1 or limit > 200:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
cursor_i: int | None = None
cursor_raw = (cursor or "").strip()
if cursor_raw:
if not cursor_raw.isdigit():
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
cursor_i = int(cursor_raw)
if cursor_i <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
rid = get_or_create_request_id(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
stmt = sa.select(AdminAudit).order_by(AdminAudit.id.desc()).limit(limit + 1)
if cursor_i is not None:
stmt = stmt.where(AdminAudit.id < cursor_i)
async with Session() as session:
rows = (await session.execute(stmt)).scalars().all()
items_rows = rows[:limit]
next_cursor = int(items_rows[-1].id) if len(rows) > limit and items_rows else None
def _parse_detail(text: str | None) -> dict[str, Any] | None:
raw = str(text or "").strip()
if not raw:
return None
try:
data = json.loads(raw)
except Exception:
return {"raw": raw}
return data if isinstance(data, dict) else {"value": data}
items = [
{
"id": str(row.id),
"created_at": row.created_at,
"actor": row.actor,
"action": row.action,
"resource": row.resource,
"record_id": row.record_id,
"request_id": row.request_id,
"detail_json": _parse_detail(row.detail_json),
}
for row in items_rows
]
return {
"ok": True,
"items": items,
"next_cursor": str(next_cursor) if next_cursor is not None else "",
"request_id": rid,
}
+51
View File
@@ -0,0 +1,51 @@
from __future__ import annotations
import hmac
from typing import Any
from fastapi import APIRouter, Request
from app.api.admin.deps import get_admin_claims
from app.core.errors import ApiError, ErrorCode
from app.core.request_id import get_or_create_request_id
from app.core.security import create_jwt
from fastapi import Depends
router = APIRouter()
async def _load_login_json(request: Request) -> tuple[str, str]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
username = str(data.get("username") or "").strip()
password = str(data.get("password") or "")
if not username or not password:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing credentials", status_code=400)
return username, password
@router.post("/login")
async def login(request: Request) -> dict[str, Any]:
username, password = await _load_login_json(request)
settings = request.app.state.settings
if username != settings.admin_username or not hmac.compare_digest(password, settings.admin_password):
raise ApiError(code=ErrorCode.UNAUTHORIZED, message="Invalid credentials", status_code=401)
token = create_jwt(secret_key=settings.secret_key, subject=settings.admin_username, ttl_s=3600)
rid = get_or_create_request_id(request)
return {"ok": True, "token": token, "request_id": rid}
@router.post("/logout")
async def logout(request: Request, _claims: dict[str, Any] = Depends(get_admin_claims)) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
return {"ok": True, "request_id": rid}
+403
View File
@@ -0,0 +1,403 @@
from __future__ import annotations
from datetime import datetime, timedelta, timezone
from typing import Any
import sqlalchemy as sa
from fastapi import APIRouter, Depends, Request
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from sqlalchemy.orm import aliased
from app.api.admin.deps import get_admin_claims
from app.core.bindings_recompute import recompute_token_proxy_bindings
from app.core.errors import ApiError, ErrorCode
from app.core.request_id import get_or_create_request_id
from app.core.time import iso_utc_ms
from app.db.models.pixiv_tokens import PixivToken
from app.db.models.proxy_endpoints import ProxyEndpoint
from app.db.models.proxy_pool_endpoints import ProxyPoolEndpoint
from app.db.models.proxy_pools import ProxyPool
from app.db.models.token_proxy_bindings import TokenProxyBinding
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
router = APIRouter()
def _parse_bool(value: Any, *, default: bool) -> bool:
if value is None:
return bool(default)
if isinstance(value, bool):
return bool(value)
if isinstance(value, int) and value in (0, 1):
return bool(value)
if isinstance(value, str):
v = value.strip().lower()
if v in {"1", "true", "yes", "y", "on"}:
return True
if v in {"0", "false", "no", "n", "off"}:
return False
return bool(default)
def _fnv1a64(text: str) -> int:
h = 14695981039346656037
prime = 1099511628211
for b in text.encode("utf-8"):
h ^= b
h = (h * prime) & 0xFFFFFFFFFFFFFFFF
return h
def _rendezvous_proxy_order(*, token_id: int, proxy_ids: list[int], salt: str) -> list[int]:
scored = [(_fnv1a64(f"{token_id}|{pid}|{salt}"), pid) for pid in proxy_ids]
scored.sort(key=lambda x: (-x[0], x[1]))
return [pid for _, pid in scored]
def _compute_primary_assignments(
*,
token_ids: list[int],
proxy_ids: list[int],
capacity_by_proxy_id: dict[int, int],
salt: str,
) -> dict[int, int]:
remaining = {pid: int(capacity_by_proxy_id.get(int(pid), 0)) for pid in proxy_ids}
out: dict[int, int] = {}
for token_id in token_ids:
for pid in _rendezvous_proxy_order(token_id=token_id, proxy_ids=proxy_ids, salt=salt):
if remaining.get(pid, 0) > 0:
out[token_id] = pid
remaining[pid] -= 1
break
return out
def _compute_primary_assignments_soft(
*,
token_ids: list[int],
proxy_ids: list[int],
capacity_by_proxy_id: dict[int, int],
salt: str,
) -> tuple[dict[int, int], int]:
out = _compute_primary_assignments(
token_ids=token_ids,
proxy_ids=proxy_ids,
capacity_by_proxy_id=capacity_by_proxy_id,
salt=salt,
)
over_capacity = 0
for token_id in token_ids:
if token_id in out:
continue
order = _rendezvous_proxy_order(token_id=token_id, proxy_ids=proxy_ids, salt=salt)
if not order:
continue
out[token_id] = int(order[0])
over_capacity += 1
return out, int(over_capacity)
async def _load_recompute_json(request: Request) -> dict[str, Any]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
try:
pool_id = int(data.get("pool_id"))
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid pool_id", status_code=400) from exc
if pool_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid pool_id", status_code=400)
raw_max = data.get("max_tokens_per_proxy", 2)
try:
max_tokens_per_proxy = int(raw_max)
except Exception as exc:
raise ApiError(
code=ErrorCode.BAD_REQUEST,
message="Invalid max_tokens_per_proxy",
status_code=400,
) from exc
if max_tokens_per_proxy <= 0 or max_tokens_per_proxy > 1000:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid max_tokens_per_proxy", status_code=400)
strict = _parse_bool(data.get("strict"), default=True)
return {"pool_id": pool_id, "max_tokens_per_proxy": max_tokens_per_proxy, "strict": strict}
async def _load_override_json(request: Request) -> dict[str, Any]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
try:
override_proxy_id = int(data.get("override_proxy_id"))
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid override_proxy_id", status_code=400) from exc
if override_proxy_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid override_proxy_id", status_code=400)
try:
ttl_ms = int(data.get("ttl_ms"))
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid ttl_ms", status_code=400) from exc
if ttl_ms <= 0 or ttl_ms > 30 * 24 * 60 * 60 * 1000:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid ttl_ms", status_code=400)
reason = str(data.get("reason") or "").strip()
if reason and len(reason) > 200:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid reason", status_code=400)
return {"override_proxy_id": override_proxy_id, "ttl_ms": ttl_ms, "reason": reason}
@router.get("/bindings")
async def list_bindings(
request: Request,
pool_id: int,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if int(pool_id) <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid pool_id", status_code=400)
rid = get_or_create_request_id(request)
now = iso_utc_ms()
engine = request.app.state.engine
Session = create_sessionmaker(engine)
Primary = aliased(ProxyEndpoint)
Override = aliased(ProxyEndpoint)
async with Session() as session:
rows = (
(
await session.execute(
sa.select(TokenProxyBinding, PixivToken.label, ProxyPool.name, Primary, Override)
.join(PixivToken, PixivToken.id == TokenProxyBinding.token_id)
.join(ProxyPool, ProxyPool.id == TokenProxyBinding.pool_id)
.join(Primary, Primary.id == TokenProxyBinding.primary_proxy_id)
.outerjoin(Override, Override.id == TokenProxyBinding.override_proxy_id)
.where(TokenProxyBinding.pool_id == int(pool_id))
.order_by(TokenProxyBinding.id.asc())
)
)
.all()
)
counts = (
(
await session.execute(
sa.select(
sa.func.count(ProxyPoolEndpoint.endpoint_id),
sa.func.sum(
sa.case(
(sa.and_(ProxyPoolEndpoint.enabled == 1, ProxyEndpoint.enabled == 1), 1),
else_=0,
)
),
)
.select_from(ProxyPoolEndpoint)
.join(ProxyEndpoint, ProxyEndpoint.id == ProxyPoolEndpoint.endpoint_id)
.where(ProxyPoolEndpoint.pool_id == int(pool_id))
)
)
.first()
)
endpoints_total = int(counts[0] or 0) if counts is not None else 0
endpoints_enabled = int(counts[1] or 0) if counts is not None else 0
items: list[dict[str, Any]] = []
for binding, token_label, pool_name, primary_proxy, override_proxy in rows:
override_active = False
if binding.override_proxy_id is not None and binding.override_expires_at:
override_active = str(binding.override_expires_at) > now
effective_proxy_id = binding.override_proxy_id if override_active else binding.primary_proxy_id
effective_mode = "override" if override_active else "primary"
items.append(
{
"id": str(binding.id),
"created_at": binding.created_at,
"updated_at": binding.updated_at,
"token": {"id": str(binding.token_id), "label": token_label},
"pool": {"id": str(binding.pool_id), "name": pool_name},
"primary_proxy": {
"id": str(primary_proxy.id),
"scheme": primary_proxy.scheme,
"host": primary_proxy.host,
"port": int(primary_proxy.port),
"username": primary_proxy.username,
},
"override_proxy": (
{
"id": str(override_proxy.id),
"scheme": override_proxy.scheme,
"host": override_proxy.host,
"port": int(override_proxy.port),
"username": override_proxy.username,
}
if override_proxy is not None
else None
),
"override_expires_at": binding.override_expires_at,
"effective_proxy_id": str(effective_proxy_id),
"effective_mode": effective_mode,
}
)
return {
"ok": True,
"items": items,
"summary": {
"pool_id": str(pool_id),
"pool_endpoints_total": int(endpoints_total),
"pool_endpoints_enabled": int(endpoints_enabled),
},
"request_id": rid,
}
@router.post("/bindings/recompute")
async def recompute_bindings(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
now = iso_utc_ms()
body = await _load_recompute_json(request)
pool_id = int(body["pool_id"])
max_tokens_per_proxy = int(body["max_tokens_per_proxy"])
strict = bool(body.get("strict", True))
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
async with Session() as session:
pool = await session.get(ProxyPool, pool_id)
if pool is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Proxy pool not found", status_code=404)
result = await recompute_token_proxy_bindings(
session,
pool_id=int(pool_id),
now=str(now),
max_tokens_per_proxy=int(max_tokens_per_proxy),
strict=bool(strict),
)
await session.commit()
return {"ok": True, "pool_id": str(pool_id), "request_id": rid, **result}
return await with_sqlite_busy_retry(_op)
@router.post("/bindings/{binding_id}/override")
async def set_binding_override(
binding_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if binding_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid binding id", status_code=400)
rid = get_or_create_request_id(request)
now = iso_utc_ms()
body = await _load_override_json(request)
override_proxy_id = int(body["override_proxy_id"])
ttl_ms = int(body["ttl_ms"])
_reason = str(body["reason"] or "")
expires_at = iso_utc_ms(datetime.now(timezone.utc) + timedelta(milliseconds=ttl_ms))
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
async with Session() as session:
binding = await session.get(TokenProxyBinding, binding_id)
if binding is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Binding not found", status_code=404)
proxy = await session.get(ProxyEndpoint, override_proxy_id)
if proxy is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Proxy endpoint not found", status_code=404)
if not bool(proxy.enabled):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Proxy endpoint disabled", status_code=400)
in_pool = (
await session.execute(
sa.select(ProxyPoolEndpoint)
.where(ProxyPoolEndpoint.pool_id == int(binding.pool_id))
.where(ProxyPoolEndpoint.endpoint_id == int(override_proxy_id))
.where(ProxyPoolEndpoint.enabled == 1)
)
).scalars().first()
if in_pool is None:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Override proxy not in pool", status_code=400)
binding.override_proxy_id = int(override_proxy_id)
binding.override_expires_at = expires_at
binding.updated_at = now
await session.commit()
return {
"ok": True,
"binding_id": str(binding_id),
"override_proxy_id": str(override_proxy_id),
"override_expires_at": expires_at,
"request_id": rid,
}
return await with_sqlite_busy_retry(_op)
@router.post("/bindings/{binding_id}/clear-override")
async def clear_binding_override(
binding_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if binding_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid binding id", status_code=400)
rid = get_or_create_request_id(request)
now = iso_utc_ms()
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
async with Session() as session:
binding = await session.get(TokenProxyBinding, binding_id)
if binding is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Binding not found", status_code=404)
binding.override_proxy_id = None
binding.override_expires_at = None
binding.updated_at = now
await session.commit()
return {"ok": True, "binding_id": str(binding_id), "request_id": rid}
return await with_sqlite_busy_retry(_op)
+17
View File
@@ -0,0 +1,17 @@
from __future__ import annotations
from typing import Any
from fastapi import Request
from app.core.security import require_admin
def get_admin_claims(request: Request) -> dict[str, Any]:
settings = request.app.state.settings
return require_admin(
request.headers,
secret_key=settings.secret_key,
admin_username=settings.admin_username,
)
+500
View File
@@ -0,0 +1,500 @@
from __future__ import annotations
import json
from typing import Any
import sqlalchemy as sa
from fastapi import APIRouter, Depends, Request
from app.api.admin.deps import get_admin_claims
from app.core.errors import ApiError, ErrorCode
from app.core.request_id import get_or_create_request_id
from app.core.time import iso_utc_ms
from app.db.models.hydration_runs import HydrationRun
from app.db.models.images import Image
from app.db.models.jobs import JobRow
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
router = APIRouter()
_ALLOWED_RUN_STATUSES = {"pending", "running", "paused", "canceled", "completed", "failed"}
def _parse_json_dict(value: str | None) -> dict[str, Any]:
raw = str(value or "").strip()
if not raw:
return {}
try:
loaded = json.loads(raw)
except Exception:
return {}
if isinstance(loaded, dict):
return loaded
return {}
def _serialize_job(job: JobRow | None) -> dict[str, Any] | None:
if job is None:
return None
return {
"id": str(job.id),
"status": job.status,
"attempt": int(job.attempt),
"max_attempts": int(job.max_attempts),
"run_after": job.run_after,
"last_error": job.last_error,
"locked_by": job.locked_by,
"locked_at": job.locked_at,
"updated_at": job.updated_at,
}
def _serialize_run(run: HydrationRun, *, latest_job: JobRow | None = None) -> dict[str, Any]:
return {
"id": str(run.id),
"type": run.type,
"status": run.status,
"criteria": _parse_json_dict(run.criteria_json),
"cursor": _parse_json_dict(run.cursor_json),
"total": int(run.total) if run.total is not None else None,
"processed": int(run.processed),
"success": int(run.success),
"failed": int(run.failed),
"started_at": run.started_at,
"finished_at": run.finished_at,
"last_error": run.last_error,
"created_at": run.created_at,
"updated_at": run.updated_at,
"latest_job": _serialize_job(latest_job),
}
async def _latest_jobs_by_run_ids(session, *, run_ids: list[str]) -> dict[str, JobRow]:
ids = [str(v).strip() for v in run_ids if str(v).strip()]
if not ids:
return {}
latest_per_ref = (
sa.select(JobRow.ref_id.label("ref_id"), sa.func.max(JobRow.id).label("max_job_id"))
.where(JobRow.ref_type == "hydration_run", JobRow.ref_id.in_(ids))
.group_by(JobRow.ref_id)
.subquery()
)
rows = (
await session.execute(
sa.select(JobRow, latest_per_ref.c.ref_id)
.join(latest_per_ref, JobRow.id == latest_per_ref.c.max_job_id)
.order_by(JobRow.id.desc())
)
).all()
out: dict[str, JobRow] = {}
for job, ref_id in rows:
key = str(ref_id or "").strip()
if not key:
continue
out[key] = job
return out
async def _load_create_json(request: Request) -> dict[str, Any]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
run_type = str(data.get("type") or "backfill").strip().lower() or "backfill"
if run_type not in {"backfill", "manual"}:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid type", status_code=400)
criteria = data.get("criteria")
if criteria is None:
criteria = {}
if not isinstance(criteria, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid criteria", status_code=400)
return {"type": run_type, "criteria": criteria}
async def _load_manual_job_json(request: Request) -> dict[str, int | None]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
illust_id_raw = data.get("illust_id")
image_id_raw = data.get("image_id")
if illust_id_raw is None and image_id_raw is None:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Either illust_id or image_id is required", status_code=400)
if illust_id_raw is not None and image_id_raw is not None:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Provide only one of illust_id or image_id", status_code=400)
illust_id: int | None = None
image_id: int | None = None
if illust_id_raw is not None:
try:
illust_id = int(illust_id_raw)
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid illust_id", status_code=400) from exc
if illust_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid illust_id", status_code=400)
if image_id_raw is not None:
try:
image_id = int(image_id_raw)
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid image_id", status_code=400) from exc
if image_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid image_id", status_code=400)
return {"illust_id": illust_id, "image_id": image_id}
@router.get("/hydration-runs")
async def list_hydration_runs(
request: Request,
limit: int = 20,
cursor: str | None = None,
status: str | None = None,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if limit < 1 or limit > 200:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
cursor_i: int | None = None
cursor_raw = str(cursor or "").strip()
if cursor_raw:
if not cursor_raw.isdigit():
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
cursor_i = int(cursor_raw)
if cursor_i <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
status_norm = str(status or "").strip().lower() or None
if status_norm is not None and status_norm not in _ALLOWED_RUN_STATUSES:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported status", status_code=400)
rid = get_or_create_request_id(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
stmt = sa.select(HydrationRun).order_by(HydrationRun.id.desc()).limit(limit + 1)
if cursor_i is not None:
stmt = stmt.where(HydrationRun.id < cursor_i)
if status_norm is not None:
stmt = stmt.where(HydrationRun.status == status_norm)
rows = (await session.execute(stmt)).scalars().all()
current_rows = rows[:limit]
run_ids = [str(int(r.id)) for r in current_rows]
jobs_by_run_id = await _latest_jobs_by_run_ids(session, run_ids=run_ids)
next_cursor = int(current_rows[-1].id) if len(rows) > limit and current_rows else None
items = [_serialize_run(row, latest_job=jobs_by_run_id.get(str(int(row.id)))) for row in current_rows]
return {
"ok": True,
"items": items,
"next_cursor": str(next_cursor) if next_cursor is not None else "",
"request_id": rid,
}
@router.get("/hydration-runs/{run_id}")
async def get_hydration_run(
run_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if run_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid hydration_run id", status_code=400)
rid = get_or_create_request_id(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
run = await session.get(HydrationRun, run_id)
if run is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Hydration run not found", status_code=404)
jobs = await _latest_jobs_by_run_ids(session, run_ids=[str(run_id)])
item = _serialize_run(run, latest_job=jobs.get(str(run_id)))
return {"ok": True, "item": item, "request_id": rid}
@router.post("/hydration-runs/manual")
async def create_manual_hydration_job(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
now = iso_utc_ms()
body = await _load_manual_job_json(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
async with Session() as session:
illust_id = body.get("illust_id")
image_id = body.get("image_id")
if illust_id is None and image_id is not None:
image = await session.get(Image, int(image_id))
if image is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Image not found", status_code=404)
illust_id = int(image.illust_id)
if illust_id is None or int(illust_id) <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid illust_id", status_code=400)
existing = (
(
await session.execute(
sa.select(JobRow)
.where(
JobRow.type == "hydrate_metadata",
JobRow.ref_type == "manual_hydrate",
JobRow.ref_id == str(int(illust_id)),
JobRow.status.in_(["pending", "running"]),
)
.order_by(JobRow.id.desc())
.limit(1)
)
)
.scalars()
.first()
)
if existing is not None:
return {
"ok": True,
"created": False,
"job_id": str(int(existing.id)),
"illust_id": str(int(illust_id)),
"request_id": rid,
}
job = JobRow(
type="hydrate_metadata",
status="pending",
payload_json=json.dumps({"illust_id": int(illust_id)}, separators=(",", ":"), ensure_ascii=False),
last_error=None,
priority=0,
run_after=None,
attempt=0,
max_attempts=3,
locked_by=None,
locked_at=None,
ref_type="manual_hydrate",
ref_id=str(int(illust_id)),
updated_at=now,
)
session.add(job)
await session.flush()
await session.commit()
return {
"ok": True,
"created": True,
"job_id": str(int(job.id)),
"illust_id": str(int(illust_id)),
"request_id": rid,
}
return await with_sqlite_busy_retry(_op)
@router.post("/hydration-runs")
async def create_hydration_run(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
now = iso_utc_ms()
body = await _load_create_json(request)
run_type = str(body["type"])
criteria = dict(body["criteria"])
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> tuple[int, int]:
async with Session() as session:
run = HydrationRun(
type=run_type,
status="pending",
criteria_json=json.dumps(criteria, separators=(",", ":"), ensure_ascii=False),
cursor_json=None,
total=None,
processed=0,
success=0,
failed=0,
started_at=None,
finished_at=None,
last_error=None,
updated_at=now,
)
session.add(run)
await session.flush()
job = JobRow(
type="hydrate_metadata",
status="pending",
payload_json=json.dumps(
{"hydration_run_id": int(run.id), "criteria": criteria},
separators=(",", ":"),
ensure_ascii=False,
),
last_error=None,
priority=0,
run_after=None,
attempt=0,
max_attempts=3,
locked_by=None,
locked_at=None,
ref_type="hydration_run",
ref_id=str(int(run.id)),
updated_at=now,
)
session.add(job)
await session.flush()
await session.commit()
return int(run.id), int(job.id)
run_id, job_id = await with_sqlite_busy_retry(_op)
return {
"ok": True,
"hydration_run_id": str(run_id),
"job_id": str(job_id),
"request_id": rid,
}
async def _set_run_and_job_status(
request: Request,
*,
run_id: int,
target_status: str,
allowed_from: set[str],
job_status: str,
) -> dict[str, Any]:
if run_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid hydration_run id", status_code=400)
rid = get_or_create_request_id(request)
now = iso_utc_ms()
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
async with Session() as session:
run = await session.get(HydrationRun, run_id)
if run is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Hydration run not found", status_code=404)
if str(run.status) not in allowed_from:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported status transition", status_code=400)
run.status = target_status
run.updated_at = now
if target_status in {"canceled", "completed", "failed"} and run.finished_at is None:
run.finished_at = now
job = (
(
await session.execute(
sa.select(JobRow)
.where(JobRow.ref_type == "hydration_run", JobRow.ref_id == str(run_id))
.order_by(JobRow.id.desc())
.limit(1)
)
)
.scalars()
.first()
)
if job is not None:
job.status = job_status
if job_status in {"pending", "canceled", "paused", "dlq"}:
job.run_after = None
job.locked_by = None
job.locked_at = None
job.updated_at = now
await session.commit()
return {
"ok": True,
"hydration_run_id": str(run_id),
"status": target_status,
"job_status": job_status if job is not None else "",
"request_id": rid,
}
return await with_sqlite_busy_retry(_op)
@router.post("/hydration-runs/{run_id}/pause")
async def pause_hydration_run(
run_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
return await _set_run_and_job_status(
request,
run_id=run_id,
target_status="paused",
allowed_from={"pending", "running"},
job_status="paused",
)
@router.post("/hydration-runs/{run_id}/resume")
async def resume_hydration_run(
run_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
return await _set_run_and_job_status(
request,
run_id=run_id,
target_status="pending",
allowed_from={"paused"},
job_status="pending",
)
@router.post("/hydration-runs/{run_id}/cancel")
async def cancel_hydration_run(
run_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
return await _set_run_and_job_status(
request,
run_id=run_id,
target_status="canceled",
allowed_from={"pending", "running", "paused"},
job_status="canceled",
)
+343
View File
@@ -0,0 +1,343 @@
from __future__ import annotations
from typing import Any
import sqlalchemy as sa
from fastapi import APIRouter, Depends, Query, Request
from app.api.admin.deps import get_admin_claims
from app.core.errors import ApiError, ErrorCode
from app.core.request_id import get_or_create_request_id
from app.db.models.image_tags import ImageTag
from app.db.models.images import Image
from app.db.models.tags import Tag
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
router = APIRouter()
_ALLOWED_MISSING = {"tags", "geometry", "r18", "ai", "illust_type", "user", "title", "created_at", "popularity"}
def _parse_missing(values: list[str] | None) -> list[str]:
out: list[str] = []
seen: set[str] = set()
for raw in values or []:
for part in str(raw or "").replace(",", "|").split("|"):
key = part.strip().lower()
if not key or key in seen:
continue
if key not in _ALLOWED_MISSING:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported missing", status_code=400)
seen.add(key)
out.append(key)
return out
@router.get("/images")
async def list_admin_images(
request: Request,
limit: int = 50,
cursor: str | None = None,
missing: list[str] | None = Query(default=None),
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if limit < 1 or limit > 200:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
cursor_i: int | None = None
cursor_raw = (cursor or "").strip()
if cursor_raw:
if not cursor_raw.isdigit():
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
cursor_i = int(cursor_raw)
if cursor_i <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
missing_keys = _parse_missing(missing)
rid = get_or_create_request_id(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
tag_counts = (
sa.select(ImageTag.image_id.label("image_id"), sa.func.count().label("tag_count"))
.group_by(ImageTag.image_id)
.subquery()
)
tag_count_col = sa.func.coalesce(tag_counts.c.tag_count, 0).label("tag_count")
stmt = (
sa.select(Image, tag_count_col)
.outerjoin(tag_counts, tag_counts.c.image_id == Image.id)
.where(Image.status == 1)
.order_by(Image.id.desc())
.limit(int(limit) + 1)
)
if cursor_i is not None:
stmt = stmt.where(Image.id < int(cursor_i))
for key in missing_keys:
if key == "tags":
stmt = stmt.where(tag_count_col == 0)
elif key == "geometry":
stmt = stmt.where((Image.width.is_(None)) | (Image.height.is_(None)))
elif key == "r18":
stmt = stmt.where(Image.x_restrict.is_(None))
elif key == "ai":
stmt = stmt.where(Image.ai_type.is_(None))
elif key == "illust_type":
stmt = stmt.where(Image.illust_type.is_(None))
elif key == "user":
stmt = stmt.where(Image.user_id.is_(None))
elif key == "title":
stmt = stmt.where((Image.title.is_(None)) | (sa.func.trim(Image.title) == ""))
elif key == "created_at":
stmt = stmt.where((Image.created_at_pixiv.is_(None)) | (sa.func.trim(Image.created_at_pixiv) == ""))
elif key == "popularity":
stmt = stmt.where(
(Image.bookmark_count.is_(None)) | (Image.view_count.is_(None)) | (Image.comment_count.is_(None))
)
async with Session() as session:
rows = (await session.execute(stmt)).all()
rows_page = rows[: int(limit)]
next_cursor = int(rows_page[-1][0].id) if len(rows) > int(limit) and rows_page else None
items: list[dict[str, Any]] = []
for img, tag_count in rows_page:
tag_count_i = int(tag_count or 0)
missing_list: list[str] = []
if tag_count_i <= 0:
missing_list.append("tags")
if img.width is None or img.height is None:
missing_list.append("geometry")
if img.x_restrict is None:
missing_list.append("r18")
if img.ai_type is None:
missing_list.append("ai")
if getattr(img, "illust_type", None) is None:
missing_list.append("illust_type")
if img.user_id is None:
missing_list.append("user")
if img.title is None or not str(img.title).strip():
missing_list.append("title")
if img.created_at_pixiv is None or not str(img.created_at_pixiv).strip():
missing_list.append("created_at")
if img.bookmark_count is None or img.view_count is None or img.comment_count is None:
missing_list.append("popularity")
items.append(
{
"id": str(img.id),
"illust_id": str(img.illust_id),
"page_index": int(img.page_index),
"ext": img.ext,
"status": int(img.status),
"width": img.width,
"height": img.height,
"orientation": img.orientation,
"x_restrict": img.x_restrict,
"ai_type": img.ai_type,
"illust_type": getattr(img, "illust_type", None),
"bookmark_count": img.bookmark_count,
"view_count": img.view_count,
"comment_count": img.comment_count,
"user": {
"id": str(img.user_id) if img.user_id is not None else None,
"name": img.user_name,
},
"title": img.title,
"created_at_pixiv": img.created_at_pixiv,
"original_url": img.original_url,
"proxy_path": img.proxy_path,
"tag_count": tag_count_i,
"missing": missing_list,
}
)
return {
"ok": True,
"items": items,
"next_cursor": str(next_cursor) if next_cursor is not None else "",
"request_id": rid,
}
async def _load_bulk_delete_json(request: Request) -> dict[str, Any]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
raw_ids = data.get("image_ids", None)
if raw_ids is None:
raw_ids = data.get("ids", None)
if raw_ids is None:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing image_ids", status_code=400)
if not isinstance(raw_ids, list):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported image_ids", status_code=400)
ids: list[int] = []
seen: set[int] = set()
for raw in raw_ids:
try:
i = int(raw)
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported image_ids", status_code=400) from exc
if i <= 0 or i in seen:
continue
seen.add(i)
ids.append(i)
if not ids:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Empty image_ids", status_code=400)
if len(ids) > 20_000:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Too many image_ids", status_code=400)
return {"image_ids": ids}
def _chunks(values: list[int], *, chunk_size: int) -> list[list[int]]:
if chunk_size <= 0:
return [values]
out: list[list[int]] = []
for i in range(0, len(values), chunk_size):
out.append(values[i : i + chunk_size])
return out
def _safe_rowcount(result: Any) -> int:
try:
rc = int(getattr(result, "rowcount", 0) or 0)
except Exception:
return 0
return rc if rc > 0 else 0
@router.delete("/images/{image_id}")
async def delete_admin_image(
image_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if int(image_id) <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid image id", status_code=400)
rid = get_or_create_request_id(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
async with Session() as session:
row = await session.get(Image, int(image_id))
if row is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Image not found", status_code=404)
await session.execute(sa.delete(ImageTag).where(ImageTag.image_id == int(image_id)))
await session.delete(row)
await session.commit()
return {"ok": True, "image_id": str(int(image_id)), "request_id": rid}
return await with_sqlite_busy_retry(_op)
@router.post("/images/bulk-delete")
async def bulk_delete_admin_images(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
body = await _load_bulk_delete_json(request)
ids = list(body["image_ids"])
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
deleted = 0
found = 0
async with Session() as session:
for chunk in _chunks(ids, chunk_size=900):
rows = (await session.execute(sa.select(Image.id).where(Image.id.in_(chunk)))).scalars().all()
found += len(rows)
for chunk in _chunks(ids, chunk_size=900):
await session.execute(sa.delete(ImageTag).where(ImageTag.image_id.in_(chunk)))
result = await session.execute(sa.delete(Image).where(Image.id.in_(chunk)))
deleted += _safe_rowcount(result)
await session.commit()
missing = max(0, int(len(ids)) - int(found))
return {
"ok": True,
"requested": int(len(ids)),
"deleted": int(deleted),
"missing": int(missing),
"request_id": rid,
}
return await with_sqlite_busy_retry(_op)
async def _load_clear_images_json(request: Request) -> dict[str, Any]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
confirm = data.get("confirm", False)
if confirm not in {True, 1, "1", "true", "yes", "y", "on"}:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing confirm", status_code=400)
delete_tags = data.get("delete_tags", True)
delete_tags_bool = (
bool(delete_tags)
if isinstance(delete_tags, (bool, int))
else str(delete_tags).strip().lower() in {"1", "true", "yes", "y", "on"}
)
return {"delete_tags": bool(delete_tags_bool)}
@router.post("/images/clear")
async def clear_admin_images(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
body = await _load_clear_images_json(request)
delete_tags = bool(body["delete_tags"])
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
async with Session() as session:
result_links = await session.execute(sa.delete(ImageTag))
result_images = await session.execute(sa.delete(Image))
result_tags = None
if delete_tags:
result_tags = await session.execute(sa.delete(Tag))
await session.commit()
return {
"ok": True,
"deleted_image_tags": _safe_rowcount(result_links),
"deleted_images": _safe_rowcount(result_images),
"deleted_tags": _safe_rowcount(result_tags) if result_tags is not None else 0,
"request_id": rid,
}
return await with_sqlite_busy_retry(_op)
+435
View File
@@ -0,0 +1,435 @@
from __future__ import annotations
import json
import os
from dataclasses import asdict, dataclass
from typing import Any, Literal
from uuid import uuid4
import sqlalchemy as sa
from fastapi import APIRouter, Depends, Request
from pydantic import BaseModel, Field
from pydantic import ValidationError
from starlette.datastructures import UploadFile
from app.api.admin.deps import get_admin_claims
from app.core.data_files import get_sqlite_db_dir, make_file_ref
from app.core.errors import ApiError, ErrorCode
from app.core.request_id import get_or_create_request_id
from app.core.time import iso_utc_ms
from app.core.pixiv_urls import parse_pixiv_original_url
from app.db.models.images import Image
from app.db.models.imports import Import
from app.db.models.jobs import JobRow
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
from app.jobs.dispatch import JobDispatcher
from app.jobs.executor import execute_claimed_job
from app.jobs.handlers.import_images import build_import_images_handler
router = APIRouter()
class ImportCreateRequest(BaseModel):
text: str = Field(min_length=1)
dry_run: bool = False
hydrate_on_import: bool = False
source: str = "manual"
class ImportRollbackRequest(BaseModel):
mode: Literal["disable", "delete"]
@dataclass(frozen=True, slots=True)
class ImportErrorItem:
line: int
url: str
code: str
message: str
def _max_import_text_bytes() -> int:
default = 50 * 1024 * 1024
raw = (os.environ.get("IMPORT_MAX_BYTES") or "").strip()
if not raw:
return default
try:
value = int(raw)
except Exception:
return default
return max(1024, min(int(value), 50 * 1024 * 1024))
def _import_inline_max_accepted() -> int:
default = 200
raw = (os.environ.get("IMPORT_INLINE_MAX_ACCEPTED") or "").strip()
if not raw:
return default
try:
value = int(raw)
except Exception:
return default
return max(0, min(int(value), 10_000))
async def _claim_job_by_id(engine, *, job_id: int, worker_id: str, now: str) -> dict[str, Any] | None:
sql = """
UPDATE jobs
SET status='running',
locked_by=:worker_id,
locked_at=:now,
updated_at=:now
WHERE id=:id AND status='pending'
RETURNING *;
""".strip()
async def _op() -> dict[str, Any] | None:
async with engine.begin() as conn:
result = await conn.exec_driver_sql(sql, {"id": int(job_id), "worker_id": worker_id, "now": now})
row = result.mappings().first()
return dict(row) if row else None
return await with_sqlite_busy_retry(_op)
def _parse_import_text(
text: str, *, preview_limit: int = 20
) -> tuple[int, int, int, int, list[ImportErrorItem], list[dict[str, Any]]]:
total = 0
accepted = 0
deduped = 0
error_total = 0
errors: list[ImportErrorItem] = []
seen: set[tuple[int, int]] = set()
preview: list[dict[str, Any]] = []
for line_no, raw in enumerate(text.splitlines(), start=1):
url = raw.strip()
if not url:
continue
total += 1
try:
parsed = parse_pixiv_original_url(url)
except Exception as exc:
error_total += 1
if len(errors) < 200:
errors.append(
ImportErrorItem(
line=line_no,
url=url,
code=ErrorCode.UNSUPPORTED_URL.value,
message=str(exc) or ErrorCode.UNSUPPORTED_URL.value,
)
)
continue
key = (parsed.illust_id, parsed.page_index)
if key in seen:
deduped += 1
continue
seen.add(key)
accepted += 1
if len(preview) < int(preview_limit):
preview.append(
{
"illust_id": parsed.illust_id,
"page_index": parsed.page_index,
"ext": parsed.ext,
"url": url,
}
)
return total, accepted, deduped, error_total, errors, preview
def _parse_bool(value: Any, *, default: bool = False) -> bool:
if value is None:
return default
if isinstance(value, bool):
return value
if isinstance(value, int) and value in (0, 1):
return bool(value)
if isinstance(value, str):
v = value.strip().lower()
if v in {"1", "true", "yes", "y", "on"}:
return True
if v in {"0", "false", "no", "n", "off"}:
return False
return default
def _validate_import_create(data: dict[str, Any]) -> ImportCreateRequest:
try:
return ImportCreateRequest(**data)
except ValidationError as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="导入请求体无效", status_code=400) from exc
async def _load_import_request(request: Request) -> ImportCreateRequest:
content_type = (request.headers.get("content-type") or "").lower()
max_bytes = _max_import_text_bytes()
if content_type.startswith("application/json"):
data = await request.json()
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
body = _validate_import_create(data)
if len(body.text.encode("utf-8", errors="ignore")) > max_bytes:
raise ApiError(code=ErrorCode.PAYLOAD_TOO_LARGE, message="Payload too large", status_code=413)
return body
if content_type.startswith("multipart/form-data"):
form = await request.form()
file_obj = form.get("file")
if not isinstance(file_obj, UploadFile):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing file", status_code=400)
filename = (file_obj.filename or "").strip().lower()
if filename and not filename.endswith(".txt"):
raise ApiError(code=ErrorCode.INVALID_UPLOAD_TYPE, message="Unsupported upload type", status_code=400)
raw = await file_obj.read()
if len(raw) > max_bytes:
raise ApiError(code=ErrorCode.PAYLOAD_TOO_LARGE, message="Payload too large", status_code=413)
text = raw.decode("utf-8", errors="replace")
return _validate_import_create(
{
"text": text,
"dry_run": _parse_bool(form.get("dry_run"), default=False),
"hydrate_on_import": _parse_bool(form.get("hydrate_on_import"), default=False),
"source": str(form.get("source") or "manual"),
}
)
raise ApiError(code=ErrorCode.INVALID_UPLOAD_TYPE, message="Unsupported content type", status_code=400)
@router.post("/imports")
async def create_import(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
body = await _load_import_request(request)
rid = get_or_create_request_id(request)
total, accepted, deduped, error_total, errors, preview = _parse_import_text(body.text, preview_limit=20)
if body.dry_run:
return {
"ok": True,
"import_id": "",
"job_id": "",
"accepted": accepted,
"deduped": deduped,
"errors": [asdict(e) for e in errors[:200]],
"preview": preview,
"request_id": rid,
}
engine = request.app.state.engine
Session = create_sessionmaker(engine)
settings = request.app.state.settings
db_dir = get_sqlite_db_dir(settings.database_url)
payload_dir = db_dir / "imports_payloads"
payload_dir.mkdir(parents=True, exist_ok=True)
payload_path = payload_dir / f"import_payload_{uuid4().hex}.txt"
try:
payload_path.write_text(body.text, encoding="utf-8")
except OSError as exc:
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="写入导入内容失败", status_code=500) from exc
file_ref = make_file_ref(payload_path, base_dir=db_dir)
async def _op() -> tuple[int, int]:
async with Session() as session:
imp = Import(created_by=str(_claims.get("sub") or ""), source=body.source)
session.add(imp)
await session.flush()
imp.total = int(total)
imp.accepted = int(accepted)
imp.success = 0
imp.failed = int(error_total)
imp.detail_json = json.dumps(
{
"deduped": int(deduped),
"errors": [asdict(e) for e in errors[:200]],
},
ensure_ascii=False,
)
job = JobRow(
type="import_images",
status="pending",
payload_json=json.dumps(
{
"import_id": int(imp.id),
"file_ref": file_ref,
"hydrate_on_import": bool(body.hydrate_on_import),
},
ensure_ascii=False,
separators=(",", ":"),
),
ref_type="import",
ref_id=str(imp.id),
)
session.add(job)
await session.flush()
await session.commit()
return int(imp.id), int(job.id)
try:
import_id, job_id = await with_sqlite_busy_retry(_op)
except Exception:
try:
payload_path.unlink(missing_ok=True)
except Exception:
pass
raise
inline_max = _import_inline_max_accepted()
executed_inline = False
if accepted > 0 and accepted <= inline_max:
now = iso_utc_ms()
actor = str(_claims.get("sub") or "admin").strip() or "admin"
worker_id = f"inline-import:{actor}"
claimed = await _claim_job_by_id(engine, job_id=int(job_id), worker_id=worker_id, now=now)
if claimed is not None:
dispatcher = JobDispatcher()
dispatcher.register("import_images", build_import_images_handler(engine))
await execute_claimed_job(engine, dispatcher, job_row=claimed, worker_id=worker_id)
executed_inline = True
return {
"ok": True,
"import_id": str(import_id),
"job_id": str(job_id),
"executed_inline": executed_inline,
"accepted": accepted,
"deduped": deduped,
"errors": [asdict(e) for e in errors[:200]],
"preview": preview,
"request_id": rid,
}
@router.post("/imports/{import_id}/rollback")
async def rollback_import(
import_id: int,
body: ImportRollbackRequest,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if import_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid import id", status_code=400)
rid = get_or_create_request_id(request)
target_status = 2 if body.mode == "disable" else 4
now_expr = sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))")
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> int:
async with Session() as session:
imp = await session.get(Import, import_id)
if imp is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Import not found", status_code=404)
result = await session.execute(
sa.update(Image)
.where(Image.created_import_id == import_id)
.values(status=target_status, updated_at=now_expr)
)
updated = int(result.rowcount or 0)
await session.commit()
return int(updated)
updated = await with_sqlite_busy_retry(_op)
return {
"ok": True,
"mode": body.mode,
"updated": updated,
"request_id": rid,
}
@router.get("/imports/{import_id}")
async def get_import(
import_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if import_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid import id", status_code=400)
rid = get_or_create_request_id(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
imp = await session.get(Import, import_id)
if imp is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Import not found", status_code=404)
job = (
(
await session.execute(
sa.select(JobRow)
.where(JobRow.ref_type == "import", JobRow.ref_id == str(import_id))
.order_by(JobRow.id.desc())
.limit(1)
)
)
.scalars()
.first()
)
detail: dict[str, Any] = {}
if imp.detail_json:
try:
parsed = json.loads(imp.detail_json)
if isinstance(parsed, dict):
detail = parsed
except Exception:
detail = {}
return {
"ok": True,
"item": {
"import": {
"id": str(imp.id),
"created_at": imp.created_at,
"created_by": imp.created_by,
"source": imp.source,
"total": int(imp.total or 0),
"accepted": int(imp.accepted or 0),
"success": int(imp.success or 0),
"failed": int(imp.failed or 0),
},
"job": (
{
"id": str(job.id),
"type": job.type,
"status": job.status,
"attempt": job.attempt,
"max_attempts": job.max_attempts,
"last_error": job.last_error,
}
if job is not None
else None
),
"detail": detail,
},
"request_id": rid,
}
+251
View File
@@ -0,0 +1,251 @@
from __future__ import annotations
import json
from typing import Any
import sqlalchemy as sa
from fastapi import APIRouter, Depends, Request
from app.api.admin.deps import get_admin_claims
from app.core.errors import ApiError, ErrorCode
from app.core.request_id import get_or_create_request_id
from app.core.time import iso_utc_ms
from app.db.models.jobs import JobRow
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
router = APIRouter()
_ALLOWED_JOB_STATUSES = {"pending", "running", "paused", "canceled", "completed", "failed", "dlq"}
@router.get("/jobs")
async def list_jobs(
request: Request,
limit: int = 50,
cursor: str | None = None,
status: str | None = None,
type: str | None = None,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if limit < 1 or limit > 200:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
cursor_i: int | None = None
cursor_raw = (cursor or "").strip()
if cursor_raw:
if not cursor_raw.isdigit():
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
cursor_i = int(cursor_raw)
if cursor_i <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
status_norm = (status or "").strip().lower() or None
if status_norm is not None and status_norm not in _ALLOWED_JOB_STATUSES:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported status", status_code=400)
type_norm = (type or "").strip() or None
if type_norm is not None and len(type_norm) > 100:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported type", status_code=400)
rid = get_or_create_request_id(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
stmt = sa.select(JobRow).order_by(JobRow.id.desc()).limit(limit + 1)
if cursor_i is not None:
stmt = stmt.where(JobRow.id < cursor_i)
if status_norm is not None:
stmt = stmt.where(JobRow.status == status_norm)
if type_norm is not None:
stmt = stmt.where(JobRow.type == type_norm)
async with Session() as session:
rows = ((await session.execute(stmt)).scalars().all())
items_rows = rows[:limit]
next_cursor = int(items_rows[-1].id) if len(rows) > limit and items_rows else None
items = [
{
"id": str(row.id),
"type": row.type,
"status": row.status,
"priority": int(row.priority),
"run_after": row.run_after,
"attempt": int(row.attempt),
"max_attempts": int(row.max_attempts),
"last_error": row.last_error,
"locked_by": row.locked_by,
"locked_at": row.locked_at,
"ref_type": row.ref_type,
"ref_id": row.ref_id,
"created_at": row.created_at,
"updated_at": row.updated_at,
}
for row in items_rows
]
return {
"ok": True,
"items": items,
"next_cursor": str(next_cursor) if next_cursor is not None else "",
"request_id": rid,
}
@router.get("/jobs/{job_id}")
async def get_job(
job_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if job_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid job id", status_code=400)
rid = get_or_create_request_id(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
row = await session.get(JobRow, job_id)
if row is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Job not found", status_code=404)
payload_json = str(row.payload_json or "")
payload: Any = None
if payload_json.strip():
try:
payload = json.loads(payload_json)
except Exception:
payload = None
return {
"ok": True,
"item": {
"id": str(row.id),
"type": row.type,
"status": row.status,
"priority": int(row.priority),
"run_after": row.run_after,
"attempt": int(row.attempt),
"max_attempts": int(row.max_attempts),
"payload": payload,
"payload_json": payload_json,
"last_error": row.last_error,
"locked_by": row.locked_by,
"locked_at": row.locked_at,
"ref_type": row.ref_type,
"ref_id": row.ref_id,
"created_at": row.created_at,
"updated_at": row.updated_at,
},
"request_id": rid,
}
@router.post("/jobs/{job_id}/retry")
async def retry_job(
job_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if job_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid job id", status_code=400)
rid = get_or_create_request_id(request)
now = iso_utc_ms()
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
async with Session() as session:
row = await session.get(JobRow, job_id)
if row is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Job not found", status_code=404)
if row.status == "running":
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Job is running", status_code=400)
row.status = "pending"
row.run_after = None
row.locked_by = None
row.locked_at = None
row.updated_at = now
await session.commit()
return {"ok": True, "job_id": str(job_id), "status": "pending", "request_id": rid}
return await with_sqlite_busy_retry(_op)
@router.post("/jobs/{job_id}/cancel")
async def cancel_job(
job_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if job_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid job id", status_code=400)
rid = get_or_create_request_id(request)
now = iso_utc_ms()
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
async with Session() as session:
row = await session.get(JobRow, job_id)
if row is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Job not found", status_code=404)
row.status = "canceled"
row.locked_by = None
row.locked_at = None
row.updated_at = now
await session.commit()
return {"ok": True, "job_id": str(job_id), "status": "canceled", "request_id": rid}
return await with_sqlite_busy_retry(_op)
@router.post("/jobs/{job_id}/move-to-dlq")
async def move_job_to_dlq(
job_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if job_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid job id", status_code=400)
rid = get_or_create_request_id(request)
now = iso_utc_ms()
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
async with Session() as session:
row = await session.get(JobRow, job_id)
if row is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Job not found", status_code=404)
row.status = "dlq"
row.run_after = None
row.locked_by = None
row.locked_at = None
row.updated_at = now
await session.commit()
return {"ok": True, "job_id": str(job_id), "status": "dlq", "request_id": rid}
return await with_sqlite_busy_retry(_op)
+118
View File
@@ -0,0 +1,118 @@
from __future__ import annotations
from typing import Any
from fastapi import APIRouter, Depends, Request
from app.api.admin.deps import get_admin_claims
from app.core.errors import ApiError, ErrorCode
from app.core.request_id import get_or_create_request_id
from app.db.request_logs_cleanup import (
DEFAULT_REQUEST_LOGS_CHUNK_SIZE,
DEFAULT_REQUEST_LOGS_KEEP_DAYS,
DEFAULT_REQUEST_LOGS_MAX_DELETE_ROWS,
cleanup_request_logs,
preview_request_logs_cleanup,
)
router = APIRouter()
def _parse_bool(value: Any, *, default: bool) -> bool:
if value is None:
return default
if isinstance(value, bool):
return value
if isinstance(value, int) and value in (0, 1):
return bool(value)
if isinstance(value, str):
v = value.strip().lower()
if v in {"1", "true", "yes", "y", "on"}:
return True
if v in {"0", "false", "no", "n", "off"}:
return False
return default
async def _load_cleanup_request_logs_json(request: Request) -> dict[str, Any]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
keep_days_raw = data.get("keep_days", DEFAULT_REQUEST_LOGS_KEEP_DAYS)
try:
keep_days = int(keep_days_raw)
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported keep_days", status_code=400) from exc
if keep_days < 0 or keep_days > 36500:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported keep_days", status_code=400)
max_delete_rows_raw = data.get("max_delete_rows", DEFAULT_REQUEST_LOGS_MAX_DELETE_ROWS)
try:
max_delete_rows = int(max_delete_rows_raw)
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported max_delete_rows", status_code=400) from exc
if max_delete_rows < 1 or max_delete_rows > 10_000_000:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported max_delete_rows", status_code=400)
chunk_size_raw = data.get("chunk_size", DEFAULT_REQUEST_LOGS_CHUNK_SIZE)
try:
chunk_size = int(chunk_size_raw)
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported chunk_size", status_code=400) from exc
if chunk_size < 1 or chunk_size > 100_000:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported chunk_size", status_code=400)
dry_run = _parse_bool(data.get("dry_run"), default=False)
return {
"keep_days": int(keep_days),
"max_delete_rows": int(max_delete_rows),
"chunk_size": int(chunk_size),
"dry_run": bool(dry_run),
}
@router.post("/maintenance/request-logs/cleanup")
async def cleanup_request_logs_endpoint(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
cfg = await _load_cleanup_request_logs_json(request)
engine = request.app.state.engine
if bool(cfg["dry_run"]):
preview = await preview_request_logs_cleanup(
engine,
keep_days=int(cfg["keep_days"]),
max_delete_rows=int(cfg["max_delete_rows"]),
)
return {
"ok": True,
"dry_run": True,
"cutoff": preview.cutoff,
"would_delete": int(preview.would_delete),
"has_more": bool(preview.has_more),
"request_id": rid,
}
result = await cleanup_request_logs(
engine,
keep_days=int(cfg["keep_days"]),
max_delete_rows=int(cfg["max_delete_rows"]),
chunk_size=int(cfg["chunk_size"]),
)
return {
"ok": True,
"dry_run": False,
"cutoff": result.cutoff,
"deleted": int(result.deleted),
"has_more": bool(result.has_more),
"request_id": rid,
}
File diff suppressed because it is too large. Load diff
+318
View File
@@ -0,0 +1,318 @@
from __future__ import annotations
from typing import Any
import sqlalchemy as sa
from fastapi import APIRouter, Depends, Request
from sqlalchemy.exc import IntegrityError
from app.api.admin.deps import get_admin_claims
from app.core.errors import ApiError, ErrorCode
from app.core.request_id import get_or_create_request_id
from app.db.models.proxy_endpoints import ProxyEndpoint
from app.db.models.proxy_pool_endpoints import ProxyPoolEndpoint
from app.db.models.proxy_pools import ProxyPool
from app.db.session import create_sessionmaker
router = APIRouter()
def _parse_bool(value: Any, *, default: bool) -> bool:
if value is None:
return default
if isinstance(value, bool):
return value
if isinstance(value, int) and value in (0, 1):
return bool(value)
if isinstance(value, str):
v = value.strip().lower()
if v in {"1", "true", "yes", "y", "on"}:
return True
if v in {"0", "false", "no", "n", "off"}:
return False
return default
async def _load_create_json(request: Request) -> dict[str, Any]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
name = str(data.get("name") or "").strip()
if not name:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing name", status_code=400)
if len(name) > 100:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported name", status_code=400)
desc_raw = data.get("description")
description = str(desc_raw).strip() if desc_raw is not None else None
description = description if description else None
enabled = _parse_bool(data.get("enabled"), default=True)
return {"name": name, "description": description, "enabled": enabled}
async def _load_update_json(request: Request) -> dict[str, Any]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
out: dict[str, Any] = {}
if "name" in data:
name = str(data.get("name") or "").strip()
if not name:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported name", status_code=400)
if len(name) > 100:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported name", status_code=400)
out["name"] = name
if "description" in data:
desc_raw = data.get("description")
description = str(desc_raw).strip() if desc_raw is not None else None
out["description"] = description if description else None
if "enabled" in data:
out["enabled"] = _parse_bool(data.get("enabled"), default=True)
if not out:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing fields", status_code=400)
return out
async def _load_set_endpoints_json(request: Request) -> list[dict[str, Any]]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
raw_items = data.get("items")
if raw_items is None:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing items", status_code=400)
if not isinstance(raw_items, list):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid items", status_code=400)
out: list[dict[str, Any]] = []
seen: set[int] = set()
for raw in raw_items:
if not isinstance(raw, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid items", status_code=400)
try:
endpoint_id = int(raw.get("endpoint_id"))
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid endpoint_id", status_code=400) from exc
if endpoint_id <= 0 or endpoint_id in seen:
continue
seen.add(endpoint_id)
enabled = _parse_bool(raw.get("enabled"), default=True)
weight_raw = raw.get("weight", 1)
try:
weight = int(weight_raw)
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid weight", status_code=400) from exc
if weight < 0 or weight > 1000:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid weight", status_code=400)
out.append({"endpoint_id": endpoint_id, "enabled": enabled, "weight": weight})
return out
@router.get("/proxy-pools")
async def list_proxy_pools(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
pools = (
(await session.execute(sa.select(ProxyPool).order_by(ProxyPool.id.desc())))
.scalars()
.all()
)
items = [
{
"id": str(p.id),
"name": p.name,
"description": p.description,
"enabled": bool(p.enabled),
}
for p in pools
]
return {"ok": True, "items": items, "request_id": rid}
@router.post("/proxy-pools")
async def create_proxy_pool(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
body = await _load_create_json(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
row = ProxyPool(
name=str(body["name"]),
description=body["description"],
enabled=1 if bool(body["enabled"]) else 0,
)
session.add(row)
try:
await session.commit()
except IntegrityError as exc:
await session.rollback()
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Proxy pool name exists", status_code=400) from exc
await session.refresh(row)
return {"ok": True, "pool_id": str(row.id), "request_id": rid}
@router.put("/proxy-pools/{pool_id}")
async def update_proxy_pool(
pool_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if pool_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid pool id", status_code=400)
rid = get_or_create_request_id(request)
body = await _load_update_json(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
row = await session.get(ProxyPool, pool_id)
if row is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Proxy pool not found", status_code=404)
if "name" in body:
row.name = str(body["name"])
if "description" in body:
row.description = body["description"]
if "enabled" in body:
row.enabled = 1 if bool(body["enabled"]) else 0
try:
await session.commit()
except IntegrityError as exc:
await session.rollback()
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Proxy pool name exists", status_code=400) from exc
return {"ok": True, "pool_id": str(pool_id), "request_id": rid}
@router.post("/proxy-pools/{pool_id}/endpoints")
async def set_proxy_pool_endpoints(
pool_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if pool_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid pool id", status_code=400)
rid = get_or_create_request_id(request)
items = await _load_set_endpoints_json(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
created = 0
updated = 0
removed = 0
async with Session() as session:
pool = await session.get(ProxyPool, pool_id)
if pool is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Proxy pool not found", status_code=404)
endpoint_ids = [int(x["endpoint_id"]) for x in items]
if endpoint_ids:
existing = (
(
await session.execute(
sa.select(ProxyEndpoint.id).where(ProxyEndpoint.id.in_(endpoint_ids))
)
)
.scalars()
.all()
)
existing_set = {int(x) for x in existing}
missing = [eid for eid in endpoint_ids if eid not in existing_set]
if missing:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unknown endpoint_id", status_code=400)
current_rows = (
(
await session.execute(
sa.select(ProxyPoolEndpoint).where(ProxyPoolEndpoint.pool_id == pool_id)
)
)
.scalars()
.all()
)
current_by_eid = {int(r.endpoint_id): r for r in current_rows}
keep: set[int] = set()
for item in items:
eid = int(item["endpoint_id"])
keep.add(eid)
row = current_by_eid.get(eid)
if row is None:
session.add(
ProxyPoolEndpoint(
pool_id=pool_id,
endpoint_id=eid,
enabled=1 if bool(item["enabled"]) else 0,
weight=int(item["weight"]),
)
)
created += 1
else:
row.enabled = 1 if bool(item["enabled"]) else 0
row.weight = int(item["weight"])
updated += 1
for eid, row in current_by_eid.items():
if eid in keep:
continue
await session.delete(row)
removed += 1
await session.commit()
return {
"ok": True,
"pool_id": str(pool_id),
"created": created,
"updated": updated,
"removed": removed,
"request_id": rid,
}
+36
View File
@@ -0,0 +1,36 @@
from __future__ import annotations
from fastapi import APIRouter
from app.api.admin.auth import router as auth_router
from app.api.admin.api_keys import router as api_keys_router
from app.api.admin.audit import router as audit_router
from app.api.admin.bindings import router as bindings_router
from app.api.admin.hydration_runs import router as hydration_runs_router
from app.api.admin.images import router as images_router
from app.api.admin.imports import router as imports_router
from app.api.admin.jobs import router as jobs_router
from app.api.admin.maintenance import router as maintenance_router
from app.api.admin.proxies import router as proxies_router
from app.api.admin.proxy_pools import router as proxy_pools_router
from app.api.admin.settings import router as settings_router
from app.api.admin.stats import router as stats_router
from app.api.admin.summary import router as summary_router
from app.api.admin.tokens import router as tokens_router
router = APIRouter(prefix="/admin/api")
router.include_router(auth_router)
router.include_router(api_keys_router)
router.include_router(audit_router)
router.include_router(bindings_router)
router.include_router(hydration_runs_router)
router.include_router(images_router)
router.include_router(imports_router)
router.include_router(jobs_router)
router.include_router(maintenance_router)
router.include_router(proxies_router)
router.include_router(proxy_pools_router)
router.include_router(settings_router)
router.include_router(stats_router)
router.include_router(summary_router)
router.include_router(tokens_router)
+444
View File
@@ -0,0 +1,444 @@
from __future__ import annotations
from typing import Any
from fastapi import APIRouter, Depends, Request
from app.api.admin.deps import get_admin_claims
from app.core.errors import ApiError, ErrorCode
from app.core.request_id import get_or_create_request_id
from app.core.runtime_settings import (
fetch_runtime_settings,
runtime_config_from_values,
set_runtime_setting,
)
router = APIRouter()
_DEFAULT_SCORE_WEIGHTS: dict[str, float] = {
"bookmark": 4.0,
"view": 0.5,
"comment": 2.0,
"pixels": 1.0,
"bookmark_rate": 3.0,
}
_DEFAULT_RECOMMENDATION: dict[str, Any] = {
"pick_mode": "weighted",
"temperature": 1.0,
"score_weights": dict(_DEFAULT_SCORE_WEIGHTS),
"multipliers": {
"ai": 1.0,
"non_ai": 1.0,
"unknown_ai": 1.0,
"illust": 1.0,
"manga": 1.0,
"ugoira": 1.0,
"unknown_illust_type": 1.0,
},
}
_DEFAULT_SETTINGS = {
"random": {
"default_attempts": 3,
"default_r18_strict": True,
"fail_cooldown_ms": 600_000,
"strategy": "quality",
"quality_samples": 5,
"recommendation": dict(_DEFAULT_RECOMMENDATION),
},
"proxy": {"allowlist_domains": []},
}
def _as_str_list(value: Any) -> list[str]:
if not isinstance(value, list):
return []
out: list[str] = []
seen: set[str] = set()
for item in value:
if not isinstance(item, str):
continue
v = item.strip()
if not v or v in seen:
continue
seen.add(v)
out.append(v)
return out
def _as_bool(value: Any) -> bool | None:
if isinstance(value, bool):
return value
if isinstance(value, int) and value in (0, 1):
return bool(value)
if isinstance(value, str):
v = value.strip().lower()
if v in {"true", "1", "yes", "y", "on"}:
return True
if v in {"false", "0", "no", "n", "off"}:
return False
return None
def _as_float(value: Any) -> float | None:
if value is None:
return None
if isinstance(value, bool):
return None
try:
return float(value)
except Exception:
return None
def _normalize_recommendation(value: Any, *, strict: bool) -> dict[str, Any]:
if value is None:
return dict(_DEFAULT_RECOMMENDATION)
if not isinstance(value, dict):
if strict:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random.recommendation", status_code=400)
return dict(_DEFAULT_RECOMMENDATION)
pick_mode_default = str(_DEFAULT_RECOMMENDATION["pick_mode"])
pick_mode_raw = value.get("pick_mode")
pick_mode = pick_mode_default
if pick_mode_raw is not None:
candidate = str(pick_mode_raw or "").strip().lower()
if candidate not in {"best", "weighted"}:
if strict:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random.recommendation.pick_mode", status_code=400)
else:
pick_mode = candidate
temperature_default = float(_DEFAULT_RECOMMENDATION["temperature"])
temperature = temperature_default
if "temperature" in value:
v = _as_float(value.get("temperature"))
if v is None:
if strict:
raise ApiError(
code=ErrorCode.BAD_REQUEST,
message="Invalid random.recommendation.temperature",
status_code=400,
)
else:
temperature = float(max(0.05, min(float(v), 100.0)))
score_weights_default = _DEFAULT_RECOMMENDATION["score_weights"]
score_weights_raw = value.get("score_weights")
if score_weights_raw is None:
score_weights_obj: dict[str, Any] = {}
elif not isinstance(score_weights_raw, dict):
if strict:
raise ApiError(
code=ErrorCode.BAD_REQUEST,
message="Invalid random.recommendation.score_weights",
status_code=400,
)
score_weights_obj = {}
else:
score_weights_obj = score_weights_raw
if strict:
for k in score_weights_obj.keys():
if str(k) not in _DEFAULT_SCORE_WEIGHTS:
raise ApiError(
code=ErrorCode.BAD_REQUEST,
message="Invalid random.recommendation.score_weights",
status_code=400,
)
score_weights: dict[str, float] = {}
for key, default_value in _DEFAULT_SCORE_WEIGHTS.items():
if key in score_weights_obj:
v = _as_float(score_weights_obj.get(key))
if v is None:
if strict:
raise ApiError(
code=ErrorCode.BAD_REQUEST,
message="Invalid random.recommendation.score_weights",
status_code=400,
)
v = float(default_value)
score_weights[key] = float(max(-100.0, min(float(v), 100.0)))
else:
score_weights[key] = float(default_value)
multipliers_default = _DEFAULT_RECOMMENDATION["multipliers"]
multipliers_raw = value.get("multipliers")
if multipliers_raw is None:
multipliers_obj: dict[str, Any] = {}
elif not isinstance(multipliers_raw, dict):
if strict:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random.recommendation.multipliers", status_code=400)
multipliers_obj = {}
else:
multipliers_obj = multipliers_raw
if strict:
for k in multipliers_obj.keys():
if str(k) not in multipliers_default:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random.recommendation.multipliers", status_code=400)
multipliers: dict[str, float] = {}
for key, default_value in multipliers_default.items():
if key in multipliers_obj:
v = _as_float(multipliers_obj.get(key))
if v is None:
if strict:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random.recommendation.multipliers", status_code=400)
v = float(default_value)
multipliers[key] = float(max(0.0, min(float(v), 100.0)))
else:
multipliers[key] = float(default_value)
return {
"pick_mode": pick_mode,
"temperature": temperature,
"score_weights": score_weights,
"multipliers": multipliers,
}
async def _load_settings_json(request: Request) -> dict[str, Any]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
settings = data.get("settings") if "settings" in data else data
if not isinstance(settings, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid settings", status_code=400)
if not settings:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing fields", status_code=400)
return settings
@router.get("/settings")
async def get_settings(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
engine = request.app.state.engine
values = await fetch_runtime_settings(engine)
runtime = runtime_config_from_values(values)
random_defaults = dict(_DEFAULT_SETTINGS["random"])
if isinstance(runtime.random_defaults, dict):
for k in list(random_defaults.keys()):
if k == "recommendation":
continue
if k in runtime.random_defaults:
random_defaults[k] = runtime.random_defaults[k]
random_defaults["recommendation"] = _normalize_recommendation(runtime.random_defaults.get("recommendation"), strict=False)
else:
random_defaults["recommendation"] = _normalize_recommendation(None, strict=False)
return {
"ok": True,
"settings": {
"proxy": {
"enabled": bool(runtime.proxy_enabled),
"fail_closed": bool(runtime.proxy_fail_closed),
"route_mode": runtime.proxy_route_mode,
"allowlist_domains": list(runtime.proxy_allowlist_domains),
"default_pool_id": str(runtime.proxy_default_pool_id) if runtime.proxy_default_pool_id is not None else "",
"route_pools": {k: str(v) for k, v in runtime.proxy_route_pools.items()},
},
"random": random_defaults,
"security": {"hide_origin_url_in_public_json": bool(runtime.hide_origin_url_in_public_json)},
"rate_limit": dict(runtime.rate_limit),
},
"request_id": rid,
}
@router.put("/settings")
async def update_settings(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
rid = get_or_create_request_id(request)
body = await _load_settings_json(request)
actor = str(_claims.get("sub") or "admin").strip() or "admin"
updated_by = f"admin:{actor}"
updates: list[tuple[str, Any]] = []
proxy_enabled_override: bool | None = None
proxy_fail_closed_override: bool | None = None
proxy = body.get("proxy")
if proxy is not None:
if not isinstance(proxy, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy", status_code=400)
if "enabled" in proxy:
v = _as_bool(proxy.get("enabled"))
if v is None:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.enabled", status_code=400)
proxy_enabled_override = bool(v)
updates.append(("proxy.enabled", bool(v)))
if "fail_closed" in proxy:
v = _as_bool(proxy.get("fail_closed"))
if v is None:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.fail_closed", status_code=400)
proxy_fail_closed_override = bool(v)
updates.append(("proxy.fail_closed", bool(v)))
if "route_mode" in proxy:
route_mode = str(proxy.get("route_mode") or "").strip().lower()
if route_mode not in {"pixiv_only", "all", "allowlist", "off"}:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.route_mode", status_code=400)
updates.append(("proxy.route_mode", route_mode))
if "allowlist_domains" in proxy:
domains = _as_str_list(proxy.get("allowlist_domains"))
if len(domains) > 200 or any(len(d) > 200 for d in domains):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.allowlist_domains", status_code=400)
updates.append(("proxy.allowlist_domains", domains))
if "default_pool_id" in proxy:
raw = proxy.get("default_pool_id")
if raw is None or raw == "":
updates.append(("proxy.default_pool_id", None))
else:
try:
pool_id = int(raw)
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.default_pool_id", status_code=400) from exc
if pool_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.default_pool_id", status_code=400)
updates.append(("proxy.default_pool_id", int(pool_id)))
if "route_pools" in proxy:
raw = proxy.get("route_pools")
if raw is None:
updates.append(("proxy.route_pools", {}))
else:
if not isinstance(raw, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.route_pools", status_code=400)
route_pools: dict[str, int] = {}
if len(raw) > 200:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.route_pools", status_code=400)
for k, v in raw.items():
key = str(k or "").strip().lower().strip(".")
if not key or len(key) > 200:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.route_pools", status_code=400)
try:
pool_id = int(v)
except Exception as exc:
raise ApiError(
code=ErrorCode.BAD_REQUEST,
message="Invalid proxy.route_pools",
status_code=400,
) from exc
if pool_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid proxy.route_pools", status_code=400)
route_pools[key] = int(pool_id)
updates.append(("proxy.route_pools", route_pools))
random = body.get("random")
if random is not None:
if not isinstance(random, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random", status_code=400)
defaults: dict[str, Any] = {}
for key in ("default_attempts", "default_r18_strict", "fail_cooldown_ms", "strategy", "quality_samples"):
if key not in random:
continue
if key in {"default_attempts", "fail_cooldown_ms", "quality_samples"}:
try:
n = int(random.get(key))
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message=f"Invalid random.{key}", status_code=400) from exc
if key == "quality_samples":
if n < 1 or n > 1000:
raise ApiError(code=ErrorCode.BAD_REQUEST, message=f"Invalid random.{key}", status_code=400)
elif n < 0 or n > 10_000_000:
raise ApiError(code=ErrorCode.BAD_REQUEST, message=f"Invalid random.{key}", status_code=400)
defaults[key] = n
elif key == "strategy":
s = str(random.get(key) or "").strip().lower()
if s not in {"quality", "random"}:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid random.strategy", status_code=400)
defaults[key] = s
else:
v = _as_bool(random.get(key))
if v is None:
raise ApiError(code=ErrorCode.BAD_REQUEST, message=f"Invalid random.{key}", status_code=400)
defaults[key] = bool(v)
if "recommendation" in random:
defaults["recommendation"] = _normalize_recommendation(random.get("recommendation"), strict=True)
if defaults:
values = await fetch_runtime_settings(request.app.state.engine)
runtime = runtime_config_from_values(values)
existing_raw = runtime.random_defaults if isinstance(runtime.random_defaults, dict) else {}
if existing_raw:
merged = dict(existing_raw)
merged.update(defaults)
defaults = merged
updates.append(("random.defaults", defaults))
security = body.get("security")
if security is not None:
if not isinstance(security, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid security", status_code=400)
if "hide_origin_url_in_public_json" in security:
v = _as_bool(security.get("hide_origin_url_in_public_json"))
if v is None:
raise ApiError(
code=ErrorCode.BAD_REQUEST,
message="Invalid security.hide_origin_url_in_public_json",
status_code=400,
)
updates.append(("security.hide_origin_url_in_public_json", bool(v)))
rate_limit = body.get("rate_limit")
if rate_limit is not None:
if not isinstance(rate_limit, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid rate_limit", status_code=400)
for key, value in rate_limit.items():
k = str(key or "").strip()
if not k or len(k) > 100:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid rate_limit key", status_code=400)
updates.append((f"rate_limit.{k}", value))
if not updates:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing fields", status_code=400)
engine = request.app.state.engine
if proxy_enabled_override is not None or proxy_fail_closed_override is not None:
values = await fetch_runtime_settings(engine)
runtime = runtime_config_from_values(values)
desired_enabled = proxy_enabled_override if proxy_enabled_override is not None else bool(runtime.proxy_enabled)
desired_fail_closed = (
proxy_fail_closed_override if proxy_fail_closed_override is not None else bool(runtime.proxy_fail_closed)
)
if desired_enabled and desired_fail_closed:
async with engine.connect() as conn:
result = await conn.exec_driver_sql("SELECT COUNT(*) FROM proxy_endpoints WHERE enabled=1;")
enabled_proxy_count = int(result.scalar_one())
if enabled_proxy_count <= 0:
raise ApiError(
code=ErrorCode.PROXY_REQUIRED,
message="Proxy required (fail-closed) but no enabled proxies",
status_code=400,
)
for key, value in updates:
await set_runtime_setting(engine, key=key, value=value, updated_by=updated_by)
return {"ok": True, "updated": len(updates), "request_id": rid}
+40
View File
@@ -0,0 +1,40 @@
from __future__ import annotations
from dataclasses import asdict
from typing import Any
from fastapi import APIRouter, Depends, Request
from app.api.admin.deps import get_admin_claims
from app.core.request_id import get_or_create_request_id
router = APIRouter()
@router.get("/stats/random")
async def get_random_stats(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
stats = getattr(request.app.state, "random_request_stats", None)
if stats is None:
snapshot = {
"total_requests": 0,
"total_ok": 0,
"total_error": 0,
"in_flight": 0,
"window_seconds": 60,
"last_window_requests": 0,
"last_window_ok": 0,
"last_window_error": 0,
"last_window_success_rate": 0.0,
}
else:
snap = await stats.snapshot()
snapshot = asdict(snap)
return {"ok": True, "stats": snapshot, "request_id": rid}
+156
View File
@@ -0,0 +1,156 @@
from __future__ import annotations
import json
from typing import Any
from fastapi import APIRouter, Depends, Request
from app.api.admin.deps import get_admin_claims
from app.core.request_id import get_or_create_request_id
from app.db.session import with_sqlite_busy_retry
router = APIRouter()
def _worker_last_seen_from_value_json(value_json: str | None) -> str | None:
if value_json is None:
return None
raw = str(value_json or "").strip()
if not raw:
return None
try:
data = json.loads(raw)
except Exception:
return None
if isinstance(data, dict):
at = data.get("at")
return str(at) if isinstance(at, str) and at.strip() else None
if isinstance(data, str):
return data.strip() or None
return None
@router.get("/summary")
async def get_summary(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
engine = request.app.state.engine
async def _op() -> dict[str, Any]:
async with engine.connect() as conn:
images_total = int((await conn.exec_driver_sql("SELECT COUNT(*) FROM images;")).scalar_one())
images_enabled = int(
(await conn.exec_driver_sql("SELECT COUNT(*) FROM images WHERE status=1;")).scalar_one()
)
missing_tags = int(
(
await conn.exec_driver_sql(
"""
SELECT COUNT(*)
FROM images
WHERE status=1
AND id NOT IN (SELECT DISTINCT image_id FROM image_tags);
""".strip()
)
).scalar_one()
)
missing_geometry = int(
(
await conn.exec_driver_sql(
"SELECT COUNT(*) FROM images WHERE status=1 AND (width IS NULL OR height IS NULL);"
)
).scalar_one()
)
missing_r18 = int(
(await conn.exec_driver_sql("SELECT COUNT(*) FROM images WHERE status=1 AND x_restrict IS NULL;")).scalar_one()
)
missing_ai = int(
(await conn.exec_driver_sql("SELECT COUNT(*) FROM images WHERE status=1 AND ai_type IS NULL;")).scalar_one()
)
missing_illust_type = int(
(await conn.exec_driver_sql("SELECT COUNT(*) FROM images WHERE status=1 AND illust_type IS NULL;")).scalar_one()
)
missing_user = int(
(await conn.exec_driver_sql("SELECT COUNT(*) FROM images WHERE status=1 AND user_id IS NULL;")).scalar_one()
)
missing_title = int(
(
await conn.exec_driver_sql(
"SELECT COUNT(*) FROM images WHERE status=1 AND (title IS NULL OR TRIM(title)='');"
)
).scalar_one()
)
missing_created_at = int(
(
await conn.exec_driver_sql(
"SELECT COUNT(*) FROM images WHERE status=1 AND (created_at_pixiv IS NULL OR TRIM(created_at_pixiv)='');"
)
).scalar_one()
)
missing_popularity = int(
(
await conn.exec_driver_sql(
"SELECT COUNT(*) FROM images "
"WHERE status=1 AND (bookmark_count IS NULL OR view_count IS NULL OR comment_count IS NULL);"
)
).scalar_one()
)
tokens_total = int((await conn.exec_driver_sql("SELECT COUNT(*) FROM pixiv_tokens;")).scalar_one())
tokens_enabled = int(
(await conn.exec_driver_sql("SELECT COUNT(*) FROM pixiv_tokens WHERE enabled=1;")).scalar_one()
)
proxies_total = int((await conn.exec_driver_sql("SELECT COUNT(*) FROM proxy_endpoints;")).scalar_one())
proxies_enabled = int(
(await conn.exec_driver_sql("SELECT COUNT(*) FROM proxy_endpoints WHERE enabled=1;")).scalar_one()
)
pools_total = int((await conn.exec_driver_sql("SELECT COUNT(*) FROM proxy_pools;")).scalar_one())
pools_enabled = int(
(await conn.exec_driver_sql("SELECT COUNT(*) FROM proxy_pools WHERE enabled=1;")).scalar_one()
)
bindings_total = int((await conn.exec_driver_sql("SELECT COUNT(*) FROM token_proxy_bindings;")).scalar_one())
jobs_counts: dict[str, int] = {}
rows = (await conn.exec_driver_sql("SELECT status, COUNT(*) AS c FROM jobs GROUP BY status;")).fetchall()
for status, count in rows:
jobs_counts[str(status)] = int(count)
worker_last_seen_json = (
await conn.exec_driver_sql("SELECT value_json FROM runtime_settings WHERE key = ?;", ("worker.last_seen_at",))
).scalar_one_or_none()
worker_last_seen_at = _worker_last_seen_from_value_json(str(worker_last_seen_json) if worker_last_seen_json is not None else None)
return {
"images": {"total": images_total, "enabled": images_enabled},
"hydration": {
"enabled_images_total": images_enabled,
"missing": {
"tags": missing_tags,
"geometry": missing_geometry,
"r18": missing_r18,
"ai": missing_ai,
"illust_type": missing_illust_type,
"user": missing_user,
"title": missing_title,
"created_at": missing_created_at,
"popularity": missing_popularity,
},
},
"tokens": {"total": tokens_total, "enabled": tokens_enabled},
"proxies": {"endpoints_total": proxies_total, "endpoints_enabled": proxies_enabled},
"proxy_pools": {"total": pools_total, "enabled": pools_enabled},
"bindings": {"total": bindings_total},
"jobs": {"counts": jobs_counts},
"worker": {"last_seen_at": worker_last_seen_at},
}
counts = await with_sqlite_busy_retry(_op)
return {"ok": True, "counts": counts, "request_id": rid}
+450
View File
@@ -0,0 +1,450 @@
from __future__ import annotations
from datetime import datetime, timedelta, timezone
from typing import Any
import sqlalchemy as sa
from fastapi import APIRouter, Depends, Request
from app.api.admin.deps import get_admin_claims
from app.core.crypto import FieldEncryptor, mask_secret
from app.core.errors import ApiError, ErrorCode
from app.core.proxy_routing import select_proxy_uri_for_url
from app.core.request_id import get_or_create_request_id
from app.core.runtime_settings import load_runtime_config
from app.core.time import iso_utc_ms
from app.db.models.pixiv_tokens import PixivToken
from app.db.models.token_proxy_bindings import TokenProxyBinding
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
from app.pixiv.oauth import OAUTH_TOKEN_PATH, PixivOauthConfig, PixivOauthError, refresh_access_token
from app.pixiv.refresh_backoff import refresh_backoff_seconds
router = APIRouter()
def _parse_bool(value: Any, *, default: bool) -> bool:
if value is None:
return default
if isinstance(value, bool):
return value
if isinstance(value, int) and value in (0, 1):
return bool(value)
if isinstance(value, str):
v = value.strip().lower()
if v in {"1", "true", "yes", "y", "on"}:
return True
if v in {"0", "false", "no", "n", "off"}:
return False
return default
def _parse_bool_strict(value: Any) -> bool | None:
if isinstance(value, bool):
return value
if isinstance(value, int) and value in (0, 1):
return bool(value)
if isinstance(value, str):
v = value.strip().lower()
if v in {"1", "true", "yes", "y", "on"}:
return True
if v in {"0", "false", "no", "n", "off"}:
return False
return None
async def _load_create_token_json(request: Request) -> dict[str, Any]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
refresh_token = str(data.get("refresh_token") or "").strip()
if not refresh_token:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing refresh_token", status_code=400)
if len(refresh_token) > 2048:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported refresh_token", status_code=400)
label_raw = data.get("label")
label = str(label_raw).strip() if label_raw is not None else None
label = label if label else None
enabled = _parse_bool(data.get("enabled"), default=True)
weight_raw = data.get("weight", 1.0)
try:
weight = float(weight_raw)
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported weight", status_code=400) from exc
if weight < 0.0 or weight > 100.0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported weight", status_code=400)
return {
"label": label,
"enabled": enabled,
"weight": weight,
"refresh_token": refresh_token,
}
async def _load_update_token_json(request: Request) -> dict[str, Any]:
try:
data = await request.json()
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400) from exc
if not isinstance(data, dict):
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid JSON body", status_code=400)
if not data:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing fields", status_code=400)
out: dict[str, Any] = {}
if "label" in data:
label_raw = data.get("label")
if label_raw is None:
label = None
else:
label = str(label_raw).strip() or None
if label is not None and len(label) > 200:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported label", status_code=400)
out["label"] = label
if "enabled" in data:
enabled = _parse_bool_strict(data.get("enabled"))
if enabled is None:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported enabled", status_code=400)
out["enabled"] = bool(enabled)
if "weight" in data:
weight_raw = data.get("weight")
try:
weight = float(weight_raw)
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported weight", status_code=400) from exc
if weight < 0.0 or weight > 100.0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported weight", status_code=400)
out["weight"] = float(weight)
if not out:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Missing fields", status_code=400)
return out
@router.get("/tokens")
async def list_tokens(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
tokens = (
(
await session.execute(
sa.select(PixivToken).order_by(PixivToken.id.desc())
)
)
.scalars()
.all()
)
items = [
{
"id": str(t.id),
"label": t.label,
"enabled": bool(t.enabled),
"refresh_token_masked": t.refresh_token_masked,
"weight": float(t.weight),
"error_count": int(t.error_count or 0),
"backoff_until": t.backoff_until,
"last_ok_at": t.last_ok_at,
"last_fail_at": t.last_fail_at,
"last_error_code": t.last_error_code,
"last_error_msg": t.last_error_msg,
}
for t in tokens
]
return {"ok": True, "items": items, "request_id": rid}
@router.post("/tokens")
async def create_token(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
rid = get_or_create_request_id(request)
body = await _load_create_token_json(request)
settings = request.app.state.settings
try:
encryptor = FieldEncryptor.from_key(settings.field_encryption_key)
except Exception as exc:
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Encryption not configured", status_code=500) from exc
refresh_token = str(body["refresh_token"])
refresh_token_enc = encryptor.encrypt_text(refresh_token)
refresh_token_masked = mask_secret(refresh_token)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
row = PixivToken(
label=body["label"],
enabled=1 if bool(body["enabled"]) else 0,
refresh_token_enc=refresh_token_enc,
refresh_token_masked=refresh_token_masked,
weight=float(body["weight"]),
)
session.add(row)
await session.commit()
await session.refresh(row)
return {"ok": True, "token_id": str(row.id), "request_id": rid}
@router.put("/tokens/{token_id}")
async def update_token(
token_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if token_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid token id", status_code=400)
rid = get_or_create_request_id(request)
body = await _load_update_token_json(request)
now = iso_utc_ms()
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
row = await session.get(PixivToken, token_id)
if row is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Token not found", status_code=404)
if "label" in body:
row.label = body["label"]
if "enabled" in body:
row.enabled = 1 if bool(body["enabled"]) else 0
if "weight" in body:
row.weight = float(body["weight"])
row.updated_at = now
await session.commit()
return {"ok": True, "token_id": str(token_id), "request_id": rid}
@router.delete("/tokens/{token_id}")
async def delete_token(
token_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if token_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid token id", status_code=400)
rid = get_or_create_request_id(request)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
async with Session() as session:
row = await session.get(PixivToken, token_id)
if row is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Token not found", status_code=404)
await session.execute(sa.delete(TokenProxyBinding).where(TokenProxyBinding.token_id == int(token_id)))
await session.delete(row)
await session.commit()
return {"ok": True, "token_id": str(token_id), "request_id": rid}
return await with_sqlite_busy_retry(_op)
@router.post("/tokens/{token_id}/test-refresh")
async def test_refresh_token(
token_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if token_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid token id", status_code=400)
rid = get_or_create_request_id(request)
settings = request.app.state.settings
try:
encryptor = FieldEncryptor.from_key(settings.field_encryption_key)
except Exception as exc:
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Encryption not configured", status_code=500) from exc
client_id = (settings.pixiv_oauth_client_id or "").strip()
client_secret = (settings.pixiv_oauth_client_secret or "").strip()
if not client_id or not client_secret:
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Pixiv OAuth not configured", status_code=500)
config = PixivOauthConfig(
client_id=client_id,
client_secret=client_secret,
hash_secret=(settings.pixiv_oauth_hash_secret or "").strip() or None,
)
transport = getattr(request.app.state, "httpx_transport", None)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
row = await session.get(PixivToken, token_id)
if row is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Token not found", status_code=404)
try:
refresh_token = encryptor.decrypt_text(row.refresh_token_enc)
except Exception as exc:
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Invalid stored token", status_code=500) from exc
now = iso_utc_ms()
try:
runtime = await load_runtime_config(engine)
oauth_url = config.base_url.rstrip("/") + OAUTH_TOKEN_PATH
picked_proxy = await select_proxy_uri_for_url(
engine,
settings,
runtime,
url=oauth_url,
token_id=int(token_id),
)
proxy_uri = picked_proxy.uri if picked_proxy is not None else None
token = await refresh_access_token(
refresh_token=refresh_token,
config=config,
transport=transport,
proxy=proxy_uri,
)
except PixivOauthError as exc:
new_error_count = int(row.error_count or 0) + 1
backoff_s = refresh_backoff_seconds(attempt=new_error_count, status_code=exc.status_code)
backoff_until = (
iso_utc_ms(datetime.now(timezone.utc) + timedelta(seconds=backoff_s)) if backoff_s > 0 else None
)
row.error_count = new_error_count
row.backoff_until = backoff_until
row.last_fail_at = now
row.last_error_code = ErrorCode.TOKEN_REFRESH_FAILED.value
row.last_error_msg = str(exc)[:500]
row.updated_at = now
await session.commit()
raise ApiError(
code=ErrorCode.TOKEN_REFRESH_FAILED,
message="Token refresh failed",
status_code=502,
details={"upstream_status": exc.status_code or 0, "backoff_until": backoff_until or ""},
) from exc
except Exception as exc:
new_error_count = int(row.error_count or 0) + 1
backoff_s = refresh_backoff_seconds(attempt=new_error_count, status_code=None)
backoff_until = (
iso_utc_ms(datetime.now(timezone.utc) + timedelta(seconds=backoff_s)) if backoff_s > 0 else None
)
row.error_count = new_error_count
row.backoff_until = backoff_until
row.last_fail_at = now
row.last_error_code = ErrorCode.TOKEN_REFRESH_FAILED.value
row.last_error_msg = "exception"
row.updated_at = now
await session.commit()
raise ApiError(
code=ErrorCode.TOKEN_REFRESH_FAILED,
message="Token refresh failed",
status_code=502,
details={"backoff_until": backoff_until or ""},
) from exc
row.error_count = 0
row.backoff_until = None
row.last_ok_at = now
row.last_fail_at = None
row.last_error_code = None
row.last_error_msg = None
row.updated_at = now
rotated = token.refresh_token
if rotated:
row.refresh_token_enc = encryptor.encrypt_text(rotated)
row.refresh_token_masked = mask_secret(rotated)
await session.commit()
proxy_details: dict[str, Any] | None = None
if picked_proxy is not None:
proxy_details = {"endpoint_id": str(picked_proxy.endpoint_id), "pool_id": str(picked_proxy.pool_id)}
return {
"ok": True,
"expires_in": int(token.expires_in),
"user_id": token.user_id,
"proxy": proxy_details,
"request_id": rid,
}
@router.post("/tokens/{token_id}/reset-failures")
async def reset_failures(
token_id: int,
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> dict[str, Any]:
_ = _claims
if token_id <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid token id", status_code=400)
rid = get_or_create_request_id(request)
now = iso_utc_ms()
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
row = await session.get(PixivToken, token_id)
if row is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Token not found", status_code=404)
row.error_count = 0
row.backoff_until = None
row.last_fail_at = None
row.last_error_code = None
row.last_error_msg = None
row.updated_at = now
await session.commit()
return {"ok": True, "token_id": str(token_id), "request_id": rid}
+88
View File
@@ -0,0 +1,88 @@
from __future__ import annotations
from typing import Any
from fastapi import APIRouter, Depends, Request
from fastapi.responses import Response
from prometheus_client.exposition import CONTENT_TYPE_LATEST, generate_latest
from sqlalchemy.ext.asyncio import AsyncEngine
from app.api.admin.deps import get_admin_claims
from app.core.metrics import (
METRICS_LAST_SCRAPE_SUCCESS,
METRICS_SCRAPE_ERRORS_TOTAL,
ensure_known_keys,
set_jobs_status_counts,
set_proxy_state_counts,
)
from app.core.time import iso_utc_ms
router = APIRouter()
async def _query_job_status_counts(engine: AsyncEngine) -> dict[str, int]:
sql = "SELECT status, COUNT(*) AS c FROM jobs GROUP BY status"
async with engine.connect() as conn:
result = await conn.exec_driver_sql(sql)
counts: dict[str, int] = {}
for row in result.fetchall():
status = str(row[0])
counts[status] = int(row[1])
return counts
async def _query_proxy_state_counts(engine: AsyncEngine) -> dict[str, int]:
now = iso_utc_ms()
sql = """
SELECT
COUNT(*) AS total,
SUM(CASE WHEN enabled = 1 THEN 1 ELSE 0 END) AS enabled,
SUM(CASE WHEN enabled = 1 AND blacklisted_until IS NOT NULL AND blacklisted_until > :now THEN 1 ELSE 0 END) AS blacklisted,
SUM(CASE WHEN enabled = 1
AND (blacklisted_until IS NULL OR blacklisted_until <= :now)
AND last_ok_at IS NOT NULL
AND (last_fail_at IS NULL OR last_ok_at >= last_fail_at)
THEN 1 ELSE 0 END) AS healthy,
SUM(CASE WHEN enabled = 1
AND (blacklisted_until IS NULL OR blacklisted_until <= :now)
AND (last_ok_at IS NULL OR (last_fail_at IS NOT NULL AND last_fail_at > last_ok_at))
THEN 1 ELSE 0 END) AS unhealthy
FROM proxy_endpoints;
""".strip()
async with engine.connect() as conn:
result = await conn.exec_driver_sql(sql, {"now": now})
row = result.fetchone()
if row is None:
return {"total": 0, "enabled": 0, "healthy": 0, "unhealthy": 0, "blacklisted": 0}
return {
"total": int(row[0] or 0),
"enabled": int(row[1] or 0),
"blacklisted": int(row[2] or 0),
"healthy": int(row[3] or 0),
"unhealthy": int(row[4] or 0),
}
@router.get("/metrics")
async def metrics(
request: Request,
_claims: dict[str, Any] = Depends(get_admin_claims),
) -> Response:
engine: AsyncEngine | None = getattr(request.app.state, "engine", None)
if engine is not None:
try:
job_counts = await _query_job_status_counts(engine)
set_jobs_status_counts(job_counts)
proxy_counts = await _query_proxy_state_counts(engine)
proxy_counts = ensure_known_keys(["total", "enabled", "healthy", "unhealthy", "blacklisted"], proxy_counts)
set_proxy_state_counts(proxy_counts)
METRICS_LAST_SCRAPE_SUCCESS.set(1)
except Exception:
METRICS_SCRAPE_ERRORS_TOTAL.inc()
METRICS_LAST_SCRAPE_SUCCESS.set(0)
content = generate_latest()
return Response(content=content, media_type=CONTENT_TYPE_LATEST)
+62
View File
@@ -0,0 +1,62 @@
from __future__ import annotations
from typing import Any
from fastapi import APIRouter, Request
from fastapi.responses import JSONResponse
from app.core.errors import ApiError, ErrorCode
from app.core.request_id import get_or_create_request_id
from app.db.authors_list import list_authors as db_list_authors
from app.db.session import create_sessionmaker
router = APIRouter()
@router.get("/authors")
async def list_authors(
request: Request,
q: str | None = None,
limit: int = 50,
cursor: str | None = None,
) -> Any:
if limit < 1 or limit > 100:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
q_norm = (q or "").strip()
if q_norm and len(q_norm) > 200:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported q", status_code=400)
cursor_i: int | None = None
cursor_raw = (cursor or "").strip()
if cursor_raw:
if not cursor_raw.isdigit():
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
cursor_i = int(cursor_raw)
if cursor_i <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
items, next_cursor = await db_list_authors(session, limit=limit, cursor=cursor_i, q=q_norm or None)
rid = get_or_create_request_id(request)
resp = JSONResponse(
status_code=200,
content={
"ok": True,
"items": [
{
"user_id": str(item.user_id),
"user_name": item.user_name,
"count_images": item.count_images,
}
for item in items
],
"next_cursor": str(next_cursor) if next_cursor is not None else "",
"request_id": rid,
},
)
return resp
+361
View File
@@ -0,0 +1,361 @@
from __future__ import annotations
from fastapi import APIRouter, Request
from fastapi.responses import HTMLResponse
router = APIRouter()
def _build_docs_html(*, base_url: str) -> str:
base = (base_url or "").rstrip("/")
if not base:
base = ""
def u(path: str) -> str:
path_norm = (path or "").strip()
if not path_norm.startswith("/"):
path_norm = "/" + path_norm
return f"{base}{path_norm}"
examples = {
"img_default": u("/random"),
"img_redirect": u("/random?redirect=1"),
"json_full": u("/random?format=json"),
"json_simple": u("/random?format=simple_json"),
"r18_only": u("/random?r18=1"),
"safe_only": u("/random?r18=0"),
"any_r18": u("/random?r18=2"),
"portrait": u("/random?orientation=portrait"),
"landscape": u("/random?orientation=landscape"),
"adaptive": u("/random?adaptive=1"),
"tag_loli": u("/random?included_tags=loli"),
"pure_random": u("/random?strategy=random"),
"quality_strong": u(
"/random?strategy=quality&quality_samples=200&min_pixels=2000000&min_bookmarks=100&illust_type=illust&ai_type=0"
),
"complex": u(
"/random?format=json&r18=1&illust_type=illust&orientation=portrait&min_pixels=2500000&min_bookmarks=25&min_comments=50&included_tags=loli"
),
"tags_api": u("/tags"),
"authors_api": u("/authors"),
"swagger": u("/api/docs"),
"openapi": u("/openapi.json"),
}
return f"""<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Random Mage Docs</title>
<style>
:root {{
--bg: #0b1220;
--card: rgba(255, 255, 255, 0.06);
--card-2: rgba(255, 255, 255, 0.08);
--border: rgba(255, 255, 255, 0.12);
--text: rgba(255, 255, 255, 0.92);
--muted: rgba(255, 255, 255, 0.72);
--muted2: rgba(255, 255, 255, 0.60);
--link: #7dd3fc;
--accent: #a78bfa;
--ok: #34d399;
--warn: #fbbf24;
--mono: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace;
--sans: ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, Helvetica, Arial, "Apple Color Emoji",
"Segoe UI Emoji";
}}
* {{ box-sizing: border-box; }}
body {{
margin: 0;
font-family: var(--sans);
color: var(--text);
background:
radial-gradient(1000px 600px at 15% 10%, rgba(167, 139, 250, 0.22), transparent 55%),
radial-gradient(900px 520px at 85% 0%, rgba(125, 211, 252, 0.18), transparent 60%),
radial-gradient(700px 700px at 60% 85%, rgba(52, 211, 153, 0.12), transparent 55%),
var(--bg);
}}
a {{ color: var(--link); text-decoration: none; }}
a:hover {{ text-decoration: underline; }}
.wrap {{
max-width: 980px;
margin: 0 auto;
padding: 32px 20px 64px;
}}
.hero {{
padding: 18px 18px 16px;
border: 1px solid var(--border);
background: linear-gradient(180deg, rgba(255,255,255,0.08), rgba(255,255,255,0.04));
border-radius: 14px;
backdrop-filter: blur(10px);
}}
.hero h1 {{
margin: 0;
font-size: 22px;
letter-spacing: 0.2px;
}}
.hero p {{
margin: 10px 0 0;
color: var(--muted);
line-height: 1.55;
}}
.grid {{
display: grid;
grid-template-columns: 1fr;
gap: 14px;
margin-top: 14px;
}}
@media (min-width: 860px) {{
.grid {{ grid-template-columns: 1fr 1fr; }}
}}
.card {{
border: 1px solid var(--border);
background: var(--card);
border-radius: 14px;
padding: 16px 16px 14px;
}}
.card h2 {{
font-size: 16px;
margin: 0 0 10px;
letter-spacing: 0.2px;
}}
.card p {{
margin: 0 0 10px;
color: var(--muted);
line-height: 1.55;
}}
.kbd {{
display: inline-block;
font-family: var(--mono);
font-size: 12px;
padding: 2px 6px;
border-radius: 8px;
border: 1px solid var(--border);
background: rgba(0, 0, 0, 0.18);
color: rgba(255,255,255,0.86);
}}
pre {{
margin: 10px 0 0;
padding: 12px 12px;
border-radius: 12px;
border: 1px solid var(--border);
background: rgba(0, 0, 0, 0.22);
overflow: auto;
font-family: var(--mono);
font-size: 12px;
line-height: 1.55;
color: rgba(255,255,255,0.88);
}}
.chips {{
display: flex;
flex-wrap: wrap;
gap: 8px;
margin-top: 10px;
}}
.chip {{
padding: 6px 10px;
border-radius: 999px;
border: 1px solid var(--border);
background: var(--card-2);
font-size: 12px;
color: rgba(255,255,255,0.85);
white-space: nowrap;
}}
.chip strong {{ color: rgba(255,255,255,0.94); }}
table {{
width: 100%;
border-collapse: collapse;
margin-top: 10px;
border-radius: 12px;
overflow: hidden;
border: 1px solid var(--border);
}}
th, td {{
padding: 10px 10px;
border-bottom: 1px solid rgba(255,255,255,0.10);
vertical-align: top;
font-size: 13px;
line-height: 1.45;
}}
th {{
text-align: left;
background: rgba(255,255,255,0.06);
color: rgba(255,255,255,0.92);
font-weight: 600;
}}
td code {{
font-family: var(--mono);
font-size: 12px;
color: rgba(255,255,255,0.9);
}}
.muted {{ color: var(--muted2); }}
.note {{
margin-top: 12px;
padding: 12px 12px;
border-radius: 12px;
border: 1px solid var(--border);
background: rgba(251, 191, 36, 0.10);
color: rgba(255,255,255,0.90);
}}
.note strong {{ color: rgba(255,255,255,0.95); }}
.footer {{
margin-top: 18px;
color: var(--muted2);
font-size: 12px;
line-height: 1.6;
}}
</style>
</head>
<body>
<div class="wrap">
<div class="hero">
<h1>Random Mage · 随机图片 API</h1>
<p>质量优先随机 + 强力筛选,支持标签/热度/分辨率/R18/AI/作品类型等组合查询。</p>
<div class="chips" aria-label="quick-links">
<a class="chip" href="{examples["img_default"]}"><strong>/random</strong> 直接出图</a>
<a class="chip" href="{examples["json_full"]}"><strong>format=json</strong> 返回 JSON</a>
<a class="chip" href="{examples["pure_random"]}"><strong>strategy=random</strong> 纯随机</a>
<a class="chip" href="{examples["swagger"]}"><strong>/api/docs</strong> Swagger</a>
</div>
</div>
<div class="grid">
<section class="card">
<h2>1) 最常用(直接出图)</h2>
<p>默认返回图片(不是 JSON)。想要稳定 URL 可加 <span class="kbd">redirect=1</span> 跳转到本站缓存/代理路径。</p>
<pre><code>{examples["img_default"]}
{examples["img_redirect"]}</code></pre>
</section>
<section class="card">
<h2>2) JSON / 调试信息</h2>
<p><span class="kbd">format=json</span> 返回完整数据(含 tags / debug);<span class="kbd">format=simple_json</span> 更轻量。</p>
<pre><code>{examples["json_full"]}
{examples["json_simple"]}</code></pre>
</section>
<section class="card">
<h2>3) R18 / 作品类型 / AI</h2>
<p>R18 用 <span class="kbd">r18</span> 控制:0=仅全年龄(默认),1=仅R18,2=都可。</p>
<pre><code>{examples["safe_only"]}
{examples["r18_only"]}
{examples["any_r18"]}</code></pre>
<div class="note">
<strong>提示:</strong>如果你库里有不少作品还没补全 x_restrict,且你想“更容易命中”,可用
<span class="kbd">r18=0&amp;r18_strict=0</span>(允许未知的 x_restrict)。
</div>
</section>
<section class="card">
<h2>4) 横竖图 / 自适应</h2>
<p><span class="kbd">orientation</span> 支持:any / portrait / landscape / square。自适应用 <span class="kbd">adaptive=1</span>:移动端默认竖图+更低像素门槛,PC 默认横图+更高门槛(不会覆盖你显式传入的过滤条件)。</p>
<pre><code>{examples["portrait"]}
{examples["landscape"]}
{examples["adaptive"]}</code></pre>
</section>
</div>
<section class="card" style="margin-top: 14px;">
<h2>筛选语法速记</h2>
<table>
<thead>
<tr>
<th style="width: 170px;">参数</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>strategy</code></td>
<td>
随机策略:<code>quality</code>(默认,质量优先) / <code>random</code>(纯随机)。<br/>
<span class="muted">quality 会先抽样 N 张候选再按热度/分辨率打分选图。</span>
</td>
</tr>
<tr>
<td><code>quality_samples</code></td>
<td>
质量抽样数量(1–1000)。越大越“挑剔”,但 DB 开销也更高。<br/>
示例:<a href="{examples["quality_strong"]}">{examples["quality_strong"]}</a>
</td>
</tr>
<tr>
<td><code>included_tags</code></td>
<td>
必须包含的标签(AND)。可重复传参:<code>included_tags=a&amp;included_tags=b</code>。<br/>
也支持用 <code>|</code> 分隔:<code>included_tags=a|b</code> 等价于同时包含 a 与 b。<br/>
示例:<a href="{examples["tag_loli"]}">{examples["tag_loli"]}</a>
</td>
</tr>
<tr>
<td><code>excluded_tags</code></td>
<td>必须不包含的标签(OR)。可重复传参。示例:<code>excluded_tags=AI生成</code></td>
</tr>
<tr>
<td><code>min_width</code><br/><code>min_height</code><br/><code>min_pixels</code></td>
<td>分辨率门槛:宽/高/像素数(例如 2000000 ≈ 2MP)。</td>
</tr>
<tr>
<td><code>min_bookmarks</code><br/><code>min_views</code><br/><code>min_comments</code></td>
<td>热度门槛:收藏/浏览/评论(需要补全元数据后更准确)。</td>
</tr>
<tr>
<td><code>ai_type</code></td>
<td><code>any</code> / <code>0</code>(非AI) / <code>1</code>(AI)。</td>
</tr>
<tr>
<td><code>illust_type</code></td>
<td><code>any</code> / <code>illust</code> / <code>manga</code> / <code>ugoira</code>。</td>
</tr>
<tr>
<td><code>user_id</code></td>
<td>只返回某个作者的作品(Pixiv 用户 ID)。</td>
</tr>
<tr>
<td><code>created_from</code><br/><code>created_to</code></td>
<td>按发布时间筛选(ISO8601,例如 <code>2024-01-01T00:00:00Z</code>)。</td>
</tr>
<tr>
<td><code>seed</code></td>
<td>固定随机种子(同参数 + 同 seed 更容易复现)。</td>
</tr>
<tr>
<td><code>attempts</code></td>
<td>上游失败时重试次数(1–10)。</td>
</tr>
</tbody>
</table>
<div class="footer">
其它接口:<a href="{examples["tags_api"]}">/tags</a>(标签列表)、<a href="{examples["authors_api"]}">/authors</a>(作者列表)。<br/>
OpenAPI:<a href="{examples["openapi"]}">/openapi.json</a>,Swagger:<a href="{examples["swagger"]}">/api/docs</a>。<br/>
</div>
</section>
<section class="card" style="margin-top: 14px;">
<h2>复杂示例(组合查询)</h2>
<p>示例:R18 + 插画 + 竖图 + 250万像素以上 + 收藏≥25 + 评论≥50 + 标签包含 loli,并返回 JSON。</p>
<pre><code>{examples["complex"]}</code></pre>
</section>
</div>
</body>
</html>
"""
@router.get("/docs", include_in_schema=False)
async def docs_page(request: Request) -> HTMLResponse:
html = _build_docs_html(base_url=str(getattr(request, "base_url", "") or "").rstrip("/"))
return HTMLResponse(content=html, status_code=200, headers={"Cache-Control": "no-store"})
+180
View File
@@ -0,0 +1,180 @@
from __future__ import annotations
import json
import os
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any
from fastapi import APIRouter, Request
from fastapi.responses import JSONResponse
from sqlalchemy.ext.asyncio import AsyncEngine
from app.core.errors import ErrorCode, error_body
from app.core.request_id import get_or_create_request_id, set_request_id_header, set_request_id_on_state
from app.db.session import with_sqlite_busy_retry
router = APIRouter()
_JOB_STATUSES: tuple[str, ...] = (
"pending",
"running",
"paused",
"canceled",
"completed",
"failed",
"dlq",
)
def _ensure_sqlite_dir(engine: AsyncEngine) -> None:
url = engine.url
if url.get_backend_name() != "sqlite":
return
db_path = url.database
if not db_path or db_path == ":memory:":
return
Path(db_path).expanduser().resolve().parent.mkdir(parents=True, exist_ok=True)
async def _check_db(engine: AsyncEngine) -> bool:
try:
_ensure_sqlite_dir(engine)
async with engine.connect() as conn:
await conn.exec_driver_sql("SELECT 1")
return True
except Exception:
return False
def _parse_iso_utc(value: str) -> datetime | None:
raw = (value or "").strip()
if not raw:
return None
if raw.endswith("Z"):
raw = raw[:-1] + "+00:00"
try:
dt = datetime.fromisoformat(raw)
except Exception:
return None
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt.astimezone(timezone.utc)
async def _query_worker_last_seen(engine: AsyncEngine) -> tuple[str | None, str]:
sql = "SELECT value_json FROM runtime_settings WHERE key = ?"
async def _op() -> str | None:
async with engine.connect() as conn:
result = await conn.exec_driver_sql(sql, ("worker.last_seen_at",))
row = result.fetchone()
return str(row[0]) if row is not None else None
try:
raw = await with_sqlite_busy_retry(_op)
except Exception:
return None, "runtime_settings_unavailable"
if raw is None:
return None, "no_heartbeat"
try:
value = json.loads(raw)
except Exception:
return None, "invalid_json"
if isinstance(value, dict):
at = value.get("at")
if isinstance(at, str):
return at, "ok"
return None, "invalid_value"
if isinstance(value, str):
return value, "ok"
return None, "invalid_value"
async def _query_queue_status_counts(engine: AsyncEngine) -> tuple[dict[str, int] | None, str]:
sql = "SELECT status, COUNT(*) AS c FROM jobs GROUP BY status"
async def _op() -> dict[str, int]:
async with engine.connect() as conn:
result = await conn.exec_driver_sql(sql)
counts: dict[str, int] = {}
for row in result.fetchall():
counts[str(row[0])] = int(row[1])
return counts
try:
counts = await with_sqlite_busy_retry(_op)
except Exception:
return None, "jobs_unavailable"
for status in _JOB_STATUSES:
counts.setdefault(status, 0)
return counts, "ok"
@router.get("/healthz")
async def healthz(request: Request) -> Any:
rid = get_or_create_request_id(request)
set_request_id_on_state(request, rid)
engine: AsyncEngine | None = getattr(request.app.state, "engine", None)
db_ok = await _check_db(engine) if engine is not None else False
if db_ok:
try:
stale_after_s = int((os.environ.get("WORKER_HEARTBEAT_STALE_SECONDS") or "60").strip() or "60")
except Exception:
stale_after_s = 60
stale_after_s = max(1, min(int(stale_after_s), 24 * 60 * 60))
worker_last_seen_at, worker_reason = await _query_worker_last_seen(engine) # type: ignore[arg-type]
worker_ok = False
if worker_last_seen_at is not None:
last_seen_dt = _parse_iso_utc(worker_last_seen_at)
if last_seen_dt is None:
worker_reason = "invalid_timestamp"
else:
worker_ok = datetime.now(timezone.utc) - last_seen_dt <= timedelta(seconds=stale_after_s)
if not worker_ok:
worker_reason = "stale"
queue_counts, queue_reason = await _query_queue_status_counts(engine) # type: ignore[arg-type]
queue_ok = queue_counts is not None
resp = JSONResponse(
status_code=200,
content={
"ok": True,
"db_ok": True,
"worker_ok": worker_ok,
"queue_ok": queue_ok,
"worker": {
"last_seen_at": worker_last_seen_at,
"stale_after_s": stale_after_s,
"reason": worker_reason,
},
"queue": {
"counts": queue_counts or {},
"reason": queue_reason,
},
"request_id": rid,
},
)
else:
resp = JSONResponse(
status_code=503,
content=error_body(
code=ErrorCode.INTERNAL_ERROR,
message="Database unavailable",
request_id=rid,
details={"db_ok": False},
),
)
set_request_id_header(resp, rid)
return resp
+333
View File
@@ -0,0 +1,333 @@
from __future__ import annotations
from datetime import datetime, timezone
from typing import Any
from fastapi import APIRouter, Query, Request
from fastapi.responses import JSONResponse
from app.core.errors import ApiError, ErrorCode
from app.core.http_stream import stream_url
import sqlalchemy as sa
from app.core.pixiv_urls import ALLOWED_IMAGE_EXTS
from app.core.request_id import get_or_create_request_id, set_request_id_header, set_request_id_on_state
from app.core.runtime_settings import load_runtime_config
from app.core.time import iso_utc_ms
from app.core.proxy_routing import select_proxy_uri_for_url
from app.db.images_get import get_image_by_id
from app.db.images_list import list_images as db_list_images
from app.db.images_mark import mark_image_failure, mark_image_ok
from app.db.models.image_tags import ImageTag
from app.db.session import create_sessionmaker
from app.db.tags_get import get_tag_names_for_image
from app.jobs.enqueue import enqueue_opportunistic_hydrate_metadata
router = APIRouter()
_MAX_TAG_FILTERS = 50
@router.get("/images")
async def list_images(
request: Request,
limit: int = 50,
cursor: str | None = None,
r18: int = 0,
r18_strict: int = 1,
ai_type: str = "any",
orientation: str = "any",
min_width: int = 0,
min_height: int = 0,
min_pixels: int = 0,
included_tags: list[str] | None = Query(default=None),
excluded_tags: list[str] | None = Query(default=None),
user_id: int | None = None,
illust_id: int | None = None,
created_from: str | None = None,
created_to: str | None = None,
) -> Any:
if limit < 1 or limit > 200:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
cursor_i: int | None = None
cursor_raw = (cursor or "").strip()
if cursor_raw:
if not cursor_raw.isdigit():
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
cursor_i = int(cursor_raw)
if cursor_i <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported cursor", status_code=400)
if r18 not in {0, 1, 2}:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported r18", status_code=400)
if r18_strict not in {0, 1}:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported r18_strict", status_code=400)
ai_type_raw = (ai_type or "any").strip().lower()
ai_type_i: int | None = None
if ai_type_raw in {"", "any"}:
ai_type_i = None
elif ai_type_raw in {"0", "1"}:
ai_type_i = int(ai_type_raw)
else:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported ai_type", status_code=400)
orientation = (orientation or "").strip().lower()
orientation_map = {"any": None, "portrait": 1, "landscape": 2, "square": 3}
if orientation not in orientation_map:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported orientation", status_code=400)
if min_width < 0 or min_height < 0 or min_pixels < 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported min_*", status_code=400)
def _parse_tags(values: list[str] | None) -> list[str]:
out: list[str] = []
seen: set[str] = set()
for raw in values or []:
for part in str(raw).split("|"):
name = part.strip()
if not name or name in seen:
continue
seen.add(name)
out.append(name)
return out
included = _parse_tags(included_tags)
excluded = _parse_tags(excluded_tags)
if len(included) > _MAX_TAG_FILTERS or len(excluded) > _MAX_TAG_FILTERS:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Too many tag filters", status_code=400)
if user_id is not None and int(user_id) <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported user_id", status_code=400)
if illust_id is not None and int(illust_id) <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported illust_id", status_code=400)
def _normalize_iso_utc(value: str) -> str:
raw = (value or "").strip()
if not raw:
raise ValueError("empty datetime")
if raw.endswith("Z"):
raw = raw[:-1] + "+00:00"
dt = datetime.fromisoformat(raw)
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
dt = dt.astimezone(timezone.utc).replace(microsecond=0)
return dt.strftime("%Y-%m-%dT%H:%M:%SZ")
created_from_norm: str | None = None
created_to_norm: str | None = None
try:
if created_from is not None:
created_from_norm = _normalize_iso_utc(created_from)
if created_to is not None:
created_to_norm = _normalize_iso_utc(created_to)
except Exception:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported created_*", status_code=400)
if created_from_norm is not None and created_to_norm is not None:
if created_from_norm > created_to_norm:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="created_from > created_to", status_code=400)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
images, next_cursor = await db_list_images(
session,
limit=limit,
cursor=cursor_i,
r18=r18,
r18_strict=bool(r18_strict),
orientation=orientation_map[orientation],
ai_type=ai_type_i,
min_width=min_width,
min_height=min_height,
min_pixels=min_pixels,
included_tags=included,
excluded_tags=excluded,
user_id=user_id,
illust_id=illust_id,
created_from=created_from_norm,
created_to=created_to_norm,
)
rid = get_or_create_request_id(request)
set_request_id_on_state(request, rid)
items = [
{
"id": str(img.id),
"illust_id": str(img.illust_id),
"page_index": img.page_index,
"ext": img.ext,
"width": img.width,
"height": img.height,
"x_restrict": img.x_restrict,
"ai_type": img.ai_type,
"bookmark_count": getattr(img, "bookmark_count", None),
"view_count": getattr(img, "view_count", None),
"comment_count": getattr(img, "comment_count", None),
"user": {
"id": str(img.user_id) if img.user_id is not None else None,
"name": img.user_name,
},
"title": img.title,
"created_at_pixiv": img.created_at_pixiv,
}
for img in images
]
resp = JSONResponse(
status_code=200,
content={
"ok": True,
"items": items,
"next_cursor": str(next_cursor) if next_cursor is not None else "",
"request_id": rid,
},
)
set_request_id_header(resp, rid)
return resp
@router.get("/images/{image_id}")
async def get_image(
request: Request,
image_id: int,
) -> Any:
if int(image_id) <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported image_id", status_code=400)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
image = await get_image_by_id(session, image_id=image_id)
if image is None:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Image not found", status_code=404)
tags = await get_tag_names_for_image(session, image_id=image.id)
rid = get_or_create_request_id(request)
set_request_id_on_state(request, rid)
resp = JSONResponse(
status_code=200,
content={
"ok": True,
"item": {
"image": {
"id": str(image.id),
"illust_id": str(image.illust_id),
"page_index": image.page_index,
"ext": image.ext,
"width": image.width,
"height": image.height,
"x_restrict": image.x_restrict,
"ai_type": image.ai_type,
"bookmark_count": getattr(image, "bookmark_count", None),
"view_count": getattr(image, "view_count", None),
"comment_count": getattr(image, "comment_count", None),
"user": {
"id": str(image.user_id) if image.user_id is not None else None,
"name": image.user_name,
},
"title": image.title,
"created_at_pixiv": image.created_at_pixiv,
},
"tags": tags,
},
"request_id": rid,
},
)
set_request_id_header(resp, rid)
return resp
@router.get("/i/{image_id}.{ext}")
async def proxy_image(
request: Request,
image_id: int,
ext: str,
):
ext = (ext or "").lower()
if ext not in ALLOWED_IMAGE_EXTS:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported ext", status_code=400)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
needs_hydrate = False
should_mark_ok = False
async with Session() as session:
image = await get_image_by_id(session, image_id=image_id)
if image is None or (image.ext or "").lower() != ext:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Image not found", status_code=404)
should_mark_ok = image.last_ok_at is None or image.last_error_code is not None
missing_basic_fields = (
getattr(image, "width", None) is None
or getattr(image, "height", None) is None
or getattr(image, "x_restrict", None) is None
or getattr(image, "ai_type", None) is None
or getattr(image, "user_id", None) is None
or not str(getattr(image, "user_name", "") or "").strip()
or not str(getattr(image, "title", "") or "").strip()
or not str(getattr(image, "created_at_pixiv", "") or "").strip()
or getattr(image, "bookmark_count", None) is None
or getattr(image, "view_count", None) is None
or getattr(image, "comment_count", None) is None
)
if missing_basic_fields:
needs_hydrate = True
else:
tag_row = (
await session.execute(
sa.select(ImageTag.image_id).where(ImageTag.image_id == int(image.id)).limit(1)
)
).scalar_one_or_none()
needs_hydrate = tag_row is None
runtime = await load_runtime_config(engine)
proxy_uri = None
picked = await select_proxy_uri_for_url(
engine,
request.app.state.settings,
runtime,
url=str(image.original_url),
)
if picked is not None:
proxy_uri = picked.uri
transport = getattr(request.app.state, "httpx_transport", None)
now = iso_utc_ms()
try:
resp = await stream_url(
image.original_url,
transport=transport,
proxy=proxy_uri,
cache_control="public, max-age=31536000, immutable",
range_header=request.headers.get("Range"),
)
if should_mark_ok:
await mark_image_ok(engine, image_id=int(image.id), now=now)
if needs_hydrate:
try:
await enqueue_opportunistic_hydrate_metadata(engine, illust_id=int(image.illust_id), reason="image_proxy")
except Exception:
pass
return resp
except ApiError as exc:
if exc.code in {
ErrorCode.UPSTREAM_STREAM_ERROR,
ErrorCode.UPSTREAM_403,
ErrorCode.UPSTREAM_404,
ErrorCode.UPSTREAM_RATE_LIMIT,
}:
await mark_image_failure(
engine,
image_id=int(image.id),
now=now,
error_code=exc.code.value,
error_message=exc.message,
)
raise
+100
View File
@@ -0,0 +1,100 @@
from __future__ import annotations
from fastapi import APIRouter, Request
from app.core.errors import ApiError, ErrorCode
from app.core.http_stream import stream_url
from app.core.pixiv_urls import ALLOWED_IMAGE_EXTS
from app.core.proxy_routing import select_proxy_uri_for_url
from app.core.runtime_settings import load_runtime_config
from app.db.images_get_by_illust import get_image_by_illust_page
from app.db.session import create_sessionmaker
router = APIRouter()
@router.get("/{illust_id}-{page}.{ext}")
async def legacy_multi(
request: Request,
illust_id: int,
page: int,
ext: str,
):
if int(illust_id) <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported illust_id", status_code=400)
if int(page) <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported page", status_code=400)
ext = (ext or "").lower()
if ext not in ALLOWED_IMAGE_EXTS:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported ext", status_code=400)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
image = await get_image_by_illust_page(session, illust_id=illust_id, page_index=int(page) - 1)
if image is None or (image.ext or "").lower() != ext:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Image not found", status_code=404)
runtime = await load_runtime_config(engine)
proxy_uri = None
picked = await select_proxy_uri_for_url(
engine,
request.app.state.settings,
runtime,
url=str(image.original_url),
)
if picked is not None:
proxy_uri = picked.uri
transport = getattr(request.app.state, "httpx_transport", None)
return await stream_url(
image.original_url,
transport=transport,
proxy=proxy_uri,
cache_control="public, max-age=31536000, immutable",
range_header=request.headers.get("Range"),
)
@router.get("/{illust_id}.{ext}")
async def legacy_single(
request: Request,
illust_id: int,
ext: str,
):
if int(illust_id) <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported illust_id", status_code=400)
ext = (ext or "").lower()
if ext not in ALLOWED_IMAGE_EXTS:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported ext", status_code=400)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
image = await get_image_by_illust_page(session, illust_id=illust_id, page_index=0)
if image is None or (image.ext or "").lower() != ext:
raise ApiError(code=ErrorCode.NOT_FOUND, message="Image not found", status_code=404)
runtime = await load_runtime_config(engine)
proxy_uri = None
picked = await select_proxy_uri_for_url(
engine,
request.app.state.settings,
runtime,
url=str(image.original_url),
)
if picked is not None:
proxy_uri = picked.uri
transport = getattr(request.app.state, "httpx_transport", None)
return await stream_url(
image.original_url,
transport=transport,
proxy=proxy_uri,
cache_control="public, max-age=31536000, immutable",
range_header=request.headers.get("Range"),
)
+861
View File
@@ -0,0 +1,861 @@
from __future__ import annotations
import random
import os
import math
from datetime import datetime, timedelta, timezone
from typing import Any
from fastapi import APIRouter, Query, Request
from fastapi.responses import RedirectResponse
from app.core.errors import ApiError, ErrorCode
from app.core.http_stream import stream_url
from app.core.imgproxy import build_signed_processing_url, load_imgproxy_config_from_settings
from app.core.proxy_routing import select_proxy_uri_for_url
from app.core.runtime_settings import load_runtime_config
from app.core.time import iso_utc_ms
from app.db.images_mark import mark_image_failure, mark_image_ok
from app.db.tags_get import get_tag_names_for_image
from app.db.random_pick import pick_random_image, pick_random_images
from app.db.session import create_sessionmaker
from app.jobs.enqueue import enqueue_opportunistic_hydrate_metadata
router = APIRouter()
_MAX_TAG_FILTERS = 50
def _as_nonneg_int(value: Any) -> int:
if value is None:
return 0
if isinstance(value, bool):
return 0
try:
i = int(value)
except Exception:
return 0
return i if i > 0 else 0
_DEFAULT_SCORE_WEIGHTS: dict[str, float] = {
"bookmark": 4.0,
"view": 0.5,
"comment": 2.0,
"pixels": 1.0,
"bookmark_rate": 3.0,
}
_DEFAULT_RECOMMENDATION: dict[str, Any] = {
"pick_mode": "weighted",
"temperature": 1.0,
"score_weights": dict(_DEFAULT_SCORE_WEIGHTS),
"multipliers": {
"ai": 1.0,
"non_ai": 1.0,
"unknown_ai": 1.0,
"illust": 1.0,
"manga": 1.0,
"ugoira": 1.0,
"unknown_illust_type": 1.0,
},
}
def _as_float(value: Any, *, default: float) -> float:
if value is None:
return float(default)
if isinstance(value, bool):
return float(default)
try:
return float(value)
except Exception:
return float(default)
def _quality_score(image: Any, *, weights: dict[str, float] | None = None) -> float:
w = weights or _DEFAULT_SCORE_WEIGHTS
w_bookmark = float(w.get("bookmark", _DEFAULT_SCORE_WEIGHTS["bookmark"]))
w_view = float(w.get("view", _DEFAULT_SCORE_WEIGHTS["view"]))
w_comment = float(w.get("comment", _DEFAULT_SCORE_WEIGHTS["comment"]))
w_pixels = float(w.get("pixels", _DEFAULT_SCORE_WEIGHTS["pixels"]))
w_bookmark_rate = float(w.get("bookmark_rate", _DEFAULT_SCORE_WEIGHTS["bookmark_rate"]))
bookmark_count = _as_nonneg_int(getattr(image, "bookmark_count", None))
view_count = _as_nonneg_int(getattr(image, "view_count", None))
comment_count = _as_nonneg_int(getattr(image, "comment_count", None))
width = _as_nonneg_int(getattr(image, "width", None))
height = _as_nonneg_int(getattr(image, "height", None))
pixels = width * height if width > 0 and height > 0 else 0
rate_term = 0.0
if view_count > 0:
bookmark_rate_per_mille = (float(bookmark_count) / float(view_count)) * 1000.0
rate_term = math.log1p(max(0.0, bookmark_rate_per_mille))
score = (
float(w_bookmark) * math.log1p(bookmark_count)
+ float(w_view) * math.log1p(view_count)
+ float(w_comment) * math.log1p(comment_count)
+ float(w_pixels) * math.log1p(float(pixels) / 1_000_000.0)
+ float(w_bookmark_rate) * rate_term
)
return float(score)
@router.get("/random")
async def random_image(
request: Request,
format: str = "image",
redirect: int = 0,
attempts: int | None = None,
seed: str | None = None,
strategy: str | None = None,
quality_samples: int | None = None,
r18: int = 0,
r18_strict: int | None = None,
ai_type: str = "any",
illust_type: str = "any",
orientation: str = "any",
layout: str | None = None,
adaptive: int = 0,
min_width: int = 0,
min_height: int = 0,
min_pixels: int = 0,
min_bookmarks: int = 0,
min_views: int = 0,
min_comments: int = 0,
included_tags: list[str] | None = Query(default=None),
excluded_tags: list[str] | None = Query(default=None),
user_id: int | None = None,
illust_id: int | None = None,
created_from: str | None = None,
created_to: str | None = None,
) -> Any:
if format not in {"image", "json", "simple_json"}:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported format", status_code=400)
if redirect not in {0, 1}:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported redirect", status_code=400)
seed_norm = (seed or "").strip()
if seed is not None and not seed_norm:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported seed", status_code=400)
if seed_norm and len(seed_norm) > 128:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported seed", status_code=400)
ai_type_raw = (ai_type or "any").strip().lower()
ai_type_i: int | None = None
if ai_type_raw in {"", "any"}:
ai_type_i = None
elif ai_type_raw in {"0", "1"}:
ai_type_i = int(ai_type_raw)
else:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported ai_type", status_code=400)
illust_type_raw = (illust_type or "any").strip().lower()
illust_type_i: int | None = None
if illust_type_raw in {"", "any"}:
illust_type_i = None
elif illust_type_raw in {"0", "illust", "illustration"}:
illust_type_i = 0
elif illust_type_raw in {"1", "manga"}:
illust_type_i = 1
elif illust_type_raw in {"2", "ugoira"}:
illust_type_i = 2
else:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported illust_type", status_code=400)
if r18 not in {0, 1, 2}:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported r18", status_code=400)
if adaptive not in {0, 1}:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported adaptive", status_code=400)
layout_source = "orientation"
raw_layout = orientation
if layout is not None:
layout_source = "layout"
raw_layout = layout
layout_norm = (raw_layout or "").strip().lower()
alias_map = {"vertical": "portrait", "horizontal": "landscape"}
layout_norm = alias_map.get(layout_norm, layout_norm)
orientation_map = {"any": None, "portrait": 1, "landscape": 2, "square": 3}
if layout_norm not in orientation_map:
raise ApiError(
code=ErrorCode.BAD_REQUEST,
message="Unsupported layout" if layout_source == "layout" else "Unsupported orientation",
status_code=400,
)
min_width_i = int(min_width)
min_height_i = int(min_height)
min_pixels_i = int(min_pixels)
min_bookmarks_i = int(min_bookmarks)
min_views_i = int(min_views)
min_comments_i = int(min_comments)
if min_width_i < 0 or min_height_i < 0 or min_pixels_i < 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported min_*", status_code=400)
if min_bookmarks_i < 0 or min_views_i < 0 or min_comments_i < 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported min_*", status_code=400)
# 自适应:在用户没有显式指定的情况下,根据设备类型设置默认的方向/分辨率门槛。
# 注意:不会覆盖用户显式传入的 orientation/layout/min_*。
if int(adaptive) == 1:
qp = request.query_params
orientation_explicit = ("layout" in qp) or ("orientation" in qp)
min_explicit = ("min_width" in qp) or ("min_height" in qp) or ("min_pixels" in qp)
ch_mobile = (request.headers.get("sec-ch-ua-mobile") or request.headers.get("Sec-CH-UA-Mobile") or "").strip()
if ch_mobile == "?1":
is_mobile = True
elif ch_mobile == "?0":
is_mobile = False
else:
ua = (request.headers.get("user-agent") or request.headers.get("User-Agent") or "").lower()
is_mobile = any(x in ua for x in ("mobi", "android", "iphone", "ipad", "ipod"))
if not orientation_explicit and layout_norm == "any":
layout_norm = "portrait" if is_mobile else "landscape"
if not min_explicit and min_width_i == 0 and min_height_i == 0 and min_pixels_i == 0:
min_pixels_i = 1_000_000 if is_mobile else 2_000_000
def _parse_tags(values: list[str] | None) -> list[str]:
out: list[str] = []
seen: set[str] = set()
for raw in values or []:
for part in str(raw).split("|"):
name = part.strip()
if not name or name in seen:
continue
seen.add(name)
out.append(name)
return out
included = _parse_tags(included_tags)
excluded = _parse_tags(excluded_tags)
if len(included) > _MAX_TAG_FILTERS or len(excluded) > _MAX_TAG_FILTERS:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Too many tag filters", status_code=400)
if user_id is not None and int(user_id) <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported user_id", status_code=400)
if illust_id is not None and int(illust_id) <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported illust_id", status_code=400)
def _normalize_iso_utc(value: str) -> str:
raw = (value or "").strip()
if not raw:
raise ValueError("empty datetime")
if raw.endswith("Z"):
raw = raw[:-1] + "+00:00"
dt = datetime.fromisoformat(raw)
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
dt = dt.astimezone(timezone.utc).replace(microsecond=0)
return dt.strftime("%Y-%m-%dT%H:%M:%SZ")
created_from_norm: str | None = None
created_to_norm: str | None = None
try:
if created_from is not None:
created_from_norm = _normalize_iso_utc(created_from)
if created_to is not None:
created_to_norm = _normalize_iso_utc(created_to)
except Exception:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported created_*", status_code=400)
if created_from_norm is not None and created_to_norm is not None:
if created_from_norm > created_to_norm:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="created_from > created_to", status_code=400)
def _no_match_error() -> ApiError:
applied_filters: dict[str, Any] = {
"r18": r18,
"r18_strict": int(r18_strict),
"ai_type": ai_type_raw,
"illust_type": illust_type_raw,
"adaptive": int(adaptive),
"orientation": layout_norm,
"min_width": int(min_width_i),
"min_height": int(min_height_i),
"min_pixels": int(min_pixels_i),
"min_bookmarks": int(min_bookmarks_i),
"min_views": int(min_views_i),
"min_comments": int(min_comments_i),
}
if included:
applied_filters["included_tags"] = included
if excluded:
applied_filters["excluded_tags"] = excluded
if user_id is not None:
applied_filters["user_id"] = int(user_id)
if illust_id is not None:
applied_filters["illust_id"] = int(illust_id)
if created_from_norm is not None:
applied_filters["created_from"] = created_from_norm
if created_to_norm is not None:
applied_filters["created_to"] = created_to_norm
suggestions: list[str] = ["运行元数据补全任务以提升元数据覆盖率"]
if r18 == 0 and int(r18_strict) == 1:
suggestions.append("将 r18_strict=0 以允许未知 x_restrict(冷启动阶段更容易命中)")
if layout_norm != "any":
suggestions.append("将 orientation=any(取消方向限制)")
if int(min_width_i) > 0 or int(min_height_i) > 0 or int(min_pixels_i) > 0:
suggestions.append("降低 min_width/min_height/min_pixels(放宽分辨率门槛)")
if int(min_bookmarks_i) > 0 or int(min_views_i) > 0 or int(min_comments_i) > 0:
suggestions.append("降低 min_bookmarks/min_views/min_comments(放宽热度门槛)")
if included:
suggestions.append("放宽 included_tags(减少必须包含的标签)")
if excluded:
suggestions.append("放宽 excluded_tags(减少必须排除的标签)")
if user_id is not None:
suggestions.append("移除 user_id 过滤")
if illust_id is not None:
suggestions.append("移除 illust_id 过滤")
if ai_type_i is not None:
suggestions.append("将 ai_type=any(取消 AI 限制)")
if illust_type_i is not None:
suggestions.append("将 illust_type=any(取消作品类型限制)")
if created_from_norm is not None or created_to_norm is not None:
suggestions.append("扩大 created_from/created_to 时间范围")
if int(adaptive) == 1:
suggestions.append("若自适应导致过滤过严,可尝试 adaptive=0 或显式设置 min_*")
return ApiError(
code=ErrorCode.NO_MATCH,
message="没有匹配的图片。",
status_code=404,
details={"hints": {"applied_filters": applied_filters, "suggestions": suggestions}},
)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
runtime = await load_runtime_config(engine)
random_defaults = runtime.random_defaults if isinstance(runtime.random_defaults, dict) else {}
attempts_source = "query"
attempts_i = 3
if attempts is not None:
try:
attempts_i = int(attempts)
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported attempts", status_code=400) from exc
else:
raw = random_defaults.get("default_attempts")
if raw is None:
attempts_source = "fallback"
attempts_i = 3
else:
attempts_source = "runtime"
try:
attempts_i = int(raw)
except Exception:
attempts_i = 3
if attempts_i < 1 or attempts_i > 10:
if attempts_source == "query":
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported attempts", status_code=400)
attempts_source = "fallback"
attempts_i = 3
attempts = int(attempts_i)
r18_strict_source = "query"
r18_strict_i = 1
if r18_strict is not None:
try:
r18_strict_i = int(r18_strict)
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported r18_strict", status_code=400) from exc
else:
raw = random_defaults.get("default_r18_strict")
if raw is None:
r18_strict_source = "fallback"
r18_strict_i = 1
elif isinstance(raw, bool):
r18_strict_source = "runtime"
r18_strict_i = 1 if raw else 0
else:
r18_strict_source = "runtime"
try:
r18_strict_i = int(raw)
except Exception:
r18_strict_i = 1
if r18_strict_i not in {0, 1}:
if r18_strict_source == "query":
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported r18_strict", status_code=400)
r18_strict_source = "fallback"
r18_strict_i = 1
r18_strict = int(r18_strict_i)
fail_cooldown_source = "runtime"
fail_cooldown_ms = random_defaults.get("fail_cooldown_ms")
try:
fail_cooldown_ms_i = int(fail_cooldown_ms) if fail_cooldown_ms is not None else None
except Exception:
fail_cooldown_ms_i = None
if fail_cooldown_ms_i is None:
fail_cooldown_source = "fallback"
cooldown_s_raw = (os.environ.get("RANDOM_FAIL_COOLDOWN_SECONDS") or "600").strip()
try:
cooldown_s = int(cooldown_s_raw)
except Exception:
cooldown_s = 600
cooldown_s = max(0, min(int(cooldown_s), 24 * 60 * 60))
fail_cooldown_ms_i = int(cooldown_s) * 1000
fail_cooldown_ms_i = max(0, min(int(fail_cooldown_ms_i), 24 * 60 * 60 * 1000))
request_now = datetime.now(timezone.utc)
fail_cooldown_before = (
iso_utc_ms(request_now - timedelta(milliseconds=int(fail_cooldown_ms_i)))
if int(fail_cooldown_ms_i) > 0
else None
)
pick_kwargs: dict[str, Any] = {
"r18": r18,
"r18_strict": bool(r18_strict),
"ai_type": ai_type_i,
"illust_type": illust_type_i,
"orientation": orientation_map[layout_norm],
"min_width": int(min_width_i),
"min_height": int(min_height_i),
"min_pixels": int(min_pixels_i),
"min_bookmarks": int(min_bookmarks_i),
"min_views": int(min_views_i),
"min_comments": int(min_comments_i),
"included_tags": included,
"excluded_tags": excluded,
"user_id": user_id,
"illust_id": illust_id,
"created_from": created_from_norm,
"created_to": created_to_norm,
"fail_cooldown_before": fail_cooldown_before,
}
rng = random.Random(seed_norm) if seed_norm else random
strategy_raw = (strategy or "").strip().lower()
strategy_source = "query"
if not strategy_raw:
strategy_source = "runtime"
strategy_raw = str(random_defaults.get("strategy") or "").strip().lower()
if not strategy_raw:
strategy_source = "fallback"
strategy_raw = "quality"
if strategy_raw not in {"quality", "random"}:
if strategy_source == "query":
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported strategy", status_code=400)
strategy_source = "fallback"
strategy_raw = "quality"
strategy_norm = strategy_raw
quality_samples_i: int
quality_samples_source = "query"
if quality_samples is not None:
try:
quality_samples_i = int(quality_samples)
except Exception as exc:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported quality_samples", status_code=400) from exc
else:
raw = random_defaults.get("quality_samples")
if raw is None:
quality_samples_source = "fallback"
quality_samples_i = 5
else:
quality_samples_source = "runtime"
try:
quality_samples_i = int(raw)
except Exception:
quality_samples_i = 5
if quality_samples_i < 1 or quality_samples_i > 1000:
if quality_samples_source == "query":
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported quality_samples", status_code=400)
quality_samples_source = "fallback"
quality_samples_i = 5
recommendation_raw = random_defaults.get("recommendation")
recommendation_source = "fallback"
recommendation_obj: dict[str, Any] = {}
if isinstance(recommendation_raw, dict):
recommendation_source = "runtime"
recommendation_obj = dict(recommendation_raw)
pick_mode_raw = str(recommendation_obj.get("pick_mode") or _DEFAULT_RECOMMENDATION["pick_mode"]).strip().lower()
if pick_mode_raw not in {"best", "weighted"}:
pick_mode_raw = str(_DEFAULT_RECOMMENDATION["pick_mode"])
temperature_raw = _as_float(recommendation_obj.get("temperature"), default=float(_DEFAULT_RECOMMENDATION["temperature"]))
temperature = float(max(0.05, min(float(temperature_raw), 100.0)))
score_weights_raw = recommendation_obj.get("score_weights")
score_weights_obj = score_weights_raw if isinstance(score_weights_raw, dict) else {}
score_weights: dict[str, float] = {}
for key, default_value in _DEFAULT_SCORE_WEIGHTS.items():
v = _as_float(score_weights_obj.get(key), default=float(default_value))
score_weights[key] = float(max(-100.0, min(float(v), 100.0)))
multipliers_default = _DEFAULT_RECOMMENDATION["multipliers"]
multipliers_raw = recommendation_obj.get("multipliers")
multipliers_obj = multipliers_raw if isinstance(multipliers_raw, dict) else {}
multipliers: dict[str, float] = {}
for key, default_value in multipliers_default.items():
v = _as_float(multipliers_obj.get(key), default=float(default_value))
multipliers[key] = float(max(0.0, min(float(v), 100.0)))
debug_base = {
"attempts": int(attempts_i),
"attempts_source": attempts_source,
"r18_strict": int(r18_strict_i),
"r18_strict_source": r18_strict_source,
"fail_cooldown_ms": int(fail_cooldown_ms_i),
"fail_cooldown_source": fail_cooldown_source,
"strategy": strategy_norm,
"strategy_source": strategy_source,
"quality_samples": int(quality_samples_i),
"quality_samples_source": quality_samples_source,
"recommendation_source": recommendation_source,
}
async def _pick_with_strategy(
*,
session: Any,
exclude_image_ids: list[int] | None = None,
) -> tuple[Any, dict[str, Any]] | tuple[None, dict[str, Any]]:
if strategy_norm == "random":
image = await pick_random_image(session, r=rng.random(), exclude_image_ids=exclude_image_ids, **pick_kwargs)
if image is None:
return None, {**debug_base, "attempts_used": 1, "picked_by": "random_key"}
return image, {**debug_base, "attempts_used": 1, "picked_by": "random_key"}
def _multiplier_for_image(image: Any) -> float:
m = 1.0
ai = getattr(image, "ai_type", None)
if ai == 1:
m *= float(multipliers.get("ai", 1.0))
elif ai == 0:
m *= float(multipliers.get("non_ai", 1.0))
else:
m *= float(multipliers.get("unknown_ai", 1.0))
it = getattr(image, "illust_type", None)
if it == 0:
m *= float(multipliers.get("illust", 1.0))
elif it == 1:
m *= float(multipliers.get("manga", 1.0))
elif it == 2:
m *= float(multipliers.get("ugoira", 1.0))
else:
m *= float(multipliers.get("unknown_illust_type", 1.0))
if not math.isfinite(float(m)) or float(m) <= 0.0:
return 0.0
return float(m)
exclude_set: set[int] = set(int(x) for x in exclude_image_ids or [])
candidates: list[tuple[Any, float, float, float]] = []
# 批量抽样:一次性取 N 个候选(必要时 wrap-around 再取一次),避免 N 次 DB 循环查询。
# 若用户把某些类别倍率设为 0(例如 manga=0),直接在 SQL 抽样阶段剔除,减少无效候选。
ai_allowed: set[int | None] = set()
if float(multipliers.get("ai", 1.0)) > 0.0:
ai_allowed.add(1)
if float(multipliers.get("non_ai", 1.0)) > 0.0:
ai_allowed.add(0)
if float(multipliers.get("unknown_ai", 1.0)) > 0.0:
ai_allowed.add(None)
illust_allowed: set[int | None] = set()
if float(multipliers.get("illust", 1.0)) > 0.0:
illust_allowed.add(0)
if float(multipliers.get("manga", 1.0)) > 0.0:
illust_allowed.add(1)
if float(multipliers.get("ugoira", 1.0)) > 0.0:
illust_allowed.add(2)
if float(multipliers.get("unknown_illust_type", 1.0)) > 0.0:
illust_allowed.add(None)
if not ai_allowed or not illust_allowed:
return None, {
**debug_base,
"attempts_used": 1,
"picked_by": "quality_weighted" if pick_mode_raw == "weighted" else "quality_best",
"candidates_drawn": 0,
"candidates_accepted": 0,
"quality_pick_mode": pick_mode_raw,
"quality_temperature": float(temperature),
}
images = await pick_random_images(
session,
r=rng.random(),
limit=int(quality_samples_i),
exclude_image_ids=list(exclude_set),
ai_type_allowed=ai_allowed,
illust_type_allowed=illust_allowed,
**pick_kwargs,
)
drawn = int(len(images))
accepted = 0
for image in images:
score = _quality_score(image, weights=score_weights)
multiplier = _multiplier_for_image(image)
if multiplier <= 0.0:
continue
logit = float(score) / float(temperature) + math.log(float(multiplier))
candidates.append((image, float(score), float(multiplier), float(logit)))
accepted += 1
if not candidates:
return None, {
**debug_base,
"attempts_used": 1,
"picked_by": "quality_weighted" if pick_mode_raw == "weighted" else "quality_best",
"candidates_drawn": int(drawn),
"candidates_accepted": int(accepted),
"quality_pick_mode": pick_mode_raw,
"quality_temperature": float(temperature),
}
if pick_mode_raw == "best":
picked = max(candidates, key=lambda x: x[3])
picked_by = "quality_best"
else:
max_logit = max(x[3] for x in candidates)
weights = [math.exp(float(x[3]) - float(max_logit)) for x in candidates]
total = float(sum(weights))
if not math.isfinite(total) or total <= 0.0:
picked = max(candidates, key=lambda x: x[3])
picked_by = "quality_best"
else:
r = float(rng.random()) * total
idx = 0
for i, w in enumerate(weights):
r -= float(w)
if r <= 0:
idx = i
break
picked = candidates[int(max(0, min(idx, len(candidates) - 1)))]
picked_by = "quality_weighted"
best_image, best_score, best_multiplier, _best_logit = picked
return (
best_image,
{
**debug_base,
"attempts_used": 1,
"picked_by": picked_by,
"candidates_drawn": int(drawn),
"candidates_accepted": int(accepted),
"quality_pick_mode": pick_mode_raw,
"quality_temperature": float(temperature),
"quality_score": float(best_score),
"quality_multiplier": float(best_multiplier),
},
)
def _needs_opportunistic_hydrate(image: Any) -> bool:
return (
getattr(image, "width", None) is None
or getattr(image, "height", None) is None
or getattr(image, "x_restrict", None) is None
or getattr(image, "ai_type", None) is None
or getattr(image, "illust_type", None) is None
or getattr(image, "user_id", None) is None
or getattr(image, "bookmark_count", None) is None
or getattr(image, "view_count", None) is None
or getattr(image, "comment_count", None) is None
)
if format in {"json", "simple_json"} or (format == "image" and redirect == 1):
tags: list[str] = []
debug: dict[str, Any] = {}
async with Session() as session:
image, debug = await _pick_with_strategy(session=session)
if image is None:
raise _no_match_error()
if format == "json":
tags = await get_tag_names_for_image(session, image_id=image.id)
if _needs_opportunistic_hydrate(image):
try:
await enqueue_opportunistic_hydrate_metadata(engine, illust_id=int(image.illust_id), reason="random")
except Exception:
pass
if format == "image" and redirect == 1:
return RedirectResponse(
url=f"/i/{image.id}.{image.ext}",
status_code=302,
headers={"Cache-Control": "no-store"},
)
origin_url = None if runtime.hide_origin_url_in_public_json else image.original_url
imgproxy_url = None
try:
cfg = load_imgproxy_config_from_settings(request.app.state.settings)
except Exception:
cfg = None
if cfg is not None:
try:
if runtime.hide_origin_url_in_public_json:
base = str(getattr(request, "base_url", "") or "").rstrip("/")
source_url = f"{base}/i/{image.id}.{image.ext}"
else:
source_url = str(image.original_url)
imgproxy_url = build_signed_processing_url(cfg, source_url=source_url, extension=str(image.ext))
except Exception:
imgproxy_url = None
if format == "simple_json":
return {
"ok": True,
"code": "OK",
"request_id": getattr(getattr(request, "state", None), "request_id", None) or "req_unknown",
"data": {
"image": {
"id": str(image.id),
"illust_id": str(image.illust_id),
"page_index": image.page_index,
"ext": image.ext,
"width": image.width,
"height": image.height,
"x_restrict": image.x_restrict,
"ai_type": image.ai_type,
"illust_type": getattr(image, "illust_type", None),
"bookmark_count": getattr(image, "bookmark_count", None),
"view_count": getattr(image, "view_count", None),
"comment_count": getattr(image, "comment_count", None),
},
"urls": {
"proxy": f"/i/{image.id}.{image.ext}",
"origin": origin_url,
"imgproxy": imgproxy_url,
},
"debug": {
**debug,
},
},
}
return {
"ok": True,
"code": "OK",
"request_id": getattr(getattr(request, "state", None), "request_id", None) or "req_unknown",
"data": {
"image": {
"id": str(image.id),
"illust_id": str(image.illust_id),
"page_index": image.page_index,
"ext": image.ext,
"width": image.width,
"height": image.height,
"x_restrict": image.x_restrict,
"ai_type": image.ai_type,
"illust_type": getattr(image, "illust_type", None),
"bookmark_count": getattr(image, "bookmark_count", None),
"view_count": getattr(image, "view_count", None),
"comment_count": getattr(image, "comment_count", None),
"user": {
"id": str(image.user_id) if image.user_id is not None else None,
"name": image.user_name,
},
"title": image.title,
"created_at_pixiv": image.created_at_pixiv,
},
"tags": tags,
"urls": {
"proxy": f"/i/{image.id}.{image.ext}",
"origin": origin_url,
"imgproxy": imgproxy_url,
"legacy_single": f"/{image.illust_id}.{image.ext}",
"legacy_multi": f"/{image.illust_id}-{image.page_index + 1}.{image.ext}",
},
"debug": {
**debug,
},
},
}
tried_ids: set[int] = set()
last_error: ApiError | None = None
attempts_i = int(attempts)
runtime_stream = runtime
for _ in range(attempts_i):
async with Session() as session:
image, _debug = await _pick_with_strategy(session=session, exclude_image_ids=list(tried_ids))
if image is None:
break
image_id = int(image.id)
origin_url = str(image.original_url)
illust_id_for_hydrate = int(image.illust_id)
needs_hydrate = _needs_opportunistic_hydrate(image)
transport = getattr(request.app.state, "httpx_transport", None)
proxy_uri = None
picked = await select_proxy_uri_for_url(
engine,
request.app.state.settings,
runtime_stream,
url=origin_url,
)
if picked is not None:
proxy_uri = picked.uri
try:
resp = await stream_url(
origin_url,
transport=transport,
proxy=proxy_uri,
cache_control="no-store",
range_header=request.headers.get("Range"),
)
await mark_image_ok(engine, image_id=image_id, now=iso_utc_ms())
if needs_hydrate:
try:
await enqueue_opportunistic_hydrate_metadata(
engine,
illust_id=illust_id_for_hydrate,
reason="random",
)
except Exception:
pass
return resp
except ApiError as exc:
if exc.code in {
ErrorCode.UPSTREAM_STREAM_ERROR,
ErrorCode.UPSTREAM_403,
ErrorCode.UPSTREAM_404,
ErrorCode.UPSTREAM_RATE_LIMIT,
}:
await mark_image_failure(
engine,
image_id=image_id,
now=iso_utc_ms(),
error_code=exc.code.value,
error_message=exc.message,
)
tried_ids.add(image_id)
last_error = exc
continue
raise
if last_error is None:
raise _no_match_error()
raise ApiError(
code=ErrorCode.UPSTREAM_STREAM_ERROR,
message="多次尝试后上游请求仍失败。",
status_code=502,
details={
"attempts_used": len(tried_ids),
"last_upstream_code": last_error.code.value,
},
)
+54
View File
@@ -0,0 +1,54 @@
from __future__ import annotations
from typing import Any
from fastapi import APIRouter, Request
from fastapi.responses import JSONResponse
from app.core.errors import ApiError, ErrorCode
from app.core.request_id import get_or_create_request_id
from app.db.session import create_sessionmaker
from app.db.tags_list import list_tags as db_list_tags
router = APIRouter()
@router.get("/tags")
async def list_tags(
request: Request,
q: str | None = None,
limit: int = 50,
cursor: str | None = None,
) -> Any:
if limit < 1 or limit > 100:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported limit", status_code=400)
q_norm = (q or "").strip()
if q_norm and len(q_norm) > 200:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Unsupported q", status_code=400)
engine = request.app.state.engine
Session = create_sessionmaker(engine)
async with Session() as session:
items, next_cursor = await db_list_tags(session, limit=limit, cursor=cursor, q=q_norm or None)
rid = get_or_create_request_id(request)
resp = JSONResponse(
status_code=200,
content={
"ok": True,
"items": [
{
"id": str(item.id),
"name": item.name,
"translated_name": item.translated_name,
"count_images": item.count_images,
}
for item in items
],
"next_cursor": next_cursor or "",
"request_id": rid,
},
)
return resp
+35
View File
@@ -0,0 +1,35 @@
from __future__ import annotations
import os
from typing import Any
from fastapi import APIRouter, Request
from fastapi.responses import JSONResponse
from app.core.request_id import get_or_create_request_id, set_request_id_header, set_request_id_on_state
router = APIRouter()
def _get_env(key: str, default: str) -> str:
value = os.environ.get(key, default)
return str(value).strip()
@router.get("/version")
async def version(request: Request) -> Any:
rid = get_or_create_request_id(request)
set_request_id_on_state(request, rid)
body = {
"ok": True,
"version": _get_env("APP_VERSION", "dev"),
"build_time": _get_env("APP_BUILD_TIME", ""),
"git_commit": _get_env("APP_COMMIT", ""),
"request_id": rid,
}
resp = JSONResponse(status_code=200, content=body)
set_request_id_header(resp, rid)
return resp
+182
View File
@@ -0,0 +1,182 @@
from __future__ import annotations
import hashlib
import hmac
import time
from dataclasses import dataclass, field
from collections.abc import Mapping
import sqlalchemy as sa
from sqlalchemy.ext.asyncio import AsyncEngine
from app.core.errors import ApiError, ErrorCode
from app.db.models.api_keys import ApiKey
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
def _coerce_int(value: int, *, default: int = 0) -> int:
try:
return int(value)
except Exception:
return default
def _normalize_api_key(value: str) -> str:
v = (value or "").strip()
return v
def hmac_sha256_hex(*, secret_key: str, message: str) -> str:
secret_key = (secret_key or "").strip()
if not secret_key:
raise ValueError("SECRET_KEY is required")
mac = hmac.new(secret_key.encode("utf-8"), message.encode("utf-8"), digestmod=hashlib.sha256)
return mac.hexdigest()
def api_key_hint(api_key: str) -> str:
api_key = _normalize_api_key(api_key)
if not api_key:
return ""
digest = hashlib.sha256(api_key.encode("utf-8")).hexdigest()
return digest[:8]
@dataclass(frozen=True, slots=True)
class ApiKeyAuthConfig:
required: bool
rpm: int
burst: int
secret_key: str
@dataclass(slots=True)
class _CacheEntry:
api_key_id: int
enabled: bool
expires_at_m: float
@dataclass(slots=True)
class ApiKeyAuthenticator:
engine: AsyncEngine
cfg: ApiKeyAuthConfig
cache_ttl_s: float = 5.0
_cache: dict[str, _CacheEntry] = field(default_factory=dict)
async def _lookup(self, key_hash: str, *, now_m: float) -> _CacheEntry | None:
Session = create_sessionmaker(self.engine)
async def _op() -> _CacheEntry | None:
async with Session() as session:
row = (
(
await session.execute(
sa.select(ApiKey.id, ApiKey.enabled).where(ApiKey.key_hash == key_hash).limit(1)
)
)
.first()
)
if row is None:
return None
return _CacheEntry(
api_key_id=int(row[0]),
enabled=bool(int(row[1] or 0)),
expires_at_m=float(now_m) + float(self.cache_ttl_s),
)
return await with_sqlite_busy_retry(_op)
async def authenticate(self, api_key: str) -> int | None:
api_key = _normalize_api_key(api_key)
if not api_key:
return None
key_hash = hmac_sha256_hex(secret_key=self.cfg.secret_key, message=api_key)
now_m = time.monotonic()
cached = self._cache.get(key_hash)
if cached is not None and float(cached.expires_at_m) > float(now_m):
return int(cached.api_key_id) if bool(cached.enabled) else None
entry = await self._lookup(key_hash, now_m=now_m)
if entry is None:
self._cache[key_hash] = _CacheEntry(api_key_id=0, enabled=False, expires_at_m=float(now_m) + 2.0)
return None
self._cache[key_hash] = entry
if len(self._cache) > 10_000:
self._cache = {k: v for k, v in self._cache.items() if float(v.expires_at_m) > float(now_m)}
return int(entry.api_key_id) if bool(entry.enabled) else None
@dataclass(slots=True)
class _Bucket:
tokens: float
updated_at_m: float
@dataclass(slots=True)
class ApiKeyRateLimiter:
rpm: int
burst: int
_buckets: dict[int, _Bucket] = field(default_factory=dict)
def _params(self) -> tuple[float, float]:
rpm = max(0, _coerce_int(self.rpm))
if rpm <= 0:
return 0.0, 0.0
capacity = max(1.0, float(_coerce_int(self.burst)) or float(rpm))
refill_per_s = float(rpm) / 60.0
return capacity, refill_per_s
def allow(self, api_key_id: int) -> bool:
api_key_id_i = _coerce_int(api_key_id)
if api_key_id_i <= 0:
return False
capacity, refill_per_s = self._params()
if capacity <= 0.0 or refill_per_s <= 0.0:
return True
now_m = time.monotonic()
b = self._buckets.get(api_key_id_i)
if b is None:
self._buckets[api_key_id_i] = _Bucket(tokens=float(capacity) - 1.0, updated_at_m=now_m)
return True
elapsed = max(0.0, float(now_m) - float(b.updated_at_m))
b.tokens = min(float(capacity), float(b.tokens) + elapsed * float(refill_per_s))
b.updated_at_m = now_m
if float(b.tokens) >= 1.0:
b.tokens -= 1.0
return True
return False
def extract_api_key(headers: Mapping[str, str] | None) -> str | None:
if not headers:
return None
return (headers.get("X-API-Key") or headers.get("x-api-key") or "").strip() or None
async def require_public_api_key(
authenticator: ApiKeyAuthenticator,
limiter: ApiKeyRateLimiter,
*,
headers: Mapping[str, str] | None,
) -> int:
api_key = extract_api_key(headers)
if not api_key:
raise ApiError(code=ErrorCode.UNAUTHORIZED, message="Missing API key", status_code=401)
api_key_id = await authenticator.authenticate(api_key)
if api_key_id is None:
raise ApiError(code=ErrorCode.UNAUTHORIZED, message="Invalid API key", status_code=401)
if not limiter.allow(api_key_id):
raise ApiError(code=ErrorCode.RATE_LIMITED, message="Rate limited", status_code=429)
return int(api_key_id)
+204
View File
@@ -0,0 +1,204 @@
from __future__ import annotations
from typing import Any
import sqlalchemy as sa
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.errors import ApiError, ErrorCode
from app.db.models.pixiv_tokens import PixivToken
from app.db.models.proxy_endpoints import ProxyEndpoint
from app.db.models.proxy_pool_endpoints import ProxyPoolEndpoint
from app.db.models.token_proxy_bindings import TokenProxyBinding
def _fnv1a64(text: str) -> int:
h = 14695981039346656037
prime = 1099511628211
for b in text.encode("utf-8"):
h ^= b
h = (h * prime) & 0xFFFFFFFFFFFFFFFF
return h
def _rendezvous_proxy_order(*, token_id: int, proxy_ids: list[int], salt: str) -> list[int]:
scored = [(_fnv1a64(f"{token_id}|{pid}|{salt}"), pid) for pid in proxy_ids]
scored.sort(key=lambda x: (-x[0], x[1]))
return [pid for _, pid in scored]
def _compute_primary_assignments(
*,
token_ids: list[int],
proxy_ids: list[int],
capacity_by_proxy_id: dict[int, int],
salt: str,
) -> dict[int, int]:
remaining = {pid: int(capacity_by_proxy_id.get(int(pid), 0)) for pid in proxy_ids}
out: dict[int, int] = {}
for token_id in token_ids:
for pid in _rendezvous_proxy_order(token_id=token_id, proxy_ids=proxy_ids, salt=salt):
if remaining.get(pid, 0) > 0:
out[token_id] = pid
remaining[pid] -= 1
break
return out
def _compute_primary_assignments_soft(
*,
token_ids: list[int],
proxy_ids: list[int],
capacity_by_proxy_id: dict[int, int],
salt: str,
) -> tuple[dict[int, int], int]:
out = _compute_primary_assignments(
token_ids=token_ids,
proxy_ids=proxy_ids,
capacity_by_proxy_id=capacity_by_proxy_id,
salt=salt,
)
over_capacity = 0
for token_id in token_ids:
if token_id in out:
continue
order = _rendezvous_proxy_order(token_id=token_id, proxy_ids=proxy_ids, salt=salt)
if not order:
continue
out[token_id] = int(order[0])
over_capacity += 1
return out, int(over_capacity)
async def recompute_token_proxy_bindings(
session: AsyncSession,
*,
pool_id: int,
now: str,
max_tokens_per_proxy: int,
strict: bool,
) -> dict[str, Any]:
if int(pool_id) <= 0:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid pool_id", status_code=400)
if int(max_tokens_per_proxy) <= 0 or int(max_tokens_per_proxy) > 1000:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="Invalid max_tokens_per_proxy", status_code=400)
proxy_rows = (
(
await session.execute(
sa.select(ProxyPoolEndpoint.endpoint_id, ProxyPoolEndpoint.weight)
.join(ProxyEndpoint, ProxyEndpoint.id == ProxyPoolEndpoint.endpoint_id)
.where(ProxyPoolEndpoint.pool_id == int(pool_id))
.where(ProxyPoolEndpoint.enabled == 1)
.where(ProxyEndpoint.enabled == 1)
.order_by(ProxyPoolEndpoint.endpoint_id.asc())
)
)
.all()
)
proxies: list[tuple[int, int]] = []
for endpoint_id, weight in proxy_rows:
try:
eid = int(endpoint_id)
except Exception:
continue
try:
w = int(weight or 0)
except Exception:
w = 0
if eid <= 0:
continue
proxies.append((eid, w))
capacity_by_proxy_id = {pid: int(max_tokens_per_proxy) * max(0, int(w)) for pid, w in proxies}
proxy_ids_norm = [pid for pid, _w in proxies if capacity_by_proxy_id.get(pid, 0) > 0]
weight_sum = sum(max(0, int(w)) for pid, w in proxies if int(pid) in set(proxy_ids_norm))
if not proxy_ids_norm:
raise ApiError(code=ErrorCode.BAD_REQUEST, message="No enabled proxies in pool", status_code=400)
token_ids = (
(
await session.execute(
sa.select(PixivToken.id)
.where(PixivToken.enabled == 1)
.where(PixivToken.weight > 0)
.order_by(PixivToken.id.asc())
)
)
.scalars()
.all()
)
if not token_ids:
return {"recomputed": 0}
capacity = sum(int(capacity_by_proxy_id.get(int(pid), 0)) for pid in proxy_ids_norm)
if strict and len(token_ids) > capacity:
raise ApiError(
code=ErrorCode.BAD_REQUEST,
message="代理容量不足(请增加节点或调高单代理最多绑定令牌数)",
status_code=400,
details={
"token_count": len(token_ids),
"proxy_count": len(proxy_ids_norm),
"max_tokens_per_proxy": int(max_tokens_per_proxy),
"weight_sum": int(weight_sum),
"capacity": int(capacity),
},
)
salt = f"pool:{pool_id}"
over_capacity_assigned = 0
if strict:
assignments = _compute_primary_assignments(
token_ids=token_ids,
proxy_ids=proxy_ids_norm,
capacity_by_proxy_id=capacity_by_proxy_id,
salt=salt,
)
else:
assignments, over_capacity_assigned = _compute_primary_assignments_soft(
token_ids=token_ids,
proxy_ids=proxy_ids_norm,
capacity_by_proxy_id=capacity_by_proxy_id,
salt=salt,
)
for token_id in token_ids:
primary_proxy_id = assignments.get(int(token_id))
if primary_proxy_id is None:
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Binding recompute failed", status_code=500)
stmt = sqlite_insert(TokenProxyBinding).values(
token_id=int(token_id),
pool_id=int(pool_id),
primary_proxy_id=int(primary_proxy_id),
override_proxy_id=None,
override_expires_at=None,
updated_at=now,
)
stmt = stmt.on_conflict_do_update(
index_elements=[TokenProxyBinding.token_id, TokenProxyBinding.pool_id],
set_={"primary_proxy_id": int(primary_proxy_id), "updated_at": now},
)
await session.execute(stmt)
resp: dict[str, Any] = {"recomputed": len(token_ids)}
if not strict:
resp.update(
{
"strict": False,
"over_capacity_assigned": int(over_capacity_assigned),
"capacity": int(capacity),
"token_count": len(token_ids),
"proxy_count": len(proxy_ids_norm),
"max_tokens_per_proxy": int(max_tokens_per_proxy),
"weight_sum": int(weight_sum),
}
)
return resp
+195
View File
@@ -0,0 +1,195 @@
from __future__ import annotations
import os
from dataclasses import dataclass
from pathlib import Path
from typing import Mapping
from cryptography.fernet import Fernet
from app.core.crypto import FieldEncryptor
from app.core.logging import get_logger
log = get_logger(__name__)
@dataclass(frozen=True, slots=True)
class Settings:
app_env: str
database_url: str
secret_key: str
field_encryption_key: str
admin_username: str
admin_password: str
pixiv_oauth_client_id: str
pixiv_oauth_client_secret: str
pixiv_oauth_hash_secret: str
imgproxy_base_url: str
imgproxy_key: str
imgproxy_salt: str
imgproxy_max_dim: int
imgproxy_default_options: str
imgproxy_url_chunk_size: int
public_api_key_required: bool
public_api_key_rpm: int
public_api_key_burst: int
@property
def is_prod(self) -> bool:
return self.app_env in {"prod", "production"}
def _get(env: Mapping[str, str], key: str, default: str) -> str:
value = env.get(key, default)
return value.strip()
def _get_bool(env: Mapping[str, str], key: str, default: bool) -> bool:
raw = _get(env, key, "1" if default else "0").lower()
if raw in {"1", "true", "yes", "y", "on"}:
return True
if raw in {"0", "false", "no", "n", "off"}:
return False
return default
def _read_key_file(path: Path) -> str | None:
try:
raw = path.read_text(encoding="utf-8")
except FileNotFoundError:
return None
except Exception as exc:
log.warning("field_encryption_key_read_failed path=%s err=%s", str(path), type(exc).__name__)
return None
value = raw.strip()
return value or None
def _atomic_write(path: Path, *, content: str) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_name(path.name + ".tmp")
tmp.write_text(content, encoding="utf-8")
try:
os.chmod(tmp, 0o600)
except Exception:
pass
os.replace(tmp, path)
def _ensure_field_encryption_key(env: Mapping[str, str], *, app_env: str) -> str:
key = _get(env, "FIELD_ENCRYPTION_KEY", "")
if key:
FieldEncryptor.from_key(key)
return key
file_raw = _get(env, "FIELD_ENCRYPTION_KEY_FILE", "")
key_file = Path(file_raw) if file_raw else Path("./data/field_encryption_key")
from_file = _read_key_file(key_file)
if from_file is not None:
FieldEncryptor.from_key(from_file)
return from_file
if app_env in {"prod", "production"}:
return ""
generated = Fernet.generate_key().decode("utf-8")
try:
_atomic_write(key_file, content=generated + "\n")
log.info("field_encryption_key_generated path=%s", str(key_file))
except Exception as exc:
log.warning(
"field_encryption_key_generated_not_persisted path=%s err=%s",
str(key_file),
type(exc).__name__,
)
return generated
def _load_pixiv_oauth_config(env: Mapping[str, str]) -> tuple[str, str, str]:
# NOTE: We intentionally do NOT ship any embedded Pixiv OAuth credentials.
# If you want token refresh / metadata hydration jobs, configure these via env.
client_id = _get(env, "PIXIV_OAUTH_CLIENT_ID", "")
client_secret = _get(env, "PIXIV_OAUTH_CLIENT_SECRET", "")
hash_secret = _get(env, "PIXIV_OAUTH_HASH_SECRET", "")
return client_id, client_secret, hash_secret
def load_settings(env: Mapping[str, str] | None = None) -> Settings:
env = env or os.environ
app_env = _get(env, "APP_ENV", "dev").lower()
database_url = _get(env, "DATABASE_URL", "sqlite+aiosqlite:///./data/app.db")
secret_key = _get(env, "SECRET_KEY", "dev-secret-key" if app_env != "prod" else "")
field_encryption_key = _ensure_field_encryption_key(env, app_env=app_env)
admin_username = _get(env, "ADMIN_USERNAME", "admin")
admin_password = _get(env, "ADMIN_PASSWORD", "admin" if app_env != "prod" else "")
pixiv_oauth_client_id, pixiv_oauth_client_secret, pixiv_oauth_hash_secret = _load_pixiv_oauth_config(env)
imgproxy_base_url = _get(env, "IMGPROXY_BASE_URL", "")
imgproxy_key = _get(env, "IMGPROXY_KEY", "")
imgproxy_salt = _get(env, "IMGPROXY_SALT", "")
try:
imgproxy_max_dim = int(_get(env, "IMGPROXY_MAX_DIM", "2048") or "2048")
except Exception:
imgproxy_max_dim = 2048
imgproxy_max_dim = max(16, min(int(imgproxy_max_dim), 20_000))
imgproxy_default_options = _get(env, "IMGPROXY_DEFAULT_OPTIONS", "")
try:
imgproxy_url_chunk_size = int(_get(env, "IMGPROXY_URL_CHUNK_SIZE", "16") or "16")
except Exception:
imgproxy_url_chunk_size = 16
imgproxy_url_chunk_size = max(0, min(int(imgproxy_url_chunk_size), 128))
public_api_key_required = _get_bool(env, "PUBLIC_API_KEY_REQUIRED", False)
try:
public_api_key_rpm = int(_get(env, "PUBLIC_API_KEY_RPM", "0") or "0")
except Exception:
public_api_key_rpm = 0
public_api_key_rpm = max(0, min(int(public_api_key_rpm), 10_000_000))
try:
public_api_key_burst = int(_get(env, "PUBLIC_API_KEY_BURST", "0") or "0")
except Exception:
public_api_key_burst = 0
public_api_key_burst = max(0, min(int(public_api_key_burst), 10_000_000))
settings = Settings(
app_env=app_env,
database_url=database_url,
secret_key=secret_key,
field_encryption_key=field_encryption_key,
admin_username=admin_username,
admin_password=admin_password,
pixiv_oauth_client_id=pixiv_oauth_client_id,
pixiv_oauth_client_secret=pixiv_oauth_client_secret,
pixiv_oauth_hash_secret=pixiv_oauth_hash_secret,
imgproxy_base_url=imgproxy_base_url,
imgproxy_key=imgproxy_key,
imgproxy_salt=imgproxy_salt,
imgproxy_max_dim=imgproxy_max_dim,
imgproxy_default_options=imgproxy_default_options,
imgproxy_url_chunk_size=imgproxy_url_chunk_size,
public_api_key_required=public_api_key_required,
public_api_key_rpm=public_api_key_rpm,
public_api_key_burst=public_api_key_burst,
)
if settings.is_prod:
missing: list[str] = []
if not settings.secret_key:
missing.append("SECRET_KEY")
if not settings.field_encryption_key:
missing.append("FIELD_ENCRYPTION_KEY")
if not settings.admin_password:
missing.append("ADMIN_PASSWORD")
if settings.imgproxy_base_url and (not settings.imgproxy_key or not settings.imgproxy_salt):
missing.append("IMGPROXY_KEY/IMGPROXY_SALT")
if missing:
raise ValueError(f"Missing required env vars for prod: {', '.join(missing)}")
return settings
+44
View File
@@ -0,0 +1,44 @@
from __future__ import annotations
from dataclasses import dataclass
from cryptography.fernet import Fernet, InvalidToken
MASKED = "***"
def mask_secret(value: str | None) -> str:
value = (value or "").strip()
return MASKED if value else ""
@dataclass(frozen=True, slots=True)
class FieldEncryptor:
_fernet: Fernet
@classmethod
def from_key(cls, key: str) -> "FieldEncryptor":
key = (key or "").strip()
if not key:
raise ValueError("FIELD_ENCRYPTION_KEY is required")
try:
fernet = Fernet(key.encode("utf-8"))
except Exception as exc:
raise ValueError("Invalid FIELD_ENCRYPTION_KEY") from exc
return cls(_fernet=fernet)
def encrypt_text(self, plaintext: str) -> str:
if not isinstance(plaintext, str):
raise TypeError("plaintext must be str")
token = self._fernet.encrypt(plaintext.encode("utf-8"))
return token.decode("utf-8")
def decrypt_text(self, token: str) -> str:
if not isinstance(token, str):
raise TypeError("token must be str")
try:
plaintext = self._fernet.decrypt(token.encode("utf-8"))
except InvalidToken as exc:
raise ValueError("Invalid encrypted value") from exc
return plaintext.decode("utf-8")
+57
View File
@@ -0,0 +1,57 @@
from __future__ import annotations
from pathlib import Path
from sqlalchemy.engine.url import make_url
def get_sqlite_db_dir(database_url: str) -> Path:
"""
Returns the directory containing the SQLite database file.
For non-SQLite or in-memory DBs, falls back to ./data relative to CWD.
"""
fallback = Path("./data").resolve()
try:
url = make_url(database_url)
except Exception:
return fallback
backend = (url.get_backend_name() or "").lower()
if backend != "sqlite":
return fallback
db = str(url.database or "").strip()
if not db or db == ":memory:":
return fallback
p = Path(db)
if not p.is_absolute():
p = Path.cwd() / p
return p.resolve().parent
def make_file_ref(path: Path, *, base_dir: Path) -> str:
base = base_dir.resolve()
target = path.resolve()
return target.relative_to(base).as_posix()
def resolve_file_ref(file_ref: str, *, base_dir: Path) -> Path:
raw = (file_ref or "").strip()
if not raw:
raise ValueError("file_ref is required")
rel = Path(raw)
if rel.is_absolute():
raise ValueError("file_ref must be relative")
if any(part == ".." for part in rel.parts):
raise ValueError("file_ref must not contain '..'")
base = base_dir.resolve()
target = (base / rel).resolve()
target.relative_to(base) # raises if escaped
return target
+22
View File
@@ -0,0 +1,22 @@
from __future__ import annotations
from dataclasses import dataclass
from typing import Mapping
from sqlalchemy.ext.asyncio import AsyncEngine
from app.core.config import Settings, load_settings
from app.core.runtime_settings import RuntimeConfig, load_runtime_config
@dataclass(frozen=True, slots=True)
class EffectiveSettings:
env: Settings
runtime: RuntimeConfig
async def load_effective_settings(engine: AsyncEngine, *, env: Mapping[str, str] | None = None) -> EffectiveSettings:
settings = load_settings(env)
runtime = await load_runtime_config(engine)
return EffectiveSettings(env=settings, runtime=runtime)
+187
View File
@@ -0,0 +1,187 @@
from __future__ import annotations
from dataclasses import dataclass
from enum import Enum
from typing import Any
class ErrorCode(str, Enum):
# 4.1 通用
BAD_REQUEST = "BAD_REQUEST"
UNAUTHORIZED = "UNAUTHORIZED"
FORBIDDEN = "FORBIDDEN"
NOT_FOUND = "NOT_FOUND"
RATE_LIMITED = "RATE_LIMITED"
INTERNAL_ERROR = "INTERNAL_ERROR"
# 4.2 Random
NO_MATCH = "NO_MATCH"
UPSTREAM_STREAM_ERROR = "UPSTREAM_STREAM_ERROR"
UPSTREAM_403 = "UPSTREAM_403"
UPSTREAM_404 = "UPSTREAM_404"
UPSTREAM_RATE_LIMIT = "UPSTREAM_RATE_LIMIT"
# 4.3 Import
INVALID_UPLOAD_TYPE = "INVALID_UPLOAD_TYPE"
PAYLOAD_TOO_LARGE = "PAYLOAD_TOO_LARGE"
UNSUPPORTED_URL = "UNSUPPORTED_URL"
# 4.4 Pixiv/Token
TOKEN_REFRESH_FAILED = "TOKEN_REFRESH_FAILED"
TOKEN_BACKOFF = "TOKEN_BACKOFF"
NO_TOKEN_AVAILABLE = "NO_TOKEN_AVAILABLE"
# 4.5 Proxy
PROXY_REQUIRED = "PROXY_REQUIRED"
PROXY_AUTH_FAILED = "PROXY_AUTH_FAILED"
PROXY_CONNECT_FAILED = "PROXY_CONNECT_FAILED"
UNKNOWN_REQUEST_ID = "req_unknown"
_DEFAULT_ZH_MESSAGE_BY_CODE: dict[ErrorCode, str] = {
ErrorCode.BAD_REQUEST: "请求参数错误",
ErrorCode.UNAUTHORIZED: "未授权",
ErrorCode.FORBIDDEN: "禁止访问",
ErrorCode.NOT_FOUND: "资源不存在",
ErrorCode.RATE_LIMITED: "请求过于频繁",
ErrorCode.INTERNAL_ERROR: "服务器内部错误",
ErrorCode.NO_MATCH: "没有匹配的图片",
ErrorCode.UPSTREAM_STREAM_ERROR: "上游请求失败",
ErrorCode.UPSTREAM_403: "上游拒绝访问(403)",
ErrorCode.UPSTREAM_404: "上游资源不存在(404)",
ErrorCode.UPSTREAM_RATE_LIMIT: "上游触发限流(429)",
ErrorCode.INVALID_UPLOAD_TYPE: "上传类型不支持",
ErrorCode.PAYLOAD_TOO_LARGE: "请求体过大",
ErrorCode.UNSUPPORTED_URL: "链接格式不支持",
ErrorCode.TOKEN_REFRESH_FAILED: "令牌刷新失败",
ErrorCode.TOKEN_BACKOFF: "令牌已进入退避",
ErrorCode.NO_TOKEN_AVAILABLE: "没有可用令牌",
ErrorCode.PROXY_REQUIRED: "需要代理",
ErrorCode.PROXY_AUTH_FAILED: "代理认证失败",
ErrorCode.PROXY_CONNECT_FAILED: "代理连接失败",
}
def _is_ascii_only(text: str) -> bool:
return all(ord(ch) < 128 for ch in text)
_ASCII_MESSAGE_MAP: dict[str, str] = {
"missing api key": "缺少 API Key",
"invalid api key": "API Key 无效",
"rate limited": "请求过于频繁",
"database unavailable": "数据库不可用",
"invalid json body": "JSON 请求体无效",
"unsupported limit": "limit 参数不支持",
"unsupported cursor": "cursor 参数不支持",
"unsupported status": "status 参数不支持",
"unsupported type": "type 参数不支持",
"unsupported format": "format 参数不支持",
"unsupported redirect": "redirect 参数不支持",
"unsupported seed": "seed 参数不支持",
"unsupported ai_type": "ai_type 参数不支持",
"unsupported orientation": "orientation 参数不支持",
"unsupported r18": "r18 参数不支持",
"unsupported min_*": "min_* 参数不支持",
"too many tag filters": "标签筛选条件过多",
"pixiv oauth not configured": "未配置 Pixiv OAuth(client_id/client_secret)",
"encryption not configured": "未配置加密密钥(FIELD_ENCRYPTION_KEY)",
"invalid stored token": "令牌密文无效(无法解密)",
"token not found": "令牌不存在",
"job not found": "任务不存在",
"proxy endpoint not found": "代理节点不存在",
"proxy pool not found": "代理池不存在",
"job is running": "任务正在运行中",
"proxy authentication failed": "代理认证失败",
"proxy connect failed": "代理连接失败",
"upstream request failed": "上游请求失败",
"upstream forbidden": "上游拒绝访问(403)",
"upstream not found": "上游资源不存在(404)",
"upstream rate limited": "上游触发限流(429)",
"upstream error": "上游错误",
"upstream request failed after attempts.": "多次尝试后上游请求仍失败",
"invalid proxy endpoint": "代理节点配置无效",
"unsupported conflict_policy": "conflict_policy 参数不支持",
}
def normalize_error_message(*, code: ErrorCode, message: str) -> str:
msg = str(message or "").strip()
if not msg:
return _DEFAULT_ZH_MESSAGE_BY_CODE.get(code, "请求失败")
if not _is_ascii_only(msg):
return msg
key = msg.lower()
if key in _ASCII_MESSAGE_MAP:
return _ASCII_MESSAGE_MAP[key]
if key.startswith("unsupported "):
rest = msg[len("Unsupported ") :].strip()
return f"参数不支持:{rest}" if rest else _DEFAULT_ZH_MESSAGE_BY_CODE.get(code, "请求失败")
if key.startswith("invalid "):
rest = msg[len("Invalid ") :].strip()
return f"参数无效:{rest}" if rest else _DEFAULT_ZH_MESSAGE_BY_CODE.get(code, "请求失败")
if key.startswith("missing "):
rest = msg[len("Missing ") :].strip()
return f"缺少字段:{rest}" if rest else _DEFAULT_ZH_MESSAGE_BY_CODE.get(code, "请求失败")
return _DEFAULT_ZH_MESSAGE_BY_CODE.get(code, "请求失败")
@dataclass(frozen=True, slots=True)
class ApiError(Exception):
code: ErrorCode
message: str
status_code: int = 400
details: dict[str, Any] | None = None
def _coerce_request_id(request_id: str | None) -> str:
request_id = (request_id or "").strip()
return request_id if request_id else UNKNOWN_REQUEST_ID
def _request_id_from_request(request: Any | None) -> str | None:
if request is None:
return None
request_id = getattr(getattr(request, "state", None), "request_id", None)
if request_id:
return str(request_id)
header = request.headers.get("X-Request-Id")
return header.strip() if header else None
def error_body(
*,
code: ErrorCode,
message: str,
request_id: str | None = None,
details: dict[str, Any] | None = None,
) -> dict[str, Any]:
return {
"ok": False,
"code": code.value,
"message": normalize_error_message(code=code, message=message),
"request_id": _coerce_request_id(request_id),
"details": details or {},
}
def json_error_response(
*,
code: ErrorCode,
message: str,
status_code: int,
request: Any | None = None,
request_id: str | None = None,
details: dict[str, Any] | None = None,
) -> Any:
if request_id is None:
request_id = _request_id_from_request(request)
from fastapi.responses import JSONResponse
return JSONResponse(
status_code=status_code,
content=error_body(code=code, message=message, request_id=request_id, details=details),
)
+76
View File
@@ -0,0 +1,76 @@
from __future__ import annotations
from enum import Enum
import httpx
class OutboundErrorKind(str, Enum):
PROXY_CONNECT = "proxy_connect"
PROXY_AUTH = "proxy_auth"
PIXIV_RATE_LIMIT = "pixiv_rate_limit"
def classify_httpx_exception(exc: BaseException) -> OutboundErrorKind | None:
if isinstance(exc, httpx.ProxyError):
msg = str(exc).lower()
if "407" in msg or "proxy authentication" in msg:
return OutboundErrorKind.PROXY_AUTH
return OutboundErrorKind.PROXY_CONNECT
return None
def classify_pixiv_rate_limit(*, status_code: int, body_text: str | None) -> OutboundErrorKind | None:
if int(status_code) != 403:
return None
text = (body_text or "").strip().lower()
if "rate limit" in text:
return OutboundErrorKind.PIXIV_RATE_LIMIT
return None
def should_override_proxy(kind: OutboundErrorKind) -> bool:
return kind in {OutboundErrorKind.PROXY_CONNECT, OutboundErrorKind.PROXY_AUTH}
def should_backoff_token(kind: OutboundErrorKind) -> bool:
return kind == OutboundErrorKind.PIXIV_RATE_LIMIT
def proxy_override_ttl_seconds(*, attempt: int) -> int:
attempt_i = int(attempt)
if attempt_i <= 0:
return 0
schedule = {
1: 20 * 60,
2: 60 * 60,
3: 6 * 60 * 60,
}
if attempt_i in schedule:
return schedule[attempt_i]
base = 6 * 60 * 60
seconds = base * (2 ** (attempt_i - 3))
return min(seconds, 24 * 60 * 60)
def pixiv_rate_limit_backoff_seconds(*, attempt: int) -> int:
attempt_i = int(attempt)
if attempt_i <= 0:
return 0
schedule = {
1: 60,
2: 5 * 60,
3: 15 * 60,
4: 60 * 60,
5: 6 * 60 * 60,
}
if attempt_i in schedule:
return schedule[attempt_i]
base = 6 * 60 * 60
seconds = base * (2 ** (attempt_i - 5))
return min(seconds, 24 * 60 * 60)
+98
View File
@@ -0,0 +1,98 @@
from __future__ import annotations
from typing import Any
import httpx
from starlette.responses import StreamingResponse
from app.core.errors import ApiError, ErrorCode
from app.core.metrics import UPSTREAM_STREAM_ERRORS_TOTAL
PIXIV_REFERER = "https://www.pixiv.net/"
async def stream_url(
url: str,
*,
transport: httpx.BaseTransport | None = None,
proxy: str | None = None,
cache_control: str,
referer: str = PIXIV_REFERER,
timeout_s: float = 30.0,
range_header: str | None = None,
) -> StreamingResponse:
client = httpx.AsyncClient(
transport=transport,
proxy=proxy,
follow_redirects=True,
timeout=httpx.Timeout(timeout_s, connect=10.0),
)
request_headers: dict[str, str] = {}
if referer:
request_headers["Referer"] = referer
if range_header:
request_headers["Range"] = range_header
request = client.build_request("GET", url, headers=request_headers)
try:
upstream = await client.send(request, stream=True)
except httpx.ProxyError as exc:
UPSTREAM_STREAM_ERRORS_TOTAL.inc()
await client.aclose()
msg = str(exc).lower()
if "407" in msg or "proxy authentication" in msg:
raise ApiError(
code=ErrorCode.PROXY_AUTH_FAILED,
message="代理认证失败",
status_code=502,
) from exc
raise ApiError(
code=ErrorCode.PROXY_CONNECT_FAILED,
message="代理连接失败",
status_code=502,
) from exc
except Exception as exc:
UPSTREAM_STREAM_ERRORS_TOTAL.inc()
await client.aclose()
raise ApiError(code=ErrorCode.UPSTREAM_STREAM_ERROR, message="上游请求失败", status_code=502) from exc
if upstream.status_code not in {200, 206}:
status = upstream.status_code
UPSTREAM_STREAM_ERRORS_TOTAL.inc()
await upstream.aclose()
await client.aclose()
if status == 403:
raise ApiError(code=ErrorCode.UPSTREAM_403, message="上游拒绝访问(403)", status_code=502)
if status == 404:
raise ApiError(code=ErrorCode.UPSTREAM_404, message="上游资源不存在(404)", status_code=502)
if status == 429:
raise ApiError(code=ErrorCode.UPSTREAM_RATE_LIMIT, message="上游触发限流(429)", status_code=502)
raise ApiError(code=ErrorCode.UPSTREAM_STREAM_ERROR, message="上游错误", status_code=502)
media_type = upstream.headers.get("content-type") or "application/octet-stream"
content_length = upstream.headers.get("content-length")
async def _iter_bytes() -> Any:
try:
async for chunk in upstream.aiter_bytes():
yield chunk
except Exception:
UPSTREAM_STREAM_ERRORS_TOTAL.inc()
raise
finally:
await upstream.aclose()
await client.aclose()
accept_ranges = upstream.headers.get("accept-ranges")
content_range = upstream.headers.get("content-range")
resp = StreamingResponse(_iter_bytes(), status_code=upstream.status_code, media_type=media_type)
resp.headers["Cache-Control"] = cache_control
if content_length:
resp.headers["Content-Length"] = content_length
if accept_ranges:
resp.headers["Accept-Ranges"] = accept_ranges
if content_range:
resp.headers["Content-Range"] = content_range
return resp
+148
View File
@@ -0,0 +1,148 @@
from __future__ import annotations
import base64
import hashlib
import hmac
from dataclasses import dataclass
from typing import Mapping
from app.core.config import Settings
@dataclass(frozen=True, slots=True)
class ImgproxyConfig:
base_url: str
key: bytes
salt: bytes
max_dim: int
default_options: str
url_chunk_size: int
def _decode_hex(raw: str, *, name: str) -> bytes:
raw = (raw or "").strip()
if not raw:
raise ValueError(f"{name} is required")
try:
return bytes.fromhex(raw)
except Exception as exc:
raise ValueError(f"{name} must be hex") from exc
def load_imgproxy_config_from_settings(settings: Settings) -> ImgproxyConfig | None:
base_url = (settings.imgproxy_base_url or "").strip()
if not base_url:
return None
key = _decode_hex(settings.imgproxy_key, name="IMGPROXY_KEY")
salt = _decode_hex(settings.imgproxy_salt, name="IMGPROXY_SALT")
default_options = (settings.imgproxy_default_options or "").strip().strip("/")
if not default_options:
default_options = f"rs:fit:{int(settings.imgproxy_max_dim)}:{int(settings.imgproxy_max_dim)}"
return ImgproxyConfig(
base_url=base_url.rstrip("/"),
key=key,
salt=salt,
max_dim=int(settings.imgproxy_max_dim),
default_options=default_options,
url_chunk_size=int(settings.imgproxy_url_chunk_size),
)
def urlsafe_b64_no_pad(raw: bytes) -> str:
return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii")
def encode_source_url(source_url: str, *, chunk_size: int) -> str:
source_url = (source_url or "").strip()
if not source_url:
raise ValueError("source_url is required")
encoded = urlsafe_b64_no_pad(source_url.encode("utf-8"))
if chunk_size <= 0 or len(encoded) <= chunk_size:
return encoded
return "/".join(encoded[i : i + chunk_size] for i in range(0, len(encoded), chunk_size))
def sign_path(cfg: ImgproxyConfig, path_after_signature: str) -> str:
path = (path_after_signature or "").strip()
if not path.startswith("/"):
raise ValueError("path_after_signature must start with '/'")
mac = hmac.new(cfg.key, digestmod=hashlib.sha256)
mac.update(cfg.salt)
mac.update(path.encode("utf-8"))
return urlsafe_b64_no_pad(mac.digest())
def build_processing_path(
*,
processing_options: str,
source_url: str,
extension: str,
url_chunk_size: int,
) -> str:
processing_options = (processing_options or "").strip().strip("/")
if not processing_options:
raise ValueError("processing_options is required")
extension = (extension or "").strip().lower().lstrip(".")
if not extension or len(extension) > 10 or any(c for c in extension if not (c.isalnum() or c == "_")):
raise ValueError("extension is invalid")
encoded = encode_source_url(source_url, chunk_size=int(url_chunk_size))
return f"/{processing_options}/{encoded}.{extension}"
def build_signed_processing_url(
cfg: ImgproxyConfig,
*,
source_url: str,
extension: str,
processing_options: str | None = None,
) -> str:
path = build_processing_path(
processing_options=processing_options or cfg.default_options,
source_url=source_url,
extension=extension,
url_chunk_size=int(cfg.url_chunk_size),
)
sig = sign_path(cfg, path)
return f"{cfg.base_url}/{sig}{path}"
def load_imgproxy_config(env: Mapping[str, str]) -> ImgproxyConfig | None:
base_url = (env.get("IMGPROXY_BASE_URL") or "").strip()
if not base_url:
return None
key = _decode_hex(env.get("IMGPROXY_KEY") or "", name="IMGPROXY_KEY")
salt = _decode_hex(env.get("IMGPROXY_SALT") or "", name="IMGPROXY_SALT")
try:
max_dim = int((env.get("IMGPROXY_MAX_DIM") or "2048").strip() or "2048")
except Exception:
max_dim = 2048
max_dim = max(16, min(int(max_dim), 20_000))
default_options = (env.get("IMGPROXY_DEFAULT_OPTIONS") or "").strip().strip("/")
if not default_options:
default_options = f"rs:fit:{max_dim}:{max_dim}"
try:
url_chunk_size = int((env.get("IMGPROXY_URL_CHUNK_SIZE") or "16").strip() or "16")
except Exception:
url_chunk_size = 16
url_chunk_size = max(0, min(int(url_chunk_size), 128))
return ImgproxyConfig(
base_url=base_url.rstrip("/"),
key=key,
salt=salt,
max_dim=max_dim,
default_options=default_options,
url_chunk_size=url_chunk_size,
)
+30
View File
@@ -0,0 +1,30 @@
from __future__ import annotations
import logging
from typing import Any
from app.core.redact import redact_any
class RedactFilter(logging.Filter):
def filter(self, record: logging.LogRecord) -> bool:
try:
message = record.getMessage()
record.msg = redact_any(message)
record.args = ()
for key, value in list(record.__dict__.items()):
if key.startswith("_"):
continue
record.__dict__[key] = redact_any(value)
except Exception:
pass
return True
def configure_logging(level: int = logging.INFO) -> None:
logging.basicConfig(level=level, format="%(levelname)s %(name)s %(message)s")
logging.getLogger().addFilter(RedactFilter())
def get_logger(name: str) -> logging.Logger:
return logging.getLogger(name)
+170
View File
@@ -0,0 +1,170 @@
from __future__ import annotations
from collections.abc import Iterable
from prometheus_client import Counter, Gauge, Histogram
RANDOM_RESULTS: tuple[str, ...] = (
"ok",
"no_match",
"upstream_error",
"bad_request",
"error",
)
JOB_STATUSES: tuple[str, ...] = (
"pending",
"running",
"paused",
"canceled",
"completed",
"failed",
"dlq",
)
PROXY_STATES: tuple[str, ...] = (
"total",
"enabled",
"healthy",
"unhealthy",
"blacklisted",
)
RANDOM_REQUESTS_TOTAL = Counter(
"new_pixiv_random_requests_total",
"Total /random requests by result.",
["result"],
)
RANDOM_NO_MATCH_TOTAL = Counter(
"new_pixiv_random_no_match_total",
"Total /random NO_MATCH responses.",
)
RANDOM_OPPORTUNISTIC_HYDRATE_ENQUEUED_TOTAL = Counter(
"new_pixiv_random_opportunistic_hydrate_enqueued_total",
"Total opportunistic hydrate_metadata enqueues from /random.",
)
RANDOM_LATENCY_SECONDS = Histogram(
"new_pixiv_random_latency_seconds",
"Latency for /random endpoint (seconds).",
buckets=(
0.01,
0.025,
0.05,
0.1,
0.25,
0.5,
1.0,
2.5,
5.0,
10.0,
),
)
UPSTREAM_STREAM_ERRORS_TOTAL = Counter(
"new_pixiv_upstream_stream_errors_total",
"Total upstream stream failures (stream_url).",
)
JOBS_CLAIM_TOTAL = Counter(
"new_pixiv_jobs_claim_total",
"Total jobs claimed by workers.",
)
JOBS_FAILED_TOTAL = Counter(
"new_pixiv_jobs_failed_total",
"Total jobs transitioned to failed/dlq status.",
)
TOKEN_REFRESH_FAIL_TOTAL = Counter(
"new_pixiv_token_refresh_fail_total",
"Total token refresh failures.",
)
JOBS_STATUS_COUNT = Gauge(
"new_pixiv_jobs_status_count",
"Current jobs count by status (from SQLite).",
["status"],
)
PROXY_ENDPOINTS_STATE_COUNT = Gauge(
"new_pixiv_proxy_endpoints_state_count",
"Current proxy endpoints count by state (from SQLite).",
["state"],
)
PROXY_PROBE_LATENCY_MS = Histogram(
"new_pixiv_proxy_probe_latency_ms",
"Proxy probe latency (ms).",
buckets=(
10.0,
25.0,
50.0,
100.0,
250.0,
500.0,
1000.0,
2000.0,
5000.0,
10000.0,
),
)
METRICS_SCRAPE_ERRORS_TOTAL = Counter(
"new_pixiv_metrics_scrape_errors_total",
"Total /metrics scrape errors while querying backing dependencies.",
)
METRICS_LAST_SCRAPE_SUCCESS = Gauge(
"new_pixiv_metrics_last_scrape_success",
"Last /metrics scrape success (1=ok, 0=error).",
)
def _init_labelsets() -> None:
for result in RANDOM_RESULTS:
RANDOM_REQUESTS_TOTAL.labels(result=result).inc(0)
RANDOM_NO_MATCH_TOTAL.inc(0)
RANDOM_OPPORTUNISTIC_HYDRATE_ENQUEUED_TOTAL.inc(0)
UPSTREAM_STREAM_ERRORS_TOTAL.inc(0)
JOBS_CLAIM_TOTAL.inc(0)
JOBS_FAILED_TOTAL.inc(0)
TOKEN_REFRESH_FAIL_TOTAL.inc(0)
for status in JOB_STATUSES:
JOBS_STATUS_COUNT.labels(status=status).set(0)
for state in PROXY_STATES:
PROXY_ENDPOINTS_STATE_COUNT.labels(state=state).set(0)
METRICS_LAST_SCRAPE_SUCCESS.set(1)
_init_labelsets()
def observe_random_result(*, result: str, duration_s: float | None) -> None:
result = (result or "").strip()
if result not in RANDOM_RESULTS:
result = "error"
RANDOM_REQUESTS_TOTAL.labels(result=result).inc()
if result == "no_match":
RANDOM_NO_MATCH_TOTAL.inc()
if duration_s is not None and duration_s >= 0:
RANDOM_LATENCY_SECONDS.observe(duration_s)
def set_jobs_status_counts(counts: dict[str, int]) -> None:
for status in JOB_STATUSES:
JOBS_STATUS_COUNT.labels(status=status).set(float(int(counts.get(status, 0) or 0)))
def set_proxy_state_counts(counts: dict[str, int]) -> None:
for state in PROXY_STATES:
PROXY_ENDPOINTS_STATE_COUNT.labels(state=state).set(float(int(counts.get(state, 0) or 0)))
def ensure_known_keys(keys: Iterable[str], counts: dict[str, int]) -> dict[str, int]:
out = dict(counts)
for k in keys:
out.setdefault(k, 0)
return out
+44
View File
@@ -0,0 +1,44 @@
from __future__ import annotations
import re
from dataclasses import dataclass
from urllib.parse import urlparse
@dataclass(frozen=True, slots=True)
class PixivOriginalUrl:
illust_id: int
page_index: int
ext: str
_PIXIV_P_RE = re.compile(r"(?P<illust_id>\d+)_p(?P<page_index>\d+)(?:_master1200)?\.(?P<ext>[A-Za-z0-9]+)$")
_PIXIV_UGOIRA_RE = re.compile(r"(?P<illust_id>\d+)_ugoira(?P<page_index>\d+)\.(?P<ext>[A-Za-z0-9]+)$")
ALLOWED_IMAGE_EXTS = {"jpg", "jpeg", "png", "gif", "webp"}
def parse_pixiv_original_url(url: str) -> PixivOriginalUrl:
url = url.strip()
if not url:
raise ValueError("url is required")
parsed = urlparse(url)
if parsed.scheme.lower() not in {"http", "https"}:
raise ValueError("unsupported scheme")
host = (parsed.hostname or "").lower()
if not host.endswith("pximg.net"):
raise ValueError("unsupported host")
m = _PIXIV_P_RE.search(parsed.path) or _PIXIV_UGOIRA_RE.search(parsed.path)
if not m:
raise ValueError("unsupported pixiv original url")
illust_id = int(m.group("illust_id"))
page_index = int(m.group("page_index"))
ext = m.group("ext").lower()
if ext not in ALLOWED_IMAGE_EXTS:
raise ValueError("unsupported ext")
return PixivOriginalUrl(illust_id=illust_id, page_index=page_index, ext=ext)
+539
View File
@@ -0,0 +1,539 @@
from __future__ import annotations
import random
from dataclasses import dataclass
from typing import Any
from urllib.parse import urlparse
from urllib.parse import quote
from sqlalchemy.ext.asyncio import AsyncEngine
from app.core.config import Settings
from app.core.crypto import FieldEncryptor
from app.core.errors import ApiError, ErrorCode
from app.core.runtime_settings import RuntimeConfig
from app.core.time import iso_utc_ms
from app.db.session import with_sqlite_busy_retry
_PIXIV_HOST_SUFFIXES = (
"pixiv.net",
"pximg.net",
"secure.pixiv.net",
)
def _normalize_host(host: str) -> str:
return (host or "").strip().lower().strip(".")
def host_from_url(url: str) -> str | None:
url = (url or "").strip()
if not url:
return None
try:
parsed = urlparse(url)
except Exception:
return None
host = _normalize_host(parsed.hostname or "")
return host or None
def _suffix_match(*, host: str, suffix: str) -> bool:
host = _normalize_host(host)
suffix = _normalize_host(suffix)
if not host or not suffix:
return False
return host == suffix or host.endswith("." + suffix)
def should_use_proxy_for_host(runtime: RuntimeConfig, *, host: str) -> bool:
if not bool(runtime.proxy_enabled):
return False
host_n = _normalize_host(host)
if not host_n:
return False
mode = (runtime.proxy_route_mode or "").strip().lower()
if mode in {"off"}:
return False
if mode == "all":
return True
if mode == "allowlist":
return any(_suffix_match(host=host_n, suffix=d) for d in (runtime.proxy_allowlist_domains or []))
if mode == "pixiv_only":
return any(_suffix_match(host=host_n, suffix=s) for s in _PIXIV_HOST_SUFFIXES)
return False
def resolve_pool_id_for_host(runtime: RuntimeConfig, *, host: str) -> int | None:
host_n = _normalize_host(host)
if not host_n:
return None
best_len = -1
best: int | None = None
for suffix, pool_id in (runtime.proxy_route_pools or {}).items():
suf = _normalize_host(str(suffix))
if not suf:
continue
if not _suffix_match(host=host_n, suffix=suf):
continue
if len(suf) > best_len:
best_len = len(suf)
best = int(pool_id)
if best is not None and int(best) > 0:
return int(best)
if runtime.proxy_default_pool_id is not None and int(runtime.proxy_default_pool_id) > 0:
return int(runtime.proxy_default_pool_id)
return None
def _weighted_choice(items: list[tuple[int, int]]) -> int | None:
if not items:
return None
total = sum(max(0, int(w)) for _pid, w in items)
if total <= 0:
items2 = [pid for pid, _w in items]
return int(random.choice(items2)) if items2 else None
r = random.random() * total
for pid, w in items:
w_i = max(0, int(w))
if w_i <= 0:
continue
if r < w_i:
return int(pid)
r -= w_i
return int(items[-1][0])
@dataclass(frozen=True, slots=True)
class ProxyUri:
uri: str
endpoint_id: int
pool_id: int
async def _first_enabled_pool_id(engine: AsyncEngine) -> int | None:
sql = "SELECT id FROM proxy_pools WHERE enabled=1 ORDER BY id ASC LIMIT 1;"
async def _op() -> int | None:
async with engine.connect() as conn:
result = await conn.exec_driver_sql(sql)
value = result.scalar_one_or_none()
return int(value) if value is not None else None
return await with_sqlite_busy_retry(_op)
async def _list_enabled_pool_ids(engine: AsyncEngine) -> list[int]:
sql = "SELECT id FROM proxy_pools WHERE enabled=1 ORDER BY id ASC;"
async def _op() -> list[int]:
async with engine.connect() as conn:
rows = (await conn.exec_driver_sql(sql)).fetchall()
out: list[int] = []
seen: set[int] = set()
for (pid,) in rows:
try:
pool_id = int(pid)
except Exception:
continue
if pool_id <= 0 or pool_id in seen:
continue
seen.add(pool_id)
out.append(pool_id)
return out
return await with_sqlite_busy_retry(_op)
async def _pool_health_stats(engine: AsyncEngine, *, pool_id: int, now_iso: str) -> dict[str, Any]:
sql = """
SELECT
COUNT(*) AS endpoints_total,
COALESCE(SUM(CASE
WHEN pe.blacklisted_until IS NULL OR pe.blacklisted_until <= :now THEN 1
ELSE 0
END), 0) AS endpoints_eligible,
MIN(CASE
WHEN pe.blacklisted_until > :now THEN pe.blacklisted_until
ELSE NULL
END) AS next_available_at
FROM proxy_pools pp
JOIN proxy_pool_endpoints ppe
ON ppe.pool_id = pp.id AND ppe.enabled = 1
JOIN proxy_endpoints pe
ON pe.id = ppe.endpoint_id AND pe.enabled = 1
WHERE pp.id = :pool_id AND pp.enabled = 1;
""".strip()
async def _op() -> dict[str, Any]:
async with engine.connect() as conn:
row = (await conn.exec_driver_sql(sql, {"pool_id": int(pool_id), "now": str(now_iso)})).first()
if row is None:
return {"endpoints_total": 0, "endpoints_eligible": 0, "next_available_at": None}
total = int(row[0] or 0)
eligible = int(row[1] or 0)
next_available_at = str(row[2]).strip() if row[2] is not None else None
return {
"endpoints_total": max(0, total),
"endpoints_eligible": max(0, eligible),
"next_available_at": next_available_at or None,
}
return await with_sqlite_busy_retry(_op)
async def _pick_endpoint_in_pool(engine: AsyncEngine, *, pool_id: int, now_iso: str) -> tuple[int, str, str, int, str, str] | None:
sql = """
SELECT pe.id, pe.scheme, pe.host, pe.port, pe.username, pe.password_enc, ppe.weight, pe.last_ok_at, pe.last_fail_at
FROM proxy_pools pp
JOIN proxy_pool_endpoints ppe
ON ppe.pool_id = pp.id AND ppe.enabled = 1
JOIN proxy_endpoints pe
ON pe.id = ppe.endpoint_id AND pe.enabled = 1
WHERE pp.id = :pool_id AND pp.enabled = 1
AND (pe.blacklisted_until IS NULL OR pe.blacklisted_until <= :now)
ORDER BY pe.id ASC;
""".strip()
async def _op() -> tuple[int, str, str, int, str, str] | None:
async with engine.connect() as conn:
result = await conn.exec_driver_sql(sql, {"pool_id": int(pool_id), "now": now_iso})
rows = result.fetchall()
ok_weighted: list[tuple[int, int]] = []
unknown_weighted: list[tuple[int, int]] = []
fail_weighted: list[tuple[int, int]] = []
for r in rows:
endpoint_id = int(r[0])
weight = int(r[6] or 0)
last_ok_at = str(r[7]).strip() if r[7] is not None else ""
last_fail_at = str(r[8]).strip() if r[8] is not None else ""
if last_ok_at and (not last_fail_at or last_ok_at >= last_fail_at):
ok_weighted.append((endpoint_id, weight))
elif not last_ok_at and not last_fail_at:
unknown_weighted.append((endpoint_id, weight))
else:
fail_weighted.append((endpoint_id, weight))
weighted = ok_weighted or unknown_weighted or fail_weighted
chosen_id = _weighted_choice(weighted)
if chosen_id is None:
return None
for r in rows:
if int(r[0]) == int(chosen_id):
return (
int(r[0]),
str(r[1]),
str(r[2]),
int(r[3]),
str(r[4] or ""),
str(r[5] or ""),
)
return None
return await with_sqlite_busy_retry(_op)
async def _load_token_binding(
engine: AsyncEngine,
*,
token_id: int,
pool_id: int,
) -> tuple[int, int | None, str | None] | None:
sql = """
SELECT primary_proxy_id, override_proxy_id, override_expires_at
FROM token_proxy_bindings
WHERE token_id=:token_id AND pool_id=:pool_id
LIMIT 1;
""".strip()
async def _op() -> tuple[int, int | None, str | None] | None:
async with engine.connect() as conn:
result = await conn.exec_driver_sql(sql, {"token_id": int(token_id), "pool_id": int(pool_id)})
row = result.first()
if row is None:
return None
primary_id = int(row[0])
override_id = int(row[1]) if row[1] is not None else None
override_expires_at = str(row[2]) if row[2] is not None else None
return primary_id, override_id, override_expires_at
return await with_sqlite_busy_retry(_op)
async def _load_endpoint_in_pool(
engine: AsyncEngine,
*,
pool_id: int,
endpoint_id: int,
now_iso: str,
) -> tuple[int, str, str, int, str, str] | None:
sql = """
SELECT pe.id, pe.scheme, pe.host, pe.port, pe.username, pe.password_enc
FROM proxy_pools pp
JOIN proxy_pool_endpoints ppe
ON ppe.pool_id = pp.id AND ppe.enabled = 1
JOIN proxy_endpoints pe
ON pe.id = ppe.endpoint_id AND pe.enabled = 1
WHERE pp.id = :pool_id AND pp.enabled = 1
AND pe.id = :endpoint_id
AND (pe.blacklisted_until IS NULL OR pe.blacklisted_until <= :now)
LIMIT 1;
""".strip()
async def _op() -> tuple[int, str, str, int, str, str] | None:
async with engine.connect() as conn:
result = await conn.exec_driver_sql(
sql,
{"pool_id": int(pool_id), "endpoint_id": int(endpoint_id), "now": now_iso},
)
row = result.first()
if row is None:
return None
return (
int(row[0]),
str(row[1]),
str(row[2]),
int(row[3]),
str(row[4] or ""),
str(row[5] or ""),
)
return await with_sqlite_busy_retry(_op)
def _build_proxy_uri(
encryptor: FieldEncryptor | None,
*,
scheme: str,
host: str,
port: int,
username: str,
password_enc: str,
) -> str:
scheme = (scheme or "").strip().lower()
host = (host or "").strip()
username = (username or "").strip()
password_enc = (password_enc or "").strip()
if not scheme or not host or int(port) <= 0:
raise ValueError("invalid proxy endpoint")
password = ""
if password_enc:
if encryptor is None:
raise ValueError("FIELD_ENCRYPTION_KEY is required to decrypt proxy password")
password = encryptor.decrypt_text(password_enc) if password_enc else ""
host_part = host
if ":" in host_part and not host_part.startswith("["):
host_part = f"[{host_part}]"
auth = ""
if username:
user_q = quote(username, safe="")
pass_q = quote(password or "", safe="")
auth = f"{user_q}:{pass_q}@"
return f"{scheme}://{auth}{host_part}:{int(port)}"
def _proxy_uri_from_endpoint_row(
settings: Settings,
*,
endpoint_id: int,
pool_id: int,
scheme: str,
host: str,
port: int,
username: str,
password_enc: str,
) -> ProxyUri:
encryptor = None
if str(password_enc or "").strip():
try:
encryptor = FieldEncryptor.from_key(settings.field_encryption_key)
except Exception as exc:
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Encryption not configured", status_code=500) from exc
try:
uri = _build_proxy_uri(
encryptor,
scheme=scheme,
host=host,
port=int(port),
username=username,
password_enc=password_enc,
)
except ApiError:
raise
except Exception as exc:
raise ApiError(code=ErrorCode.INTERNAL_ERROR, message="Invalid proxy endpoint", status_code=500) from exc
return ProxyUri(uri=uri, endpoint_id=int(endpoint_id), pool_id=int(pool_id))
async def select_proxy_uri_for_url(
engine: AsyncEngine,
settings: Settings,
runtime: RuntimeConfig,
*,
url: str,
now_iso: str | None = None,
token_id: int | None = None,
) -> ProxyUri | None:
host = host_from_url(url)
if host is None:
return None
if not should_use_proxy_for_host(runtime, host=host):
return None
now_iso = now_iso or iso_utc_ms()
preferred_pool_id = resolve_pool_id_for_host(runtime, host=host)
enabled_pools = await _list_enabled_pool_ids(engine)
enabled_pool_set = {int(pid) for pid in enabled_pools}
pool_candidates: list[int] = []
if preferred_pool_id is not None and int(preferred_pool_id) > 0:
preferred_pool_id_i = int(preferred_pool_id)
if preferred_pool_id_i in enabled_pool_set:
pool_candidates.append(preferred_pool_id_i)
for pid in enabled_pools:
if pid not in pool_candidates:
pool_candidates.append(int(pid))
if not pool_candidates:
if bool(runtime.proxy_fail_closed):
raise ApiError(
code=ErrorCode.PROXY_REQUIRED,
message="需要代理,但未配置代理池",
status_code=502,
details={
"reason": "no_proxy_pool_configured",
"host": host,
"url": url,
},
)
return None
async def _select_in_pool(pool_id: int) -> ProxyUri | None:
if int(pool_id) <= 0:
return None
if token_id is not None and int(token_id) > 0:
binding = await _load_token_binding(engine, token_id=int(token_id), pool_id=int(pool_id))
if binding is not None:
primary_proxy_id, override_proxy_id, override_expires_at = binding
override_active = bool(
override_proxy_id is not None
and override_expires_at
and str(override_expires_at) > str(now_iso)
)
candidates: list[int] = []
if override_active and override_proxy_id is not None:
candidates.append(int(override_proxy_id))
candidates.append(int(primary_proxy_id))
for endpoint_id in candidates:
picked_by_binding = await _load_endpoint_in_pool(
engine,
pool_id=int(pool_id),
endpoint_id=int(endpoint_id),
now_iso=str(now_iso),
)
if picked_by_binding is None:
continue
eid, scheme, host_v, port, username, password_enc = picked_by_binding
return _proxy_uri_from_endpoint_row(
settings,
endpoint_id=int(eid),
pool_id=int(pool_id),
scheme=scheme,
host=host_v,
port=int(port),
username=username,
password_enc=password_enc,
)
picked = await _pick_endpoint_in_pool(engine, pool_id=int(pool_id), now_iso=now_iso)
if picked is None:
return None
endpoint_id, scheme, host_v, port, username, password_enc = picked
return _proxy_uri_from_endpoint_row(
settings,
endpoint_id=int(endpoint_id),
pool_id=int(pool_id),
scheme=scheme,
host=host_v,
port=int(port),
username=username,
password_enc=password_enc,
)
for candidate_pool_id in pool_candidates:
picked = await _select_in_pool(int(candidate_pool_id))
if picked is not None:
return picked
if bool(runtime.proxy_fail_closed):
pool_stats: list[dict[str, Any]] = []
next_available_at: str | None = None
total_any = 0
eligible_any = 0
any_next = False
for pid in pool_candidates:
stats = await _pool_health_stats(engine, pool_id=int(pid), now_iso=str(now_iso))
total = int(stats.get("endpoints_total") or 0)
eligible = int(stats.get("endpoints_eligible") or 0)
total_any += max(0, total)
eligible_any += max(0, eligible)
nxt = stats.get("next_available_at")
if isinstance(nxt, str) and nxt.strip():
any_next = True
if next_available_at is None or str(nxt) < str(next_available_at):
next_available_at = str(nxt)
pool_stats.append({"pool_id": int(pid), **stats})
reason = "no_healthy_proxy_available"
if total_any <= 0:
reason = "pool_has_no_endpoints"
elif eligible_any <= 0 and any_next and next_available_at:
reason = "all_endpoints_blacklisted"
primary_pool_id = (
int(preferred_pool_id)
if preferred_pool_id is not None and int(preferred_pool_id) > 0
else int(pool_candidates[0])
)
raise ApiError(
code=ErrorCode.PROXY_REQUIRED,
message="需要代理,但当前没有可用代理节点",
status_code=502,
details={
"reason": reason,
"host": host,
"url": url,
"pool_id": primary_pool_id,
"attempted_pool_ids": [int(pid) for pid in pool_candidates],
"pools": pool_stats,
"endpoints_total": int(total_any),
"endpoints_eligible": int(eligible_any),
"next_available_at": next_available_at,
},
)
return None
+100
View File
@@ -0,0 +1,100 @@
from __future__ import annotations
from dataclasses import dataclass
from urllib.parse import unquote
@dataclass(frozen=True, slots=True)
class ProxyUriParts:
scheme: str
host: str
port: int
username: str | None
password: str | None
_ALLOWED_PROXY_SCHEMES = {"http", "https", "socks4", "socks5"}
def _strip_authority(rest: str) -> str:
for sep in ("/", "?", "#"):
idx = rest.find(sep)
if idx >= 0:
return rest[:idx]
return rest
def _parse_hostport(hostport: str) -> tuple[str, int]:
hostport = hostport.strip()
if not hostport:
raise ValueError("missing hostport")
if hostport.startswith("["):
rb = hostport.find("]")
if rb <= 0:
raise ValueError("invalid ipv6 hostport")
host = hostport[1:rb].strip()
rest = hostport[rb + 1 :].strip()
if not rest.startswith(":"):
raise ValueError("missing port")
port_s = rest[1:].strip()
else:
if ":" not in hostport:
raise ValueError("missing port")
host, port_s = hostport.rsplit(":", 1)
host = host.strip()
port_s = port_s.strip()
if not host:
raise ValueError("missing host")
try:
port = int(port_s)
except Exception as exc:
raise ValueError("invalid port") from exc
if port <= 0 or port > 65535:
raise ValueError("invalid port")
return host, port
def parse_proxy_uri(uri: str) -> ProxyUriParts:
uri = uri.strip()
if not uri:
raise ValueError("uri is required")
if "://" not in uri:
raise ValueError("invalid uri")
scheme_raw, rest_raw = uri.split("://", 1)
scheme = scheme_raw.strip().lower()
if scheme not in _ALLOWED_PROXY_SCHEMES:
raise ValueError("unsupported scheme")
authority = _strip_authority(rest_raw.strip())
if not authority:
raise ValueError("invalid uri")
username: str | None = None
password: str | None = None
if "@" in authority:
userinfo, hostport = authority.rsplit("@", 1)
if ":" not in userinfo:
raise ValueError("invalid userinfo")
username, password = userinfo.split(":", 1)
username = unquote(username.strip())
password = unquote(password)
if not username:
raise ValueError("invalid userinfo")
else:
hostport = authority
host, port = _parse_hostport(hostport)
return ProxyUriParts(
scheme=scheme,
host=host,
port=port,
username=username,
password=password,
)
+83
View File
@@ -0,0 +1,83 @@
from __future__ import annotations
import asyncio
import time
from collections import deque
from dataclasses import dataclass
@dataclass(frozen=True, slots=True)
class RandomRequestStatsSnapshot:
total_requests: int
total_ok: int
total_error: int
in_flight: int
window_seconds: int
last_window_requests: int
last_window_ok: int
last_window_error: int
last_window_success_rate: float
class RandomRequestStats:
def __init__(self, *, window_seconds: int = 60) -> None:
self._window_s = max(1, int(window_seconds))
self._lock = asyncio.Lock()
self._total_requests = 0
self._total_ok = 0
self._total_error = 0
self._in_flight = 0
# (monotonic_seconds, ok)
self._events: deque[tuple[float, bool]] = deque()
def _purge(self, *, now_m: float) -> None:
cutoff = float(now_m) - float(self._window_s)
while self._events:
t, _ok = self._events[0]
if float(t) >= cutoff:
break
self._events.popleft()
async def on_begin(self) -> float:
now_m = float(time.monotonic())
async with self._lock:
self._total_requests += 1
self._in_flight += 1
return now_m
async def on_end(self, *, status_code: int) -> None:
now_m = float(time.monotonic())
ok = 200 <= int(status_code) < 400
async with self._lock:
self._in_flight = max(0, int(self._in_flight) - 1)
if ok:
self._total_ok += 1
else:
self._total_error += 1
self._events.append((now_m, bool(ok)))
self._purge(now_m=now_m)
async def snapshot(self) -> RandomRequestStatsSnapshot:
now_m = float(time.monotonic())
async with self._lock:
self._purge(now_m=now_m)
last_total = int(len(self._events))
last_ok = int(sum(1 for _t, ok in self._events if ok))
last_error = int(last_total - last_ok)
rate = float(last_ok) / float(last_total) if last_total > 0 else 0.0
return RandomRequestStatsSnapshot(
total_requests=int(self._total_requests),
total_ok=int(self._total_ok),
total_error=int(self._total_error),
in_flight=int(self._in_flight),
window_seconds=int(self._window_s),
last_window_requests=int(last_total),
last_window_ok=int(last_ok),
last_window_error=int(last_error),
last_window_success_rate=float(rate),
)
+95
View File
@@ -0,0 +1,95 @@
from __future__ import annotations
import re
from collections.abc import Mapping
from typing import Any
REDACTED = "***"
_SENSITIVE_KEY_PARTS = (
"refresh",
"token",
"api_key",
"apikey",
"authorization",
"password",
"secret",
"cookie",
)
_BEARER_RE = re.compile(r"(?i)\bBearer\s+([^\s]+)")
_REFRESH_QUERY_RE = re.compile(r"(?i)(refresh_token=)([^&\s]+)")
_URI_IN_TEXT_RE = re.compile(r"(?i)(?:https?|socks[45])://[^\s\"']+")
_TRAILING_PUNCT = ".,);:]}"
def is_sensitive_key(key: str) -> bool:
key_l = key.lower()
return any(part in key_l for part in _SENSITIVE_KEY_PARTS)
def _redact_single_proxy_uri(text: str) -> str:
# Handles both normal and the common \"password contains @\" variant by treating the last '@' as separator.
m = re.match(r"^(?P<scheme>https?|socks[45])://(?P<rest>.+)$", text, flags=re.IGNORECASE)
if not m:
return text
scheme = m.group("scheme")
rest = m.group("rest")
if "@" not in rest or ":" not in rest:
return text
userinfo, hostpart = rest.rsplit("@", 1)
if ":" not in userinfo:
return text
username, _password = userinfo.split(":", 1)
return f"{scheme}://{username}:{REDACTED}@{hostpart}"
def redact_proxy_uri(text: str) -> str:
def _strip_trailing_punct(uri: str) -> tuple[str, str]:
suffix = ""
while uri and uri[-1] in _TRAILING_PUNCT:
suffix = uri[-1] + suffix
uri = uri[:-1]
return uri, suffix
def _repl(m: re.Match[str]) -> str:
full = m.group(0)
core, suffix = _strip_trailing_punct(full)
return _redact_single_proxy_uri(core) + suffix
return _URI_IN_TEXT_RE.sub(_repl, text)
def redact_text(text: str) -> str:
text = redact_proxy_uri(text)
text = _BEARER_RE.sub("Bearer " + REDACTED, text)
text = _REFRESH_QUERY_RE.sub(r"\1" + REDACTED, text)
return text
def redact_any(value: Any) -> Any:
if value is None:
return None
if isinstance(value, str):
return redact_text(value)
if isinstance(value, bytes):
try:
return redact_text(value.decode("utf-8", errors="replace")).encode("utf-8")
except Exception:
return value
if isinstance(value, Mapping):
out: dict[Any, Any] = {}
for k, v in value.items():
if isinstance(k, str) and is_sensitive_key(k):
out[k] = REDACTED
else:
out[k] = redact_any(v)
return out
if isinstance(value, (list, tuple)):
seq = [redact_any(v) for v in value]
return type(value)(seq) if isinstance(value, tuple) else seq
return value
+67
View File
@@ -0,0 +1,67 @@
from __future__ import annotations
import secrets
from typing import Any, Mapping
REQUEST_ID_HEADER = "X-Request-Id"
def new_request_id() -> str:
return "req_" + secrets.token_hex(8)
def get_request_id_from_headers(headers: Mapping[str, str] | None) -> str | None:
if not headers:
return None
value = headers.get(REQUEST_ID_HEADER) or headers.get(REQUEST_ID_HEADER.lower())
if not value:
return None
value = value.strip()
return value if value else None
def set_request_id_on_state(request: Any, request_id: str) -> None:
state = getattr(request, "state", None)
if state is None:
class _State:
pass
state = _State()
setattr(request, "state", state)
setattr(state, "request_id", request_id)
def set_request_id_header(response: Any, request_id: str) -> None:
headers = getattr(response, "headers", None)
if headers is None:
setattr(response, "headers", {})
headers = response.headers
headers[REQUEST_ID_HEADER] = request_id
def get_or_create_request_id(request: Any) -> str:
state = getattr(request, "state", None)
if state is not None:
rid = getattr(state, "request_id", None)
if rid:
return str(rid)
headers = getattr(request, "headers", None)
rid = get_request_id_from_headers(headers)
return rid or new_request_id()
def build_request_id_middleware() -> Any | None:
try:
from starlette.middleware.base import BaseHTTPMiddleware
except Exception:
return None
class RequestIdMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request, call_next): # type: ignore[no-untyped-def]
rid = get_or_create_request_id(request)
set_request_id_on_state(request, rid)
response = await call_next(request)
set_request_id_header(response, rid)
return response
return RequestIdMiddleware
+205
View File
@@ -0,0 +1,205 @@
from __future__ import annotations
import json
from dataclasses import dataclass
from typing import Any
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncEngine
from app.core.logging import get_logger
from app.core.time import iso_utc_ms
from app.db.models.runtime_settings import RuntimeSetting
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
log = get_logger(__name__)
def _as_bool(value: Any) -> bool | None:
if isinstance(value, bool):
return value
if isinstance(value, int) and value in (0, 1):
return bool(value)
if isinstance(value, str):
v = value.strip().lower()
if v in {"true", "1", "yes", "y", "on"}:
return True
if v in {"false", "0", "no", "n", "off"}:
return False
return None
def _as_str_list(value: Any) -> list[str]:
if not isinstance(value, list):
return []
out: list[str] = []
seen: set[str] = set()
for item in value:
if not isinstance(item, str):
continue
v = item.strip()
if not v or v in seen:
continue
seen.add(v)
out.append(v)
return out
@dataclass(frozen=True, slots=True)
class RuntimeConfig:
proxy_enabled: bool
proxy_fail_closed: bool
proxy_route_mode: str
proxy_allowlist_domains: list[str]
proxy_route_pools: dict[str, int]
proxy_default_pool_id: int | None
random_defaults: dict[str, Any]
hide_origin_url_in_public_json: bool
rate_limit: dict[str, Any]
@classmethod
def defaults(cls) -> "RuntimeConfig":
return cls(
proxy_enabled=False,
proxy_fail_closed=True,
proxy_route_mode="pixiv_only",
proxy_allowlist_domains=[],
proxy_route_pools={},
proxy_default_pool_id=None,
random_defaults={},
hide_origin_url_in_public_json=True,
rate_limit={},
)
async def fetch_runtime_settings(engine: AsyncEngine) -> dict[str, Any]:
Session = create_sessionmaker(engine)
async def _op() -> dict[str, Any]:
async with Session() as session:
result = await session.execute(select(RuntimeSetting.key, RuntimeSetting.value_json))
values: dict[str, Any] = {}
for key, value_json in result.all():
try:
values[str(key)] = json.loads(value_json)
except Exception:
log.warning("runtime_settings_invalid_json key=%s", key)
return values
return await with_sqlite_busy_retry(_op)
def runtime_config_from_values(values: dict[str, Any]) -> RuntimeConfig:
defaults = RuntimeConfig.defaults()
proxy_enabled = _as_bool(values.get("proxy.enabled"))
proxy_fail_closed = _as_bool(values.get("proxy.fail_closed"))
proxy_route_mode_raw = values.get("proxy.route_mode")
proxy_route_mode = defaults.proxy_route_mode
if isinstance(proxy_route_mode_raw, str):
candidate = proxy_route_mode_raw.strip().lower()
if candidate in {"pixiv_only", "all", "allowlist", "off"}:
proxy_route_mode = candidate
proxy_allowlist_domains = _as_str_list(values.get("proxy.allowlist_domains")) or defaults.proxy_allowlist_domains
proxy_route_pools_raw = values.get("proxy.route_pools")
proxy_route_pools: dict[str, int] = {}
if isinstance(proxy_route_pools_raw, dict):
for k, v in proxy_route_pools_raw.items():
key = str(k or "").strip().lower().strip(".")
if not key or len(key) > 200:
continue
try:
pool_id = int(v)
except Exception:
continue
if pool_id <= 0:
continue
proxy_route_pools[key] = pool_id
proxy_default_pool_id: int | None = None
default_pool_raw = values.get("proxy.default_pool_id")
if default_pool_raw is not None:
try:
candidate = int(default_pool_raw)
except Exception:
candidate = 0
proxy_default_pool_id = candidate if candidate > 0 else None
random_defaults_raw = values.get("random.defaults")
random_defaults = defaults.random_defaults
if isinstance(random_defaults_raw, dict):
random_defaults = dict(random_defaults_raw)
hide_origin_url_raw = values.get("security.hide_origin_url_in_public_json")
hide_origin_url = _as_bool(hide_origin_url_raw)
rate_limit: dict[str, Any] = {}
for key, value in values.items():
if key.startswith("rate_limit."):
rate_limit[key.removeprefix("rate_limit.")] = value
return RuntimeConfig(
proxy_enabled=proxy_enabled if proxy_enabled is not None else defaults.proxy_enabled,
proxy_fail_closed=proxy_fail_closed if proxy_fail_closed is not None else defaults.proxy_fail_closed,
proxy_route_mode=proxy_route_mode,
proxy_allowlist_domains=proxy_allowlist_domains,
proxy_route_pools=proxy_route_pools,
proxy_default_pool_id=proxy_default_pool_id,
random_defaults=random_defaults,
hide_origin_url_in_public_json=hide_origin_url
if hide_origin_url is not None
else defaults.hide_origin_url_in_public_json,
rate_limit=rate_limit,
)
async def load_runtime_config(engine: AsyncEngine) -> RuntimeConfig:
values = await fetch_runtime_settings(engine)
return runtime_config_from_values(values)
async def set_runtime_setting(
engine: AsyncEngine,
*,
key: str,
value: Any,
description: str | None = None,
updated_by: str | None = None,
) -> None:
key = (key or "").strip()
if not key:
raise ValueError("key is required")
value_json = json.dumps(value, separators=(",", ":"), sort_keys=True, ensure_ascii=False)
now = iso_utc_ms()
Session = create_sessionmaker(engine)
async def _op() -> None:
async with Session() as session:
stmt = sqlite_insert(RuntimeSetting).values(
key=key,
value_json=value_json,
description=description,
updated_at=now,
updated_by=updated_by,
)
set_values: dict[str, Any] = {
"value_json": value_json,
"updated_at": now,
"updated_by": updated_by,
}
if description is not None:
set_values["description"] = description
stmt = stmt.on_conflict_do_update(
index_elements=[RuntimeSetting.key],
set_=set_values,
)
await session.execute(stmt)
await session.commit()
await with_sqlite_busy_retry(_op)
+147
View File
@@ -0,0 +1,147 @@
from __future__ import annotations
import base64
import hashlib
import hmac
import json
import time
from collections.abc import Mapping
from typing import Any
from app.core.errors import ApiError, ErrorCode
JWT_ALG_HS256 = "HS256"
JWT_TYP = "JWT"
def _b64url_encode(raw: bytes) -> str:
return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii")
def _b64url_decode(data: str) -> bytes:
data = data.strip()
padding = "=" * (-len(data) % 4)
return base64.urlsafe_b64decode(data + padding)
def create_jwt(
*,
secret_key: str,
subject: str,
ttl_s: int = 3600,
extra_claims: Mapping[str, Any] | None = None,
now_s: int | None = None,
) -> str:
secret_key = secret_key.strip()
if not secret_key:
raise ValueError("SECRET_KEY is required")
now_i = int(now_s if now_s is not None else time.time())
header = {"alg": JWT_ALG_HS256, "typ": JWT_TYP}
payload: dict[str, Any] = {"sub": subject, "iat": now_i, "exp": now_i + ttl_s}
if extra_claims:
payload.update(dict(extra_claims))
header_b64 = _b64url_encode(json.dumps(header, separators=(",", ":"), sort_keys=True).encode("utf-8"))
payload_b64 = _b64url_encode(json.dumps(payload, separators=(",", ":"), sort_keys=True).encode("utf-8"))
signing_input = f"{header_b64}.{payload_b64}".encode("ascii")
signature = hmac.new(secret_key.encode("utf-8"), signing_input, hashlib.sha256).digest()
sig_b64 = _b64url_encode(signature)
return f"{header_b64}.{payload_b64}.{sig_b64}"
def decode_jwt(token: str, *, secret_key: str, leeway_s: int = 0, now_s: int | None = None) -> dict[str, Any]:
secret_key = secret_key.strip()
if not secret_key:
raise ValueError("SECRET_KEY is required")
token = token.strip()
parts = token.split(".")
if len(parts) != 3:
raise ValueError("Invalid token format")
header_raw = _b64url_decode(parts[0])
payload_raw = _b64url_decode(parts[1])
signature_raw = _b64url_decode(parts[2])
try:
header = json.loads(header_raw)
payload = json.loads(payload_raw)
except Exception as exc:
raise ValueError("Invalid token JSON") from exc
if not isinstance(header, dict) or not isinstance(payload, dict):
raise ValueError("Invalid token JSON")
if header.get("alg") != JWT_ALG_HS256 or header.get("typ") != JWT_TYP:
raise ValueError("Unsupported token")
signing_input = f"{parts[0]}.{parts[1]}".encode("ascii")
expected_sig = hmac.new(secret_key.encode("utf-8"), signing_input, hashlib.sha256).digest()
if not hmac.compare_digest(signature_raw, expected_sig):
raise ValueError("Invalid token signature")
now_i = int(now_s if now_s is not None else time.time())
exp = payload.get("exp")
if exp is not None:
try:
exp_i = int(exp)
except Exception as exc:
raise ValueError("Invalid exp") from exc
if now_i > exp_i + leeway_s:
raise ValueError("Token expired")
iat = payload.get("iat")
if iat is not None:
try:
iat_i = int(iat)
except Exception as exc:
raise ValueError("Invalid iat") from exc
if iat_i > now_i + leeway_s:
raise ValueError("Token issued in the future")
return payload
def _authorization_from_headers(headers: Mapping[str, str] | None) -> str | None:
if not headers:
return None
return headers.get("Authorization") or headers.get("authorization")
def parse_bearer_token(authorization: str | None) -> str | None:
if not authorization:
return None
parts = authorization.strip().split(None, 1)
if len(parts) != 2:
return None
scheme, token = parts
if scheme.lower() != "bearer":
return None
token = token.strip()
return token if token else None
def require_admin(
headers: Mapping[str, str] | None,
*,
secret_key: str,
admin_username: str,
) -> dict[str, Any]:
token = parse_bearer_token(_authorization_from_headers(headers))
if not token:
raise ApiError(code=ErrorCode.UNAUTHORIZED, message="Missing admin token", status_code=401)
try:
claims = decode_jwt(token, secret_key=secret_key)
except Exception as exc:
raise ApiError(code=ErrorCode.UNAUTHORIZED, message="Invalid admin token", status_code=401) from exc
subject = str(claims.get("sub") or "")
if subject != admin_username:
raise ApiError(code=ErrorCode.FORBIDDEN, message="Forbidden", status_code=403)
return claims
+11
View File
@@ -0,0 +1,11 @@
from __future__ import annotations
from datetime import datetime, timezone
def iso_utc_ms(dt: datetime | None = None) -> str:
dt = dt or datetime.now(timezone.utc)
dt = dt.astimezone(timezone.utc)
ms = dt.microsecond // 1000
return dt.strftime("%Y-%m-%dT%H:%M:%S.") + f"{ms:03d}Z"
+99
View File
@@ -0,0 +1,99 @@
from __future__ import annotations
from dataclasses import dataclass
import sqlalchemy as sa
from sqlalchemy import func, select
from sqlalchemy.exc import DBAPIError
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models.images import Image
from app.db.sqlite_utils import sqlite_fts_phrase_query, sqlite_table_exists
@dataclass(frozen=True, slots=True)
class AuthorListItem:
user_id: int
user_name: str | None
count_images: int
async def list_authors(
session: AsyncSession,
*,
limit: int,
cursor: int | None = None,
q: str | None = None,
) -> tuple[list[AuthorListItem], int | None]:
limit_i = int(limit)
if limit_i < 1:
raise ValueError("limit must be >= 1")
cursor_i: int | None = None
if cursor is not None:
cursor_i = int(cursor)
if cursor_i <= 0:
raise ValueError("cursor must be > 0")
q_norm = (q or "").strip()
use_fts = False
fts_q = ""
if q_norm and len(q_norm) >= 3:
use_fts = await sqlite_table_exists(session, name="authors_fts")
if use_fts:
fts_q = sqlite_fts_phrase_query(q_norm)
def _build_stmt(*, use_fts_filter: bool) -> sa.Select:
clauses: list[object] = [Image.status == 1, Image.user_id.is_not(None)]
if cursor_i is not None:
clauses.append(Image.user_id > cursor_i)
if q_norm:
if use_fts_filter:
fts_user_ids = (
sa.text("SELECT rowid AS user_id FROM authors_fts WHERE authors_fts MATCH :q")
.bindparams(sa.bindparam("q", fts_q))
.columns(user_id=sa.Integer)
)
fts_user_ids_sq = fts_user_ids.subquery()
clauses.append(Image.user_id.in_(sa.select(fts_user_ids_sq.c.user_id)))
else:
clauses.append(Image.user_name.like(f"%{q_norm}%"))
return (
select(
Image.user_id,
func.max(Image.user_name).label("user_name"),
func.count(Image.id).label("count_images"),
)
.where(*clauses)
.group_by(Image.user_id)
.order_by(Image.user_id.asc())
.limit(limit_i + 1)
)
stmt = _build_stmt(use_fts_filter=use_fts)
try:
rows = (await session.execute(stmt)).all()
except DBAPIError:
if not use_fts:
raise
stmt = _build_stmt(use_fts_filter=False)
rows = (await session.execute(stmt)).all()
next_cursor: int | None = None
if len(rows) > limit_i:
rows = rows[:limit_i]
next_cursor = int(rows[-1][0] or 0)
items = [
AuthorListItem(
user_id=int(row[0]),
user_name=str(row[1]) if row[1] is not None else None,
count_images=int(row[2] or 0),
)
for row in rows
]
return items, next_cursor
+72
View File
@@ -0,0 +1,72 @@
from __future__ import annotations
import os
from typing import Any
from sqlalchemy import event
from sqlalchemy.engine.url import make_url
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
SQLITE_BUSY_TIMEOUT_MS = 30_000
SQLITE_POOL_SIZE = 5
SQLITE_MAX_OVERFLOW = 0
def apply_sqlite_pragmas(dbapi_connection: Any) -> None:
try:
busy_timeout_ms = int((os.environ.get("SQLITE_BUSY_TIMEOUT_MS") or str(SQLITE_BUSY_TIMEOUT_MS)).strip() or SQLITE_BUSY_TIMEOUT_MS)
except Exception:
busy_timeout_ms = int(SQLITE_BUSY_TIMEOUT_MS)
busy_timeout_ms = max(1000, min(int(busy_timeout_ms), 5 * 60_000))
cursor = dbapi_connection.cursor()
try:
cursor.execute("PRAGMA foreign_keys = ON")
cursor.execute("PRAGMA journal_mode = WAL")
cursor.fetchone()
cursor.execute("PRAGMA synchronous = NORMAL")
cursor.execute("PRAGMA temp_store = MEMORY")
cursor.execute(f"PRAGMA busy_timeout = {busy_timeout_ms}")
finally:
cursor.close()
def _is_sqlite_file_url(database_url: str) -> bool:
try:
url = make_url(database_url)
except Exception:
return False
if (url.get_backend_name() or "").lower() != "sqlite":
return False
db = str(url.database or "").strip()
return bool(db and db != ":memory:")
def create_engine(database_url: str) -> AsyncEngine:
kwargs: dict[str, Any] = {}
if database_url.lower().startswith("sqlite"):
try:
busy_timeout_ms = int((os.environ.get("SQLITE_BUSY_TIMEOUT_MS") or str(SQLITE_BUSY_TIMEOUT_MS)).strip() or SQLITE_BUSY_TIMEOUT_MS)
except Exception:
busy_timeout_ms = int(SQLITE_BUSY_TIMEOUT_MS)
busy_timeout_ms = max(1000, min(int(busy_timeout_ms), 5 * 60_000))
kwargs["connect_args"] = {"timeout": float(busy_timeout_ms) / 1000.0}
if _is_sqlite_file_url(database_url):
# 限制单进程内同时打开的 SQLite 连接数,减少并发写导致的 "database is locked"。
kwargs.update(
{
"pool_size": int(SQLITE_POOL_SIZE),
"max_overflow": int(SQLITE_MAX_OVERFLOW),
}
)
engine = create_async_engine(database_url, **kwargs)
if database_url.lower().startswith("sqlite"):
def _on_connect(dbapi_connection: Any, _record: Any) -> None:
apply_sqlite_pragmas(dbapi_connection)
event.listen(engine.sync_engine, "connect", _on_connect)
return engine
+12
View File
@@ -0,0 +1,12 @@
from __future__ import annotations
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models.images import Image
async def get_image_by_id(session: AsyncSession, *, image_id: int) -> Image | None:
stmt = select(Image).where(Image.id == int(image_id), Image.status == 1).limit(1)
return (await session.execute(stmt)).scalars().first()
+25
View File
@@ -0,0 +1,25 @@
from __future__ import annotations
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models.images import Image
async def get_image_by_illust_page(
session: AsyncSession,
*,
illust_id: int,
page_index: int,
) -> Image | None:
stmt = (
select(Image)
.where(
Image.status == 1,
Image.illust_id == int(illust_id),
Image.page_index == int(page_index),
)
.limit(1)
)
return (await session.execute(stmt)).scalars().first()
+91
View File
@@ -0,0 +1,91 @@
from __future__ import annotations
from collections.abc import Sequence
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models.images import Image
from app.db.random_pick import (
_excluded_tags_where_clause,
_included_tags_where_clause,
_orientation_where_clause,
_r18_where_clause,
)
async def list_images(
session: AsyncSession,
*,
limit: int,
cursor: int | None = None,
r18: int = 0,
r18_strict: bool = True,
orientation: int | None = None,
ai_type: int | None = None,
min_width: int = 0,
min_height: int = 0,
min_pixels: int = 0,
included_tags: Sequence[str] | None = None,
excluded_tags: Sequence[str] | None = None,
user_id: int | None = None,
illust_id: int | None = None,
created_from: str | None = None,
created_to: str | None = None,
) -> tuple[list[Image], int | None]:
limit_i = int(limit)
if limit_i < 1:
raise ValueError("limit must be >= 1")
r18_clause = _r18_where_clause(r18=r18, r18_strict=r18_strict)
orientation_clause = _orientation_where_clause(orientation=orientation)
included_tags_clause = _included_tags_where_clause(tag_names=included_tags or [])
excluded_tags_clause = _excluded_tags_where_clause(tag_names=excluded_tags or [])
min_width_i = int(min_width)
min_height_i = int(min_height)
min_pixels_i = int(min_pixels)
if min_width_i < 0 or min_height_i < 0 or min_pixels_i < 0:
raise ValueError("min_* must be >= 0")
clauses: list[object] = [Image.status == 1]
if cursor is not None:
clauses.append(Image.id < int(cursor))
if r18_clause is not None:
clauses.append(r18_clause)
if orientation_clause is not None:
clauses.append(orientation_clause)
if ai_type is not None:
ai_type_i = int(ai_type)
if ai_type_i not in {0, 1}:
raise ValueError("ai_type must be 0, 1, or None")
clauses.append(Image.ai_type == ai_type_i)
if included_tags_clause is not None:
clauses.append(included_tags_clause)
if excluded_tags_clause is not None:
clauses.append(excluded_tags_clause)
if min_width_i > 0:
clauses.append(Image.width >= min_width_i)
if min_height_i > 0:
clauses.append(Image.height >= min_height_i)
if min_pixels_i > 0:
clauses.append((Image.width * Image.height) >= min_pixels_i)
if user_id is not None:
clauses.append(Image.user_id == int(user_id))
if illust_id is not None:
clauses.append(Image.illust_id == int(illust_id))
if created_from is not None:
clauses.append(Image.created_at_pixiv >= str(created_from))
if created_to is not None:
clauses.append(Image.created_at_pixiv <= str(created_to))
stmt = select(Image).where(*clauses).order_by(Image.id.desc()).limit(limit_i + 1)
images = (await session.execute(stmt)).scalars().all()
next_cursor: int | None = None
if len(images) > limit_i:
images = images[:limit_i]
next_cursor = int(images[-1].id)
return images, next_cursor
+62
View File
@@ -0,0 +1,62 @@
from __future__ import annotations
from sqlalchemy import update
from sqlalchemy.ext.asyncio import AsyncEngine
from app.core.redact import redact_text
from app.db.models.images import Image
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
def _truncate(text: str, *, max_len: int = 500) -> str:
if len(text) <= max_len:
return text
return text[: max_len - 3] + "..."
async def mark_image_failure(
engine: AsyncEngine,
*,
image_id: int,
now: str,
error_code: str,
error_message: str,
) -> None:
Session = create_sessionmaker(engine)
msg = _truncate(redact_text(error_message or ""))
async def _op() -> None:
async with Session() as session:
await session.execute(
update(Image)
.where(Image.id == int(image_id))
.values(
fail_count=Image.fail_count + 1,
last_fail_at=str(now),
last_error_code=str(error_code),
last_error_msg=str(msg),
)
)
await session.commit()
await with_sqlite_busy_retry(_op)
async def mark_image_ok(engine: AsyncEngine, *, image_id: int, now: str) -> None:
Session = create_sessionmaker(engine)
async def _op() -> None:
async with Session() as session:
await session.execute(
update(Image)
.where(Image.id == int(image_id))
.values(
last_ok_at=str(now),
last_error_code=None,
last_error_msg=None,
)
)
await session.commit()
await with_sqlite_busy_retry(_op)
+44
View File
@@ -0,0 +1,44 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models.images import Image
async def upsert_image_by_illust_page(
session: AsyncSession,
*,
illust_id: int,
page_index: int,
ext: str,
original_url: str,
proxy_path: str,
random_key: float,
created_import_id: int | None,
) -> int:
stmt = sqlite_insert(Image).values(
illust_id=illust_id,
page_index=page_index,
ext=ext,
original_url=original_url,
proxy_path=proxy_path,
random_key=random_key,
created_import_id=created_import_id,
)
now_expr = sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))")
stmt = stmt.on_conflict_do_update(
index_elements=["illust_id", "page_index"],
set_={
"ext": stmt.excluded.ext,
"original_url": stmt.excluded.original_url,
"proxy_path": stmt.excluded.proxy_path,
"created_import_id": stmt.excluded.created_import_id,
"updated_at": now_expr,
},
).returning(Image.id)
result = await session.execute(stmt)
return int(result.scalar_one())
+32
View File
@@ -0,0 +1,32 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class AdminAudit(Base):
__tablename__ = "admin_audit"
__table_args__ = (
sa.Index("idx_admin_audit_created_at", "created_at"),
sa.Index("idx_admin_audit_action", "action"),
sa.Index("idx_admin_audit_resource", "resource"),
sa.Index("idx_admin_audit_record_id", "record_id"),
)
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
created_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
actor: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
action: Mapped[str] = mapped_column(sa.Text(), nullable=False)
resource: Mapped[str] = mapped_column(sa.Text(), nullable=False)
record_id: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
request_id: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
ip: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
user_agent: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
detail_json: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
+38
View File
@@ -0,0 +1,38 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class ApiKey(Base):
__tablename__ = "api_keys"
__table_args__ = (
sa.UniqueConstraint("name", name="uq_api_keys_name"),
sa.CheckConstraint("enabled IN (0,1)", name="ck_api_keys_enabled"),
sa.Index("idx_api_keys_enabled", "enabled"),
sa.Index("idx_api_keys_created_at", "created_at"),
)
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
created_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
updated_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
name: Mapped[str] = mapped_column(sa.Text(), nullable=False)
description: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
key_hash: Mapped[str] = mapped_column(sa.Text(), nullable=False)
hint: Mapped[str] = mapped_column(sa.Text(), nullable=False, server_default=sa.text("''"))
enabled: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
last_used_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
+26
View File
@@ -0,0 +1,26 @@
from __future__ import annotations
from sqlalchemy.orm import DeclarativeBase
class Base(DeclarativeBase):
pass
# Import all model modules so Base.metadata is fully populated for create_all().
# This keeps tests and one-off scripts consistent, regardless of import order.
from app.db.models import admin_audit as _admin_audit # noqa: F401,E402
from app.db.models import api_keys as _api_keys # noqa: F401,E402
from app.db.models import hydration_runs as _hydration_runs # noqa: F401,E402
from app.db.models import image_tags as _image_tags # noqa: F401,E402
from app.db.models import images as _images # noqa: F401,E402
from app.db.models import imports as _imports # noqa: F401,E402
from app.db.models import jobs as _jobs # noqa: F401,E402
from app.db.models import pixiv_tokens as _pixiv_tokens # noqa: F401,E402
from app.db.models import proxy_endpoints as _proxy_endpoints # noqa: F401,E402
from app.db.models import proxy_pool_endpoints as _proxy_pool_endpoints # noqa: F401,E402
from app.db.models import proxy_pools as _proxy_pools # noqa: F401,E402
from app.db.models import request_logs as _request_logs # noqa: F401,E402
from app.db.models import runtime_settings as _runtime_settings # noqa: F401,E402
from app.db.models import tags as _tags # noqa: F401,E402
from app.db.models import token_proxy_bindings as _token_proxy_bindings # noqa: F401,E402
+42
View File
@@ -0,0 +1,42 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class HydrationRun(Base):
__tablename__ = "hydration_runs"
__table_args__ = (
sa.CheckConstraint(
"status IN ('pending','running','paused','canceled','completed','failed')",
name="ck_hr_status",
),
sa.Index("idx_hr_status_updated", "status", "updated_at"),
)
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
created_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
updated_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
type: Mapped[str] = mapped_column(sa.Text(), nullable=False)
status: Mapped[str] = mapped_column(sa.Text(), nullable=False)
criteria_json: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
cursor_json: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
total: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
processed: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
success: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
failed: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
started_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
finished_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
last_error: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
+23
View File
@@ -0,0 +1,23 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class ImageTag(Base):
__tablename__ = "image_tags"
__table_args__ = (sa.Index("idx_image_tags_tag_image", "tag_id", "image_id"),)
image_id: Mapped[int] = mapped_column(
sa.Integer(),
sa.ForeignKey("images.id", ondelete="CASCADE"),
primary_key=True,
)
tag_id: Mapped[int] = mapped_column(
sa.Integer(),
sa.ForeignKey("tags.id", ondelete="CASCADE"),
primary_key=True,
)
+78
View File
@@ -0,0 +1,78 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class Image(Base):
__tablename__ = "images"
__table_args__ = (
sa.UniqueConstraint("illust_id", "page_index", name="uq_images_illust_page"),
sa.CheckConstraint("status IN (1,2,3,4)", name="ck_images_status"),
sa.CheckConstraint("random_key >= 0.0 AND random_key < 1.0", name="ck_images_random_key"),
sa.Index(
"idx_images_filter",
"status",
"x_restrict",
"orientation",
"width",
"height",
"random_key",
),
sa.Index("idx_images_illust_type_random", "status", "illust_type", "random_key"),
sa.Index("idx_images_user_random", "status", "user_id", "random_key"),
sa.Index("idx_images_created_at_pixiv", "created_at_pixiv"),
sa.Index("idx_images_created_import_id", "created_import_id"),
)
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
illust_id: Mapped[int] = mapped_column(sa.Integer(), nullable=False)
page_index: Mapped[int] = mapped_column(sa.Integer(), nullable=False)
ext: Mapped[str] = mapped_column(sa.Text(), nullable=False)
original_url: Mapped[str] = mapped_column(sa.Text(), nullable=False)
proxy_path: Mapped[str] = mapped_column(sa.Text(), nullable=False)
random_key: Mapped[float] = mapped_column(sa.Float(), nullable=False)
width: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
height: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
aspect_ratio: Mapped[float | None] = mapped_column(sa.Float(), nullable=True)
orientation: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
x_restrict: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
ai_type: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
illust_type: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
user_id: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
user_name: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
title: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
created_at_pixiv: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
bookmark_count: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
view_count: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
comment_count: Mapped[int | None] = mapped_column(sa.Integer(), nullable=True)
status: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
fail_count: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
last_fail_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
last_ok_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
last_error_code: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
last_error_msg: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
created_import_id: Mapped[int | None] = mapped_column(
sa.Integer(),
sa.ForeignKey("imports.id", ondelete="SET NULL"),
nullable=True,
)
added_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
updated_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
+27
View File
@@ -0,0 +1,27 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class Import(Base):
__tablename__ = "imports"
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
created_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
created_by: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
source: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
total: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
accepted: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
success: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
failed: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
detail_json: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
+49
View File
@@ -0,0 +1,49 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class JobRow(Base):
__tablename__ = "jobs"
__table_args__ = (
sa.CheckConstraint(
"status IN ('pending','running','paused','canceled','completed','failed','dlq')",
name="ck_jobs_status",
),
sa.Index("idx_jobs_status_priority", "status", "priority", "id"),
sa.Index("idx_jobs_run_after", "run_after"),
sa.Index("idx_jobs_ref", "ref_type", "ref_id"),
)
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
created_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
updated_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
type: Mapped[str] = mapped_column(sa.Text(), nullable=False)
status: Mapped[str] = mapped_column(sa.Text(), nullable=False)
priority: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
run_after: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
attempt: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
max_attempts: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("3"))
payload_json: Mapped[str] = mapped_column(sa.Text(), nullable=False)
last_error: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
locked_by: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
locked_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
ref_type: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
ref_id: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
+41
View File
@@ -0,0 +1,41 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class PixivToken(Base):
__tablename__ = "pixiv_tokens"
__table_args__ = (
sa.CheckConstraint("enabled IN (0,1)", name="ck_pixiv_tokens_enabled"),
sa.Index("idx_pixiv_tokens_enabled", "enabled"),
)
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
created_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
updated_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
label: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
enabled: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
refresh_token_enc: Mapped[str] = mapped_column(sa.Text(), nullable=False)
refresh_token_masked: Mapped[str] = mapped_column(sa.Text(), nullable=False)
weight: Mapped[float] = mapped_column(sa.Float(), nullable=False, server_default=sa.text("1.0"))
error_count: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
backoff_until: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
last_ok_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
last_fail_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
last_error_code: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
last_error_msg: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
+47
View File
@@ -0,0 +1,47 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class ProxyEndpoint(Base):
__tablename__ = "proxy_endpoints"
__table_args__ = (
sa.UniqueConstraint("scheme", "host", "port", "username", name="uq_proxy_identity"),
sa.CheckConstraint("enabled IN (0,1)", name="ck_proxy_enabled"),
sa.Index("idx_proxy_endpoints_enabled", "enabled"),
sa.Index("idx_proxy_endpoints_source", "source"),
)
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
created_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
updated_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
scheme: Mapped[str] = mapped_column(sa.Text(), nullable=False)
host: Mapped[str] = mapped_column(sa.Text(), nullable=False)
port: Mapped[int] = mapped_column(sa.Integer(), nullable=False)
username: Mapped[str] = mapped_column(sa.Text(), nullable=False, server_default=sa.text("''"))
password_enc: Mapped[str] = mapped_column(sa.Text(), nullable=False, server_default=sa.text("''"))
enabled: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
source: Mapped[str] = mapped_column(sa.Text(), nullable=False, server_default=sa.text("'manual'"))
source_ref: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
last_latency_ms: Mapped[float | None] = mapped_column(sa.Float(), nullable=True)
last_ok_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
last_fail_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
success_count: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
failure_count: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("0"))
blacklisted_until: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
last_error: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
@@ -0,0 +1,41 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class ProxyPoolEndpoint(Base):
__tablename__ = "proxy_pool_endpoints"
__table_args__ = (
sa.Index("idx_ppe_pool_enabled", "pool_id", "enabled"),
sa.Index("idx_ppe_endpoint_pool", "endpoint_id", "pool_id"),
sa.CheckConstraint("enabled IN (0,1)", name="ck_ppe_enabled"),
)
pool_id: Mapped[int] = mapped_column(
sa.Integer(),
sa.ForeignKey("proxy_pools.id", ondelete="CASCADE"),
primary_key=True,
)
endpoint_id: Mapped[int] = mapped_column(
sa.Integer(),
sa.ForeignKey("proxy_endpoints.id", ondelete="CASCADE"),
primary_key=True,
)
enabled: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
weight: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
created_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
updated_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
+32
View File
@@ -0,0 +1,32 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class ProxyPool(Base):
__tablename__ = "proxy_pools"
__table_args__ = (
sa.UniqueConstraint("name", name="uq_proxy_pools_name"),
sa.CheckConstraint("enabled IN (0,1)", name="ck_proxy_pools_enabled"),
sa.Index("idx_proxy_pools_enabled", "enabled"),
)
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
created_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
updated_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
name: Mapped[str] = mapped_column(sa.Text(), nullable=False)
description: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
enabled: Mapped[int] = mapped_column(sa.Integer(), nullable=False, server_default=sa.text("1"))
+31
View File
@@ -0,0 +1,31 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class RequestLog(Base):
__tablename__ = "request_logs"
__table_args__ = (
sa.Index("idx_request_logs_created_at", "created_at"),
sa.Index("idx_request_logs_route", "route"),
sa.Index("idx_request_logs_status", "status"),
)
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
created_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
request_id: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
method: Mapped[str] = mapped_column(sa.Text(), nullable=False)
route: Mapped[str] = mapped_column(sa.Text(), nullable=False)
status: Mapped[int] = mapped_column(sa.Integer(), nullable=False)
duration_ms: Mapped[int] = mapped_column(sa.Integer(), nullable=False)
ip: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
user_agent: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
sample_rate: Mapped[float | None] = mapped_column(sa.Float(), nullable=True)
+21
View File
@@ -0,0 +1,21 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class RuntimeSetting(Base):
__tablename__ = "runtime_settings"
key: Mapped[str] = mapped_column(sa.Text(), primary_key=True)
value_json: Mapped[str] = mapped_column(sa.Text(), nullable=False)
description: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
updated_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
updated_by: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
+26
View File
@@ -0,0 +1,26 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class Tag(Base):
__tablename__ = "tags"
__table_args__ = (sa.UniqueConstraint("name", name="uq_tags_name"),)
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
name: Mapped[str] = mapped_column(sa.Text(), nullable=False)
translated_name: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
created_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
updated_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
@@ -0,0 +1,51 @@
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.orm import Mapped, mapped_column
from app.db.models.base import Base
class TokenProxyBinding(Base):
__tablename__ = "token_proxy_bindings"
__table_args__ = (
sa.UniqueConstraint("token_id", "pool_id", name="uq_token_pool"),
sa.Index("idx_tpb_pool", "pool_id"),
sa.Index("idx_tpb_primary", "primary_proxy_id"),
sa.Index("idx_tpb_override", "override_proxy_id"),
)
id: Mapped[int] = mapped_column(sa.Integer(), primary_key=True, autoincrement=True)
created_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
updated_at: Mapped[str] = mapped_column(
sa.Text(),
nullable=False,
server_default=sa.text("(strftime('%Y-%m-%dT%H:%M:%fZ','now'))"),
)
token_id: Mapped[int] = mapped_column(
sa.Integer(),
sa.ForeignKey("pixiv_tokens.id", ondelete="CASCADE"),
nullable=False,
)
pool_id: Mapped[int] = mapped_column(
sa.Integer(),
sa.ForeignKey("proxy_pools.id", ondelete="CASCADE"),
nullable=False,
)
primary_proxy_id: Mapped[int] = mapped_column(
sa.Integer(),
sa.ForeignKey("proxy_endpoints.id", ondelete="RESTRICT"),
nullable=False,
)
override_proxy_id: Mapped[int | None] = mapped_column(
sa.Integer(),
sa.ForeignKey("proxy_endpoints.id", ondelete="SET NULL"),
nullable=True,
)
override_expires_at: Mapped[str | None] = mapped_column(sa.Text(), nullable=True)
+366
View File
@@ -0,0 +1,366 @@
from __future__ import annotations
from collections.abc import Sequence
from sqlalchemy import distinct, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models.image_tags import ImageTag
from app.db.models.images import Image
from app.db.models.tags import Tag
def _r18_where_clause(*, r18: int, r18_strict: bool) -> object | None:
r18_i = int(r18)
if r18_i not in {0, 1, 2}:
raise ValueError("r18 must be 0, 1, or 2")
if r18_i == 2:
return None
if r18_i == 1:
return Image.x_restrict == 1
if bool(r18_strict):
return Image.x_restrict == 0
return (Image.x_restrict == 0) | (Image.x_restrict.is_(None))
def _orientation_where_clause(*, orientation: int | None) -> object | None:
if orientation is None:
return None
orientation_i = int(orientation)
if orientation_i not in {1, 2, 3}:
raise ValueError("orientation must be 1, 2, or 3")
return Image.orientation == orientation_i
def _clean_tag_names(values: Sequence[str] | None) -> list[str]:
if not values:
return []
out: list[str] = []
seen: set[str] = set()
for v in values:
name = str(v or "").strip()
if not name:
continue
if name in seen:
continue
seen.add(name)
out.append(name)
return out
def _included_tags_where_clause(*, tag_names: Sequence[str]) -> object | None:
names = _clean_tag_names(tag_names)
if not names:
return None
subq = (
select(ImageTag.image_id)
.join(Tag, Tag.id == ImageTag.tag_id)
.where(Tag.name.in_(names))
.group_by(ImageTag.image_id)
.having(func.count(distinct(Tag.name)) == len(names))
)
return Image.id.in_(subq)
def _excluded_tags_where_clause(*, tag_names: Sequence[str]) -> object | None:
names = _clean_tag_names(tag_names)
if not names:
return None
subq = (
select(ImageTag.image_id)
.join(Tag, Tag.id == ImageTag.tag_id)
.where(Tag.name.in_(names))
)
return Image.id.not_in(subq)
def _exclude_image_ids_where_clause(*, image_ids: Sequence[int] | None) -> object | None:
if not image_ids:
return None
ids: list[int] = []
seen: set[int] = set()
for raw in image_ids:
try:
value = int(raw)
except Exception:
continue
if value <= 0 or value in seen:
continue
seen.add(value)
ids.append(value)
if not ids:
return None
return Image.id.not_in(ids)
async def pick_random_image(
session: AsyncSession,
*,
r: float,
r18: int = 0,
r18_strict: bool = True,
orientation: int | None = None,
ai_type: int | None = None,
illust_type: int | None = None,
min_width: int = 0,
min_height: int = 0,
min_pixels: int = 0,
min_bookmarks: int = 0,
min_views: int = 0,
min_comments: int = 0,
included_tags: Sequence[str] | None = None,
excluded_tags: Sequence[str] | None = None,
user_id: int | None = None,
illust_id: int | None = None,
created_from: str | None = None,
created_to: str | None = None,
exclude_image_ids: Sequence[int] | None = None,
fail_cooldown_before: str | None = None,
) -> Image | None:
r = float(r)
if r < 0.0:
r = 0.0
if r >= 1.0:
r = 0.999999999
r18_clause = _r18_where_clause(r18=r18, r18_strict=r18_strict)
orientation_clause = _orientation_where_clause(orientation=orientation)
included_tags_clause = _included_tags_where_clause(tag_names=included_tags or [])
excluded_tags_clause = _excluded_tags_where_clause(tag_names=excluded_tags or [])
exclude_ids_clause = _exclude_image_ids_where_clause(image_ids=exclude_image_ids)
min_width_i = int(min_width)
min_height_i = int(min_height)
min_pixels_i = int(min_pixels)
min_bookmarks_i = int(min_bookmarks)
min_views_i = int(min_views)
min_comments_i = int(min_comments)
if min_width_i < 0 or min_height_i < 0 or min_pixels_i < 0:
raise ValueError("min_* must be >= 0")
if min_bookmarks_i < 0 or min_views_i < 0 or min_comments_i < 0:
raise ValueError("min_* must be >= 0")
clauses = [Image.status == 1]
if r18_clause is not None:
clauses.append(r18_clause)
if orientation_clause is not None:
clauses.append(orientation_clause)
if ai_type is not None:
ai_type_i = int(ai_type)
if ai_type_i not in {0, 1}:
raise ValueError("ai_type must be 0, 1, or None")
clauses.append(Image.ai_type == ai_type_i)
if illust_type is not None:
illust_type_i = int(illust_type)
if illust_type_i not in {0, 1, 2}:
raise ValueError("illust_type must be 0, 1, 2, or None")
clauses.append(Image.illust_type == illust_type_i)
if included_tags_clause is not None:
clauses.append(included_tags_clause)
if excluded_tags_clause is not None:
clauses.append(excluded_tags_clause)
if exclude_ids_clause is not None:
clauses.append(exclude_ids_clause)
if min_width_i > 0:
clauses.append(Image.width >= min_width_i)
if min_height_i > 0:
clauses.append(Image.height >= min_height_i)
if min_pixels_i > 0:
clauses.append((Image.width * Image.height) >= min_pixels_i)
if min_bookmarks_i > 0:
clauses.append(Image.bookmark_count >= min_bookmarks_i)
if min_views_i > 0:
clauses.append(Image.view_count >= min_views_i)
if min_comments_i > 0:
clauses.append(Image.comment_count >= min_comments_i)
if user_id is not None:
clauses.append(Image.user_id == int(user_id))
if illust_id is not None:
clauses.append(Image.illust_id == int(illust_id))
if created_from is not None:
clauses.append(Image.created_at_pixiv >= str(created_from))
if created_to is not None:
clauses.append(Image.created_at_pixiv <= str(created_to))
if fail_cooldown_before is not None:
clauses.append((Image.last_fail_at.is_(None)) | (Image.last_fail_at <= str(fail_cooldown_before)))
stmt = (
select(Image)
.where(*clauses, Image.random_key >= r)
.order_by(Image.random_key.asc())
.limit(1)
)
image = (await session.execute(stmt)).scalars().first()
if image is not None:
return image
stmt2 = (
select(Image)
.where(*clauses)
.order_by(Image.random_key.asc())
.limit(1)
)
return (await session.execute(stmt2)).scalars().first()
def _allowed_int_or_null_clause(column: object, *, allowed: set[int | None]) -> object | None:
allowed_norm: set[int | None] = set(allowed)
ints = sorted({int(v) for v in allowed_norm if v is not None})
has_null = None in allowed_norm
if ints and has_null and set(ints) in ({0, 1}, {0, 1, 2}):
return None
if ints and has_null:
return or_(column.in_(ints), column.is_(None)) # type: ignore[attr-defined]
if ints:
if len(ints) == 1:
return column == ints[0] # type: ignore[operator]
return column.in_(ints) # type: ignore[attr-defined]
if has_null:
return column.is_(None) # type: ignore[attr-defined]
return None
async def pick_random_images(
session: AsyncSession,
*,
r: float,
limit: int,
r18: int = 0,
r18_strict: bool = True,
orientation: int | None = None,
ai_type: int | None = None,
illust_type: int | None = None,
ai_type_allowed: set[int | None] | None = None,
illust_type_allowed: set[int | None] | None = None,
min_width: int = 0,
min_height: int = 0,
min_pixels: int = 0,
min_bookmarks: int = 0,
min_views: int = 0,
min_comments: int = 0,
included_tags: Sequence[str] | None = None,
excluded_tags: Sequence[str] | None = None,
user_id: int | None = None,
illust_id: int | None = None,
created_from: str | None = None,
created_to: str | None = None,
exclude_image_ids: Sequence[int] | None = None,
fail_cooldown_before: str | None = None,
) -> list[Image]:
limit_i = int(limit)
if limit_i <= 0:
return []
if limit_i > 5000:
limit_i = 5000
r = float(r)
if r < 0.0:
r = 0.0
if r >= 1.0:
r = 0.999999999
r18_clause = _r18_where_clause(r18=r18, r18_strict=r18_strict)
orientation_clause = _orientation_where_clause(orientation=orientation)
included_tags_clause = _included_tags_where_clause(tag_names=included_tags or [])
excluded_tags_clause = _excluded_tags_where_clause(tag_names=excluded_tags or [])
exclude_ids_clause = _exclude_image_ids_where_clause(image_ids=exclude_image_ids)
min_width_i = int(min_width)
min_height_i = int(min_height)
min_pixels_i = int(min_pixels)
min_bookmarks_i = int(min_bookmarks)
min_views_i = int(min_views)
min_comments_i = int(min_comments)
if min_width_i < 0 or min_height_i < 0 or min_pixels_i < 0:
raise ValueError("min_* must be >= 0")
if min_bookmarks_i < 0 or min_views_i < 0 or min_comments_i < 0:
raise ValueError("min_* must be >= 0")
clauses = [Image.status == 1]
if r18_clause is not None:
clauses.append(r18_clause)
if orientation_clause is not None:
clauses.append(orientation_clause)
if ai_type is not None:
ai_type_i = int(ai_type)
if ai_type_i not in {0, 1}:
raise ValueError("ai_type must be 0, 1, or None")
clauses.append(Image.ai_type == ai_type_i)
if illust_type is not None:
illust_type_i = int(illust_type)
if illust_type_i not in {0, 1, 2}:
raise ValueError("illust_type must be 0, 1, 2, or None")
clauses.append(Image.illust_type == illust_type_i)
if ai_type_allowed is not None:
allowed = set(ai_type_allowed)
if not allowed:
return []
clause = _allowed_int_or_null_clause(Image.ai_type, allowed=allowed)
if clause is not None:
clauses.append(clause)
if illust_type_allowed is not None:
allowed = set(illust_type_allowed)
if not allowed:
return []
clause = _allowed_int_or_null_clause(Image.illust_type, allowed=allowed)
if clause is not None:
clauses.append(clause)
if included_tags_clause is not None:
clauses.append(included_tags_clause)
if excluded_tags_clause is not None:
clauses.append(excluded_tags_clause)
if exclude_ids_clause is not None:
clauses.append(exclude_ids_clause)
if min_width_i > 0:
clauses.append(Image.width >= min_width_i)
if min_height_i > 0:
clauses.append(Image.height >= min_height_i)
if min_pixels_i > 0:
clauses.append((Image.width * Image.height) >= min_pixels_i)
if min_bookmarks_i > 0:
clauses.append(Image.bookmark_count >= min_bookmarks_i)
if min_views_i > 0:
clauses.append(Image.view_count >= min_views_i)
if min_comments_i > 0:
clauses.append(Image.comment_count >= min_comments_i)
if user_id is not None:
clauses.append(Image.user_id == int(user_id))
if illust_id is not None:
clauses.append(Image.illust_id == int(illust_id))
if created_from is not None:
clauses.append(Image.created_at_pixiv >= str(created_from))
if created_to is not None:
clauses.append(Image.created_at_pixiv <= str(created_to))
if fail_cooldown_before is not None:
clauses.append((Image.last_fail_at.is_(None)) | (Image.last_fail_at <= str(fail_cooldown_before)))
stmt = (
select(Image)
.where(*clauses, Image.random_key >= r)
.order_by(Image.random_key.asc())
.limit(limit_i)
)
items = (await session.execute(stmt)).scalars().all()
if len(items) >= limit_i:
return list(items)
remain = int(limit_i - len(items))
if remain <= 0:
return list(items)
stmt2 = (
select(Image)
.where(*clauses, Image.random_key < r)
.order_by(Image.random_key.asc())
.limit(remain)
)
more = (await session.execute(stmt2)).scalars().all()
if not more:
return list(items)
return list(items) + list(more)
+130
View File
@@ -0,0 +1,130 @@
from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
import sqlalchemy as sa
from sqlalchemy.ext.asyncio import AsyncEngine
from app.core.time import iso_utc_ms
from app.db.session import create_sessionmaker, with_sqlite_busy_retry
DEFAULT_REQUEST_LOGS_KEEP_DAYS = 30
DEFAULT_REQUEST_LOGS_MAX_DELETE_ROWS = 50_000
DEFAULT_REQUEST_LOGS_CHUNK_SIZE = 1_000
@dataclass(frozen=True, slots=True)
class RequestLogsCleanupResult:
cutoff: str
deleted: int
has_more: bool
@dataclass(frozen=True, slots=True)
class RequestLogsCleanupPreview:
cutoff: str
would_delete: int
has_more: bool
def _clamp_int(value: int, *, min_v: int, max_v: int) -> int:
return max(min_v, min(int(value), max_v))
def _cutoff_iso(*, keep_days: int, now: datetime | None = None) -> str:
keep_days_i = int(keep_days)
if keep_days_i < 0:
raise ValueError("keep_days must be >= 0")
now_dt = now or datetime.now(timezone.utc)
cutoff_dt = now_dt - timedelta(days=keep_days_i)
return iso_utc_ms(cutoff_dt)
async def preview_request_logs_cleanup(
engine: AsyncEngine,
*,
keep_days: int = DEFAULT_REQUEST_LOGS_KEEP_DAYS,
max_delete_rows: int = DEFAULT_REQUEST_LOGS_MAX_DELETE_ROWS,
now: datetime | None = None,
) -> RequestLogsCleanupPreview:
cutoff = _cutoff_iso(keep_days=int(keep_days), now=now)
max_delete_rows_i = _clamp_int(int(max_delete_rows), min_v=1, max_v=10_000_000)
Session = create_sessionmaker(engine)
sql = """
SELECT id
FROM request_logs
WHERE created_at < :cutoff
ORDER BY created_at ASC
LIMIT :limit;
""".strip()
async def _op() -> RequestLogsCleanupPreview:
async with Session() as session:
res = await session.execute(sa.text(sql), {"cutoff": cutoff, "limit": int(max_delete_rows_i) + 1})
rows = res.all()
would_delete = min(len(rows), int(max_delete_rows_i))
has_more = len(rows) > int(max_delete_rows_i)
return RequestLogsCleanupPreview(cutoff=cutoff, would_delete=would_delete, has_more=has_more)
return await with_sqlite_busy_retry(_op)
async def cleanup_request_logs(
engine: AsyncEngine,
*,
keep_days: int = DEFAULT_REQUEST_LOGS_KEEP_DAYS,
max_delete_rows: int = DEFAULT_REQUEST_LOGS_MAX_DELETE_ROWS,
chunk_size: int = DEFAULT_REQUEST_LOGS_CHUNK_SIZE,
now: datetime | None = None,
) -> RequestLogsCleanupResult:
cutoff = _cutoff_iso(keep_days=int(keep_days), now=now)
max_delete_rows_i = _clamp_int(int(max_delete_rows), min_v=1, max_v=10_000_000)
chunk_size_i = _clamp_int(int(chunk_size), min_v=1, max_v=100_000)
chunk_size_i = min(int(chunk_size_i), int(max_delete_rows_i))
Session = create_sessionmaker(engine)
delete_sql = """
DELETE FROM request_logs
WHERE id IN (
SELECT id
FROM request_logs
WHERE created_at < :cutoff
ORDER BY created_at ASC
LIMIT :chunk
);
""".strip()
has_more_sql = """
SELECT 1
FROM request_logs
WHERE created_at < :cutoff
LIMIT 1;
""".strip()
async def _op() -> RequestLogsCleanupResult:
deleted = 0
has_more = False
async with Session() as session:
while deleted < int(max_delete_rows_i):
chunk = min(int(chunk_size_i), int(max_delete_rows_i) - int(deleted))
result = await session.execute(sa.text(delete_sql), {"cutoff": cutoff, "chunk": int(chunk)})
n = int(result.rowcount or 0)
if n <= 0:
await session.commit()
break
deleted += n
await session.commit()
if deleted >= int(max_delete_rows_i):
res = await session.execute(sa.text(has_more_sql), {"cutoff": cutoff})
has_more = res.first() is not None
return RequestLogsCleanupResult(cutoff=cutoff, deleted=int(deleted), has_more=bool(has_more))
return await with_sqlite_busy_retry(_op)
Loaded 100 of 309 files, more files were not shown because too many files have changed in this diff. Show more