Commit Graph
952 Commits
Author SHA1 Message Date
ForgQi c2a52973a7 Refactor hook command execution; bump version
Unify and simplify how hook Run steps execute external commands: execute_command now accepts raw stdin bytes (src: &[u8]) and runs the full command string via the platform shell (cmd /C or sh -c), writing src to the child's stdin, streaming stdout/stderr lines to logs, and checking exit status. execute_with for the Run case now builds a newline-separated paths string (when called with file paths) and delegates to execute_command. Removed manual arg parsing and duplicated process-handling logic. Also bumped workspace/package version to 1.1.29 (Cargo.toml) and updated lockfile accordingly.
2026-03-01 21:14:50 +08:00
sora3 d666dc7bd8 New Configuration File Parameters (#1600)
* Update config.toml

* Update config.yaml
2026-02-23 10:27:07 +00:00
112292454 fa0075aa8c feat: 添加合集(SEASON)管理功能 (#1598)
实现 B站新版合集(SEASON 类型)的增删改查接口:
- list_seasons: 列出用户的所有合集
- get_season_section: 获取合集内视频列表
- get_video_info: 获取视频信息(添加合集的前置步骤)
- add_to_season: 将视频添加到合集
- remove_from_season: 从合集移除视频
- sort_season_episodes: 合集内视频排序

API 格式通过逆向创作中心前端 JS 和浏览器 F12 抓包验证,
所有接口均在真实环境测试通过。
2026-02-19 10:00:57 +00:00
zhangaoyun 04d017416b feat: 命令行支持通过 Web 接口投稿 (#1580) 2025-12-27 16:45:48 +00:00
lovegaoshi 5096d60010 fix: filename extraction using os.path.basename (#1579) 2025-12-22 14:26:21 +00:00
XuanRan_Nyan caf106c3f8 Fix(Huya): refactor anticode generation using getCdnTokenInfoEx API (#1575)
* Fix(Huya): refactor anticode generation using getCdnTokenInfoEx API

* chore: cleanup

* chore: get random sHuYaUA

* chore: 代码复用
2025-12-21 13:03:52 +00:00
ForgQi 218b8bd127 Refactor code for conciseness and clarity
Simplified conditional logic using Rust's let-else syntax, removed unused imports, and improved code readability in extractor.rs, lib.rs, uploader/line.rs, and stream-gears/server.rs. These changes reduce code complexity and improve maintainability.
2025-12-15 00:43:24 +08:00
ForgQi b57dc29e96 Add cover downloader and update dependencies
Introduced a new cover_downloader module in the downloader core. Updated several dependencies in Cargo.toml and Cargo.lock, including major upgrades to image processing, HTTP, and async libraries. Adjusted related code in util, download, plugin, and uploader modules to support new features and dependency changes.
2025-12-15 00:41:34 +08:00
Dzming Li 175be7625e feat: 断点续传、标题自动截断及限流重试优化 (#1558)
* feat: 添加断点续传功能

- 实现自动保存上传进度到临时文件
- 支持上传中断后自动恢复并跳过已上传文件
- 为 Video 结构体添加 Clone trait
- 添加 .envrc 用于 direnv 自动加载 Nix 环境
- 更新 flake.lock 依赖

* fix: Auto-truncate video titles exceeding 80 characters

B站API限制分P视频标题不能超过80个字符。当文件名过长时,上传会在提交阶段失败并报错"第(N)个视频的标题过长,已经超过80个字符"。

此次修改在所有上传线路(Upos/Kodo/Cos)中添加自动截断逻辑:
- 添加 Video::truncate_title() 辅助函数,按字符(非字节)截断
- 文件名超过80字符时,自动设置title字段为截断版本(77字符+"...")
- 文件名不超过80字符时,title保持None,使用完整文件名

影响范围:
- crates/biliup/src/uploader/bilibili.rs - 新增truncate_title方法
- crates/biliup/src/uploader/line/upos.rs - Upos线路应用截断
- crates/biliup/src/uploader/line/kodo.rs - Kodo线路应用截断
- crates/biliup/src/uploader/line/cos.rs - Cos线路应用截断

* feat: Auto-retry on upload rate limit (code 601)

当遇到B站限流错误(code: 601, "您上传视频过快")时,自动等待后重试,最多重试5次,等待时间递增。

- 在 pre_upload 中添加重试循环
- 解析错误响应JSON,检测 code: 601
- 递增等待时间:第1次等待1分钟,第2次2分钟,第3次3分钟,第4次4分钟
- 最多重试5次(从原来的3次增加)
- 记录警告日志,显示重试进度和等待时间
- 配合断点续传功能,提高批量上传成功率

* refactor: 将重试逻辑从库层移至 CLI 层

主要变更:
- 新增 RateLimit 错误类型以区分限流错误(code: 601)
- 移除 line.rs 中 pre_upload() 的重试循环,遇到 601 错误直接返回
- 在 CLI 层 (uploader.rs) 实现重试逻辑

* fix: 使用跨平台路径获取断点续传文件位置

- 添加 dirs 依赖
- 使用 dirs::data_local_dir() 获取平台相关的数据目录
- Windows 下使用 %LOCALAPPDATA%,避免 /tmp 路径不存在导致的 panic
- 如果无法获取数据目录,回退到 std::env::temp_dir()

* refactor: 使用通用 retry 工具函数重构重试逻辑

主要变更:
- 改造 lib.rs 中的 retry 函数,新增 retry_with_config 支持:
  - 自定义最大重试次数
  - 条件重试(可指定只在特定错误时重试)
  - 保留原有的指数退避和抖动策略
- 在 CLI 层使用 retry_with_config 替换手写的重试循环
- 代码从 30+ 行简化为 15 行
- 更新 dirs 依赖从 5.0 到 6.0

* feat: 添加账号级上传互斥锁机制

为防止多个进程同时使用同一账号上传触发限流,实现账号级互斥锁:

- 新增 upload_lock.rs 模块:
  - 基于文件锁实现跨进程互斥
  - 使用账号 mid 作为锁标识符
  - 自动清理超过30分钟的僵尸锁
  - Drop 时自动释放锁

- 集成到上传流程(uploader.rs):
  - 在上传前检查是否有其他进程正在等待限流恢复
  - 检测到限流错误时尝试获取锁
  - 如果无法获取锁(其他进程已持有),本进程立即退出
  - 使用 Arc<Mutex<>> 在闭包间共享锁状态

工作流程:
1. 进程1 上传时遇到限流 (Rate Limit)
2. 进程1 获取锁并进入重试等待
3. 进程2 启动并检测到锁已被占用
4. 进程2 立即退出,避免继续触发限流
5. 进程1 重试成功后释放锁
2025-12-06 16:12:44 +00:00
XuanRan_Nyan f0bc3536ad fix(Douyu): extract stream_id from HLS stream (#1560) 2025-12-06 16:11:54 +00:00
ForgQi 08c8370c49 Refactor context and downloader logic for modularity
This commit refactors the context handling in the server, introducing PluginContext and improving separation between plugin and runtime contexts. It updates downloader and uploader logic to use the new context structure, streamlines file suffix handling, and removes redundant fields and code. The changes enhance code maintainability, modularity, and clarity, especially around download/upload workflows and plugin integration.
2025-12-05 09:09:31 +08:00
ForgQi 4fa4b90a37 Update README.md 2025-12-01 03:04:56 +08:00
ForgQi 9837d39d24 Update README.md 2025-12-01 03:00:35 +08:00
ForgQi eefbacd89f Update README.md 2025-12-01 02:53:37 +08:00
ForgQi c172efb6cd Add dynamic log level configuration support
Introduces a new `loggers_level` field in the server config to allow dynamic adjustment of log levels via the API and UI. Updates the developer UI to reflect the new log level controls and ensures the backend applies changes to the logging filter at runtime. Also bumps the version to 1.1.27.
2025-12-01 02:18:20 +08:00
ForgQi 07da196a9d Add frontend build step to release workflow
Updated the GitHub Actions release workflow to include Node.js setup and frontend build steps. Also updated the README to clarify the Windows release download link and removed extra blank lines in the disclaimer section.
2025-11-29 23:37:33 +08:00
ForgQi 9e9c9617d4 Refactor plugin config handling and add release workflow
Removed biliup/config.py and refactored all plugin and downloader classes to accept a config parameter, passing configuration explicitly instead of relying on a global import. Updated Rust backend to pass config objects to Python plugins and added a OnceConfig Python class for config access. Introduced a GitHub Actions release workflow. Updated version to 1.1.26 and added a disclaimer to the README. Minor bug fixes and improvements in Rust uploader and config handling.
2025-11-29 23:24:53 +08:00
sora3 2743cac6e2 fix(douyu): force use of constructed URLs for hs-h5 (#1549)
* Update douyu.py

* Update config.yaml
2025-11-28 10:07:17 +00:00
Dzming Li 37abf9222e fix: 修复 shellexpand::tilde 类型错误并优化 Nix 构建 (#1552)
1. 修复 shellexpand::tilde 类型处理
   - shellexpand::tilde() 返回 Cow<'_, str> 而不是 Result
   - 移除了错误的 Result 模式匹配,直接使用返回值

2. 优化 Nix flake 配置
   - 将耗时的 buildNpmPackage 替换为轻量级占位符,跳过前端构建
   - 禁用测试以避免 server 模块测试依赖前端的问题
   - 减少构建时间和网络依赖
   - 命令行用户可通过 biliup login 的多种方式登录
2025-11-28 06:18:49 +00:00
Dzming Li 4bbc909854 feat: 添加 biliup-cli的Nix flake 支持和路径处理改进等 (#1550)
* fix: 修复空视频路径导致的 panic

在 upload 和 append 命令中添加空视频路径检查,避免 video_path[0] 访问时出现 index out of bounds panic

* feat: 添加 Nix flake 支持

- 在 biliup-cli 中添加 flake.nix 配置文件,支持可复现的构建环境
- 使用 crane 构建系统打包 biliup-cli
- 包含开发环境配置
- 添加 mainProgram 元数据

* feat: 改进配置文件格式错误提示

- 添加详细的错误位置信息(行号和列号)
- 提供友好的错误消息和修复提示
- 优化文件打开失败的错误提示

* feat: 为 Studio 结构体字段添加默认值支持

- 为大部分字符串字段添加 serde(default) 支持
- 为 copyright 添加默认值 1(自制)
- tid 和 title 保持为必填字段
- 用户可在配置文件中省略非必填字段

* feat: 添加对波浪号(~)路径的支持

- 添加 shellexpand 依赖用于路径扩展
- 实现 expand_path 函数处理 ~ 路径
- 在所有路径参数(user_cookie, video_path, config, cover)中应用路径扩展
- 用户现在可以使用 ~/path/to/file 格式的路径
2025-11-28 03:04:14 +00:00
ForgQi 55370dd87a Add dynamic log level reload support to server
Introduces tracing_subscriber::reload to enable dynamic log level changes at runtime. Updates the main server initialization and ServiceRegister to accept and store a log handle. Refactors related code to support this feature and removes disabled state from developer UI log level selectors.
2025-11-27 03:41:09 +08:00
ForgQi 3698bcf28a Add struct-patch support and improve config handling
Introduces the struct-patch crate for partial config updates and applies it to configuration models. Refactors config access patterns, adds PartialEq to several types, and updates LiveStreamer to use ConfigPatch for override_cfg. Also updates the CI workflow to use macos-15-intel, improves download task cancellation logic, and upgrades @douyinfe/semi-ui and related dependencies in the frontend.
2025-11-20 03:26:40 +08:00
ForgQi 4fdd0f81c1 Add Bilibili anonymous origin option and improve hook step execution
Introduces a new 'bili_anonymous_origin' option to the config for Bilibili original quality without login. Refactors hook step command execution to use platform-specific shells, capture and log stdout/stderr in real time, and improves process handling. Also updates workspace version to 1.1.23 and makes minor code cleanups.
2025-11-19 02:28:49 +08:00
ForgQi 7763a4cf3a Refactor download workflow and add is_only_self support
Refactored the download workflow to improve danmaku client handling, status transitions, and resource cleanup. Enhanced logging for better traceability. Added support for the 'is_only_self' field in upload streamer models and the Studio struct, ensuring correct type usage and propagation through the upload pipeline. Updated pythonize dependency to version 0.27.0 and improved queue management logic in the monitor.
2025-11-18 03:28:04 +08:00
ForgQi 5de3dcaa6d Refactor monitor and download manager for improved cleanup
Refactored the monitor system by merging RoomsHandle into Monitor, centralizing plugin and worker management, and improving monitor task spawning. Updated DownloadManager to use the new Monitor structure, added a cleanup method for graceful shutdown, and ensured worker statuses are reset on exit. Adjusted related modules and API endpoints to accommodate these changes, and improved code clarity and error handling throughout.
2025-11-17 03:18:01 +08:00
ForgQi 2810ac9a88 Refactor and clean up unused imports and code paths
Removed unused imports, simplified code by eliminating unnecessary variables and clones, and applied Rust idioms such as let-else and .cloned(). Added Default implementations for certain structs. These changes improve code clarity, maintainability, and reduce potential for confusion or errors.
2025-11-17 01:07:00 +08:00
ForgQi 064111669e Refactor download manager and downloader architecture
Refactored the download manager to unify plugin and worker management, replacing ActorHandle with DownloadManager. Updated API endpoints and download logic to use the new DownloadManager interface. Introduced DownloaderRuntime enum to abstract over different downloader implementations. Added initial ytdlp.rs for YouTube downloads. Refactored StreamGears and Streamlink to fit the new architecture, and updated worker and status handling throughout the codebase.
2025-11-17 01:05:30 +08:00
ForgQi 58e7d3c80f Add Streamlink downloader and refactor Twitch plugin
Introduces a new Streamlink downloader implementation supporting multiple platforms and output modes. Refactors the Twitch plugin to use async traits and separates TwitchDownloader logic. Cleans up endpoint logging and updates workspace version to 1.1.22.
2025-11-05 01:44:31 +08:00
ForgQi 673a1e7eb8 Refactor plugin system and update dependencies
Refactored the plugin system, including changes to plugin loading and Twitch plugin implementation. Updated dependencies in Cargo.lock to their latest versions, improving compatibility and security. Adjusted downloader and download logic to align with the new plugin structure.
2025-11-04 00:56:51 +08:00
ForgQi 99b00d3bc3 Add Twitch plugin and refactor ffmpeg downloader
Introduces a new Twitch plugin module for stream handling, including GraphQL queries and stream URL extraction. Refactors ffmpeg_downloader to extract process logging and waiting logic into a reusable async function. Updates dependencies to use workspace versions for rand and serde_urlencoded, and bumps version numbers to 1.1.21.
2025-11-03 00:58:43 +08:00
ForgQi d1cb0047a5 Refactor segment event processing and downloader callbacks
Refactored SegmentEventProcessor to manage its channel internally and removed unnecessary cloning. Updated downloader traits and implementations to use FnMut callbacks with explicit lifetimes, improving flexibility and correctness. Adjusted file and segment types in the downloader module to be generic over lifetimes, ensuring proper resource management. These changes improve event handling, callback usage, and lifetime safety across the download pipeline.
2025-10-31 02:47:32 +08:00
ForgQi 6051dc8fe9 Refactor downloader architecture and config handling
Refactored the downloader system to pass DownloadConfig explicitly to all downloaders, removing internal state from FfmpegDownloader and StreamGears. Simplified DownloadPlugin::create_downloader to only require the downloader type, and updated DownloadTask to manage cancellation and retries more robustly. Adjusted related code to support the new DownloadConfig flow, improved resource cleanup, and updated trait signatures and usages throughout the codebase.
2025-10-30 02:10:17 +08:00
ForgQi b497749744 Update download task execution and improve docs
Refactored the DownloadTask::execute method to return DownloadStatus and adjusted its visibility. Enhanced DActor message handling to update stream info after download execution. Updated README and introduction guide with improved instructions for background running and systemd service configuration.
2025-10-27 02:05:23 +08:00
ForgQi f5f2998d16 Add streamer file info API and improve file listing
Introduces a new API endpoint `/v1/streamer-info/files/{id}` to fetch file lists for a specific streamer, and updates the frontend to fetch file lists dynamically per row. Also updates serialization for streamer date fields, improves comments and error handling, and changes the default downloader type for plugins. Minor documentation and code cleanup included.
2025-10-27 01:28:08 +08:00
ForgQi 2ede5cda33 Refactor YY plugin to Rust and update downloader logic
Migrates the YY plugin from Python to Rust, implementing it under crates/biliup-cli/src/server/core/plugin/yy.rs and registering it in the service register. Removes the old Python plugin. Refactors downloader instantiation logic into DownloadPlugin, and updates header construction utility. Updates CLI and README to clarify the --auth option. Cleans up duplicate code in stream-gears and synchronizes workspace version numbers.
2025-10-26 17:01:45 +08:00
ForgQi ff3fd2ef15 Refactor tracing and type casting in stream-gears
Removed unused tracing initialization and related code from stream-gears. Replaced deprecated `downcast` calls with `cast` for PyO3 types. Cleaned up imports and minor code improvements in biliup-cli and stream-gears modules.
2025-10-25 00:10:00 +08:00
ForgQi 3ad7c6ac0f Move tracing initialization to server entry point
Refactored tracing initialization from lib.rs to server.rs, ensuring logging setup occurs only when the server starts. Updated README and docker-compose.yml to reflect new server startup command and options for authentication.
2025-10-24 23:07:43 +08:00
ForgQi 9e155493f0 Refactor imports and formatting across modules
This commit cleans up and reorders imports for consistency, improves formatting (especially for multi-line expressions and function signatures), and fixes minor style issues in several files. It also updates the server command to pass the 'auth' parameter and makes minor logic clarifications in the stream-gears Python entrypoint.
2025-10-24 03:21:44 +08:00
ForgQi b829731ad9 Refactor CLI, update dependencies, and improve server logic
Refactored CLI argument parsing and main entrypoint in both Python and Rust, updated pyo3 and pythonize dependencies, and improved modularization in biliup-cli. Added an 'auth' flag to the server command, fixed several import and ownership issues, and improved plugin and server initialization logic. Also cleaned up unused imports and streamlined code for better maintainability.
2025-10-24 03:11:27 +08:00
ForgQi a0b6800c0a Refactor async worker status and improve file move logic
Refactored worker status management to use async RwLock and updated all related methods to be async. Improved file move logic in hook steps to support cross-filesystem moves by falling back to copy+delete when necessary. Updated API endpoints, download/upload logic, and monitor state transitions to be fully async and more robust. Also added a Debug implementation for Credentials to avoid logging sensitive data, and made minor UI tweaks to disable certain developer options.
2025-10-22 02:13:37 +08:00
ForgQi 2fcdb0fe7b Move associated XML file with video in HookStep
When moving a video file, also check for an associated XML file (e.g., subtitles or metadata) and move it to the target directory if it exists. This ensures related files are kept together during the move operation.
2025-10-01 01:52:04 +08:00
ForgQi af82185618 Refactor recorder usage and context handling
Refactored the handling of the Recorder struct to include recording start time and improved filename templating. The Context struct now holds a Recorder instance, and its initialization was updated throughout the codebase. Adjusted upload and download logic to use the new Recorder, and improved logging in hook steps. Bumped version to 1.1.15.
2025-10-01 01:45:55 +08:00
ForgQi ea17305b0a Refactor actor system and add upload status tracking
Refactored download and upload actor implementations into separate modules for better maintainability. Added upload_status field to LiveStreamerEntity and API responses, and updated the frontend to display upload status. Also added Huya protocol config option and improved Huya stream header handling in stream-gears. Minor bug fixes and code cleanups included.
2025-10-01 00:26:07 +08:00
ForgQi e2da70b9ea Update config.rs
Bump workspace version to 1.1.13

Updated the workspace and related package versions from 1.1.12 to 1.1.13 in Cargo.toml and Cargo.lock for release and dependency consistency.
2025-09-30 01:52:54 +08:00
ForgQi 247f231640 Refactor log tabs and improve download/upload handling
Merged download and upload logs into a single tab in the log viewer and commented out the separate upload log tab. Removed unused and legacy methods from the Python upload and download engine classes. Improved logging and error handling in Rust server modules, including more informative log messages and better context propagation. Updated plugin interface to pass context instead of just URL, and enhanced dictionary filtering in config state. Bumped workspace version to 1.1.12.
2025-09-30 01:44:46 +08:00
ForgQi 3e76841206 Update README.md 2025-09-30 00:07:39 +08:00
ForgQi bd1ccdd0ff Refactor upload pipeline and improve code consistency
Refactored upload pipeline to use stream inspection for file item insertion, improving modularity and resource handling. Cleaned up imports, improved formatting, and removed redundant logging and code. Minor fixes in module ordering and code style for better maintainability.
2025-09-29 04:19:22 +08:00
ForgQi 92b5a30df5 Add streamer pause/resume feature and UI updates
Introduces a pause/resume button for streamers in the UI and backend, including API endpoint and status handling. Refactors streamer status display, updates related TypeScript interfaces, and improves modal forms. Also updates @douyinfe/semi-ui and related dependencies to v2.86.0, and bumps workspace version to 1.1.10. Minor logging and trait changes in Rust backend for improved clarity and error handling.
2025-09-29 03:18:21 +08:00
ForgQi d4b6a5fb22 Refactor models and context for streamer separation
Split LiveStreamer and UploadStreamer models into separate modules and update references throughout the codebase. Refactor StreamInfo to StreamInfoExt, embedding StreamerInfo for improved database integration. Update context, plugin, monitor, and related logic to support new model structure and database operations. Bump version to 1.1.9.
2025-09-27 16:42:53 +08:00
ForgQi e41acef883 Refactor upload logic and add WebSocket log endpoint
Refactored upload logic to decouple upload and submission, moving UploadLine and related logic to biliup-cli crate root. Added a new WebSocket log endpoint in a dedicated ws.rs module and routed it accordingly. Updated database migration to set certain fields to null instead of empty strings. Adjusted context and download/upload manager logic to pass StreamInfo and Context as needed. Added static file serving route and improved filename generation for uploads. Bumped version to 1.1.8 and enabled 'fs' feature for tower-http.
2025-09-27 01:13:18 +08:00