Commit Graph
995 Commits
Author SHA1 Message Date
Noah 2c3592f117 fix(bilibili): 稿件分页查询复用登录客户端,不再每页新建 Client (#1675)
archives() 之前在分页循环中每请求一页都重新 build 一个
reqwest::Client 和 cookie jar:连接无法复用,代理配置丢失导致
查询绕过 --proxy 直连;且 Jar::add_cookie_str 会把 "; " 拼接的
整串 Cookie 当作单条 Set-Cookie 解析,实际只有第一个 Cookie
生效,其余键值被当作 Cookie 属性丢弃。

改为复用登录时构建的 self.client:cookie store 已含全部登录
Cookie,UA 与原实现一致,并保留代理配置与连接池,同时按原
行为为该请求保留 60 秒总超时。
2026-08-30 14:52:47 +00:00
Noah 5c6c720fdc fix(upload): 测速时单条线路失败不再中断整体测速 (#1674)
Probe::probe 对每条线路的 ping 使用 ? 传播错误,任意一条线路
DNS/连接/读取失败都会让整个测速立即返回错误,即使后续线路
完全可用,调用方也会因 unwrap_or_default 直接退回默认线路。

现在单条线路网络错误或异常状态码只记录警告并跳过该线路,
继续测速其余候选;全部线路失败时返回统一错误。
2026-08-30 14:51:09 +00:00
NoahandNoah 616432d190 feat(sync): 恢复边录边传下载器的流式上传管线 (#1673)
Rust 重写后 sync-downloader 被静默映射为 stream-gears 落盘下载,
边录边传完全失效(#1612)。本次恢复 ffmpeg stdout 到 UPOS 流式
分片上传、逐段投稿/追加分 P 的完整链路,并修复可靠性问题:

- UPOS 拆分为上传 parts 与显式 complete 两阶段,parts 按
  partNumber 排序并校验连续,complete 带状态检查与重试
- 录制同时写临时文件(有界通道背压);短流/预传失败时按实际
  长度回退为文件重传,投稿成功后清理临时文件
- SyncSession 跨拉流重连保留 aid,分 P 严格按录制顺序提交,
  首投结果不确定时停止自动重试避免重复建稿
- 上传/投稿错误向上传播;外层重试只在分 P 推进后重置退避
- 修复 www.douyin.com/live/{id} 被解析成 web_rid=live 的问题

Co-authored-by: Noah <17682131+ransxd@users.noreply.github.com>
2026-08-30 14:50:37 +00:00
yjywybb 051f5c7eb0 fix(upload): preserve filename-invalid characters in submission titles (#1671) 2026-08-25 07:50:23 +00:00
ForgQi 03b7a84f55 Allow remote Web UI bootstrap with auth
This change updates the server default bind to 127.0.0.1 for safer local-only access while documenting how to expose the Web UI via --bind 0.0.0.0 --auth for remote or Docker use. It also removes the previous restriction that blocked first-time admin bootstrap on non-loopback addresses, allowing authenticated remote initialization and bumping the workspace version to 1.2.4.
2026-08-24 01:13:29 +08:00
NoahandNoah c351c39771 fix(server): 会话 Cookie 不再强制 Secure,修复 HTTP 远程访问登录后无法保持登录态 (#1669)
Co-authored-by: Noah <17682131+ransxd@users.noreply.github.com>
2026-08-23 15:20:11 +00:00
NoahandNoah 6cef8afe22 fix(stream-gears): 登录函数释放 GIL 后再执行阻塞网络请求 (#1667)
Co-authored-by: Noah <17682131+ransxd@users.noreply.github.com>
2026-08-22 19:25:48 +00:00
ForgQi eb8699b203 Add recording policy and time-range checks
This change adds a recording policy layer that enforces time-window eligibility and excluded title keywords before probing and before recording. It updates the streamer API to surface reasons like OutOfSchedule and TitleExcluded, clamps ffmpeg/segment durations to the configured window, and exposes the last rejection reason from the worker state for the frontend.
2026-08-22 19:15:48 +08:00
NoahandNoah 1affe97d87 fix(webup): Data 声明 desc_v2 字段,避免提交时被 asdict 静默丢弃 (#1666)
Co-authored-by: Noah <17682131+ransxd@users.noreply.github.com>
2026-08-22 08:52:05 +00:00
cls3389 8fe338b5f5 fix(douyin): 下播后回放房间仍被误判为直播中 (#1663)
抖音主播下播后,直播间跳转 reflow 回放页,接口仍返回 status=2
和可拉取的回放流,导致 monitor 循环误判开播并反复录制垫片流。

两处独立防线:
- get_room_info: finish_time > 0(抖音已写入直播结束时间)判定未开播
- select_stream_url: 流 URL 含 rtm_expr_tag=reflow_room_info
  (回放流标记)时拒绝作为直播流返回

Fixes #1532
2026-08-22 08:39:50 +00:00
NoahandNoah 3750ced583 fix(hls): respect playlist refresh lifecycle (#1665)
Co-authored-by: Noah <17682131+ransxd@users.noreply.github.com>
2026-08-22 08:37:05 +00:00
Noahandransxd e1c4b6887e fix(downloader): 纯视频流分段时避免序列头 expect 导致的 panic (#1664)
分段重建逻辑此前对 on_meta_data / aac_sequence_header / h264_sequence_header 使用 expect。纯视频流(无音频,aac_sequence_header 恒为 None)在首次分段时必然 panic,导致 7x24 录制任务中断。改为缺失时跳过并告警,不再中断录制。

新增回归测试 pure_video_stream_segments_without_panic:构造仅含 onMetaData 与 H264 序列头、无音频标签的 FLV 流并触发分段;修复前该测试会 panic(aac_sequence_header does not exist),修复后通过。

Co-authored-by: ransxd <ransxd@users.noreply.github.com>
2026-08-22 08:35:19 +00:00
Mostly_Harmless b438d72415 feat(web): 完善账号管理、稿件查询与安全边界 (#1660)
启动 Web 服务时自动登记 CLI 指定的 B 站凭据,并为账号登记、刷新和删除统一补上并发去重、原子写入及引用检查。

同时收紧默认监听、管理员初始化、认证路由、静态文件和日志 WebSocket,恢复 server 的 ds_update.log,并避免凭据及上游响应进入日志。

新增按账号查询远程稿件的 API 与 Web 页面,补齐严格分页和响应校验。

关联 biliup/biliup#1624

关联 biliup/biliup#1658

关联 biliup/biliup#724
2026-08-16 14:05:20 +00:00
vector0902androotuc ccb016be4f fix(upload): include lock file path in rate-limit lock error message (#1661)
Co-authored-by: rootuc <rootuc@notunixchan>
2026-08-15 06:40:18 +00:00
PingouinFerreux 0c3bbe2ec1 fix: switch star history chart to a working provider (#1659)
The star history chart in the README is no longer loading because the upstream API it depended on stopped working. Point the chart and its link to the star-history service hosted at star-history.dera.page, which uses a different data source and requires no API key, so the chart renders again for everyone.

This keeps the same query and hash parameters as before, so only the domain in the two URLs changes.
2026-08-14 07:59:58 +00:00
dreammu 3e8b55c2c4 fix(upload): update upload lines (#1656) 2026-08-07 09:28:12 +00:00
Micuks adf6a1c03b fix(hls): 选取最高画质的流,而非固定取 variants[0] (#1649)
HLS 下载器一直取 master playlist 里的第一个变体。Twitch 会把各档
转码清晰度排在源流前面,所以经常录到较低画质的转码流,而不是源流。

改为选取带 RESOLUTION、且码率最高的变体,并跳过 I-frame(trick-play)
流。若没有变体带分辨率,则回退到码率最高的非 I-frame 变体,最后回退
到第一个变体,保证在异常 playlist 下行为不退化。
2026-07-14 14:35:35 +00:00
ForgQi 03c36e1da7 Add batch live-check support + fixes
Introduce batch live-check infrastructure and implementations. Add BatchCheckRequest and LivePlugin::supports_batch_check/batch_check API; Monitor uses BatchLiveCache and batch_verdict to prefilter rooms (platform-level checks + caching and fast rotation). RoomsActor gains PlatformUrls. Implement batch_check for Picarto (explore pagination) and Twitch (GQL batching with auth retry). Other fixes: Douyu auth-failure detection and key invalidation + retry, Inke optional response handling, streamlink runtime tweaks for yt-dlp/Twitch ffmpeg, small test additions and import refactors.
2026-07-14 21:33:23 +08:00
ForgQi 083313257f Improve downloader reliability and add caching
- Add process-level caching for expensive API calls: login tokens (Afreecatv, Douyin), encryption keys (Douyu), room IDs (Douyu, Huya)
- Implement Huya mini-app API option (huya_mobile_api) with WUP protocol support for getCdnTokenInfoEx
- Add cdn_fallback for Huya with health checks and protocol negotiation
- Improve Bilibili stream selection with protocol fallback (hls_fmp4→stream) and qn-based grouping
- Fix Twitch auth token invalidation handling and improve cookie file management
- Refactor YouTube entry selection with proper playlist recursion and date filtering
- Add retry logic for network failures (Douyu betard) and improve error recovery
- Share WBI signer state across Bilibili check_stream calls to respect 2-hour key cache interval
- Always pass --download-archive for YouTube (including live) to prevent post-stream duplicates
- Add field aliases (inke: live_name, kilakila: backPic) and fix Niconico suffix (flv→ts)
2026-07-14 00:32:29 +08:00
ForgQi 18c5bf086e Define AT_HWCAP2 for ppc64le; enable tokio
Set CFLAGS_powerpc64le_unknown_linux_gnu=-DAT_HWCAP2=26 in CI to work around aws-lc-sys (via rustls) calling getauxval(AT_HWCAP2) on manylinux2014 where glibc 2.17 headers lack AT_HWCAP2; glibc still provides getauxval so the value is safe and only used for ppc64le cc invocations. Also enable the tokio "io-std" feature for the biliup-cli crate to opt into the io-std I/O implementation.
2026-07-11 03:53:51 +08:00
ForgQi ddc29f63b4 Update deps & fix server upload/download
Bump dependencies (Cargo.toml + Cargo.lock) to newer crate versions and adapt code to API changes. Refactor server-side upload/download logic, the upload_streamer model, and related API endpoints to improve streaming handling, buffer/error propagation and compatibility with updated crates (e.g. tokio-tungstenite/futures/etc.). These changes align the codebase with the upgraded dependency set and fix streaming/endpoint issues introduced by those upgrades.
2026-07-11 01:57:48 +08:00
ForgQi 58dc75b6ca Build free-threaded wheels for Python 3.14t (#1648)
maturin 1.14 dropped support for free-threaded Python 3.13
("Free-threaded Python interpreter is only supported on 3.14 and
later"), which broke the linux/musllinux/macos free-threaded wheel
steps since maturin-action always uses the latest maturin. Switch
the free-threaded builds from python3.13t to python3.14t, matching
what maturin generate-ci now emits.
2026-07-10 17:08:08 +00:00
Micuks 33649fb568 fix server submit_api web routing (#1645) 2026-07-10 15:11:39 +00:00
Aluneu 165d4eac68 Update CHANGELOG.md (#1646)
更新发行版信息
2026-07-05 11:08:39 +00:00
XuanRan_Nyan 08b96b73f8 feat(douyu): build huos url for forced hs (#1644) 2026-06-26 11:59:42 +00:00
dplei 0fe20687a3 fix(upload): fall back to live URL when copyright_source is blank (#1640)
When the upload template's reprint source (copyright_source) is left
empty, the web form submits an empty string rather than null, so it is
stored as Some(""). build_studio used `unwrap_or_else`, which only fires
on None, so the submitted source became an empty string instead of the
live room URL — contradicting the UI hint
"如不填写转载来源默认为直播间地址".

Treat None and whitespace-only values as "unset" and fall back to the
live room URL. Extract the resolution into a pure `resolve_source`
helper with unit tests covering none / empty / whitespace / real value.
2026-06-11 02:54:59 +00:00
ForgQi b2ab974431 Update README with new features and cleanup
Reorganize features and remove command line tool section.
2026-06-08 01:32:00 +08:00
ForgQi b5ddff0564 Refactor Python package and simplify entry
Remove legacy engine and many plugin modules, refactor upload plugins to standalone classes, and simplify package entry points. __init__.py now provides a safe __version__ fallback; __main__.py directly invokes stream_gears.main_loop. BiliWeb/BiliWebAsync plugins were adapted to use a FileInfo NamedTuple and no longer depend on engine.UploadBase/Plugin decorator. Documentation updated (README expanded with new commands and CLI/SKILL integration) and a new SKILL.md added. Workspace version bumped to 1.2.1 in Cargo.toml (lock updated accordingly).
2026-06-06 03:22:54 +08:00
ForgQi 431d1f61ab Support optional segment limits and danmaku files
Make file_size and segment_time optional and normalize/validate their values so segmentation can be disabled. Frontend: send nulls for empty fields and allow empty segment_time via validation; add showClear to inputs. Downloader: extend SegmentInfo with optional danmaku_file_path and propagate it from various downloaders; DanmakuClient::rolling now returns a bool to signal whether a danmaku file was produced. Uploader: add segment_paths helper to include danmaku files in upload/postprocessing, and preserve processed paths; add unit tests. HookStep: stream child process stdout/stderr as raw bytes to terminal and append to a log file (tee), and simplify run/move logic to use explicit path lists. Cargo.toml: bump reqwest to 0.13.4. Tests and minor adjustments added throughout to support these changes.
2026-06-04 01:19:27 +08:00
ForgQi a529bbcbd2 Update lib.rs 2026-06-01 02:20:59 +08:00
ForgQi 3655679a58 Refactor live downloader architecture
Migrate the server-side live downloading to the new biliup::downloader::live API and simplify the actor model. Key changes:

- Introduce a new server/core/live module that builds LiveRequest/LiveOptions/LiveCredentials, maps runtimes, and creates danmaku clients.
- Replace legacy plugin/DownloadBase actor flow with LivePlugin-based workflows and a start_download_workflow entrypoint; monitor now acquires a Semaphore permit before starting a download to enforce concurrency.
- Remove the old plugin module and per-platform plugin plumbing; many platform implementations were moved/renamed under crates/biliup/src/downloader/live.
- Adjust DownloadManager, Monitor, and common download/upload code to use LivePlugin, LiveStream and related types; pass uploader channel and download slot semaphore into Monitor.
- Add download_stream helper to support direct FLV/HLS downloads via StatelessClient, and extend Streamlink downloader to support Niconico credentials/options.
- Update usage of context/streamer info and danmaku initialization to the new LiveStream-based model.
- Bump workspace version to 1.2.0 (Cargo.toml) and related package versions.

This refactor centralizes live logic on the shared biliup downloader live API, simplifies concurrency control, and removes the previous actor-based plugin indirection.
2026-06-01 01:50:00 +08:00
ForgQi 219334f936 fix ci 2026-05-30 02:24:52 +08:00
ForgQi df898c5d35 Update Dockerfile 2026-05-30 01:37:52 +08:00
ForgQi bde581fa07 Update Dockerfile 2026-05-30 01:27:28 +08:00
ForgQi c395b31148 Replace Python Danmaku with Rust implementation
Remove the legacy Python biliup/Danmaku package and related douyin utilities, and adjust PyInstaller spec to stop bundling those resources. Introduce and expand Rust-based danmaku and plugin implementations (crates/danmaku, crates/biliup-cli server/core/plugin/*, and stream-gears changes) and update various server/downloader modules to integrate them. Also add a TOML dependency to Cargo.toml to support the new Rust changes.
2026-05-30 01:17:18 +08:00
XenoAmess 62bd16de8e chore(doc): update CHANGELOG.md and other docs (#1631) 2026-05-28 02:02:30 +00:00
方程 697a525b55 Add comment listing and reply commands (#1628) 2026-05-25 06:38:50 +00:00
Micuks b2b92e6fee feat(hook): 启用 segment_processor 阶段,新增 remux:mp4 内建钩子 (#1626)
* feat(hook): built-in remux:mp4 hook + activate segment_processor stage

B站 transcoder rejects raw .ts uploads with PTS jumps ("您的视频未能成功
转码 ... 该视频内部存在时间戳跳变"), which is common on HLS reconnects
mid-segment. The fix is to feed it an mp4 with monotonic PTS — a plain
ffmpeg copy-remux is enough.

Add a built-in `HookStep::Remux { remux: String }` variant that runs
`ffmpeg -fflags +genpts+igndts -c copy -bsf:a aac_adtstoasc
-movflags +faststart -avoid_negative_ts make_zero` on .ts/.m2ts inputs.
Lossless and idempotent (reuses existing .mp4 sibling). Currently
target="mp4" only.

To make this useful, segment_processor (defined in config but never
previously invoked) is now executed in pipeline_upload_videos BEFORE
each upload_single_file. That's the right stage for ts→mp4 — must
run before pre_upload, not after.

Path-mutating processor pipeline:
- new execute_paths(&mut Vec<PathBuf>) API on HookStep
- new process_video_paths helper
- Remux replaces .ts entries with the new .mp4 entries so downstream
  stages (upload, postprocessor) see the right paths.
The legacy execute(&[&Path]) path stays for postprocessor-style steps
that don't mutate the working set.

Config form:

  segment_processor:
    - {remux: "mp4"}

Tests cover YAML/JSON deserialization of all 4 HookStep variants
and Remux skipping non-.ts inputs.

* fix(upload): 单段 segment_processor / upload 失败不再拖死整批

之前 pipeline_upload_videos 里两个 ? 把单段失败直接抛到
process_with_upload,让后续 submit + execute_postprocessor(含 rm)
全部跳过。在磁盘紧的场景下,ffmpeg remux 写 mp4 header 时遇到
"No space left on device" → segment_processor Err → 整个直播会话已
成功上传的段也不会触发本地 rm → 文件堆积 → 磁盘更紧 → 下一段又失败,
正反馈把磁盘吃完。

把两处 ? 改成 log + continue:
- segment_processor 失败:跳过当前段,下个事件继续
- upload_single_file 失败:跳过,已上传的段照常进入 uploaded.{videos,paths}

直播会话结束(channel close)时,submit_to_bilibili 仍然能用收集到
的 videos 提交,execute_postprocessor 仍能 rm 已成功的本地文件。

修复对应残留 1.6 TB 历史文件的根因(35 次/5 天 "Process segment
event failed: ffmpeg remux failed ... No space left on device")。

* test(hook): remux 在 ffmpeg 失败时返回 Err(供 pipeline 容错前置条件)

复现 pipeline_upload_videos 容错路径的触发条件:segment_processor
(Remux step) 在 ffmpeg 退非 0 时返回 Err。生产事故的具体表现就是磁盘
满让 ffmpeg 写 mp4 header 失败,调用方必须能从该 Err 优雅恢复
(log + continue),否则整批已上传的段都拿不到 postprocessor 的 rm。

完整 pipeline_upload_videos 的端到端测试要 mock UploadContext
(BiliBili client + 登录态),这是后续重构 seam 的方向。
2026-05-21 02:15:23 +00:00
XuanRan_Nyan a24d46aae7 refactor: Implementing the Danmaku client in Rust (#1570)
* refactor: Implementing the Danmaku client in Rust

* test
2026-05-18 04:16:23 +00:00
Micuks e70679dbd7 fix(server): 上传失败后重建 segment 通道,避免后续段被静默丢弃 (#1623)
`SegmentEventProcessor` 第一次为某个 streamer 处理事件时会创建一条
`async_channel::bounded(32)` 把段送给 `UActor`,并把 `tx` 缓存到
`self.channel`。一旦后续 `process_with_upload` 因为 `upload_single_file`
失败而 `?` 抛错,`UActor` 在 `handle_message` 走完 `Err` 分支就把 `rx`
drop 掉,但 `processor` 这边的 `tx` 仍然挂着。

之后只要 `execute()` 还在重试循环里没退出(streamer 在播 + 下载继续),
新段都会落到 `Some(tx)` 分支:`tx.force_send` 对已关闭通道返回 Err →
`Failed to send to buffer` → 段被静默丢弃。直到下载侧 `max_retries` 用
完、`processor` 才被 drop 重建。

实测:单台机器 24h 内出现 64 次 `Failed to send to buffer`,集中在两位
upload 失败过的 streamer 上,每 30min(segment_time)和每次 HLS
`#EXT-X-DISCONTINUITY` 各丢一段。

修复:进入 `match` 之前先用 `Sender::is_closed()` 探测,发现挂掉的 tx
就把 `self.channel` 置 None,复用下面的「首次创建」分支重新发 rx 给
UActor,恢复管道。
2026-05-09 04:03:50 +00:00
XuanRan_Nyan 28b0bd7105 fix(Huya): WEB_ROOM_DATA_REGEX 正则错误匹配 html 转义 (#1606) 2026-03-09 06:13:25 +00:00
gwy15 57a7741302 feat: upload_by_config 支持通过 Web 提交视频配置 (#1605) 2026-03-06 15:24:30 +00:00
XuanRan_Nyan 826fcede27 chore(Douyu): 适配新版本斗鱼 (#1603) 2026-03-03 15:41:55 +00:00
ForgQi c2a52973a7 Refactor hook command execution; bump version
Unify and simplify how hook Run steps execute external commands: execute_command now accepts raw stdin bytes (src: &[u8]) and runs the full command string via the platform shell (cmd /C or sh -c), writing src to the child's stdin, streaming stdout/stderr lines to logs, and checking exit status. execute_with for the Run case now builds a newline-separated paths string (when called with file paths) and delegates to execute_command. Removed manual arg parsing and duplicated process-handling logic. Also bumped workspace/package version to 1.1.29 (Cargo.toml) and updated lockfile accordingly.
2026-03-01 21:14:50 +08:00
sora3 d666dc7bd8 New Configuration File Parameters (#1600)
* Update config.toml

* Update config.yaml
2026-02-23 10:27:07 +00:00
112292454 fa0075aa8c feat: 添加合集(SEASON)管理功能 (#1598)
实现 B站新版合集(SEASON 类型)的增删改查接口:
- list_seasons: 列出用户的所有合集
- get_season_section: 获取合集内视频列表
- get_video_info: 获取视频信息(添加合集的前置步骤)
- add_to_season: 将视频添加到合集
- remove_from_season: 从合集移除视频
- sort_season_episodes: 合集内视频排序

API 格式通过逆向创作中心前端 JS 和浏览器 F12 抓包验证,
所有接口均在真实环境测试通过。
2026-02-19 10:00:57 +00:00
zhangaoyun 04d017416b feat: 命令行支持通过 Web 接口投稿 (#1580) 2025-12-27 16:45:48 +00:00
lovegaoshi 5096d60010 fix: filename extraction using os.path.basename (#1579) 2025-12-22 14:26:21 +00:00
XuanRan_Nyan caf106c3f8 Fix(Huya): refactor anticode generation using getCdnTokenInfoEx API (#1575)
* Fix(Huya): refactor anticode generation using getCdnTokenInfoEx API

* chore: cleanup

* chore: get random sHuYaUA

* chore: 代码复用
2025-12-21 13:03:52 +00:00
ForgQi 218b8bd127 Refactor code for conciseness and clarity
Simplified conditional logic using Rust's let-else syntax, removed unused imports, and improved code readability in extractor.rs, lib.rs, uploader/line.rs, and stream-gears/server.rs. These changes reduce code complexity and improve maintainability.
2025-12-15 00:43:24 +08:00