fix(route): reserve deadline stages for fallback (#214)

Prevents silent peer candidates from starving selected/SFU fallback while preserving one room-serial operation and one total deadline.
This commit is contained in:
AAA authored and GitHub committed 2026-08-23 23:59:56 +08:00
1 parent cc44764e74
commit de6f6aa466
15 files changed
+449 -21

No files matched your search

@@ -0,0 +1,9 @@
# Install as /etc/systemd/system/screener.service.d/livekit-readiness.conf only
# when this Screener instance is configured with the same-host LiveKit owner.
[Unit]
Wants=livekit.service
After=livekit.service
[Service]
ExecStartPre=/bin/sh -c 'attempt=0; until curl -sS --max-time 1 -o /dev/null http://127.0.0.1:7880 2>/dev/null; do attempt=$((attempt + 1)); [ "$attempt" -ge 30 ] && exit 1; sleep 1; done'
+1
View File
@@ -1,5 +1,6 @@
[Unit]
Description=Screener WebRTC signaling and web application
# Same-host LiveKit deployments also install the tracked readiness drop-in.
After=network-online.target
Wants=network-online.target
@@ -144,6 +144,12 @@ and one total operation deadline. Candidate failure advances the cursor without
resetting that deadline. These fields do not become separate gates or state
machines, and this route wave adds no
assignment, media-binding, or proof generation to the wire.
The same one timer derives wake boundaries from the route classes actually
present in the deterministic list: direct peer, selected TURN, and SFU. The
total deadline is divided equally between those semantic stages, without fixed
per-candidate milliseconds. Hard failures may advance through multiple tuples
inside a stage; a silent tuple at its boundary skips the remaining tuples of
that class so it cannot consume the selected/SFU suffix.
A `prepare` route update names that operation's exact child, selected transport,
and server-issued candidate connection identity. Parent and child therefore
prepare the same connection even when the parent's child set is unchanged by a
@@ -309,7 +315,7 @@ deployment-wide.
## Current Deployment Boundary
Production release `352c457` runs the `screener-v8` Browser runtime. Its exact
Production release `d3ff9e7` runs the `screener-v8` Browser runtime. Its exact
configuration, rollback artifacts, and postflight evidence are owned by the
deployment document; real heterogeneous-network and SFU/TURN media validation
remains open.
+12 -7
View File
@@ -2,7 +2,7 @@
Last verified against upstream documentation: 2026-08-23.
This page records exact release `352c457` production facts and the current source
This page records exact release `d3ff9e7` production facts and the current source
deployment contract. Product direction and pending migrations are owned by
[project memory](./project-memory.md) and [the TODO ledger](./todo.md).
@@ -25,13 +25,13 @@ tracked coturn example remains `stun-only`; the shared host retains its older
authenticated-relay daemon configuration and firewall range. The application
does not pre-advertise TURN credentials to ordinary peers.
Production runs exact `352c4578954c399e56a2578cf3e22f5e0cfc2bad`, release
`352c457`, from `/opt/screener/releases/352c457`. The immutable runtime archive
Production runs exact `d3ff9e7b7b4a8fe58db700565971aaeda638d2e9`, release
`d3ff9e7`, from `/opt/screener/releases/d3ff9e7`. The immutable runtime archive
SHA-256 is
`d7feb021821b2af3db5180aed780da67ca582abfb3fe7a8805f6c6d88042bd3c`.
Immediate rollback is `/opt/screener/releases/6b87732`, with the verified
pre-cutover SQLite/environment/LiveKit/unit backup at
`/opt/screener/backups/352c457-precutover-20260823T135843Z`. Local and public
`a7c3b7b7cf593f3933a1c194a07370131fd76b3d4c30768b5206e84ccf93b63d`.
Immediate rollback is `/opt/screener/releases/352c457`, with the verified
pre-cutover environment/LiveKit/unit backup at
`/opt/screener/backups/d3ff9e7-precutover-20260823T150618Z`. Local and public
health return 200; Screener, LiveKit, coturn, and nginx are active with
`NRestarts=0`.
@@ -125,6 +125,11 @@ private to nginx and the Screener process uses
`LIVEKIT_API_URL=http://127.0.0.1:7880` for `RoomService`; WebRTC media reaches
LiveKit directly on UDP 7882. No unrelated application may create rooms on this
instance, and its tracked configuration sets `room.auto_create: false`.
Install
[`screener-livekit-readiness.conf.example`](../deploy/systemd/screener-livekit-readiness.conf.example)
as a `screener.service.d` drop-in on this layout. `After=` orders the services,
while the bounded `ExecStartPre` waits for TCP 7880 readiness; ActiveState alone
does not prove the LiveKit control listener is accepting connections.
## Candidate boundary and rollback
+2 -1
View File
@@ -29,8 +29,9 @@ Last updated: 2026-08-23
- Canonical source is the root `main` branch. Audit/integration boundaries require it to be clean, and new branches/worktrees start from its exact commit only after accepted truth is merged.
- Current source admits up to `20` Viewers and implements the uniform `1/2/3` endpoint cap, single Browser `screener-v8` wire, exact prepare candidate, first-decoded-frame commit, strict rollback revision, decoded-progress stall, and one event-driven room controller. Older Browser and executable-sender wires fail before room authority. SFU admission uses exact publication and Viewer-subscription handles with explicit no-default ingress/egress capacities and a listener-fenced dedicated LiveKit owner. Selected TURN uses its own explicit no-default deployment capacity and exact logical-allocation ledger shared by independent peer edges and Host-SFU ingress.
- Current source and production implement the memory-resident room model: random free four-digit codes, the configurable dormant lease, exact-Host resume, restart loss, local creation-preference replay, orthogonal Viewer grants and `open | password | disabled` code entry, and direct invitation/policy controls. SQLite room persistence, `ROOM_DATABASE_PATH`, `ROOM_TTL_SECONDS`, and the writable room StateDirectory are removed from the v8 runtime; the stopped v7 SQLite snapshot exists only in the verified rollback backup.
- Production runs exact `352c4578954c399e56a2578cf3e22f5e0cfc2bad`, release `352c457`, wire `screener-v8`; `6b87732` is the immediate rollback release.
- Production runs exact `d3ff9e7b7b4a8fe58db700565971aaeda638d2e9`, release `d3ff9e7`, wire `screener-v8`; `352c457` is the immediate rollback release.
- Production implements the current endpoint-cap and route-controller contract, admits 20 Viewers, uses LiveKit admission of one publication ingress plus 20 subscription egress handles on a dedicated non-auto-create instance, and bounds selected TURN to two logical allocations.
- Browser SFU subscribers reconcile eligible Host screen publications across connect/activate event ordering, participant/track arrival, and reconnect. Production LiveKit logs only warn/error, and the application service has a bounded same-host LiveKit readiness preflight.
- Production retains access controls, self-check, diagnostic export, signaling watchdog behavior, 64/128/256 audio choices, and selected-edge transport. Real heterogeneous-network and SFU/TURN media evidence remains open.
- `screener`, LiveKit, coturn, and nginx are healthy; local and public health return 200 and `screener` has `NRestarts=0`. Deployment detail is owned by [deployment](./deployment.md).
- Chrome loopback, cap2/cap3, short resource, recovery, MBB, and A/V fixtures are bounded evidence only. They do not establish quality, resource limits, or product policy.
+1 -1
View File
@@ -32,7 +32,7 @@ Current high-signal boundary:
- docs/status.md owns the exact source/production/hold state.
- docs/todo.md is the only executable work ledger and owns ordering.
- ADR-0005 owns the accepted automatic route model. The Browser route remains
implemented and deployed in release `352c457`; real external-network
implemented and deployed in release `d3ff9e7`; real external-network
and SFU/TURN media evidence remains open. Do not rerun broad or
executable/native test suites unless the current acceptance boundary
specifically requires them.
+6 -4
View File
@@ -6,10 +6,11 @@ This is the current execution index. Git history owns completed timelines; [veri
## Production
- `https://share.bonfire.icu` runs exact `352c4578954c399e56a2578cf3e22f5e0cfc2bad`, release `352c457`, wire `screener-v8`; `6b87732` is the immediate rollback release and `/opt/screener/backups/352c457-precutover-20260823T135843Z` retains its verified environment/unit/LiveKit/SQLite boundary.
- `https://share.bonfire.icu` runs exact `d3ff9e7b7b4a8fe58db700565971aaeda638d2e9`, release `d3ff9e7`, wire `screener-v8`; `352c457` is the immediate rollback release and `/opt/screener/backups/d3ff9e7-precutover-20260823T150618Z` retains its verified environment/unit/LiveKit boundary.
- `screener`, LiveKit, coturn, and nginx are active; local/public health return 200 and all services report `NRestarts=0`. The v8 unit has no writable room `StateDirectory`, the process has no SQLite descriptor, and the old live SQLite path is absent.
- Production admits one Host plus 20 Viewers, applies endpoint capacity `2`, rejects stale v7 Browser and executable-sender wires before room authority, and runs the one-controller exact-candidate route model. Rooms use random free four-digit codes, a 24-hour dormant lease, restart loss, local Host creation preferences, independent Viewer grants, and `open | password | disabled` code entry. LiveKit remains dedicated with `room.auto_create: false`, `max_participants: 21`, global admission `1` publication ingress / `20` subscription egress; selected TURN has `2` logical allocations.
- Production still has access/privacy controls, self-check, diagnostic export, signaling watchdog behavior, 64/128/256 audio choices, and selected-edge transport. These deployment facts do not close the real heterogeneous-network, SFU/TURN media, mobile, or endurance evidence boundaries.
- SFU subscribers reconcile Host screen publications across connect, activation, participant arrival, track publication, and reconnect. A controlled production canary forced two SFU roots; both committed `route-ready`, reached active SFU assignments, and decoded 1920x1080 video. LiveKit runs at `warn`/Pion `error`, and the Screener unit waits boundedly for the same-host LiveKit control listener before startup.
## Current Source
@@ -42,6 +43,7 @@ This is the current execution index. Git history owns completed timelines; [veri
## Current Hold
The Browser route and memory-room slices are implemented and deployed. Later TODO work is
intentionally paused until the owner chooses the next slice; do not start live-media deployment,
route/room rollback, Native/executable work, or broad cleanup without a new instruction.
The Browser route, SFU subscription reconciliation, and memory-room slices are deployed, but room
`9403` exposed a P1 fallback-budget regression: silent direct/selected candidates can consume the
whole operation before an existing SFU publication is reused. Fix and deploy the one-timer staged
deadline invariant before resuming live-media, Native/executable, or broad cleanup work.
+9 -5
View File
@@ -6,11 +6,15 @@ Only items in **Now** are executable after their stated decision gate. A branch
## Now
1. **Switch screen-audio ceilings live.** Keep Opus fixed and serialize 64/128/256 kbps desired profile updates across every current Host, relay, and SFU audio sender. Each endpoint uses fresh `getParameters()`/`setParameters()` readback, ignores stale completions, retains media plus its prior applied value on failure, and applies the latest desired value to future edges. Do not add a remote applied-ack protocol or claim room-wide atomic convergence in this slice.
2. **Switch video codec while paused.** Serialize the change with route mutation, freeze exact current route/media bindings, hold new joins, renegotiate P2P/relay and republish SFU while paused, then verify actual codec plus decoded frames on resume. A binding invalidated by leave or hard route failure exits this proof set and waits for routing to rebuild from the final committed preference after the bounded codec transaction settles. On failure pause again and perform bounded rollback; rollback proves the prior preference was reapplied plus a newly negotiated actual codec and decoded progress, not literal equality with the old actual codec. Zero targets update only the future-edge preference. Reset `automatic` with empty codec preferences. This is a paused bounded-gap transaction; unpaused make-before-break is a later extension.
3. **Diagnose the production H.264/blur report.** Reproduce exact release `769de201f7cc` against the deployed route release on the same Host/browser/driver/game and one wired direct Viewer, changing one variable at a time. Implement a codec-specific repair only if codec/encoder evidence identifies it; otherwise fix the proven capture, sender, BWE, or lifecycle cause.
4. **Gate the strict-NAT guessed-candidate experiment.** Build only the isolated endpoint-dependent NAT emulator and browser harness first. Standard ICE must fail in the owned fixture, the capped candidate set must pass the documented repeatability, deadline, privacy, and cleanup gates in stable Chrome and Firefox, and fallback must remain unchanged. Only then may one manually approved, operator-owned test room run a default-off, kill-switched canary with explicit consent from both endpoints and exact room/edge/generation/budget bounds; failure closes the candidate. Do not ship guessed candidates to ordinary rooms or infer NAT type from IP, UA, carrier, or two STUN observations.
5. **Make Viewer connection progress explicit.** Derive one stable stage from existing access, route, peer/SFU, media, autoplay, and bounded-failure state. Keep the overlay until a composited current-generation frame, show `Play` only for actual autoplay rejection, preserve a still-valid old frame during recovery, and expose retry only through the bounded controller. Measure the existing token-free SFU prewarm and route stages before adding another connection mechanism.
1. **Restore stage budget for SFU fallback.** Room `9403` proved that silent direct recovery can consume the whole child-operation deadline: Host-full bootstrap correctly moved one existing direct child to SFU, but the cellular child's next operation never reached the healthy publication's `sfu:reuse` tuple. Divide the one total deadline equally across the route classes actually present, use the same timer to skip silent candidates at each derived boundary, and prove direct/selected stages cannot starve SFU. Deploy and recheck the exact cellular Viewer before resuming later work.
2. **Switch screen-audio ceilings live.** Keep Opus fixed and serialize 64/128/256 kbps desired profile updates across every current Host, relay, and SFU audio sender. Each endpoint uses fresh `getParameters()`/`setParameters()` readback, ignores stale completions, retains media plus its prior applied value on failure, and applies the latest desired value to future edges. Do not add a remote applied-ack protocol or claim room-wide atomic convergence in this slice.
3. **Switch video codec while paused.** Serialize the change with route mutation, freeze exact current route/media bindings, hold new joins, renegotiate P2P/relay and republish SFU while paused, then verify actual codec plus decoded frames on resume. A binding invalidated by leave or hard route failure exits this proof set and waits for routing to rebuild from the final committed preference after the bounded codec transaction settles. On failure pause again and perform bounded rollback; rollback proves the prior preference was reapplied plus a newly negotiated actual codec and decoded progress, not literal equality with the old actual codec. Zero targets update only the future-edge preference. Reset `automatic` with empty codec preferences. This is a paused bounded-gap transaction; unpaused make-before-break is a later extension.
4. **Diagnose the production H.264/blur report.** Reproduce exact release `769de201f7cc` against the deployed route release on the same Host/browser/driver/game and one wired direct Viewer, changing one variable at a time. Implement a codec-specific repair only if codec/encoder evidence identifies it; otherwise fix the proven capture, sender, BWE, or lifecycle cause.
5. **Diagnose desktop Host background/minimized performance.** On the same machine, dynamic scene, direct Viewer, browser, and fresh share, compare foreground, occluded, minimized, and hidden states while separately verifying actual negotiated VP8/H.264. Correlate capture settings/FPS, frames captured/encoded/sent/decoded, bitrate, `qualityLimitationReason`, encode time, browser/game CPU and GPU, and the existing local-preview pause boundary. Treat browser scheduling/throttling and codec effects as hypotheses until measured. Do not spoof page activity, add keepalive timers, or misuse Wake Lock; if current Chrome/Edge cannot preserve the accepted Host behavior through standard Web APIs, record that capability boundary before considering the packaged/native Host.
6. **Gate the strict-NAT guessed-candidate experiment.** Build only the isolated endpoint-dependent NAT emulator and browser harness first. Standard ICE must fail in the owned fixture, and every guessed candidate must remain inside the existing direct-stage share rather than extending the room operation. The capped set must pass the documented repeatability, privacy, cleanup, and total-deadline gates in stable Chrome and Firefox, while fallback remains unchanged. Only then may one manually approved, operator-owned test room run a default-off, kill-switched canary with explicit consent from both endpoints and exact room/edge/ICE-generation/deadline/aggregate-budget bounds; failure closes the candidate. Do not ship guessed candidates to ordinary rooms or infer NAT type from IP, UA, carrier, or two STUN observations.
7. **Make Viewer connection progress and failure recovery explicit.** Derive one stable stage from existing access, Host presence, route, peer/SFU, media, autoplay, recovery, and bounded-failure state. Keep the overlay until a composited current-generation frame, show `Play` only for actual autoplay rejection, preserve a still-valid old frame during recovery, and expose retry only through the bounded controller. Use one Chinese user-facing vocabulary and map server restart/room loss, Host stop, Host signaling loss with healthy media, Host crash/offline, route exhaustion, SFU/TURN admission, autoplay, and stale-client refresh to distinct actionable presentations without exposing private topology identifiers. A failed mobile page must retain privacy-safe stage/revision/code buckets for diagnostic export instead of collapsing everything into `No usable media route is available`. Measure the existing token-free SFU prewarm and route stages before adding another connection mechanism.
8. **Clarify malformed and unusable room entry without enumeration.** Reject non-four-digit input locally with a direct format message. For a well-formed code that cannot be admitted, stop automatically implying that a password is required; show one neutral result covering an unknown room, disabled code entry, wrong/absent password, or bounded admission failure, and offer an optional password retry plus the invitation-link path. Keep `open` rooms direct and do not expose room existence or code-entry policy through distinct server errors.
9. **Refine recommended quality presets.** Change the default recommended profile to the middle `1080p30` option and add one fully specified 480p preset only after its exact resolution, frame rate, and bitrate ceiling are accepted. Update capture constraints, current/future peer/relay/SFU sender settings, local preference/default selection, readback, and tests together; do not add a label that leaves old sender parameters active.
## Accepted Later Roadmap
+3 -1
View File
@@ -10,10 +10,12 @@ Capacity and routing entries below describe the exact tested source or release o
## Deployment Evidence
- Production runs exact `352c4578954c399e56a2578cf3e22f5e0cfc2bad`, release `352c457`, wire `screener-v8`, from `/opt/screener/releases/352c457`; the immutable runtime archive SHA-256 is `d7feb021821b2af3db5180aed780da67ca582abfb3fe7a8805f6c6d88042bd3c`. `6b87732` is the immediate rollback release, and `/opt/screener/backups/352c457-precutover-20260823T135843Z` contains the verified environment/unit/LiveKit/SQLite boundary.
- Production runs exact `d3ff9e7b7b4a8fe58db700565971aaeda638d2e9`, release `d3ff9e7`, wire `screener-v8`, from `/opt/screener/releases/d3ff9e7`; the immutable runtime archive SHA-256 is `a7c3b7b7cf593f3933a1c194a07370131fd76b3d4c30768b5206e84ccf93b63d`. `352c457` is the immediate rollback release, and `/opt/screener/backups/d3ff9e7-precutover-20260823T150618Z` contains the verified environment/unit/LiveKit boundary.
- The 2026-08-23 postflight found `screener`, LiveKit, coturn, and nginx active with zero restarts; local and public `/healthz` returned 200 and the Screener journal contained no error line. The deployed browser bundle is `assets/index-BVK8yQk2.js`, SHA-256 `388a093734e8dc37a07b859651ea15d4098a8c8f13864ca9d70eb99128a911a3`, matching the locally built artifact and containing one `screener-v8` and zero `screener-v7` literals.
- Production uses endpoint capacity `2`, room admission `20`, random free four-digit memory rooms, `ROOM_LEASE_SECONDS=86400`, a dedicated LiveKit instance with `room.auto_create: false` and `max_participants: 21`, global SFU admission `1` ingress / `20` egress, and selected TURN admission `2`. The v8 service has no StateDirectory or SQLite descriptor; the old live SQLite file/directory is absent and LiveKit listed zero rooms after startup.
- A public control-plane smoke passed site access, four-digit creation, Host v8 authentication, stale-v7 rejection before authority, independent code-only and grant Viewer admission, grant rotation that preserved the code Viewer, signaling challenge, and exact room abandonment. A separate service-restart smoke rejected the old Host token and Viewer grant, atomically created a password-policy replacement room, and abandoned it.
- The first reported SFU failure showed Host screen tracks published and the Viewer active over LiveKit UDP, but the candidate was deleted before a usable route committed. Exact `d3ff9e7` fixed missing subscription reconciliation across pending connect, participant/track arrival, activation, and reconnect; 550 Web tests passed, and a production Host plus three zero-downstream Viewer canary forced one peer and two active 1920x1080 SFU roots. Room `9403` then exposed a separate earlier blocker: repeated cellular attempts never entered LiveKit. With Host peer slots full, bootstrap correctly converted one existing direct child into a healthy Host publication plus one SFU subscriber, but the cellular child's next operation again exhausted in direct/selected work before `sfu:reuse`. This is the active P1 stage-budget regression, not evidence that the publication or LiveKit UDP path is unavailable.
- The first hotfix cutover start raced LiveKit ActiveState before TCP 7880 and produced one supervised Screener restart. A production-only `After/Wants` plus bounded listener-ready `ExecStartPre` was installed; a simultaneous LiveKit/Screener restart then completed with both `NRestarts=0`. Pinned LiveKit info records were confirmed to contain raw PublisherOffer SDP, so production now uses warn/Pion-error. Three archived journal files containing prior raw SDP (90,404,248 bytes total) were deleted; the active journal and the post-hardening SFU canary contain zero `PublisherOffer` record.
- Rollback evidence confirms release `6b87732` and its old environment/runtime imports are readable, every precutover backup checksum passes, the backed-up database remains schema v3/integrity-ok with six rooms, and the current and rollback release regular-file sets share zero inodes.
- Share audio presets, local-only diagnostics, self-check, diagnostic export, signaling-partition recovery, and selected-edge transport remain deployed surfaces. Their real-route media behavior and target-browser field availability remain unverified.
+2
View File
@@ -372,6 +372,8 @@ Token-free SFU standby prewarm 保持可选且不创建 participant、publicatio
`reconcileRoom` 在没有 pending operation 时先删除 disconnected childless participant,再找一个 waiting Viewer,或 parent offline/effective-capacity 不足、exact edge 已失效的 child。Candidate planning 拒绝 stale session、不可 source-reachable、成环或没有稳态/overlap/bounded-gap producer slot 的 tuple;server resource admission 在执行当前 cursor 时单独检查。由 failed edge 启动时只为当前 fact version 把它的 `parent + transport` tuple记为已尝试;新的外部事实允许重新评估。对剩余 upstream 依次比较结果深度、剩余 sender slots、稳定加入顺序与 peer identity,先建立其他 direct/STUN,再建立 selected TURN 与 SFU 的有界顺序;同一 parent 的不同 transport 仍是不同 tuple。每次只 reserve/prepare cursor 指向的 candidate。WebRTC 负责 ICE/DTLS/consent;exact candidate child 解码出第一帧新画面后发送一次 ready,controller 先同步 commit typed admission,再原子提交 graph。失败释放 candidate、广播严格更新的 rollback revision 并推进 cursor,operation 总 deadline 不重置。耗尽时 hard-invalid、confirmed-departed 或 deterministic overflow edge 被退休并进入明确 waiting;healthy bootstrap edge 保留。无环和房间 admission 已约束最大深度,不增加独立 depth hard cap。
一个 operation 仍只有一个 timer 和一个总 deadline,但 timer 的下一次 wake 从当前 candidate list 派生。Controller 提取实际存在的 direct、selected-TURN、SFU 阶段并等分总窗口;显式失败可以在阶段内继续尝试其他 tuple,silent candidate 到达该阶段 wake boundary 后释放 reservation、广播 rollback 并跳到下一类。只有 direct 时仍使用完整 deadline;direct+SFU 时各用一半;三类同时存在时各用三分之一。这样不引入每-candidate magic milliseconds,也不允许一条 silent direct 把 SFU suffix 饿死。
首次需要 SFU 但 Host 已满时不建立另一条 rescue 流程。若 affected child 已是 Host direct child,就用它;否则用稳定加入顺序中最新的 connected Host direct child。`reconcileRoom` 为该 child 建立普通 SFU candidate,创建唯一 publication,保持其 subtree,并在该 child 首帧后提交。Host publication 取代释放的 peer slot,随后同一 fixed-point scan 再处理原 waiting/failed child。若旧 Host peer sender 在首帧前仍保留,candidate 必须持有同一 `TransitionOverlapSlot`。没有 overlap 时先 preflight server resource,再退休 exact producer;operation 保持原总 deadline、按新 graph 重算未尝试 suffix,并为被切断的 healthy direct edge追加一个最后 restore tuple。Forward candidate 与 restore 都失败才进入 waiting,不得产生第四份 endpoint copy或另一套恢复状态。
所有非服务器端点共享一个 server-authoritative steady outbound media-copy cap `C`:唯一配置 `ENDPOINT_MEDIA_COPY_CAPACITY` 默认 `2`,只接受 `1`、`2` 或 `3`,不兼容旧键。服务端将 endpoint 当前广告的 `0..C` 可用值 clamp 为唯一 `effectiveDownstreamCapacity`;该值只表示当前可用 sender slots,不建立角色、浏览器、UA、设备或页面可见性 tier。peer child 或 Host publication 各占一份,upstream receive 免费,committed selected TURN 只替换同一份 edge transport;SFU subscriber egress 与 TURN allocation 使用独立 server admission。Browser `screener-v8` 的 `authenticated.endpointMediaCopyCapacity` 把部署 `C` 交给 Host 与 Viewer;两端用该值校验稳态 assignment,并让 active+pending 的物理 copy 只在 controller 已授权 overlap 时达到 `min(C + 1, 3)`。代码通过一个 shared accounting guard 执行,不重复 magic number;旧 wire 与 Native executable 在 authority 前拒绝,不保留兼容路径。Selected TURN 使用显式无默认值的正 safe-integer deployment capacity;同一个进程内 ledger 对每条 exact peer-selected edge 和 Host-SFU ingress 在 credential 签发前 reserve `1`,并让 reserved、committed、draining authorization 持续计费至 exact logical release。`C=3` 满载时先收到 active bounded-gap retirement再收到 prepare;任何会产生第四份 endpoint media copy 的 token/grant都不得签发。同一 edge 改用 TURN 不增加稳态 endpoint copy,但仍需独立 TURN admission。
+1
View File
@@ -31,6 +31,7 @@ SFU/TURN 的 publication、subscription、transport、server-resource accounting
- 路径隔离双表示:direct/peer 继续使用每条 PeerConnection 独立的 stock WebRTC GCC,不把 viewer 反馈合成全房目标。每条 SFU path 从 `HIGH` ceiling 开始;优先使用 pinned LiveKit 的标准两层 screen-share simulcast,让内建 per-subscriber BWE 独立降层和恢复,Screener 不实现媒体层 selector。应用只诊断实际接收层、硬件编码、游戏性能、资源和上传;SFU 自动降层不改变拓扑。`LOW` 停编只是资源优化,active representation/layer 硬上限为二且不随 viewer 增长。当前 Web P2P simulcast 与 Web/LiveKit SVC 不能证明跨 direct/peer 共享编码或可移植硬件路径;全部标准路径失败后才考虑 custom/native dual encode。
- 路由失效输入:应用不推断 parent-wide 质量,也不计算综合分数。当前 authenticated child 只在 PeerConnection `failed/closed`,或非权威暂停时现有 periodic stats 在统一 deadline 内没有新 decoded frame 时,使自己的 exact edge 失效并唤醒同一个 reconcile。decoded progress 只保存 route-keyed `lastDecodedAt`,route/connection变化或新帧重置、暂停抑制,同一 edge只报告一次,不增加轮询器。该 operation 把失败的 `parent + transport` tuple 在当前 fact version 直接视为已尝试,先选择其他 direct parent;同一 parent 只有换用另一 transport 时才可在本次继续,新的外部事实才允许原 tuple 重新进入。Track availability、低码率、低 FPS、模糊、sender/viewer stats 与 SFU layer只用于stock WebRTC/LiveKit恢复和诊断。relay自身ingress失效时,它作为child换父并保留subtree;disconnect或effective downstream capacity降低只让direct/overflow children逐个走同一事务。
- 断线恢复:临时网络变化先交给 WebRTC/LiveKit 的 ICE、consent、重连和流状态。Direct edge 可在 exact current parent/child sessions 内采用 framework rebuild 的新 connection ID而不改变topology;selected transport随session变化统一retire。应用只在上述exact edge失效后运行一次有界reconcile;任一步成功即停止。候选耗尽时hard-invalid、confirmed-departed或overflow edge被退休并进入明确waiting,healthy bootstrap edge保留。所有异步结果受authenticated session、share generation、base/pending route revision与candidate connection identity约束,未受影响分支保持sticky。权威暂停abort未提交candidate、保留active graph、抑制stall,并在resume后重新reconcile。
- 同一 operation 的一个总 deadline 按实际存在的 `direct | selected-turn | sfu` 语义阶段等分,不配置每个 candidate 的 magic timeout。阶段内的硬失败继续推进 cursor;silent ICE 到达阶段边界时跳过剩余同类 tuple,必须把后续阶段的派生份额留给 selected transport 与 SFU create/reuse。任何 strict-NAT guessed candidate 以后也只能处于 direct 阶段,不能延长总 deadline。
- 可观测性:按目标 RTP transport 与不透明连接/ICE 代次关联 selected path、`iceGatheringState`/ICE/connection state、事件推导的建立时长、generation-baselined pair-change delta,以及 RTT、RTP 丢包率、抖动、码率、帧率、编码耗时和受限原因。候选与错误的 wire、日志和导出只使用[低服务器成本媒体路由](./research/low-server-media-routes.md)的脱敏字段和 code bucket,禁止 raw SDP、candidate、地址、URL、`errorText` 或网络标识。只有创建该 `RTCPeerConnection` 的本机可在折叠详情中显示浏览器对 selected local/remote candidate 实际暴露的地址与端口;该值不上传、不持久化、不用于地理/NAT 推断,缺失或被隐私保护时显示未知。
- 渐进式诊断:分享者和观看者默认只看到房间、画面、核心状态和常用控制;Host 的 Viewer 卡片只把本机连接或当前接收证据称为“已连接”,其余状态保持中性的“线路分配中”或具体连接状态,不把 active assignment 当作连接成功。展开连接详情也只在当前媒体证据成立后显示媒体方式:`P2P` 使用绿色与直观图标,`SFU` 使用黄色的 `SFU fallback`;实际 relay 时另标 `TURN`,只有本地 relay protocol 可见时才附加 `/UDP` 等实际值。尚未连接时不预告 P2P、SFU 或未知 ICE,只保留中性状态。Host/Peer 上下游仍只在独立的权威拓扑视图中展示。传输协议、候选路径、RTT、视频/音频 RTP 区间丢包率、实际编解码器、音频区间码率/jitter 和视频编码耗时等技术详情由一个明确的“显示连接详情”开关展开。远端 Viewer evidence 的详情 shell 在同一 viewer/parent/connection/route identity 内保持稳定;非空字段从各自最后一次实际观测起精确保留五秒,`connecting`/`disconnected` 只降为非 fresh,`failed`/`closed`、identity/session/share/route 变化或非单调 sequence 则清空/换代。非 fresh evidence 不得继续证明“已连接”或被其他 current-state 消费者使用。音频 codec MIME、RTP clock、channels 与 fmtp 只能来自当前 RTP 对象引用的 `RTCCodecStats`,缺失即未知;`opus/48000/2` 不得写成 48 kHz 音源或实际立体声。隐藏诊断不能隐藏失败原因、音频缺失、持续画质受限或需要用户操作的告警。
- 站点访问:生产部署必须配置独立的 `SITE_ACCESS_PASSWORD` 控制建房、Host 信令角色和所有 code-only Viewer 入口;只有本地开发/测试可显式关闭。非空值接受 8 至 128 个可见 ASCII bytes,不能复用 TURN、LiveKit、TLS 或其他部署 secret。`POST /api/site-access` 验证成功签发 12 小时、无服务端会话状态的 HttpOnly、`SameSite=Strict` site-access cookie。有效 room-scoped fragment grant 可绕过站点 gate 直接进入对应房间;其余 `/r/{code}` 和 `/join` 必须先持有效站点 cookie,之后才检查 `open | password | disabled` code-only policy。失败统一且不得泄露房间是否存在或 policy;建房 HTTP 只认该 cookie。已移除的旧环境变量、endpoint 和 cookie 不保留兼容。不引入账号、JWT、服务端 session Map 或通用 logout。
+1 -1
View File
@@ -1404,7 +1404,7 @@ export class HybridMediaRouter {
);
}
this.requestPump(roomId);
}, Math.max(0, operation.deadlineAtMs - this.now()));
}, Math.max(0, operation.wakeAtMs - this.now()));
timer.unref();
room.deadlineTimer = timer;
}
+84
View File
@@ -129,6 +129,7 @@ export interface OperationSnapshot {
candidates: readonly CandidatePlan[];
cursor: number;
deadlineAtMs: number;
wakeAtMs: number;
current?: { tuple: CandidateTuple; revision: number; connectionId: string };
}
@@ -677,6 +678,9 @@ export class RoomRouteController<Resource = unknown> {
this.blockAndClear(operation, !factsChanged, released);
return { ...result, exhausted: !factsChanged, expired: true };
}
if (this.advanceExpiredCandidateStage(operation, nowMs, released)) {
result.expired = true;
}
if (this.participants.get(operation.childPeerId)?.sessionId !== operation.childSessionId) {
if (operation.current) released.push(...reservationResources(operation.current.reservation));
if (operation.current) this.advanceActiveRevision(operation);
@@ -1084,10 +1088,72 @@ export class RoomRouteController<Resource = unknown> {
baseRevision: operation.baseRevision, factVersion: operation.builtAtFactVersion,
candidates: operation.candidates.map(cloneCandidatePlan),
cursor: operation.cursor, deadlineAtMs: operation.deadlineAtMs,
wakeAtMs: this.operationWakeAt(operation),
current: operation.current ? { tuple: { ...operation.current.tuple }, revision: operation.current.revision,
connectionId: operation.current.connectionId } : undefined };
}
private advanceExpiredCandidateStage(
operation: ChildOperation<Resource>,
nowMs: number,
released: Resource[],
): boolean {
const plan = operation.current
? { tuple: operation.current.tuple }
: operation.candidates[operation.cursor];
if (!plan) {
return false;
}
const stage = candidateStage(plan.tuple);
const stageDeadlineAtMs = this.stageDeadlineAt(operation, stage);
if (
stageDeadlineAtMs >= operation.deadlineAtMs ||
nowMs < stageDeadlineAtMs
) {
return false;
}
if (operation.current) {
released.push(...reservationResources(operation.current.reservation));
this.advanceActiveRevision(operation);
operation.current = undefined;
operation.cursor += 1;
}
while (
operation.cursor < operation.candidates.length &&
candidateStage(operation.candidates[operation.cursor]!.tuple) === stage
) {
operation.cursor += 1;
}
return true;
}
private operationWakeAt(operation: ChildOperation<Resource>): number {
const tuple =
operation.current?.tuple ?? operation.candidates[operation.cursor]?.tuple;
return tuple
? this.stageDeadlineAt(operation, candidateStage(tuple))
: operation.deadlineAtMs;
}
private stageDeadlineAt(
operation: ChildOperation<Resource>,
stage: CandidateStage,
): number {
const stages = candidateStages(operation.candidates);
const index = stages.indexOf(stage);
if (index === -1 || index === stages.length - 1) {
return operation.deadlineAtMs;
}
const startedAtMs =
operation.deadlineAtMs - this.options.operationTimeoutMs;
return (
startedAtMs +
Math.floor(
(this.options.operationTimeoutMs * (index + 1)) / stages.length,
)
);
}
private guardMatches(guard: CandidateGuard, operation: ChildOperation<Resource>, attempt: Attempt<Resource>): boolean {
return guard.childPeerId === operation.childPeerId && guard.childSessionId === operation.childSessionId &&
guard.revision === attempt.revision && guard.connectionId === attempt.connectionId;
@@ -1299,6 +1365,24 @@ function tupleKey(tuple: CandidateTuple): string {
return tuple.kind === "peer" ? `peer:${tuple.parentPeerId}:${tuple.transport}` : `sfu:${tuple.publication}:${tuple.ingress}`;
}
type CandidateStage = "direct" | "selected-turn" | "sfu";
function candidateStage(tuple: CandidateTuple): CandidateStage {
if (tuple.kind === "sfu") {
return "sfu";
}
return tuple.transport;
}
function candidateStages(candidates: readonly CandidatePlan[]): CandidateStage[] {
const stages = new Set(
candidates.map((candidate) => candidateStage(candidate.tuple)),
);
return (["direct", "selected-turn", "sfu"] as const).filter((stage) =>
stages.has(stage),
);
}
function edgeTupleKey<Resource>(edge: CommittedEdge<Resource>): string {
return edge.kind === "peer" ? `peer:${edge.parentPeerId}:${edge.transport}` : "sfu:reuse:existing";
}
+39
View File
@@ -97,6 +97,7 @@ function harness(
endpointMediaCopyCapacity: 1 | 2 | 3,
withSfu = false,
withSelectedTurn = false,
prepareTimeoutMs?: number,
) {
const store = createStore();
const sent = new Map<string, ServerMessage[]>();
@@ -117,6 +118,7 @@ function harness(
url: "wss://sfu.example.test",
admission,
roomControl,
...(prepareTimeoutMs ? { prepareTimeoutMs } : {}),
tokenIssuer: {
async issueToken({ peerId }) {
return `token-${peerId}`;
@@ -158,6 +160,43 @@ function harness(
}
describe("HybridMediaRouter v7 runtime", () => {
it("wakes at the derived direct boundary and prepares SFU automatically", async () => {
vi.useFakeTimers();
vi.setSystemTime(0);
const { store, sent, router } = harness(2, true, false, 300);
try {
const room = await store.createRoom();
const host = connectHost(store, room);
complete(router, host);
const viewer = connectViewer(store, room, "staged-deadline");
complete(router, viewer);
await vi.waitFor(() =>
expect(preparedFor(sent, viewer.sessionId)?.candidate.transport).toBe(
"direct",
),
);
await vi.advanceTimersByTimeAsync(150);
await vi.waitFor(() =>
expect(preparedFor(sent, viewer.sessionId)?.candidate.transport).toBe(
"sfu",
),
);
expect(
sent
.get(viewer.sessionId)
?.some(
(message) =>
message.type === "error" &&
message.message === "No usable media route is available",
),
).toBe(false);
} finally {
await router.close();
vi.useRealTimers();
}
});
it("orders exact prepare, commits only child proof, and rolls back above P", async () => {
const { store, sent, router } = harness(2);
const room = await store.createRoom();
+272
View File
@@ -148,6 +148,113 @@ describe("RoomRouteController", () => {
},
);
it("keeps healthy branches while 18 silent Viewers fall back serially", () => {
const routes = controller(2, {
selectedTurnEnabled: true,
sfuEnabled: true,
});
addViewer(routes, B, 2);
routes.hydrateHostPublication(
"publication_generation",
"publication_resource",
);
routes.hydrateEdge(B, {
kind: "sfu",
publicationGeneration: "publication_generation",
transport: "sfu",
connectionId: "b_sfu",
usable: true,
physicalActive: true,
resource: "b_subscription",
});
addViewer(routes, C, 0);
routes.hydrateEdge(C, peerEdge(HOST, "c_from_host"));
const waiting = Array.from(
{ length: 18 },
(_, index) => `silent_${String(index).padStart(2, "0")}_12345678`,
);
waiting.forEach((peerId) => addViewer(routes, peerId, 0));
let nowMs = 0;
waiting.forEach((peerId, index) => {
const directOperation = routes.reconcile(nowMs).operation!;
expect(directOperation.childPeerId).toBe(peerId);
beginCandidate(routes, {
nowMs: nowMs + 1,
connectionId: `${peerId}_silent_direct`,
reservation: { kind: "direct" },
});
routes.operationExpired(directOperation.wakeAtMs);
const selectedOperation = routes.snapshot().operation!;
expect(
selectedOperation.candidates[selectedOperation.cursor]!.tuple,
).toMatchObject({ kind: "peer", transport: "selected-turn" });
beginCandidate(routes, {
nowMs: selectedOperation.wakeAtMs - 1,
connectionId: `${peerId}_silent_turn`,
reservation: {
kind: "selected-turn",
edge: `${peerId}_turn_resource`,
},
});
const selectedExpired = routes.operationExpired(
selectedOperation.wakeAtMs,
);
expect(selectedExpired.released).toEqual([
`${peerId}_turn_resource`,
]);
const sfuOperation = routes.snapshot().operation!;
expect(sfuOperation.candidates[sfuOperation.cursor]!.tuple).toEqual({
kind: "sfu",
publication: "reuse",
ingress: "existing",
});
const prepared = beginCandidate(routes, {
nowMs: selectedOperation.wakeAtMs + 1,
connectionId: `${peerId}_sfu`,
reservation: {
kind: "sfu-reuse",
edge: `${peerId}_subscription`,
},
}).operation!;
expect(
routes.candidateReady(
{
childPeerId: peerId,
childSessionId: `${peerId}_session`,
revision: prepared.current!.revision,
connectionId: `${peerId}_sfu`,
},
selectedOperation.wakeAtMs + 2,
).accepted,
).toBe(true);
nowMs = selectedOperation.wakeAtMs + 3;
expect(routes.snapshot().upstreamByViewer.get(B)).toMatchObject({
kind: "sfu",
connectionId: "b_sfu",
});
expect(routes.snapshot().upstreamByViewer.get(C)).toMatchObject({
kind: "peer",
parentPeerId: HOST,
connectionId: "c_from_host",
});
expect(routes.snapshot().upstreamByViewer.get(peerId)).toMatchObject({
kind: "sfu",
usable: true,
});
expect(routes.snapshot().upstreamByViewer.size).toBe(index + 3);
});
expect(routes.snapshot().upstreamByViewer.size).toBe(20);
expect(routes.snapshot().hostPublication).toMatchObject({
generation: "publication_generation",
usable: true,
});
});
it.each([1, 2, 3] as const)(
"builds a deterministic acyclic route without a depth cap at C=%i",
(capacity) => {
@@ -333,6 +440,171 @@ describe("RoomRouteController", () => {
]);
});
it("reserves one derived deadline stage for direct, selected TURN, and SFU", () => {
const routes = controller(2, {
selectedTurnEnabled: true,
sfuEnabled: true,
});
addViewer(routes, B, 2);
routes.hydrateEdge(B, peerEdge(HOST, "b_from_host"));
addViewer(routes, A, 0);
const operation = routes.reconcile(0).operation!;
expect(operation.deadlineAtMs).toBe(10_000);
expect(operation.wakeAtMs).toBe(Math.floor(10_000 / 3));
expect(operation.candidates.map((candidate) => candidate.tuple)).toEqual(
expect.arrayContaining([
{ kind: "peer", parentPeerId: B, transport: "direct" },
{ kind: "peer", parentPeerId: B, transport: "selected-turn" },
{ kind: "sfu", publication: "create", ingress: "direct" },
]),
);
beginCandidate(routes, {
nowMs: 1,
connectionId: "silent_direct",
reservation: { kind: "direct" },
});
const directExpired = routes.operationExpired(operation.wakeAtMs);
expect(directExpired).toMatchObject({ accepted: true });
expect(directExpired.exhausted).toBeUndefined();
expect(routes.snapshot().operation).toMatchObject({
deadlineAtMs: 10_000,
wakeAtMs: Math.floor((10_000 * 2) / 3),
current: undefined,
});
expect(
routes.snapshot().operation!.candidates[
routes.snapshot().operation!.cursor
]!.tuple,
).toMatchObject({ kind: "peer", transport: "selected-turn" });
beginCandidate(routes, {
nowMs: operation.wakeAtMs + 1,
connectionId: "silent_selected_turn",
reservation: { kind: "selected-turn", edge: "turn_edge" },
});
const selectedExpired = routes.operationExpired(
Math.floor((10_000 * 2) / 3),
);
expect(selectedExpired).toMatchObject({ accepted: true });
expect(selectedExpired.released).toEqual(["turn_edge"]);
expect(
routes.snapshot().operation!.candidates[
routes.snapshot().operation!.cursor
]!.tuple,
).toMatchObject({ kind: "sfu" });
expect(routes.snapshot().operation!.wakeAtMs).toBe(10_000);
const prepared = beginCandidate(routes, {
nowMs: 7_000,
connectionId: "sfu_subscription",
publicationGeneration: "publication_generation",
publicationConnectionId: "publication_connection",
reservation: {
kind: "sfu-create",
edge: "sfu_edge",
publication: "sfu_publication",
},
}).operation!;
expect(
routes.candidateReady(
{
childPeerId: A,
childSessionId: `${A}_session`,
revision: prepared.current!.revision,
connectionId: "sfu_subscription",
},
9_000,
).accepted,
).toBe(true);
expect(routes.snapshot().upstreamByViewer.get(A)).toMatchObject({
kind: "sfu",
usable: true,
});
});
it("reaches a healthy SFU reuse stage after silent peer transports", () => {
const routes = controller(2, {
selectedTurnEnabled: true,
sfuEnabled: true,
});
addViewer(routes, B, 2);
routes.hydrateHostPublication(
"publication_generation",
"publication_resource",
);
routes.hydrateEdge(B, {
kind: "sfu",
publicationGeneration: "publication_generation",
transport: "sfu",
connectionId: "b_sfu",
usable: true,
physicalActive: true,
resource: "b_subscription",
});
addViewer(routes, C, 0);
routes.hydrateEdge(C, peerEdge(HOST, "c_from_host"));
addViewer(routes, A, 0);
const operation = routes.reconcile(0).operation!;
expect(operation.candidates.map((candidate) => candidate.tuple)).toEqual(
expect.arrayContaining([
{ kind: "peer", parentPeerId: B, transport: "direct" },
{ kind: "peer", parentPeerId: B, transport: "selected-turn" },
{ kind: "sfu", publication: "reuse", ingress: "existing" },
]),
);
beginCandidate(routes, {
nowMs: 1,
connectionId: "silent_direct",
reservation: { kind: "direct" },
});
routes.operationExpired(operation.wakeAtMs);
beginCandidate(routes, {
nowMs: operation.wakeAtMs + 1,
connectionId: "silent_selected",
reservation: { kind: "selected-turn", edge: "turn_edge" },
});
routes.operationExpired(Math.floor((10_000 * 2) / 3));
expect(
routes.snapshot().operation!.candidates[
routes.snapshot().operation!.cursor
]!.tuple,
).toEqual({
kind: "sfu",
publication: "reuse",
ingress: "existing",
});
const prepared = beginCandidate(routes, {
nowMs: 7_000,
connectionId: "a_sfu_reuse",
reservation: { kind: "sfu-reuse", edge: "a_subscription" },
}).operation!;
expect(
routes.candidateReady(
{
childPeerId: A,
childSessionId: `${A}_session`,
revision: prepared.current!.revision,
connectionId: "a_sfu_reuse",
},
8_000,
).accepted,
).toBe(true);
expect(routes.snapshot().hostPublication).toMatchObject({
generation: "publication_generation",
usable: true,
});
expect(routes.snapshot().upstreamByViewer.get(A)).toMatchObject({
kind: "sfu",
publicationGeneration: "publication_generation",
usable: true,
});
});
it("retains another exact failed edge while one child operation is busy", () => {
const routes = controller(3);
addViewer(routes, A, 1);