diff --git a/docs/README.md b/docs/README.md index ca462fd..a3f5b2f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,7 +11,7 @@ - [Low-server-cost media routes](./research/low-server-media-routes.md): SFU-root/peer distribution, optional selected-edge TURN, hop-level accounting, privacy-safe ICE evidence, and the bounded transport canary. - [Advanced peer distribution](./research/advanced-peer-distribution.md): multi-tree/SVC, encoded-object relay, native RTP forwarding, FEC/network-coding, MoQ, and measurable go/no-go gates. - [Native shared-encode sender](./research/native-shared-encode-sender.md): consolidated Draft #16/#18/#22/#23/#25/#28 ladder, unclassified product-gate failure, staged revalidation, and stop line. -- [Realtime quality adaptation](./research/realtime-quality-adaptation.md): correlated capture/send/receive diagnosis, production generation-rebuild gate, demand-driven dual representations, and bounded simulcast/Dynacast/SVC checks. +- [Realtime quality adaptation](./research/realtime-quality-adaptation.md): correlated capture/send/receive diagnosis, production generation-rebuild gate, path-isolated dual representations, and bounded simulcast/Dynacast/SVC checks. - [Browser screen-audio quality](./research/browser-screen-audio-quality.md): capture/source compatibility, standard readback limits, Opus negotiation boundaries, and the retained no-control decision. - [Agent context governance research](./research/agent-context-governance.md): official Codex, Claude Code, Hermes Agent, and GitHub practices adopted by this repository. - [ADR-0001](./adr/0001-p2p-first-media-topology.md): accepted P2P-first media topology and its consequences. @@ -19,6 +19,6 @@ - [ADR-0004](./adr/0004-peer-assisted-media-experiment.md): proposed, experiment-only standard-WebRTC peer-assisted spike with hard fanout and failure gates. - [ADR-0005](./adr/0005-automatic-hybrid-media-routing.md): accepted direct/peer UDP -> SFU-root -> optional exceptional-edge TURN direction, with the current default-off controller and migration gates. - [ADR-0006](./adr/0006-fixed-high-native-sender-canary.md): proposed fixed-`HIGH` native canary, current no-go result, and staged revalidation boundary. -- [ADR-0007](./adr/0007-demand-driven-dual-representation-quality.md): accepted demand-driven `HIGH + at most one on-demand LOW` quality policy, path-level hysteresis, evidence contract, and SVC boundary. +- [ADR-0007](./adr/0007-demand-driven-dual-representation-quality.md): accepted path-isolated `HIGH + at most one LOW` policy, optional idle-layer stop, evidence contract, and SVC boundary. Documentation and project memory are part of the product source of truth. Update them in the same change that alters the corresponding requirement or architecture. diff --git a/docs/adr/0004-peer-assisted-media-experiment.md b/docs/adr/0004-peer-assisted-media-experiment.md index 9502d5f..208d407 100644 --- a/docs/adr/0004-peer-assisted-media-experiment.md +++ b/docs/adr/0004-peer-assisted-media-experiment.md @@ -205,9 +205,9 @@ Negative: whole-room SFU history. ADR-0005 and merged PR #17 own the default-off automatic optional-SFU fallback; this ADR does not accept that route for production. -- ADR-0007 owns demand-driven per-path quality representations. Its accepted - product target does not add SVC, multiple representations, or adaptive - switching to this bounded browser-relay experiment. +- ADR-0007 owns path-isolated quality representations and topology eligibility. + Its accepted product target does not add SVC, multiple representations, or + adaptive switching to this bounded browser-relay experiment. - If a later ADR accepts peer-assisted media, it must state exactly which parts of ADR-0001 and ADR-0005 it supersedes. diff --git a/docs/adr/0006-fixed-high-native-sender-canary.md b/docs/adr/0006-fixed-high-native-sender-canary.md index 7276e1f..2256993 100644 --- a/docs/adr/0006-fixed-high-native-sender-canary.md +++ b/docs/adr/0006-fixed-high-native-sender-canary.md @@ -63,8 +63,8 @@ expand the feature to make the diagnostic gate pass. ADR-0006 records a bounded candidate and its failed first product gate, not an accepted architecture or shipped sender. The deployed Web sender and ordinary -viewer remain unchanged. ADR-0007 continues to own demand-driven -`HIGH`/on-demand `LOW`; this canary cannot advance or replace it. +viewer remain unchanged. ADR-0007 continues to own path-isolated +`HIGH + at most one LOW`; this canary cannot advance or replace it. ## References diff --git a/docs/adr/0007-demand-driven-dual-representation-quality.md b/docs/adr/0007-demand-driven-dual-representation-quality.md index 0a42364..746510d 100644 --- a/docs/adr/0007-demand-driven-dual-representation-quality.md +++ b/docs/adr/0007-demand-driven-dual-representation-quality.md @@ -1,4 +1,4 @@ -# ADR-0007: Demand-Driven Dual-Representation Quality +# ADR-0007: Path-Isolated Dual-Representation Quality - Status: Accepted - Staged Implementation - Date: 2026-08-19 @@ -21,34 +21,42 @@ relay/SFU. Per-viewer encoders would violate the game's performance budget. ## Decision -Use demand-driven dual representations: +Use path-isolated dual representations: -1. A healthy room runs one shared `HIGH` representation. -2. Every viewing path starts in `HIGH`. It moves to `FALLBACK` only after - multiple consecutive sampling windows show that the path cannot sustain - `HIGH`, using correlated sender and viewer evidence. -3. The first verified fallback path must start exactly one shared `LOW` - representation. Every verified weak path moves to that same representation. - Healthy paths remain on `HIGH`. -4. A fallback path returns to `HIGH` only after a longer, independently defined - stable-recovery window. When no path needs `LOW`, deactivate the - representation and verify its bytes, frames, and resource cost stop. -5. The representation count is a hard `HIGH + at most one on-demand LOW <= 2`; - it never grows with viewer count. +1. Direct/peer paths keep their requested `HIGH` target and independent stock + WebRTC congestion control. No viewer feedback changes another path's target. +2. Each SFU path starts with a `HIGH` ceiling. The preferred implementation + publishes one shared `HIGH+LOW` pair and lets LiveKit BWE independently + choose and recover the forwarded layer for each subscriber. If that gate + passes, Screener does not add a media-layer selector. +3. Correlated sender/viewer evidence and asymmetric windows classify topology + eligibility and diagnostics only. An observed downshift is `suspect`; a + confirmed `FALLBACK` endpoint cannot remain a parent. These application + states do not command ordinary built-in layer changes. +4. Explicit subscriber quality or sender layer activation is considered only + if the built-in candidate fails its bounded gate. Deactivating an unused + `LOW` is a resource optimization, not an acceptance requirement. +5. The active representation/layer count is a hard `HIGH + at most one LOW <= + 2`; it never grows with viewer count. -Starting `LOW` is conditional on a qualified hardware/power-efficient media -path and measured spare game-performance budget. If that path is absent or the -additional representation causes unacceptable CPU/GPU/game load, fail closed for that -attempt: preserve `HIGH` and show the weak path an explicit degraded/unavailable -state. This is exceptional damage containment, not permission to ignore a weak -path indefinitely. A supported sender cohort that cannot reliably start `LOW` -on demand fails automatic-quality-control acceptance; optimize its encode path -or mark that cohort unsupported. Never protect a weak path by reducing `HIGH`. +Making `LOW` available is conditional on positive hardware-encoder evidence and +measured spare game-performance and upload budget. A permanently active +two-representation path is acceptable when that gate passes; stopping unused +`LOW` remains a later optimization. If the hardware path is absent or the +additional representation causes unacceptable CPU/GPU/game/upload cost, fail +closed for that attempt: preserve `HIGH` and show the weak path an explicit +degraded/unavailable state. This is exceptional damage containment, not +permission to ignore a weak path indefinitely. A supported sender cohort that +cannot reliably provide the one shared `LOW`, either always-on or on demand, +fails quality acceptance; optimize its encode path or mark +that cohort unsupported. Never protect a weak path by reducing `HIGH`. -The two path states use explicit predicates and asymmetric entry/exit windows. -There is no weighted score, device ranking, machine-learning controller, or -continuous room-wide optimizer. Exact thresholds remain implementation inputs -until the controlled quality matrix establishes them. +The durable topology states use explicit predicates and asymmetric +entry/recovery windows; `suspect` is the sampling interval before a transition, +not a third media mode. There is no weighted score, device ranking, +machine-learning controller, custom media selector, or continuous room-wide +optimizer. Exact thresholds remain implementation inputs until the controlled +quality matrix establishes them. The topology and representation budgets are independent invariants. The host still has at most two downstream media edges. A native relay forwards selected @@ -56,10 +64,23 @@ encoded packets without decoding or re-encoding. Future dual-tree or striped delivery may reduce host upload from about two full copies toward one copy plus necessary redundancy, but it does not block this decision. +The application must not stably retain a confirmed `FALLBACK` endpoint as a +parent. A later planned or explicit `LOW` fallback is leaf-only: complete a +generation-guarded child evacuation before changing its quality, and preserve +the prior assignment and quality if evacuation fails or becomes stale. An SFU +may make an unannounced congestion-protection downshift before the application +can react. Treat that observation as `suspect`, not confirmed `FALLBACK`; if +correlated evidence confirms it across the bounded entry window, evacuate its +children and set downstream capacity to zero. This cannot promise packet-level +preemption, so the root-with-children gate must bound temporary descendant +impact before default enablement. Re-advertising relay capacity still needs the +longer recovery window and a cooldown. A viewer's +advisory request cannot trigger this sequence by itself. + ## Evidence Contract -Before implementing automatic switching, correlate the same time interval and -stream generation across three stages: +Before enabling quality-aware topology changes, correlate the same time +interval and stream generation across three stages: | Stage | Required evidence | | --- | --- | @@ -78,14 +99,14 @@ Interpretation is deliberately ordered: | Inbound metrics healthy, image still blurry | insufficient bitrate/quantization, negotiated codec/profile/layer, or display scaling problem | WebRTC stats are sampled dictionaries and many useful counters are cumulative. -The controller must use adjacent, non-overlapping deltas and rebase on a missing +The evidence sampler must use adjacent, non-overlapping deltas and rebase on a missing field, stream/stat ID change, or counter reset. Missing values remain unknown; they are not zeros. Implementation is staged. First correlate A and B locally in one sampling tick, with an explicit interval, media/stat identity, and valid deltas. Only after that probe is trustworthy may the product add a minimal authenticated C report -for the few receive/decode signals required by the two-state predicate. The B/C +for the few receive/decode signals required by the topology predicate. The B/C correlation uses normalized fields derived from negotiated parameters/stats and actual decode behavior; it must never upload raw SDP, raw stats, candidate addresses, or raw device/network identifiers. Opaque server-issued path and @@ -94,7 +115,7 @@ Do not build a general telemetry schema. A viewer may request `LOW`, but the request is advisory. It must be carried on an authenticated, current room/path session and be rate-limited and deduplicated. -The sender/controller accepts it only when viewer receive/decode behavior and +The topology classifier accepts it only when viewer receive/decode behavior and derived negotiated codec/profile/parameters (including applicable `scalabilityMode`) agree with sender transport/GCC and encode evidence. A request alone cannot lower quality or start a representation. @@ -110,7 +131,8 @@ Inspect standard capabilities in order and stop at the first accepted path before custom dual-representation media. Static API or pinned-protocol incompatibilities can close a candidate without a browser run; runtime media gates start only after the A+B/C evidence contract is trustworthy. These -results do not change the accepted two-state controller: +results do not create an application media-layer selector; the asymmetric +evidence windows remain limited to topology eligibility and diagnostics: 1. **Web P2P simulcast: rejected (`no-go-web-p2p-simulcast-layer-selection`).** WebRTC exposes @@ -124,64 +146,108 @@ results do not change the accepted two-state controller: required product semantic is absent. In particular, the normative rule that `active=false` stops sending an encoding is not evidence that its physical encoder, CPU work, or GPU allocation is released. -2. **Pinned LiveKit Dynacast: rejected - (`no-go-livekit-1.13.5-dynacast-cumulative-layers`).** Server 1.13.5 takes - the maximum quality requested across subscribers and subscriber nodes, then - enables every quality `q <= maxQuality`; client 2.22.0 applies those flags to - simulcast encoding `active`. Any root requesting `HIGH` consequently keeps - `LOW` enabled. This cannot satisfy healthy-room `HIGH` only, one shared `LOW` - while a weak cohort exists, and `LOW` off after that cohort recovers. The - Firefox client does not rely on `active=false` being honored and also reduces - a disabled encoding to 4x scale and 10 bps; it writes non-standard - `maxFrameRate`, so a 2 fps cap or a stopped layer cannot be relied upon. No - browser run could change this pinned control contract, so none was run. - Screener's current publisher still sets `simulcast: false` and constructs a - Room with Dynacast at its default `false`; - this result is a capability rejection, not a runtime behavior claim. -3. **SVC: next bounded candidate.** Compare the applied codec and - `scalabilityMode` with Media Capabilities `powerEfficient` and the - game-performance matrix. Adopt it only if it meets the on-demand selection - and resource gates, then stop before custom media. Reject silent software - fallback. +2. **Pinned LiveKit two-layer simulcast: next bounded runtime candidate.** + Client 2.22.0 can publish a screen-share original plus one lower simulcast + encoding. `RemoteTrackPublication.setVideoQuality(HIGH)` sets a per-subscriber + spatial-quality ceiling. Server 1.13.5 maps quality, dimensions, and FPS to + maximum spatial/temporal layers and can adapt each SFU downtrack to its own + bandwidth and recover it independently. Server Dynacast + takes the maximum requested quality and enables every quality at or below + it, so any `HIGH` root keeps `LOW` active. That cumulative behavior prevents + dynamic `LOW` stop while `HIGH` is subscribed, but dynamic stop is now an + optimization rather than a hard requirement. + + The next gate therefore publishes exactly `HIGH+LOW` with standard + simulcast/send encodings, leaves each LiveKit root's ceiling at `HIGH`, and + first tests built-in per-subscriber SFU bandwidth adaptation on + zero-descendant leaf viewers. Start with deterministic two-layer publication + and Dynacast off; its pinned cumulative behavior can later be measured as the + equivalent always-on case. Screener's + subscriber does not attach a `RemoteTrack`, so LiveKit `adaptiveStream` is not + directly usable without changing that ownership; it is not required for the + SFU bandwidth-adaptation candidate. Before default enablement, a separate + root-with-children gate must inject an autonomous downshift, observe it as + suspect, evacuate on bounded confirmation, and limit temporary descendant + impact; no confirmed `FALLBACK` root may retain children. If built-in selection + fails the product gates, test explicit standard subscriber quality selection + before sender activation/deactivation. The gate must prove that a healthy + P2P/`HIGH` path is unchanged, no third layer appears, the expected layer is + actually received, and hardware encoder, game FPS/p1 low, CPU/GPU, interval + encode cost, host upload, and `HIGH+LOW` bytes fit budget. Screener currently + sets `simulcast: false`, leaves Dynacast at its default `false`, and does not + call `setVideoQuality()`, so no runtime claim follows and no browser is run + in this static SVC review. If the always-on cost fails, test manual standard + sender activation/deactivation next; custom/native dual encoding follows + only if built-in and manual standard primitives fail. +3. **Current Web/LiveKit SVC: rejected + (`no-go-web-svc-cross-path-hardware-contract`).** WebRTC-SVC adds + `scalabilityMode` to sender encoding parameters, but it adds no receiver + setter for choosing a base or enhancement layer. The setting is scoped to + one `RTCRtpSender`; separate viewer PeerConnections still have no portable + shared-encoder or cross-connection layer-selection contract. An applied + `scalabilityMode` proves only the browser's configured mode. The SVC + specification explicitly says successful mode probing cannot distinguish a + hardware encoder from a software encoder, while Media Capabilities + `powerEfficient` and RTCStats `powerEfficientEncoder` are user-agent-defined + signals rather than hardware guarantees. + + Pinned LiveKit narrows the result further. Client 2.22.0 overwrites SVC + screen-share publication to `L1T3`: one spatial resolution and three temporal + layers, so it provides no low-resolution base and exceeds this product's + two-active-layer ceiling. Its `RemoteTrackPublication.setVideoQuality()` + controls per-subscriber spatial quality, while server 1.13.5 maps + quality/dimensions/FPS to maximum spatial/temporal layers. That selection + exists only on an SFU downtrack; it does not extend to + Screener's direct/peer receivers. The current subscriber only requests the + publication, and LiveKit documents that Dynacast can pause an entire SVC + stream but not individual SVC layers. Chrome ships the outgoing-track API; + current Firefox and WebKit WebIDL do not expose the standard + `scalabilityMode` member, and Firefox's implementation remains behind a + preference. A pinned-SDK Safari compatibility branch is not a portable + standard contract. These static blockers cannot be changed by a Chrome + media run, so no harness was written and no browser was run. None may reuse PR #28's minimum-of-two target, bypass a per-edge stock WebRTC congestion controller, or expand the representation limit beyond two. -## SVC Boundary +## SVC Static Verdict -SVC is the third standard candidate after simulcast and LiveKit/Dynacast. A path -may receive only the base layer or the base plus enhancement layers. Adopt it -when the earlier candidates fail and the exact negotiated codec and -`scalabilityMode` are read back, a power-efficient/hardware path is positively -established on the supported sender cohort, and game-performance and latency -gates pass. Do not silently fall back to software SVC. It is not the preselected -default; a strict-one-output requirement would strengthen, not create, its case. +This static verdict rejects current Web/LiveKit SVC, not the separate LiveKit +two-layer simulcast candidate above. Requested and post-negotiation applied +`scalabilityMode` remain useful diagnostics, because the browser may apply a +different mode or omit the field when none was requested. They are not proof +of the actual hardware implementation, and neither a support query nor a +browser resource run can add SVC's missing direct/peer receiver-selection +contract. -The WebRTC-SVC specification exposes `scalabilityMode` and permits the browser -to return a different configured mode after negotiation. Media Capabilities can -report support, expected smoothness, and power efficiency for a specific -configuration. WebCodecs `hardwareAcceleration` is only a hint that a user -agent may ignore. These APIs are useful evidence, not a portable guarantee of a -particular physical hardware encoder. +This rejection is scoped to the current standard Web/LiveKit shortcut, not to +scalable codecs forever. A future native sender may reopen SVC only if its media +API positively selects a hardware encoder without silent software fallback, +publishes at most two decodable layers, reuses one encoded output across direct, +peer, and SFU transports, preserves independent path selection, and passes the +same game-performance and latency gates. That would be a new native capability +decision, not completion of this browser spike. ## Consequences Positive: - Healthy viewers are not reduced to the worst path. -- Normal operation sends one active representation; one additional - representation is paid only while a verified weak path needs it. Physical - encoder instances and resource release remain measured outcomes. +- One shared `LOW` is available to all SFU paths and LiveKit selects it per + subscriber; its idle deactivation is allowed but not required. Physical + encoder instances and resource cost remain measured outcomes. - Encode cost is bounded independently of viewer count. -- The state machine and its evidence remain inspectable. +- The topology classifier and its evidence remain inspectable. Negative: -- A weak-path interval adds representation bandwidth and may require another - physical encoder; the exact instance/resource cost is implementation-specific. -- Starting/stopping `LOW` and switching paths need a keyframe-safe transition. +- An always-on `LOW` adds representation bandwidth and may require another + physical encoder even without a weak path; the exact cost is an acceptance + measurement, not an assumed negligible overhead. +- Any later explicit quality/layer fallback needs a keyframe-safe transition. - Browser capability reporting cannot by itself prove hardware or sustained - decode performance, so the real-device matrix remains mandatory. + decode performance. The current SVC path is rejected before that matrix; + custom/native candidates still require it. ## Product Stop Lines @@ -192,22 +258,31 @@ Negative: - Do not create one representation or encoder per viewer. - Do not use UA/device identity as a quality signal. - Do not enable software SVC as an invisible fallback. -- Do not start `LOW` when its measured resource cost violates the game budget. -- Do not add a composite health score when the two-state predicates suffice. +- Do not start or retain `LOW` when its measured hardware, game, or upload cost + violates budget. +- Do not add a composite health score or media selector when built-in adaptation + plus topology predicates suffice. ## Verification Gates -- A healthy room emits only `HIGH`. -- One sustained weak path starts exactly one `LOW`; healthy viewers remain on - `HIGH`, and additional weak viewers reuse `LOW`. -- After all weak paths sustain the longer recovery window, every path returns - to `HIGH`; `LOW` bytes/frames stop and its physical resource behavior is - recorded rather than assumed. +- Direct/peer paths retain independent stock WebRTC congestion control. Each SFU + path starts with a `HIGH` ceiling; under shaping, built-in LiveKit BWE sends a + weak leaf the shared `LOW` while a healthy leaf stays `HIGH`, then restores the + weak leaf without an application media selector. +- An idle `LOW` may remain active only after hardware, game-performance, and + upload cost passes. If idle stopping exists, verify bytes/frames stop; + otherwise record and accept the bounded always-on cost. - Representation count never exceeds two and host media edges never exceed two. +- A later planned/explicit `LOW` fallback evacuates children first and preserves the + prior state on stale/failed evacuation. An autonomous BWE downshift enters + `suspect`; bounded confirmation evacuates its children, and no confirmed + `FALLBACK` endpoint retains downstream capacity. The root-with-children + gate measures temporary descendant impact before default enablement. +- Relay capacity returns only after the longer stable recovery plus cooldown. - An unavailable/over-budget `LOW` fails visibly for weak paths while healthy paths and `HIGH` remain unchanged. -- Every supported sender cohort can reliably start the one shared `LOW` on - demand; otherwise automatic quality control does not pass acceptance. +- Every supported sender cohort can reliably provide the one shared `LOW`, + always-on or on demand; otherwise the quality gate does not pass. - Native relays show packet forwarding without an added decode/encode stage. - Spoofed, stale, duplicated, or rate-excessive viewer requests have no effect. - Controlled capture/CPU/bandwidth/TURN/receiver/display cases produce the @@ -223,7 +298,17 @@ Primary sources checked 2026-08-19: - [W3C Media Capabilities](https://www.w3.org/TR/media-capabilities/) - [W3C WebCodecs](https://www.w3.org/TR/webcodecs/) - [W3C WebRTC simulcast](https://www.w3.org/TR/webrtc/#simulcast-functionality) +- [Chrome 111 WebRTC SVC extension](https://developer.chrome.com/blog/chrome-111-beta) +- [Chromium `RTCRtpEncodingParameters` WebIDL](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/renderer/modules/peerconnection/rtc_rtp_encoding_parameters.idl) +- [Firefox `RTCRtpEncodingParameters` WebIDL](https://searchfox.org/firefox-main/source/dom/webidl/RTCRtpParameters.webidl) +- [Firefox WebRTC-SVC implementation status](https://bugzilla.mozilla.org/show_bug.cgi?id=1571470) +- [WebKit `RTCRtpEncodingParameters` WebIDL](https://github.com/WebKit/WebKit/blob/main/Source/WebCore/Modules/mediastream/RTCRtpEncodingParameters.idl) - [LiveKit video simulcast and Dynacast](https://docs.livekit.io/transport/media/advanced/) +- [LiveKit client 2.22.0 SVC defaults](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/track/options.ts) +- [LiveKit client 2.22.0 screen-share SVC override](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/participant/LocalParticipant.ts) +- [LiveKit client 2.22.0 SVC encoding construction](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/participant/publishUtils.ts) +- [LiveKit client 2.22.0 subscriber quality control](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/track/RemoteTrackPublication.ts) +- [LiveKit server 1.13.5 per-subscriber layer application](https://github.com/livekit/livekit/blob/v1.13.5/pkg/rtc/subscribedtrack.go) - [LiveKit client 2.22.0 Dynacast layer control](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/track/LocalVideoTrack.ts) - [LiveKit server 1.13.5 Dynacast quality aggregation](https://github.com/livekit/livekit/blob/v1.13.5/pkg/rtc/dynacast/dynacastqualityvideo.go) - [LiveKit server 1.13.5 enabled-quality generation](https://github.com/livekit/livekit/blob/v1.13.5/pkg/rtc/dynacast/dynacastmanagervideo.go) diff --git a/docs/project-memory.md b/docs/project-memory.md index 4e309df..c4e0803 100644 --- a/docs/project-memory.md +++ b/docs/project-memory.md @@ -10,7 +10,8 @@ Last updated: 2026-08-19 - Minimize server bandwidth. The accepted invisible ladder is direct/peer UDP, then SFU/UDP roots, then optional TURN for a selected exceptional edge, then bounded failure. TURN is transport, not topology. - Non-server nodes have at most two downstream edges; browser relays stay at one until resource gates pass. SFU normally feeds one or two roots that retain peer descendants; separately capped server edges may serve exceptional viewers that cannot attach behind a healthy root. - Two-tree packet/layer striping may reduce endpoint upload toward one stream bitrate, but needs a bounded multi-parent, loss, sync, churn, and latency experiment; it is not in the current full-stream chains. -- Flagship native media encodes each active representation once and reuses it across direct, peer, and SFU transports; per-connection packetization, pacing, encryption, feedback, and upload remain independent. Healthy rooms have one shared `HIGH`; verified weak paths may add one shared on-demand `LOW`, never per-viewer encoders. Standard Web P2P simulcast and pinned LiveKit 1.13.5/JS 2.22.0 Dynacast are static no-go paths for that exact selection policy; SVC is next and has no software fallback. +- Direct/peer keeps per-PC stock GCC. SFU paths start at a `HIGH` ceiling and share at most one `LOW`; next test LiveKit exactly-two built-in BWE on zero-descendant leaves. If it passes, no app media selector is built. Explicit quality, manual activation, then custom/native are later fallbacks; always-on `LOW` needs resource gates. +- A later explicit quality fallback evacuates children under a generation guard first. Autonomous BWE enters `suspect`; confirmation evacuates children, and no confirmed `FALLBACK` parent remains. Root-with-children impact is a default-on gate; capacity returns after longer recovery plus cooldown. Self-report alone never triggers it. - Separate abuse control from watching: production Host admission protects room creation/Host role, while a default private capability grants Viewer access to one room; public-watch is explicit. Keep raw grants out of localStorage/cookies/query/logs/SQLite; retain no account, ACL, user, or session table. - Keep decisions, snapshots, research, code, `AGENTS.md`, and `.codex/` in Git; rewrite memory/status in place. Research current primary sources before material work and reject speculative machinery. - Migrate client, server, and deployment atomically. After a canary, delete superseded config/wire/parsers/tests; do not retain compatibility layers, dual writes, or a second architecture without a current consumer. Git history owns the old implementation. @@ -26,7 +27,7 @@ Last updated: 2026-08-19 - Keep the failure-only controller default-off until the isolated STUN/SFU exact-room gates pass. Preserve break-before-make, sticky healthy subtrees, mobile leaves, and PR #20 prewarm; public transport/load remains open. - Local reparenting is a later candidate: start with unassigned relay admission rescue; do not block current work. - Flagship media is UDP; HTTPS/WSS stays TLS/TCP. Old production retains coturn TURN for rollback. The candidate uses self-hosted STUN-only ordinary ICE and separate LiveKit SFU/UDP, with no TURN/media-TCP wire. -- Treat settings as ceilings and deployed degradation as unclassified. Use correlated Host A+B/Viewer C to compare exact `769de201f7cc` against current `main` for Host-only recovery, game FPS/load, preview cost, actual codec/encoder and one-variable rebuilds; then gate SVC before custom `LOW`. Never use UA or a composite score. +- Treat settings as ceilings and deployed degradation as unclassified. Compare `769de201f7cc` with current `main` using Host A+B/Viewer C, game/load, preview, codec/encoder, and one-variable rebuilds. The SVC gate is closed; never use UA or a composite score. - Production reports poor film audio and self-echo when system capture includes voice software. Diagnose audio A/B/C and sync; Web cannot isolate arbitrary processes and `maxBitrate` is not quality-up. A Windows 11 native candidate defaults to game-process-tree audio and never widens silently; Windows 10 remains unresolved/unsupported. See `docs/research/browser-screen-audio-quality.md`. - Candidate UI has local Viewer volume/mute, copyable room codes, and a favicon. Names/roster, endpoint details, and RTP loss remain pending; no wire, media, or routing effects. - Do not add scene detection, dynamic-FPS control, or forced AV1 until stats and target hardware prove the need. Codec acceptance requires actual negotiation, power-efficient candidate evidence, interval encode cost, game FPS, CPU/GPU and sender count; Discord's native capture/hardware tuning is comparison evidence, not proof of server re-encoding or a reusable preset. @@ -47,8 +48,8 @@ Last updated: 2026-08-19 - Chrome 151/LiveKit localhost A/B cut failure-to-active/render from 1.481/2.257 seconds to 0.200/0.320; 31 new frames and 25 ms sampling kept host edges at two. It is headless synthetic 720p30 and includes SDK/network prewarm, not public-network evidence. - Native drafts #16/#18/#22/#23/#25/#28 remain experiments: one WebCodecs object is not hardware proof; stock GCC+RTX is no-go, and no-RTX remains outside product code. - Chrome 151 synthetic topology/quality-control runs kept fanout 2/1 and all viewers decoding; one relay close recovered in 5.32 seconds. This is control evidence only. -- Quality controls remain ceilings with diagnostics. ADR-0007's per-path `HIGH`/`FALLBACK`, one on-demand shared `LOW`, and controller are unimplemented. Web P2P simulcast lacks receiver RID selection and LiveKit Dynacast cumulatively enables `LOW`; both are static no-go results without Chrome or resource-release claims. SFU publishing remains non-simulcast with Dynacast off; SVC is next. -- Local Host A+B aligns capture with one unique outbound RTP/remote/transport pair and rejects ambiguous or stale generations. Authenticated Viewer C now adds a nullable, sanitized two-second receive window of at most 2 KiB for each current ordinary or peer-assisted P2P hop; the server derives identity and current connection/revision, and the parent correlates only matching local B. It retains no raw fmtp/SDP/stats, performs no media action, and fails closed for SFU-fed roots. Automatic control and `LOW` remain absent. +- ADR-0007's dual publication/topology classifier is absent. Next measure LiveKit `HIGH+LOW` BWE on SFU leaves, then root suspect/evacuation; app evidence never duplicates normal layer selection. Current publishing is non-simulcast/Dynacast-off. +- Local Host A+B aligns one capture/outbound/transport generation; authenticated Viewer C adds a sanitized, read-only P2P window and fails closed for stale, ambiguous, or SFU-fed evidence. It retains no raw fmtp/SDP/stats and performs no media action. Dual publication and topology classification remain absent. ## Provisional Quality Targets diff --git a/docs/research/native-shared-encode-sender.md b/docs/research/native-shared-encode-sender.md index c010f65..a55ce7a 100644 --- a/docs/research/native-shared-encode-sender.md +++ b/docs/research/native-shared-encode-sender.md @@ -180,43 +180,50 @@ conservative feedback-derived reconfiguration while feeding two transports. It does not establish heterogeneous feedback behavior and must not evolve into a room-wide minimum controller. -ADR-0007 defines the product boundary: normally run one shared `HIGH` encoded -representation; start one shared `LOW` only while correlated sender/viewer -evidence proves that paths persistently cannot sustain `HIGH`; keep healthy -paths on `HIGH`; and stop `LOW` after every weak path sustains the longer -recovery window. The representation limit is two, never one per viewer. The -independent host budget remains at most two downstream media edges. +ADR-0007 keeps stock WebRTC GCC per direct/peer path. Each SFU path starts with +a `HIGH` ceiling and built-in BWE selects from one shared `HIGH+LOW` pair. App +evidence affects topology eligibility and diagnostics, not ordinary layer +selection. `LOW` may already be active, and idle stop depends on the resource +gate. The active representation/layer limit is two, never one per viewer; the +host still has at most two downstream media edges. If a qualified hardware/power-efficient `LOW` path is unavailable or its -measured encoder/CPU/GPU game load is unacceptable, `LOW` fails closed for weak paths while -healthy paths keep `HIGH`. +measured encoder/CPU/GPU game or upload load is unacceptable, `LOW` fails closed +for weak paths while healthy paths keep `HIGH`. `LOW` may remain active when +that measured budget passes; stopping it while idle is an optimization. -Each path has only `HIGH` and `FALLBACK` state with asymmetric consecutive -entry/recovery windows, not a composite score. Viewer requests are -authenticated, rate-limited, deduplicated advice and need sender transport and -encode evidence plus viewer receive/decode corroboration. UA and device -identity do not participate in quality control. +The app's `HIGH`/`FALLBACK` states and asymmetric windows only classify topology +eligibility, not media layers, and do not form a composite score. Viewer +requests are authenticated, rate-limited, deduplicated advice and need sender +transport/encode plus viewer receive/decode corroboration. UA and device +identity do not participate. A native relay forwards the selected encoded packets and must not decode or re-encode them. An ordinary non-scalable representation cannot be forwarded -into a second quality; the bounded choices are a temporary second encode, SVC, -or transcoding. Test standard simulcast, LiveKit/Dynacast, then SVC and stop at -the first path that meets on-demand selection and resource gates. SVC still -requires the exact applied codec/mode, encoder/native evidence, and the real -game/power matrix; a future strict-one-output requirement strengthens but does -not create that case. Media Capabilities or RTCStats -power-efficiency signals are admission evidence, not hardware proof; an -unestablished path stays disabled rather than silently using software. Future -dual-tree/striped distribution may reduce two-copy host upload toward one copy -plus redundancy, but does not block dual-representation work. +into a second quality; the bounded choices are a second encode, SVC, or +transcoding. ADR-0007 statically closes current Web P2P simulcast and +Web/LiveKit SVC as cross-path shortcuts, but keeps pinned LiveKit exactly-two +simulcast with built-in SFU bandwidth adaptation on zero-descendant leaves as +the priority runtime candidate. A root-with-children downshift/evacuation gate +must pass before default enablement. Explicit subscriber quality is next if +built-in selection fails; manual sender activation follows if always-on cost +fails; custom/native dual encode is last. A future native SVC decision may +reopen only with an explicit hardware encoder contract, at most two decodable +layers, one encoded +output reused across direct/peer/SFU, independent path selection, and the real +game/power matrix. Media Capabilities or RTCStats power-efficiency signals are +diagnostics, not hardware proof; an unestablished path stays disabled rather +than silently using software. Future dual-tree/striped distribution may reduce +two-copy host upload toward one copy plus redundancy, but does not block +dual-representation work. Simulcast does not change this native proof boundary: one sender may negotiate `HIGH`/`LOW`, but separate direct PeerConnections have no portable shared-encode contract and inactive API state is not physical resource proof. ADR-0007 and [Realtime Quality Adaptation](./realtime-quality-adaptation.md) own the -post-A+B/C simulcast, pinned LiveKit -Dynacast, and SVC gates. This path still requires measured per-representation -traffic/resources and cannot bypass #28's stock-GCC/RTX stop line. +static standard-path results and any future native reopen gate. This path still +requires measured per-representation traffic/resources and cannot bypass #28's +stock-GCC/RTX stop line. ## Product Stop Line @@ -226,7 +233,7 @@ congestion controller, interceptor fork, or private transport. The no-RTX primary-SSRC path stays research-only because it sacrifices retransmission- specific receiver statistics and can distort RTP/RTCP loss accounting. -No stacked PR is connected to Screener's product controller, capture path, or +No stacked PR is connected to Screener's product path, capture path, or audio path. Before product consideration, a target native sender must still prove one physical encoder invocation on representative hardware, live PLI/FIR-to-encoder control, heterogeneous downstream estimates, bounded burst diff --git a/docs/research/realtime-quality-adaptation.md b/docs/research/realtime-quality-adaptation.md index a2723b2..6d7e908 100644 --- a/docs/research/realtime-quality-adaptation.md +++ b/docs/research/realtime-quality-adaptation.md @@ -58,8 +58,8 @@ hypothesis, not a conclusion. A further report that quality may remain low after the old viewer leaves and a new viewer joins must likewise be tested against lifecycle evidence rather than assumed. -Use this as the first A+B/C reproduction before the viewer controller or -`LOW` runtime: +Use this as the first A+B/C reproduction before quality-aware topology +classification or dual-layer publication: 1. Reproduce one wired LAN/direct viewer first, then Wi-Fi, two/three viewers, and forced TURN, using the same high-motion scene for 60 to 90 seconds. @@ -80,7 +80,7 @@ viewers neither migrate nor interrupt, and the host-edge cap holds. Only proven generation-specific failure can justify a later automatic one-edge recovery with sustained-bandwidth and healthy-counterpart evidence, cooldown, and a generation guard. Do not add periodic reconnect, blindly raise a ceiling, or -change the current controller before this gate. +change the current routing behavior before this gate. Missing `scaleResolutionDownBy` is not itself a root cause for a single encoding because its effective default is 1.0. Likewise, the current 3/5/8 Mbps values @@ -176,7 +176,7 @@ reset evidence remains unknown and rebases the interval. The smallest implementation sequence is local A+B correlation in one host sampling tick, followed by a minimal authenticated C report carrying the -receive/decode and derived negotiation signals needed by the two-state +receive/decode and derived negotiation signals needed by the topology predicate. It never carries raw SDP, raw stats, candidate addresses, or raw device/network identifiers. Server-authoritative path and connection generations provide authorization and correlation; a general remote stats stream or @@ -216,8 +216,9 @@ one unambiguous encoding. Current quality settings do not request a mode, so the requested value remains null and a browser-reported default is not called a mismatch; multiple encodings remain unknown. Inbound stats provide no current standard `scalabilityMode` source, so C does not carry a null-only placeholder. -The two-state controller and on-demand `LOW` runtime remain unimplemented, and -browser support remains subject to the controlled matrix. +The topology classifier and shared `HIGH+LOW` publication remain unimplemented, +and browser support remains subject to the controlled matrix. Physically +stopping an unused `LOW` is a resource optimization rather than a prerequisite. Official W3C text checked 2026-08-19 defines names ending in `Id` as stats-object references. In particular, outbound [`mediaSourceId`](https://www.w3.org/TR/webrtc-stats/#dom-rtcoutboundrtpstreamstats-mediasourceid) @@ -244,23 +245,29 @@ member, which remains null rather than a capability conclusion. ## Accepted Adaptation Direction -ADR-0007 rejects room-wide worst-link adaptation. Healthy operation has one -shared `HIGH` representation. A viewing path enters `FALLBACK` only after -multiple consecutive windows show insufficient bandwidth, freezes, or decode -pressure in correlated sender/viewer evidence. The first verified weak path -starts one shared `LOW`; all weak paths reuse it while healthy paths remain on -`HIGH`. Recovery requires a longer stable window than entry. When the last weak -path recovers, stop `LOW`. The hard representation limit is two, never one per -viewer. +ADR-0007 rejects room-wide worst-link adaptation. Direct/peer paths retain their +independent stock WebRTC congestion control. Each SFU path starts with a `HIGH` +ceiling; the preferred candidate publishes one shared `HIGH+LOW` pair and lets +LiveKit BWE independently select and recover its forwarded layer. If that gate +passes, Screener does not add a media-layer selector. The hard active +representation/layer limit is two, never one per viewer, and stopping idle +`LOW` remains optional. + +Correlated sender/viewer evidence and asymmetric entry/recovery windows classify +topology eligibility and diagnostics only. An observed BWE downshift is +`suspect`; sustained evidence confirms `FALLBACK`, removes parent capacity, and +longer recovery plus cooldown can restore that capacity. These application +states do not command ordinary built-in layer changes. `LOW` itself is conditional: if no qualified hardware/power-efficient media -path exists or the additional representation exceeds the measured CPU/GPU/game budget, the -controller preserves `HIGH` and fails visibly for the weak path. It never buys -weak-path recovery by degrading healthy paths. +path exists or the additional representation exceeds the measured +CPU/GPU/game/upload budget, the system preserves `HIGH` and fails visibly for +the weak path. It never intentionally buys weak-path recovery by degrading +healthy paths. That fail-closed result is exceptional damage containment, not a supported -steady state. A supported sender cohort that cannot reliably start the one -shared `LOW` on demand fails automatic-quality-control acceptance; its encoding +steady state. A supported sender cohort that cannot reliably provide the one +shared `LOW`, either always-on or on demand, fails the quality gate; its encoding path must be improved or the cohort explicitly marked unsupported. `HIGH` remains protected in either case. @@ -268,23 +275,26 @@ A viewer's `LOW` request is advisory and must be authenticated, session-bound, rate-limited, deduplicated, and corroborated by sender transport/encode and viewer receive/decode stats. UA or device-model detection is not quality evidence; the current mobile/iPad heuristic remains restricted to relay -capacity. +capacity. A valid report may affect confirmed topology eligibility; it does not +command built-in WebRTC/LiveKit media adaptation. + +The application must not stably retain a confirmed `FALLBACK` parent. For a +later planned or explicit quality fallback it evacuates children under the current generation +before setting `LOW`, preserving the prior state on failure. Built-in SFU BWE +can downshift before application evidence exists; record that as `suspect`, then +evacuate and set capacity zero only after bounded correlated confirmation. The +system cannot promise packet-level preemption of that congestion response, so a +root-with-children gate must measure and bound temporary descendant impact +before default enablement. An ordinary non-scalable stream cannot yield a second independent quality by packet forwarding alone. The alternatives are a second representation, -scalable layers, or relay/SFU transcoding. Screener tests standard capabilities -before custom media. SVC is the third short-circuit candidate when simulcast -and LiveKit/Dynacast fail; adoption requires an exact negotiated mode, a -positively established hardware or power-efficient path, and measured game -performance. WebRTC-SVC permits the browser to return a -different configured `scalabilityMode`; Media Capabilities reports support and -expected smoothness/power efficiency for a specified configuration; WebCodecs -defines `hardwareAcceleration` only as a hint the user agent may ignore. -Therefore none is, by itself, proof of a particular hardware encoder, and a -software SVC fallback must not be silent. +scalable layers, or relay/SFU transcoding. Static review closes current Web P2P +simulcast and Web/LiveKit SVC as cross-path shortcuts, while pinned LiveKit +two-layer simulcast remains the priority bounded runtime candidate. This SVC +gate does not run a browser or select a custom implementation. -The first two standard capability candidates close on static semantics, without -a Chrome run. WebRTC provides sender-side encoding control but no +WebRTC provides sender-side encoding control but no `RTCRtpReceiver.setParameters()` or other standard per-RID subscription method. Consequently, direct P2P receivers cannot explicitly choose `HIGH` versus `LOW` from one shared simulcast sender, while separate viewer PeerConnections have no @@ -293,35 +303,95 @@ portable shared-encoder contract. This is claim about per-RID traffic or whether `active=false` releases a physical encoder, CPU work, or GPU allocation. -Pinned LiveKit is also a static no-go for this exact policy. Server 1.13.5 takes -the maximum requested quality across subscribers and subscriber nodes and -enables every quality at or below that maximum. Client 2.22.0 applies those -flags to simulcast encoding `active`. Thus any `HIGH` root also keeps `LOW` -enabled, which cannot produce healthy-room `HIGH` only, a shared `LOW` only -while weak roots need it, and `LOW` off after recovery. Because its Firefox -path does not rely on `active=false` being honored, it also applies 4x scale and -10 bps; its `maxFrameRate` field is not the standard `maxFramerate`, so neither -a 2 fps cap nor a stopped layer can be treated as applied. This is -`no-go-livekit-1.13.5-dynacast-cumulative-layers`; a browser resource run -could not change the pinned control contract and was not performed. Screener's -current publisher remains `simulcast: false` with default-disabled Dynacast, so -this is not a claim about current runtime behavior. +Pinned LiveKit 2.22.0 can publish screen-share original plus one lower +simulcast encoding. `RemoteTrackPublication.setVideoQuality(HIGH)` sets a +per-subscriber spatial-quality ceiling. Server 1.13.5 maps quality, dimensions, +and FPS to maximum spatial/temporal layers and can adapt each SFU downtrack to +its own bandwidth and recover it independently. Server Dynacast +takes the maximum quality requested across subscribers and subscriber nodes, +then enables every quality at or below that maximum; a `HIGH` root therefore +keeps `LOW` active. That prevents dynamic `LOW` stop while `HIGH` is subscribed, +but idle-layer stop is only an optimization, so this cumulative behavior is not +a product no-go. The Firefox branch's 4x scale, 10 bps, and non-standard +`maxFrameRate` fallback likewise cautions against assuming a clean stopped +layer; it does not invalidate the always-on two-layer candidate. -The next standard candidate is a bounded SVC spike. Its static codec/layer -semantics may be checked before A+B/C; browser media and resource gates wait for -trustworthy A+B/C. Compare requested/applied mode and Media Capabilities -`powerEfficient`, with no silent software fallback. Stop before custom media if -it meets the on-demand selection and resource gates. None of these paths may -bypass PR #28's stock-GCC/RTX stop line. +The next runtime gate must publish exactly `HIGH+LOW` with standard +simulcast/send encodings, leave each zero-descendant SFU leaf's ceiling at +`HIGH`, and first test built-in per-subscriber SFU bandwidth adaptation and +recovery. Start with Dynacast off for a deterministic always-on measurement; +pinned cumulative +Dynacast-on is another always-on form, not a third representation. Prove the +received layer, the two-layer ceiling, unchanged healthy P2P/`HIGH`, and +hardware encoder, game FPS/p1 low, CPU/GPU, interval encode cost, upload, and +per-layer byte budgets. Before default enablement, a separate root-with-children +gate must inject an autonomous downshift, observe `suspect`, evacuate after +bounded confirmation, and limit temporary descendant impact; no confirmed +`FALLBACK` root may retain children. Screener's current publisher remains `simulcast: false`, +leaves Dynacast at default `false`, and does not call `setVideoQuality()`. +Its subscriber also does not attach a `RemoteTrack`, so SDK `adaptiveStream` is +not directly usable without changing that ownership; built-in SFU bandwidth +adaptation does not depend on enabling that feature. If built-in selection fails +the product gates, test explicit standard subscriber quality selection before +manual sender activation/deactivation. This is a candidate rather than a +current runtime claim, and no browser run belongs in this static SVC review. -If SVC does not pass and custom dual representation remains necessary, retain a -favorable, testable hypothesis: one low-rate, low-resolution hardware `LOW` may -have no material game impact. Before accepting that custom path, compare `HIGH` -against `HIGH+LOW` under one scene using game FPS/p1 low, CPU, GPU -video-encode/copy activity, interval encode cost, actual encoder identity, and -`LOW` bytes/frames. Passing permits the custom on-demand path; failure preserves -`HIGH` and fails the weak path visibly. It cannot reopen an earlier static no-go -or bypass the two-state policy and resource budget. +SVC is the third static no-go, +`no-go-web-svc-cross-path-hardware-contract`. WebRTC-SVC configures an outgoing +`RTCRtpSender`; `RTCRtpReceiver` has no matching `setParameters()` experiment or +standard base/enhancement selector. That lets an SFM/SFU selectively forward +layers, but it does not let separate direct/peer PeerConnections consume one +portable shared encoding and independently select layers. Post-negotiation +`getParameters()` can read the currently configured mode when one was +requested, including a browser-selected replacement, but a successful request +does not reveal whether the encoder is hardware or software. + +The remaining capability signals cannot enforce the product's no-silent- +software-fallback rule. Media Capabilities defines `powerEfficient` as an +optimal-power judgment left to the user agent and explicitly notes that +software can qualify. RTCStats exposes `encoderImplementation` and +`powerEfficientEncoder` only when hardware exposure is allowed; the latter +should reflect acceleration but may use other information. These fields are +valuable diagnostics, not an affirmative portable hardware contract. + +Official implementation surfaces checked 2026-08-19 are deliberately treated +as boundaries, not UA quality rankings: + +| Implementation | Static SVC surface | +| --- | --- | +| Chromium/Chrome | Chrome 111 shipped outgoing-track SVC selection; current Blink WebIDL exposes `scalabilityMode` behind its runtime feature. Exact codec/mode and hardware use remain runtime outcomes. | +| Firefox | Current Firefox WebIDL omits `scalabilityMode`; Mozilla's implementation issue remains assigned and behind a preference. | +| Safari/WebKit | Current WebKit WebIDL omits the standard member. LiveKit 2.22.0 carries a Safari-specific legacy encoding branch, but that is an SDK workaround rather than a portable standard setter/readback contract. | + +Pinned LiveKit can select layers per SFU subscriber: client 2.22.0 exposes +`RemoteTrackPublication.setVideoQuality()`, and server 1.13.5 maps each +subscriber's quality/dimensions/FPS to maximum spatial/temporal layers on that +subscriber's downtrack. That useful SFU contract still does not extend to +direct/peer receivers. Screener's current SFU subscriber calls only +`setSubscribed(true)`, so it does not own a layer choice today. + +For this screen-share product the pin has an additional hard mismatch. Client +2.22.0 overwrites SVC screen-share publication to `L1T3`, even when another +mode was supplied: one spatial resolution and three temporal layers. It +therefore supplies no low-resolution base and exceeds the two-active-layer +ceiling. LiveKit also documents that Dynacast can pause only an entire SVC +stream, not individual SVC layers. Changing those contracts would require a +different dependency/native design decision, not a runtime proof of the pinned +path. No SVC harness was written and no Chrome run was performed. None of the +rejected standard paths may bypass PR #28's stock-GCC/RTX stop line. + +If built-in LiveKit selection fails the product gates, test explicit standard +subscriber quality selection. If the always-on representation cost itself +exceeds budget, test manual activation/deactivation through standard sender +primitives. Only if these standard forms fail may custom/native dual +representation remain a candidate. Retain the +favorable, testable hypothesis that one low-rate, low-resolution hardware `LOW` +may have no material game impact, but compare `HIGH` against `HIGH+LOW` under +one scene using game FPS/p1 low, CPU, GPU video-encode/copy activity, interval +encode cost, actual encoder identity, upload, and per-layer bytes before +accepting any custom path. Failure preserves `HIGH` and fails the weak path +visibly. It cannot reopen an earlier static no-go or bypass the topology policy +and resource budget. ## Why Offline Encoding Presets Do Not Transfer @@ -435,10 +505,13 @@ pipeline. Compare image readability and motion continuity instead of declaring success from FPS alone. If reproducible evidence later shows that `balanced` still oscillates or makes the wrong tradeoff on supported machines, revise the fixed -profiles before enabling ADR-0007. Its acceptance matrix must prove one healthy -viewer remains `HIGH` while another enters/recover from `FALLBACK`, that all weak -viewers move to exactly one shared `LOW`, that `LOW` stops after sustained -recovery, and that each supported sender cohort can start it reliably. +profiles before enabling ADR-0007. Its acceptance matrix must prove that built-in +LiveKit BWE independently moves one shaped SFU leaf to the shared `LOW` and back +while a healthy leaf remains `HIGH`, without an application media selector. The +application's asymmetric evidence windows must affect only confirmed topology +eligibility and diagnostics. Each supported sender cohort must provide `LOW` +within measured hardware, game, and upload budgets; idle-layer resource release +is a separate optimization. ## Primary Sources @@ -450,6 +523,11 @@ recovery, and that each supported sender cohort can start it reliably. - [W3C WebRTC SVC](https://www.w3.org/TR/webrtc-svc/) - [W3C Media Capabilities](https://www.w3.org/TR/media-capabilities/) - [W3C WebCodecs](https://www.w3.org/TR/webcodecs/) +- [Chrome 111 WebRTC SVC extension](https://developer.chrome.com/blog/chrome-111-beta) +- [Chromium `RTCRtpEncodingParameters` WebIDL](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/renderer/modules/peerconnection/rtc_rtp_encoding_parameters.idl) +- [Firefox `RTCRtpEncodingParameters` WebIDL](https://searchfox.org/firefox-main/source/dom/webidl/RTCRtpParameters.webidl) +- [Firefox WebRTC-SVC implementation status](https://bugzilla.mozilla.org/show_bug.cgi?id=1571470) +- [WebKit `RTCRtpEncodingParameters` WebIDL](https://github.com/WebKit/WebKit/blob/main/Source/WebCore/Modules/mediastream/RTCRtpEncodingParameters.idl) - [RFC 6184 H.264 RTP payload format](https://www.rfc-editor.org/rfc/rfc6184.html) - [RFC 7741 VP8 RTP payload format](https://www.rfc-editor.org/rfc/rfc7741.html) - [RFC 9628 VP9 RTP payload format](https://www.rfc-editor.org/rfc/rfc9628.html) @@ -467,6 +545,12 @@ recovery, and that each supported sender cohort can start it reliably. - [libwebrtc removal of the automatic animation-detection experiment, 2024-05-22](https://webrtc.googlesource.com/src/+/1d7d0e6e2c5002815853be251ce43fe88779ac85) - [LiveKit screen-share presets](https://github.com/livekit/client-sdk-js/blob/main/src/room/track/options.ts) - [LiveKit degradation defaults](https://github.com/livekit/client-sdk-js/blob/main/src/room/participant/publishUtils.ts) +- [LiveKit video simulcast and Dynacast](https://docs.livekit.io/transport/media/advanced/) +- [LiveKit client 2.22.0 SVC defaults](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/track/options.ts) +- [LiveKit client 2.22.0 screen-share SVC override](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/participant/LocalParticipant.ts) +- [LiveKit client 2.22.0 SVC encoding construction](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/participant/publishUtils.ts) +- [LiveKit client 2.22.0 subscriber quality control](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/track/RemoteTrackPublication.ts) +- [LiveKit server 1.13.5 per-subscriber layer application](https://github.com/livekit/livekit/blob/v1.13.5/pkg/rtc/subscribedtrack.go) - [LiveKit client 2.22.0 Dynacast layer control](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/track/LocalVideoTrack.ts) - [LiveKit server 1.13.5 Dynacast quality aggregation](https://github.com/livekit/livekit/blob/v1.13.5/pkg/rtc/dynacast/dynacastqualityvideo.go) - [LiveKit server 1.13.5 enabled-quality generation](https://github.com/livekit/livekit/blob/v1.13.5/pkg/rtc/dynacast/dynacastmanagervideo.go) diff --git a/docs/status.md b/docs/status.md index b4fc6ce..8ac746b 100644 --- a/docs/status.md +++ b/docs/status.md @@ -15,14 +15,16 @@ unconfigured, so production remains one host connection per viewer and P2P/TURN. - Current SQLite still stores only room ID and Host-token digest. Amended ADR-0002 accepts a later atomic `HOST_ADMISSION_PASSWORD`/`screener-v2` migration with default room-scoped private Viewer grants, explicit public-watch, rotate/revoke, and one nullable digest column; no runtime is implemented. - PR #44 merged the default-off exact-room candidate code: ordinary ICE is process-wide STUN-only, LiveKit SFU/UDP feeds at most two roots, browser relays stay at one child, the host stays at two, and exhaustion fails boundedly. Old production/coturn remains rollback; HTTPS/WSS stays TLS/TCP. - Fallback prewarm is token-free and dormant without configuration. Mobile/iPad viewers are leaves and healthy edges stay sticky. -- Production has memory-only video settings and manual readback. Automatic `HIGH`/`FALLBACK` plus one shared on-demand `LOW` is accepted but not deployed; local A+B and authenticated P2P Viewer C are read-only evidence. Browser audio is request/presence-only. +- Manual readback and read-only A+B/C exist; dual publication/topology classification do not. Always-on `LOW` needs exactly-two, hardware, game, and upload gates. Browser audio is request/presence-only. +- A later explicit fallback evacuates first. Autonomous BWE enters `suspect`; confirmation evacuates children, and no confirmed `FALLBACK` parent remains. Root impact is a default-on gate; self-report alone cannot trigger it. ## Verified Evidence - Release `769de201f7cc` passed CI/hygiene/typecheck, 274 tests, both builds, zero production dependency vulnerabilities, atomic health/access/bundle/SQLite/service/log checks, and responsive 320/375/390 px checks without live media; rollback is ready. - Chrome 151 synthetic `1/3/5/8` and 720p30 quality-change runs kept 2/1 fanout and decoding; slowest first frame was 1.05 seconds and one relay close recovered in 5.32 seconds. This is control evidence only. - Chrome 151/LiveKit localhost A/B measured cold active/render at 1.481/2.257 seconds versus standby 0.200/0.320 with host edges at two. It includes SDK/prewarm and is not public-network evidence. -- Host A+B aligns capture/outbound identity, deltas, path, and nullable transport-bound codec/`scalabilityMode`. Authenticated Viewer C is limited to the current ordinary or peer-assisted P2P hop and its connection/revision; two-second windows are at most 2 KiB, sanitized, generation-bound, and read-only. Stale, ambiguous, and SFU-fed evidence fails closed; no raw media metadata is retained. +- Host A+B and authenticated P2P Viewer C are sanitized, generation-bound, read-only, and fail closed for stale, ambiguous, or SFU-fed evidence; no raw media metadata is retained. +- SVC is `no-go-web-svc-cross-path-hardware-contract`: no direct/peer selection, cross-PC shared encode, or portable hardware proof; pinned screen share is `L1T3`. No browser run was warranted. - HTTPS/WSS, access and room/WebSocket auth, renewal, public STUN, and authenticated TURN/UDP/TCP relay-only paths pass; TURN/TLS is off. - Draft #16/#18/#22/#23/#25/#28 passed isolated experiments only. ADR-0006 reached host setup/one encoder output, then timed out before viewer 1 decoded/rendered. @@ -33,10 +35,9 @@ unconfigured, so production remains one host connection per viewer and P2P/TURN. - Silent partitions can wait 30 to 60 seconds for heartbeat detection before the default 5-second grace; this remains unverified. - Real audio and heterogeneous clients remain unverified. Production reports poor film audio and voice-call self-echo under system capture; there is no app audio ceiling or Web process isolation. Diagnose A/B/C and sync, then gate Windows 11 game-process audio; Windows 10 stays unresolved without system fallback. - Production `769de201f7cc` reportedly sustains bandwidth-limited blur on a capable LAN; Host refresh recovered while Viewer refresh did not, and persistence across churn was also reported. This prioritizes, but does not prove, host sender/PC/GCC/capture generation. -- ADR-0007 remains incomplete: the reported case/A+B/C matrix is unverified; the controller, `LOW`, and SFU last-hop C are absent. Weak paths share one `LOW`, and an unreliable supported cohort fails acceptance. - Browser relays re-encode. ADR-0006 is no-go-unclassified: downstream checkpoints and two-edge/FIFO/TURN gates are absent. Spikes prove one WebCodecs object, not hardware; GCC+RTX is no-go and no-RTX weakens stats. - Browser fanout is host two/viewer one; any accepted endpoint relay stays capped at two downstream edges. -- Web P2P simulcast and pinned LiveKit 1.13.5/JS 2.22.0 Dynacast are static no-go: no receiver RID-selection/shared-encoder contract, and `HIGH` cumulatively enables `LOW`. No Chrome/resource-release claim follows; current SFU publishing is non-simulcast/Dynacast-off. SVC is next. +- ADR-0007/SFU C are incomplete. Next is LiveKit exactly-two BWE on SFU leaves, then root suspect/evacuation. Web P2P/current SVC shortcuts are no-go; publishing is non-simulcast/Dynacast-off. - The corrected standby smoke is localhost/headless/video-only; public DNS/TLS reuse, transport, audio, shaping, load, mobile, endurance, and sub-25 ms overlap remain open. - Scoped Viewer access is designed but unimplemented. Tests owe role bounds, fragment clearing/no-leak, expiry/cross-room/rotate/revoke, v1 tab termination, and SQLite locked migration/restore. Human passwords/accounts stay out. @@ -48,11 +49,15 @@ A+B while separately rebuilding one `HostPeer`, replacing capture, and fully refreshing Host. Only a proven stuck generation justifies guarded recovery; never reconnect periodically, force AV1, or raise ceilings blindly. -After that reproduction, run the bounded SVC capability gate and stop if it -fits before custom `LOW`, independently of ADR-0006. Diagnose audio A/B/C, +The bounded SVC capability gate is closed statically. After that reproduction, +gate LiveKit exactly-two/Dynacast-off BWE on zero-descendant SFU leaves, then +bound root-with-children downshift/evacuation. Prove healthy `HIGH`, hardware, +game, upload, and received layers. Explicit quality, manual activation, and +custom/native are later fallbacks. +Diagnose audio A/B/C, sync, and voice-source leakage in parallel when it does not displace that P0; `maxBitrate` is not a quality fix. WebRTC/LiveKit owns congestion/layers; the app -owns two-state policy. An isolated exact room later gates peer/SFU UDP and +owns evidence and topology guardrails. An isolated exact room later gates peer/SFU UDP and bounded failure before replacing the old release. Selected-edge TURN, if later justified, is a separate complete change rather than part of this canary. Candidate UI has volume, room-code copy and favicon; names/roster/diagnostics remain. diff --git a/docs/方案设计.md b/docs/方案设计.md index bcd9547..971a08c 100644 --- a/docs/方案设计.md +++ b/docs/方案设计.md @@ -145,7 +145,7 @@ Zod 在服务边界拒绝未知类型、超长字符串、非法 SDP/candidate 浏览器可返回更低设置，界面以 `track.getSettings()` 显示实际捕获结果。生产 release `769de201f7cc` 已部署有界手动质量控制：视频 track 保留 `contentHint = "motion"`，默认使用清晰优先的 `degradationPreference = "maintain-resolution"`，并允许显式选择 balanced 或 fluid。折叠高级区将设置严格限制为 720p/1080p/1440p、15 至 60 整数 FPS、2 至 12 Mbps，以及 `maintain-resolution`、`balanced`、`maintain-framerate` 三种偏好。它们只是 ceiling 和浏览器偏好，不保证实际分辨率、帧率或码率。 -每次 `setParameters()` 都必须从当次 `getParameters()` 派生更新，然后读回 requested/applied `maxBitrate`、`maxFramerate`、`scaleResolutionDownBy` 和 degradation preference。拒绝或浏览器改写必须进入可见状态，不能只写 `console.warn`。线上已是 clarity-first bounded controls，但用户报告的严重降级尚未完成受控分类，ADR-0007 自动质量控制也尚未实现；不得据此宣称画质提升。下一次真实质量验收仍需在同一高动态场景采样 1、2、3 名观看者和 TURN 路径。不得只提高 ceiling、强制 codec 或修改 SDP。 +每次 `setParameters()` 都必须从当次 `getParameters()` 派生更新，然后读回 requested/applied `maxBitrate`、`maxFramerate`、`scaleResolutionDownBy` 和 degradation preference。拒绝或浏览器改写必须进入可见状态，不能只写 `console.warn`。线上已是 clarity-first bounded controls，但用户报告的严重降级尚未完成受控分类，ADR-0007 双层发布与拓扑分类也尚未实现；不得据此宣称画质提升。下一次真实质量验收仍需在同一高动态场景采样 1、2、3 名观看者和 TURN 路径。不得只提高 ceiling、强制 codec 或修改 SDP。 PoC 不强制 codec，也不做 SDP munging。先通过 stats 记录实际协商 codec、`encoderImplementation` 和 `powerEfficientEncoder`，再依据目标机器测量决定是否调整 H.264、VP8、VP9 或 AV1 优先级。`getCapabilities()` 只证明可协商，Media Capabilities 的 supported/smooth/power-efficient 只提供候选证据；最终必须同时检查实际 outbound codec、区间 encode time、游戏 FPS、CPU/GPU encode 引擎和 sender 数量。AV1 压缩效率不能证明目标机器具有 WebRTC 硬件编码，强制后可能落入更重的软件路径。 @@ -165,9 +165,9 @@ Host 本地预览直接绑定现有 capture stream，不创建 viewer、`RTCPeer `frameRate` 约束和 sender 的 `maxFramerate` 都是目标或上限；浏览器仍控制实际捕获、重复帧处理、编码和拥塞降级。静态画面可能由浏览器减少实际编码数据，但标准没有保证每个实现以同一种方式节省采集或 GPU 开销。本 PoC 不添加画面差分、定时改约束或自定义动态 FPS 状态机，先用 `framesEncoded`、实际发送码率、`totalEncodeTime`、质量受限原因和主机资源占用测量真实问题。 -自动质量控制采用 ADR-0007 的按需双表示，而不是全房间按最差链路降档。正常状态只运行一套 active `HIGH` representation；第一条被真实统计持续证明无法承受的观看路径出现时，必须启动唯一一套共享 `LOW` 并迁移所有已验证弱路径，健康路径继续接收 `HIGH`。最后一条弱路径经过更长的稳定恢复窗口回到 `HIGH` 后关闭 `LOW`。active representation 数量硬限制为 `HIGH + at most one on-demand LOW <= 2`，不能按 viewer 增长为 N 套。标准浏览器不保证跨 PeerConnection 共用一个物理编码器，inactive 层也不保证释放 GPU；实例数、CPU/GPU 与释放行为必须实测。若没有合格 hardware/power-efficient 路径或启动 `LOW` 实测破坏游戏预算，则当次 fail closed，保留 `HIGH` 并明确显示弱路径降级或失败；这只是异常止损。supported sender cohort 若不能可靠按需启动 `LOW`，则自动质量控制验收失败，必须优化编码路径或明确标记不支持，不能长期忽略弱路径。 +质量自适应采用 ADR-0007 的 built-in-first 路径隔离双表示，而不是全房间按最差链路降档。Direct/peer 保留每条 PeerConnection 的 stock WebRTC GCC。每条 SFU path 从 `HIGH` ceiling 开始；优先发布共享 `HIGH+LOW` 并让 LiveKit BWE 独立降层和恢复。若该 gate 通过，Screener 不实现媒体层 selector。应用证据只决定下述 parent/leaf 拓扑资格与诊断。`LOW` 可以常驻；关闭空闲层只是资源优化。active representation/layer 数量硬限制为二，不能按 viewer 增长。标准浏览器不保证跨 PeerConnection 共用一个物理编码器，inactive 层也不保证释放 GPU；实际层数、实例数、CPU/GPU、游戏 FPS/p1 low、区间 encode cost、上传与释放行为必须实测。若没有合格 hardware/power-efficient 路径或额外表示破坏游戏/上传预算，则 fail closed，保留健康路径并明确显示弱路径降级或失败。 -每条观看路径只维护 `HIGH`/`FALLBACK` 两态。`HIGH -> FALLBACK` 需要多个连续采样窗口中的带宽不足、冻结或解码跟不上；`FALLBACK -> HIGH` 使用不同门槛和更长的稳定窗口，避免来回抖动。不把 RTT、CPU、码率、设备信息等压成一个综合分数。具体窗口和阈值必须由真实质量矩阵确定，而不是先写死一个猜测。 +应用只为拓扑资格维护 `HIGH`/`FALLBACK` 两个 durable states；`suspect` 是采样中的候选而不是第三种媒体模式。连续窗口确认 `FALLBACK` 后撤销 parent capacity，更长稳定窗口和 cooldown 后才恢复。它不命令正常的 LiveKit/WebRTC 媒体层，也不把 RTT、CPU、码率、设备信息压成综合分数。具体窗口和阈值由真实质量矩阵确定。 状态决策必须关联同一时间区间和媒体代次的三段证据： @@ -179,15 +179,17 @@ Host 本地预览直接绑定现有 capture stream，不创建 viewer、`RTCPeer 判断顺序固定为：Capture 低归为采集或约束；Capture 高而 outbound 低且 reason=`cpu` 归为编码资源；Capture 高而 outbound 低且 reason=`bandwidth` 归为 GCC、上行或 TURN/路径；outbound 正常而 inbound 低归为传输或接收；inbound 指标正常但肉眼模糊归为码率/量化、协商 codec/profile/parameters、适用的 scalable layer 或显示缩放。B/C 必须关联同一派生协商结果与实际解码表现。累计统计只计算相邻、不重叠区间的 delta；缺字段、stats ID/媒体代次变化或计数器回退时重新建立基线，不能把未知当零。 -探针按最小顺序落地：先在 host 同一个采样 tick 内关联 A+B，并携带明确 window、媒体/stats identity 和有效 delta；验证可靠后才增加只包含两态判定所需字段的受限、已鉴权 C 上报。不建设通用遥测 schema，只传派生后的协商字段，绝不上传 raw stats、raw SDP、candidate 地址或原始设备/网络标识；鉴权与关联只使用服务端签发的不透明 path/connection generation ID。 +探针按最小顺序落地：先在 host 同一个采样 tick 内关联 A+B，并携带明确 window、媒体/stats identity 和有效 delta；验证可靠后才增加只包含拓扑判定所需字段的受限、已鉴权 C 上报。不建设通用遥测 schema，只传派生后的协商字段，绝不上传 raw stats、raw SDP、candidate 地址或原始设备/网络标识；鉴权与关联只使用服务端签发的不透明 path/connection generation ID。 -Viewer 可通过已鉴权、绑定当前房间/路径代次的控制面请求 `LOW`，但请求只是一条观测输入，必须限频和去重，并与 sender 的 Transport-CC/GCC、实际码率、RTT、丢包、重传及编码限制、派生协商 codec/profile/parameters（适用时含 `scalabilityMode`）和 viewer 的实际解码/丢帧/冻结表现交叉验证。UA、平台和设备型号不参与质量决策；现有 mobile/iPad 检测只用于保守设置 relay capacity。 +Viewer 可通过已鉴权、绑定当前房间/路径代次的控制面报告需要 `LOW`，但它只是一条拓扑观测输入，不能命令内建 WebRTC/LiveKit 媒体层。报告必须限频、去重，并与 sender 的 Transport-CC/GCC、实际码率、RTT、丢包、重传及编码限制、派生协商 codec/profile/parameters（适用时含 `scalabilityMode`）和 viewer 的实际解码/丢帧/冻结表现交叉验证。UA、平台和设备型号不参与质量决策；现有 mobile/iPad 检测只用于保守设置 relay capacity。 -SVC 是 simulcast 与 LiveKit/Dynacast 均未满足后的第三个标准候选；若它能独立选层、读回实际 codec/`scalabilityMode`、确认目标 hardware/power-efficient 路径并通过游戏性能和亚秒延迟门槛，即采用并停止自研。浏览器的硬件偏好不是保证，不能静默回退为软件 SVC；它不是预选默认方案，未来严格单输出要求只会加强而不是创建采用理由。普通非分层编码包不能只靠转发生成另一档画质，实现路径差异只能用第二表示、SVC 或 relay/SFU 转码。 +SVC 的标准能力门禁已经静态关闭为 `no-go-web-svc-cross-path-hardware-contract`。WebRTC-SVC 只配置单个 outgoing sender，direct/peer receiver 没有对应的按层选择 API，独立 PeerConnection 之间也没有 shared-encode 契约；requested/applied codec/`scalabilityMode`、Media Capabilities `powerEfficient` 与 RTCStats encoder 字段不能证明编码器一定是硬件。Pinned LiveKit 虽能在 SFU 内为每个 subscriber 设置最大空间/时间层，但 client 2.22.0 会把 screen-share SVC 强制成三层 temporal-only `L1T3`，既没有低分辨率 base，也超过最多两层的门槛；这种 SFU 选层也不延伸到 direct/peer descendants。当前静态结论不运行 browser harness。未来只能由能明确锁定硬编、最多两层、跨 direct/peer/SFU 复用同一输出且通过游戏性能/延迟门槛的 native sender 以新决策重开。普通非分层编码包不能只靠转发生成另一档画质，实现路径差异只能用第二表示、合格的 scalable layers 或 relay/SFU 转码。 -质量控制不放宽媒体拓扑预算：host 仍最多两条下游媒体边；native relay 只转发选定表示的编码包，不解码重编码。后续双树或条带化可把 host 上传从约两份完整码率推向一份加必要冗余，但不阻塞按需双表示的实现。 +质量控制不放宽媒体拓扑预算：host 仍最多两条下游媒体边；native relay 只转发选定表示的编码包，不解码重编码。后续双树或条带化可把 host 上传从约两份完整码率推向一份加必要冗余，但不阻塞路径隔离双表示的实现。 -标准表示能力排在任何自定义双表示媒体实现之前，并可与 ADR-0006 native sender 复核并行。静态 API 或 pinned 协议不满足可在完整 A+B/C 前短路否决；只有 runtime media/resource gate 等待可信 A+B/C。序列依次为同 sender simulcast、最多两个 LiveKit root 的 Dynacast 和 SVC；前两项已静态 no-go，SVC 是下一候选，首个满足语义与资源门槛的路径立即结束序列。具体门槛由 ADR-0007 和[实时质量调研](./research/realtime-quality-adaptation.md)持有，任何方案都不得复用 PR #28 的 minimum-of-two 或绕过 stock WebRTC congestion control。 +应用不得稳定保留经交叉证据确认的 `FALLBACK` parent。SFU BWE 可能先自主 downshift；该观测进入 `suspect`，跨有界窗口确认后再迁走 children 并撤销 capacity，不能承诺 packet-level 先迁后降。Root-with-children gate 必须限制 descendants 的短暂影响。计划或显式质量 fallback 只在内建路径失败后考虑，并先在 assignment/connection generation guard 下迁走全部 children；失败或 stale 操作保持原 assignment 与 quality。更长恢复窗口和 cooldown 后才能重新发布 capacity。Viewer 自报不能单独改变拓扑。详细 wire 由后续 routing owner 持有。 + +标准表示能力排在任何自定义双表示媒体实现之前。当前 Web P2P simulcast 因 receiver 不能选 RID 且跨 PeerConnection 无 shared-encode 契约而静态 no-go；当前 Web/LiveKit SVC 因 direct/peer 无独立选层、硬件不可证明且 pinned screen share 为 `L1T3` 而静态 no-go。下一 gate 优先验证 pinned LiveKit 的标准 exactly-two screen-share simulcast：Dynacast off 常驻 `HIGH+LOW`，zero-descendant SFU leaf 以 `HIGH` 为 maximum，由 SFU BWE 独立降层和恢复；通过后不实现自定义媒体 selector。Root-with-children 的 suspect/确认/有界迁移是 default-on 前硬门。只有内建选择不满足门槛时才显式设置标准 subscriber quality；常驻成本违反硬件、游戏或上传预算时才测标准 sender 手动启停，全部标准路径失败后才进入 custom/native dual encode。当前 subscriber 未 attach `RemoteTrack`，因此不直接启用 `adaptiveStream`，但这不阻塞 SFU BWE leaf 候选。Cumulative Dynacast-on 只作为另一种 always-on 形态。本 SVC 静态审查不跑 browser。具体门槛由 ADR-0007 和[实时质量调研](./research/realtime-quality-adaptation.md)持有，任何方案都不得复用 PR #28 的 minimum-of-two 或绕过 stock WebRTC congestion control。 ## 媒体传输配置 @@ -343,7 +345,7 @@ Viewer 在处理 peer-assisted 鉴权快照时先更新 `ViewerRelay` 的 desire spike 复用现有 screen stream 和浏览器 codec 协商，覆盖当前桌面 Chrome/Edge relay、按测试顺序位于链尾的当前 Android Chrome 与 iOS Safari leaf、最多八名 viewer；捕获存在时沿同一标准 WebRTC 路径带上音频。Encoded Transform/DataChannel/WebCodecs 媒体、SVC/simulcast、多树、转码、codec ladder、综合评分、移动端后台 relay 和跨模式 SFU 迁移不在本次 spike 内实现。该范围限制不能解释为拒绝产品所需的自动兜底；验收与失败即放弃条件以 ADR-0004 为准，自动路由由后续独立 ADR 定义。 -浏览器不保证多个 `RTCPeerConnection` 共享一次编码。实验允许 host 对一至两个 seed 各自编码。后续 native sender 必须让每个 active representation 只编码一次并跨 direct、peer 与 SFU transport 复用；每条连接仍拥有独立 packetizer、SSRC、pacing、加密、反馈和上传。健康时只有共享 `HIGH`，弱路径期间最多增加一份共享 `LOW`。该计划不依赖本次浏览器 relay 实验成败，也不在本 spike 中提前抽象或实现。 +浏览器不保证多个 `RTCPeerConnection` 共享一次编码。实验允许 host 对一至两个 seed 各自编码。后续 native sender 必须让每个 active representation 只编码一次并跨 direct、peer 与 SFU transport 复用；每条连接仍拥有独立 packetizer、SSRC、pacing、加密、反馈和上传。健康路径选择共享 `HIGH`，最多另有一份弱路径共用的 `LOW`，是否常驻由硬件、游戏和上传门槛决定。该计划不依赖本次浏览器 relay 实验成败，也不在本 spike 中提前抽象或实现。 详细证据见 `docs/research/peer-assisted-media.md`，提案见 `docs/adr/0004-peer-assisted-media-experiment.md`。 diff --git a/docs/需求理解.md b/docs/需求理解.md index 7fc6535..fbdfedf 100644 --- a/docs/需求理解.md +++ b/docs/需求理解.md @@ -24,8 +24,8 @@ - [ ] P2P 优先：每条逻辑媒体边的 ICE 必须先尝试直接 UDP 路径，服务器不得默认中转可直连的媒体。一个或两个观看者优先直接连接分享端；更多观看者的 peer-assisted 行为须先通过 ADR-0004 的实验门槛。 - [ ] NAT 与传输兜底：STUN 必须可用。普通 direct/peer 边只接收 STUN ICE servers；其路径不可用时改由 SFU virtual parent 承担必要 root，而不是给每条 peer edge 常驻 TURN candidates。当前可执行梯级为 `direct/peer UDP -> SFU ICE/UDP -> 明确失败`。应用配置、信令协议和客户端不得保留无消费者的全房 TURN URL、credential 或续期 wire。未来只有经受限网络证据证明必要时，才可由独立 PR 新增绑定当前 assignment/connection generation 的 selected-edge coturn grant；一个 TURN seed root 收到媒体后仍优先服务 bounded peer descendants。LiveKit publisher/subscriber 到 SFU 是另一 ICE 域，pinned 1.13.5 的 embedded/external TURN 当前按 LiveKit participant 广告，不是 Screener selected-edge grant。两者必须分开配置、鉴权和验收。当前候选不配置 LiveKit TURN、媒体 ICE/TCP、TURN/TCP 或 TURN/TLS；HTTPS/WSS 仍使用 TLS/TCP。 - [ ] 质量控制：保留 1080p60（8 Mbps）、1080p30（5 Mbps）、720p30（3 Mbps）三档推荐上限，默认采用清晰优先；分享期间可直接换档或应用有界高级设置，不重新选择来源或重建健康连接。Peer-assisted relay 与已配置的 SFU 必须接收同一个严格校验的房间级权威设置，覆盖当前和以后建立的 sender；普通 P2P 信令结构保持不变。任何设置都只是浏览器上限和偏好，必须读回实际 sender 参数并显示参数拒绝、改写和持续受限原因。不做自定义动态 FPS、综合评分或周期性全房间调档。 -- [ ] 按需双表示：健康房间只运行一套 active `HIGH` representation；第一条观看路径被同一时段的 sender 与 viewer 统计持续证明无法承受时，系统必须启动唯一一套由全部弱路径共用的 `LOW` 并迁移弱路径。健康路径继续使用 `HIGH`，全部弱路径稳定恢复后关闭 `LOW`。active representation 硬上限为二，禁止按观看者增长为 N 套，也禁止把最差路径的目标应用到整个房间。实现必须依次验证标准 WebRTC simulcast、LiveKit layer selection/Dynacast 和 SVC；任一方案满足按需停启、独立选层与资源门槛即停止，不继续自研媒体层。标准浏览器不保证跨 PeerConnection 共用一个物理编码器，inactive 层也不保证释放 GPU；物理 encoder instance、CPU/GPU 和释放行为必须实测。没有合格 hardware/power-efficient 路径或启动 `LOW` 造成不可接受负载时，当次必须 fail closed：保护 `HIGH` 并明确显示弱路径降级或不可用；这只是异常止损，supported sender cohort 若不能可靠按需启动 `LOW`，则自动质量控制验收失败，必须优化编码路径或明确标记不支持。 -- [ ] 路径级质量状态：每条观看路径只使用 `HIGH` 与 `FALLBACK` 两态。进入 `FALLBACK` 需要多个连续窗口的带宽不足、冻结或解码跟不上证据，回到 `HIGH` 需要更长的稳定窗口；进入与退出使用不同门槛且不计算综合分数。Viewer 的低档请求必须经当前会话鉴权、限频和去重，并由 sender 的 GCC/Transport-CC、实际码率、RTT、丢包、重传、编码限制、派生后的实际协商视频 codec/profile/parameters（适用时含 `scalabilityMode`）与 viewer 的实际解码、丢帧、冻结证据交叉验证，不能仅凭自报生效；上报不得包含 raw SDP、candidate 地址或原始设备/网络标识，鉴权与关联只使用服务端签发的不透明路径/连接代次 ID。 +- [ ] 路径隔离双表示：direct/peer 继续使用每条 PeerConnection 独立的 stock WebRTC GCC，不把任何 viewer 反馈合成全房目标。每条 SFU path 从 `HIGH` ceiling 开始；优先用 pinned LiveKit 2.22.0/1.13.5 的标准两层 screen-share simulcast，在 Dynacast off 下常驻 exactly `HIGH+LOW`，让内建 per-subscriber BWE 独立降层和恢复。若该 gate 通过，Screener 不实现媒体层 selector。首 gate 使用 zero-descendant SFU leaf；root-with-children 的自动 downshift 与有界迁移是 default-on 前独立硬门。只有内建选择不满足门槛时才用标准 subscriber quality 显式限制，常驻成本不合格时再测标准 sender 手动启停。`LOW` 停编只是资源优化；active representation/layer 硬上限为二，禁止按 viewer 增长。必须实测实际接收层、硬件编码证据、游戏 FPS/p1 low、CPU/GPU、区间 encode cost、上传和分层 bytes。当前 Web P2P simulcast 缺少 receiver 选 RID 和跨 PeerConnection shared-encode 契约；当前 Web/LiveKit SVC 也缺少 direct/peer 独立选层与可移植硬件保证，且 pinned screen share 强制三层 temporal-only `L1T3`，二者按现契约静态 no-go。全部标准路径失败后才考虑 custom/native dual encode。没有合格 hardware/power-efficient 路径或额外表示破坏游戏/上传预算时 fail closed：保护健康路径并明确显示弱路径降级或不可用。 +- [ ] 路径级质量状态：应用的 `HIGH`/`FALLBACK` 非对称窗口只决定拓扑资格和诊断，不命令正常的媒体层切换，也不计算综合分数。SFU BWE downshift 先进入 `suspect`；多个连续窗口的 sender/viewer 关联证据确认 `FALLBACK` 后，应用不得稳定保留该 parent，并把 downstream capacity 降为 0。计划或显式质量 fallback 仅是内建路径失败后的候选：已有 children 时先在 connection/assignment generation guard 下全部迁走，失败或代次过期则保持原 assignment/quality。Autonomous downshift 无法 packet-level 预先迁移，default-on 前必须验收 descendants 的短暂影响。只有更长的稳定恢复窗口和 cooldown 后才能重新声明 relay capacity。Viewer 请求必须鉴权、限频、去重，并由 sender 的 GCC/Transport-CC、实际码率、RTT、丢包、重传、编码限制、协商 codec/profile/parameters 与 viewer 解码、丢帧、冻结证据交叉验证；不能仅凭自报生效，也不得上传 raw SDP、candidate 地址或原始设备/网络标识。 - [ ] 断线恢复：临时网络变化时尝试 ICE restart 和重连，并向用户显示当前连接状态。 - [ ] 可观测性：按目标 RTP transport 与不透明连接/ICE 代次关联 selected path、`iceGatheringState`/ICE/connection state、事件推导的建立时长、generation-baselined pair-change delta，以及 RTT、RTP 丢包率、抖动、码率、帧率、编码耗时和受限原因。候选与错误的 wire、日志和导出只使用[低服务器成本媒体路由](./research/low-server-media-routes.md)的脱敏字段和 code bucket，禁止 raw SDP、candidate、地址、URL、`errorText` 或网络标识。只有创建该 `RTCPeerConnection` 的本机可在折叠详情中显示浏览器对 selected local/remote candidate 实际暴露的地址与端口；该值不上传、不持久化、不用于地理/NAT 推断，缺失或被隐私保护时显示未知。 - [ ] 渐进式诊断：分享者和观看者默认只看到房间、画面、核心状态和常用控制；候选路径、RTT、丢包、编解码器、编码耗时等技术详情由一个明确的“显示连接详情”开关展开。隐藏诊断不能隐藏失败原因、音频缺失、持续画质受限或需要用户操作的告警。 @@ -40,9 +40,9 @@ - [ ] 质量控制扩展：推荐档位仍是默认入口；折叠高级区只允许 720p/1080p/1440p 分辨率 ceiling、15 至 60 的整数 FPS ceiling、2 至 12 Mbps 视频码率 ceiling，以及清晰/平衡/流畅三种偏好。直播中显式应用后才生效，避免滑杆产生连续控制流量。当前 Web 发送端不提供音频声道、采样率、Opus 码率、stereo、DTX 或 FEC 控件：display capture 未把声道和采样率定义为可移植控制，标准音频 `maxBitrate` 只是可能导致更低质量或停播的发送上限，其他项依赖 codec/fmtp 协商而没有可移植 sender setter。除非真实音频矩阵证明用户问题、标准 API 能应用并读回且实际结果可独立观测，否则不得为这些控件新增应用层 SDP munging 或用默认值制造伪控制。实际质量持续低于所选上限时必须显示可解释的限制原因；在真实拥塞、编码器和接收端证据完成前，不把任一组合称为“完美平衡”。 - [ ] 本地预览性能候选：在相同动态场景中 A/B 测量 Host 动态预览开/关的 CPU、GPU、encode time 和实际 FPS。只有稳定收益且捕获、编码和远端画面不受影响时，才允许直播开始后默认显示静止缩略图或隐藏动态预览并提供按需展开；未测前不宣称收益。 - [ ] Peer-assisted 媒体实验：分享端最多两个 child、每名 viewer 最多一个 child；信令服务只按深度和服务端加入序号确定性地形成两条 sticky 平衡链，不加入综合评分或主动重平衡。每个 viewer 将收到的标准 remote track 通过新的 `RTCPeerConnection` 转发，因此会逐跳解码和重编码；必须测量而不能称为 shared encode。服务端只在受房间数限制的内存状态中保存当前严格校验的质量设置，不写入 SQLite；host 鉴权后先重报设置，在线 viewers 接收更新，后加入者从鉴权快照取得当前值。该功能在 ADR-0004 的实验门槛通过并由后续 ADR 正式接受前不得成为产品路径。 -- [ ] 原生 shared encode 分享端：后续独立的 Electron/原生分享端必须让每个 active representation 只产生一份兼容编码结果，并在 direct、peer 与 SFU transport 间复用；健康时只有共享 `HIGH`，弱路径期间最多再有一份共享 `LOW`。每条连接仍保留独立 WebRTC packetization、SSRC、pacing、加密和反馈，且各自消耗上传带宽。浏览器不保证跨 `RTCPeerConnection` 复用编码器，因此当前 Web spike 不能宣称满足该保证。 +- [ ] 原生 shared encode 分享端：后续独立的 Electron/原生分享端必须让每个 active representation 只产生一份兼容编码结果，并在 direct、peer 与 SFU transport 间复用；健康路径选择共享 `HIGH`，最多另有一份弱路径共用的 `LOW`，是否常驻由硬件、游戏和上传门槛决定。每条连接仍保留独立 WebRTC packetization、SSRC、pacing、加密和反馈，且各自消耗上传带宽。浏览器不保证跨 `RTCPeerConnection` 复用编码器，因此当前 Web spike 不能宣称满足该保证。 - [ ] 原生 shared encode relay：具备原生 relay 能力的 audience 同样最多维护两条下游媒体边，并应复用兼容编码输出而不再次编码。普通浏览器 audience 继续作为 leaf 或使用当前会逐跳重编码的实验路径，不伪称 shared encode。 -- [ ] 条件式 SVC：标准能力按 simulcast、LiveKit/Dynacast、SVC 顺序短路，并可与 ADR-0006 native sender 复核并行；静态 API 或 pinned 协议不满足可在完整 A+B/C 前否决，只有 runtime media/resource gate 必须等待可信 A+B/C。前两项已静态 no-go；SVC 满足独立选层、requested/applied codec/`scalabilityMode`、hardware/power-efficient、真实游戏性能和亚秒延迟门槛时即可采用并停止自研，不得静默回退为高负载软件 SVC。它不是预选默认方案，未来严格单输出要求只会加强而不是创建采用理由。 +- [ ] SVC 重开条件：当前标准 Web/LiveKit SVC 已静态拒绝为 `no-go-web-svc-cross-path-hardware-contract`。sender `scalabilityMode` 没有 direct/peer receiver 对应的按层选择或跨 PeerConnection shared-encode 契约；requested/applied mode、Media Capabilities `powerEfficient` 和 RTCStats encoder 字段也不能证明没有静默软件 fallback。Pinned client 2.22.0 还会把 screen-share SVC 覆盖为三层 temporal-only `L1T3`，不满足低分辨率 base 与最多两层。只有未来 native sender 能明确选择硬件编码器、最多发布两层、将同一编码输出复用于 direct/peer/SFU、保持每条路径独立选择并通过真实游戏和亚秒延迟门槛时，才可用新的 capability decision 重开 SVC；不能把浏览器 API 支持或 UA 推断当成采用理由。 - [ ] 分包/分层多树实验：把编码包、对象或可独立解码层拆到多个 peer 路径以分散上传，只能在独立 spike 中验证。必须保持亚秒级，处理多父重组、排序、丢包恢复、关键帧、同步和节点退出；未通过门槛前不进入产品路由控制器。 - [ ] SFU-root 中央兜底：产品部署目标必须提供经 canary 验收的单节点 SFU 资源；系统在 direct/peer UDP 不可用或明确拓扑预算无法接纳时自动把它作为主要中央兜底，不要求用户选模式。它只向总计一至两个必要 roots 出口；root 优先承担受限 peer descendants，没有可靠 relay root 时必要 viewer 只作为同一总上限内的零 descendants root。已关闭 PR #12 的显式整房 SFU 模式已被取代。当前 PR #17 实现仍是 default-off、failure-only 且生产未配置；按 ADR-0005 的 exact-room canary 验收并修改配置契约后，才能启用新的默认梯级。 - [ ] 分享端桌面封装：用 Electron 固定 Chromium 版本、改善选源/热键/系统音频体验，同时保留 Web 观看端。 @@ -123,16 +123,18 @@ - Given peer-assisted host 鉴权或直播中连续修改质量设置，When 服务端校验其分辨率、FPS、码率和偏好，Then 只有 host 的完整有界设置被接受，last-wins 地应用到当前和以后建立的 relay child sender 以及启用的 SFU publisher；越界、缺字段和多余字段被拒绝，普通 P2P 的鉴权 wire 不新增该字段且不接受同步消息。 - Given 标准浏览器把收到的 remote track 直接加入下游 `RTCPeerConnection`，When 检查 relay 统计和 libwebrtc 路径，Then 明确记录新的 outbound encoder、编码耗时和 CPU/GPU 限制，不把它当作 encoded forwarding 或 shared encode 成果。 - Given ADR-0004 的任一硬门槛只有在引入 Encoded Transform/DataChannel/WebCodecs 媒体、自定义拥塞控制、FEC/RTX、多树、综合评分、转码、codec ladder、浏览器 RTP 注入 hack 或放宽 host fanout 后才能满足，When 结束 bounded spike，Then 放弃并删除浏览器 relay 实验运行时代码，保留标准 P2P 和 SFU-root fallback；后续 native shared encode sender 仍按独立 ADR 推进，但不能据此把失败的 relay 拓扑判为通过。 -- Given 新观看者会使预计总上行超过安全预算，When 其请求加入，Then 系统先遵守两边 fanout 预算选择 peer-assisted 或已配置的 SFU；没有合格路线时明确等待或拒绝，而不是创建第三条 host 边或持续堆积发送队列。质量不足只对经证据确认的观看路径使用共享 `LOW`，不能降低健康路径。 +- Given 新观看者会使预计总上行超过安全预算，When 其请求加入，Then 系统先遵守两边 fanout 预算选择 peer-assisted 或已配置的 SFU；没有合格路线时明确等待或拒绝，而不是创建第三条 host 边或持续堆积发送队列。Direct/peer GCC 与 SFU BWE 各自适应该路径，应用证据只改变拓扑资格，不能合成全房降档。 - Given TURN 或 SFU 被某条路线使用，When 媒体已抵达一至两个可靠 root，Then 其余可行 viewers 仍由 sticky、受深度与故障半径限制的 peer descendants 服务；host/root fanout 不超过二，当前 browser relay 不超过一个 child，中央服务和任一 endpoint 都不得无界集中 fanout。 - Given 没有任何满足能力、深度、路径和恢复门槛的可靠 relay root，When 控制器穷尽 peer recovery，Then SFU 可把必要 viewers 作为零-descendant roots；正常仍限一至二 roots，多个异常 viewers 只能使用另行配置的中央 egress cap，否则明确等待或失败，不得变成无界整房 SFU。 ### 质量和恢复 - Given 用户选择任一推荐档位或高级质量参数，When 接收端实际分辨率、帧率或码率持续显著低于该目标，Then 诊断数据能区分采集约束、发送编码、带宽/拥塞、TURN 路径和接收端限制；产品不得只改变档位标签却继续隐藏实际降级。 -- Given 房间所有观看路径持续健康，When 自动质量控制运行，Then 只存在一套 active `HIGH` representation，且任何单路径短暂抖动或未经验证的低档请求都不启动 `LOW` 或降低其他路径；物理编码器实例数另行实测。 -- Given 一个或多个路径在连续窗口中被 sender 与 viewer 证据共同确认无法承受 `HIGH`，When 它们进入 `FALLBACK`，Then 系统必须启动且最多只启动一套共享 `LOW`，只迁移这些弱路径，其他路径继续 `HIGH`；所有弱路径经过更长稳定恢复窗口后返回 `HIGH` 并关闭 `LOW`。 -- Given 启动 `LOW` representation 的实测 encoder/CPU/GPU 成本超过游戏预算或没有合格 hardware/power-efficient 路径，When 弱路径请求 `LOW`，Then 当次保留健康路径的 `HIGH` 并向弱路径明确显示降级或不可用，不得退回全房间最低目标或静默软件编码；若 supported sender cohort 无法可靠按需启动 `LOW`，Then 自动质量控制验收失败，必须优化编码路径或明确标记不支持，不能把 fail closed 当作长期忽略弱路径的许可。 +- Given 一个健康 SFU leaf 和一个受控弱 SFU leaf 都以 `HIGH` ceiling 订阅，When 内建 LiveKit BWE 运行，Then 弱 leaf 自动使用唯一共享 `LOW` 并在恢复后回到 `HIGH`，健康 leaf 不受影响，且 Screener 没有实现媒体层 selector；direct/peer 继续使用各自的 stock WebRTC GCC。 +- Given 一个路径的 sender/viewer 证据跨连续窗口确认 `FALLBACK`，When 应用更新状态，Then 只改变其 parent/leaf 拓扑资格和诊断；不得重复命令内建媒体层。更长稳定恢复窗口和 cooldown 后才可重新声明 relay capacity。 +- Given 内建路径失败后才启用显式质量 fallback，When 一个当前承载 children 的 relay 准备切换，Then 必须先在匹配 generation 下迁走全部 children；失败或代次过期保持原状态，且 Viewer 自报不能单独触发该过程。 +- Given SFU BWE 在应用确认前把一个仍有 children 的 root 瞬时降层，When 收到该观测，Then 先标记 `suspect`；只有跨有界连续窗口的关联证据确认后才迁走 children 并进入 `FALLBACK`，不得稳定保留 confirmed `FALLBACK` parent。默认启用前必须在 root-with-children gate 中记录并限制这段不可避免的 descendants 短暂画质影响，不能声称 packet-level 先迁后降。 +- Given exactly-two publication 的实测 encoder/CPU/GPU 或上传成本超过游戏预算，或没有合格 hardware/power-efficient 路径，When 评估该候选，Then 保留健康路径并明确显示弱路径降级或不可用，不得退回全房间最低目标或静默软件编码；若 supported sender cohort 无法以常驻或按需方式提供共享 `LOW`，Then 质量验收失败，必须优化编码路径或明确标记不支持。 - Given 同一时刻采集、host outbound 与 viewer inbound 证据，When 进行根因分类，Then 关联派生后的实际协商视频 codec/profile/parameters、适用时的 `scalabilityMode` 与真实解码表现，依次区分 Capture 低、Capture 高但 outbound 因 CPU 低、Capture 高但 outbound 因 bandwidth 低、outbound 正常但 inbound 低、以及 inbound 正常但肉眼模糊五类；不用 UA 或设备型号代替运行时证据，也不上传 raw SDP。 - Given Viewer 发送低档请求，When 请求未鉴权、过频、重复、过期或没有匹配的 sender/viewer 统计，Then 请求不改变表示或路径状态；有效请求也只能作为交叉验证输入。 - Given 受控 RTT/丢包测试条件，When 连续分享高动态游戏画面 30 分钟，Then 延迟、帧率与丢帧达到上述目标，且没有不可恢复的冻结。