From b4548c88c5186a71fa9c0d2fb57fefb03f887f4d Mon Sep 17 00:00:00 2001 From: AAA <35992542+TNTcraftHIM@users.noreply.github.com> Date: Wed, 19 Aug 2026 20:18:08 +0800 Subject: [PATCH 1/2] docs(media): define deployed quality and audio gates --- ...mand-driven-dual-representation-quality.md | 22 +++--- docs/project-memory.md | 17 ++--- docs/research/browser-screen-audio-quality.md | 69 +++++++++++++++++-- docs/research/native-shared-encode-sender.md | 9 +-- docs/research/realtime-quality-adaptation.md | 67 ++++++++++++++++-- docs/status.md | 35 ++++------ docs/方案设计.md | 18 +++-- docs/需求理解.md | 14 ++-- 8 files changed, 183 insertions(+), 68 deletions(-) diff --git a/docs/adr/0007-demand-driven-dual-representation-quality.md b/docs/adr/0007-demand-driven-dual-representation-quality.md index fa0d845..cc91700 100644 --- a/docs/adr/0007-demand-driven-dual-representation-quality.md +++ b/docs/adr/0007-demand-driven-dual-representation-quality.md @@ -107,9 +107,9 @@ but runtime encode/decode behavior is authoritative. ## Deferred Capability Spikes These spikes start only after the A+B/C evidence contract is trustworthy. They -may proceed independently of ADR-0006 native-sender revalidation and must run -before any custom dual-representation media implementation. They do not change -the accepted two-state controller: +run in order and stop at the first accepted path, may proceed independently of +ADR-0006 native-sender revalidation, and precede any custom dual-representation +media implementation. They do not change the accepted two-state controller: 1. Negotiate exactly two `HIGH`/`LOW` simulcast encodings on one sender in its initial envelope, with `LOW` inactive. Verify requested/applied parameters, @@ -130,22 +130,22 @@ the accepted two-state controller: on-demand-`LOW` requirement. 3. Run a bounded SVC viability spike and compare the applied codec/`scalabilityMode` with Media Capabilities `powerEfficient` and the - game-performance matrix. This avoids prematurely building a custom second - representation; product adoption still requires a future strict-one-output - requirement. Reject silent software fallback. + game-performance matrix. If simulcast and LiveKit/Dynacast fail while this + path meets current on-demand selection and resource gates, adopt it and stop + before custom media. Reject silent software fallback. None may reuse PR #28's minimum-of-two target, bypass a per-edge stock WebRTC congestion controller, or expand the representation limit beyond two. ## SVC Boundary -The bounded spike may measure SVC now, but adopt it only when a later accepted -requirement demands one encoded output at all times. A path may then receive only the base layer or the base plus -enhancement layers. Enable it only when the exact negotiated codec and +SVC is the third standard candidate after simulcast and LiveKit/Dynacast. A path +may receive only the base layer or the base plus enhancement layers. Adopt it +when the earlier candidates fail and the exact negotiated codec and `scalabilityMode` are read back, a power-efficient/hardware path is positively established on the supported sender cohort, and game-performance and latency -gates pass. Do not silently fall back to software SVC. Persistent SVC is not the -default design. +gates pass. Do not silently fall back to software SVC. It is not the preselected +default; a strict-one-output requirement would strengthen, not create, its case. The WebRTC-SVC specification exposes `scalabilityMode` and permits the browser to return a different configured mode after negotiation. Media Capabilities can diff --git a/docs/project-memory.md b/docs/project-memory.md index d1f2a89..adbe22a 100644 --- a/docs/project-memory.md +++ b/docs/project-memory.md @@ -8,25 +8,26 @@ Last updated: 2026-08-19 - Viewers join from normal desktop/mobile browsers. A later native sender may share encoding and improve capture/audio without changing the Web viewer requirement. - Use a small central service for access, rooms, signaling, deterministic topology, STUN, observability, and bounded SFU-root capacity; TURN is optional extreme-network transport. - Minimize server bandwidth. The accepted invisible ladder is direct/peer UDP, then SFU/UDP roots, then optional TURN for a selected exceptional edge, then bounded failure. TURN is transport, not topology. -- Non-server nodes have at most two downstream edges; browser relays stay at one until resource gates pass. SFU normally feeds one or two roots that retain peer descendants; separately capped server edges may serve multiple exceptional viewers that cannot attach behind a healthy root. Native/object paths should reuse one encode while browsers re-encode. +- Non-server nodes have at most two downstream edges; browser relays stay at one until resource gates pass. SFU normally feeds one or two roots that retain peer descendants; separately capped server edges may serve exceptional viewers that cannot attach behind a healthy root. - Two-tree packet/layer striping may reduce endpoint upload toward one stream bitrate, but needs a bounded multi-parent, loss, sync, churn, and latency experiment; it is not in the current full-stream chains. -- Shared encoding reduces compute, not each viewer's last-hop copy. Automatic quality uses one active `HIGH` representation plus at most one on-demand `LOW`; all verified weak paths share `LOW` and recovery stops its bytes/frames. This does not prove one/two physical encoders or GPU release. An over-budget media path fails closed for that attempt while protecting `HIGH`, but a supported cohort that cannot start `LOW` reliably fails acceptance. SVC has no software fallback. +- Flagship native media encodes each active representation once and reuses it across direct, peer, and SFU transports; per-connection packetization, pacing, encryption, feedback, and upload remain independent. Healthy rooms have one shared `HIGH`; verified weak paths may add one shared on-demand `LOW`, never per-viewer encoders. Test simulcast, LiveKit/Dynacast, then SVC and stop at the first standard path that meets on-demand/resource gates; SVC has no software fallback. - Keep decisions, snapshots, research, code, `AGENTS.md`, and `.codex/` in Git; rewrite memory/status in place. Research current primary sources before material work and reject speculative machinery. +- Migrate client, server, and deployment atomically. After a canary, delete superseded config/wire/parsers/tests; do not retain compatibility layers, dual writes, or a second architecture without a current consumer. Git history owns the old implementation. ## Current Recommendation - Keep desktop Chrome/Edge and the responsive Web viewer as the baseline. Validate Android Chrome and iOS Safari as leaves. - Use direct host P2P for one or two viewers. Keep ADR-0004 off broadly; use an exact-room canary until resource, quality, recovery, SFU/UDP, optional-TURN, and browser/mobile gates pass. - Browser relays resend remote `MediaStreamTrack` values and re-encode at each hop; WebRTC does not guarantee a shared encoder across peer connections, so measure the cost. -- Keep browser experiments bounded; spike native RTP relay, encoded-object striping, SVC, and FEC separately. -- ADR-0006 retains a fixed-`HIGH` native canary boundary, but its sole product-wiring run reached only host setup and one encoder output before the first-viewer decoded/rendered gate timed out. Downstream checkpoints were not retained, so the result is no-go-unclassified, not a diagnosed product bug. No native product code is accepted; revisit only through the staged evidence gate. +- Keep experiments bounded: the standard representation sequence is below; native RTP relay, encoded-object striping, and FEC remain separate. +- ADR-0006's fixed-`HIGH` canary reached host setup and one encoder output but retained no downstream checkpoint before the first-viewer timeout. It is no-go-unclassified; no native product code is accepted, and any revisit starts at its staged evidence gate. - ADR-0005 accepts SFU roots as the primary central fallback after direct/peer UDP, never default whole-room fanout. Optional TURN is issued only to an exceptional assigned edge; PR #12 is superseded. - Keep the current failure-only controller default-off until config migration and exact-room gates pass. Preserve break-before-make, sticky healthy subtrees, mobile leaves, and PR #20 prewarm; public transport/load remains open. - Local reparenting is a later candidate: start with unassigned relay admission rescue; do not block current work. - Flagship media defaults to UDP; HTTPS/WSS remains TLS/TCP. Current production still requires coturn UDP/TCP. Future TURN absence is normal, partial config fails, and any media-TCP/port compatibility mode is chosen only by canary. -- Treat video settings as ceilings and degradation as unclassified. Local host A+B now covers same-tick capture/outbound identity, windows, deltas, remote linkage, path, and nullable negotiated codec/profile token/allowlisted parameters/current stream `scalabilityMode`. Use it first to reproduce the reported Host-only recovery, then add minimal authenticated C and test standard simulcast/LiveKit/SVC before custom `LOW`; this need not wait for ADR-0006. Never use UA or a composite score. -- Reported poor movie/video audio is unclassified. After video A+B, separately diagnose audio A/B/C and A/V sync across capture settings, codec/fmtp, actual bitrate, loss, jitter, concealment, and jitter buffer. `maxBitrate` is not quality-up; do not expand runtime before evidence. See `docs/research/browser-screen-audio-quality.md`. -- Do not add custom scene detection or dynamic-FPS control until WebRTC statistics and host resource measurements prove a material gap. Keep browser codec order until target hardware proves a more efficient common codec. +- Treat settings as ceilings and deployed degradation as unclassified. Use local Host A+B to compare exact `769de201f7cc` against current `main` for Host-only recovery, game FPS/load, preview cost, actual codec/encoder and one-variable rebuilds; then add authenticated C and test standard simulcast/LiveKit/SVC before custom `LOW`. Never use UA or a composite score. +- Production reports poor film audio and self-echo when system capture includes voice software. Diagnose audio A/B/C and sync; Web cannot isolate arbitrary processes and `maxBitrate` is not quality-up. A Windows 11 native candidate defaults to game-process-tree audio and never widens silently; Windows 10 remains unresolved/unsupported. See `docs/research/browser-screen-audio-quality.md`. +- Do not add scene detection, dynamic-FPS control, or forced AV1 until stats and target hardware prove the need. Codec acceptance requires actual negotiation, power-efficient candidate evidence, interval encode cost, game FPS, CPU/GPU and sender count; Discord's native capture/hardware tuning is comparison evidence, not proof of server re-encoding or a reusable preset. - Keep room policy deployment-driven: public/password-only rooms are random and temporary; password plus SQLite enables sequential persistent rooms and reusable stopped links. - Deferred architecture audit: `docs/maintenance.md`. @@ -46,7 +47,7 @@ Last updated: 2026-08-19 - Chrome 151 short synthetic `1/3/5/8` runs kept host edges at two, relay edges at one, and all viewers decoding; a three-viewer relay close recovered in 5.32 seconds. This proves topology/control only, not quality, load, endurance, or silent partitions. - The current quality stack uses one strict, non-persistent `QualitySettings` value with bounded ceilings and clarity/balanced/fluid preference. Sender readback and limitation warnings remain manual diagnostics. ADR-0007's automatic per-path `HIGH`/`FALLBACK` and unique on-demand shared `LOW` are accepted mainline behavior, but the C report, state controller, and `LOW` runtime are unimplemented. - Chrome 151 synthetic 720p30 propagated balanced/clarity to one host and three viewers without changing peer identities; all viewers kept decoding and fanout stayed host two/relay one. This proves control continuity only. -- Local host A+B binds capture settings to one uniquely matched outbound RTP stream in the same tick, follows `remoteId` and the RTP-specific transport/candidate pair, and blocks stale generations across source replacement. Adjacent `getStats()` deltas rebase on first, empty, ambiguous, changed-identity, or reset intervals. The same RTP's transport-bound codec supplies only nullable MIME, self-describing profile token, allowlisted format parameters, and current stream `scalabilityMode`; this diagnostic adds no upload or persistence of raw fmtp/SDP/stats, defaults are not inferred, profile tokens imply no quality/hardware/support conclusion, and multiple RTP/encodings remain unknown. Authenticated C, automatic control, and `LOW` remain unimplemented. +- Local Host A+B aligns current capture settings with one unique outbound RTP/remote/transport pair per tick, rebases deltas on ambiguity, identity change, or reset, and rejects stale source generations. Codec evidence is nullable and transport-bound; no defaults or profile capability are inferred, raw fmtp/SDP/stats are not retained, and multiple RTP/encodings remain unknown. Authenticated C, automatic control, and `LOW` remain absent. - Release `769de201f7cc` passed CI/hygiene/typecheck, 22 files/274 tests, both builds, zero production dependency vulnerabilities, atomic activation, public health/access, exact bundle, and service/log checks. ## Provisional Quality Targets diff --git a/docs/research/browser-screen-audio-quality.md b/docs/research/browser-screen-audio-quality.md index 4c36b88..49dee8d 100644 --- a/docs/research/browser-screen-audio-quality.md +++ b/docs/research/browser-screen-audio-quality.md @@ -20,10 +20,27 @@ The Web sender keeps the current minimal behavior: - expose no channel-count, sample-rate, Opus bitrate, stereo, DTX, or FEC quality control. -No runtime or UI change is justified. In particular, Screener will not add -application-owned SDP munging for these controls or add a control whose value -cannot be separated into requested, applied, negotiated, and observed states -using standard APIs. +No codec-quality control is justified yet. In particular, Screener will not add +application-owned SDP munging or a control whose value cannot be separated into +requested, applied, negotiated, and observed states using standard APIs. The +deployed reports below do justify an explicit source-scope acceptance gate and +a later native process-audio candidate; neither is a bitrate knob. + +## Deployed Reports And Echo Boundary + +Users report poor movie/video audio and delayed self-echo when production +`769de201f7cc` captures whole-system audio while a voice application is active. +These reports apply to that deployed release and remain unclassified until the +same fixture compares it with current `main`. The release sets no application +audio bitrate, channel, sample-rate, or SDP option; its Host preview is muted, +and peer/SFU relay is disabled. Therefore the current evidence does not point +to an application ceiling, Host preview playback, or relay re-encoding. + +The echo case is source-content leakage: a viewer's voice is rendered by a +separate voice application on the Host, enters the system mix, and is captured +and sent back. It is not ordinary microphone acoustic echo. AEC, noise +suppression, or post-mix Web Audio cannot recover the originating process and +must not be presented as a fix. ## Standard API Boundary @@ -31,6 +48,7 @@ using standard APIs. | --- | --- | --- | --- | --- | | Audio presence | `getDisplayMedia({ audio: true, video: ... })` expresses interest | `stream.getAudioTracks().length` proves only whether a track was returned | The browser may still return video only; a track does not identify system, window, tab, or selected-game audio | Keep the existing presence check and visible no-audio warning | | Audio source choice | `systemAudio`, `windowAudio`, and `audioSelection` are picker hints that a user agent may ignore | There is no standard audio-source category readback corresponding to those hints | The selected audio scope, per-application isolation, or cross-browser availability | Do not present these hints as quality or source guarantees | +| Voice-app exclusion | `restrictOwnAudio` concerns audio produced by the document that invoked capture; `suppressLocalAudioPlayback` concerns local playback of a captured browser surface | The app may observe whether a returned track exists, not which OS processes it contains | Excluding Discord, KOOK, WeChat, notifications, or any other independent process from system audio | Web keeps video-only available and warns that system audio may include calls/notifications; it does not claim isolation | | Capture channels | The Screen Capture specification does not list generic `channelCount` as applicable to display audio | `track.getSettings().channelCount` may describe the returned track when the browser supplies it | That the app controlled the value, or that the RTP encoder sends stereo | Observe only in a future diagnostic; absent means unknown | | Capture sample rate | The Screen Capture specification does not list generic `sampleRate` as applicable to display audio | `track.getSettings().sampleRate` may be present in an implementation | The Opus mode, RTP clock semantics, receiver output rate, or end-to-end fidelity | Observe only in a future diagnostic; absent means unknown | | RTP send bitrate | `RTCRtpSender.setParameters()` can set `encodings[].maxBitrate` for audio | A following `getParameters()` can show the applied ceiling; outbound byte deltas show actual traffic | A target, minimum, quality increase, or Opus `maxaveragebitrate`; other limits may keep traffic lower | Do not add a control without a measured need to reduce audio bandwidth | @@ -133,13 +151,47 @@ If audio diagnostics become necessary, keep four layers separate: when the browser returns numeric values. 2. **Requested/applied:** an RTP ceiling only if Screener actually calls `setParameters()`, followed by immediate `getParameters()` readback. -3. **Negotiated:** the in-use codec MIME type and raw fmtp from `RTCCodecStats`, - labelled as negotiated parameters rather than actual stereo/DTX/FEC. +3. **Negotiated:** the in-use codec MIME type and bounded derived fmtp evidence + from `RTCCodecStats`, labelled as negotiated parameters rather than actual + stereo/DTX/FEC; raw SDP/fmtp is not uploaded or persisted. 4. **Observed:** interval audio bitrate from byte deltas and receiver FEC counters when implemented; missing values remain unknown. Never substitute `48000`, `2`, `false`, or `0` for an unavailable field. +## Native Windows Process-Audio Candidate + +The smallest honest native path is Windows WASAPI application loopback through +`ActivateAudioInterfaceAsync` and `AUDIOCLIENT_PROCESS_LOOPBACK_PARAMS` with +`INCLUDE_TARGET_PROCESS_TREE`. After the user selects a game window, the sender +resolves its PID locally and captures that process plus children. The API's +minimum build 20348 is not available on ordinary Windows 10 22H2 (19045), so +the current product candidate covers Windows 11 only. Windows 10 game-only +audio remains unresolved or explicitly unsupported until another bounded path +passes. + +Default native behavior is game-only. Whole-system audio requires an explicit +choice. No render stream, process exit, or capture failure produces a visible +silent/failed state and never widens capture to the whole system. Excluding one +known voice process is not a sufficient default because other calls, +notifications, or browser audio could still leak. Protected content may be +absent and an exclusive/no-render path may yield silence rather than a +classifiable error; the sender reports silence or unknown and never bypasses +protection. PID, path, window title, device identity, and raw audio remain local +and are not retained or uploaded. + +Run one bounded matrix rather than a full route Cartesian product: exact +production and current `main` on Windows Chrome/Edge for tab/window/monitor, +audio selected/unselected, and a simultaneous voice call; then the native +candidate on current Windows 11 with game parent and child audio, an independent +voice process, notifications, no render stream, and process restart. Windows 10 +records the explicit unsupported/unresolved result rather than a fake fallback. +Direct is the primary route; one browser-relay and one +SFU-root audio check cover route preservation, while TURN needs only a +connectivity smoke. Correlate capture settings, negotiated codec/derived fmtp, +actual outbound/inbound bitrate, loss, jitter, concealment, jitter buffer, and +A/V playout timing using a distinguishable stereo fixture plus game/film audio. + ## Revisit Gates Add a user-facing audio setting only when all of these are true: @@ -159,6 +211,7 @@ Add a user-facing audio setting only when all of these are true: - [W3C Screen Capture](https://www.w3.org/TR/screen-capture/) - [W3C WebRTC](https://www.w3.org/TR/webrtc/) - [W3C WebRTC Statistics](https://www.w3.org/TR/webrtc-stats/) +- [W3C Web Audio](https://www.w3.org/TR/webaudio-1.1/) - [RFC 7587: RTP Payload Format for Opus](https://www.rfc-editor.org/rfc/rfc7587.html) - [RFC 7874: WebRTC Audio Codec and Processing Requirements](https://www.rfc-editor.org/rfc/rfc7874.html) - [MDN `getDisplayMedia()`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia) @@ -168,3 +221,7 @@ Add a user-facing audio setting only when all of these are true: - [Chrome screen-sharing controls](https://developer.chrome.com/docs/web-platform/screen-sharing-controls) - [Chrome 141 window-audio release note](https://developer.chrome.com/release-notes/141) - [Chromium media-capture architecture index](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/media/capture/) +- [Microsoft Application Loopback sample](https://learn.microsoft.com/en-us/samples/microsoft/windows-classic-samples/applicationloopbackaudio-sample/) +- [Windows process-loopback parameters](https://learn.microsoft.com/en-us/windows/win32/api/audioclientactivationparams/ns-audioclientactivationparams-audioclient_process_loopback_params) +- [Windows 10 release information](https://learn.microsoft.com/en-us/windows/release-health/release-information) +- [Microsoft WASAPI loopback recording](https://learn.microsoft.com/en-us/windows/win32/coreaudio/loopback-recording) diff --git a/docs/research/native-shared-encode-sender.md b/docs/research/native-shared-encode-sender.md index f30f65c..c010f65 100644 --- a/docs/research/native-shared-encode-sender.md +++ b/docs/research/native-shared-encode-sender.md @@ -200,10 +200,11 @@ identity do not participate in quality control. A native relay forwards the selected encoded packets and must not decode or re-encode them. An ordinary non-scalable representation cannot be forwarded into a second quality; the bounded choices are a temporary second encode, SVC, -or transcoding. The default is the temporary second representation. SVC is -considered only for a later strict-one-output requirement and only after the -exact applied codec/mode, available encoder-implementation/native evidence, and -the real game/power matrix establish the path. Media Capabilities or RTCStats +or transcoding. Test standard simulcast, LiveKit/Dynacast, then SVC and stop at +the first path that meets on-demand selection and resource gates. SVC still +requires the exact applied codec/mode, encoder/native evidence, and the real +game/power matrix; a future strict-one-output requirement strengthens but does +not create that case. Media Capabilities or RTCStats power-efficiency signals are admission evidence, not hardware proof; an unestablished path stays disabled rather than silently using software. Future dual-tree/striped distribution may reduce two-copy host upload toward one copy diff --git a/docs/research/realtime-quality-adaptation.md b/docs/research/realtime-quality-adaptation.md index b166a25..d68724a 100644 --- a/docs/research/realtime-quality-adaptation.md +++ b/docs/research/realtime-quality-adaptation.md @@ -88,6 +88,47 @@ are ceilings rather than targets. Raising them cannot repair CPU or bandwidth limitation and should only follow evidence that the encoder is already pinned to the ceiling while spare transport capacity remains. +The same production release is also reported to reduce game-stream frame rate +and consume noticeable Host resources. That report applies only to deployed +`769de201f7cc`, not automatically to the newer diagnostics on `main`. The Web +sender creates one independent `RTCRtpSender` per viewer and has no cross-PC +shared-encoder guarantee; its muted local preview creates no media edge or +server traffic but may still consume compositor/GPU work. Compare the exact +release and current `main` under one fixture, with preview on/off as a separate +binary intervention. + +## Codec And Hardware Gate + +Do not default to AV1 from compression efficiency alone. The deployed release +does not call `setCodecPreferences()`, and a browser may expose a negotiable +codec without a power-efficient WebRTC encoder. `getCapabilities()` establishes +only an optimistic negotiation set; Media Capabilities supplies +supported/smooth/power-efficient candidate evidence; outbound `codecId`, +`encoderImplementation`, and `powerEfficientEncoder` describe the stream only +when the browser exposes them. Final acceptance still requires interval encode +cost, game FPS, CPU/GPU video-encode activity, and active sender count. Chromium +`main` on the access date gates WebRTC AV1 hardware encoding off by default on +Windows even when a platform accelerator exists; this is an implementation +snapshot, not a permanent browser contract. + +First diagnose the Host-refresh case using the browser-selected codec. Only if +encode cost or `cpu` limitation is abnormal, hold scene, resolution/FPS/bitrate, +network, and viewer constant while comparing browser default, H.264, and VP8. +VP9 or AV1 enters that spike only when both endpoints can negotiate it and +Media Capabilities reports the exact configuration supported, smooth, and +power-efficient. A preference change must retain negotiated repair codecs and +is successful only when outbound stats prove the codec actually in use; silent +software fallback fails the gate. + +Discord's published Go Live material is a useful architecture comparison, not +a preset to copy. It describes native OS/driver-integrated capture and encoding, +GPU hardware encoding, WebRTC transport, and product-specific rate-control +tuning. It also documents a feedback loop that could lock 60 fps output down to +30 fps. This supports measuring the complete capture/encoder/congestion loop; +it does not show that Discord servers re-encode each viewer stream or that AV1 +is universally cheaper. No equivalent first-party implementation evidence was +found for KOOK or Oopz, so they are not used as design facts. + ## Evidence Before Adaptation Verified specification facts: the W3C stats model supports the needed @@ -212,19 +253,20 @@ capacity. An ordinary non-scalable stream cannot yield a second independent quality by packet forwarding alone. The alternatives are a second representation, scalable layers, or relay/SFU transcoding. Screener tests standard capabilities -before custom media. The bounded SVC spike is scheduled now, but product -adoption remains conditional on a future strict-one-output need, an exact -negotiated mode, a positively established hardware or power-efficient -path, and measured game performance. WebRTC-SVC permits the browser to return a +before custom media. SVC is the third short-circuit candidate when simulcast +and LiveKit/Dynacast fail; adoption requires an exact negotiated mode, a +positively established hardware or power-efficient path, and measured game +performance. WebRTC-SVC permits the browser to return a different configured `scalabilityMode`; Media Capabilities reports support and expected smoothness/power efficiency for a specified configuration; WebCodecs defines `hardwareAcceleration` only as a hint the user agent may ignore. Therefore none is, by itself, proof of a particular hardware encoder, and a software SVC fallback must not be silent. -After trustworthy A+B/C, run three bounded capability spikes before custom -dual-representation media work; they do not wait for ADR-0006 native-sender -product acceptance. First, negotiate `HIGH`/`LOW` simulcast in one sender's +After trustworthy A+B/C, evaluate three bounded capability spikes in order and +stop at the first accepted path before custom dual-representation media work; +they do not wait for ADR-0006 native-sender product acceptance. First, negotiate +`HIGH`/`LOW` simulcast in one sender's initial envelope with `LOW` inactive and prove applied parameters, per-RID bytes/frames, and CPU/GPU/encoder release; separate PeerConnections have no portable shared-encode contract. Second, test at most two independently @@ -236,6 +278,14 @@ requirement unless runtime and resource counters disprove that boundary. Third, compare requested/applied SVC mode and Media Capabilities `powerEfficient`, with no software fallback. None may bypass PR #28's stock-GCC/RTX stop line. +Retain a favorable, testable hypothesis: on target GPUs, adding one low-rate, +low-resolution hardware `LOW` representation may have no material game impact. +Compare `HIGH` against `HIGH+LOW` under one scene using game FPS/p1 low, CPU, +GPU video-encode/copy activity, interval encode cost, actual encoder identity, +and LOW bytes/frames. If the increment stays inside the accepted game budget, +adopt the simpler on-demand dual representation and stop; do not continue into +SVC or custom media merely for theoretical encoder-count elegance. + ## Why Offline Encoding Presets Do Not Transfer [Maruko Toolbox](https://github.com/wzxjohn/marukotoolbox) is an Apache-2.0 @@ -358,6 +408,7 @@ recovery, and that each supported sender cohort can start it reliably. - [W3C MediaStreamTrack Content Hints](https://www.w3.org/TR/mst-content-hint/) - [W3C Screen Capture](https://www.w3.org/TR/screen-capture/) - [W3C WebRTC](https://www.w3.org/TR/webrtc/) +- [W3C WebRTC codec preferences](https://www.w3.org/TR/webrtc/#dom-rtcrtptransceiver-setcodecpreferences) - [W3C WebRTC Statistics](https://www.w3.org/TR/webrtc-stats/) - [W3C WebRTC SVC](https://www.w3.org/TR/webrtc-svc/) - [W3C Media Capabilities](https://www.w3.org/TR/media-capabilities/) @@ -367,6 +418,8 @@ recovery, and that each supported sender cohort can start it reliably. - [RFC 9628 VP9 RTP payload format](https://www.rfc-editor.org/rfc/rfc9628.html) - [IANA AV1 media type and format parameters](https://www.iana.org/assignments/media-types/video/AV1) - [Chromium/libwebrtc video stats origins](https://webrtc.googlesource.com/src/+/HEAD/video/g3doc/stats.md) +- [Chromium WebRTC hardware encoder factory](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/renderer/platform/peerconnection/rtc_video_encoder_factory.cc) +- [Chromium WebRTC AV1 hardware feature gate](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/media/webrtc/webrtc_features.cc) - [MDN `RTCRtpSender.setParameters()`](https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpSender/setParameters) - [Chromium `motion` to libwebrtc `kFluid` bridge](https://chromium.googlesource.com/chromium/src/third_party/+/refs/heads/main/blink/renderer/modules/peerconnection/media_stream_video_webrtc_sink.cc) - [libwebrtc `motion`/`kFluid` sender classification](https://webrtc.googlesource.com/src/+/3b1eab8a69cb5078befb021c5492d3f204a7d6a2/pc/rtp_sender.cc) diff --git a/docs/status.md b/docs/status.md index c30f6ea..b4a7d25 100644 --- a/docs/status.md +++ b/docs/status.md @@ -26,16 +26,16 @@ unconfigured, so production remains one host connection per viewer and P2P/TURN. - Synthetic Chrome 151 720p30 propagated balanced/clarity to one host and three viewers without peer changes; decoding and 2/1 fanout continued. This proves control continuity only. - Chrome 151/LiveKit localhost A/B measured cold active/render at 1.481/2.257 seconds versus standby 0.200/0.320 with host edges at two. It includes SDK/prewarm and is not public-network evidence. - Atomic activation passed health/access, exact bundle, SQLite preservation, services, and logs; the old release is rollback-ready. Responsive 320/375/390 px checks passed without live media. -- Host A+B aligns capture with one outbound RTP, interval/deltas, `remoteId`, path, and nullable transport-bound codec/profile/fmtp/`scalabilityMode`. It infers no quality/capability, retains no raw fmtp/SDP/stats, and treats stale, ambiguous, reset, or cross-transport data as unknown. Review found no P1/P2; 22 files/299 tests and both builds passed. +- Host A+B aligns capture/outbound identity, deltas, path, and nullable transport-bound codec/`scalabilityMode`; stale or ambiguous data stays unknown and no raw media metadata is retained. Review found no P1/P2; 22 files/299 tests and both builds passed. - HTTPS/WSS, access and room/WebSocket auth, renewal, public STUN, and authenticated TURN/UDP/TCP relay-only paths pass; TURN/TLS is off. - Draft #16/#18/#22/#23/#25/#28 passed isolated experiments only. ADR-0006 reached host setup/one encoder output, then timed out before viewer 1 decoded/rendered. ## Unverified Boundaries -- Full-resolution 30-minute runs, relay CPU/GPU, generational quality, controlled loss/RTT, depth latency, and game capture are unverified. +- Full-resolution 30-minute runs, relay CPU/GPU, controlled loss/RTT, and depth latency are unverified. Production `769de201f7cc` game sharing is reported low-FPS/high-load; latest `main`, preview cost, codec, and hardware encoding are not yet compared. - Android Chrome/iOS Safari leaves are unverified; runtime conservatively marks detected mobile/iPad clients as leaves. - Silent partitions can wait 30 to 60 seconds for heartbeat detection before the default 5-second grace; this remains unverified. -- Real audio, heterogeneous clients, mobile lifecycle, quality/pause, room `1` republish, and sustained profiles are unverified. Poor movie/video audio is user-reported but unclassified; after video A+B, diagnose capture settings, codec/fmtp, actual bitrate, loss, jitter, concealment, jitter buffer, and A/V sync. +- Real audio and heterogeneous clients remain unverified. Production reports poor film audio and voice-call self-echo under system capture; there is no app audio ceiling or Web process isolation. Diagnose A/B/C and sync, then gate Windows 11 game-process audio; Windows 10 stays unresolved without system fallback. - Production `769de201f7cc` reportedly sustains native `qualityLimitationReason=bandwidth` and severe blur with one capable LAN/direct viewer; only Host refresh restores quality. Within the five-second grace, stable-clientId Viewer refresh retains its `peerId`/`HostPeer`, while Host refresh rebuilds peers/capture; churn persistence is also reported. This prioritizes but does not prove sender/PC/GCC/capture generation over network or lifecycle causes. - ADR-0007 remains incomplete. Host A+B codec fields exist, but browser support and the reported case are unverified; authenticated C, the controller, and `LOW` are absent. Weak paths must share one `LOW`; fail closed protects `HIGH` only exceptionally, and an unreliable supported cohort fails acceptance. - Browser relays re-encode. ADR-0006 is no-go-unclassified: downstream checkpoints and two-edge/FIFO/TURN gates are absent. Spikes prove one WebCodecs object, not hardware; GCC+RTX is no-go and no-RTX weakens stats. @@ -46,19 +46,18 @@ unconfigured, so production remains one host connection per viewer and P2P/TURN. ## Next Milestone -Use the completed local host A+B diagnostics to reproduce the same-LAN/direct -case. First rebuild one affected `HostPeer`/`connectionId` while retaining -capture and healthy peers; separately compare capture-only and full Host rebuilds with -A+B/C evidence and opaque generations. The affected viewer must recover without -moving healthy peers or breaking the host-edge cap. Only proven stuck -generations justify guarded/cooldown recovery; never reconnect periodically or -raise ceilings blindly. +Use Host A+B to compare exact production and current `main` on one wired/direct +game fixture. Record actual codec/encoder, game FPS, CPU/GPU, preview on/off and +A+B while separately rebuilding one `HostPeer`, replacing capture, and fully +refreshing Host; retain the matching Viewer-local sample until authenticated C +lands. Only a proven stuck generation justifies guarded recovery; never +reconnect periodically, force AV1, or raise ceilings blindly. -After video A+B, run the audio A/B/C and sync diagnosis; `maxBitrate` is not a -quality fix. Then add authenticated video C and test standard -simulcast/LiveKit/SVC before custom `LOW`, independently of ADR-0006. The app -owns only the two-state policy; WebRTC/LiveKit owns congestion/layers. Later one exact room must pass peer/SFU UDP, optional -compatibility, and bounded-failure gates before changing coturn/TCP. +After authenticated video C, test simulcast/LiveKit/SVC in order and stop at the +first fit before custom `LOW`, independently of ADR-0006. Diagnose audio A/B/C, +sync, and voice-source leakage in parallel when it does not displace that P0; +`maxBitrate` is not a quality fix. WebRTC/LiveKit owns congestion/layers; the app +owns two-state policy. An exact room later gates peer/SFU UDP and coturn changes. ADR-0004 still requires a full-resolution 30-minute `1/3/5/8` network, resource, quality, latency, recovery, and browser/mobile-leaf matrix. @@ -69,19 +68,13 @@ picture; silent partitions add detection delay. ADR-0004 fails closed. Native shared encode and packet/layer striping remain separate experiments, not ways to relabel a failed browser route. -The ladder stops at #28; minimum-of-two/custom GCC stays out. ADR-0006 remains -no-go until an authorized staged gate retains host/RTP, signaling/PC, -decode/render, 2/3-viewer, and TURN evidence. - ## Blockers And Decisions Peer assistance and automatic routing remain No-Go until ADR-0004/0005 migration gates pass. Without LiveKit, two mobile leaves can fill both host roots; use an isolated exact-room canary only. -Native shared encode and striping remain separate experiments. Deferred architecture audit: `docs/maintenance.md`. - Whole-system versus selected-game audio for the first release. - Initial deployment region and network cohort. - Project license and distribution model. -- Whether peer assistance passes every non-encoding gate; any failure closes it. diff --git a/docs/方案设计.md b/docs/方案设计.md index 3fe30cf..07ab00c 100644 --- a/docs/方案设计.md +++ b/docs/方案设计.md @@ -107,6 +107,8 @@ WebSocket 使用同源 `/signal`。启用全站密码时，upgrade 必须先携 Zod 在服务边界拒绝未知类型、超长字符串、非法 SDP/candidate 结构和越权目标。服务端根据已认证会话决定路由：观看者不能给另一名观看者发信令，分享者也不能向本房间以外的 peerId 发送。 +不兼容迁移使用一个精确的当前 release/protocol fingerprint，而不是版本协商。旧页面连接新服务端时收到专用 fatal mismatch，停止自动重连并提示刷新；服务端终止旧 session。客户端和服务端不解析旧 wire、不双写、不维护多版本 handler，部署切换完成后删除旧 schema 与测试。 + ## WebRTC 协商 分享者是唯一 offerer。每名观看者对应一个 `RTCPeerConnection`，以独立 `connectionId` 区分重建代次： @@ -137,9 +139,11 @@ Zod 在服务边界拒绝未知类型、超长字符串、非法 SDP/candidate 每次 `setParameters()` 都必须从当次 `getParameters()` 派生更新，然后读回 requested/applied `maxBitrate`、`maxFramerate`、`scaleResolutionDownBy` 和 degradation preference。拒绝或浏览器改写必须进入可见状态，不能只写 `console.warn`。线上已是 clarity-first bounded controls，但用户报告的严重降级尚未完成受控分类，ADR-0007 自动质量控制也尚未实现；不得据此宣称画质提升。下一次真实质量验收仍需在同一高动态场景采样 1、2、3 名观看者和 TURN 路径。不得只提高 ceiling、强制 codec 或修改 SDP。 -PoC 不强制 codec，也不做 SDP munging。先通过 stats 记录实际协商 codec、`encoderImplementation` 和 `powerEfficientEncoder`，再依据目标机器测量决定是否优先 H.264。这样避免静态 codec 偏好意外触发软件编码。 +PoC 不强制 codec，也不做 SDP munging。先通过 stats 记录实际协商 codec、`encoderImplementation` 和 `powerEfficientEncoder`，再依据目标机器测量决定是否调整 H.264、VP8、VP9 或 AV1 优先级。`getCapabilities()` 只证明可协商，Media Capabilities 的 supported/smooth/power-efficient 只提供候选证据；最终必须同时检查实际 outbound codec、区间 encode time、游戏 FPS、CPU/GPU encode 引擎和 sender 数量。AV1 压缩效率不能证明目标机器具有 WebRTC 硬件编码，强制后可能落入更重的软件路径。 -`audio: true` 只是请求。若 `getAudioTracks()` 为空，分享者界面必须显示“当前来源没有可共享音频”。`systemAudio`、`windowAudio` 和 `audioSelection` 只影响选源器偏好，浏览器可以忽略且没有对应的标准音频来源类型读回，因此不能显示为系统、窗口或所选游戏音频保证。纯 Web 首版接受浏览器实际提供的标签页或系统音频，不承诺仅捕获某个游戏进程。 +`audio: true` 只是请求。若 `getAudioTracks()` 为空，分享者界面必须显示“当前来源没有可共享音频”。`systemAudio`、`windowAudio` 和 `audioSelection` 只影响选源器偏好，浏览器可以忽略且没有对应的标准音频来源类型读回，因此不能显示为系统、窗口或所选游戏音频保证。纯 Web 首版接受浏览器实际提供的标签页或系统音频，不承诺仅捕获某个游戏进程；系统音频可能把语音软件中观众自己的声音再次采入，这属于源内容泄漏而非 Host 页面回放，AEC 或 Web Audio 无法从已混合 track 恢复进程身份。Web 端必须保留 video-only 选择并明确风险，不能声称已隔离语音。 + +后续 Windows sender 以所选游戏 PID 为本地边界，优先使用 WASAPI process loopback 的 include-target-process-tree 捕获游戏及其子进程；“全部系统声音”仅为显式选择。目标进程无渲染流、退出或捕获失败时显示无声/未知，不回退为整机音频，也不上传 PID、路径、窗口标题、设备标识或原始音频。该 API 的最低 build 20348 不存在于普通 Windows 10 22H2（19045）产品矩阵，因此当前候选只覆盖 Windows 11；Windows 10 game-only 音频保持未解决或明确不支持，直到独立候选通过。 直播中切换来源仍由明确的按钮手势重新调用 `getDisplayMedia()`，不改变房间、host generation 或媒体拓扑。选源取消或失败时继续发送旧流；成功后为每个现有 sender 调用 `replaceTrack()`，健康连接不重新协商。初始 offer 预留 send-only audio transceiver，使音频可以在 `null -> track -> null` 间切换。若某个 peer 的替换或回滚失败，只重建该 peer；新流提交后才停止旧流，并以当前 stream 身份保护旧 track 的迟到 `ended` 事件。统计码率累加器在换源后重置，避免把不同 track 的计数器混算。 @@ -147,6 +151,8 @@ PoC 不强制 codec，也不做 SDP munging。先通过 stats 记录实际协商 Host 本地预览直接绑定现有 capture stream，不创建 viewer、`RTCPeerConnection` 或额外媒体边，也不产生 TURN/SFU/应用服务器出口。预览仍可能有本地合成、缩放和 GPU 显示成本；在同一动态场景对预览开/关采集 Host CPU、GPU、encode time 和实际 FPS。只有稳定收益且捕获、编码和远端画面不变时，才把直播后的默认动态预览改为静止缩略图或隐藏并允许按需展开；隐藏预览绝不停止 capture track 或 sender encode。 +部署版 `769de201f7cc` 的模糊、游戏低帧、持续 `bandwidth` 提示和 Host 刷新恢复都只作为同一 release 的用户报告。首轮复现必须在相同机器、Chrome/Edge、驱动、游戏场景和单个有线 direct Viewer 上对照该 release 与最新 `main`，并分别执行只重建受影响 `HostPeer`、只替换 capture、完整 Host 重建和预览开/关；任何一次只改变一个变量。若实际 codec/encoder 不变而 BWE/发送恢复，则下调 codec 根因优先级；只有编码成本异常时才进入受控 codec spike。 + 当前 Web 高级区不暴露音频质量控件。Screen Capture 规范未把 `channelCount`、`sampleRate` 列为 display-audio 可控属性；`track.getSettings()` 中偶尔出现的值只能作为捕获观测。标准 `RTCRtpSender.encodings[].maxBitrate` 可设置和读回，但只降低发送 ceiling，既不能要求编码器提高质量，也不等于 Opus `maxaveragebitrate`。Opus stereo、DTX 和 in-band FEC 属于协商的 codec/fmtp 行为，没有跨主流浏览器的标准 sender setter；`opus/48000/2` 也只是 RTP 信令约定，不能证明 48 kHz 源或实际 stereo。设计不为这些控件新增应用层 SDP munging，也不把 SDK 私有设置包装成浏览器保证。详细证据与复核门槛见 `docs/research/browser-screen-audio-quality.md`。 `frameRate` 约束和 sender 的 `maxFramerate` 都是目标或上限；浏览器仍控制实际捕获、重复帧处理、编码和拥塞降级。静态画面可能由浏览器减少实际编码数据，但标准没有保证每个实现以同一种方式节省采集或 GPU 开销。本 PoC 不添加画面差分、定时改约束或自定义动态 FPS 状态机，先用 `framesEncoded`、实际发送码率、`totalEncodeTime`、质量受限原因和主机资源占用测量真实问题。 @@ -169,11 +175,11 @@ Host 本地预览直接绑定现有 capture stream，不创建 viewer、`RTCPeer Viewer 可通过已鉴权、绑定当前房间/路径代次的控制面请求 `LOW`，但请求只是一条观测输入，必须限频和去重，并与 sender 的 Transport-CC/GCC、实际码率、RTT、丢包、重传及编码限制、派生协商 codec/profile/parameters（适用时含 `scalabilityMode`）和 viewer 的实际解码/丢帧/冻结表现交叉验证。UA、平台和设备型号不参与质量决策；现有 mobile/iPad 检测只用于保守设置 relay capacity。 -固定排期的 SVC spike 只验证标准能力和资源边界；只有以后要求任何时刻严格一个编码输出时才可采用 SVC 产品路径。每条连接可独立选择基础层或增强层，但必须读回实际 codec/`scalabilityMode`，确认目标 hardware/power-efficient 路径，并通过游戏性能和亚秒延迟验收。浏览器的硬件偏好不是保证，不能静默回退为软件 SVC；常驻 SVC 不是默认方案。普通非分层编码包不能只靠转发生成另一档画质，实现路径差异只能用第二表示、SVC 或 relay/SFU 转码，当前优先验证标准表示能力。 +SVC 是 simulcast 与 LiveKit/Dynacast 均未满足后的第三个标准候选；若它能独立选层、读回实际 codec/`scalabilityMode`、确认目标 hardware/power-efficient 路径并通过游戏性能和亚秒延迟门槛，即采用并停止自研。浏览器的硬件偏好不是保证，不能静默回退为软件 SVC；它不是预选默认方案，未来严格单输出要求只会加强而不是创建采用理由。普通非分层编码包不能只靠转发生成另一档画质，实现路径差异只能用第二表示、SVC 或 relay/SFU 转码。 质量控制不放宽媒体拓扑预算：host 仍最多两条下游媒体边；native relay 只转发选定表示的编码包，不解码重编码。后续双树或条带化可把 host 上传从约两份完整码率推向一份加必要冗余，但不阻塞按需双表示的实现。 -标准表示能力排在完整 A+B/C 证据之后、任何自定义双表示媒体实现之前，并可与 ADR-0006 native sender 复核并行：依次做同 sender simulcast、最多两个 LiveKit root 的 Dynacast 验证/否决和 SVC spike。具体语义与资源门槛由 ADR-0007 和[实时质量调研](./research/realtime-quality-adaptation.md)持有，任何方案都不得复用 PR #28 的 minimum-of-two 或绕过 stock WebRTC congestion control。 +标准表示能力排在完整 A+B/C 证据之后、任何自定义双表示媒体实现之前，并可与 ADR-0006 native sender 复核并行：依次做同 sender simulcast、最多两个 LiveKit root 的 Dynacast 验证/否决和 SVC spike，首个满足语义与资源门槛的路径立即结束序列。具体门槛由 ADR-0007 和[实时质量调研](./research/realtime-quality-adaptation.md)持有，任何方案都不得复用 PR #28 的 minimum-of-two 或绕过 stock WebRTC congestion control。 ## 媒体传输配置与可选 TURN @@ -257,7 +263,7 @@ SFU 是主要中央兜底，但正常只作为 virtual parent 向一至两个必 - 同机双标签页与同 LAN direct； - 真实跨网络、同房间 direct/peer UDP + SFU/UDP root； - CGNAT、双 NAT、移动热点和普通家宽下使用临时 canary 配置移除 `rtc.tcp_port`、关闭公网 7881 且不配置 TURN/TCP/TLS，验证 host 与 roots 使用 SFU/UDP；人为封锁 UDP 时必须有界明确失败。只在部署选择兼容层时另测 LiveKit participant TURN 或 ordinary peer coturn； -- Windows Chrome/Edge 的音轨有/无； +- Windows Chrome/Edge 的 tab/window/monitor 音轨有/无、同时语音通话和 video-only；Windows 11 原生候选另测 game process tree、通知、无渲染流与进程重启，Windows 10 记录明确不支持/未解决； - Android Chrome 与 iOS Safari 的播放手势、旋转、前后台和网络切换； - 一名分享者、八名异构观看者连续 30 分钟，同时记录较低人数基线、实际上行、编码负载和首帧时间。真实 1:8 未完成前不得把默认上限写成性能承诺。 - 真实桌面 Chrome 对 balanced 与 clarity 做控制传播和连接身份 smoke；该项只证明直播中应用与不重建，不代替清晰度、负载或公网矩阵。 @@ -298,7 +304,7 @@ Viewer 在处理 peer-assisted 鉴权快照时先更新 `ViewerRelay` 的 desire spike 复用现有 screen stream 和浏览器 codec 协商，覆盖当前桌面 Chrome/Edge relay、按测试顺序位于链尾的当前 Android Chrome 与 iOS Safari leaf、最多八名 viewer；捕获存在时沿同一标准 WebRTC 路径带上音频。Encoded Transform/DataChannel/WebCodecs 媒体、SVC/simulcast、多树、转码、codec ladder、综合评分、移动端后台 relay 和跨模式 SFU 迁移不在本次 spike 内实现。该范围限制不能解释为拒绝产品所需的自动兜底；验收与失败即放弃条件以 ADR-0004 为准，自动路由由后续独立 ADR 定义。 -浏览器不保证多个 `RTCPeerConnection` 共享一次编码。实验允许 host 对一至两个 seed 各自编码。后续独立的 native sender 阶段必须用 shared encoder 将一份编码结果交给最多两套标准 WebRTC packetizer，并由新的 ADR 约束；它降低 host 编码负载，但两条出边仍各自消耗上传带宽。该计划不依赖本次浏览器 relay 实验成败，也不在本 spike 中提前抽象或实现。 +浏览器不保证多个 `RTCPeerConnection` 共享一次编码。实验允许 host 对一至两个 seed 各自编码。后续 native sender 必须让每个 active representation 只编码一次并跨 direct、peer 与 SFU transport 复用；每条连接仍拥有独立 packetizer、SSRC、pacing、加密、反馈和上传。健康时只有共享 `HIGH`，弱路径期间最多增加一份共享 `LOW`。该计划不依赖本次浏览器 relay 实验成败，也不在本 spike 中提前抽象或实现。 详细证据见 `docs/research/peer-assisted-media.md`，提案见 `docs/adr/0004-peer-assisted-media-experiment.md`。 diff --git a/docs/需求理解.md b/docs/需求理解.md index 749f621..78426b2 100644 --- a/docs/需求理解.md +++ b/docs/需求理解.md @@ -22,7 +22,7 @@ - [ ] P2P 优先：每条逻辑媒体边的 ICE 必须先尝试直接 UDP 路径，服务器不得默认中转可直连的媒体。一个或两个观看者优先直接连接分享端；更多观看者的 peer-assisted 行为须先通过 ADR-0004 的实验门槛。 - [ ] NAT 与传输兜底：STUN 必须可用。普通 direct/peer 边默认只使用 ICE/UDP；其路径不可用时改由 SFU virtual parent 承担必要 root，而不是给每条 peer edge 常驻 TURN candidates。旗舰默认媒体路径为 `direct/peer UDP -> SFU ICE/UDP -> 明确失败`。部署者可为已验证的受限网络额外启用短期鉴权 TURN。普通 Host 到 peer root 或 root 到 viewer 的异常边需要新增、绑定当前 assignment/connection generation 的独立 coturn grant/wire；一个 TURN seed root 收到媒体后仍优先服务 bounded peer descendants。LiveKit publisher/subscriber 到 SFU 是另一 ICE 域，pinned 1.13.5 的 embedded/external TURN 当前按 LiveKit participant 广告，不是 Screener selected-edge grant。两者必须分开配置、鉴权和验收。媒体 ICE/TCP、TURN/TCP 和 TURN/TLS 不是默认质量路径。HTTPS/WSS 仍使用 TLS/TCP。全部可选 TURN 配置缺失表示正常禁用且不提示，部分配置必须启动即失败。 - [ ] 质量控制：保留 1080p60（8 Mbps）、1080p30（5 Mbps）、720p30（3 Mbps）三档推荐上限，默认采用清晰优先；分享期间可直接换档或应用有界高级设置，不重新选择来源或重建健康连接。Peer-assisted relay 与已配置的 SFU 必须接收同一个严格校验的房间级权威设置，覆盖当前和以后建立的 sender；普通 P2P 信令结构保持不变。任何设置都只是浏览器上限和偏好，必须读回实际 sender 参数并显示参数拒绝、改写和持续受限原因。不做自定义动态 FPS、综合评分或周期性全房间调档。 -- [ ] 按需双表示：健康房间只运行一套 active `HIGH` representation；第一条观看路径被同一时段的 sender 与 viewer 统计持续证明无法承受时，系统必须启动唯一一套由全部弱路径共用的 `LOW` 并迁移弱路径。健康路径继续使用 `HIGH`，全部弱路径稳定恢复后关闭 `LOW`。active representation 硬上限为二，禁止按观看者增长为 N 套，也禁止把最差路径的目标应用到整个房间。标准浏览器不保证跨 PeerConnection 共用一个物理编码器，inactive 层也不保证释放 GPU；物理 encoder instance、CPU/GPU 和释放行为必须实测。没有合格 hardware/power-efficient 路径或启动 `LOW` 造成不可接受负载时，当次必须 fail closed：保护 `HIGH` 并明确显示弱路径降级或不可用；这只是异常止损，supported sender cohort 若不能可靠按需启动 `LOW`，则自动质量控制验收失败，必须优化编码路径或明确标记不支持。 +- [ ] 按需双表示：健康房间只运行一套 active `HIGH` representation；第一条观看路径被同一时段的 sender 与 viewer 统计持续证明无法承受时，系统必须启动唯一一套由全部弱路径共用的 `LOW` 并迁移弱路径。健康路径继续使用 `HIGH`，全部弱路径稳定恢复后关闭 `LOW`。active representation 硬上限为二，禁止按观看者增长为 N 套，也禁止把最差路径的目标应用到整个房间。实现必须依次验证标准 WebRTC simulcast、LiveKit layer selection/Dynacast 和 SVC；任一方案满足按需停启、独立选层与资源门槛即停止，不继续自研媒体层。标准浏览器不保证跨 PeerConnection 共用一个物理编码器，inactive 层也不保证释放 GPU；物理 encoder instance、CPU/GPU 和释放行为必须实测。没有合格 hardware/power-efficient 路径或启动 `LOW` 造成不可接受负载时，当次必须 fail closed：保护 `HIGH` 并明确显示弱路径降级或不可用；这只是异常止损，supported sender cohort 若不能可靠按需启动 `LOW`，则自动质量控制验收失败，必须优化编码路径或明确标记不支持。 - [ ] 路径级质量状态：每条观看路径只使用 `HIGH` 与 `FALLBACK` 两态。进入 `FALLBACK` 需要多个连续窗口的带宽不足、冻结或解码跟不上证据，回到 `HIGH` 需要更长的稳定窗口；进入与退出使用不同门槛且不计算综合分数。Viewer 的低档请求必须经当前会话鉴权、限频和去重，并由 sender 的 GCC/Transport-CC、实际码率、RTT、丢包、重传、编码限制、派生后的实际协商视频 codec/profile/parameters（适用时含 `scalabilityMode`）与 viewer 的实际解码、丢帧、冻结证据交叉验证，不能仅凭自报生效；上报不得包含 raw SDP、candidate 地址或原始设备/网络标识，鉴权与关联只使用服务端签发的不透明路径/连接代次 ID。 - [ ] 断线恢复：临时网络变化时尝试 ICE restart 和重连，并向用户显示当前连接状态。 - [ ] 可观测性：按目标 RTP transport 与不透明连接/ICE 代次关联 selected path、`iceGatheringState`/ICE/connection state、事件推导的建立时长、generation-baselined pair-change delta，以及 RTT、丢包、抖动、码率、帧率、编码耗时和受限原因。候选与错误仅使用[低服务器成本媒体路由](./research/low-server-media-routes.md)的脱敏字段和 code bucket，禁止 raw SDP、candidate、地址、URL、`errorText` 或网络标识。 @@ -37,16 +37,16 @@ - [ ] 质量控制扩展：推荐档位仍是默认入口；折叠高级区只允许 720p/1080p/1440p 分辨率 ceiling、15 至 60 的整数 FPS ceiling、2 至 12 Mbps 视频码率 ceiling，以及清晰/平衡/流畅三种偏好。直播中显式应用后才生效，避免滑杆产生连续控制流量。当前 Web 发送端不提供音频声道、采样率、Opus 码率、stereo、DTX 或 FEC 控件：display capture 未把声道和采样率定义为可移植控制，标准音频 `maxBitrate` 只是可能导致更低质量或停播的发送上限，其他项依赖 codec/fmtp 协商而没有可移植 sender setter。除非真实音频矩阵证明用户问题、标准 API 能应用并读回且实际结果可独立观测，否则不得为这些控件新增应用层 SDP munging 或用默认值制造伪控制。实际质量持续低于所选上限时必须显示可解释的限制原因；在真实拥塞、编码器和接收端证据完成前，不把任一组合称为“完美平衡”。 - [ ] 本地预览性能候选：在相同动态场景中 A/B 测量 Host 动态预览开/关的 CPU、GPU、encode time 和实际 FPS。只有稳定收益且捕获、编码和远端画面不受影响时，才允许直播开始后默认显示静止缩略图或隐藏动态预览并提供按需展开；未测前不宣称收益。 - [ ] Peer-assisted 媒体实验：分享端最多两个 child、每名 viewer 最多一个 child；信令服务只按深度和服务端加入序号确定性地形成两条 sticky 平衡链，不加入综合评分或主动重平衡。每个 viewer 将收到的标准 remote track 通过新的 `RTCPeerConnection` 转发，因此会逐跳解码和重编码；必须测量而不能称为 shared encode。服务端只在受房间数限制的内存状态中保存当前严格校验的质量设置，不写入 SQLite；host 鉴权后先重报设置，在线 viewers 接收更新，后加入者从鉴权快照取得当前值。该功能在 ADR-0004 的实验门槛通过并由后续 ADR 正式接受前不得成为产品路径。 -- [ ] 原生 shared encode 分享端：后续独立的 Electron/原生分享端阶段必须保证一份兼容编码输出扇出到最多两套独立 WebRTC packetizer，并由单独 ADR 和测量约束。它降低分享端重复编码负载，但不消除每条出边各自的上传带宽。浏览器不保证跨 `RTCPeerConnection` 复用编码器，因此当前 Web spike 不实现、也不能宣称满足 shared encode 保证。 +- [ ] 原生 shared encode 分享端：后续独立的 Electron/原生分享端必须让每个 active representation 只产生一份兼容编码结果，并在 direct、peer 与 SFU transport 间复用；健康时只有共享 `HIGH`，弱路径期间最多再有一份共享 `LOW`。每条连接仍保留独立 WebRTC packetization、SSRC、pacing、加密和反馈，且各自消耗上传带宽。浏览器不保证跨 `RTCPeerConnection` 复用编码器，因此当前 Web spike 不能宣称满足该保证。 - [ ] 原生 shared encode relay：具备原生 relay 能力的 audience 同样最多维护两条下游媒体边，并应复用兼容编码输出而不再次编码。普通浏览器 audience 继续作为 leaf 或使用当前会逐跳重编码的实验路径，不伪称 shared encode。 -- [ ] 条件式 SVC：在完整 A+B/C 证据之后、任何自定义双表示媒体实现之前执行有界 viability spike，并可与 ADR-0006 native sender 复核并行。只有后续需求要求任何时刻严格一个编码输出时才可采用 SVC 产品路径；必须读回 requested/applied codec/`scalabilityMode`，由 Media Capabilities 提供 power-efficient 候选证据并通过真实游戏性能验证，不得静默回退为高负载软件 SVC，常驻 SVC 不是默认方案。 +- [ ] 条件式 SVC：在完整 A+B/C 证据之后，按 simulcast、LiveKit/Dynacast、SVC 顺序短路验证，并可与 ADR-0006 native sender 复核并行。前两项失败而 SVC 满足独立选层、requested/applied codec/`scalabilityMode`、hardware/power-efficient、真实游戏性能和亚秒延迟门槛时即可采用并停止自研；不得静默回退为高负载软件 SVC。它不是预选默认方案，未来严格单输出要求只会加强而不是创建采用理由。 - [ ] 分包/分层多树实验：把编码包、对象或可独立解码层拆到多个 peer 路径以分散上传，只能在独立 spike 中验证。必须保持亚秒级，处理多父重组、排序、丢包恢复、关键帧、同步和节点退出；未通过门槛前不进入产品路由控制器。 - [ ] SFU-root 中央兜底：产品部署目标必须提供经 canary 验收的单节点 SFU 资源；系统在 direct/peer UDP 不可用或明确拓扑预算无法接纳时自动把它作为主要中央兜底，不要求用户选模式。它只向总计一至两个必要 roots 出口；root 优先承担受限 peer descendants，没有可靠 relay root 时必要 viewer 只作为同一总上限内的零 descendants root。已关闭 PR #12 的显式整房 SFU 模式已被取代。当前 PR #17 实现仍是 default-off、failure-only 且生产未配置；按 ADR-0005 的 exact-room canary 验收并修改配置契约后，才能启用新的默认梯级。 - [ ] 分享端桌面封装：用 Electron 固定 Chromium 版本、改善选源/热键/系统音频体验，同时保留 Web 观看端。 -- [ ] Windows 应用音频：仅捕获所选游戏及其子进程的音频，避免把语音聊天或通知声一并分享。 +- [ ] Windows 应用音频：Windows 11 原生候选默认仅捕获所选游戏及其子进程的音频，避免把语音聊天或通知声一并分享；“全部系统声音”只能由分享者显式选择。普通 Windows 10 不具备当前 process-loopback API 门槛，必须明确标为未解决/不支持而不是伪兼容。浏览器无法证明或隔离任意进程音频时必须如实提示，原生捕获失败、无声或进程退出不得静默扩大为整机捕获。 - [ ] 连接自检：在开始分享前验证信令、STUN、SFU ICE/UDP 及部署实际启用的可选 TURN 或媒体 TCP 可达性，并给出不暴露凭据的诊断结果。未配置可选兼容层不是告警。 - [ ] 受限网络增强：部署者仅在明确选择覆盖 UDP 受限网络且 canary 证明收益时，才可启用短期鉴权 TURN 或媒体 TCP；它只能用于控制器选中的异常 transport，不能成为所有普通 peer 边的默认候选，也不能被描述为更高清的传输方式。 -- [ ] 运行时编解码选择：根据端点能力和统计数据选择实际硬件编码的 H.264、VP8、VP9 或 AV1，并始终保留兼容回退。 +- [ ] 运行时编解码选择：先保留浏览器默认协商；只有端点能力相交、Media Capabilities 对目标分辨率/FPS/码率报告 supported/smooth/power-efficient，且实际 outbound codec、区间 encode time、游戏 FPS、CPU/GPU 与 sender 数量共同通过时，才调整 H.264、VP8、VP9 或 AV1 优先级。AV1 不得因压缩率推断而默认启用，任何候选都不得静默落入高负载软件编码。 - [ ] 诊断页面：用户可导出不含敏感地址和凭据的连接诊断信息。 ### P2 - 可选功能 @@ -63,6 +63,7 @@ - 安全：应用控制面全程 HTTPS/WSS；P2P/TURN 边在 WebRTC 端点间使用 DTLS-SRTP，普通 `turn:` over TCP 不额外套 TLS，但不移除该媒体层加密。普通 SFU 是逐跳 DTLS-SRTP，只有另行实现并验收应用层 E2EE 才能对 SFU operator 保持媒体端到端机密。TURN 使用短期凭据、配额和速率限制；仓库不得包含真实密钥。 - 隐私：P2P 会向房间内好友暴露网络地址，首版威胁模型仅覆盖可信熟人；若要求隐藏 IP，必须切换为服务器转发。 - 可维护性：代码、需求、调研、项目记忆、`AGENTS.md` 和 `.codex/` 均在 Git 中跟踪；开发配置不得依赖单一操作系统的绝对路径。 +- 迁移完整性：同一 release 的客户端、服务端和部署配置原子迁移；canary 结束后删除被替代的配置、wire、parser、双写与测试，不为无当前消费者的旧版本保留兼容层或第二套架构。Git 历史承担旧实现追溯。 - 兼容性：分享端首个目标矩阵为 Windows 10/11 的当前 Chrome/Edge；观看端还必须覆盖当前 Android Chrome 和 iOS Safari，并实测自动播放限制、横竖屏、锁屏恢复与移动网络切换。Peer-assisted 实验只允许能力完整的前台桌面 Chrome/Edge 作为 relay，手机必须保持 leaf。 ## 边界与约束 @@ -85,6 +86,7 @@ - Given 分享期间只有 host 信令短暂断开，When 现有 P2P 媒体仍健康，Then 客户端继续播放媒体并恢复信令，而不是错误显示“已停止分享”。 - Given `ROOM_DATABASE_PATH` 已配置但 `ACCESS_PASSWORD` 为空，When 应用启动，Then 配置校验明确失败，不能在公开网站暴露可枚举的持久房间。 - Given `ROOM_DATABASE_PATH` 为空，When 创建房间，Then 无论是否启用全站密码都使用随机临时房间码并遵守 `ROOM_TTL_SECONDS`；停止分享本身不立即销毁房间。 +- Given 部署切换了不兼容的当前协议，When 仍打开的旧页面连接新服务端，Then 单一 release/protocol mismatch 边界终止该 session、停止自动重连并明确提示刷新；不得通过旧 parser、双写或双协议继续假连接。 ### P2P、SFU 与可选 TURN 兜底 @@ -122,10 +124,12 @@ - Given 同一时刻采集、host outbound 与 viewer inbound 证据，When 进行根因分类，Then 关联派生后的实际协商视频 codec/profile/parameters、适用时的 `scalabilityMode` 与真实解码表现，依次区分 Capture 低、Capture 高但 outbound 因 CPU 低、Capture 高但 outbound 因 bandwidth 低、outbound 正常但 inbound 低、以及 inbound 正常但肉眼模糊五类；不用 UA 或设备型号代替运行时证据，也不上传 raw SDP。 - Given Viewer 发送低档请求，When 请求未鉴权、过频、重复、过期或没有匹配的 sender/viewer 统计，Then 请求不改变表示或路径状态；有效请求也只能作为交叉验证输入。 - Given 受控 RTT/丢包测试条件，When 连续分享高动态游戏画面 30 分钟，Then 延迟、帧率与丢帧达到上述目标，且没有不可恢复的冻结。 +- Given 部署版出现游戏低帧、持续 `bandwidth` 限制或只有 Host 刷新才能恢复，When 归因该报告，Then 用同一机器、浏览器、驱动、场景和单个有线 direct Viewer 对照精确 release `769de201f7cc` 与最新 `main`，依次比较受影响状态、只重建该 `HostPeer`、只替换 capture、完整 Host 重建和本地预览开/关；记录实际 codec/encoder、A+B/C、游戏 FPS、CPU 与 GPU encode 使用，不能把部署版报告直接归为最新代码缺陷。 - Given 分享过程中切换 Wi-Fi/有线网络，When ICE 连接失败，Then 客户端触发 ICE restart，并在成功或超时后给出明确状态。 - Given 浏览器未返回系统音轨，When 分享开始，Then 分享者在启播前看到“当前来源没有可共享音频”的提示。 - Given 浏览器返回屏幕音轨，When 产品展示音频状态或未来连接详情，Then 只能把存在性、浏览器实际返回的 track settings、实际协商 codec/fmtp 和实际流量分别标为捕获、协商或观测事实；字段缺失必须显示为未知，且不得从 Opus 的 `48000/2` RTP 声明推断源采样率或实际 stereo。 - Given Web 或后续 native sender 分享电影/视频音频，When 进行产品验收，Then 必须用独立 A/B/C 证据关联 capture audio track/settings、实际 negotiated codec/fmtp、outbound/inbound 实际码率、loss、jitter、concealment、jitter-buffer 与 A/V sync；native sender 必须通过该音质和同步矩阵，且不得把 audio `maxBitrate` ceiling 当作提质证明。 +- Given 分享者同时运行语音软件，When 选择仅分享游戏声音，Then supported Windows 11 native sender 只保留目标游戏及其子进程的渲染音频，观众、通知和其他应用声音不得进入分享；浏览器 system audio 无法满足时必须允许 video-only 或明确告知其可能包含语音/通知，不能用 AEC、降噪或来源标签伪称已隔离。 ### 仓库可移植性 From a949fbaf159d2f8441718aff10a5bf14d34b0a6d Mon Sep 17 00:00:00 2001 From: AAA <35992542+TNTcraftHIM@users.noreply.github.com> Date: Wed, 19 Aug 2026 20:39:56 +0800 Subject: [PATCH 2/2] feat(quality): add authenticated viewer evidence --- docs/project-memory.md | 6 +- docs/research/realtime-quality-adaptation.md | 27 +- docs/status.md | 13 +- src/client/media/viewer-quality-evidence.ts | 250 ++++++++++++++ src/client/pages/HostPage.tsx | 128 ++++++- src/client/pages/ViewerPage.tsx | 102 ++++++ src/client/types.ts | 10 + src/client/webrtc/stats.ts | 163 +++------ src/server/hybrid-media-router.ts | 32 +- src/server/signaling.ts | 188 +++++++++- src/shared/protocol.ts | 106 ++++++ src/shared/video-codec-evidence.ts | 166 +++++++++ tests/client-state.test.ts | 113 +++++- tests/protocol.test.ts | 131 +++++++ tests/server-signal.test.ts | 343 +++++++++++++++++++ tests/viewer-quality-evidence.test.ts | 247 +++++++++++++ 16 files changed, 1862 insertions(+), 163 deletions(-) create mode 100644 src/client/media/viewer-quality-evidence.ts create mode 100644 src/shared/video-codec-evidence.ts create mode 100644 tests/viewer-quality-evidence.test.ts diff --git a/docs/project-memory.md b/docs/project-memory.md index adbe22a..57c11cb 100644 --- a/docs/project-memory.md +++ b/docs/project-memory.md @@ -25,7 +25,7 @@ Last updated: 2026-08-19 - Keep the current failure-only controller default-off until config migration and exact-room gates pass. Preserve break-before-make, sticky healthy subtrees, mobile leaves, and PR #20 prewarm; public transport/load remains open. - Local reparenting is a later candidate: start with unassigned relay admission rescue; do not block current work. - Flagship media defaults to UDP; HTTPS/WSS remains TLS/TCP. Current production still requires coturn UDP/TCP. Future TURN absence is normal, partial config fails, and any media-TCP/port compatibility mode is chosen only by canary. -- Treat settings as ceilings and deployed degradation as unclassified. Use local Host A+B to compare exact `769de201f7cc` against current `main` for Host-only recovery, game FPS/load, preview cost, actual codec/encoder and one-variable rebuilds; then add authenticated C and test standard simulcast/LiveKit/SVC before custom `LOW`. Never use UA or a composite score. +- Treat settings as ceilings and deployed degradation as unclassified. Use correlated Host A+B/Viewer C to compare exact `769de201f7cc` against current `main` for Host-only recovery, game FPS/load, preview cost, actual codec/encoder and one-variable rebuilds; then test standard simulcast/LiveKit/SVC before custom `LOW`. Never use UA or a composite score. - Production reports poor film audio and self-echo when system capture includes voice software. Diagnose audio A/B/C and sync; Web cannot isolate arbitrary processes and `maxBitrate` is not quality-up. A Windows 11 native candidate defaults to game-process-tree audio and never widens silently; Windows 10 remains unresolved/unsupported. See `docs/research/browser-screen-audio-quality.md`. - Do not add scene detection, dynamic-FPS control, or forced AV1 until stats and target hardware prove the need. Codec acceptance requires actual negotiation, power-efficient candidate evidence, interval encode cost, game FPS, CPU/GPU and sender count; Discord's native capture/hardware tuning is comparison evidence, not proof of server re-encoding or a reusable preset. - Keep room policy deployment-driven: public/password-only rooms are random and temporary; password plus SQLite enables sequential persistent rooms and reusable stopped links. @@ -45,9 +45,9 @@ Last updated: 2026-08-19 - Draft native ladder #16/#18/#22/#23/#25/#28 has one bounded Chrome 720p30 loop: one WebCodecs object feeds two Pion/browser legs, applies an experiment-only minimum stock-GCC target, and recovers one isolated loss. It does not prove one hardware encode. - Stock Pion GCC plus negotiated RTX is a no-go. The no-RTX candidate can distort loss statistics and stays outside product code until audio, heterogeneous estimates, broader loss, TURN/reconnect, and browser-diversity gates pass; do not add custom congestion control to bypass them. - Chrome 151 short synthetic `1/3/5/8` runs kept host edges at two, relay edges at one, and all viewers decoding; a three-viewer relay close recovered in 5.32 seconds. This proves topology/control only, not quality, load, endurance, or silent partitions. -- The current quality stack uses one strict, non-persistent `QualitySettings` value with bounded ceilings and clarity/balanced/fluid preference. Sender readback and limitation warnings remain manual diagnostics. ADR-0007's automatic per-path `HIGH`/`FALLBACK` and unique on-demand shared `LOW` are accepted mainline behavior, but the C report, state controller, and `LOW` runtime are unimplemented. +- The current quality stack uses one strict, non-persistent `QualitySettings` value with bounded ceilings and clarity/balanced/fluid preference. Sender readback and limitation warnings remain manual diagnostics. ADR-0007's automatic per-path `HIGH`/`FALLBACK` and unique on-demand shared `LOW` are accepted mainline behavior, but the state controller and `LOW` runtime are unimplemented. - Chrome 151 synthetic 720p30 propagated balanced/clarity to one host and three viewers without changing peer identities; all viewers kept decoding and fanout stayed host two/relay one. This proves control continuity only. -- Local Host A+B aligns current capture settings with one unique outbound RTP/remote/transport pair per tick, rebases deltas on ambiguity, identity change, or reset, and rejects stale source generations. Codec evidence is nullable and transport-bound; no defaults or profile capability are inferred, raw fmtp/SDP/stats are not retained, and multiple RTP/encodings remain unknown. Authenticated C, automatic control, and `LOW` remain absent. +- Local Host A+B aligns capture with one unique outbound RTP/remote/transport pair and rejects ambiguous or stale generations. Authenticated Viewer C now adds a nullable, sanitized two-second receive window of at most 2 KiB for each current ordinary or peer-assisted P2P hop; the server derives identity and current connection/revision, and the parent correlates only matching local B. It retains no raw fmtp/SDP/stats, performs no media action, and fails closed for SFU-fed roots. Automatic control and `LOW` remain absent. - Release `769de201f7cc` passed CI/hygiene/typecheck, 22 files/274 tests, both builds, zero production dependency vulnerabilities, atomic activation, public health/access, exact bundle, and service/log checks. ## Provisional Quality Targets diff --git a/docs/research/realtime-quality-adaptation.md b/docs/research/realtime-quality-adaptation.md index d68724a..b7bdd52 100644 --- a/docs/research/realtime-quality-adaptation.md +++ b/docs/research/realtime-quality-adaptation.md @@ -164,20 +164,28 @@ This is a diagnostic classification, not a weighted health score. Missing or reset evidence remains unknown and rebases the interval. The smallest implementation sequence is local A+B correlation in one host -sampling tick, including the interval, media/stat identity, and valid deltas. -Only after that is trustworthy should a minimal authenticated C report carry -the receive/decode and derived negotiation signals needed by the two-state +sampling tick, followed by a minimal authenticated C report carrying the +receive/decode and derived negotiation signals needed by the two-state predicate. It never carries raw SDP, raw stats, candidate addresses, or raw -device/network identifiers. Opaque server-issued path and connection-generation -IDs provide authorization and correlation; a general remote stats stream or +device/network identifiers. Server-authoritative path and connection generations +provide authorization and correlation; a general remote stats stream or telemetry pipeline is unnecessary. -The repository now implements the local host A+B foundation: same-tick capture +The repository implements the local host A+B foundation: same-tick capture settings plus one uniquely matched outbound RTP sample, explicit sample/media identity and adjacent deltas, `remoteId` linkage, and the selected path reached through that RTP stream's transport. Source replacement blocks sampling and invalidates in-flight generations. +Authenticated Viewer C is also implemented for each current ordinary or +peer-assisted P2P hop. A viewer sends one nullable, sanitized aggregate window +on the existing two-second stats cadence, bounded to 2 KiB; the server derives +the current viewer, parent, connection, and route revision, while the parent +accepts only the matching local generation and expires it. The report contains +no raw SDP, stats, candidates, addresses, device identifiers, or room identity, +is neither stored nor used for media or routing action, and fails closed for an +SFU-fed root until a real SFU last-hop B/generation exists. + Codec evidence follows only that outbound RTP object's `codecId`, and the referenced `RTCCodecStats` must use the same transport. The local diagnostics expose nullable, bounded MIME plus a normalized profile token and an ASCII allowlist of @@ -195,9 +203,10 @@ The same unique outbound object supplies nullable current configured `scalabilityMode`. Sender parameter readback exposes an applied mode only for one unambiguous encoding. Current quality settings do not request a mode, so the requested value remains null and a browser-reported default is not called -a mismatch; multiple encodings remain unknown. This still does not implement -an authenticated C report, two-state controller, or on-demand `LOW` runtime, -and browser support remains subject to the controlled matrix. +a mismatch; multiple encodings remain unknown. Inbound stats provide no current +standard `scalabilityMode` source, so C does not carry a null-only placeholder. +The two-state controller and on-demand `LOW` runtime remain unimplemented, and +browser support remains subject to the controlled matrix. Official W3C text checked 2026-08-19 defines names ending in `Id` as stats-object references. In particular, outbound [`mediaSourceId`](https://www.w3.org/TR/webrtc-stats/#dom-rtcoutboundrtpstreamstats-mediasourceid) diff --git a/docs/status.md b/docs/status.md index b4a7d25..45c3eb4 100644 --- a/docs/status.md +++ b/docs/status.md @@ -17,7 +17,7 @@ unconfigured, so production remains one host connection per viewer and P2P/TURN. - SFU normally feeds one or two roots. Downstream cap is two, browser relay one; exceptional central edges need a separate cap. Production routing is off. - Configured fallback adds a non-secret standby URL and token-free DNS/TLS warmup; absent configuration adds no field, import, request, participant, or edge. - Each viewer advertises relay capacity 0/1; detected mobile/iPad clients are leaves. Withdrawal does not move a healthy edge. Browser relays stay one-child and host two-child. -- Production has memory-only video settings and manual readback. Automatic `HIGH`/`FALLBACK` plus one shared on-demand `LOW` is accepted but not deployed; host A+B exists, while C/actions do not. Browser audio is request/presence-only. +- Production has memory-only video settings and manual readback. Automatic `HIGH`/`FALLBACK` plus one shared on-demand `LOW` is accepted but not deployed; local A+B and authenticated P2P Viewer C are read-only evidence. Browser audio is request/presence-only. ## Verified Evidence @@ -26,7 +26,7 @@ unconfigured, so production remains one host connection per viewer and P2P/TURN. - Synthetic Chrome 151 720p30 propagated balanced/clarity to one host and three viewers without peer changes; decoding and 2/1 fanout continued. This proves control continuity only. - Chrome 151/LiveKit localhost A/B measured cold active/render at 1.481/2.257 seconds versus standby 0.200/0.320 with host edges at two. It includes SDK/prewarm and is not public-network evidence. - Atomic activation passed health/access, exact bundle, SQLite preservation, services, and logs; the old release is rollback-ready. Responsive 320/375/390 px checks passed without live media. -- Host A+B aligns capture/outbound identity, deltas, path, and nullable transport-bound codec/`scalabilityMode`; stale or ambiguous data stays unknown and no raw media metadata is retained. Review found no P1/P2; 22 files/299 tests and both builds passed. +- Host A+B aligns capture/outbound identity, deltas, path, and nullable transport-bound codec/`scalabilityMode`. Authenticated Viewer C is limited to the current ordinary or peer-assisted P2P hop and its connection/revision; two-second windows are at most 2 KiB, sanitized, generation-bound, and read-only. Stale, ambiguous, and SFU-fed evidence fails closed; no raw media metadata is retained. - HTTPS/WSS, access and room/WebSocket auth, renewal, public STUN, and authenticated TURN/UDP/TCP relay-only paths pass; TURN/TLS is off. - Draft #16/#18/#22/#23/#25/#28 passed isolated experiments only. ADR-0006 reached host setup/one encoder output, then timed out before viewer 1 decoded/rendered. @@ -37,7 +37,7 @@ unconfigured, so production remains one host connection per viewer and P2P/TURN. - Silent partitions can wait 30 to 60 seconds for heartbeat detection before the default 5-second grace; this remains unverified. - Real audio and heterogeneous clients remain unverified. Production reports poor film audio and voice-call self-echo under system capture; there is no app audio ceiling or Web process isolation. Diagnose A/B/C and sync, then gate Windows 11 game-process audio; Windows 10 stays unresolved without system fallback. - Production `769de201f7cc` reportedly sustains native `qualityLimitationReason=bandwidth` and severe blur with one capable LAN/direct viewer; only Host refresh restores quality. Within the five-second grace, stable-clientId Viewer refresh retains its `peerId`/`HostPeer`, while Host refresh rebuilds peers/capture; churn persistence is also reported. This prioritizes but does not prove sender/PC/GCC/capture generation over network or lifecycle causes. -- ADR-0007 remains incomplete. Host A+B codec fields exist, but browser support and the reported case are unverified; authenticated C, the controller, and `LOW` are absent. Weak paths must share one `LOW`; fail closed protects `HIGH` only exceptionally, and an unreliable supported cohort fails acceptance. +- ADR-0007 remains incomplete. A+B/C browser support and the reported case are unverified; the controller, `LOW`, and an authenticated SFU last-hop C generation are absent. Weak paths must share one `LOW`; fail closed protects `HIGH` only exceptionally, and an unreliable supported cohort fails acceptance. - Browser relays re-encode. ADR-0006 is no-go-unclassified: downstream checkpoints and two-edge/FIFO/TURN gates are absent. Spikes prove one WebCodecs object, not hardware; GCC+RTX is no-go and no-RTX weakens stats. - Browser fanout is host two/viewer one; any accepted endpoint relay stays capped at two downstream edges. - ADR-0005 accepts SFU-root fallback, but code is failure-only/TURN-required; UDP-first config, edge grants, bounded failure, cost/trust, and public canary are absent. @@ -46,14 +46,13 @@ unconfigured, so production remains one host connection per viewer and P2P/TURN. ## Next Milestone -Use Host A+B to compare exact production and current `main` on one wired/direct +Use correlated Host A+B/Viewer C to compare exact production and current `main` on one wired/direct game fixture. Record actual codec/encoder, game FPS, CPU/GPU, preview on/off and A+B while separately rebuilding one `HostPeer`, replacing capture, and fully -refreshing Host; retain the matching Viewer-local sample until authenticated C -lands. Only a proven stuck generation justifies guarded recovery; never +refreshing Host. Only a proven stuck generation justifies guarded recovery; never reconnect periodically, force AV1, or raise ceilings blindly. -After authenticated video C, test simulcast/LiveKit/SVC in order and stop at the +After that reproduction, test simulcast/LiveKit/SVC in order and stop at the first fit before custom `LOW`, independently of ADR-0006. Diagnose audio A/B/C, sync, and voice-source leakage in parallel when it does not displace that P0; `maxBitrate` is not a quality fix. WebRTC/LiveKit owns congestion/layers; the app diff --git a/src/client/media/viewer-quality-evidence.ts b/src/client/media/viewer-quality-evidence.ts new file mode 100644 index 0000000..2f7973d --- /dev/null +++ b/src/client/media/viewer-quality-evidence.ts @@ -0,0 +1,250 @@ +import { + MAX_MEDIA_ROUTE_REVISION, + MAX_VIEWER_QUALITY_EVIDENCE_BYTES, + VIEWER_QUALITY_EVIDENCE_INTERVAL_MS, + viewerQualityEvidenceMessageSchema, + viewerQualityEvidenceMetricsSchema, + type ClientMessage, + type ServerMessage, + type ViewerQualityEvidenceMetrics, +} from "../../shared/protocol"; +import { + EMPTY_METRICS, + type ConnectionMetrics, + type PeerSnapshot, +} from "../types"; + +type ViewerQualityEvidence = Extract< + ServerMessage, + { type: "viewer-quality-evidence" } +>; + +type ViewerQualityEvidenceWindow = Pick< + Extract, + "windowMs" | "metrics" +>; + +function boundedNumber(value: number | null, maximum: number): number | null { + return value !== null && Number.isFinite(value) && value >= 0 && value <= maximum + ? value + : null; +} + +function boundedInteger(value: number | null, maximum: number): number | null { + return value !== null && Number.isInteger(value) && value >= 0 && value <= maximum + ? value + : null; +} + +function boundedString( + value: string | null, + maximumLength: number, + pattern: RegExp, +): string | null { + return value !== null && + value.length <= maximumLength && + pattern.test(value) + ? value + : null; +} + +export function qualityEvidenceWindowFromMetrics( + metrics: ConnectionMetrics, +): ViewerQualityEvidenceWindow | null { + if (metrics.sampleWindowMs === null || !Number.isFinite(metrics.sampleWindowMs)) { + return null; + } + const windowMs = Math.round(metrics.sampleWindowMs); + if (windowMs < 1_000 || windowMs > 5_000) { + return null; + } + + const boundedWidth = boundedInteger(metrics.frameWidth, 16_384); + const boundedHeight = boundedInteger(metrics.frameHeight, 16_384); + const width = boundedWidth !== null && boundedWidth >= 1 ? boundedWidth : null; + const height = + boundedHeight !== null && boundedHeight >= 1 ? boundedHeight : null; + const dimensions = + width !== null && height !== null + ? { width, height } + : { width: null, height: null }; + const freezeDuration = boundedNumber( + metrics.intervalFreezeDurationMs, + windowMs, + ); + const candidate: ViewerQualityEvidenceMetrics = { + ...dimensions, + framesPerSecond: boundedNumber(metrics.framesPerSecond, 240), + bitrateKbps: boundedNumber(metrics.bitrateKbps, 100_000), + packetsReceivedDelta: boundedInteger( + metrics.intervalPacketsReceived, + 1_000_000, + ), + packetsLostDelta: boundedInteger(metrics.intervalPacketsLost, 1_000_000), + jitterMs: boundedNumber(metrics.jitterMs, 60_000), + framesDecodedDelta: boundedInteger( + metrics.intervalFramesDecoded, + 10_000, + ), + framesDroppedDelta: boundedInteger( + metrics.intervalFramesDropped, + 10_000, + ), + decodeMsPerFrame: boundedNumber(metrics.intervalDecodeMs, 60_000), + freezeCountDelta: boundedInteger(metrics.intervalFreezeCount, 10_000), + freezeDurationMsDelta: freezeDuration, + codec: boundedString( + metrics.codec, + 64, + /^video\/[A-Za-z0-9.+-]{1,32}$/i, + ), + codecProfile: boundedString( + metrics.codecProfile, + 64, + /^[a-z0-9-]+=[a-z0-9]+$/, + ), + codecParameters: boundedString( + metrics.codecParameters, + 128, + /^[a-z0-9-]+=[a-z0-9]+(?:; [a-z0-9-]+=[a-z0-9]+)*$/, + ), + }; + const parsed = viewerQualityEvidenceMetricsSchema.safeParse(candidate); + return parsed.success ? { windowMs, metrics: parsed.data } : null; +} + +export class ViewerQualityEvidenceReporter { + private connectionId: string | null = null; + private routeRevision: number | null = null; + private sequence = 0; + private lastSentAtMs: number | null = null; + private lastSampleTimestampMs: number | null = null; + + constructor( + private readonly send: (message: ClientMessage) => boolean, + private readonly now: () => number = Date.now, + ) {} + + offer(snapshot: PeerSnapshot, routeRevision: number): boolean { + if ( + !Number.isSafeInteger(routeRevision) || + routeRevision < 0 || + routeRevision > MAX_MEDIA_ROUTE_REVISION + ) { + return false; + } + const window = qualityEvidenceWindowFromMetrics(snapshot.metrics); + const sampleTimestampMs = snapshot.metrics.sampleTimestampMs; + if ( + !window || + sampleTimestampMs === null || + !Number.isFinite(sampleTimestampMs) || + sampleTimestampMs < 0 + ) { + return false; + } + if (this.connectionId !== snapshot.connectionId) { + this.connectionId = snapshot.connectionId; + this.routeRevision = routeRevision; + this.sequence = 0; + this.lastSentAtMs = null; + this.lastSampleTimestampMs = null; + } else if (this.routeRevision !== routeRevision) { + this.routeRevision = routeRevision; + this.lastSentAtMs = null; + this.lastSampleTimestampMs = null; + } + if (this.sequence > Number.MAX_SAFE_INTEGER) { + return false; + } + + if ( + this.lastSampleTimestampMs !== null && + sampleTimestampMs <= this.lastSampleTimestampMs + ) { + return false; + } + const now = this.now(); + if ( + !Number.isFinite(now) || + (this.lastSentAtMs !== null && + now - this.lastSentAtMs < VIEWER_QUALITY_EVIDENCE_INTERVAL_MS) + ) { + return false; + } + + const message = { + type: "viewer-quality-evidence" as const, + guard: { + connectionId: snapshot.connectionId, + routeRevision, + }, + sequence: this.sequence, + ...window, + }; + const parsed = viewerQualityEvidenceMessageSchema.safeParse(message); + if ( + !parsed.success || + new TextEncoder().encode(JSON.stringify(parsed.data)).byteLength > + MAX_VIEWER_QUALITY_EVIDENCE_BYTES || + !this.send(parsed.data) + ) { + return false; + } + + this.sequence += 1; + this.lastSentAtMs = now; + this.lastSampleTimestampMs = sampleTimestampMs; + return true; + } + + reset(): void { + this.connectionId = null; + this.routeRevision = null; + this.sequence = 0; + this.lastSentAtMs = null; + this.lastSampleTimestampMs = null; + } +} + +export function qualityEvidenceMatchesSnapshot( + evidence: ViewerQualityEvidence, + snapshot: PeerSnapshot | null, +): boolean { + return ( + snapshot !== null && + snapshot.peerId === evidence.viewerPeerId && + snapshot.connectionId === evidence.guard.connectionId + ); +} + +export function metricsFromQualityEvidence( + evidence: ViewerQualityEvidence, +): ConnectionMetrics { + const metrics = evidence.metrics; + return { + ...EMPTY_METRICS, + sampleWindowMs: evidence.windowMs, + framesPerSecond: metrics.framesPerSecond, + frameWidth: metrics.width, + frameHeight: metrics.height, + resolution: + metrics.width !== null && metrics.height !== null + ? `${metrics.width}x${metrics.height}` + : null, + bitrateKbps: metrics.bitrateKbps, + packetsLost: metrics.packetsLostDelta, + intervalPacketsReceived: metrics.packetsReceivedDelta, + intervalPacketsLost: metrics.packetsLostDelta, + jitterMs: metrics.jitterMs, + framesDropped: metrics.framesDroppedDelta, + intervalFramesDecoded: metrics.framesDecodedDelta, + intervalFramesDropped: metrics.framesDroppedDelta, + intervalDecodeMs: metrics.decodeMsPerFrame, + intervalFreezeCount: metrics.freezeCountDelta, + intervalFreezeDurationMs: metrics.freezeDurationMsDelta, + codec: metrics.codec, + codecProfile: metrics.codecProfile, + codecParameters: metrics.codecParameters, + }; +} diff --git a/src/client/pages/HostPage.tsx b/src/client/pages/HostPage.tsx index ad8e9a7..7150390 100644 --- a/src/client/pages/HostPage.tsx +++ b/src/client/pages/HostPage.tsx @@ -12,6 +12,7 @@ import { import { useEffect, useMemo, useRef, useState } from "react"; import { DEFAULT_QUALITY_SETTINGS, + VIEWER_QUALITY_EVIDENCE_EXPIRY_MS, type CreateRoomResponse, type IceConfig, type ServerMessage, @@ -52,6 +53,7 @@ import { } from "../media/quality"; import { HostSfuRoute } from "../media/host-sfu-route"; import { SfuStandbyPrewarmer } from "../media/sfu-standby-prewarmer"; +import { metricsFromQualityEvidence } from "../media/viewer-quality-evidence"; import type { PeerSnapshot, SignalConnectionState, @@ -65,6 +67,11 @@ import { sourceSwitchNotice } from "./host-page-notices"; type HostPhase = "idle" | "starting" | "live" | "ended" | "error"; +type ViewerQualityEvidence = Extract< + ServerMessage, + { type: "viewer-quality-evidence" } +>; + interface CaptureDetails { resolution: string; frameRate: number | null; @@ -142,6 +149,9 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) { const [peerSnapshots, setPeerSnapshots] = useState>( () => new Map(), ); + const [viewerQualityEvidence, setViewerQualityEvidence] = useState< + Map + >(() => new Map()); const [notice, setNotice] = useState(null); const [copied, setCopied] = useState(false); const [switchingSource, setSwitchingSource] = useState(false); @@ -154,7 +164,14 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) { const signalRef = useRef(null); const iceConfigRef = useRef(null); const peersRef = useRef(new Map()); + const viewerQualityEvidenceRef = useRef( + new Map(), + ); + const viewerQualityEvidenceTimersRef = useRef( + new Map(), + ); const peerAssistedRef = useRef(false); + const activeRouteRevisionRef = useRef(0); const generationRef = useRef(0); const activeGenerationRef = useRef(null); const sourceSwitchRef = useRef(null); @@ -193,6 +210,12 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) { signalRef.current?.stop(); peersRef.current.forEach((peer) => peer.dispose()); peersRef.current.clear(); + viewerQualityEvidenceTimersRef.current.forEach((timer) => + window.clearTimeout(timer), + ); + viewerQualityEvidenceTimersRef.current.clear(); + viewerQualityEvidenceRef.current.clear(); + activeRouteRevisionRef.current = 0; void hostSfuRouteRef.current?.disconnect(); hostSfuRouteRef.current = null; sfuStandbyPrewarmerRef.current?.dispose(); @@ -301,6 +324,13 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) { setRelayAvailable(false); setMaxViewers(null); setPeerSnapshots(new Map()); + viewerQualityEvidenceTimersRef.current.forEach((timer) => + window.clearTimeout(timer), + ); + viewerQualityEvidenceTimersRef.current.clear(); + viewerQualityEvidenceRef.current = new Map(); + setViewerQualityEvidence(new Map()); + activeRouteRevisionRef.current = 0; setSignalStatus("offline"); setSwitchingSource(false); setChangingQuality(false); @@ -327,6 +357,13 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) { } function updatePeerSnapshot(snapshot: PeerSnapshot): void { + const evidence = viewerQualityEvidenceRef.current.get(snapshot.peerId); + if ( + evidence && + evidence.guard.connectionId !== snapshot.connectionId + ) { + clearViewerQualityEvidence(snapshot.peerId); + } setPeerSnapshots((current) => { const next = new Map(current); next.set(snapshot.peerId, snapshot); @@ -334,6 +371,59 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) { }); } + function clearViewerQualityEvidence(peerId: string): void { + const timer = viewerQualityEvidenceTimersRef.current.get(peerId); + if (timer !== undefined) { + window.clearTimeout(timer); + viewerQualityEvidenceTimersRef.current.delete(peerId); + } + if (!viewerQualityEvidenceRef.current.has(peerId)) { + return; + } + const next = new Map(viewerQualityEvidenceRef.current); + next.delete(peerId); + viewerQualityEvidenceRef.current = next; + setViewerQualityEvidence(next); + } + + function clearAllViewerQualityEvidence(): void { + viewerQualityEvidenceTimersRef.current.forEach((timer) => + window.clearTimeout(timer), + ); + viewerQualityEvidenceTimersRef.current.clear(); + viewerQualityEvidenceRef.current = new Map(); + setViewerQualityEvidence(new Map()); + } + + function acceptViewerQualityEvidence(evidence: ViewerQualityEvidence): void { + const peer = peersRef.current.get(evidence.viewerPeerId); + if ( + !peer || + peer.connectionId !== evidence.guard.connectionId || + evidence.guard.routeRevision !== activeRouteRevisionRef.current + ) { + return; + } + const next = new Map(viewerQualityEvidenceRef.current); + next.set(evidence.viewerPeerId, evidence); + viewerQualityEvidenceRef.current = next; + setViewerQualityEvidence(next); + const previousTimer = viewerQualityEvidenceTimersRef.current.get( + evidence.viewerPeerId, + ); + if (previousTimer !== undefined) { + window.clearTimeout(previousTimer); + } + const timer = window.setTimeout(() => { + if ( + viewerQualityEvidenceRef.current.get(evidence.viewerPeerId) === evidence + ) { + clearViewerQualityEvidence(evidence.viewerPeerId); + } + }, VIEWER_QUALITY_EVIDENCE_EXPIRY_MS); + viewerQualityEvidenceTimersRef.current.set(evidence.viewerPeerId, timer); + } + function watchCaptureEnd( captured: MediaStream, generation: number, @@ -471,6 +561,7 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) { } function removePeer(peerId: string): void { + clearViewerQualityEvidence(peerId); peersRef.current.get(peerId)?.dispose(); peersRef.current.delete(peerId); setPeerSnapshots((current) => { @@ -625,6 +716,7 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) { return; } if (message.type === "authenticated" && message.role === "host") { + clearAllViewerQualityEvidence(); setSfuStandbyUrl( "sfuStandbyUrl" in message ? message.sfuStandbyUrl : null, ); @@ -633,6 +725,7 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) { "mediaMode" in message && message.mediaMode === "peer-assisted" ) { + activeRouteRevisionRef.current = message.routeRevision; peerAssistedRef.current = true; signalRef.current?.send({ type: "set-quality-settings", @@ -648,6 +741,7 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) { .then(() => showHostSfuQualityWarning(route, generation)); return; } + activeRouteRevisionRef.current = 0; peerAssistedRef.current = false; clearHostSfuRoute(); const currentViewerIds = new Set(message.viewerPeerIds); @@ -660,6 +754,13 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) { } if (message.type === "route-update") { if (peerAssistedRef.current) { + if ( + message.phase === "active" && + message.revision !== activeRouteRevisionRef.current + ) { + activeRouteRevisionRef.current = message.revision; + clearAllViewerQualityEvidence(); + } const route = ensureHostSfuRoute(generation); void route .acceptAndWait(message) @@ -667,6 +768,10 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) { } return; } + if (message.type === "viewer-quality-evidence") { + acceptViewerQualityEvidence(message); + return; + } if (message.type === "sfu-config") { if (peerAssistedRef.current) { const route = ensureHostSfuRoute(generation); @@ -1432,11 +1537,24 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) { )} {showConnectionDetails && ( - + <> + + {viewerQualityEvidence.get(viewer.peerId) && ( + <> +

观看端接收

+ + + )} + )} {viewer.error &&

{viewer.error}

} diff --git a/src/client/pages/ViewerPage.tsx b/src/client/pages/ViewerPage.tsx index 7b9f1d4..d6d5e87 100644 --- a/src/client/pages/ViewerPage.tsx +++ b/src/client/pages/ViewerPage.tsx @@ -9,6 +9,7 @@ import { import { useEffect, useMemo, useRef, useState } from "react"; import { DEFAULT_QUALITY_SETTINGS, + VIEWER_QUALITY_EVIDENCE_EXPIRY_MS, type IceConfig, type MediaAssignment, type ServerMessage, @@ -28,6 +29,11 @@ import { SignalingClient } from "../lib/signaling"; import type { QualitySettings } from "../media/quality"; import { relayCapacityMessageForBrowser } from "../media/relay-capability"; import { SfuStandbyPrewarmer } from "../media/sfu-standby-prewarmer"; +import { + metricsFromQualityEvidence, + qualityEvidenceMatchesSnapshot, + ViewerQualityEvidenceReporter, +} from "../media/viewer-quality-evidence"; import { ViewerMessageAuthority } from "../media/viewer-message-authority"; import { ViewerSfuRoute } from "../media/viewer-sfu-route"; import type { @@ -48,6 +54,11 @@ interface ViewerPageProps { onAuthorizationRequired: () => void; } +type ViewerQualityEvidence = Extract< + ServerMessage, + { type: "viewer-quality-evidence" } +>; + export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) { const forceRelay = useMemo( () => new URLSearchParams(window.location.search).get("relay") === "1", @@ -61,6 +72,8 @@ export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) const [remoteStream, setRemoteStream] = useState(null); const [peerSnapshot, setPeerSnapshot] = useState(null); const [relaySnapshot, setRelaySnapshot] = useState(null); + const [relayChildEvidence, setRelayChildEvidence] = + useState(null); const [playbackBlocked, setPlaybackBlocked] = useState(false); const [muted, setMuted] = useState(false); const [showConnectionDetails, setShowConnectionDetails] = useState(false); @@ -83,6 +96,8 @@ export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) let currentIceConfig: IceConfig | null = null; let currentHostOnline = false; let peerAssisted = false; + let currentPeerId: string | null = null; + let currentRouteRevision = 0; let currentQualitySettings: QualitySettings = DEFAULT_QUALITY_SETTINGS; let currentAssignment: MediaAssignment = { parentPeerId: null, @@ -96,6 +111,8 @@ export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) > = []; const messageAuthority = new ViewerMessageAuthority(); let sfuStandbyPrewarmer: SfuStandbyPrewarmer | null = null; + let relayChildEvidenceCurrent: ViewerQualityEvidence | null = null; + let relayChildEvidenceTimer: number | null = null; function setSfuStandbyUrl(url: string | null | undefined): void { if (!url) { @@ -150,6 +167,37 @@ export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) }, }, ); + const qualityEvidenceReporter = new ViewerQualityEvidenceReporter( + (message) => active && signal.send(message), + ); + + function clearRelayChildEvidence(): void { + relayChildEvidenceCurrent = null; + setRelayChildEvidence(null); + if (relayChildEvidenceTimer !== null) { + window.clearTimeout(relayChildEvidenceTimer); + relayChildEvidenceTimer = null; + } + } + + function acceptRelayChildEvidence(evidence: ViewerQualityEvidence): void { + if ( + !peerAssisted || + evidence.parentPeerId !== currentPeerId || + evidence.guard.routeRevision !== currentRouteRevision || + !qualityEvidenceMatchesSnapshot(evidence, viewerRelay?.getSnapshot() ?? null) + ) { + return; + } + clearRelayChildEvidence(); + relayChildEvidenceCurrent = evidence; + setRelayChildEvidence(evidence); + relayChildEvidenceTimer = window.setTimeout(() => { + if (relayChildEvidenceCurrent === evidence) { + clearRelayChildEvidence(); + } + }, VIEWER_QUALITY_EVIDENCE_EXPIRY_MS); + } function ensureViewerRelay(): ViewerRelay | null { if (viewerRelay) { @@ -175,6 +223,15 @@ export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) onUpdate: (snapshot) => { if (active) { setRelaySnapshot(snapshot); + if ( + relayChildEvidenceCurrent && + !qualityEvidenceMatchesSnapshot( + relayChildEvidenceCurrent, + snapshot, + ) + ) { + clearRelayChildEvidence(); + } } }, }, @@ -301,6 +358,7 @@ export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) } function clearUpstreamState(): void { + qualityEvidenceReporter.reset(); peerRef.current?.dispose(); peerRef.current = null; setRemoteStream(null); @@ -319,12 +377,16 @@ export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) MAX_VIEWER_MEDIA_CHILDREN, ); const previousParentId = currentAssignment.parentPeerId; + const previousChildId = currentAssignment.childPeerIds[0] ?? null; currentAssignment = nextAssignment; if (previousParentId !== nextAssignment.parentPeerId) { clearUpstreamState(); setStatusText("正在恢复连接"); } + if (previousChildId !== (nextAssignment.childPeerIds[0] ?? null)) { + clearRelayChildEvidence(); + } ensureViewerRelay()?.setChild(nextAssignment.childPeerIds[0] ?? null); } @@ -362,6 +424,7 @@ export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) }, onUpdate: (snapshot) => { if (active) { + qualityEvidenceReporter.offer(snapshot, currentRouteRevision); if ( !currentHostOnline && snapshot.connectionState === "failed" @@ -395,6 +458,8 @@ export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) authorityToken: number, ): Promise { if (message.type === "authenticated") { + clearRelayChildEvidence(); + currentPeerId = message.peerId; setSfuStandbyUrl( "sfuStandbyUrl" in message ? message.sfuStandbyUrl : null, ); @@ -408,6 +473,14 @@ export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) currentAssignment = { parentPeerId: null, childPeerIds: [] }; } peerAssisted = nextPeerAssisted; + const nextRouteRevision = + nextPeerAssisted && "routeRevision" in message + ? message.routeRevision + : 0; + if (nextRouteRevision !== currentRouteRevision) { + clearRelayChildEvidence(); + } + currentRouteRevision = nextRouteRevision; const relayCapacity = relayCapacityMessageForBrowser(nextPeerAssisted); if (relayCapacity) { signal.send(relayCapacity); @@ -467,10 +540,20 @@ export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) } if (message.type === "route-update") { if (peerAssisted) { + if (message.phase === "active") { + if (message.revision !== currentRouteRevision) { + clearRelayChildEvidence(); + } + currentRouteRevision = message.revision; + } ensureViewerSfuRoute().accept(message); } return; } + if (message.type === "viewer-quality-evidence") { + acceptRelayChildEvidence(message); + return; + } if (message.type === "sfu-config") { if (peerAssisted) { await ensureViewerSfuRoute().acceptConfig(message); @@ -591,6 +674,13 @@ export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) signal.start(); return () => { active = false; + currentPeerId = null; + qualityEvidenceReporter.reset(); + relayChildEvidenceCurrent = null; + if (relayChildEvidenceTimer !== null) { + window.clearTimeout(relayChildEvidenceTimer); + relayChildEvidenceTimer = null; + } sfuStandbyPrewarmer?.dispose(); signal.stop(); void viewerSfuRoute?.disconnect(); @@ -797,6 +887,18 @@ export function ViewerPage({ roomId, onAuthorizationRequired }: ViewerPageProps) /> )} + {showConnectionDetails && relayChildEvidence && ( +
+

下游接收数据

+ +
+ )} ); diff --git a/src/client/types.ts b/src/client/types.ts index 5ac4efa..2e72678 100644 --- a/src/client/types.ts +++ b/src/client/types.ts @@ -26,10 +26,15 @@ export interface ConnectionMetrics { bitrateKbps: number | null; availableOutgoingKbps: number | null; framesPerSecond: number | null; + frameWidth: number | null; + frameHeight: number | null; resolution: string | null; packetsLost: number | null; + intervalPacketsReceived: number | null; + intervalPacketsLost: number | null; jitterMs: number | null; framesDropped: number | null; + intervalFramesDecoded: number | null; intervalFramesDropped: number | null; intervalFreezeCount: number | null; intervalFreezeDurationMs: number | null; @@ -77,10 +82,15 @@ export const EMPTY_METRICS: ConnectionMetrics = { bitrateKbps: null, availableOutgoingKbps: null, framesPerSecond: null, + frameWidth: null, + frameHeight: null, resolution: null, packetsLost: null, + intervalPacketsReceived: null, + intervalPacketsLost: null, jitterMs: null, framesDropped: null, + intervalFramesDecoded: null, intervalFramesDropped: null, intervalFreezeCount: null, intervalFreezeDurationMs: null, diff --git a/src/client/webrtc/stats.ts b/src/client/webrtc/stats.ts index 475540e..95d38c6 100644 --- a/src/client/webrtc/stats.ts +++ b/src/client/webrtc/stats.ts @@ -2,6 +2,7 @@ import { EMPTY_METRICS, type ConnectionMetrics, } from "../types"; +import { deriveVideoCodecEvidence } from "../../shared/video-codec-evidence"; type StatsRecord = Record & { id: string; @@ -18,6 +19,8 @@ export interface StatsAccumulator { timestamp: number | null; previousTotalEncodeTime: number | null; previousTotalDecodeTime: number | null; + previousPacketsReceived: number | null; + previousPacketsLost: number | null; previousFramesDropped: number | null; previousFreezeCount: number | null; previousTotalFreezesDuration: number | null; @@ -39,6 +42,8 @@ export function createStatsAccumulator(): StatsAccumulator { timestamp: null, previousTotalEncodeTime: null, previousTotalDecodeTime: null, + previousPacketsReceived: null, + previousPacketsLost: null, previousFramesDropped: null, previousFreezeCount: null, previousTotalFreezesDuration: null, @@ -200,56 +205,6 @@ function linkedRemoteInbound( : null; } -interface CodecEvidence { - codec: string | null; - profile: string | null; - parameters: string | null; -} - -type CodecParameterRule = { - profileKey: string | null; - parameterKeys: readonly string[]; - validators: Readonly>; -}; - -const CODEC_PARAMETER_RULES: Readonly> = { - "video/h264": { - profileKey: "profile-level-id", - parameterKeys: ["packetization-mode", "level-asymmetry-allowed"], - validators: { - "profile-level-id": /^[0-9a-f]{6}$/i, - "packetization-mode": /^[0-2]$/, - "level-asymmetry-allowed": /^[01]$/, - }, - }, - "video/vp9": { - profileKey: "profile-id", - parameterKeys: ["max-fr", "max-fs"], - validators: { - "profile-id": /^[0-3]$/, - "max-fr": /^[1-9][0-9]{0,9}$/, - "max-fs": /^[1-9][0-9]{0,9}$/, - }, - }, - "video/vp8": { - profileKey: null, - parameterKeys: ["max-fr", "max-fs"], - validators: { - "max-fr": /^[1-9][0-9]{0,9}$/, - "max-fs": /^[1-9][0-9]{0,9}$/, - }, - }, - "video/av1": { - profileKey: "profile", - parameterKeys: ["level-idx", "tier"], - validators: { - profile: /^[0-2]$/, - "level-idx": /^(?:[0-9]|[12][0-9]|3[01])$/, - tier: /^[01]$/, - }, - }, -}; - function linkedVideoCodec( report: RTCStatsReport, media: StatsRecord | null, @@ -266,62 +221,12 @@ function linkedVideoCodec( : null; } -function deriveCodecEvidence(codec: StatsRecord | null): CodecEvidence { +function deriveCodecEvidence(codec: StatsRecord | null) { const mimeType = stringValue(codec, "mimeType"); - if (!mimeType) { - return { codec: null, profile: null, parameters: null }; - } - const rule = CODEC_PARAMETER_RULES[mimeType.toLowerCase()]; - const fmtp = stringValue(codec, "sdpFmtpLine"); - if (!rule || !fmtp || fmtp.length > 2_048) { - return { codec: mimeType, profile: null, parameters: null }; - } - - const segments = fmtp.split(";"); - if (segments.length > 32) { - return { codec: mimeType, profile: null, parameters: null }; - } - const values = new Map(); - const seenKeys = new Set(); - for (const segment of segments) { - const separator = segment.indexOf("="); - if (separator < 1) { - continue; - } - const key = segment.slice(0, separator).trim().toLowerCase(); - const validator = rule.validators[key]; - if (!validator) { - continue; - } - if (seenKeys.has(key)) { - values.delete(key); - continue; - } - seenKeys.add(key); - const value = segment.slice(separator + 1).trim().toLowerCase(); - if (!validator.test(value)) { - continue; - } - values.set(key, value); - } - - const profileValue = rule.profileKey - ? values.get(rule.profileKey) - : undefined; - const parameters = rule.parameterKeys - .flatMap((key) => { - const value = values.get(key); - return value ? [`${key}=${value}`] : []; - }) - .join("; "); - return { - codec: mimeType, - profile: - rule.profileKey && profileValue - ? `${rule.profileKey}=${profileValue}` - : null, - parameters: parameters || null, - }; + return deriveVideoCodecEvidence( + mimeType, + stringValue(codec, "sdpFmtpLine"), + ); } function scalabilityModeValue(media: StatsRecord | null): string | null { @@ -383,6 +288,11 @@ export async function collectConnectionMetrics( const timestamp = numberValue(media, "timestamp"); const framesEncoded = numberValue(media, "framesEncoded"); const framesDecoded = numberValue(media, "framesDecoded"); + const packetsReceived = numberValue(media, "packetsReceived"); + const packetsLost = numberValue( + direction === "send" ? remoteInbound : media, + "packetsLost", + ); const totalEncodeTime = numberValue(media, "totalEncodeTime"); const totalDecodeTime = numberValue(media, "totalDecodeTime"); const framesDropped = numberValue(media, "framesDropped"); @@ -441,6 +351,30 @@ export async function collectConnectionMetrics( sampleWindowMs !== null ? previous.frames : null, ) : null; + const intervalPacketsReceived = + direction === "receive" + ? intervalDelta( + packetsReceived, + previous.previousPacketsReceived, + sampleWindowMs !== null, + ) + : null; + const intervalPacketsLost = + direction === "receive" + ? intervalDelta( + packetsLost, + previous.previousPacketsLost, + sampleWindowMs !== null, + ) + : null; + const intervalFramesDecoded = + direction === "receive" + ? intervalDelta( + framesDecoded, + previous.frames, + sampleWindowMs !== null, + ) + : null; const intervalFramesDropped = intervalDelta( framesDropped, previous.previousFramesDropped, @@ -474,6 +408,8 @@ export async function collectConnectionMetrics( previous.timestamp = timestamp; previous.previousTotalEncodeTime = totalEncodeTime; previous.previousTotalDecodeTime = totalDecodeTime; + previous.previousPacketsReceived = packetsReceived; + previous.previousPacketsLost = packetsLost; previous.previousFramesDropped = framesDropped; previous.previousFreezeCount = freezeCount; previous.previousTotalFreezesDuration = totalFreezesDuration; @@ -481,14 +417,7 @@ export async function collectConnectionMetrics( previous.previousRetransmittedBytes = retransmittedBytes; const linkedCodec = linkedVideoCodec(report, media, transport); - const codecEvidence = - direction === "send" - ? deriveCodecEvidence(linkedCodec) - : { - codec: stringValue(linkedCodec, "mimeType"), - profile: null, - parameters: null, - }; + const codecEvidence = deriveCodecEvidence(linkedCodec); const width = numberValue(media, "frameWidth"); const height = numberValue(media, "frameHeight"); const iceProtocol = @@ -523,15 +452,19 @@ export async function collectConnectionMetrics( ? numberValue(pair, "availableOutgoingBitrate")! / 1_000 : null, framesPerSecond: numberValue(media, "framesPerSecond") ?? derivedFps, + frameWidth: width, + frameHeight: height, resolution: width !== null && height !== null ? `${width}x${height}` : null, - packetsLost: - numberValue(direction === "send" ? remoteInbound : media, "packetsLost"), + packetsLost, + intervalPacketsReceived, + intervalPacketsLost, jitterMs: numberValue(direction === "send" ? remoteInbound : media, "jitter") !== null ? numberValue(direction === "send" ? remoteInbound : media, "jitter")! * 1_000 : null, framesDropped, + intervalFramesDecoded, intervalFramesDropped, intervalFreezeCount, intervalFreezeDurationMs: diff --git a/src/server/hybrid-media-router.ts b/src/server/hybrid-media-router.ts index e6fe652..941c4bf 100644 --- a/src/server/hybrid-media-router.ts +++ b/src/server/hybrid-media-router.ts @@ -207,17 +207,35 @@ export class HybridMediaRouter { parentPeerId: string, childPeerId: string, ): boolean { + return ( + this.resolveActivePeerEdge(roomId, childPeerId)?.parentPeerId === + parentPeerId + ); + } + + resolveActivePeerEdge( + roomId: string, + childPeerId: string, + ): { revision: number; parentPeerId: string } | undefined { const controller = this.mediaRouteControllers.get(roomId); if (!controller) { - return false; + return undefined; } - const assignments = controller.getActiveRoute().assignments; + const active = controller.getActiveRoute(); + const assignments = active.assignments; const child = assignments.get(childPeerId); - return ( - child?.upstream.kind === "peer" && - child.upstream.peerId === parentPeerId && - assignments.get(parentPeerId)?.childPeerIds.includes(childPeerId) === true - ); + if ( + child?.upstream.kind !== "peer" || + assignments + .get(child.upstream.peerId) + ?.childPeerIds.includes(childPeerId) !== true + ) { + return undefined; + } + return { + revision: active.revision, + parentPeerId: child.upstream.peerId, + }; } setViewerRelayCapacity( diff --git a/src/server/signaling.ts b/src/server/signaling.ts index 41edf99..ef84c99 100644 --- a/src/server/signaling.ts +++ b/src/server/signaling.ts @@ -7,13 +7,19 @@ import WebSocket, { WebSocketServer } from "ws"; import { DEFAULT_QUALITY_SETTINGS, MAX_SIGNAL_BYTES, + MAX_VIEWER_QUALITY_EVIDENCE_BYTES, + VIEWER_QUALITY_EVIDENCE_INTERVAL_MS, decodeClientMessage, type ClientMessage, type QualitySettings, type Role, type ServerMessage, } from "../shared/protocol.js"; -import { RoomStore, RoomStoreError } from "./room-store.js"; +import { + RoomStore, + RoomStoreError, + type ConnectedPeer, +} from "./room-store.js"; import { HybridMediaRouter, type SfuFallbackOptions, @@ -41,6 +47,16 @@ interface SocketState { authenticated?: AuthenticatedSession; } +interface ViewerQualityEvidenceGate { + viewerSessionId: string; + parentSessionId: string; + parentPeerId: string; + connectionId: string; + routeRevision: number; + sequence: number; + acceptedAtMs: number; +} + export interface SignalingOptions { server: HttpServer; roomStore: RoomStore; @@ -70,6 +86,10 @@ export class SignalingServer { private readonly socketsBySessionId = new Map(); private readonly viewerGraceTimers = new Map(); private readonly connectionIdsByViewer = new Map(); + private readonly viewerQualityEvidenceGates = new Map< + string, + ViewerQualityEvidenceGate + >(); private readonly qualitySettingsByRoom = new Map(); private readonly shareGenerationsByRoom = new Map(); private readonly hybridMediaRouter?: HybridMediaRouter; @@ -122,9 +142,7 @@ export class SignalingServer { viewerConnectionKey(roomId, viewerPeerId), ), deleteConnectionId: (roomId, viewerPeerId) => - this.connectionIdsByViewer.delete( - viewerConnectionKey(roomId, viewerPeerId), - ), + this.deleteViewerConnectionId(roomId, viewerPeerId), }); } @@ -185,6 +203,7 @@ export class SignalingServer { } this.viewerGraceTimers.clear(); this.hybridMediaRouter?.close(); + this.viewerQualityEvidenceGates.clear(); this.shareGenerationsByRoom.clear(); this.options.server.off("upgrade", this.upgradeHandler); @@ -242,6 +261,13 @@ export class SignalingServer { this.rejectInvalidMessage(socket); return; } + if ( + message.type === "viewer-quality-evidence" && + Buffer.byteLength(encoded, "utf8") > MAX_VIEWER_QUALITY_EVIDENCE_BYTES + ) { + this.rejectInvalidMessage(socket); + return; + } const state = this.socketStates.get(socket); if (!state) { @@ -339,6 +365,11 @@ export class SignalingServer { : Date.parse(participant.expiresAt), shareGeneration, }; + if (participant.role === "viewer") { + this.viewerQualityEvidenceGates.delete( + viewerConnectionKey(participant.roomId, participant.peerId), + ); + } this.clearViewerGrace(participant.roomId, participant.peerId); const routeParticipant = { roomId: participant.roomId, @@ -570,6 +601,9 @@ export class SignalingServer { message.revision, ); return; + case "viewer-quality-evidence": + this.handleViewerQualityEvidence(socket, authenticated, message); + return; case "stop-sharing": case "close-room": if (authenticated.role !== "host") { @@ -613,6 +647,98 @@ export class SignalingServer { } } + private handleViewerQualityEvidence( + socket: WebSocket, + source: AuthenticatedSession, + message: Extract, + ): void { + if (source.role !== "viewer") { + return; + } + const viewerState = this.socketStates.get(socket); + if (!viewerState || viewerState.authenticated !== source) { + return; + } + + const connectionId = this.connectionIdsByViewer.get( + viewerConnectionKey(source.roomId, source.peerId), + ); + if (!connectionId || message.guard.connectionId !== connectionId) { + return; + } + + let routeRevision = 0; + let parent: ConnectedPeer | undefined; + if (this.isPeerAssistedRoom(source.roomId)) { + const edge = this.hybridMediaRouter?.resolveActivePeerEdge( + source.roomId, + source.peerId, + ); + if (!edge) { + return; + } + routeRevision = edge.revision; + parent = this.connectedPeer(source.roomId, edge.parentPeerId); + } else { + parent = this.options.roomStore.getConnectedHost(source.roomId); + } + if ( + !parent || + message.guard.routeRevision !== routeRevision || + parent.peerId === source.peerId + ) { + return; + } + + const gateKey = viewerConnectionKey(source.roomId, source.peerId); + const previous = this.viewerQualityEvidenceGates.get(gateKey); + const sameGeneration = + previous?.viewerSessionId === viewerState.sessionId && + previous.parentSessionId === parent.sessionId && + previous.parentPeerId === parent.peerId && + previous.connectionId === connectionId && + previous.routeRevision === routeRevision; + const now = this.now(); + if ( + sameGeneration && + previous && + (message.sequence <= previous.sequence || + now - previous.acceptedAtMs < VIEWER_QUALITY_EVIDENCE_INTERVAL_MS) + ) { + return; + } + + const forwarded = { + type: "viewer-quality-evidence" as const, + viewerPeerId: source.peerId, + parentPeerId: parent.peerId, + guard: { + connectionId, + routeRevision, + }, + sequence: message.sequence, + windowMs: message.windowMs, + metrics: message.metrics, + }; + const encoded = JSON.stringify(forwarded); + if ( + Buffer.byteLength(encoded, "utf8") > + MAX_VIEWER_QUALITY_EVIDENCE_BYTES || + !this.sendEncodedToSession(parent.sessionId, encoded) + ) { + return; + } + this.viewerQualityEvidenceGates.set(gateKey, { + viewerSessionId: viewerState.sessionId, + parentSessionId: parent.sessionId, + parentPeerId: parent.peerId, + connectionId, + routeRevision, + sequence: message.sequence, + acceptedAtMs: now, + }); + } + private routeSignal( sourceSocket: WebSocket, source: AuthenticatedSession, @@ -640,8 +766,9 @@ export class SignalingServer { return; } if (description?.type === "offer") { - this.connectionIdsByViewer.set( - viewerConnectionKey(source.roomId, viewer.peerId), + this.setViewerConnectionId( + source.roomId, + viewer.peerId, message.payload.connectionId, ); } @@ -713,8 +840,9 @@ export class SignalingServer { } if (description?.type === "offer") { - this.connectionIdsByViewer.set( - viewerConnectionKey(source.roomId, targetPeerId), + this.setViewerConnectionId( + source.roomId, + targetPeerId, message.payload.connectionId, ); } @@ -819,8 +947,9 @@ export class SignalingServer { disconnected.peerId, ) ) { - this.connectionIdsByViewer.delete( - viewerConnectionKey(disconnected.roomId, disconnected.peerId), + this.deleteViewerConnectionId( + disconnected.roomId, + disconnected.peerId, ); if (this.isPeerAssistedRoom(disconnected.roomId)) { this.hybridMediaRouter!.removeViewer( @@ -977,6 +1106,19 @@ export class SignalingServer { } } + private sendEncodedToSession(sessionId: string, encoded: string): boolean { + const socket = this.socketsBySessionId.get(sessionId); + if (!socket || socket.readyState !== WebSocket.OPEN) { + return false; + } + if (socket.bufferedAmount > MAX_BUFFERED_SIGNAL_BYTES) { + socket.terminate(); + return false; + } + socket.send(encoded); + return true; + } + private send(socket: WebSocket, message: ServerMessage): void { if (socket.readyState !== WebSocket.OPEN) { return; @@ -1023,6 +1165,32 @@ export class SignalingServer { this.connectionIdsByViewer.delete(key); } } + for (const key of this.viewerQualityEvidenceGates.keys()) { + if (key.startsWith(prefix)) { + this.viewerQualityEvidenceGates.delete(key); + } + } + } + + private setViewerConnectionId( + roomId: string, + viewerPeerId: string, + connectionId: string, + ): void { + const key = viewerConnectionKey(roomId, viewerPeerId); + if (this.connectionIdsByViewer.get(key) !== connectionId) { + this.viewerQualityEvidenceGates.delete(key); + } + this.connectionIdsByViewer.set(key, connectionId); + } + + private deleteViewerConnectionId( + roomId: string, + viewerPeerId: string, + ): void { + const key = viewerConnectionKey(roomId, viewerPeerId); + this.connectionIdsByViewer.delete(key); + this.viewerQualityEvidenceGates.delete(key); } } diff --git a/src/shared/protocol.ts b/src/shared/protocol.ts index 1c9db32..15c94ff 100644 --- a/src/shared/protocol.ts +++ b/src/shared/protocol.ts @@ -1,10 +1,15 @@ import { z } from "zod"; +import { isCanonicalVideoCodecEvidence } from "./video-codec-evidence.js"; + export const MAX_VIEWERS_PER_ROOM_LIMIT = 16; export const MAX_SIGNAL_BYTES = 64 * 1024; export const ROOM_CODE_LENGTH = 12; export const MAX_MEDIA_ROUTE_REVISION = Number.MAX_SAFE_INTEGER; export const MAX_SFU_TOKEN_LENGTH = 8 * 1024; +export const MAX_VIEWER_QUALITY_EVIDENCE_BYTES = 2 * 1024; +export const VIEWER_QUALITY_EVIDENCE_INTERVAL_MS = 2_000; +export const VIEWER_QUALITY_EVIDENCE_EXPIRY_MS = 5_000; const opaqueIdSchema = z .string() @@ -181,6 +186,94 @@ export type ParticipantRouteAssignment = z.infer< typeof participantRouteAssignmentSchema >; +const nullableEvidenceNumber = (maximum: number) => + z.number().finite().min(0).max(maximum).nullable(); + +const nullableEvidenceInteger = (maximum: number) => + z.number().int().min(0).max(maximum).nullable(); + +export const viewerQualityEvidenceMetricsSchema = z + .object({ + width: z.number().int().min(1).max(16_384).nullable(), + height: z.number().int().min(1).max(16_384).nullable(), + framesPerSecond: nullableEvidenceNumber(240), + bitrateKbps: nullableEvidenceNumber(100_000), + packetsReceivedDelta: nullableEvidenceInteger(1_000_000), + packetsLostDelta: nullableEvidenceInteger(1_000_000), + jitterMs: nullableEvidenceNumber(60_000), + framesDecodedDelta: nullableEvidenceInteger(10_000), + framesDroppedDelta: nullableEvidenceInteger(10_000), + decodeMsPerFrame: nullableEvidenceNumber(60_000), + freezeCountDelta: nullableEvidenceInteger(10_000), + freezeDurationMsDelta: nullableEvidenceNumber(5_000), + codec: z + .string() + .max(64) + .regex(/^video\/[A-Za-z0-9.+-]{1,32}$/i) + .nullable(), + codecProfile: z + .string() + .max(64) + .regex(/^[a-z0-9-]+=[a-z0-9]+$/) + .nullable(), + codecParameters: z + .string() + .max(128) + .regex(/^[a-z0-9-]+=[a-z0-9]+(?:; [a-z0-9-]+=[a-z0-9]+)*$/) + .nullable(), + }) + .strict() + .refine( + (metrics) => + (metrics.width === null && metrics.height === null) || + (metrics.width !== null && metrics.height !== null), + { message: "Viewer quality dimensions must be present together" }, + ) + .refine(isCanonicalVideoCodecEvidence, { + message: "Viewer codec evidence is not canonical", + path: ["codecParameters"], + }); +export type ViewerQualityEvidenceMetrics = z.infer< + typeof viewerQualityEvidenceMetricsSchema +>; + +const viewerQualityEvidenceGuardSchema = z + .object({ + connectionId: opaqueIdSchema, + routeRevision: mediaRouteRevisionSchema, + }) + .strict(); + +const viewerQualityEvidenceWindowShape = { + sequence: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER), + windowMs: z.number().int().min(1_000).max(5_000), + metrics: viewerQualityEvidenceMetricsSchema, +}; + +function freezeFitsEvidenceWindow(value: { + windowMs: number; + metrics: ViewerQualityEvidenceMetrics; +}): boolean { + return ( + value.metrics.freezeDurationMsDelta === null || + value.metrics.freezeDurationMsDelta <= value.windowMs + ); +} + +export const viewerQualityEvidenceMessageSchema = z + .object({ + type: z.literal("viewer-quality-evidence"), + guard: viewerQualityEvidenceGuardSchema, + ...viewerQualityEvidenceWindowShape, + }) + .strict() + .refine(freezeFitsEvidenceWindow, { + message: "Viewer freeze duration exceeds its evidence window", + }); +export type ViewerQualityEvidenceMessage = z.infer< + typeof viewerQualityEvidenceMessageSchema +>; + const authenticateMessageSchema = z.discriminatedUnion("role", [ z .object({ @@ -253,6 +346,7 @@ export const clientMessageSchema = z.union([ revision: mediaRouteRevisionSchema, }) .strict(), + viewerQualityEvidenceMessageSchema, z .object({ type: z.literal("stop-sharing"), @@ -361,6 +455,18 @@ export const serverMessageSchema = z.union([ qualitySettings: qualitySettingsSchema, }) .strict(), + z + .object({ + type: z.literal("viewer-quality-evidence"), + viewerPeerId: opaqueIdSchema, + parentPeerId: opaqueIdSchema, + guard: viewerQualityEvidenceGuardSchema, + ...viewerQualityEvidenceWindowShape, + }) + .strict() + .refine(freezeFitsEvidenceWindow, { + message: "Viewer freeze duration exceeds its evidence window", + }), z .object({ type: z.literal("ice-config"), diff --git a/src/shared/video-codec-evidence.ts b/src/shared/video-codec-evidence.ts new file mode 100644 index 0000000..81b596e --- /dev/null +++ b/src/shared/video-codec-evidence.ts @@ -0,0 +1,166 @@ +export interface VideoCodecEvidence { + codec: string | null; + profile: string | null; + parameters: string | null; +} + +interface CodecParameterRule { + profileKey: string | null; + parameterKeys: readonly string[]; + validators: Readonly>; +} + +const VIDEO_CODEC_PARAMETER_RULES: Readonly< + Record +> = { + "video/h264": { + profileKey: "profile-level-id", + parameterKeys: ["packetization-mode", "level-asymmetry-allowed"], + validators: { + "profile-level-id": /^[0-9a-f]{6}$/, + "packetization-mode": /^[0-2]$/, + "level-asymmetry-allowed": /^[01]$/, + }, + }, + "video/vp9": { + profileKey: "profile-id", + parameterKeys: ["max-fr", "max-fs"], + validators: { + "profile-id": /^[0-3]$/, + "max-fr": /^[1-9][0-9]{0,9}$/, + "max-fs": /^[1-9][0-9]{0,9}$/, + }, + }, + "video/vp8": { + profileKey: null, + parameterKeys: ["max-fr", "max-fs"], + validators: { + "max-fr": /^[1-9][0-9]{0,9}$/, + "max-fs": /^[1-9][0-9]{0,9}$/, + }, + }, + "video/av1": { + profileKey: "profile", + parameterKeys: ["level-idx", "tier"], + validators: { + profile: /^[0-2]$/, + "level-idx": /^(?:[0-9]|[12][0-9]|3[01])$/, + tier: /^[01]$/, + }, + }, +}; + +const VIDEO_MIME_TYPE = /^video\/[A-Za-z0-9.+-]{1,32}$/i; + +export function deriveVideoCodecEvidence( + codec: string | null, + rawParameters: string | null, +): VideoCodecEvidence { + if (!codec || !VIDEO_MIME_TYPE.test(codec)) { + return { codec: null, profile: null, parameters: null }; + } + const rule = VIDEO_CODEC_PARAMETER_RULES[codec.toLowerCase()]; + if (!rule || !rawParameters || rawParameters.length > 2_048) { + return { codec, profile: null, parameters: null }; + } + + const segments = rawParameters.split(";"); + if (segments.length > 32) { + return { codec, profile: null, parameters: null }; + } + const values = new Map(); + const seenKeys = new Set(); + for (const segment of segments) { + const separator = segment.indexOf("="); + if (separator < 1) { + continue; + } + const key = segment.slice(0, separator).trim().toLowerCase(); + const validator = rule.validators[key]; + if (!validator) { + continue; + } + if (seenKeys.has(key)) { + values.delete(key); + continue; + } + seenKeys.add(key); + const value = segment.slice(separator + 1).trim().toLowerCase(); + if (validator.test(value)) { + values.set(key, value); + } + } + + const profileValue = rule.profileKey + ? values.get(rule.profileKey) + : undefined; + const parameters = rule.parameterKeys + .flatMap((key) => { + const value = values.get(key); + return value ? [`${key}=${value}`] : []; + }) + .join("; "); + return { + codec, + profile: + rule.profileKey && profileValue + ? `${rule.profileKey}=${profileValue}` + : null, + parameters: parameters || null, + }; +} + +export function isCanonicalVideoCodecEvidence(value: { + codec: string | null; + codecProfile: string | null; + codecParameters: string | null; +}): boolean { + const rule = value.codec + ? VIDEO_CODEC_PARAMETER_RULES[value.codec.toLowerCase()] + : undefined; + if (!rule) { + return value.codecProfile === null && value.codecParameters === null; + } + return ( + canonicalValuesMatch( + value.codecProfile, + rule.profileKey ? [rule.profileKey] : [], + rule.validators, + ) && + canonicalValuesMatch( + value.codecParameters, + rule.parameterKeys, + rule.validators, + ) + ); +} + +function canonicalValuesMatch( + encoded: string | null, + orderedKeys: readonly string[], + validators: Readonly>, +): boolean { + if (encoded === null) { + return true; + } + const segments = encoded.split("; "); + if (segments.length > orderedKeys.length) { + return false; + } + let previousKeyIndex = -1; + for (const segment of segments) { + const separator = segment.indexOf("="); + const key = separator < 1 ? "" : segment.slice(0, separator); + const keyIndex = orderedKeys.indexOf(key); + const candidate = separator < 1 ? "" : segment.slice(separator + 1); + if ( + keyIndex <= previousKeyIndex || + keyIndex < 0 || + !validators[key]?.test(candidate) + ) { + return false; + } + previousKeyIndex = keyIndex; + } + return true; +} diff --git a/tests/client-state.test.ts b/tests/client-state.test.ts index f5333cc..d6a0963 100644 --- a/tests/client-state.test.ts +++ b/tests/client-state.test.ts @@ -17,6 +17,7 @@ import { shouldReconnectSignaling, SignalingClient, } from "../src/client/lib/signaling.ts"; +import { qualityEvidenceWindowFromMetrics } from "../src/client/media/viewer-quality-evidence.ts"; import { createStatsAccumulator, collectConnectionMetrics } from "../src/client/webrtc/stats.ts"; afterEach(() => { @@ -574,18 +575,27 @@ describe("WebRTC stats parsing", () => { getStats: async () => report, } as unknown as RTCPeerConnection; - await expect( - collectConnectionMetrics( - connection, - "send", - createStatsAccumulator(), - ), - ).resolves.toMatchObject({ + const metrics = await collectConnectionMetrics( + connection, + "send", + createStatsAccumulator(), + ); + expect(metrics).toMatchObject({ codec: mimeType, codecProfile: profile, codecParameters: parameters, scalabilityMode: null, }); + expect( + qualityEvidenceWindowFromMetrics({ + ...metrics, + sampleWindowMs: 2_000, + })?.metrics, + ).toMatchObject({ + codec: mimeType, + codecProfile: profile, + codecParameters: parameters, + }); }, ); @@ -996,4 +1006,93 @@ describe("WebRTC stats parsing", () => { intervalRetransmittedBytes: 800, }); }); + + it("derives a bounded inbound C window from adjacent RTP counters", async () => { + const inbound = ( + timestamp: number, + packetsReceived: number, + packetsLost: number, + framesDecoded: number, + totalDecodeTime: number, + ) => + new Map([ + [ + "transport", + { id: "transport", type: "transport", timestamp }, + ], + [ + "codec", + { + id: "codec", + type: "codec", + timestamp, + transportId: "transport", + mimeType: "video/H264", + sdpFmtpLine: + "profile-level-id=42e01f; packetization-mode=1; " + + "sprop-parameter-sets=must-not-leave-stats", + }, + ], + [ + "inbound", + { + id: "inbound", + type: "inbound-rtp", + timestamp, + kind: "video", + ssrc: 101, + transportId: "transport", + codecId: "codec", + bytesReceived: timestamp * 10, + packetsReceived, + packetsLost, + framesDecoded, + framesDropped: 3, + freezeCount: 1, + totalFreezesDuration: 0.25, + totalDecodeTime, + frameWidth: 1_920, + frameHeight: 1_080, + scalabilityMode: "L3T3_KEY", + }, + ], + ]) as unknown as RTCStatsReport; + const reports = [ + inbound(1_000, 1_000, 10, 60, 0.12), + inbound(3_000, 2_500, 12, 180, 0.42), + inbound(5_000, 4_000, 11, 300, 0.72), + ]; + const connection = { + getStats: async () => reports.shift()!, + } as unknown as RTCPeerConnection; + const accumulator = createStatsAccumulator(); + + await collectConnectionMetrics(connection, "receive", accumulator); + const stable = await collectConnectionMetrics( + connection, + "receive", + accumulator, + ); + const correctedLoss = await collectConnectionMetrics( + connection, + "receive", + accumulator, + ); + + expect(stable).toMatchObject({ + sampleWindowMs: 2_000, + frameWidth: 1_920, + frameHeight: 1_080, + intervalPacketsReceived: 1_500, + intervalPacketsLost: 2, + intervalFramesDecoded: 120, + intervalDecodeMs: 2.5, + codec: "video/H264", + codecProfile: "profile-level-id=42e01f", + codecParameters: "packetization-mode=1", + scalabilityMode: null, + }); + expect(JSON.stringify(stable)).not.toContain("sprop-parameter-sets"); + expect(correctedLoss.intervalPacketsLost).toBeNull(); + }); }); diff --git a/tests/protocol.test.ts b/tests/protocol.test.ts index 15845e7..e262ba2 100644 --- a/tests/protocol.test.ts +++ b/tests/protocol.test.ts @@ -3,6 +3,7 @@ import { describe, expect, it } from "vitest"; import { MAX_MEDIA_ROUTE_REVISION, MAX_SFU_TOKEN_LENGTH, + MAX_VIEWER_QUALITY_EVIDENCE_BYTES, MAX_VIEWERS_PER_ROOM_LIMIT, clientMessageSchema, decodeClientMessage, @@ -19,6 +20,34 @@ const qualitySettings = { degradationPreference: "maintain-resolution", } as const; +const qualityEvidence = { + type: "viewer-quality-evidence", + guard: { + connectionId: "connection_12345678", + routeRevision: 0, + }, + sequence: 0, + windowMs: 2_000, + metrics: { + width: 1_920, + height: 1_080, + framesPerSecond: 59.8, + bitrateKbps: 7_500, + packetsReceivedDelta: 1_500, + packetsLostDelta: 2, + jitterMs: 3.5, + framesDecodedDelta: 120, + framesDroppedDelta: 1, + decodeMsPerFrame: 2.4, + freezeCountDelta: 0, + freezeDurationMsDelta: 0, + codec: "video/H264", + codecProfile: "profile-level-id=42e01f", + codecParameters: + "packetization-mode=1; level-asymmetry-allowed=1", + }, +} as const; + describe("client signaling protocol", () => { it("accepts a bounded authentication message", () => { expect( @@ -220,6 +249,87 @@ describe("client signaling protocol", () => { ).toBe(false); }); + it("accepts only strict, bounded viewer quality evidence", () => { + expect(clientMessageSchema.safeParse(qualityEvidence).success).toBe(true); + expect( + Buffer.byteLength(JSON.stringify(qualityEvidence), "utf8"), + ).toBeLessThanOrEqual(MAX_VIEWER_QUALITY_EVIDENCE_BYTES); + + for (const invalid of [ + { ...qualityEvidence, roomId }, + { + ...qualityEvidence, + guard: { ...qualityEvidence.guard, peerId: "viewer_12345678" }, + }, + { ...qualityEvidence, windowMs: 999 }, + { ...qualityEvidence, sequence: -1 }, + { + ...qualityEvidence, + metrics: { ...qualityEvidence.metrics, width: null }, + }, + { + ...qualityEvidence, + metrics: { ...qualityEvidence.metrics, bitrateKbps: Number.POSITIVE_INFINITY }, + }, + { + ...qualityEvidence, + metrics: { ...qualityEvidence.metrics, freezeDurationMsDelta: 2_001 }, + }, + { + ...qualityEvidence, + metrics: { + ...qualityEvidence.metrics, + codecParameters: "sprop-parameter-sets=deadbeef", + }, + }, + { + ...qualityEvidence, + metrics: { + ...qualityEvidence.metrics, + codecProfile: "device-id=deadbeef", + }, + }, + { + ...qualityEvidence, + metrics: { + ...qualityEvidence.metrics, + scalabilityMode: "DEVICE_ABC123", + }, + }, + { + ...qualityEvidence, + metrics: { + ...qualityEvidence.metrics, + decoderImplementation: "device-specific-decoder", + }, + }, + ]) { + expect(clientMessageSchema.safeParse(invalid).success).toBe(false); + } + }); + + it.each([ + ["video/VP8", null, "max-fr=60; max-fs=8160"], + ["video/VP9", "profile-id=2", "max-fs=8160"], + ["video/AV1", "profile=1", "level-idx=31; tier=0"], + ["video/unknown", null, null], + ])( + "accepts canonical viewer codec evidence for %s", + (codec, codecProfile, codecParameters) => { + expect( + clientMessageSchema.safeParse({ + ...qualityEvidence, + metrics: { + ...qualityEvidence.metrics, + codec, + codecProfile, + codecParameters, + }, + }).success, + ).toBe(true); + }, + ); + it("accepts bounded route acknowledgements, failures, and SFU refreshes", () => { expect( clientMessageSchema.safeParse({ @@ -500,6 +610,27 @@ describe("server signaling protocol", () => { } }); + it("accepts only canonical server-derived viewer evidence envelopes", () => { + const forwarded = { + ...qualityEvidence, + viewerPeerId: "viewer_12345678", + parentPeerId: "host_12345678", + }; + expect(serverMessageSchema.safeParse(forwarded).success).toBe(true); + expect( + Buffer.byteLength(JSON.stringify(forwarded), "utf8"), + ).toBeLessThanOrEqual(MAX_VIEWER_QUALITY_EVIDENCE_BYTES); + expect( + serverMessageSchema.safeParse({ ...forwarded, roomId }).success, + ).toBe(false); + expect( + serverMessageSchema.safeParse({ + ...qualityEvidence, + viewerPeerId: "viewer_12345678", + }).success, + ).toBe(false); + }); + it.each([0, 1.5, MAX_VIEWERS_PER_ROOM_LIMIT + 1])( "rejects viewer limit %s outside the protocol boundary", (maxViewers) => { diff --git a/tests/server-signal.test.ts b/tests/server-signal.test.ts index 64d8313..24c7743 100644 --- a/tests/server-signal.test.ts +++ b/tests/server-signal.test.ts @@ -6,6 +6,7 @@ import WebSocket from "ws"; import { MAX_SIGNAL_BYTES, + MAX_VIEWER_QUALITY_EVIDENCE_BYTES, decodeServerMessage, type Role, type ServerMessage, @@ -161,6 +162,7 @@ async function startHarness( persistent?: boolean; peerAssistedMedia?: boolean; peerAssistedPrimaryRoomOnly?: boolean; + now?: () => number; } = {}, ): Promise { const config = testConfig(); @@ -182,6 +184,7 @@ async function startHarness( config, roomStore, serveFrontend: false, + now: overrides.now, authenticationTimeoutMs: overrides.authenticationTimeoutMs ?? 500, viewerDisconnectGraceMs: overrides.viewerDisconnectGraceMs ?? 50, heartbeatIntervalMs: 60_000, @@ -199,6 +202,38 @@ async function startHarness( }; } +function viewerQualityEvidence( + connectionId: string, + routeRevision: number, + sequence: number, + bitrateKbps = 7_500, +) { + return { + type: "viewer-quality-evidence", + guard: { connectionId, routeRevision }, + sequence, + windowMs: 2_000, + metrics: { + width: 1_920, + height: 1_080, + framesPerSecond: 60, + bitrateKbps, + packetsReceivedDelta: 1_500, + packetsLostDelta: 2, + jitterMs: 3.5, + framesDecodedDelta: 120, + framesDroppedDelta: 1, + decodeMsPerFrame: 2.4, + freezeCountDelta: 0, + freezeDurationMsDelta: 0, + codec: "video/H264", + codecProfile: "profile-level-id=42e01f", + codecParameters: + "packetization-mode=1; level-asymmetry-allowed=1", + }, + } as const; +} + async function startSfuHarness(options: { tokenIssuer: SfuTokenIssuer; prepareTimeoutMs?: number; @@ -1119,6 +1154,314 @@ describe("WebSocket signaling", () => { expect(harness.roomStore.size).toBe(0); }); + it("forwards only current, bounded ordinary-P2P viewer C evidence", async () => { + let now = Date.now(); + const harness = await startHarness({ now: () => now }); + const host = await openClient(harness.webSocketUrl); + const hostAuth = await authenticate( + host, + harness.room, + "host", + "quality-host-client", + ); + const viewer = await openClient(harness.webSocketUrl); + const viewerAuth = await authenticate( + viewer, + harness.room, + "viewer", + "quality-viewer-client", + ); + await host.inbox.next("peer-joined"); + + host.socket.send( + JSON.stringify({ + type: "signal", + targetPeerId: viewerAuth.peerId, + payload: { + kind: "description", + connectionId: "quality_connection_first", + description: { type: "offer", sdp: "v=0\r\n" }, + }, + }), + ); + await viewer.inbox.next("signal"); + + viewer.socket.send( + JSON.stringify(viewerQualityEvidence("quality_connection_first", 0, 0)), + ); + expect(await host.inbox.next("viewer-quality-evidence")).toEqual({ + ...viewerQualityEvidence("quality_connection_first", 0, 0), + viewerPeerId: viewerAuth.peerId, + parentPeerId: hostAuth.peerId, + }); + + viewer.socket.send( + JSON.stringify(viewerQualityEvidence("quality_connection_first", 0, 1)), + ); + viewer.socket.send( + JSON.stringify( + viewerQualityEvidence("quality_connection_first", 0, 1, 7_000), + ), + ); + await host.inbox.expectNone(30); + + now += 2_000; + viewer.socket.send( + JSON.stringify( + viewerQualityEvidence("quality_connection_first", 0, 1, 7_000), + ), + ); + expect( + (await host.inbox.next("viewer-quality-evidence")).metrics.bitrateKbps, + ).toBe(7_000); + + now += 2_000; + viewer.socket.send( + JSON.stringify( + viewerQualityEvidence("quality_connection_first", 0, 2, 7_000), + ), + ); + expect( + (await host.inbox.next("viewer-quality-evidence")).metrics.bitrateKbps, + ).toBe(7_000); + + viewer.socket.send( + JSON.stringify( + viewerQualityEvidence("quality_connection_stale", 0, 3, 6_500), + ), + ); + viewer.socket.send( + JSON.stringify( + viewerQualityEvidence("quality_connection_first", 1, 3, 6_500), + ), + ); + await host.inbox.expectNone(30); + + host.socket.send( + JSON.stringify({ + type: "signal", + targetPeerId: viewerAuth.peerId, + payload: { + kind: "description", + connectionId: "quality_connection_second", + description: { type: "offer", sdp: "v=0\r\n" }, + }, + }), + ); + await viewer.inbox.next("signal"); + viewer.socket.send( + JSON.stringify( + viewerQualityEvidence("quality_connection_first", 0, 3, 6_000), + ), + ); + await host.inbox.expectNone(30); + viewer.socket.send( + JSON.stringify( + viewerQualityEvidence("quality_connection_second", 0, 0, 6_000), + ), + ); + expect( + (await host.inbox.next("viewer-quality-evidence")).guard.connectionId, + ).toBe("quality_connection_second"); + + host.socket.send( + JSON.stringify( + viewerQualityEvidence("quality_connection_second", 0, 1, 5_500), + ), + ); + await host.inbox.expectNone(30); + + const replacementViewer = await openClient(harness.webSocketUrl); + const replacementAuth = await authenticate( + replacementViewer, + harness.room, + "viewer", + "quality-viewer-client", + ); + expect(replacementAuth).toMatchObject({ + peerId: viewerAuth.peerId, + connectionId: "quality_connection_second", + }); + expect(await host.inbox.next("peer-joined")).toMatchObject({ + peerId: viewerAuth.peerId, + }); + replacementViewer.socket.send( + JSON.stringify( + viewerQualityEvidence("quality_connection_second", 0, 0, 5_000), + ), + ); + expect(await host.inbox.next("viewer-quality-evidence")).toMatchObject({ + viewerPeerId: viewerAuth.peerId, + guard: { connectionId: "quality_connection_second" }, + sequence: 0, + metrics: { bitrateKbps: 5_000 }, + }); + }); + + it("derives peer-assisted C routing from the active assignment", async () => { + const harness = await startHarness({ peerAssistedMedia: true }); + const host = await openClient(harness.webSocketUrl); + const hostAuth = peerAssisted( + await authenticate(host, harness.room, "host", "peer-quality-host"), + ); + host.inbox.ignore("route-update"); + host.inbox.ignore("media-assignment"); + const viewer = await openClient(harness.webSocketUrl); + const viewerAuth = peerAssisted( + await authenticate( + viewer, + harness.room, + "viewer", + "peer-quality-viewer", + ), + ); + expect(viewerAuth.routeAssignment.upstream).toEqual({ + kind: "peer", + peerId: hostAuth.peerId, + }); + + host.socket.send( + JSON.stringify({ + type: "signal", + targetPeerId: viewerAuth.peerId, + payload: { + kind: "description", + connectionId: "peer_quality_connection", + description: { type: "offer", sdp: "v=0\r\n" }, + }, + }), + ); + await viewer.inbox.next("signal"); + viewer.socket.send( + JSON.stringify( + viewerQualityEvidence( + "peer_quality_connection", + viewerAuth.routeRevision, + 0, + ), + ), + ); + expect(await host.inbox.next("viewer-quality-evidence")).toMatchObject({ + viewerPeerId: viewerAuth.peerId, + parentPeerId: hostAuth.peerId, + guard: { + connectionId: "peer_quality_connection", + routeRevision: viewerAuth.routeRevision, + }, + }); + + viewer.socket.send( + JSON.stringify( + viewerQualityEvidence( + "peer_quality_connection", + viewerAuth.routeRevision + 1, + 1, + 7_000, + ), + ), + ); + await host.inbox.expectNone(30); + + viewer.inbox.ignore("route-update"); + viewer.inbox.ignore("media-assignment"); + const secondViewer = await openClient(harness.webSocketUrl); + await authenticate( + secondViewer, + harness.room, + "viewer", + "peer-quality-viewer-two", + ); + const relayChild = await openClient(harness.webSocketUrl); + const relayChildAuth = peerAssisted( + await authenticate( + relayChild, + harness.room, + "viewer", + "peer-quality-relay-child", + ), + ); + expect(relayChildAuth.routeAssignment.upstream).toEqual({ + kind: "peer", + peerId: viewerAuth.peerId, + }); + + viewer.socket.send( + JSON.stringify( + viewerQualityEvidence( + "peer_quality_connection", + relayChildAuth.routeRevision, + 1, + 6_500, + ), + ), + ); + expect(await host.inbox.next("viewer-quality-evidence")).toMatchObject({ + viewerPeerId: viewerAuth.peerId, + guard: { + connectionId: "peer_quality_connection", + routeRevision: relayChildAuth.routeRevision, + }, + sequence: 1, + }); + + viewer.socket.send( + JSON.stringify({ + type: "signal", + targetPeerId: relayChildAuth.peerId, + payload: { + kind: "description", + connectionId: "relay_child_quality_connection", + description: { type: "offer", sdp: "v=0\r\n" }, + }, + }), + ); + await relayChild.inbox.next("signal"); + relayChild.socket.send( + JSON.stringify( + viewerQualityEvidence( + "relay_child_quality_connection", + relayChildAuth.routeRevision, + 0, + ), + ), + ); + expect(await viewer.inbox.next("viewer-quality-evidence")).toMatchObject({ + viewerPeerId: relayChildAuth.peerId, + parentPeerId: viewerAuth.peerId, + guard: { + connectionId: "relay_child_quality_connection", + routeRevision: relayChildAuth.routeRevision, + }, + }); + await host.inbox.expectNone(30); + }); + + it("rejects viewer C evidence above the byte cap", async () => { + const harness = await startHarness(); + const viewer = await openClient(harness.webSocketUrl); + await authenticate( + viewer, + harness.room, + "viewer", + "oversized-quality-viewer", + ); + const closed = new Promise((resolve) => + viewer.socket.once("close", (code) => resolve(code)), + ); + const valid = JSON.stringify( + viewerQualityEvidence("oversized_quality_connection", 0, 0), + ); + + viewer.socket.send( + `${valid}${" ".repeat(MAX_VIEWER_QUALITY_EVIDENCE_BYTES + 1)}`, + ); + + expect(await viewer.inbox.next("error")).toMatchObject({ + code: "INVALID_MESSAGE", + }); + expect(await closed).toBe(1008); + }); + it("routes offer and answer only between the host and the targeted viewer", async () => { const harness = await startHarness(); const secondRoom = harness.roomStore.createRoom(); diff --git a/tests/viewer-quality-evidence.test.ts b/tests/viewer-quality-evidence.test.ts new file mode 100644 index 0000000..ad42da5 --- /dev/null +++ b/tests/viewer-quality-evidence.test.ts @@ -0,0 +1,247 @@ +import { describe, expect, it } from "vitest"; + +import type { ClientMessage, ServerMessage } from "../src/shared/protocol.ts"; +import { + metricsFromQualityEvidence, + qualityEvidenceMatchesSnapshot, + qualityEvidenceWindowFromMetrics, + ViewerQualityEvidenceReporter, +} from "../src/client/media/viewer-quality-evidence.ts"; +import { + EMPTY_METRICS, + type ConnectionMetrics, + type PeerSnapshot, +} from "../src/client/types.ts"; + +function receiveMetrics( + overrides: Partial = {}, +): ConnectionMetrics { + return { + ...EMPTY_METRICS, + sampleTimestampMs: 10_000, + sampleWindowMs: 2_000, + framesPerSecond: 60, + frameWidth: 1_920, + frameHeight: 1_080, + resolution: "1920x1080", + bitrateKbps: 7_500, + intervalPacketsReceived: 1_500, + intervalPacketsLost: 2, + jitterMs: 3.5, + intervalFramesDecoded: 120, + intervalFramesDropped: 1, + intervalDecodeMs: 2.4, + intervalFreezeCount: 0, + intervalFreezeDurationMs: 0, + codec: "video/H264", + codecProfile: "profile-level-id=42e01f", + codecParameters: + "packetization-mode=1; level-asymmetry-allowed=1", + scalabilityMode: "L3T3_KEY", + ...overrides, + }; +} + +function snapshot( + connectionId: string, + metrics = receiveMetrics(), +): PeerSnapshot { + return { + peerId: "host_12345678", + connectionId, + connectionState: "connected", + iceConnectionState: "connected", + metrics, + error: null, + }; +} + +describe("viewer quality evidence", () => { + it("normalizes only bounded adjacent receive evidence", () => { + expect(qualityEvidenceWindowFromMetrics(receiveMetrics())).toEqual({ + windowMs: 2_000, + metrics: { + width: 1_920, + height: 1_080, + framesPerSecond: 60, + bitrateKbps: 7_500, + packetsReceivedDelta: 1_500, + packetsLostDelta: 2, + jitterMs: 3.5, + framesDecodedDelta: 120, + framesDroppedDelta: 1, + decodeMsPerFrame: 2.4, + freezeCountDelta: 0, + freezeDurationMsDelta: 0, + codec: "video/H264", + codecProfile: "profile-level-id=42e01f", + codecParameters: + "packetization-mode=1; level-asymmetry-allowed=1", + }, + }); + + expect( + qualityEvidenceWindowFromMetrics( + receiveMetrics({ + sampleWindowMs: 9_000, + rtpStatsId: "must-not-leave-the-client", + }), + ), + ).toBeNull(); + const bounded = qualityEvidenceWindowFromMetrics( + receiveMetrics({ + frameWidth: 0, + frameHeight: 0, + bitrateKbps: 200_000, + intervalPacketsLost: -1, + codecParameters: "raw fmtp; secret=value/with/slash", + }), + ); + expect(bounded?.metrics).toMatchObject({ + width: null, + height: null, + bitrateKbps: null, + packetsLostDelta: null, + codecParameters: null, + }); + expect(JSON.stringify(bounded)).not.toContain("rtpStatsId"); + expect(JSON.stringify(bounded)).not.toContain("must-not-leave-the-client"); + expect(JSON.stringify(bounded)).not.toContain("scalabilityMode"); + expect(JSON.stringify(bounded)).not.toContain("L3T3_KEY"); + }); + + it("rate-limits, deduplicates, and resets sequence per connection", () => { + let now = 10_000; + const sent: ClientMessage[] = []; + const reporter = new ViewerQualityEvidenceReporter( + (message) => { + sent.push(message); + return true; + }, + () => now, + ); + const first = snapshot("connection_first_12345678"); + + expect(reporter.offer(first, 0)).toBe(true); + expect(reporter.offer(first, 0)).toBe(false); + now += 1_000; + expect( + reporter.offer( + snapshot( + first.connectionId, + receiveMetrics({ sampleTimestampMs: 11_000, bitrateKbps: 7_000 }), + ), + 0, + ), + ).toBe(false); + now += 1_000; + expect( + reporter.offer( + snapshot( + first.connectionId, + receiveMetrics({ sampleTimestampMs: 12_000, bitrateKbps: 6_500 }), + ), + 0, + ), + ).toBe(true); + expect( + reporter.offer( + snapshot( + first.connectionId, + receiveMetrics({ sampleTimestampMs: 12_000, bitrateKbps: 6_000 }), + ), + 2, + ), + ).toBe(true); + expect( + reporter.offer(snapshot("connection_second_12345678"), 2), + ).toBe(true); + now += 2_000; + expect( + reporter.offer( + snapshot( + "connection_second_12345678", + receiveMetrics({ sampleTimestampMs: 14_000 }), + ), + 2, + ), + ).toBe(true); + + expect( + sent.map((message) => + message.type === "viewer-quality-evidence" + ? { + connectionId: message.guard.connectionId, + revision: message.guard.routeRevision, + sequence: message.sequence, + } + : null, + ), + ).toEqual([ + { + connectionId: "connection_first_12345678", + revision: 0, + sequence: 0, + }, + { + connectionId: "connection_first_12345678", + revision: 0, + sequence: 1, + }, + { + connectionId: "connection_first_12345678", + revision: 2, + sequence: 2, + }, + { + connectionId: "connection_second_12345678", + revision: 2, + sequence: 0, + }, + { + connectionId: "connection_second_12345678", + revision: 2, + sequence: 1, + }, + ]); + }); + + it("pairs and renders only the current parent connection", () => { + const evidence = { + type: "viewer-quality-evidence", + viewerPeerId: "viewer_12345678", + parentPeerId: "host_12345678", + guard: { + connectionId: "connection_12345678", + routeRevision: 0, + }, + sequence: 3, + ...qualityEvidenceWindowFromMetrics(receiveMetrics())!, + } as Extract; + const current = { + ...snapshot(evidence.guard.connectionId), + peerId: evidence.viewerPeerId, + }; + + expect(qualityEvidenceMatchesSnapshot(evidence, current)).toBe(true); + expect( + qualityEvidenceMatchesSnapshot(evidence, { + ...current, + connectionId: "connection_replaced_12345678", + }), + ).toBe(false); + expect(metricsFromQualityEvidence(evidence)).toMatchObject({ + sampleWindowMs: 2_000, + frameWidth: 1_920, + frameHeight: 1_080, + resolution: "1920x1080", + bitrateKbps: 7_500, + packetsLost: 2, + framesDropped: 1, + intervalFramesDecoded: 120, + intervalFramesDropped: 1, + intervalFreezeCount: 0, + codec: "video/H264", + }); + }); +});