diff --git a/docs/README.md b/docs/README.md index 546af8a..7e4dbb4 100644 --- a/docs/README.md +++ b/docs/README.md @@ -18,6 +18,6 @@ - [ADR-0002](./adr/0002-persistent-protected-rooms.md): optional SQLite persistence, protected sequential room IDs, and room lifecycle. - [ADR-0004](./adr/0004-peer-assisted-media-experiment.md): proposed, experiment-only standard-WebRTC peer-assisted spike with hard fanout and failure gates. - [ADR-0005](./adr/0005-automatic-hybrid-media-routing.md): proposed, default-off automatic P2P/peer/SFU preparation, token-free standby prewarm, commit, failback, and rollback with a hard host-edge budget. -- [ADR-0007](./adr/0007-demand-driven-dual-representation-quality.md): accepted demand-driven `HIGH + optional LOW` quality policy, path-level hysteresis, evidence contract, and SVC boundary. +- [ADR-0007](./adr/0007-demand-driven-dual-representation-quality.md): accepted demand-driven `HIGH + at most one on-demand LOW` quality policy, path-level hysteresis, evidence contract, and SVC boundary. Documentation and project memory are part of the product source of truth. Update them in the same change that alters the corresponding requirement or architecture. diff --git a/docs/adr/0007-demand-driven-dual-representation-quality.md b/docs/adr/0007-demand-driven-dual-representation-quality.md index c3a50ed..b147f7a 100644 --- a/docs/adr/0007-demand-driven-dual-representation-quality.md +++ b/docs/adr/0007-demand-driven-dual-representation-quality.md @@ -1,6 +1,6 @@ # ADR-0007: Demand-Driven Dual-Representation Quality -- Status: Accepted - Implementation Pending +- Status: Accepted - Staged Implementation - Date: 2026-08-19 ## Context @@ -27,20 +27,23 @@ Use demand-driven dual representations: 2. Every viewing path starts in `HIGH`. It moves to `FALLBACK` only after multiple consecutive sampling windows show that the path cannot sustain `HIGH`, using correlated sender and viewer evidence. -3. The first verified fallback path starts one shared `LOW` representation. - Every verified weak path uses that same representation. Healthy paths remain - on `HIGH`. +3. The first verified fallback path must start exactly one shared `LOW` + representation. Every verified weak path moves to that same representation. + Healthy paths remain on `HIGH`. 4. A fallback path returns to `HIGH` only after a longer, independently defined stable-recovery window. When no path needs `LOW`, stop its encoder and free its resources. -5. The representation count is a hard `HIGH + optional LOW <= 2`. It never - grows with viewer count. +5. The representation count is a hard `HIGH + at most one on-demand LOW <= 2`; + it never grows with viewer count. Starting `LOW` is conditional on a qualified hardware/power-efficient encoder path and measured spare game-performance budget. If that path is absent or the -second encoder causes unacceptable CPU/GPU/game load, fail closed: preserve -`HIGH` for healthy paths and show the weak path an explicit degraded/unavailable -state. Never protect a weak path by reducing healthy paths. +second encoder causes unacceptable CPU/GPU/game load, fail closed for that +attempt: preserve `HIGH` and show the weak path an explicit degraded/unavailable +state. This is exceptional damage containment, not permission to ignore a weak +path indefinitely. A supported sender cohort that cannot reliably start `LOW` +on demand fails automatic-quality-control acceptance; optimize its encode path +or mark that cohort unsupported. Never protect a weak path by reducing `HIGH`. The two path states use explicit predicates and asymmetric entry/exit windows. There is no weighted score, device ranking, machine-learning controller, or @@ -85,7 +88,9 @@ that probe is trustworthy may the product add a minimal authenticated C report for the few receive/decode signals required by the two-state predicate. The B/C correlation uses normalized fields derived from negotiated parameters/stats and actual decode behavior; it must never upload raw SDP, raw stats, candidate -addresses, or other identifiers. Do not build a general telemetry schema. +addresses, or raw device/network identifiers. Opaque server-issued path and +connection-generation IDs remain required for authorization and correlation. +Do not build a general telemetry schema. A viewer may request `LOW`, but the request is advisory. It must be carried on an authenticated, current room/path session and be rate-limited and deduplicated. @@ -154,6 +159,8 @@ Negative: - Representation count never exceeds two and host media edges never exceed two. - An unavailable/over-budget `LOW` fails visibly for weak paths while healthy paths and `HIGH` remain unchanged. +- Every supported sender cohort can reliably start the one shared `LOW` on + demand; otherwise automatic quality control does not pass acceptance. - Native relays show packet forwarding without an added decode/encode stage. - Spoofed, stale, duplicated, or rate-excessive viewer requests have no effect. - Controlled capture/CPU/bandwidth/TURN/receiver/display cases produce the diff --git a/docs/project-memory.md b/docs/project-memory.md index 8ebe002..7e6ab31 100644 --- a/docs/project-memory.md +++ b/docs/project-memory.md @@ -4,15 +4,14 @@ Last updated: 2026-08-19 ## Confirmed Intent -- Build Discord/KOOK/Oopz/TeamSpeak-like low-latency game screen sharing for one broadcaster and a small trusted friend group. Public or large broadcasts belong on OBS/Twitch-class services. -- Viewers should join from a normal desktop or mobile browser. A packaged/native sender is a planned later stage for shared encoding and may also improve capture and audio without changing the Web viewer requirement. +- Build low-latency game screen sharing for one broadcaster and a small trusted group; public or large broadcasts belong on OBS/Twitch-class services. +- Viewers join from normal desktop/mobile browsers. A later native sender may share encoding and improve capture/audio without changing the Web viewer requirement. - Use a small central service for access, rooms, signaling, deterministic topology, STUN/TURN, and observability while clients carry media whenever practical. -- Server bandwidth cost is a primary constraint. The ordered media ladder is direct P2P, then measured peer assistance, then an enabled user-operated or central SFU fallback. Route selection and failure recovery must be automatic and viewer-transparent, while remaining deterministic, budget-driven, and small enough to reason about. This has always been the product intent; earlier text prohibiting silent migration was an incorrect interpretation and is superseded. -- Host and relay-capable endpoints each have a target budget of at most two downstream media edges. Every P2P edge independently uses direct ICE or authenticated TURN fallback. With two edges, the long-term native/encoded-object path should reuse one compatible encoded output; the current browser experiment is stricter at one viewer child and re-encodes. -- Packet or layer striping across two peer trees is a recorded candidate for reducing endpoint upload toward one full-stream bitrate. It requires a separate bounded experiment for multi-parent assembly, loss recovery, synchronization, churn, and sub-second latency; it is not part of the current full-stream two-chain implementation. -- Shared encoding reduces compute, not each viewer's last-hop copy. Quality uses shared `HIGH` plus at most one on-demand `LOW`; without a qualified efficient second encoder, weak paths fail visibly while `HIGH` remains protected. SVC has no software fallback. -- Durable decisions, current snapshots, research, code, `AGENTS.md`, and `.codex/` belong in Git. Memory and status are rewritten in place rather than kept as transcripts. -- Research current official sources and established implementations before material work. Apply Occam's razor and reject speculative protocols, services, scoring, and scale. +- Minimize server bandwidth. Automatic, viewer-transparent routing follows direct P2P, measured peer assistance, then enabled user-operated/central SFU fallback under deterministic budgets; earlier anti-migration text is superseded. +- Host and relay targets are at most two downstream edges; each independently uses direct ICE or authenticated TURN. A long-term native/object path should reuse one encode, while the browser experiment has one viewer child and re-encodes. +- Two-tree packet/layer striping may reduce endpoint upload toward one stream bitrate, but needs a bounded multi-parent, loss, sync, churn, and latency experiment; it is not in the current full-stream chains. +- Shared encoding reduces compute, not each viewer's last-hop copy. Automatic quality uses shared `HIGH` plus exactly one on-demand `LOW` whenever verified weak paths exist; all weak paths share it and recovery stops it. An unavailable efficient second encoder fails closed for that attempt while protecting `HIGH`, but a supported sender cohort that cannot start `LOW` reliably fails controller acceptance and must be optimized or marked unsupported. SVC has no software fallback. +- Keep decisions, snapshots, research, code, `AGENTS.md`, and `.codex/` in Git; rewrite memory/status in place. Research current primary sources before material work and reject speculative machinery. ## Current Recommendation @@ -20,34 +19,35 @@ Last updated: 2026-08-19 - Use direct host P2P for one or two viewers. Keep ADR-0004 off for broad production traffic: two mobile leaves can consume both host roots. Use the exact-room allowlist only on an isolated canary until its resource, quality, recovery, TURN, and browser/mobile gates pass. - Browser relays resend remote `MediaStreamTrack` values and re-encode at each hop; WebRTC does not guarantee a shared encoder across peer connections, so measure the cost. - Keep browser experiments bounded; spike native RTP relay, encoded-object striping, SVC, and FEC separately. -- Plan a separate native-sender ADR for TeamSpeak-style shared encoding regardless of the browser relay result. One compatible encoded output may feed at most two standard WebRTC packetizers to reduce host encode work, but each edge still consumes upload bandwidth. This work is not part of the current Draft and cannot rescue another failed browser-relay gate. -- Keep user-operated or central single-node SFU capacity optional. When configured, `main`'s #17 route controller may select it only as the final fallback; its fanout egress belongs to its operator and does not justify Redis or multi-node infrastructure. Closed PR #12's explicit whole-room SFU mode is superseded, not a current option. -- Keep automatic routing default-off behind ADR-0005: discrete failures drive peer recovery and optional SFU fallback, host media uses break-before-make under two active edges, and a binary relay capability keeps mobile/iPad clients as leaves. Merged PR #20's token-free prewarm passes the local sub-second gate; public transport and load gates remain. +- Plan a separate native-sender ADR regardless of browser relay results. One encode may feed at most two WebRTC packetizers, reducing encode work but not per-edge upload; it cannot rescue a failed browser gate. +- Optional single-node SFU capacity is final fallback only; operator egress does not justify Redis/multi-node infrastructure. PR #12's whole-room SFU mode is superseded. +- Keep ADR-0005 routing default-off: discrete failures drive peer/SFU recovery, full host fanout uses break-before-make, and mobile/iPad clients stay leaves. PR #20 prewarm passes locally; public transport/load gates remain. - Local reparenting is a later candidate: start with unassigned relay admission rescue; do not block current work. -- Prefer direct UDP, then TURN/UDP, with TURN/TCP as the required non-UDP fallback. Optional TURN/TLS uses TCP 5349 by default; TCP 443 needs a dedicated address or validated L4/SNI routing. -- Treat video settings as ceilings. Production defaults clarity-first with bounded manual controls, but reported degradation is unclassified. Before ADR-0007 implementation, correlate same-tick capture settings and outbound deltas/identity, then add only the minimal authenticated viewer report needed for per-path `HIGH`/`FALLBACK`. Do not use UA for quality or build a composite score. Keep browser audio request/presence-only. +- Prefer direct UDP, then TURN/UDP, with TURN/TCP required; optional TURN/TLS defaults to 5349, while 443 needs a dedicated address or validated L4/SNI routing. +- Treat video settings as ceilings and degradation as unclassified. The first local A+B alignment foundation covers same-tick capture/outbound identity, windows, deltas, remote linkage, and path; next derive codec profile/parameters/`scalabilityMode`, then add minimal authenticated C, a two-state controller, and shared `LOW`. Never use UA or a composite score. +- Reported poor movie/video audio is unclassified. After video A+B, separately diagnose audio A/B/C and A/V sync across capture settings, codec/fmtp, actual bitrate, loss, jitter, concealment, and jitter buffer. `maxBitrate` is not quality-up; do not expand runtime before evidence. See `docs/research/browser-screen-audio-quality.md`. - Do not add custom scene detection or dynamic-FPS control until WebRTC statistics and host resource measurements prove a material gap. Keep browser codec order until target hardware proves a more efficient common codec. -- Keep room policy deployment-driven. Public/password-only deployments use random temporary rooms. A site password plus SQLite path enables sequential persistent rooms; stopping leaves the room and viewer link available. +- Keep room policy deployment-driven: public/password-only rooms are random and temporary; password plus SQLite enables sequential persistent rooms and reusable stopped links. - Deferred architecture audit: `docs/maintenance.md`. ## Current Implementation - The repository is one npm package using Node.js 24, React, TypeScript, Vite, native WebRTC, `ws`, Zod, Vitest, and separate coturn. -- Production commit `769de201f7cc` runs at `https://share.bonfire.icu`. It adds equal share/join entry actions, default-closed details, clarity-first bounded quality settings, live source/quality changes, video pause, and visible sender warnings. +- Production `769de201f7cc` at `https://share.bonfire.icu` has equal entry actions, default-closed details, bounded clarity-first quality, live source/quality changes, pause, and sender warnings. - Production still uses one host `RTCPeerConnection` per viewer because peer assistance and LiveKit are unconfigured; above two viewers it still exceeds the target host-edge budget. Its real game-capture quality/pause cycle remains unverified. -- Access stays small: optional `ACCESS_PASSWORD`, a 12-hour stateless HMAC HttpOnly `SameSite=Strict` cookie, internal host token, role-bound signaling, Origin/payload checks, and no accounts, JWTs, or session map. -- With `ROOM_DATABASE_PATH` and the site password, SQLite stores only room ID and host-token digest. Links persist and stopping leaves viewers waiting. Room `1` survived the `769de201f7cc` deployment; without the path, rooms are random and temporary. -- Production runs behind nginx with Node.js 24.19.0. Authenticated coturn 4.17.2 at `turn.bonfire.icu:3478` has verified public STUN, TURN/UDP, TURN/TCP, and relay-only traffic. TURN/TLS is intentionally disabled. -- Production contains the default-off ADR-0005 controller but has no peer/SFU configuration, so it remains P2P/TURN. Repository configuration can restrict enabled peer/SFU routing to exact `PEER_ASSISTED_ROOM_IDS`; unlisted rooms stay legacy P2P, while empty or missing means all rooms. -- When enabled later, peer recovery precedes an allowlisted SFU root; session-bound revisions prepare/commit/abort, and an active SFU gets one token refresh before share-local failback. Browser relays re-encode; native shared encoding remains outside product code. -- Corrected Chrome 151/LiveKit 1.13.5 same-leaf/two-root cold/standby A/B reduced failure-to-active from 1.481 seconds to 200 ms and failure-to-render from 2.257 seconds to 319.7 ms. The leaf decoded/rendered 31 new frames and 25 ms sampling kept host edge peak two. It is localhost, headless, synthetic 720p30 video only; the roughly 86% result combines SDK download/parse and network prewarm and must not be extrapolated to public networks. -- Draft native ladder #16/#18/#22/#23/#25/#28 reaches one bounded Chrome 720p30 loop: one WebCodecs object feeds two Pion/browser legs, applies a conservative minimum stock-GCC target, and recovers one isolated loss. The minimum policy is experiment-only and forbidden in product; it is not proof of one physical/hardware encode. +- Access uses optional `ACCESS_PASSWORD`, a 12-hour stateless HMAC HttpOnly Strict cookie, internal host token, role-bound signaling, Origin/payload checks, and no accounts/JWT/session map. +- With `ROOM_DATABASE_PATH` and the password, SQLite stores room ID and host-token digest; links persist and room `1` survived deployment. Without it, rooms are temporary. +- Production uses nginx, Node.js 24.19.0, and authenticated coturn 4.17.2 at `turn.bonfire.icu:3478`; public STUN, TURN/UDP, TURN/TCP, and relay-only traffic pass. TURN/TLS is off. +- The default-off ADR-0005 controller has no production peer/SFU configuration. `PEER_ASSISTED_ROOM_IDS` restricts exact canary rooms; unlisted rooms stay P2P and empty/missing means all. +- When enabled, peer recovery precedes allowlisted SFU; session revisions prepare/commit/abort and active SFU gets one token refresh before failback. Browser relays re-encode; native sharing is outside product code. +- Chrome 151/LiveKit localhost A/B cut failure-to-active/render from 1.481/2.257 seconds to 0.200/0.320; 31 new frames and 25 ms sampling kept host edges at two. It is headless synthetic 720p30 and includes SDK/network prewarm, not public-network evidence. +- Draft native ladder #16/#18/#22/#23/#25/#28 has one bounded Chrome 720p30 loop: one WebCodecs object feeds two Pion/browser legs, applies an experiment-only minimum stock-GCC target, and recovers one isolated loss. It does not prove one hardware encode. - Stock Pion GCC plus negotiated RTX is a no-go. The no-RTX candidate can distort loss statistics and stays outside product code until audio, heterogeneous estimates, broader loss, TURN/reconnect, and browser-diversity gates pass; do not add custom congestion control to bypass them. - Chrome 151 short synthetic `1/3/5/8` runs kept host edges at two, relay edges at one, and all viewers decoding; a three-viewer relay close recovered in 5.32 seconds. This proves topology/control only, not quality, load, endurance, or silent partitions. -- The current quality stack uses one strict, non-persistent `QualitySettings` value with bounded ceilings and clarity/balanced/fluid preference. Sender readback and limitation warnings are manual diagnostics; ADR-0007's demand-driven `HIGH + optional LOW <= 2` controller is accepted but unimplemented. +- The current quality stack uses one strict, non-persistent `QualitySettings` value with bounded ceilings and clarity/balanced/fluid preference. Sender readback and limitation warnings remain manual diagnostics. ADR-0007's automatic per-path `HIGH`/`FALLBACK` and unique on-demand shared `LOW` are accepted mainline behavior, but the C report, state controller, and `LOW` runtime are unimplemented. - Chrome 151 synthetic 720p30 propagated balanced/clarity to one host and three viewers without changing peer identities; all viewers kept decoding and fanout stayed host two/relay one. This proves control continuity only. -- Per-frame encode/decode diagnostics use adjacent non-overlapping `getStats()` deltas rather than connection-lifetime averages; first, empty, changed-stream, and reset intervals remain unknown and rebase. -- Release `769de201f7cc` passed main CI, repository hygiene, type checking, 22 Vitest files/274 tests, both builds, zero production dependency vulnerabilities, atomic activation, public health/access checks, exact bundle verification, and service/log checks. +- The first local A+B alignment foundation binds host capture settings to one uniquely matched outbound track in the same tick, follows `remoteId` and the RTP-specific transport/candidate pair, and blocks stale generations across source replacement. Adjacent `getStats()` deltas rebase on first, empty, ambiguous, changed-identity, or reset intervals. Codec evidence remains MIME-only; this is not the complete A+B contract. +- Release `769de201f7cc` passed CI/hygiene/typecheck, 22 files/274 tests, both builds, zero production dependency vulnerabilities, atomic activation, public health/access, exact bundle, and service/log checks. ## Provisional Quality Targets @@ -73,6 +73,6 @@ These are measurement gates, not performance claims. - Documentation index and current phase: `docs/README.md` and `docs/status.md` - Requirements and design: `docs/需求理解.md` and `docs/方案设计.md` -- Media research: `docs/research/webrtc-p2p-screen-sharing.md`, `docs/research/realtime-quality-adaptation.md`, `docs/research/browser-screen-audio-quality.md`, `docs/research/peer-assisted-media.md`, `docs/research/low-server-media-routes.md`, `docs/research/advanced-peer-distribution.md`, and `docs/research/native-shared-encode-sender.md` +- Media research: `docs/research/`, indexed by `docs/README.md` - Architecture: ADR-0001/0002, proposed ADR-0004/0005, accepted ADR-0007, and rejected/superseded ADR-0003 - Deployment and maintenance: `docs/deployment.md`, `docs/maintenance.md`, `AGENTS.md`, and `.codex/` diff --git a/docs/research/realtime-quality-adaptation.md b/docs/research/realtime-quality-adaptation.md index 4c4165e..5f1fb16 100644 --- a/docs/research/realtime-quality-adaptation.md +++ b/docs/research/realtime-quality-adaptation.md @@ -106,8 +106,30 @@ The smallest implementation sequence is local A+B correlation in one host sampling tick, including the interval, media/stat identity, and valid deltas. Only after that is trustworthy should a minimal authenticated C report carry the receive/decode and derived negotiation signals needed by the two-state -predicate. It never carries raw SDP, raw stats, candidate addresses, or other -identifiers; a general remote stats stream or telemetry pipeline is unnecessary. +predicate. It never carries raw SDP, raw stats, candidate addresses, or raw +device/network identifiers. Opaque server-issued path and connection-generation +IDs provide authorization and correlation; a general remote stats stream or +telemetry pipeline is unnecessary. + +The repository now implements only the first local A+B alignment foundation: +same-tick capture settings plus one uniquely matched outbound RTP sample, +explicit sample/media identity and adjacent deltas, `remoteId` linkage, and the +selected path reached through that RTP stream's transport. Source replacement +blocks sampling and invalidates in-flight generations. This is not the complete +A+B contract: codec evidence is still MIME-only, derived codec +profile/parameters and applicable `scalabilityMode` are absent, and there is no +authenticated C report, two-state controller, or on-demand `LOW` runtime. + +Official W3C text checked 2026-08-19 defines names ending in `Id` as stats-object +references. In particular, outbound [`mediaSourceId`](https://www.w3.org/TR/webrtc-stats/#dom-rtcoutboundrtpstreamstats-mediasourceid) +references the sender's current media source, whose +[`trackIdentifier`](https://www.w3.org/TR/webrtc-stats/#dom-rtcmediasourcestats-trackidentifier) +is the `MediaStreamTrack.id`; RTP [`transportId`](https://www.w3.org/TR/webrtc-stats/#dom-rtcrtpstreamstats-transportid) +references its transport, and that transport's +[`selectedCandidatePairId`](https://www.w3.org/TR/webrtc-stats/#dom-rtctransportstats-selectedcandidatepairid) +references its selected pair. Therefore an ambiguous RTP set or a broken or +cross-transport reference remains unknown; an opaque stats ID is not a safe +substitute for those relationships. ## Accepted Adaptation Direction @@ -125,6 +147,12 @@ path exists or the second encoder exceeds the measured CPU/GPU/game budget, the controller preserves `HIGH` and fails visibly for the weak path. It never buys weak-path recovery by degrading healthy paths. +That fail-closed result is exceptional damage containment, not a supported +steady state. A supported sender cohort that cannot reliably start the one +shared `LOW` on demand fails automatic-quality-control acceptance; its encoding +path must be improved or the cohort explicitly marked unsupported. `HIGH` +remains protected in either case. + A viewer's `LOW` request is advisory and must be authenticated, session-bound, rate-limited, deduplicated, and corroborated by sender transport/encode and viewer receive/decode stats. UA or device-model detection is not quality @@ -257,7 +285,8 @@ from FPS alone. If reproducible evidence later shows that `balanced` still oscillates or makes the wrong tradeoff on supported machines, revise the fixed profiles before enabling ADR-0007. Its acceptance matrix must prove one healthy viewer remains `HIGH` while another enters/recover from `FALLBACK`, that all weak -viewers share at most one `LOW`, and that `LOW` stops after sustained recovery. +viewers move to exactly one shared `LOW`, that `LOW` stops after sustained +recovery, and that each supported sender cohort can start it reliably. ## Primary Sources diff --git a/docs/status.md b/docs/status.md index 9781259..61153c6 100644 --- a/docs/status.md +++ b/docs/status.md @@ -4,66 +4,64 @@ Last updated: 2026-08-19 ## Phase -`https://share.bonfire.icu` runs commit `769de201f7cc` with simplified entry, -default-closed diagnostics, bounded clarity-first quality settings, persistent -rooms, and reusable links. Peer assistance and LiveKit are unconfigured, so the -deployed binary still uses one host peer connection per viewer and P2P/TURN. +`https://share.bonfire.icu` runs `769de201f7cc` with simplified entry, bounded +clarity-first controls, persistent rooms, and reusable links. Peer/LiveKit is +unconfigured, so production remains one host connection per viewer and P2P/TURN. ## Current Snapshot -- Capture precedes room creation. Live source/quality changes preserve healthy peers; picture pause keeps audio and connections active. Quality defaults to `maintain-resolution`, with balanced and fluid options. -- Whole-site `ACCESS_PASSWORD` is optional. Protected sessions use a stateless 12-hour HMAC HttpOnly `SameSite=Strict` cookie; host authentication remains internal and signaling is role-bound. -- Without `ROOM_DATABASE_PATH`, rooms are random and temporary. With both the database path and site password, room IDs start at `1`, links persist, and stopping a share leaves viewers waiting. SQLite stores only room ID and host-token digest. +- Capture precedes room creation; live source/quality changes preserve peers and picture pause keeps audio/connections. Quality defaults to `maintain-resolution` with balanced/fluid options. +- Optional `ACCESS_PASSWORD` uses a stateless 12-hour HMAC HttpOnly Strict cookie; host auth stays internal and signaling role-bound. +- Without `ROOM_DATABASE_PATH`, rooms are temporary. With it and the password, sequential links persist after stop; SQLite stores only room ID and host-token digest. - Direct ICE is preferred independently per media edge. Authenticated TURN/UDP and TURN/TCP are required production fallbacks; TURN/TLS is optional. -- Hidden routing is `direct P2P -> peer-assisted -> optional SFU`. `PEER_ASSISTED_ROOM_IDS` canaries exact rooms; others stay legacy P2P. Empty or missing means all, production is disabled, and the 22-file/284-test full suite covers both paths. -- Complete fallback configuration adds a non-secret standby URL to peer-assisted authentication. Host and viewers import the SDK and make one token-free DNS/TLS warmup; no configuration means no field, import, request, participant, or media edge. -- A viewer starts as a leaf each session and explicitly advertises relay capacity zero or one; the Web client reports detected mobile/iPad clients as zero and desktop-class browsers as one. Withdrawal stops future assignment without moving a healthy edge. Browser relays remain one-child; the host remains two-child. -- Production uses one strict, memory-only video quality setting with bounded manual ceilings and sender readback. ADR-0007's per-path `HIGH`/`FALLBACK` and demand-driven shared `LOW` are accepted design, not deployed behavior. Browser audio remains request/presence-only. +- Hidden routing is `direct P2P -> peer-assisted -> optional SFU`; exact-room canaries use `PEER_ASSISTED_ROOM_IDS`. Production is off and 22 files/290 tests cover both paths. +- Configured fallback adds a non-secret standby URL and token-free DNS/TLS warmup; absent configuration adds no field, import, request, participant, or edge. +- Each viewer advertises relay capacity 0/1; detected mobile/iPad clients are leaves. Withdrawal does not move a healthy edge. Browser relays stay one-child and host two-child. +- Production has one memory-only video setting with manual ceilings/readback. Automatic per-path `HIGH`/`FALLBACK` and one on-demand shared `LOW` are accepted but not deployed; only the first local A+B foundation exists. Browser audio is request/presence-only. ## Verified Evidence -- Release `769de201f7cc` passed main CI, repository hygiene, type checking, 22 Vitest files/274 tests, both builds, and zero production dependency vulnerabilities. -- A short Chrome 151 synthetic `1/3/5/8` benchmark passed every topology check: host active edges peaked at two, relay edges at one, every viewer kept increasing decoded frames through the measurement window, and the slowest first decoded frame was about 1.05 seconds. Closing a first-level relay in the three-viewer run recovered in about 5.32 seconds without exceeding host fanout two. -- Chrome 151 synthetic 720p30 propagated balanced/clarity to one host and three viewers without changing peer identities; all kept decoding and fanout stayed host two/relay one. This proves control continuity only. -- Chrome 151/LiveKit localhost A/B measured cold active/render at 1.481/2.257 seconds and standby at 0.200/0.320 seconds while host edges stayed at two. This video-only local result includes SDK startup plus prewarm and does not predict public networks. -- Atomic production activation passed public/local health, exact served-bundle hash, unauthenticated 401, room `1` page, SQLite preservation, active Screener/nginx/coturn, and a clean warning log. The old release remains available for rollback. -- Chrome 151 checks at 320/375/390 CSS px passed equal 44 px entry actions, waiting-page overflow, and default-closed/resetting details. Live media was not covered. -- Local diagnostics use adjacent non-overlapping `getStats()` deltas; empty, changed-stream, and reset intervals rebase instead of publishing lifetime averages. -- Production HTTPS/WSS, access cookie, room/WebSocket authorization, certificate renewal, public STUN, and authenticated TURN/UDP and TURN/TCP relay-only bidirectional paths are verified. TURN/TLS is intentionally disabled. -- Draft native ladder #16/#18/#22/#23/#25/#28 passed a bounded 720p30 two-leg loop: one WebCodecs object accepted a conservative minimum stock-GCC target, one isolated loss recovered, and leg 2 stayed clean. That minimum rule is experiment-only and prohibited from productization. +- Release `769de201f7cc` passed CI/hygiene/typecheck, 22 files/274 tests, both builds, and zero production dependency vulnerabilities. +- Chrome 151 synthetic `1/3/5/8` kept host/relay edges at 2/1 and all viewers decoding; slowest first frame was 1.05 seconds and one three-viewer relay close recovered in 5.32 seconds. +- Synthetic Chrome 151 720p30 propagated balanced/clarity to one host and three viewers without peer changes; decoding and 2/1 fanout continued. This proves control continuity only. +- Chrome 151/LiveKit localhost A/B measured cold active/render at 1.481/2.257 seconds versus standby 0.200/0.320 with host edges at two. It includes SDK/prewarm and is not public-network evidence. +- Atomic activation passed health/access, exact bundle, SQLite preservation, services, and logs; the old release is rollback-ready. Responsive 320/375/390 px checks passed without live media. +- The local A+B alignment foundation samples capture settings with one uniquely matched host outbound track, explicit window/media identity and valid adjacent deltas, `remoteId`, and the RTP-bound selected path. Source replacement blocks sampling and discards stale in-flight generations; ambiguous, changed, and reset evidence rebases to unknown. +- HTTPS/WSS, access and room/WebSocket auth, renewal, public STUN, and authenticated TURN/UDP/TCP relay-only paths pass; TURN/TLS is off. +- Draft native ladder #16/#18/#22/#23/#25/#28 passed one bounded 720p30 two-leg loop with an experiment-only minimum GCC target and one recovered loss; it is not product behavior. ## Unverified Boundaries -- Full-resolution 30-minute runs, relay CPU/GPU, generational quality, controlled loss/RTT, depth latency, and actual game capture remain unverified. -- Android Chrome and iOS Safari remain unverified leaves. Runtime capability conservatively marks detected mobile/iPad clients as leaves; real UA/lifecycle behavior remains open. +- Full-resolution 30-minute runs, relay CPU/GPU, generational quality, controlled loss/RTT, depth latency, and game capture are unverified. +- Android Chrome/iOS Safari leaves are unverified; runtime conservatively marks detected mobile/iPad clients as leaves. - Silent partitions can wait 30 to 60 seconds for heartbeat detection before the default 5-second grace; this remains unverified. -- Real screen/game audio, heterogeneous machines/networks, mobile lifecycle, the new production quality/pause cycle, room `1` stop/re-publish, and sustained profiles remain unverified. -- Earlier production showed severe degradation across all profiles. The new controls permit clarity/balanced comparison, but a controlled 1/2/3-viewer and TURN sample must still isolate capture, CPU, path, and receiver limits before any quality claim or automatic controller. -- ADR-0007 is unimplemented. First correlate same-tick capture plus outbound window/identity/deltas; only then add a minimal authenticated viewer receive/decode report. Runtime evidence, not UA/device identity, must drive `HIGH`/`FALLBACK`; representations stay at `HIGH + optional LOW <= 2`. -- Browser relays re-encode. The native ladder proves one WebCodecs object, not one physical/hardware encode. Stock GCC plus RTX is no-go; no-RTX passed one controlled loss but weakens statistics. Audio, heterogeneous estimates, broader loss, direct/TURN, reconnect, browser diversity, product wiring, striping, and multi-parent assembly remain unverified. +- Real audio, heterogeneous clients, mobile lifecycle, quality/pause, room `1` republish, and sustained profiles are unverified. Poor movie/video audio is user-reported but unclassified; after video A+B, diagnose capture settings, codec/fmtp, actual bitrate, loss, jitter, concealment, jitter buffer, and A/V sync. +- Earlier production degraded across profiles. Controlled 1/2/3-viewer and TURN samples must isolate capture, CPU, path, and receiver limits before quality/controller claims. +- ADR-0007 remains incomplete. The first local A+B alignment foundation is implemented, but codec evidence is MIME-only and still lacks derived profile/parameters/`scalabilityMode`; the minimal authenticated C report, two-state controller, and `LOW` runtime do not exist. Verified weak paths must ultimately move to the one shared `LOW`, and supported sender cohorts that cannot start it reliably fail acceptance; fail closed protects `HIGH` only as exceptional damage containment. +- Browser relays re-encode. The native ladder proves one WebCodecs object, not hardware encode. GCC+RTX is no-go; no-RTX weakens stats. Broader audio/loss/routes/reconnect/browser/product/striping gates remain. - The endpoint budget remains host/relay at most two downstream edges; the browser path is host two/viewer one. - The corrected standby smoke is localhost/headless/video-only; public DNS/TLS reuse, transport, audio, shaping, load, mobile, endurance, and sub-25 ms overlap remain open. ## Next Milestone -First validate the deployed quality controls on real game capture and implement -the bounded A+B probe described above. Before any -broad route enablement, use one exact-room canary and validate standby gains -across public transports, rollback, reconnect, edge counts, egress, and load. +Validate production controls on real game capture and finish video A+B codec +derivation. Then run the separate movie/video audio A/B/C and sync diagnosis in +`docs/research/browser-screen-audio-quality.md`; `maxBitrate` is a ceiling, not +a quality fix. Next add minimal authenticated video C, the two-state controller, +and on-demand shared `LOW`. Route enablement still needs one exact-room canary +across public transports, recovery, edge counts, egress, and load. ADR-0004 still requires a full-resolution 30-minute `1/3/5/8` network, resource, quality, latency, recovery, and browser/mobile-leaf matrix. -For a disconnect immediately observed by the server, the gate retains the -default 5-second grace followed by at most 3 seconds to restore a decodable -picture, about 8 seconds total. Silent partitions include their detection delay. +Observed disconnects retain 5-second grace plus at most 3 seconds to a decodable +picture; silent partitions add detection delay. ADR-0004 fails closed. Native shared encode and packet/layer striping remain separate experiments, not ways to relabel a failed browser route. -The native ladder stops at #28. Its minimum-of-two feedback policy is not a -product candidate. Do not bypass stock GCC/RTX with custom transport/control; -no-RTX remains research-only. +The native ladder stops at #28; its minimum-of-two policy is not a product +candidate. Do not bypass stock GCC/RTX; no-RTX remains research-only. ## Blockers And Decisions @@ -76,4 +74,4 @@ Deferred architecture audit: `docs/maintenance.md`. - Whole-system versus selected-game audio for the first release. - Initial deployment region and network cohort. - Project license and distribution model. -- Whether peer assistance passes every non-encoding gate; any other failure closes that route. +- Whether peer assistance passes every non-encoding gate; any failure closes it. diff --git a/docs/方案设计.md b/docs/方案设计.md index 3b4e0c9..4e036a1 100644 --- a/docs/方案设计.md +++ b/docs/方案设计.md @@ -149,7 +149,7 @@ PoC 不强制 codec，也不做 SDP munging。先通过 stats 记录实际协商 `frameRate` 约束和 sender 的 `maxFramerate` 都是目标或上限；浏览器仍控制实际捕获、重复帧处理、编码和拥塞降级。静态画面可能由浏览器减少实际编码数据，但标准没有保证每个实现以同一种方式节省采集或 GPU 开销。本 PoC 不添加画面差分、定时改约束或自定义动态 FPS 状态机，先用 `framesEncoded`、实际发送码率、`totalEncodeTime`、质量受限原因和主机资源占用测量真实问题。 -自动质量控制采用 ADR-0007 的按需双表示，而不是全房间按最差链路降档。正常状态只运行一套共享 `HIGH` 编码；第一条被真实统计持续证明无法承受的观看路径出现时，临时启动一套共享 `LOW`，所有弱路径复用它，健康路径继续接收 `HIGH`。最后一条弱路径经过更长的稳定恢复窗口回到 `HIGH` 后关闭 `LOW`。表示数量硬限制为 `HIGH + optional LOW <= 2`，不能按 viewer 增长为 N 套编码。若没有合格硬件/power-efficient 编码路径或第二编码实测破坏游戏 CPU/GPU 预算，则不启动 `LOW`，保留健康路径的 `HIGH` 并让弱路径明确降级或失败。 +自动质量控制采用 ADR-0007 的按需双表示，而不是全房间按最差链路降档。正常状态只运行一套共享 `HIGH` 编码；第一条被真实统计持续证明无法承受的观看路径出现时，必须启动唯一一套共享 `LOW` 并迁移所有已验证弱路径，健康路径继续接收 `HIGH`。最后一条弱路径经过更长的稳定恢复窗口回到 `HIGH` 后关闭 `LOW`。表示数量硬限制为 `HIGH + at most one on-demand LOW <= 2`，不能按 viewer 增长为 N 套编码。若没有合格硬件/power-efficient 编码路径或第二编码实测破坏游戏 CPU/GPU 预算，则当次 fail closed，保留 `HIGH` 并明确显示弱路径降级或失败；这只是异常止损。supported sender cohort 若不能可靠按需启动 `LOW`，则自动质量控制验收失败，必须优化编码路径或明确标记不支持，不能长期忽略弱路径。 每条观看路径只维护 `HIGH`/`FALLBACK` 两态。`HIGH -> FALLBACK` 需要多个连续采样窗口中的带宽不足、冻结或解码跟不上；`FALLBACK -> HIGH` 使用不同门槛和更长的稳定窗口，避免来回抖动。不把 RTT、CPU、码率、设备信息等压成一个综合分数。具体窗口和阈值必须由真实质量矩阵确定，而不是先写死一个猜测。 @@ -163,7 +163,7 @@ PoC 不强制 codec，也不做 SDP munging。先通过 stats 记录实际协商 判断顺序固定为：Capture 低归为采集或约束；Capture 高而 outbound 低且 reason=`cpu` 归为编码资源；Capture 高而 outbound 低且 reason=`bandwidth` 归为 GCC、上行或 TURN/路径；outbound 正常而 inbound 低归为传输或接收；inbound 指标正常但肉眼模糊归为码率/量化、协商 codec/profile/parameters、适用的 scalable layer 或显示缩放。B/C 必须关联同一派生协商结果与实际解码表现。累计统计只计算相邻、不重叠区间的 delta；缺字段、stats ID/媒体代次变化或计数器回退时重新建立基线，不能把未知当零。 -探针按最小顺序落地：先在 host 同一个采样 tick 内关联 A+B，并携带明确 window、媒体/stats identity 和有效 delta；验证可靠后才增加只包含两态判定所需字段的受限、已鉴权 C 上报。不建设通用遥测 schema，只传派生后的协商字段，绝不上传 raw stats、raw SDP、candidate 地址或其他标识符。 +探针按最小顺序落地：先在 host 同一个采样 tick 内关联 A+B，并携带明确 window、媒体/stats identity 和有效 delta；验证可靠后才增加只包含两态判定所需字段的受限、已鉴权 C 上报。不建设通用遥测 schema，只传派生后的协商字段，绝不上传 raw stats、raw SDP、candidate 地址或原始设备/网络标识；鉴权与关联只使用服务端签发的不透明 path/connection generation ID。 Viewer 可通过已鉴权、绑定当前房间/路径代次的控制面请求 `LOW`，但请求只是一条观测输入，必须限频和去重，并与 sender 的 Transport-CC/GCC、实际码率、RTT、丢包、重传及编码限制、派生协商 codec/profile/parameters（适用时含 `scalabilityMode`）和 viewer 的实际解码/丢帧/冻结表现交叉验证。UA、平台和设备型号不参与质量决策；现有 mobile/iPad 检测只用于保守设置 relay capacity。 diff --git a/docs/需求理解.md b/docs/需求理解.md index 593b57e..ca2c5cc 100644 --- a/docs/需求理解.md +++ b/docs/需求理解.md @@ -22,8 +22,8 @@ - [ ] P2P 优先：每条逻辑媒体边的 ICE 必须先尝试直接 UDP 路径，服务器不得默认中转可直连的媒体。一个或两个观看者优先直接连接分享端；更多观看者的 peer-assisted 行为须先通过 ADR-0004 的实验门槛。 - [ ] NAT 兜底：每条 parent-child 边独立进行 ICE；生产环境必须提供 STUN、TURN/UDP 与 TURN/TCP candidates，通过优先级偏好直连和 UDP，同时允许无法使用 UDP 的网络选择 TCP，不能因为某条边的网络与其他边不同而静默失败。 - [ ] 质量控制：保留 1080p60（8 Mbps）、1080p30（5 Mbps）、720p30（3 Mbps）三档推荐上限，默认采用清晰优先；分享期间可直接换档或应用有界高级设置，不重新选择来源或重建健康连接。Peer-assisted relay 与可选 SFU 必须接收同一个严格校验的房间级权威设置，覆盖当前和以后建立的 sender；普通 P2P 信令结构保持不变。任何设置都只是浏览器上限和偏好，必须读回实际 sender 参数并显示参数拒绝、改写和持续受限原因。不做自定义动态 FPS、综合评分或周期性全房间调档。 -- [ ] 按需双表示：健康房间只运行一套共享 `HIGH` 编码；只有一个或多个观看路径被同一时段的 sender 与 viewer 统计持续证明无法承受时，才临时启动一套由全部弱路径共用的 `LOW` 编码。健康路径继续使用 `HIGH`，全部弱路径稳定恢复后关闭 `LOW`。编码表示硬上限为二，禁止按观看者增长为 N 套，也禁止把最差路径的目标应用到整个房间。没有合格硬件/power-efficient 路径或第二编码造成不可接受的 CPU/GPU/游戏负载时必须 fail closed：保护 `HIGH`，向弱路径明确显示降级或不可用。 -- [ ] 路径级质量状态：每条观看路径只使用 `HIGH` 与 `FALLBACK` 两态。进入 `FALLBACK` 需要多个连续窗口的带宽不足、冻结或解码跟不上证据，回到 `HIGH` 需要更长的稳定窗口；进入与退出使用不同门槛且不计算综合分数。Viewer 的低档请求必须经当前会话鉴权、限频和去重，并由 sender 的 GCC/Transport-CC、实际码率、RTT、丢包、重传、编码限制、派生后的实际协商视频 codec/profile/parameters（适用时含 `scalabilityMode`）与 viewer 的实际解码、丢帧、冻结证据交叉验证，不能仅凭自报生效；上报不得包含 raw SDP、candidate 地址或其他标识符。 +- [ ] 按需双表示：健康房间只运行一套共享 `HIGH` 编码；第一条观看路径被同一时段的 sender 与 viewer 统计持续证明无法承受时，系统必须启动唯一一套由全部弱路径共用的 `LOW` 并迁移弱路径。健康路径继续使用 `HIGH`，全部弱路径稳定恢复后关闭 `LOW`。编码表示硬上限为二，禁止按观看者增长为 N 套，也禁止把最差路径的目标应用到整个房间。没有合格硬件/power-efficient 路径或第二编码造成不可接受负载时，当次必须 fail closed：保护 `HIGH` 并明确显示弱路径降级或不可用；这只是异常止损，supported sender cohort 若不能可靠按需启动 `LOW`，则自动质量控制验收失败，必须优化编码路径或明确标记不支持。 +- [ ] 路径级质量状态：每条观看路径只使用 `HIGH` 与 `FALLBACK` 两态。进入 `FALLBACK` 需要多个连续窗口的带宽不足、冻结或解码跟不上证据，回到 `HIGH` 需要更长的稳定窗口；进入与退出使用不同门槛且不计算综合分数。Viewer 的低档请求必须经当前会话鉴权、限频和去重，并由 sender 的 GCC/Transport-CC、实际码率、RTT、丢包、重传、编码限制、派生后的实际协商视频 codec/profile/parameters（适用时含 `scalabilityMode`）与 viewer 的实际解码、丢帧、冻结证据交叉验证，不能仅凭自报生效；上报不得包含 raw SDP、candidate 地址或原始设备/网络标识，鉴权与关联只使用服务端签发的不透明路径/连接代次 ID。 - [ ] 断线恢复：临时网络变化时尝试 ICE restart 和重连，并向用户显示当前连接状态。 - [ ] 可观测性：采集候选类型、RTT、丢包、抖动、实际码率、帧率、编码耗时和质量受限原因。 - [ ] 渐进式诊断：分享者和观看者默认只看到房间、画面、核心状态和常用控制；候选路径、RTT、丢包、编解码器、编码耗时等技术详情由一个明确的“显示连接详情”开关展开。隐藏诊断不能隐藏失败原因、音频缺失、持续画质受限或需要用户操作的告警。 @@ -112,14 +112,15 @@ - Given 用户选择任一推荐档位或高级质量参数，When 接收端实际分辨率、帧率或码率持续显著低于该目标，Then 诊断数据能区分采集约束、发送编码、带宽/拥塞、TURN 路径和接收端限制；产品不得只改变档位标签却继续隐藏实际降级。 - Given 房间所有观看路径持续健康，When 自动质量控制运行，Then 只存在一套共享 `HIGH` 表示，且任何单路径短暂抖动或未经验证的低档请求都不启动第二套编码或降低其他路径。 -- Given 一个或多个路径在连续窗口中被 sender 与 viewer 证据共同确认无法承受 `HIGH`，When 它们进入 `FALLBACK`，Then 系统最多启动一套共享 `LOW`，只切换这些弱路径，其他路径继续 `HIGH`；所有弱路径经过更长稳定恢复窗口后返回 `HIGH` 并关闭 `LOW`。 -- Given 启动第二套编码会超过游戏 CPU/GPU 预算或没有合格硬件/power-efficient 路径，When 弱路径请求 `LOW`，Then 系统保留健康路径的 `HIGH` 并向弱路径明确显示降级或不可用，不得退回全房间最低目标或静默软件编码。 +- Given 一个或多个路径在连续窗口中被 sender 与 viewer 证据共同确认无法承受 `HIGH`，When 它们进入 `FALLBACK`，Then 系统必须启动且最多只启动一套共享 `LOW`，只迁移这些弱路径，其他路径继续 `HIGH`；所有弱路径经过更长稳定恢复窗口后返回 `HIGH` 并关闭 `LOW`。 +- Given 启动第二套编码会超过游戏 CPU/GPU 预算或没有合格硬件/power-efficient 路径，When 弱路径请求 `LOW`，Then 当次保留健康路径的 `HIGH` 并向弱路径明确显示降级或不可用，不得退回全房间最低目标或静默软件编码；若 supported sender cohort 无法可靠按需启动 `LOW`，Then 自动质量控制验收失败，必须优化编码路径或明确标记不支持，不能把 fail closed 当作长期忽略弱路径的许可。 - Given 同一时刻采集、host outbound 与 viewer inbound 证据，When 进行根因分类，Then 关联派生后的实际协商视频 codec/profile/parameters、适用时的 `scalabilityMode` 与真实解码表现，依次区分 Capture 低、Capture 高但 outbound 因 CPU 低、Capture 高但 outbound 因 bandwidth 低、outbound 正常但 inbound 低、以及 inbound 正常但肉眼模糊五类；不用 UA 或设备型号代替运行时证据，也不上传 raw SDP。 - Given Viewer 发送低档请求，When 请求未鉴权、过频、重复、过期或没有匹配的 sender/viewer 统计，Then 请求不改变表示或路径状态；有效请求也只能作为交叉验证输入。 - Given 受控 RTT/丢包测试条件，When 连续分享高动态游戏画面 30 分钟，Then 延迟、帧率与丢帧达到上述目标，且没有不可恢复的冻结。 - Given 分享过程中切换 Wi-Fi/有线网络，When ICE 连接失败，Then 客户端触发 ICE restart，并在成功或超时后给出明确状态。 - Given 浏览器未返回系统音轨，When 分享开始，Then 分享者在启播前看到“当前来源没有可共享音频”的提示。 - Given 浏览器返回屏幕音轨，When 产品展示音频状态或未来连接详情，Then 只能把存在性、浏览器实际返回的 track settings、实际协商 codec/fmtp 和实际流量分别标为捕获、协商或观测事实；字段缺失必须显示为未知，且不得从 Opus 的 `48000/2` RTP 声明推断源采样率或实际 stereo。 +- Given Web 或后续 native sender 分享电影/视频音频，When 进行产品验收，Then 必须用独立 A/B/C 证据关联 capture audio track/settings、实际 negotiated codec/fmtp、outbound/inbound 实际码率、loss、jitter、concealment、jitter-buffer 与 A/V sync；native sender 必须通过该音质和同步矩阵，且不得把 audio `maxBitrate` ceiling 当作提质证明。 ### 仓库可移植性 diff --git a/src/client/types.ts b/src/client/types.ts index 4179996..b687bed 100644 --- a/src/client/types.ts +++ b/src/client/types.ts @@ -7,6 +7,16 @@ export type SignalConnectionState = export type MediaPath = "direct" | "relay" | "unknown"; export interface ConnectionMetrics { + sampleTimestampMs: number | null; + sampleWindowMs: number | null; + rtpStatsId: string | null; + rtpSsrc: number | null; + rtpMid: string | null; + trackIdentifier: string | null; + selectedCandidatePairId: string | null; + captureWidth: number | null; + captureHeight: number | null; + captureFramesPerSecond: number | null; path: MediaPath; iceProtocol: string | null; localRelayProtocol: string | null; @@ -20,6 +30,11 @@ export interface ConnectionMetrics { packetsLost: number | null; jitterMs: number | null; framesDropped: number | null; + intervalFramesDropped: number | null; + intervalFreezeCount: number | null; + intervalFreezeDurationMs: number | null; + intervalRetransmittedPackets: number | null; + intervalRetransmittedBytes: number | null; codec: string | null; encoderImplementation: string | null; powerEfficientEncoder: boolean | null; @@ -40,6 +55,16 @@ export interface PeerSnapshot { } export const EMPTY_METRICS: ConnectionMetrics = { + sampleTimestampMs: null, + sampleWindowMs: null, + rtpStatsId: null, + rtpSsrc: null, + rtpMid: null, + trackIdentifier: null, + selectedCandidatePairId: null, + captureWidth: null, + captureHeight: null, + captureFramesPerSecond: null, path: "unknown", iceProtocol: null, localRelayProtocol: null, @@ -53,6 +78,11 @@ export const EMPTY_METRICS: ConnectionMetrics = { packetsLost: null, jitterMs: null, framesDropped: null, + intervalFramesDropped: null, + intervalFreezeCount: null, + intervalFreezeDurationMs: null, + intervalRetransmittedPackets: null, + intervalRetransmittedBytes: null, codec: null, encoderImplementation: null, powerEfficientEncoder: null, diff --git a/src/client/webrtc/host-peer.ts b/src/client/webrtc/host-peer.ts index fa60623..3aa40cf 100644 --- a/src/client/webrtc/host-peer.ts +++ b/src/client/webrtc/host-peer.ts @@ -7,6 +7,7 @@ import { } from "../media/quality"; import { EMPTY_METRICS, + type ConnectionMetrics, type PeerSnapshot, } from "../types"; import { @@ -25,6 +26,38 @@ interface HostPeerEvents { onUpdate: (snapshot: PeerSnapshot) => void; } +type CaptureMetrics = Pick< + ConnectionMetrics, + "captureWidth" | "captureHeight" | "captureFramesPerSecond" +>; + +function captureMetrics(track: MediaStreamTrack): CaptureMetrics { + if (typeof track.getSettings !== "function") { + return { + captureWidth: null, + captureHeight: null, + captureFramesPerSecond: null, + }; + } + + try { + const settings = track.getSettings(); + const finiteNumber = (value: unknown): number | null => + typeof value === "number" && Number.isFinite(value) ? value : null; + return { + captureWidth: finiteNumber(settings.width), + captureHeight: finiteNumber(settings.height), + captureFramesPerSecond: finiteNumber(settings.frameRate), + }; + } catch { + return { + captureWidth: null, + captureHeight: null, + captureFramesPerSecond: null, + }; + } +} + export class HostPeer { readonly connectionId = createOpaqueId(); @@ -35,6 +68,7 @@ export class HostPeer { private audioSender: RTCRtpSender | null = null; private statsTimer: number | null = null; private statsInFlight = false; + private statsSamplingBlocked = false; private senderWarning: string | null = null; private limitationReason: string | null = null; private limitationSamples = 0; @@ -115,28 +149,37 @@ export class HostPeer { const nextAudioTrack = nextStream.getAudioTracks()[0] ?? null; const previousVideoTrack = videoSender.track; const previousAudioTrack = audioSender.track; + this.statsSamplingBlocked = true; + this.statsAccumulator = createStatsAccumulator(); try { - await videoSender.replaceTrack(nextVideoTrack); - await audioSender.replaceTrack(nextAudioTrack); - } catch (error) { - await Promise.allSettled([ - videoSender.replaceTrack(previousVideoTrack), - audioSender.replaceTrack(previousAudioTrack), - ]); - this.setError(error, "切换共享源失败"); - return false; - } + try { + await videoSender.replaceTrack(nextVideoTrack); + await audioSender.replaceTrack(nextAudioTrack); + } catch (error) { + await Promise.allSettled([ + videoSender.replaceTrack(previousVideoTrack), + audioSender.replaceTrack(previousAudioTrack), + ]); + this.setError(error, "切换共享源失败"); + return false; + } - if (this.disposed) { - return false; + if (this.disposed) { + return false; + } + this.stream = nextStream; + this.limitationReason = null; + this.limitationSamples = 0; + this.snapshot = { ...this.snapshot, metrics: { ...EMPTY_METRICS } }; + await this.configureSender(videoSender); + this.snapshot = { ...this.snapshot, error: null }; + this.emit(); + return true; + } finally { + this.statsAccumulator = createStatsAccumulator(); + this.statsSamplingBlocked = false; } - this.stream = nextStream; - this.statsAccumulator = createStatsAccumulator(); - await this.configureSender(videoSender); - this.snapshot = { ...this.snapshot, error: null }; - this.emit(); - return true; }); } @@ -309,19 +352,35 @@ export class HostPeer { if ( this.disposed || this.connection.connectionState === "closed" || - this.statsInFlight + this.statsInFlight || + this.statsSamplingBlocked ) { return; } + const captureTrack = this.videoSender?.track ?? null; + if (!captureTrack || this.stream.getVideoTracks()[0] !== captureTrack) { + return; + } this.statsInFlight = true; const statsAccumulator = this.statsAccumulator; try { - const metrics = await collectConnectionMetrics( + const metricsPromise = collectConnectionMetrics( this.connection, "send", statsAccumulator, + { trackIdentifier: captureTrack.id }, ); - if (this.disposed || this.statsAccumulator !== statsAccumulator) { + const capture = captureMetrics(captureTrack); + const metrics = { ...(await metricsPromise), ...capture }; + if ( + this.disposed || + this.statsSamplingBlocked || + this.statsAccumulator !== statsAccumulator || + this.videoSender?.track !== captureTrack || + this.stream.getVideoTracks()[0] !== captureTrack || + (metrics.trackIdentifier !== null && + metrics.trackIdentifier !== captureTrack.id) + ) { return; } this.updateLimitationWarning(metrics.qualityLimitationReason); diff --git a/src/client/webrtc/stats.ts b/src/client/webrtc/stats.ts index 3609aa0..fc77ace 100644 --- a/src/client/webrtc/stats.ts +++ b/src/client/webrtc/stats.ts @@ -11,24 +11,58 @@ type StatsRecord = Record & { export interface StatsAccumulator { mediaId: string | null; + ssrc: number | null; + trackIdentifier: string | null; bytes: number | null; frames: number | null; timestamp: number | null; previousTotalEncodeTime: number | null; previousTotalDecodeTime: number | null; + previousFramesDropped: number | null; + previousFreezeCount: number | null; + previousTotalFreezesDuration: number | null; + previousRetransmittedPackets: number | null; + previousRetransmittedBytes: number | null; +} + +export interface StatsMediaSelector { + trackIdentifier: string | null; } export function createStatsAccumulator(): StatsAccumulator { return { mediaId: null, + ssrc: null, + trackIdentifier: null, bytes: null, frames: null, timestamp: null, previousTotalEncodeTime: null, previousTotalDecodeTime: null, + previousFramesDropped: null, + previousFreezeCount: null, + previousTotalFreezesDuration: null, + previousRetransmittedPackets: null, + previousRetransmittedBytes: null, }; } +function intervalDelta( + current: number | null, + previous: number | null, + hasInterval: boolean, +): number | null { + if ( + !hasInterval || + current === null || + previous === null || + current < previous + ) { + return null; + } + return current - previous; +} + function intervalAverageMs( totalTime: number | null, previousTotalTime: number | null, @@ -73,76 +107,125 @@ function getRecord(report: RTCStatsReport, id: string | null): StatsRecord | nul return (report.get(id) as StatsRecord | undefined) ?? null; } -function selectedCandidatePair(report: RTCStatsReport): StatsRecord | null { - let transport: StatsRecord | null = null; +function mediaTrackIdentifier( + report: RTCStatsReport, + media: StatsRecord, + direction: "send" | "receive", +): string | null { + if (direction === "receive") { + return stringValue(media, "trackIdentifier"); + } + const source = getRecord(report, stringValue(media, "mediaSourceId")); + return source?.type === "media-source" + ? stringValue(source, "trackIdentifier") + : null; +} + +function mediaRecord( + report: RTCStatsReport, + direction: "send" | "receive", + selector: StatsMediaSelector | null, +): StatsRecord | null { + const expectedType = direction === "send" ? "outbound-rtp" : "inbound-rtp"; + const isExpectedMedia = (record: StatsRecord): boolean => + record.type === expectedType && + record.kind === "video" && + record.isRemote !== true; + + const candidates: StatsRecord[] = []; + report.forEach((raw) => { + const record = raw as StatsRecord; + if (isExpectedMedia(record)) { + candidates.push(record); + } + }); + const selectedTrackIdentifier = selector?.trackIdentifier ?? null; + const narrowed = selectedTrackIdentifier !== null + ? candidates.filter( + (candidate) => + mediaTrackIdentifier(report, candidate, direction) === + selectedTrackIdentifier, + ) + : candidates; + return narrowed.length === 1 ? narrowed[0]! : null; +} + +function transportRecord( + report: RTCStatsReport, + media: StatsRecord | null, +): StatsRecord | null { + const referenced = getRecord(report, stringValue(media, "transportId")); + return referenced?.type === "transport" ? referenced : null; +} + +function selectedCandidatePair( + report: RTCStatsReport, + transport: StatsRecord | null, +): StatsRecord | null { + if (!transport) { + return null; + } const candidatePairs: StatsRecord[] = []; report.forEach((raw) => { const record = raw as StatsRecord; - if (record.type === "transport") { - transport = record; - } else if (record.type === "candidate-pair") { + if (record.type === "candidate-pair") { candidatePairs.push(record); } }); const pairId = stringValue(transport, "selectedCandidatePairId"); if (pairId) { - return getRecord(report, pairId); + const referenced = getRecord(report, pairId); + return referenced?.type === "candidate-pair" && + stringValue(referenced, "transportId") === transport.id + ? referenced + : null; } - return ( - candidatePairs.find( - (pair) => - pair.state === "succeeded" && - (pair.nominated === true || pair.selected === true), - ) ?? null + const selected = candidatePairs.filter( + (pair) => + stringValue(pair, "transportId") === transport.id && + pair.state === "succeeded" && + (pair.nominated === true || pair.selected === true), ); + return selected.length === 1 ? selected[0]! : null; } -function mediaRecord( +function linkedRemoteInbound( report: RTCStatsReport, - direction: "send" | "receive", + outbound: StatsRecord | null, ): StatsRecord | null { - let result: StatsRecord | null = null; - report.forEach((raw) => { - const record = raw as StatsRecord; - const expectedType = direction === "send" ? "outbound-rtp" : "inbound-rtp"; - if ( - record.type === expectedType && - record.kind === "video" && - record.isRemote !== true - ) { - result = record; - } - }); - return result; -} - -function remoteInboundVideo(report: RTCStatsReport): StatsRecord | null { - let result: StatsRecord | null = null; - report.forEach((raw) => { - const record = raw as StatsRecord; - if (record.type === "remote-inbound-rtp" && record.kind === "video") { - result = record; - } - }); - return result; + const remote = getRecord(report, stringValue(outbound, "remoteId")); + return remote?.type === "remote-inbound-rtp" && remote.kind === "video" + ? remote + : null; } export async function collectConnectionMetrics( connection: RTCPeerConnection, direction: "send" | "receive", previous: StatsAccumulator, + selector: StatsMediaSelector | null = null, ): Promise { const report = await connection.getStats(); - const pair = selectedCandidatePair(report); - const localCandidate = getRecord( + const media = mediaRecord(report, direction, selector); + const transport = transportRecord(report, media); + let pair = selectedCandidatePair(report, transport); + let localCandidate = getRecord( report, stringValue(pair, "localCandidateId"), ); - const remoteCandidate = getRecord( + let remoteCandidate = getRecord( report, stringValue(pair, "remoteCandidateId"), ); + if ( + localCandidate?.type !== "local-candidate" || + remoteCandidate?.type !== "remote-candidate" + ) { + pair = null; + localCandidate = null; + remoteCandidate = null; + } const localType = stringValue(localCandidate, "candidateType"); const remoteType = stringValue(remoteCandidate, "candidateType"); const path = @@ -152,29 +235,54 @@ export async function collectConnectionMetrics( ? "direct" : "unknown"; - const media = mediaRecord(report, direction); const mediaId = media?.id ?? null; - const sameMedia = mediaId !== null && mediaId === previous.mediaId; - const remoteInbound = direction === "send" ? remoteInboundVideo(report) : null; + const ssrc = numberValue(media, "ssrc"); + const trackIdentifier = media + ? mediaTrackIdentifier(report, media, direction) + : null; + const sameMedia = + mediaId !== null && + mediaId === previous.mediaId && + ssrc === previous.ssrc && + trackIdentifier === previous.trackIdentifier; + const remoteInbound = + direction === "send" ? linkedRemoteInbound(report, media) : null; const bytesKey = direction === "send" ? "bytesSent" : "bytesReceived"; const bytes = numberValue(media, bytesKey); const framesKey = direction === "send" ? "framesEncoded" : "framesDecoded"; const frames = numberValue(media, framesKey); - const timestamp = media?.timestamp ?? null; + const timestamp = numberValue(media, "timestamp"); const framesEncoded = numberValue(media, "framesEncoded"); const framesDecoded = numberValue(media, "framesDecoded"); const totalEncodeTime = numberValue(media, "totalEncodeTime"); const totalDecodeTime = numberValue(media, "totalDecodeTime"); + const framesDropped = numberValue(media, "framesDropped"); + const freezeCount = numberValue(media, "freezeCount"); + const totalFreezesDuration = numberValue(media, "totalFreezesDuration"); + const retransmittedPackets = numberValue( + media, + direction === "send" + ? "retransmittedPacketsSent" + : "retransmittedPacketsReceived", + ); + const retransmittedBytes = numberValue( + media, + direction === "send" + ? "retransmittedBytesSent" + : "retransmittedBytesReceived", + ); let bitrateKbps: number | null = null; let derivedFps: number | null = null; - - if ( + const sampleWindowMs = sameMedia && timestamp !== null && previous.timestamp !== null && timestamp > previous.timestamp - ) { - const elapsedSeconds = (timestamp - previous.timestamp) / 1_000; + ? timestamp - previous.timestamp + : null; + + if (sampleWindowMs !== null) { + const elapsedSeconds = sampleWindowMs / 1_000; if (bytes !== null && previous.bytes !== null && bytes >= previous.bytes) { bitrateKbps = ((bytes - previous.bytes) * 8) / elapsedSeconds / 1_000; } @@ -190,26 +298,58 @@ export async function collectConnectionMetrics( direction === "send" ? intervalAverageMs( totalEncodeTime, - sameMedia ? previous.previousTotalEncodeTime : null, + sampleWindowMs !== null ? previous.previousTotalEncodeTime : null, framesEncoded, - sameMedia ? previous.frames : null, + sampleWindowMs !== null ? previous.frames : null, ) : null; const intervalDecodeMs = direction === "receive" ? intervalAverageMs( totalDecodeTime, - sameMedia ? previous.previousTotalDecodeTime : null, + sampleWindowMs !== null ? previous.previousTotalDecodeTime : null, framesDecoded, - sameMedia ? previous.frames : null, + sampleWindowMs !== null ? previous.frames : null, ) : null; + const intervalFramesDropped = intervalDelta( + framesDropped, + previous.previousFramesDropped, + sampleWindowMs !== null, + ); + const intervalFreezeCount = intervalDelta( + freezeCount, + previous.previousFreezeCount, + sampleWindowMs !== null, + ); + const freezeDurationDelta = intervalDelta( + totalFreezesDuration, + previous.previousTotalFreezesDuration, + sampleWindowMs !== null, + ); + const intervalRetransmittedPackets = intervalDelta( + retransmittedPackets, + previous.previousRetransmittedPackets, + sampleWindowMs !== null, + ); + const intervalRetransmittedBytes = intervalDelta( + retransmittedBytes, + previous.previousRetransmittedBytes, + sampleWindowMs !== null, + ); previous.mediaId = mediaId; + previous.ssrc = ssrc; + previous.trackIdentifier = trackIdentifier; previous.bytes = bytes; previous.frames = frames; previous.timestamp = timestamp; previous.previousTotalEncodeTime = totalEncodeTime; previous.previousTotalDecodeTime = totalDecodeTime; + previous.previousFramesDropped = framesDropped; + previous.previousFreezeCount = freezeCount; + previous.previousTotalFreezesDuration = totalFreezesDuration; + previous.previousRetransmittedPackets = retransmittedPackets; + previous.previousRetransmittedBytes = retransmittedBytes; const codec = getRecord(report, stringValue(media, "codecId")); const width = numberValue(media, "frameWidth"); @@ -222,6 +362,13 @@ export async function collectConnectionMetrics( return { ...EMPTY_METRICS, + sampleTimestampMs: timestamp, + sampleWindowMs, + rtpStatsId: mediaId, + rtpSsrc: ssrc, + rtpMid: stringValue(media, "mid"), + trackIdentifier, + selectedCandidatePairId: pair?.id ?? null, path, iceProtocol, localRelayProtocol, @@ -247,7 +394,13 @@ export async function collectConnectionMetrics( ? numberValue(direction === "send" ? remoteInbound : media, "jitter")! * 1_000 : null, - framesDropped: numberValue(media, "framesDropped"), + framesDropped, + intervalFramesDropped, + intervalFreezeCount, + intervalFreezeDurationMs: + freezeDurationDelta === null ? null : freezeDurationDelta * 1_000, + intervalRetransmittedPackets, + intervalRetransmittedBytes, codec: stringValue(codec, "mimeType"), encoderImplementation: stringValue(media, "encoderImplementation"), powerEfficientEncoder: booleanValue(media, "powerEfficientEncoder"), diff --git a/tests/client-state.test.ts b/tests/client-state.test.ts index 841d4eb..d8b2857 100644 --- a/tests/client-state.test.ts +++ b/tests/client-state.test.ts @@ -266,6 +266,237 @@ describe("client signaling recovery policy", () => { }); describe("WebRTC stats parsing", () => { + it("binds media, remote evidence, transport, and pair without guessing", async () => { + const entry = ( + id: string, + type: string, + values: Record, + ): [string, Record] => [ + id, + { id, type, timestamp: 2_000, ...values }, + ]; + const report = new Map>([ + entry("transport-a", "transport", { + selectedCandidatePairId: "pair-a", + }), + entry("transport-z", "transport", { + selectedCandidatePairId: "pair-z", + }), + entry("pair-a", "candidate-pair", { + transportId: "transport-a", + localCandidateId: "local-a", + remoteCandidateId: "remote-candidate-a", + currentRoundTripTime: 0.02, + }), + entry("pair-z", "candidate-pair", { + transportId: "transport-z", + localCandidateId: "local-z", + remoteCandidateId: "remote-candidate-z", + }), + entry("local-a", "local-candidate", { + candidateType: "host", + protocol: "udp", + }), + entry("remote-candidate-a", "remote-candidate", { + candidateType: "srflx", + protocol: "udp", + }), + entry("local-z", "local-candidate", { + candidateType: "relay", + protocol: "tcp", + }), + entry("remote-candidate-z", "remote-candidate", { + candidateType: "relay", + protocol: "tcp", + }), + entry("outbound-a", "outbound-rtp", { + kind: "video", + ssrc: 111, + mid: "0", + transportId: "transport-a", + mediaSourceId: "source-a", + remoteId: "remote-inbound-a", + bytesSent: 2_000, + framesEncoded: 60, + }), + entry("outbound-z", "outbound-rtp", { + kind: "video", + ssrc: 999, + transportId: "transport-z", + mediaSourceId: "source-z", + remoteId: "remote-inbound-z", + bytesSent: 99_000, + framesEncoded: 99, + }), + entry("source-a", "media-source", { + kind: "video", + trackIdentifier: "capture-track-a", + }), + entry("source-z", "media-source", { + kind: "video", + trackIdentifier: "capture-track-z", + }), + entry("remote-inbound-a", "remote-inbound-rtp", { + kind: "video", + packetsLost: 2, + jitter: 0.004, + }), + entry("remote-inbound-z", "remote-inbound-rtp", { + kind: "video", + packetsLost: 999, + jitter: 0.9, + }), + ]); + const connection = { + getStats: async () => report as unknown as RTCStatsReport, + } as unknown as RTCPeerConnection; + + const accumulator = createStatsAccumulator(); + const metrics = await collectConnectionMetrics( + connection, + "send", + accumulator, + { trackIdentifier: "capture-track-a" }, + ); + + expect(metrics).toMatchObject({ + sampleTimestampMs: 2_000, + sampleWindowMs: null, + rtpStatsId: "outbound-a", + rtpSsrc: 111, + rtpMid: "0", + trackIdentifier: "capture-track-a", + selectedCandidatePairId: "pair-a", + path: "direct", + iceProtocol: "udp", + packetsLost: 2, + jitterMs: 4, + rttMs: 20, + }); + + report.get("transport-a")!.selectedCandidatePairId = "pair-z"; + const wrongTransportPair = await collectConnectionMetrics( + connection, + "send", + accumulator, + { trackIdentifier: "capture-track-a" }, + ); + expect(wrongTransportPair).toMatchObject({ + rtpStatsId: "outbound-a", + selectedCandidatePairId: null, + path: "unknown", + }); + + report.get("transport-a")!.selectedCandidatePairId = "pair-a"; + report.get("local-a")!.type = "remote-candidate"; + report.get("remote-candidate-a")!.type = "local-candidate"; + const malformedCandidateReferences = await collectConnectionMetrics( + connection, + "send", + accumulator, + { trackIdentifier: "capture-track-a" }, + ); + expect(malformedCandidateReferences).toMatchObject({ + selectedCandidatePairId: null, + path: "unknown", + localCandidateType: null, + remoteCandidateType: null, + }); + report.get("local-a")!.type = "local-candidate"; + report.get("remote-candidate-a")!.type = "remote-candidate"; + + delete report.get("outbound-a")!.transportId; + const unboundTransport = await collectConnectionMetrics( + connection, + "send", + accumulator, + { trackIdentifier: "capture-track-a" }, + ); + expect(unboundTransport).toMatchObject({ + rtpStatsId: "outbound-a", + selectedCandidatePairId: null, + path: "unknown", + }); + report.get("outbound-a")!.transportId = "transport-a"; + + report.set("outbound-a-layer-2", { + id: "outbound-a-layer-2", + type: "outbound-rtp", + timestamp: 3_000, + kind: "video", + ssrc: 112, + transportId: "transport-a", + mediaSourceId: "source-a", + bytesSent: 3_000, + framesEncoded: 90, + }); + const ambiguous = await collectConnectionMetrics( + connection, + "send", + accumulator, + { trackIdentifier: "capture-track-a" }, + ); + expect(ambiguous).toMatchObject({ + rtpStatsId: null, + selectedCandidatePairId: null, + packetsLost: null, + }); + }); + + it("rebases sender retransmission deltas after counter rollback", async () => { + const outbound = ( + timestamp: number, + retransmittedPacketsSent: number, + retransmittedBytesSent: number, + ) => + new Map([ + [ + "outbound", + { + id: "outbound", + type: "outbound-rtp", + timestamp, + kind: "video", + ssrc: 101, + bytesSent: timestamp * 10, + framesEncoded: timestamp / 10, + retransmittedPacketsSent, + retransmittedBytesSent, + }, + ], + ]) as unknown as RTCStatsReport; + const reports = [ + outbound(1_000, 2, 200), + outbound(2_000, 5, 800), + outbound(3_000, 1, 100), + outbound(4_000, 4, 700), + ]; + const connection = { + getStats: async () => reports.shift()!, + } as unknown as RTCPeerConnection; + const accumulator = createStatsAccumulator(); + const sample = () => + collectConnectionMetrics(connection, "send", accumulator); + const first = await sample(); + const stable = await sample(); + const reset = await sample(); + const afterReset = await sample(); + + expect(first.intervalRetransmittedPackets).toBeNull(); + expect(stable).toMatchObject({ + intervalRetransmittedPackets: 3, + intervalRetransmittedBytes: 600, + }); + expect(reset).toMatchObject({ + intervalRetransmittedPackets: null, + intervalRetransmittedBytes: null, + }); + expect(afterReset).toMatchObject({ + intervalRetransmittedPackets: 3, + intervalRetransmittedBytes: 600, + }); + }); + it("separates the local TURN transport from the ICE protocol", async () => { const report = new Map>([ [ @@ -283,6 +514,7 @@ describe("WebRTC stats parsing", () => { id: "pair", type: "candidate-pair", timestamp: 2_000, + transportId: "transport", state: "succeeded", nominated: true, localCandidateId: "local", @@ -317,6 +549,7 @@ describe("WebRTC stats parsing", () => { type: "inbound-rtp", timestamp: 2_000, kind: "video", + transportId: "transport", bytesReceived: 2_000, framesDecoded: 100, totalDecodeTime: 0.5, @@ -329,6 +562,7 @@ describe("WebRTC stats parsing", () => { type: "inbound-rtp", timestamp: 4_000, kind: "video", + transportId: "transport", bytesReceived: 4_000, framesDecoded: 150, totalDecodeTime: 1.5, @@ -379,6 +613,7 @@ describe("WebRTC stats parsing", () => { id: "pair", type: "candidate-pair", timestamp: 1_000, + transportId: "transport", localCandidateId: "local", remoteCandidateId: "remote", state: "succeeded", @@ -405,6 +640,18 @@ describe("WebRTC stats parsing", () => { protocol: "udp", }, ], + [ + "inbound", + { + id: "inbound", + type: "inbound-rtp", + timestamp: 1_000, + kind: "video", + transportId: "transport", + bytesReceived: 1_000, + framesDecoded: 30, + }, + ], ]) as unknown as RTCStatsReport; const connection = { getStats: async () => report, @@ -502,4 +749,105 @@ describe("WebRTC stats parsing", () => { }); expect(afterCounterReset.intervalDecodeMs).toBeCloseTo(20); }); + + it("rebases receiver event deltas on SSRC changes and counter resets", async () => { + const inbound = ( + timestamp: number, + ssrc: number, + framesDropped: number, + freezeCount: number, + totalFreezesDuration: number, + retransmittedPacketsReceived: number, + retransmittedBytesReceived: number, + ) => + new Map([ + [ + "inbound", + { + id: "inbound", + type: "inbound-rtp", + timestamp, + kind: "video", + ssrc, + bytesReceived: timestamp * 10, + framesDecoded: timestamp / 10, + framesDropped, + freezeCount, + totalFreezesDuration, + retransmittedPacketsReceived, + retransmittedBytesReceived, + }, + ], + ]) as unknown as RTCStatsReport; + const reports = [ + inbound(1_000, 101, 2, 1, 0.25, 3, 300), + inbound(2_000, 101, 5, 2, 0.75, 7, 900), + inbound(3_000, 202, 20, 8, 4, 30, 4_000), + inbound(4_000, 202, 22, 9, 4.25, 33, 4_600), + inbound(5_000, 202, 1, 0, 0.25, 2, 200), + inbound(6_000, 202, 4, 1, 0.5, 6, 1_000), + ]; + const connection = { + getStats: async () => reports.shift()!, + } as unknown as RTCPeerConnection; + const accumulator = createStatsAccumulator(); + const sample = () => + collectConnectionMetrics(connection, "receive", accumulator); + const first = await sample(); + const stable = await sample(); + const changedSsrc = await sample(); + const afterSsrcChange = await sample(); + const reset = await sample(); + const afterReset = await sample(); + + expect(first).toMatchObject({ + sampleWindowMs: null, + intervalFramesDropped: null, + intervalFreezeCount: null, + intervalFreezeDurationMs: null, + intervalRetransmittedPackets: null, + intervalRetransmittedBytes: null, + }); + expect(stable).toMatchObject({ + sampleWindowMs: 1_000, + intervalFramesDropped: 3, + intervalFreezeCount: 1, + intervalFreezeDurationMs: 500, + intervalRetransmittedPackets: 4, + intervalRetransmittedBytes: 600, + }); + expect(changedSsrc).toMatchObject({ + rtpSsrc: 202, + sampleWindowMs: null, + intervalFramesDropped: null, + intervalFreezeCount: null, + intervalFreezeDurationMs: null, + intervalRetransmittedPackets: null, + intervalRetransmittedBytes: null, + }); + expect(afterSsrcChange).toMatchObject({ + sampleWindowMs: 1_000, + intervalFramesDropped: 2, + intervalFreezeCount: 1, + intervalFreezeDurationMs: 250, + intervalRetransmittedPackets: 3, + intervalRetransmittedBytes: 600, + }); + expect(reset).toMatchObject({ + sampleWindowMs: 1_000, + intervalFramesDropped: null, + intervalFreezeCount: null, + intervalFreezeDurationMs: null, + intervalRetransmittedPackets: null, + intervalRetransmittedBytes: null, + }); + expect(afterReset).toMatchObject({ + sampleWindowMs: 1_000, + intervalFramesDropped: 3, + intervalFreezeCount: 1, + intervalFreezeDurationMs: 250, + intervalRetransmittedPackets: 4, + intervalRetransmittedBytes: 800, + }); + }); }); diff --git a/tests/host-peer-tracks.test.ts b/tests/host-peer-tracks.test.ts index dfd8e8e..edc472b 100644 --- a/tests/host-peer-tracks.test.ts +++ b/tests/host-peer-tracks.test.ts @@ -9,10 +9,13 @@ const statsCallbacks: Array<() => void> = []; class FakeSender { failNextReplace = false; + deferReplaceCall: number | null = null; failNextSetParameters = false; deferNextSetParameters = false; readonly appliedMaxBitrates: Array = []; private parameters = { encodings: [{}] } as RTCRtpSendParameters; + private replaceCallCount = 0; + private releaseReplaceTrack: (() => void) | null = null; private releaseSetParameters: (() => void) | null = null; readonly setParameters = vi.fn( async (parameters: RTCRtpSendParameters) => { @@ -31,10 +34,17 @@ class FakeSender { }, ); readonly replaceTrack = vi.fn(async (track: MediaStreamTrack | null) => { + this.replaceCallCount += 1; if (this.failNextReplace) { this.failNextReplace = false; throw new Error("replaceTrack failed"); } + if (this.deferReplaceCall === this.replaceCallCount) { + this.deferReplaceCall = null; + await new Promise((resolve) => { + this.releaseReplaceTrack = resolve; + }); + } this.track = track; }); @@ -48,6 +58,11 @@ class FakeSender { this.releaseSetParameters?.(); this.releaseSetParameters = null; } + + releaseDeferredReplaceTrack(): void { + this.releaseReplaceTrack?.(); + this.releaseReplaceTrack = null; + } } class FakePeerConnection { @@ -117,12 +132,14 @@ function sendStatsReport({ timestamp, totalEncodeTime, qualityLimitationReason, + trackIdentifier = "video", }: { bytesSent: number; framesEncoded: number; timestamp: number; totalEncodeTime?: number; qualityLimitationReason: string; + trackIdentifier?: string; }): RTCStatsReport { return new Map>([ [ @@ -140,6 +157,7 @@ function sendStatsReport({ id: "pair", type: "candidate-pair", timestamp, + transportId: "transport", state: "succeeded", nominated: true, localCandidateId: "local", @@ -175,6 +193,8 @@ function sendStatsReport({ type: "outbound-rtp", timestamp, kind: "video", + transportId: "transport", + mediaSourceId: "video-source", bytesSent, framesEncoded, framesPerSecond: 30, @@ -186,6 +206,16 @@ function sendStatsReport({ codecId: "codec", }, ], + [ + "video-source", + { + id: "video-source", + type: "media-source", + timestamp, + kind: "video", + trackIdentifier, + }, + ], [ "codec", { @@ -202,6 +232,19 @@ function createTrack(kind: "video" | "audio", id: string): MediaStreamTrack { return { id, kind } as MediaStreamTrack; } +function createConfiguredVideoTrack( + id: string, + width: number, + height: number, + frameRate: number, +): MediaStreamTrack { + return { + id, + kind: "video", + getSettings: () => ({ width, height, frameRate }), + } as unknown as MediaStreamTrack; +} + function createStream( videoTrack: MediaStreamTrack, audioTrack: MediaStreamTrack | null, @@ -488,6 +531,173 @@ describe("HostPeer source replacement", () => { ).toBe(false); }); + it("publishes capture settings with outbound evidence from the same stats tick", async () => { + const getSettings = vi.fn(() => ({ + width: 1920, + height: 1080, + frameRate: 59.94, + })); + const videoTrack = { + id: "capture-video", + kind: "video", + getSettings, + } as unknown as MediaStreamTrack; + const updates: PeerSnapshot[] = []; + const peer = createPeer( + createStream(videoTrack, null), + (snapshot) => updates.push(snapshot), + ); + await expect(peer.start()).resolves.toBe(true); + getSettings.mockClear(); + const connection = FakePeerConnection.latest!; + connection.statsReports.push( + sendStatsReport({ + bytesSent: 1_000_000, + framesEncoded: 30, + timestamp: 1_234, + qualityLimitationReason: "none", + trackIdentifier: "capture-video", + }), + ); + + expect(getSettings).not.toHaveBeenCalled(); + statsCallbacks[0]!(); + await vi.waitFor(() => + expect(updates.at(-1)?.metrics.sampleTimestampMs).toBe(1_234), + ); + + expect(connection.getStats).toHaveBeenCalledOnce(); + expect(getSettings).toHaveBeenCalledOnce(); + expect(updates.at(-1)?.metrics).toMatchObject({ + sampleTimestampMs: 1_234, + rtpStatsId: "outbound-video", + resolution: "1280x720", + captureWidth: 1920, + captureHeight: 1080, + captureFramesPerSecond: 59.94, + }); + }); + + it.each([ + { + outcome: "succeeds", + rollback: false, + deferredCall: 1, + result: true, + committed: "next" as const, + }, + { + outcome: "rolls back", + rollback: true, + deferredCall: 2, + result: false, + committed: "old" as const, + }, + ])("blocks stale stats while replacement $outcome", async ({ + rollback, + deferredCall, + result, + committed, + }) => { + const scenario = rollback ? "rollback" : "success"; + let resolveOldStats!: (report: RTCStatsReport) => void; + const oldStats = new Promise((resolve) => { + resolveOldStats = resolve; + }); + const oldVideo = createConfiguredVideoTrack( + `${scenario}-old-capture`, + 1280, + 720, + 30, + ); + const nextVideo = createConfiguredVideoTrack( + `${scenario}-next-capture`, + 1920, + 1080, + 60, + ); + const oldAudio = rollback ? createTrack("audio", "old-audio") : null; + const nextAudio = rollback ? createTrack("audio", "next-audio") : null; + const updates: PeerSnapshot[] = []; + const peer = createPeer( + createStream(oldVideo, oldAudio), + (snapshot) => updates.push(snapshot), + ); + await expect(peer.start()).resolves.toBe(true); + const connection = FakePeerConnection.latest!; + const videoSender = connection.senders[0]!; + const audioSender = connection.senders[1]!; + connection.statsReports.push( + sendStatsReport({ + bytesSent: 500_000, + framesEncoded: 15, + timestamp: 500, + qualityLimitationReason: "none", + trackIdentifier: oldVideo.id, + }), + ); + statsCallbacks[0]!(); + await vi.waitFor(() => + expect(updates.at(-1)?.metrics.sampleTimestampMs).toBe(500), + ); + connection.statsReports.push(oldStats); + statsCallbacks[0]!(); + await vi.waitFor(() => expect(connection.getStats).toHaveBeenCalledTimes(2)); + + videoSender.deferReplaceCall = deferredCall; + audioSender.failNextReplace = rollback; + const replacing = peer.replaceStream( + createStream(nextVideo, nextAudio), + ); + await vi.waitFor(() => + expect(videoSender.replaceTrack).toHaveBeenCalledTimes(deferredCall), + ); + + resolveOldStats( + sendStatsReport({ + bytesSent: 1_000_000, + framesEncoded: 30, + timestamp: 1_000, + qualityLimitationReason: "none", + trackIdentifier: oldVideo.id, + }), + ); + await new Promise((resolve) => setImmediate(resolve)); + expect( + updates.some(({ metrics }) => metrics.sampleTimestampMs === 1_000), + ).toBe(false); + statsCallbacks[0]!(); + expect(connection.getStats).toHaveBeenCalledTimes(2); + + videoSender.releaseDeferredReplaceTrack(); + await expect(replacing).resolves.toBe(result); + const committedVideo = committed === "old" ? oldVideo : nextVideo; + expect(videoSender.track).toBe(committedVideo); + expect(updates.at(-1)?.metrics.sampleTimestampMs).toBe( + committed === "old" ? 500 : null, + ); + + connection.statsReports.push( + sendStatsReport({ + bytesSent: 2_000_000, + framesEncoded: 60, + timestamp: 2_000, + qualityLimitationReason: "none", + trackIdentifier: committedVideo.id, + }), + ); + statsCallbacks[0]!(); + await vi.waitFor(() => + expect(updates.at(-1)?.metrics.sampleTimestampMs).toBe(2_000), + ); + expect(updates.at(-1)?.metrics).toMatchObject({ + trackIdentifier: committedVideo.id, + captureWidth: committed === "old" ? 1280 : 1920, + captureHeight: committed === "old" ? 720 : 1080, + captureFramesPerSecond: committed === "old" ? 30 : 60, + }); + }); + it("explains a sustained browser quality limitation without changing settings", async () => { const updates: PeerSnapshot[] = []; const peer = createPeer( @@ -589,6 +799,7 @@ describe("ViewerRelay downstream ownership", () => { timestamp: 1_000, totalEncodeTime: 0.15, qualityLimitationReason: "cpu", + trackIdentifier: "metrics-video", }), sendStatsReport({ bytesSent: 1_500_000, @@ -596,6 +807,7 @@ describe("ViewerRelay downstream ownership", () => { timestamp: 2_000, totalEncodeTime: 0.75, qualityLimitationReason: "bandwidth", + trackIdentifier: "metrics-video", }), );