diff --git a/docs/project-memory.md b/docs/project-memory.md index 3dec2c6..0588c13 100644 --- a/docs/project-memory.md +++ b/docs/project-memory.md @@ -31,7 +31,7 @@ Last updated: 2026-08-21 - C+B quality reparenting is edge-local and cooldown-bound; no score, timer, or global parent penalty. - Flagship media is UDP; HTTPS/WSS stays TLS/TCP; the old release remains rollback-only. - Treat settings as ceilings and degradation as unclassified. Use correlated Host A+B/Viewer C and one-variable evidence; never force AV1, infer by UA, or create a composite score. -- Web window/system-audio requests are browser hints; tracks use `music`. P2P/relay/SFU senders request 128,000 bit/s and SFU disables DTX; this is a ceiling, not a quality/stereo guarantee. Native Win11 process audio stays source-only; one Viewer got 495 Opus packets. +- Screen audio is route-consistent stereo: Viewer `stereo=1`; SFU uses 128 kbps stereo/`forceStereo`, DTX off, RED retained. Other SDP stays browser-generated; congestion/FEC and stream remain. Production proof open; no audio UI; voice separate. Native audio source-only. - Presence reports exact active upstream and feeds Host topology; it does not prove media. Status and details stay neutral until local/current media evidence, then label P2P/SFU and actual TURN; append its protocol only from local relay stats. Web Host names stay socket/localStorage-only; Native remains outside the boundary. - Do not add scene detection, dynamic-FPS control, or forced AV1 without negotiation, encode, game, CPU/GPU, and sender evidence. - ADR-0002 grant/public access, the v3 room-password migration, and Web Host display names are deployed. Names/presence remain session-only without account or roster tables. diff --git a/docs/research/browser-screen-audio-quality.md b/docs/research/browser-screen-audio-quality.md index c726f50..8838227 100644 --- a/docs/research/browser-screen-audio-quality.md +++ b/docs/research/browser-screen-audio-quality.md @@ -2,7 +2,9 @@ Accessed: 2026-08-21 -Status: fixed 128 kbps send ceiling accepted; user-facing codec controls remain no-go. +Status: the route-consistent screen-media default is source-complete; production +still has a 128 kbps ceiling without an explicit P2P or SFU stereo contract, and +audible route evidence remains open. ## Scope And Decision @@ -11,7 +13,8 @@ browser `RTCPeerConnection`. It does not decide whether the product ultimately needs Windows per-application capture; that remains a separate native-sender requirement. -The Web sender keeps the current minimal behavior: +The product keeps one screen-media audio mode rather than exposing codec +plumbing as quality controls: - request audio with `audio: true`; - request window audio for a selected window and offer system audio for a full @@ -20,36 +23,45 @@ The Web sender keeps the current minimal behavior: - treat an audio track as optional and warn before publishing when none exists; - preserve returned audio through source changes, picture pause, P2P, relay, and configured SFU routes; -- apply a fixed `128000` bit/s ceiling to each browser audio sender at initial - binding and source replacement, and the same audio preset with DTX disabled - when publishing through the pinned LiveKit SDK; -- expose no channel-count, sample-rate, Opus bitrate, stereo, DTX, or FEC user - control; and +- keep a `128000` bit/s sender ceiling, Opus stereo receive preference and DTX + off consistently across Web peer and SFU routes; +- let the browser's existing WebRTC congestion control reduce actual audio and + video traffic; do not add an application audio adaptation loop; +- expose no channel-count, sample-rate, codec, bitrate, stereo, DTX, RED, or FEC + user control; and - expose local, read-only audio RTP diagnostics without treating negotiated codec fields as source-quality facts. The music hint is metadata, not a codec or quality mode. Chromium accepts and reads it back, but the inspected libwebrtc audio-track interface has no matching content-hint input to the Opus encoder. Screener therefore does not credit the -hint with a quality change. The fixed ceiling is a measured product default, -not a user control or a reason to add application-owned SDP munging. Requested, -applied, negotiated, and observed states remain separate. +hint with a quality change. RFC 7587 and the pinned LiveKit presets, rather than +an ordinary-PC benchmark, define the 128 kbps ceiling: the RFC places full-band +stereo music in a 64--128 kbps sweet spot, and LiveKit 2.22.0 names 128 kbps +`musicHighQualityStereo`. Requested, applied, negotiated, and observed states +remain separate. + +There is no audio setting or panel rename in this slice. Screen capture has one +reasonable media default, while a future microphone/voice feature would be a +separate track and processing path with its own AEC/noise-suppression/DTX +contract. It must not turn movie or game audio into a voice-processed source. ## Deployed Reports And Echo Boundary -Users report poor movie/video audio and delayed self-echo when production -`769de201f7cc` captures whole-system audio while a voice application is active. -These reports apply to that deployed release and remain unclassified until the -same fixture compares it with current `main`. The release sets no application -audio bitrate, channel, sample-rate, or SDP option; its Host preview is muted, -and peer/SFU relay is disabled. Therefore the current evidence does not point -to an application ceiling, Host preview playback, or relay re-encoding. +Users report that current production +`6ccb516a47261054f91dfa2fafa408d39ced59fc` still sounds poor for movie/video +screen audio. That exact release has the 128 kbps ceiling described +above but no explicit stereo contract on peer or SFU routes, making mono +negotiation the strongest code-level candidate. It remains a diagnosis, not an +audible-quality measurement; capture scope, actual fmtp/bitrate, loss, jitter +and concealment still need the existing read-only diagnostics. -The echo case is source-content leakage: a viewer's voice is rendered by a -separate voice application on the Host, enters the system mix, and is captured -and sent back. It is not ordinary microphone acoustic echo. AEC, noise -suppression, or post-mix Web Audio cannot recover the originating process and -must not be presented as a fix. +The earlier delayed self-echo report on `769de201f7cc` has a different boundary: +a viewer's voice is rendered by a separate voice application on the Host, +enters the system mix, and is captured and sent back. It is source-content +leakage, not ordinary microphone acoustic echo. AEC, noise suppression, or +post-mix Web Audio cannot recover the originating process and must not be +presented as a fix. On 2026-08-21 the production picker was user-verified to keep tab and window audio scoped as intended in that browser environment. Current source requests @@ -85,21 +97,17 @@ constraints added to `getDisplayMedia()`. | Capture sample rate | The Screen Capture specification does not list generic `sampleRate` as applicable to display audio | `track.getSettings().sampleRate` may be present in an implementation | The Opus mode, RTP clock semantics, receiver output rate, or end-to-end fidelity | Observe only in a future diagnostic; absent means unknown | | RTP send bitrate | `RTCRtpSender.setParameters()` can set `encodings[].maxBitrate` for audio | A following `getParameters()` can show the applied ceiling; outbound byte deltas show actual traffic | A target, minimum, audible improvement, or Opus `maxaveragebitrate`; other limits may keep traffic lower | Request a fixed 128 kbps ceiling, read it back, and retain actual bitrate as the acceptance fact | | Codec | The user agent chooses among negotiated send codecs unless a separately negotiated codec selection is available | `RTCCodecStats.mimeType` and `sdpFmtpLine` identify the codec and negotiated format parameters in use | That Opus was selected before stats exist, or that negotiated preferences describe actual content | Label only observed negotiated data; do not force a codec | -| Stereo | No stable sender parameter controls Opus stereo | Capture settings and negotiated codec/fmtp data can be inspected separately | Capture channel count does not prove encoded stereo; `opus/48000/2` does not prove stereo content | No control and no inferred stereo badge | -| DTX | The current WebRTC `RTCRtpEncodingParameters` dictionary has no `dtx` member | No portable standard sender readback proves DTX operation | A deprecated, non-standard browser field or SDK option is not a cross-browser contract | No control | +| Stereo | No stable sender parameter controls Opus stereo; RFC 7587 defines `stereo` as the receiver's one-way preference | Capture settings and negotiated codec/fmtp data can be inspected separately | Capture channel count and `opus/48000/2` do not prove encoded stereo | Use one bounded Opus answer transform on the declared Chrome/Edge baseline; no user control or inferred badge | +| DTX | The current WebRTC `RTCRtpEncodingParameters` dictionary has no `dtx` member; RFC 7587 defaults absent `usedtx` to `0` | No portable standard sender readback proves DTX operation | A deprecated browser field or SDK option is not a cross-browser contract | Keep peer default-off and set LiveKit `dtx: false`; no user control | | FEC | There is no direct, portable audio FEC on/off sender parameter | Negotiated codec/fmtp data and inbound `fecPacketsReceived`/`fecBytesReceived` may provide evidence after use | Negotiation does not prove recovery occurred; zero counters do not prove FEC was disabled | No control; retain stats only as future diagnostic evidence | The WebRTC specification defines audio `maxBitrate` as a maximum and allows -other limits to constrain the sender further. A narrow Chrome 151 local -loopback on 2026-08-21 used a complex synthetic 48 kHz, two-channel source and -`contentHint = "music"`. With no sender ceiling, the offer selected -`audio/opus` with `minptime=10;useinbandfec=1` and sent 32.21 kbps over the -four-second measurement window. The same fixture with only -`encodings[0].maxBitrate = 128000` read back `128000` and sent 127.91 kbps; -the offer and fmtp were unchanged. This establishes that the standard setter -materially changed Chrome's encoder budget for that fixture. It does not prove -an audible improvement, stereo, a target bitrate, or equivalent behavior in -another browser or under congestion. +other limits to constrain the sender further. A retained Chrome 151 loopback +showed only that the browser applied `maxBitrate` and changed its encoder budget +for that synthetic fixture; the ordinary PC, short interval and synthetic +source are not calibration evidence. RFC 7587's full-band stereo music range +and the pinned LiveKit preset justify the product value. Audible quality, +actual traffic and congestion behavior remain runtime observations. ## Capture And Browser Compatibility @@ -159,12 +167,22 @@ Chrome observation, whose ordinary offer had no stereo fmtp and sent about 32 kbps. It is an implementation-level explanation for the tested default, not a portable stereo or bitrate contract. +This makes the current poor movie/music report unsurprising. Production and the +pre-slice source set a 128 kbps sender ceiling, but their Web peer negotiation +does not ask the receiving Viewer to accept stereo. The SFU publisher sets the +same ceiling and `dtx: false`, but its options merge with LiveKit 2.22.0 room +defaults, including `forceStereo: false` and `red: true`; an explicit false +overrides LiveKit's track-settings stereo heuristic. The strongest current +code-level cause is therefore a mono-by-default contract, not evidence that +WebRTC congestion control is broken. Capture/source loss, packet loss, jitter +and concealment can still degrade audio and remain separate diagnostic facts. + ## Reference Implementations No code was copied. All repositories were inspected at pinned commits on 2026-08-19 and their cited source pages were rechecked on 2026-08-21. -- LiveKit client-sdk-js, Apache-2.0, commit +- LiveKit client-sdk-js 2.22.0, Apache-2.0, commit [`0a2110d`](https://github.com/livekit/client-sdk-js/tree/0a2110d39904a06722a0c4d1ddbb9390bb06ad4d): it treats [`getSettings().channelCount` or requested constraints](https://github.com/livekit/client-sdk-js/blob/0a2110d39904a06722a0c4d1ddbb9390bb06ad4d/src/room/participant/LocalParticipant.ts#L899-L920) @@ -172,10 +190,12 @@ No code was copied. All repositories were inspected at pinned commits on `RTCRtpEncodingParameters.maxBitrate`, but stereo/DTX/RED also travel in LiveKit signaling. Its transport then [munges Opus bitrate fmtp](https://github.com/livekit/client-sdk-js/blob/0a2110d39904a06722a0c4d1ddbb9390bb06ad4d/src/room/PCTransport.ts#L269-L333) - in SDP. The pinned 2.22.0 SDK models `audioPreset` as a `maxBitrate`, defaults - music to 48 kbps, and keeps `forceStereo` separate. Screener uses its - supported preset surface at 128 kbps with DTX disabled and leaves stereo to - actual negotiation. + in SDP. Its room defaults are music 48 kbps, DTX on, RED on and + `forceStereo: false`; its named presets include stereo music at 64 kbps and + high-quality stereo music at 128 kbps. Per-track options merge after those + defaults. Its stereo-only `??=` fallback disables DTX/RED only if still + undefined, which the pinned room defaults are not. Screener's pre-slice source + overrode DTX but not `forceStereo` or RED, yielding mono, DTX off and RED on. - lib-jitsi-meet, Apache-2.0, commit [`63a04ec`](https://github.com/jitsi/lib-jitsi-meet/tree/63a04ecabd972ea75e877f9ba12086c13cb68210): [`mungeOpus()`](https://github.com/jitsi/lib-jitsi-meet/blob/63a04ecabd972ea75e877f9ba12086c13cb68210/modules/RTC/TPCUtils.ts#L871-L929) @@ -188,9 +208,18 @@ No code was copied. All repositories were inspected at pinned commits on [`sdpTransform`](https://github.com/feross/simple-peer/blob/f1a492d1999ce727fa87193ebdea20ac89c1fc6d/index.js#L608-L664) hook for SDP changes. -The Apache and MIT implementations support the same conclusion: a library can -offer product-specific audio modes, but the stereo/DTX/FEC/Opus-fmtp portion is -not equivalent to a portable browser sender control. +The Apache and MIT implementations support the same conclusion: one narrow, +structured Opus fmtp transform is established practice, but it is still an +application-owned negotiation contract rather than a portable sender setter. +`sdp-transform` 2.15.0 is MIT-licensed and already arrives transitively through +LiveKit. Runtime code that imports it must declare it directly, together with +its TypeScript types, rather than depend on LiveKit's dependency graph. +As supplementary practice evidence, historical +[LiveKit server issue 970](https://github.com/livekit/livekit/issues/970) and +the 2026 [Rust SDK issue 1018](https://github.com/livekit/rust-sdks/issues/1018) +both report stereo sources arriving as mono when the product/SDK did not signal +stereo. Issue reports are not treated as normative behavior or as proof of +Screener's runtime state; pinned source and local stats remain authoritative. Discord and Oopz have a narrower evidence boundary: @@ -202,11 +231,16 @@ Discord and Oopz have a narrower evidence boundary: audio parameters for each consumer Go Live session. This is useful native product evidence, not a preset Screener can copy or an excuse to infer codec state without stats. -- The retained read-only - [Oopz 0.87.425 package inspection](./native-shared-encode-sender.md) found a native Agora - RTC library, while its public help center does not publish screen-share audio - codec, bitrate, stereo, DTX, or FEC settings. Those artifacts do not prove - the codec or parameters used by a live Oopz session. +- Oopz's official help separates program-audio screen sharing from voice + channels: it documents Windows program-audio capture, while separate voice + articles expose channel quality/latency choices and microphone AI noise + suppression. The retained read-only + [Oopz 0.87.425 package inspection](./native-shared-encode-sender.md) found a + native Agora RTC library, program/system screen-audio switches, and Agora + profile symbols including high-quality stereo music and game-streaming + scenarios. This proves only separate native capture/voice/profile surfaces. + Neither the package nor public help proves the active screen-share codec, + bitrate, stereo, DTX, FEC or Agora profile; those values remain unknown. ## Honest Diagnostics Contract @@ -228,6 +262,73 @@ Audio diagnostics keep four layers separate: Never substitute `48000`, `2`, `false`, or `0` for an unavailable field. +## Minimal Route-Consistent Runtime Slice + +The source-complete runtime slice changes negotiation, not capture or congestion +control: + +1. Keep `contentHint = "music"`, one audio track and the existing 128 kbps + sender ceiling. Do not add Web Audio mixing, resampling, a second audio + representation or a new rate controller. +2. On every Web Viewer answer, including ICE restart/rebuild answers, parse SDP + with direct dependencies on MIT-licensed `sdp-transform` 2.15.0 and its + TypeScript declarations. Select the single non-rejected audio media section, + resolve its Opus payload from structured `rtp` entries, and idempotently + upsert only `stereo=1` in that payload's fmtp before both + `setLocalDescription()` and signaling the same description. Preserve every + other media section, codec, fmtp key and session attribute. Empty SDP, no + unambiguous active audio/Opus payload, multiple active audio sections, + malformed fmtp, or parser/write failure returns the original answer object + and SDP unchanged. This keeps a browser-generated mono/video answer usable; + no broad regex or malformed partial answer is allowed. RFC 7587 makes stereo + an optional receive preference, while W3C guarantees the unmodified + `createAnswer()` remains usable by `setLocalDescription()`. +3. Do not add `sprop-stereo`: RFC 7587 defines it as a sender-likelihood hint, + not the receiver preference that permits the remote encoder to send stereo, + and offer/answer parameters are orthogonal. Do not add `usedtx=0` or + `maxaveragebitrate=128000` either: absent `usedtx` already means off, absent + `maxaveragebitrate` already uses the corresponding Opus mode's recommended + maximum, and `RTCRtpSender.maxBitrate` remains Screener's send ceiling. Keep + the browser's existing `useinbandfec` negotiation untouched. +4. For SFU publication use LiveKit 2.22.0 + `AudioPresets.musicHighQualityStereo`, `forceStereo: true` and `dtx: false`. + Do not override the pinned room default `red: true`. RED and Opus in-band FEC + are distinct repair mechanisms; retaining one neither proves, replaces nor + disables the other, and no FEC switch is exposed or reimplemented. + +Applying the receive preference in every browser answer covers direct P2P, +browser-relay children and selected-edge TURN because TURN changes the ICE +transport, not the peer SDP contract. SFU uses the equivalent SDK contract. +An SFU-only change is not acceptable: moving between peer and SFU routes would +otherwise change channel behavior. Native/Pion sender routes retain their own +explicit encoded-track contract and need a focused compatibility check before +claiming the same result. + +At the fixed ceiling, audio contributes at most 128 kbps RTP payload per active +outbound edge, or 256 kbps for the current two-child endpoint cap, excluding +RTP/SRTP/UDP/IP overhead. A Host SFU publication contributes one such ingress; +each root/peer forward still sends its own copy. This is small beside the +current 3--8 Mbps video presets, but stereo can use more encoder work and +bandwidth than mono. No second track or representation is added, and browser +relays still make no shared-encoder promise. + +Preserve the existing persistent Viewer `MediaStream`/media element and add or +remove its audio/video tracks in place. Introducing a separate audio element or +Web Audio graph would create a new clock/buffering boundary and is outside this +fix; the stable stream already deployed is the A/V synchronization baseline. + +The implementation budget is deliberately bounded to dependency +manifest/lockfile entries, one SDP helper, `ViewerPeer` answer wiring, SFU +publisher options, focused unit/integration tests and the owning docs. Parser +tests cover CRLF serialization, an existing/missing fmtp line, unknown fmtp +preservation, idempotence, payload-number lookup, rejected/non-audio sections, +and identity-preserving fallback for empty SDP, no/ambiguous Opus and parser +exceptions. Peer tests prove the exact selected answer is both applied and +signaled on initial/restart/rebuild paths; SFU tests assert the three explicit +options and absence of a RED override. One Chrome/Edge left/right fixture checks +channel correctness and a P2P/SFU route switch checks the persistent stream; +neither is a performance benchmark or parameter-calibration gate. + ## Native Windows Process-Audio Candidate The smallest honest native path is Windows WASAPI application loopback through @@ -307,24 +408,25 @@ only one post-SFU connectivity smoke. Correlate capture settings, negotiated cod actual outbound/inbound bitrate, loss, jitter, concealment, jitter buffer, and A/V playout timing using a distinguishable stereo fixture plus game/film audio. -## Revisit Gates +## UI And Voice Revisit Gate -Add a user-facing audio setting only when all of these are true: +Do not rename “高级视频设置” or add an audio control in this slice. One screen +media default is simpler and the browser already adapts actual traffic below +the ceiling. Revisit a single bounded “省流/高保真” choice only if sanitized +production stats establish that 128 kbps audio materially blocks a real target +network while video adaptation cannot preserve the core path. Do not expose +sample rate, channel count, codec, DTX, RED, FEC or an arbitrary bitrate slider. -1. A real Windows game/system-audio matrix identifies a reproducible user - problem and a desired outcome, such as a measured bandwidth reduction. -2. The setting uses a standards-track API without SDP munging or private - signaling and is supported across the declared sender-browser matrix. -3. The app can read back the applied value and separately observe the - negotiated codec and actual traffic or recovery behavior. -4. A controlled audible stereo/frequency fixture and packet-loss test shows - the intended result without silence, channel collapse, or route-specific - divergence. +If microphone voice enters scope, treat it as a separate source/track with an +independent privacy, AEC, noise-suppression, gain and DTX design. Oopz's public +product surfaces support that separation but do not supply parameters that can +be copied into Screener's screen-media path. ## Primary Sources - [W3C Screen Capture](https://www.w3.org/TR/screen-capture/) - [W3C WebRTC](https://www.w3.org/TR/webrtc/) +- [W3C WebRTC Priority Control](https://www.w3.org/TR/webrtc-priority/) - [W3C WebRTC Statistics](https://www.w3.org/TR/webrtc-stats/) - [W3C Web Audio](https://www.w3.org/TR/webaudio-1.1/) - [W3C WebCodecs Opus registration](https://www.w3.org/TR/webcodecs-opus-codec-registration/) @@ -333,11 +435,18 @@ Add a user-facing audio setting only when all of these are true: - [Chromium `MediaStreamTrack` content hint](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/renderer/modules/mediastream/media_stream_track_impl.cc) - [Chromium content-capture audio defaults](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/content/renderer/media/stream/media_stream_constraints_util_audio_unittest.cc) - [LiveKit 2.22.0 track options](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/track/options.ts) +- [LiveKit 2.22.0 publish defaults](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/defaults.ts) +- [LiveKit 2.22.0 local publication](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/participant/LocalParticipant.ts) +- [LiveKit 2.22.0 package dependencies](https://github.com/livekit/client-sdk-js/blob/v2.22.0/package.json) +- [Jitsi Opus SDP transform, pinned commit](https://github.com/jitsi/lib-jitsi-meet/blob/63a04ecabd972ea75e877f9ba12086c13cb68210/modules/RTC/TPCUtils.ts) +- [`sdp-transform` 2.15.0](https://github.com/clux/sdp-transform/tree/v2.15.0) - [RFC 7587: RTP Payload Format for Opus](https://www.rfc-editor.org/rfc/rfc7587.html) - [RFC 7874: WebRTC Audio Codec and Processing Requirements](https://www.rfc-editor.org/rfc/rfc7874.html) - [Discord Go Live architecture](https://discord.com/blog/how-it-all-goes-live-an-overview-of-discords-streaming-technology) - [Discord Voice protocol](https://docs.discord.com/developers/topics/voice-connections) -- [Oopz help center](https://help.oopz.cn/) +- [Oopz program/screen audio help](https://help.oopz.cn/42f9/54e2) +- [Oopz voice quality/latency help](https://help.oopz.cn/42f9/3020) +- [Oopz microphone-noise help](https://help.oopz.cn/42f9/f5ae) - [MDN `getDisplayMedia()`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia) - [MDN `MediaTrackSettings.channelCount`](https://developer.mozilla.org/en-US/docs/Web/API/MediaTrackSettings/channelCount) - [MDN Browser Compatibility Data: `MediaDevices`](https://github.com/mdn/browser-compat-data/blob/main/api/MediaDevices.json) diff --git a/docs/research/realtime-quality-adaptation.md b/docs/research/realtime-quality-adaptation.md index 082d3b1..33ef386 100644 --- a/docs/research/realtime-quality-adaptation.md +++ b/docs/research/realtime-quality-adaptation.md @@ -534,8 +534,9 @@ subscriber's quality/dimensions/FPS to maximum spatial/temporal layers on that subscriber's downtrack. The actual selector remains codec-dependent: VP8 has a temporal selector, whereas H.264/H.265 simulcast is spatial-only. That useful SFU contract still does not extend to direct/peer receivers. Screener's current -SFU subscriber calls only `setSubscribed(true)`, so it does not own a layer -choice today. +SFU subscriber calls `setSubscribed(true)` and requests +`setVideoQuality(HIGH)`. `HIGH` is a ceiling, not a guarantee that BWE forwards +the high layer, so the application does not own the actual layer choice. For this screen-share product the pin has an additional hard mismatch. Client 2.22.0 overwrites SVC screen-share publication to `L1T3`, even when another @@ -604,14 +605,16 @@ resolution, frame rate, or bitrate. resolution or rate. - `maxBitrate` and `maxFramerate` are ceilings. They are neither minimums nor target guarantees, and the project does not use SDP bitrate hacks. -- The folded advanced panel accepts only 720p/1080p/1440p, integer 15-60 fps, - 2-12 Mbps, and the three preferences. It exposes no audio quality controls. - Display capture does not standardize channel-count or sample-rate control; - the portable audio `maxBitrate` field is only a ceiling, though the fixed - 128 kbps value materially raised Chrome's encoder budget in one narrow - fixture; stereo, DTX, and FEC require negotiated fmtp/codec behavior that has - no portable sender setter. The full boundary is recorded in - `docs/research/browser-screen-audio-quality.md`. +- The folded “advanced video” panel accepts only 720p/1080p/1440p, integer + 15-60 fps, 2-12 Mbps, and the three preferences. It exposes no audio quality + controls and is not renamed in the screen-audio slice. Display capture does + not standardize channel-count or sample-rate control; the portable audio + `maxBitrate` field remains only a 128 kbps ceiling. The one accepted peer + exception is a structured Viewer-answer Opus `stereo=1` receive preference, + paired with pinned LiveKit's explicit high-quality stereo/forceStereo option. + DTX stays fixed off, RED retains the pinned SDK default, and FEC remains + browser/SDK negotiation, not a control or custom adaptation algorithm. The + full boundary is recorded in `docs/research/browser-screen-audio-quality.md`. - Every sender update derives from `getParameters()`, calls `setParameters()`, then reads requested/applied bitrate, frame rate, scale, and preference. Rejection or browser rewriting is visible rather than console-only. @@ -646,8 +649,10 @@ CPU/GPU cost, public networks, or sustained behavior. - No copied x264 CRF/preset recipe in the browser path. - No forced codec order until target hardware measurements identify the actual power-efficient encoder. -- No channel-count, sample-rate, Opus bitrate, stereo, DTX, or FEC control, and - no inference of actual stereo or sample rate from `opus/48000/2`. +- No channel-count, sample-rate, codec, arbitrary bitrate, stereo, DTX, RED or + FEC control, and no inference of actual stereo or sample rate from + `opus/48000/2`. Screen media uses one route-consistent stereo contract; + microphone voice, if ever accepted, remains a separate track and policy. - No promise that all browsers honor `applyConstraints` or degradation preference identically. - No automatic composite quality score or periodic profile controller before diff --git a/docs/status.md b/docs/status.md index 516e8a5..36798f8 100644 --- a/docs/status.md +++ b/docs/status.md @@ -56,6 +56,7 @@ Recovery uses one ICE restart, one same-parent rebuild, one alternate peer, then Native WGC/MF passed one-Viewer hardware; package download, Viewer2/FIFO and game A/V remain. Use representative target-device or production evidence for performance, not an ordinary-PC synthetic gate. Ordinary peers stay STUN-only; initial ingress, `peer-selected`, and healthy-reselection browser gates remain open. ADR-0004 still needs its resource/quality matrix; the 20-viewer gate precedes a default change. Mobile uses the same capacity and is a compatibility observation. +Stereo: open. ## Blockers And Decisions diff --git a/docs/方案设计.md b/docs/方案设计.md index 7472b58..69718ad 100644 --- a/docs/方案设计.md +++ b/docs/方案设计.md @@ -152,11 +152,11 @@ Zod 在服务边界拒绝未知类型、超长字符串、非法 SDP/candidate | 1080p30 | 1920x1080、30 fps | 5 Mbps/peer | | 720p30 | 1280x720、30 fps | 3 Mbps/peer | -浏览器可返回更低设置，界面以 `track.getSettings()` 显示实际捕获结果。Current source 保留有界手动质量控制：视频 track 使用 `contentHint = "motion"`，返回的音频 track 使用 `contentHint = "music"`；二者都是内容提示而不是 codec 或质量档位。三档推荐质量与折叠高级区初始值统一采用 `degradationPreference = "maintain-resolution"`（clarity-first），并允许显式选择 `balanced` 或 `maintain-framerate`（fluid）。高级区将设置严格限制为 720p/1080p/1440p、15 至 60 整数 FPS、2 至 12 Mbps，以及 `maintain-resolution`、`balanced`、`maintain-framerate` 三种偏好。它们只是 ceiling 和浏览器偏好，不保证实际分辨率、帧率或码率。 +浏览器可返回更低设置，界面以 `track.getSettings()` 显示实际捕获结果。Current source 保留有界手动质量控制：视频 track 使用 `contentHint = "motion"`，返回的音频 track 使用 `contentHint = "music"`；二者都是内容提示而不是 codec 或质量档位。三档推荐质量与折叠高级区初始值统一采用 `degradationPreference = "maintain-resolution"`（clarity-first），并允许显式选择 `balanced` 或 `maintain-framerate`（fluid）。高级区将设置严格限制为 720p/1080p/1440p、15 至 60 整数 FPS、2 至 12 Mbps，以及 `maintain-resolution`、`balanced`、`maintain-framerate` 三种偏好。它们只是 ceiling 和浏览器偏好，不保证实际分辨率、帧率或码率。本期屏幕音频只有一个固定媒体默认，不改面板名称也不增加音频 UI。 每次 `setParameters()` 都必须从当次 `getParameters()` 派生更新，然后读回 requested/applied `maxBitrate`、`maxFramerate`、`scaleResolutionDownBy` 和 degradation preference。浏览器负责具体降级，实际结果由 stats 观测；拒绝或浏览器改写必须进入可见状态，不能只写 `console.warn`。 -Web 端不做 SDP munging。P2P 与浏览器 relay 在首个 offer 前读取标准 sender capabilities；存在 H.264 时用 `setCodecPreferences()` 将全部 H.264 条目置前，同时保留完整的 browser codec/repair 列表，能力缺失或 API 拒绝时维持浏览器默认。SFU publication 同样在 H.264 能力存在时请求 `videoCodec: "h264"`，否则请求 VP8，并始终保持 `backupCodec: false`，因此不会为兼容回退同时维护第二份 codec publication。实际 outbound codec/profile 与 stats 是协商结果真相。这个可逆偏好只要求定向互通和回退验证；硬件、CPU/GPU、游戏 FPS 和目标网络证据用于性能结论及未来动态选择，不作为上线前置。AV1 仍不得仅凭压缩率推断默认启用。 +Web 视频端不做 SDP munging。P2P 与浏览器 relay 在首个 offer 前读取标准 sender capabilities；存在 H.264 时用 `setCodecPreferences()` 将全部 H.264 条目置前，同时保留完整的 browser codec/repair 列表，能力缺失或 API 拒绝时维持浏览器默认。SFU publication 同样在 H.264 能力存在时请求 `videoCodec: "h264"`，否则请求 VP8，并始终保持 `backupCodec: false`，因此不会为兼容回退同时维护第二份 codec publication。实际 outbound codec/profile 与 stats 是协商结果真相。这个可逆偏好只要求定向互通和回退验证；硬件、CPU/GPU、游戏 FPS 和目标网络证据用于性能结论及未来动态选择，不作为上线前置。AV1 仍不得仅凭压缩率推断默认启用。唯一获准的 Web SDP 变换是下述 Viewer answer 的 Opus stereo receive preference，不得借此扩展视频码率、codec 或通用 munging。 `audio: true` 只是请求。若 `getAudioTracks()` 为空，分享者界面必须显示“当前来源没有可共享音频”。Web 端同时传入 `windowAudio: "window"` 与 `systemAudio: "include"`：窗口选项请求该窗口/应用音频，整个屏幕选项允许用户勾选系统音频；标签页使用浏览器自身的标签页音频选择。浏览器可以忽略这些 picker hints，且没有对应的标准来源类型读回，因此 UI 不额外推断或标注来源。系统音频由用户显式选择，可能包含语音或通知；video-only 始终可用。 @@ -172,7 +172,11 @@ Host 本地预览直接绑定现有 capture stream，不创建 viewer、`RTCPeer 部署版 `769de201f7cc` 的模糊、游戏低帧、持续 `bandwidth` 提示和 Host 刷新恢复都只作为同一 release 的用户报告。首轮复现必须在相同机器、Chrome/Edge、驱动、游戏场景和单个有线 direct Viewer 上对照该 release 与最新 `main`，并分别执行只重建受影响 `HostPeer`、只替换 capture、完整 Host 重建和预览开/关；任何一次只改变一个变量。若实际 codec/encoder 不变而 BWE/发送恢复，则下调 codec 根因优先级；只有编码成本异常时才进入受控 codec spike。 -当前 Web 高级区不暴露音频质量控件。Screen Capture 规范未把 `channelCount`、`sampleRate` 列为 display-audio 可控属性；`track.getSettings()` 中偶尔出现的值只能作为捕获观测。每个 Web Host、浏览器 relay 或 selected-edge 的实际音频 sender 在首次绑定和换源后通过标准 `RTCRtpSender.setParameters()` 请求并读回 128 kbps ceiling；SFU 发布使用同值 `audioPreset` 且关闭 DTX。该 ceiling 不是目标码率或听感保证，实际结果仍由 stats 观测，也不等于 Opus `maxaveragebitrate`。Opus stereo、DTX 和 in-band FEC 属于协商的 codec/fmtp 行为，没有跨主流浏览器的标准 sender setter；`opus/48000/2` 也只是 RTP 信令约定，不能证明 48 kHz 源或实际 stereo。设计不为这些控件新增应用层 SDP munging，也不把 SDK 设置包装成浏览器保证。详细证据与复核门槛见 `docs/research/browser-screen-audio-quality.md`。 +当前 Web “高级视频设置”不暴露音频质量控件。Screen Capture 规范未把 `channelCount`、`sampleRate` 列为 display-audio 可控属性；`track.getSettings()` 中偶尔出现的值只能作为捕获观测。每个 Web Host、浏览器 relay 或 selected-edge 的实际音频 sender 在首次绑定和换源后通过标准 `RTCRtpSender.setParameters()` 请求并读回 128 kbps ceiling；这只是上限，浏览器 WebRTC 拥塞控制仍可降低实际流量，不新增应用层音频降级器。RFC 7587 将 20 ms full-band stereo music 的建议区间列为 64--128 kbps，pinned LiveKit 2.22.0 也把 128 kbps 命名为 `musicHighQualityStereo`；参数由这些外部证据确定，普通 PC synthetic 只验证 setter/互通，不用于标定听感或性能。 + +Web peer 的最小 stereo 契约只改 Viewer 生成的 answer。实现把 MIT `sdp-transform` 2.15.0 与 `@types/sdp-transform` 声明为 root direct dependencies，不依赖 LiveKit 的 transitive install，也不用字符串或正则拼接 SDP。Helper 仅在解析出唯一非 rejected audio media、通过结构化 `rtp` entry 找到唯一 Opus payload 时，才在该 payload 的 fmtp 幂等 upsert `stereo=1`；随后 `ViewerPeer` 把同一个结果交给 `setLocalDescription()` 和 signaling。首次连接、ICE restart、同 parent rebuild、浏览器 relay child 与 selected-edge TURN 都走该边界；TURN 只改变 ICE transport。空 SDP、无或歧义 Opus、多个有效 audio、畸形 fmtp、parse/write 异常都按对象 identity 返回浏览器原 answer，保住仍可继续的 mono/video edge，不发送半修改 SDP。RFC 7587 说明 `stereo` 是可选单向 receiver preference，缺省为 mono preference；RFC 7874 要求 WebRTC endpoint 实现 Opus，而 W3C 只保证未修改的 `createAnswer()` 可用于 `setLocalDescription()`，所以原 answer 是兼容 fallback。`sprop-stereo` 只是 sender likelihood hint，`usedtx` 缺省已是 0，`maxaveragebitrate` 缺省采用对应模式建议最大值，因此本 slice 不增加这三项，也不改浏览器协商的 `useinbandfec`。 + +SFU publisher 使用 pinned SDK 的 `AudioPresets.musicHighQualityStereo`、`forceStereo: true` 与 `dtx: false`。LiveKit 2.22.0 先合并 room publish defaults，其中 `red: true`；本实现不传 `red` 覆盖值，保留既有 RED request。RED 与 Opus in-band FEC 是不同修复机制，保留前者不声称实际启用、替代或等价于后者。只改 SFU 会导致 route switch 声道行为分叉，因此 P2P/browser relay 与 SFU 必须同一 slice 落地。Viewer 继续在同一个持久 `MediaStream`/媒体元素中增删 screen video/audio track；不新建 audio element、Web Audio mix/resample graph、第二 audio representation 或独立音频时钟。当前每条 outbound edge 的 audio RTP payload ceiling 为 128 kbps，cap2 节点最多 256 kbps（不含 RTP/SRTP/UDP/IP overhead），相对 3--8 Mbps 视频档位较小；stereo 仍可能增加编码与网络成本，且浏览器 relay 不保证跨 PeerConnection shared encode。未来 microphone voice 若进入范围，另建独立 source/track 与 AEC、降噪、gain、DTX 设计，不能作用于电影/游戏屏幕音频。详细证据与 Oopz 未知边界见 `docs/research/browser-screen-audio-quality.md`。 `frameRate` 约束和 sender 的 `maxFramerate` 都是目标或上限；浏览器仍控制实际捕获、重复帧处理、编码和拥塞降级。静态画面可能由浏览器减少实际编码数据，但标准没有保证每个实现以同一种方式节省采集或 GPU 开销。本 PoC 不添加画面差分、定时改约束或自定义动态 FPS 状态机，先用 `framesEncoded`、实际发送码率、`totalEncodeTime`、质量受限原因和主机资源占用测量真实问题。 @@ -340,7 +344,7 @@ hop 按[低服务器成本媒体路由](./research/low-server-media-routes.md) - SQLite 房间库从 `1` 自增、重启恢复、v1/v2 -> v3 单事务迁移、nullable 32-byte grant 与 48-byte password material CHECK、畸形值 fail-closed、rotate/revoke/password 写失败原子性和数据最小化；数据库路径无 site access 时启动失败； - STUN/TURN UDP URL 严格校验、生产 STUN 必需、selected-edge tuple 完整性/TTL/secret 独立性、issuer HMAC/expiry/opaque tag、当前 route/session/revision/old-new connection generation 与 one-use rebuild guard，以及 ordinary peer、未命中房间和 Native 的 STUN-only isolation；LiveKit `sfu-config` 保持独立边界； - 单一 `screener-v2` 信令 literal、v1 旧页面 mismatch 的一次性刷新终态，以及协议非法 JSON、未知消息、错误方向、跨房间目标和 payload 限制； -- 质量协议的 host 授权、严格字段与数值范围、last-wins、当前/未来 relay、已配置 SFU 和普通 P2P wire 兼容；sender 的四字段 requested/applied 读回、拒绝与持续受限提示； +- 质量协议的 host 授权、严格字段与数值范围、last-wins、当前/未来 relay、已配置 SFU 和普通 P2P wire 兼容；sender 的四字段 requested/applied 读回、拒绝与持续受限提示；screen audio SDP helper 覆盖 CRLF、existing/missing fmtp、unknown key preservation、幂等、payload number、rejected/non-audio，以及 empty/no-Opus/ambiguous/parser 异常原对象 fallback，`ViewerPeer` 覆盖 initial/restart/rebuild 时同一结果的 local/signaling 一致性，SFU publisher 精确断言 high-quality stereo/forceStereo/DTX 且不覆盖 RED； - HTTP/WS 集成：nginx per-source site-access `limit_req` 与统一错误不泄露 secret，应用不建立 IP limiter 状态；Viewer 可在 Origin/容量门下 upgrade；Host 首条鉴权同时要求 upgrade-captured site access 与 Host token，Viewer 要么持 exact-room grant，要么先有 site access 再按 room policy；site-access cookie 不绕过 private grant/password，grant/password 不能建房、取得 Host 或跨房。覆盖 fragment/sessionStorage/replaceState、无 site access 的 public/password code-only 统一拒绝、通过 site access 后的 public/password 进入、不含明文于 URL/浏览器持久存储/SQLite/log、v2 Native wire，以及 rotate/revoke 后 grace/route 清理、旧 Viewer/上游媒体边关闭和 reconnect 拒绝； - 客户端策略与 stats 解析：会话替换不重连，ordinary ICE、LiveKit 与 selected-edge TURN protocol 分离，远端 TURN transport 保持未知； - TypeScript 类型检查、Vite production build 和仓库 hygiene。 @@ -350,7 +354,7 @@ hop 按[低服务器成本媒体路由](./research/low-server-media-routes.md) - 同机双标签页与同 LAN direct； - 真实跨网络、同房间 direct/peer UDP + SFU/UDP root； - CGNAT、双 NAT、移动热点和普通家宽下使用隔离 candidate，先验证 STUN-only direct/peer 与 SFU/UDP，再只对 controller 选中的单条失败 edge 验证 forced relay；记录 selected attempt 的 allocation/relay port/RSS/CPU/latency 与 RX/TX/loss。人为封锁全部 UDP 时仍必须有界明确失败。旧 production release 留在独立实例作为回滚，不在同进程保留旧 transport； -- Windows Chrome/Edge 的 tab/window/monitor 音轨有/无、同时语音通话和 video-only；Windows 11 原生候选另测 game process tree、通知、无渲染流与进程重启，Windows 10 记录明确不支持/未解决； +- Windows Chrome/Edge 的 tab/window/monitor 音轨有/无、同时语音通话和 video-only；一个可区分左右声道的 fixture 只验证 Web P2P 与 SFU 的 channel correctness、协商 fmtp 和 route switch 后持久 `MediaStream`，不做听感/性能标定；Windows 11 原生候选另测 game process tree、通知、无渲染流与进程重启，Windows 10 记录明确不支持/未解决； - Android Chrome 与 iOS Safari 的播放手势、旋转、前后台和网络切换； - 一名分享者、八名异构观看者连续 30 分钟，同时记录较低人数基线、实际上行、编码负载和首帧时间。真实 1:8 未完成前不得把默认上限写成性能承诺。 - 真实桌面 Chrome 对 balanced 与 clarity 做控制传播和连接身份 smoke；该项只证明直播中应用与不重建，不代替清晰度、负载或公网矩阵。 @@ -367,6 +371,7 @@ hop 按[低服务器成本媒体路由](./research/low-server-media-routes.md) 6. 可选 selected-edge TURN tuple、revision-bound one-use grant 与 parent/child rebuild 已在 source 和 production 配置闭环；下一功能验收只需一条 forced relay 媒体 smoke，完整网络/资源 benchmark 后置。ordinary STUN-only、SFU/UDP、coturn rollout 仍保持独立边界。 7. 在媒体主线阶段验收后，用一个原子 PR 迁移 site access、`screener-v2`、room-scoped Viewer grant 和 SQLite v2；同一部署切换客户端/服务端/环境和数据库备份，随后删除 v1 配置、wire 与测试。用户名/roster 保持独立。 8. Windows Native fixed-`HIGH` 保持可选且不改变 Web 默认；移动端保持 Viewer-only 且不单独分配 relay capacity。Viewer-to-TV 只采用已验证的平台本地输出，不改变媒体路由。 +9. 屏幕音频在一个原子 Web slice 中声明 `sdp-transform`/types direct dependencies，加入 answer helper 与 `ViewerPeer` wiring，并同步设置 SFU stereo options、focused tests 和 owning docs；不增加音频 UI、Web Audio、第二表示或自定义降级算法。 任何一步只实现当前消费者需要的接口。不得为未来账号、第三种媒体拓扑或原生 sender 预建抽象层。 diff --git a/docs/需求理解.md b/docs/需求理解.md index 9d8347d..1f17863 100644 --- a/docs/需求理解.md +++ b/docs/需求理解.md @@ -42,7 +42,7 @@ - [ ] 私密房间双入口：private room 接受现有 `/r/{code}#v={grant}` 合法邀请直接进入，或“站点访问 + 房间号 + 该房间观看密码”。没有 grant 的 `/join` 或 `/r/{code}` 先显示明确标为“站点访问”的整站验证，通过后才显示明确标为“加入房间”的当前房间密码输入；有效 site-access cookie 直接跳过第一步，但两类密码不得复用、代填或互相替代。Host 可设置、轮换或移除房间观看密码。密码接受 1 至 64 个可见 ASCII 字符；异步 scrypt 全局最多执行 2 个、等待 16 个派生。站点口令和房间密码都不得进入 URL/query/storage/log/SQLite；不存在、错误密码、队满和正确密码后的满房都返回通用结果。 - [ ] 统一自动路由：旗舰部署开启 `PEER_ASSISTED_MEDIA` 后，所有正常房间都进入同一个 direct/peer -> SFU -> selected-edge controller；每个房间独立保存 assignment/session/revision/connection generation 和容量状态。合格父节点构成网状候选关系，但实际媒体保持无环且每个 Viewer 只有一个 active upstream；加入、容量变化、掉线和当前边失败触发有界重选，切换重叠必须受现有边预算与当前代媒体证据约束，不做常驻多源分片。`PEER_ASSISTED_ROOM_IDS` 已退役，room `1` 只作为历史 smoke fixture，不能作为功能开关。未启用总开关的开发/轻量部署仍可保留 ordinary P2P；普通 peer 始终 STUN-only。selected-edge TURN 只能复用当前 controller 选中的异常 edge，不能向普通 peer edge 预发 candidates。该能力不增加 UI、用户选择、百分比发布或第二套路由器。 -- [ ] 质量控制扩展：推荐档位仍是默认入口；折叠高级区只允许 720p/1080p/1440p 分辨率 ceiling、15 至 60 的整数 FPS ceiling、2 至 12 Mbps 视频码率 ceiling，以及清晰/平衡/流畅三种偏好。直播中显式应用后才生效，避免滑杆产生连续控制流量。Web 不提供音频声道、采样率、Opus 码率、stereo、DTX 或 FEC 控件；每个实际浏览器音频 sender 固定请求并读回 128 kbps ceiling，SFU 音频发布使用同值 preset 且关闭 DTX。该值只是标准 API/SDK 支持的发送上限，实际码率仍以 stats 为准。display capture 未把声道和采样率定义为可移植控制，其他项依赖 codec/fmtp 协商而没有可移植 sender setter。不得为这些控件新增应用层 SDP munging 或用默认值制造伪控制。实际质量持续低于所选上限时必须显示可解释的限制原因；在真实拥塞、编码器和接收端证据完成前，不把任一组合称为“完美平衡”。 +- [ ] 质量控制扩展：推荐档位仍是默认入口；“高级视频设置”只允许 720p/1080p/1440p 分辨率 ceiling、15 至 60 的整数 FPS ceiling、2 至 12 Mbps 视频码率 ceiling，以及清晰/平衡/流畅三种偏好，初始值采用 `maintain-resolution`（清晰优先），其余偏好保留为显式选择。直播中显式应用后才生效，避免滑杆产生连续控制流量。本期不重命名该区域，也不增加音频 UI：屏幕音频固定使用一个媒体默认，所有实际浏览器 sender 请求并读回 128 kbps ceiling，DTX 关闭，Web peer 与 SFU 都显式允许 Opus stereo；实际码率仍由浏览器拥塞控制和 stats 决定。Peer 只在唯一明确的 active Opus audio 上做结构化、幂等的 Viewer answer `stereo=1` 变换；非目标形态或 parser/writer 异常必须原样保留浏览器 answer，不得阻断可继续的 mono/video edge。不得用宽泛字符串/正则改 SDP，也不得改写浏览器协商的 FEC。SFU 使用 pinned SDK 的 128 kbps high-quality stereo preset、`forceStereo = true`、`dtx = false`，并保留 pinned room default `red = true`。不得暴露声道、采样率、codec、任意码率、stereo、DTX、RED 或 FEC 控件。实际质量持续低于所选上限时必须显示可解释的限制原因；在真实拥塞、编码器和接收端证据完成前，不把任一组合称为“完美平衡”。未来 microphone voice 若进入范围，必须是独立 track/处理链，不能把 AEC/降噪/DTX 套到电影或游戏音频。 - [ ] SFU 质量设置生命周期：直播中修改档位时，浏览器 sender 的 applied readback、LiveKit `LocalVideoTrack` 保存的 degradation preference、track restart options 与 `LocalTrackPublication` 的 republish options 必须作为同一次串行操作更新；任一步失败恢复旧 sender 与旧 SDK 保存态，旧 publication generation 不得回写当前状态。LiveKit full reconnect/republish 不得恢复初次发布的旧偏好。该修复不改变 codec、码率上限、层数或路由，也不能单独解释即时低帧。 - [ ] 本地预览性能候选：在相同动态场景中 A/B 测量 Host 动态预览开/关的 CPU、GPU、encode time 和实际 FPS。只有稳定收益且捕获、编码和远端画面不受影响时，才允许直播开始后默认显示静止缩略图或隐藏动态预览并提供按需展开；未测前不宣称收益。 - [ ] Peer-assisted 媒体路径：分享端与每名普通 Web viewer 都最多两个 child。信令服务维护可从多个合格节点选择 parent 的候选图，但每名 viewer 任一时刻只有一个 active upstream，活动媒体图保持无环；只在新加入、容量释放、节点断开或当前 edge 失败时按深度和服务端加入序号做有界分配或换父，不加入综合评分、主动重平衡、多源媒体或持续优化。每个 viewer 将收到的标准 remote track 通过最多两个新的 `RTCPeerConnection` 转发，因此会逐跳解码和分别重编码；必须测量而不能称为 shared encode。服务端只在受房间数限制的内存状态中保存当前严格校验的质量设置，不写入 SQLite；host 鉴权后先重报设置，在线 viewers 接收更新，后加入者从鉴权快照取得当前值。 @@ -158,9 +158,10 @@ - Given 浏览器未返回系统音轨，When 分享开始，Then 分享者在启播前看到“当前来源没有可共享音频”的提示。 - Given 分享者选择标签页或窗口，When 浏览器支持对应音频捕获，Then picker 提供该来源音频而不扩大为系统混音。Given 分享者选择整个屏幕，When 浏览器支持系统音频，Then picker 提供由用户决定的系统音频开关。应用同时请求 `windowAudio: "window"` 和 `systemAudio: "include"`，不根据 track 存在性伪造来源标签。 - Given 浏览器返回分享音轨，When 开始发布，Then 应用设置标准 `contentHint = "music"` 作为内容意图元数据；该值可读回，但不作为 Opus、码率、声道、采样率、FEC、DTX 或立体声的质量证据。 -- Given Web Host、浏览器 relay 或 selected-edge 建立或替换分享音轨，When 对实际音频 sender 应用发送参数，Then 从当次 `getParameters()` 派生 `encodings[0].maxBitrate = 128000`、调用 `setParameters()` 并立即读回；Given 通过 SFU 发布音轨，Then 使用 `audioPreset.maxBitrate = 128000` 与 `dtx = false`。验收分别核对 applied ceiling 和 stats 实际码率，不从 preset 名称推断 stereo。 +- Given Web Host、浏览器 relay 或 selected-edge 建立或替换分享音轨，When 对实际音频 sender 应用发送参数，Then 从当次 `getParameters()` 派生 `encodings[0].maxBitrate = 128000`、调用 `setParameters()` 并立即读回；Given Web Viewer 为含唯一有效 audio/Opus payload 的 offer 生成 answer，Then 用结构化 SDP parser 只 upsert 该 payload 的 `stereo=1`，并把同一个结果用于 `setLocalDescription()` 和 signaling。首次连接、ICE restart、同 parent rebuild 和 relay child 都适用；空 SDP、无/歧义 Opus、多个有效 audio、畸形 fmtp 或 parse/write 异常必须按对象 identity 返回浏览器原 answer 并继续，不能发送部分修改或静默冒充 stereo。只有实际 `setLocalDescription()` 失败才进入既有连接错误路径。`sprop-stereo`、`usedtx` 与 `maxaveragebitrate` 不新增，其他 fmtp/FEC/session/media 字段逐项保留。 +- Given 通过 SFU 发布分享音轨，When 构建 LiveKit 2.22.0 publish options，Then 使用 `AudioPresets.musicHighQualityStereo`、`forceStereo = true` 与 `dtx = false`，不覆盖 pinned room default `red = true`；验收同时核对 SDK 选项、applied ceiling、当前 RTP codec/fmtp 与 stats 实际码率，不能从 preset 名称或 `opus/48000/2` 单独推断实际声道内容。 - Given 浏览器返回屏幕音轨，When 产品展示音频状态或未来连接详情，Then 只能把存在性、浏览器实际返回的 track settings、实际协商 codec/fmtp 和实际流量分别标为捕获、协商或观测事实；字段缺失必须显示为未知，且不得从 Opus 的 `48000/2` RTP 声明推断源采样率或实际 stereo。 -- Given Web 或后续 native sender 分享电影/视频音频，When 进行产品验收，Then 必须用独立 A/B/C 证据关联 capture audio track/settings、实际 negotiated codec/fmtp、outbound/inbound 实际码率、loss、jitter、concealment、jitter-buffer 与 A/V sync；native sender 必须通过该音质和同步矩阵，且不得把 audio `maxBitrate` ceiling 当作提质证明。 +- Given Web 或后续 native sender 分享电影/视频音频，When 进行产品验收，Then 必须用独立 A/B/C 证据关联 capture audio track/settings、实际 negotiated codec/fmtp、outbound/inbound 实际码率、loss、jitter、concealment、jitter-buffer 与 A/V sync；P2P、浏览器 relay 与 SFU route switch 保持同一持久 Viewer `MediaStream`/媒体元素和 stereo 协商，不新增独立 audio element 或 Web Audio 时钟边界。native sender 必须通过该音质和同步矩阵，且不得把 audio `maxBitrate` ceiling 当作提质证明。 - Given 分享者同时运行语音软件，When 选择仅分享游戏声音，Then supported Windows 11 native sender 只保留目标游戏及其子进程的渲染音频，观众、通知和其他应用声音不得进入分享；浏览器 system audio 无法满足时必须允许 video-only 或明确告知其可能包含语音/通知，不能用 AEC、降噪或来源标签伪称已隔离。 - Given 分享者显式选择 Native window H.264 与一个本地窗口，When 分享开始，Then helper 必须在再次验证 HWND/PID/creation-time 后以 WGC/D3D11 和 hardware-only MF H.264 产生一份 `42c01f` Annex-B 输出并直接复用现有 Go/Pion fanout；Viewer 仍使用普通浏览器。硬件 MFT、WGC、音频或目标身份任一失败都明确停止该模式，不得静默改用软件编码、浏览器画面、显示器画面或系统混音。 diff --git a/package-lock.json b/package-lock.json index e22da31..4742254 100644 --- a/package-lock.json +++ b/package-lock.json @@ -13,6 +13,7 @@ "lucide-react": "1.31.0", "react": "19.2.8", "react-dom": "19.2.8", + "sdp-transform": "2.15.0", "sirv": "3.0.2", "ws": "8.21.3", "zod": "4.4.3" @@ -21,6 +22,7 @@ "@types/node": "24.13.3", "@types/react": "19.2.18", "@types/react-dom": "19.2.4", + "@types/sdp-transform": "2.15.0", "@types/ws": "8.18.1", "@vitejs/plugin-react": "6.0.5", "tsx": "4.23.12", @@ -850,6 +852,13 @@ "@types/react": "^19.2.0" } }, + "node_modules/@types/sdp-transform": { + "version": "2.15.0", + "resolved": "https://registry.npmjs.org/@types/sdp-transform/-/sdp-transform-2.15.0.tgz", + "integrity": "sha512-ikIFF0EaYt/2XetIYYVeMj6SB52oVXFasJUXDzWHgzNJS5ep2Pbsu7f8f3Za+dEie8HQtt3Zr9mHYBpWT0XgxQ==", + "dev": true, + "license": "MIT" + }, "node_modules/@types/ws": { "version": "8.18.1", "resolved": "https://registry.npmjs.org/@types/ws/-/ws-8.18.1.tgz", diff --git a/package.json b/package.json index 298d303..a806c1d 100644 --- a/package.json +++ b/package.json @@ -28,6 +28,7 @@ "lucide-react": "1.31.0", "react": "19.2.8", "react-dom": "19.2.8", + "sdp-transform": "2.15.0", "sirv": "3.0.2", "ws": "8.21.3", "zod": "4.4.3" @@ -36,6 +37,7 @@ "@types/node": "24.13.3", "@types/react": "19.2.18", "@types/react-dom": "19.2.4", + "@types/sdp-transform": "2.15.0", "@types/ws": "8.18.1", "@vitejs/plugin-react": "6.0.5", "tsx": "4.23.12", diff --git a/src/client/sfu/publisher.ts b/src/client/sfu/publisher.ts index bb0ebb9..2b58bf8 100644 --- a/src/client/sfu/publisher.ts +++ b/src/client/sfu/publisher.ts @@ -8,7 +8,6 @@ import type { import { configureTwoLayerVideoSender, QUALITY_RESOLUTIONS, - SCREEN_AUDIO_MAX_BITRATE, SCREEN_SHARE_LOW_SCALE, screenShareLowBitrate, senderParameterWarning, @@ -170,7 +169,7 @@ export class SfuPublisher { room, audioTrack, sdk.Track.Source.ScreenShareAudio, - audioPublishOptions(), + audioPublishOptions(sdk), ); if (!this.owns(room, generation)) { return false; @@ -281,7 +280,7 @@ export class SfuPublisher { room, nextAudioTrack, sdk.Track.Source.ScreenShareAudio, - audioPublishOptions(), + audioPublishOptions(sdk), ); if (!this.owns(room, generation)) { return false; @@ -630,9 +629,10 @@ function videoPublishOptions( }; } -function audioPublishOptions(): TrackPublishOptions { +function audioPublishOptions(sdk: LiveKit): TrackPublishOptions { return { - audioPreset: { maxBitrate: SCREEN_AUDIO_MAX_BITRATE }, + audioPreset: sdk.AudioPresets.musicHighQualityStereo, + forceStereo: true, dtx: false, }; } diff --git a/src/client/webrtc/screen-audio-sdp.ts b/src/client/webrtc/screen-audio-sdp.ts new file mode 100644 index 0000000..beb3ab7 --- /dev/null +++ b/src/client/webrtc/screen-audio-sdp.ts @@ -0,0 +1,81 @@ +import { parse, parsePayloads, write } from "sdp-transform"; + +export function preferScreenAudioStereo( + answer: RTCSessionDescriptionInit, +): RTCSessionDescriptionInit { + try { + if (answer.type !== "answer" || !answer.sdp?.trim()) { + return answer; + } + const session = parse(answer.sdp); + if ( + session.version !== 0 || + !session.origin || + typeof session.name !== "string" || + !session.timing || + !Array.isArray(session.media) + ) { + return answer; + } + + const audioSections = session.media.filter( + (media) => media.type === "audio" && media.port !== 0, + ); + if (audioSections.length !== 1) { + return answer; + } + + const audio = audioSections[0]!; + const payloads = new Set(parsePayloads(audio.payloads ?? "")); + const opus = audio.rtp.filter( + (codec) => + payloads.has(codec.payload) && codec.codec.toLowerCase() === "opus", + ); + if (opus.length !== 1) { + return answer; + } + + const payload = opus[0]!.payload; + const formats = audio.fmtp.filter((format) => format.payload === payload); + if (formats.length > 1) { + return answer; + } + if (formats[0]) { + const config = upsertStereo(formats[0].config); + if (config === null) { + return answer; + } + formats[0].config = config; + } else { + audio.fmtp.push({ payload, config: "stereo=1" }); + } + + const sdp = write(session); + return sdp.trim() ? { type: answer.type, sdp } : answer; + } catch { + return answer; + } +} + +function upsertStereo(config: string): string | null { + const parameters = config.split(";").map((parameter) => parameter.trim()); + let stereoIndex = -1; + for (const [index, parameter] of parameters.entries()) { + const equals = parameter.indexOf("="); + if (equals <= 0 || equals === parameter.length - 1) { + return null; + } + if (parameter.slice(0, equals).trim().toLowerCase() === "stereo") { + if (stereoIndex !== -1) { + return null; + } + stereoIndex = index; + } + } + if (stereoIndex === -1) { + parameters.push("stereo=1"); + } else { + parameters[stereoIndex] = "stereo=1"; + } + return parameters.join(";"); +} diff --git a/src/client/webrtc/viewer-peer.ts b/src/client/webrtc/viewer-peer.ts index abae160..4325928 100644 --- a/src/client/webrtc/viewer-peer.ts +++ b/src/client/webrtc/viewer-peer.ts @@ -8,6 +8,7 @@ import { createStatsAccumulator, type StatsAccumulator, } from "./stats"; +import { preferScreenAudioStereo } from "./screen-audio-sdp"; const MAX_PENDING_CANDIDATES = 64; const MAX_AUTOMATIC_RECOVERY_REQUESTS = 2; @@ -101,7 +102,7 @@ export class ViewerPeer { if (!this.isCurrentConnection(connection, connectionId)) { return; } - const answer = await connection.createAnswer(); + const answer = preferScreenAudioStereo(await connection.createAnswer()); if (!this.isCurrentConnection(connection, connectionId)) { return; } diff --git a/tests/screen-audio-sdp.test.ts b/tests/screen-audio-sdp.test.ts new file mode 100644 index 0000000..fec6061 --- /dev/null +++ b/tests/screen-audio-sdp.test.ts @@ -0,0 +1,112 @@ +import { describe, expect, it } from "vitest"; +import { parse } from "sdp-transform"; + +import { preferScreenAudioStereo } from "../src/client/webrtc/screen-audio-sdp.ts"; + +function answerSdp(opusFmtp: string | null = "minptime=10;useinbandfec=1"): string { + return [ + "v=0", + "o=- 1 1 IN IP4 127.0.0.1", + "s=-", + "t=0 0", + "m=audio 9 UDP/TLS/RTP/SAVPF 111 0", + "c=IN IP4 0.0.0.0", + "a=rtpmap:111 opus/48000/2", + ...(opusFmtp === null ? [] : [`a=fmtp:111 ${opusFmtp}`]), + "a=rtpmap:0 PCMU/8000", + "a=recvonly", + "m=video 9 UDP/TLS/RTP/SAVPF 96", + "c=IN IP4 0.0.0.0", + "a=rtpmap:96 H264/90000", + "a=fmtp:96 packetization-mode=1;profile-level-id=42e01f", + "a=recvonly", + "", + ].join("\r\n"); +} + +describe("preferScreenAudioStereo", () => { + it("adds one exact stereo token without changing FEC or video fmtp", () => { + const result = preferScreenAudioStereo({ + type: "answer", + sdp: answerSdp("minptime=10;useinbandfec=1;x-stereo=keep"), + }); + const session = parse(result.sdp!); + + expect(session.media[0]?.fmtp).toContainEqual({ + payload: 111, + config: "minptime=10;useinbandfec=1;x-stereo=keep;stereo=1", + }); + expect(session.media[1]?.fmtp).toEqual([ + { payload: 96, config: "packetization-mode=1;profile-level-id=42e01f" }, + ]); + expect(preferScreenAudioStereo(result)).toEqual(result); + }); + + it("replaces only an exact stereo token", () => { + const result = preferScreenAudioStereo({ + type: "answer", + sdp: answerSdp("stereo=0;x-stereo=0;minptime=10"), + }); + + expect(parse(result.sdp!).media[0]?.fmtp[0]?.config).toBe( + "stereo=1;x-stereo=0;minptime=10", + ); + }); + + it("creates the Opus fmtp row when it is absent", () => { + const result = preferScreenAudioStereo({ + type: "answer", + sdp: answerSdp(null), + }); + + expect(parse(result.sdp!).media[0]?.fmtp).toContainEqual({ + payload: 111, + config: "stereo=1", + }); + }); + + it("ignores a rejected audio section", () => { + const rejected = [ + "m=audio 0 UDP/TLS/RTP/SAVPF 112", + "a=rtpmap:112 opus/48000/2", + "a=fmtp:112 stereo=0", + ].join("\r\n"); + const result = preferScreenAudioStereo({ + type: "answer", + sdp: answerSdp().replace("m=audio 9", `${rejected}\r\nm=audio 9`), + }); + const audio = parse(result.sdp!).media.filter((media) => media.type === "audio"); + + expect(audio[0]?.fmtp[0]?.config).toBe("stereo=0"); + expect(audio[1]?.fmtp[0]?.config).toContain("stereo=1"); + }); + + it.each([ + ["empty", { type: "answer", sdp: "" }], + ["malformed", { type: "answer", sdp: "not SDP" }], + ["parser exception", { type: "answer", sdp: { trim: () => "x" } as unknown as string }], + ["not an answer", { type: "offer", sdp: answerSdp() }], + [ + "missing Opus", + { type: "answer", sdp: answerSdp().replace("opus/48000/2", "ISAC/16000") }, + ], + [ + "multiple active audio sections", + { type: "answer", sdp: `${answerSdp()}m=audio 9 RTP/AVP 0\r\n` }, + ], + [ + "duplicate Opus fmtp rows", + { + type: "answer", + sdp: answerSdp().replace( + "a=fmtp:111 minptime=10;useinbandfec=1", + "a=fmtp:111 minptime=10\r\na=fmtp:111 useinbandfec=1", + ), + }, + ], + ] as const)("returns the original answer for %s", (_case, answer) => { + const result = preferScreenAudioStereo(answer); + expect(result).toBe(answer); + expect(result.sdp).toBe(answer.sdp); + }); +}); diff --git a/tests/sfu-client.test.ts b/tests/sfu-client.test.ts index ecc20b1..8410060 100644 --- a/tests/sfu-client.test.ts +++ b/tests/sfu-client.test.ts @@ -217,7 +217,12 @@ const livekit = vi.hoisted(() => { nextSenderParameterError: null, }; + const AudioPresets = { + musicHighQualityStereo: { maxBitrate: 128_000 }, + } as const; + return { + AudioPresets, FakeRemoteParticipant, FakeRemotePublication, FakeRoom, @@ -250,6 +255,7 @@ const VideoQuality = { } as const; vi.mock("livekit-client", () => ({ + AudioPresets: livekit.AudioPresets, Room: livekit.FakeRoom, RoomEvent, Track, @@ -460,6 +466,7 @@ describe("SfuPublisher", () => { expect(room.localParticipant.publishTrack).toHaveBeenNthCalledWith(2, audio, { source: Track.Source.ScreenShareAudio, audioPreset: { maxBitrate: 128_000 }, + forceStereo: true, dtx: false, }); expect(sender.setParameters).toHaveBeenCalledOnce(); @@ -916,6 +923,7 @@ describe("SfuPublisher", () => { expect(room.localParticipant.publishTrack).toHaveBeenNthCalledWith(2, audio, { source: Track.Source.ScreenShareAudio, audioPreset: { maxBitrate: 128_000 }, + forceStereo: true, dtx: false, }); }); diff --git a/tests/viewer-peer-generation.test.ts b/tests/viewer-peer-generation.test.ts index 3979e08..c376bbc 100644 --- a/tests/viewer-peer-generation.test.ts +++ b/tests/viewer-peer-generation.test.ts @@ -17,6 +17,19 @@ function createDeferred(): Deferred { return { promise, resolve }; } +const ANSWER_SDP = [ + "v=0", + "o=- 1 1 IN IP4 127.0.0.1", + "s=-", + "t=0 0", + "m=audio 9 UDP/TLS/RTP/SAVPF 111", + "c=IN IP4 0.0.0.0", + "a=rtpmap:111 opus/48000/2", + "a=fmtp:111 minptime=10;useinbandfec=1", + "a=recvonly", + "", +].join("\r\n"); + interface ConnectionPlan { answerError?: Error; candidateGates?: Promise[]; @@ -95,7 +108,7 @@ class FakePeerConnection extends EventTarget { } return { type: "answer", - sdp: `answer-${FakePeerConnection.instances.indexOf(this)}`, + sdp: ANSWER_SDP, }; } @@ -194,6 +207,30 @@ afterEach(() => { }); describe("ViewerPeer connection generations", () => { + it("sets and signals stereo for initial, restart, and rebuild answers", async () => { + const signals: SignalPayload[] = []; + const peer = createPeer(signals, []); + + await peer.acceptSignal("host", offer("screen-audio")); + await peer.acceptSignal("host", offer("screen-audio")); + await peer.acceptSignal("host", offer("screen-audio-rebuild")); + + const answers = signals.filter((signal) => signal.kind === "description"); + const localDescriptions = FakePeerConnection.instances.flatMap((connection) => + connection.setLocalDescription.mock.calls.map(([description]) => description), + ); + expect(answers).toHaveLength(3); + for (const [index, description] of localDescriptions.entries()) { + expect(description).toEqual(expect.objectContaining({ + type: "answer", + sdp: expect.stringContaining( + "a=fmtp:111 minptime=10;useinbandfec=1;stereo=1", + ), + })); + expect(answers[index]?.description.sdp).toBe(description.sdp); + } + }); + it("applies STUN-only ICE configuration at creation and update", async () => { const peer = createPeer([], [], [], { iceServers: [{ urls: ["stun:stun-a.example.test:3478"] }],