feat: prefer proven H264 browser senders

This commit is contained in:
AAA committed 2026-08-26 19:05:21 +08:00
1 parent 6abd5c8030
commit 6b3c3de8a0
19 files changed
+812 -66

No files matched your search

@@ -23,7 +23,8 @@ it does not need a second bitrate, resolution, FPS, or layer-control system.
## Decision
1. Browser video makes one internal, sender-scoped codec decision. A bounded
1. Browser video makes one internal, share-generation-scoped codec decision per
sending endpoint. A bounded
actual-`RTCPeerConnection` preflight measures H.264 encoded progress against
a deterministic moving probe track at the current share's effective target
dimensions and frame rate. The real capture supplies those target settings,
@@ -43,7 +44,7 @@ it does not need a second bitrate, resolution, FPS, or layer-control system.
After accepting an answer, the Host reapplies the current video profile to
the negotiated sender because the browser may replace or rewrite encoding
parameters during negotiation.
5. Direct and Browser-relay offers with a proved source order H.264 before VP8
5. Direct and Browser-relay offers with a proved share source order H.264 before VP8
in one standard codec-preference list; both endpoints select their first
common codec without a parallel media connection. A failed source gate
offers VP8 only. Existing edges are not renegotiated when a later relay gate
+5 -3
View File
@@ -87,7 +87,7 @@ Last updated: 2026-08-26
- WebRTC owns direct/peer media adaptation. The Host reapplies the selected video
profile after answer negotiation; no periodic application controller exists.
- The production SFU publisher uses the Host's H.264/VP8 decision and selected
ceiling but no custom simulcast layers. Pinned LiveKit defaults own
ceiling but no backup codec or custom simulcast layers. Pinned LiveKit defaults own
representations, Dynacast owns aggregate
demand, and server send-side BWE owns subscriber forwarding. AdaptiveStream
stays disabled because any Viewer may relay its received track.
@@ -118,8 +118,10 @@ Last updated: 2026-08-26
## Current Source And Production
- Production and the matching canonical application/runtime tree use strict
`screener-v12`; exact identity is indexed by [status](./status.md).
- Production and canonical source use strict `screener-v12`, but the pending
application candidate adds the internal H.264/VP8 decision while production
remains on its prior fixed-VP8 artifact. Exact identities are indexed by
[status](./status.md).
- Production uses random memory rooms, 20-Viewer admission, endpoint cap `2`,
fixed 9,000-publication ingress and `9000 * 20` subscription egress admission,
STUN UDP 3478, LiveKit media UDP 7882, and Web TCP 80/443. Node 8787 and
+4 -4
View File
@@ -2,7 +2,7 @@
## 2026-08-18 14:45
状态:production 与 canonical application/runtime tree 使用严格 `screener-v12`。Browser 视频固定 VP8 并使用 `contentHint = "motion"`,音频使用 `music`;codec UI/state/wire、自定义 SFU layer list与应用媒体质量 controller 均不存在。SFU 由 pinned LiveKit 默认 representations、Dynacast、server send-side BWE 和 disabled AdaptiveStream 拥有质量适配;screen audio 使用 96/128/192 kbps ceiling、stereo、DTX off 与 RED off。Viewer 页面恢复、原生控件和当前线路重连已进入 source 与 production,精确执行状态由 `docs/status.md` 持有。
状态:production 与 canonical source 使用严格 `screener-v12`。Canonical source 已实现每分享代次的 H.264 实际 sender gate 与 VP8 fallback,production 仍是待切换的固定 VP8 artifact;两者都使用 video `contentHint = "motion"` 和 audio `music`,且 codec UI/state/wire、自定义 SFU layer list与应用媒体质量 controller 均不存在。SFU 由 pinned LiveKit 默认 representations、Dynacast、server send-side BWE 和 disabled AdaptiveStream 拥有质量适配;screen audio 使用 96/128/192 kbps ceiling、stereo、DTX off 与 RED off。精确执行状态由 `docs/status.md` 持有。
## 目标与边界
@@ -162,7 +162,7 @@ accepted 的唯一 wire 是 `screener-v12`,不做版本协商、旧 parser、
每次 `setParameters()` 都必须从当次 `getParameters()` 派生更新,然后读回 requested/applied `maxBitrate`、`maxFramerate`、`scaleResolutionDownBy` 和 degradation preference。浏览器负责具体降级,实际结果由 stats 观测;拒绝或浏览器改写必须进入可见状态,不能只写 `console.warn`。
Accepted `screener-v12` Web 视频使用页面内 sender-scoped H.264/VP8 决策。初始 Host capture 在取得 room/route authority 前,以一对无 ICE server 的本地标准 `RTCPeerConnection` 运行一个有界 H.264-only sender probe。Probe 使用按当前分享 effective target dimensions/frame rate 生成的短时确定性运动 Canvas track,并以 `motion` 表达同一内容意图;真实 capture/upstream track 只提供目标规格,其当前内容是否变化不参与 encoder capability 判断。Gate 只用实际协商 codec、同一身份的 probe media-source/encoded progress 与 bounded completion 判断,不读取 UA/GPU/MFT allowlist,也不单独相信 capability 或 `powerEfficientEncoder`;标准 Canvas capture 不可用、失败或 inconclusive 时使用 VP8。成功结果让后续 P2P/browser-relay offer 按浏览器原生 H.264 capability 顺序列出 H.264,再列 VP8 与 RTX/RED/FEC repair capabilities;失败或 inconclusive 只列 VP8 与 repair。Relay 收到新 upstream stream 后在后台运行同一 gate,完成前建立的 child 使用 VP8,完成后只影响未来 child。无法应用 codec preferences 或缺少 VP8 时明确失败。SFU publication 使用 Host source 当时的同一 H.264/VP8 决定并关闭 backup codec。所有新 Web peer sender 与 SFU publication 保留 `motion` 和用户选择的分辨率、帧率及码率 ceiling;当 desired degradation preference 允许分辨率下降时,启动阶段的 effective preference 暂为 `maintain-resolution`,当前 sender 累计编码至少五帧后由既有 stats 路径一次性应用 desired preference。该切换使用标准 `setParameters()` 进入或离开 balanced 时的原生 restriction reset,不增加周期重写、质量评分或应用层调档器。UI 和 quality state/wire 没有 codec 字段;gate 结果不持久化、不改写已连接 edge,实际 outbound codec/profile、encoder implementation 与 stats 是只读结果真相。应用 SDP 变换边界仍只有 Viewer answer 的 Opus stereo receive preference。
Accepted `screener-v12` Web 视频使用页面内 share-generation-scoped H.264/VP8 决策。初始 Host capture 在取得 room/route authority 前,以一对无 ICE server 的本地标准 `RTCPeerConnection` 运行一个有界 H.264-only sender probe。Probe 使用按当前分享 effective target dimensions/frame rate 生成的短时确定性运动 Canvas track,并以 `motion` 表达同一内容意图;真实 capture/upstream track 只提供目标规格,其当前内容是否变化不参与 encoder capability 判断。Gate 只用实际协商 codec、同一身份的 probe media-source/encoded progress 与 bounded completion 判断,不读取 UA/GPU/MFT allowlist,也不单独相信 capability 或 `powerEfficientEncoder`;标准 Canvas capture 不可用、失败或 inconclusive 时使用 VP8。成功结果让后续 P2P/browser-relay offer 按浏览器原生 H.264 capability 顺序列出 H.264,再列 VP8 与 RTX/RED/FEC repair capabilities;失败或 inconclusive 只列 VP8 与 repair。Relay 在该分享代首次收到 upstream video 且没有 active/prepared child 时运行同一 gate;若 child prepare 先到,则等待已经在运行的 bounded gate 结束后只创建一份真实 child sender。质量设置、换源和换父不重复 gate。无法应用 codec preferences 或缺少 VP8 时明确失败。SFU publication 使用 Host share generation 的同一 H.264/VP8 决定并关闭 backup codec。所有新 Web peer sender 与 SFU publication 保留 `motion` 和用户选择的分辨率、帧率及码率 ceiling;当 desired degradation preference 允许分辨率下降时,启动阶段的 effective preference 暂为 `maintain-resolution`,当前 sender 累计编码至少五帧后由既有 stats 路径一次性应用 desired preference。该切换使用标准 `setParameters()` 进入或离开 balanced 时的原生 restriction reset,不增加周期重写、质量评分或应用层调档器。UI 和 quality state/wire 没有 codec 字段;gate 结果不持久化、不改写已连接 edge,实际 outbound codec/profile、encoder implementation 与 stats 是只读结果真相。应用 SDP 变换边界仍只有 Viewer answer 的 Opus stereo receive preference。
Web display-audio 请求显式关闭 `echoCancellation`、`noiseSuppression`、`autoGainControl` 和 `voiceIsolation`,并以 `channelCount: { ideal: 2 }` 偏好双声道;这是当前 Chrome/Edge Host 基线的实现契约,用于避开网页 `getDisplayMedia()` 默认的语音处理和单声道 APM 候选,不宣称跨浏览器保证,也不固定采样率。所有值都是非强制 post-selection preference;不使用 `exact`、`min` 或 `advanced`。若 `getAudioTracks()` 为空,分享者界面必须显示“当前来源没有可共享音频”。Web 端同时传入 `windowAudio: "window"` 与 `systemAudio: "include"`:窗口选项请求该窗口/应用音频,整个屏幕选项允许用户勾选系统音频;标签页使用浏览器自身的标签页音频选择。浏览器可以忽略这些约束或 picker hints,且没有对应的标准来源类型读回,因此 UI 不额外推断或标注来源。系统音频由用户显式选择,可能包含语音或通知;video-only 始终可用。
@@ -172,7 +172,7 @@ Web display-audio 请求显式关闭 `echoCancellation`、`noiseSuppression`、`
直播中切换来源仍由明确的按钮手势重新调用 `getDisplayMedia()`,不改变房间、host generation 或媒体拓扑。选源取消或失败时继续发送旧流;成功后为每个现有 sender 调用 `replaceTrack()`,健康连接不重新协商。初始 offer 预留 send-only audio transceiver,使音频可以在 `null -> track -> null` 间切换。若某个 peer 的替换或回滚失败,只重建该 peer;新流提交后才停止旧流,并以当前 stream 身份保护旧 track 的迟到 `ended` 事件。统计码率累加器在换源后重置,避免把不同 track 的计数器混算。
直播中切换质量不重新打开选源器:对当前视频 track 调用 `applyConstraints()`,再更新所有现有 sender 的参数;quality mutation 不重新运行 codec gate,新加入或重建的 Browser sender 使用当前 source 的内存结果与最新其他权威设置。失败时保留当前分享并给出状态,不以重建整个房间补偿。临时暂停禁用当前 capture stream 的全部音视频 tracks;Host 通过当前 session 与 `shareGeneration` 发送一个 paused boolean,服务端同步 `host-status`、route pause 与 `authenticated` 快照,Viewer 在舞台显示“分享者已暂停”。Resume 使用同一状态更新并重新启用同一批 tracks,不另设 authorization、source acknowledgement 或 proof。晚加入和信令重连从快照恢复,恢复、停止、新分享或会话替换清除旧状态;不得把 track mute、黑帧或网络停顿推断成主动暂停。房间与连接保持不变。
直播中切换质量不重新打开选源器:对当前视频 track 调用 `applyConstraints()`,再更新所有现有 sender 的参数;quality mutation 不重新运行 codec gate,新加入或重建的 Browser sender 使用当前分享代次的内存结果与最新其他权威设置。失败时保留当前分享并给出状态,不以重建整个房间补偿。临时暂停禁用当前 capture stream 的全部音视频 tracks;Host 通过当前 session 与 `shareGeneration` 发送一个 paused boolean,服务端同步 `host-status`、route pause 与 `authenticated` 快照,Viewer 在舞台显示“分享者已暂停”。Resume 使用同一状态更新并重新启用同一批 tracks,不另设 authorization、source acknowledgement 或 proof。晚加入和信令重连从快照恢复,恢复、停止、新分享或会话替换清除旧状态;不得把 track mute、黑帧或网络停顿推断成主动暂停。房间与连接保持不变。
Host 本地预览直接绑定现有 capture stream,不创建 viewer、`RTCPeerConnection` 或额外媒体边,也不产生 SFU/应用服务器出口。预览仍可能有本地合成、缩放和 GPU 显示成本;在同一动态场景对预览开/关采集 Host CPU、GPU、encode time 和实际 FPS。后台性能诊断必须把 Host 页面前台/遮挡/最小化、被捕获 source 前台/遮挡/最小化以及 preview 播放/暂停分成独立变量,并关联 document visibility/focus、track mute/readyState 与 capture/outbound/inbound stats。页面 timer、伪活跃和 Screen Wake Lock 都不构成屏幕捕获保活方案;若标准浏览器无法维持目标行为,应记录平台能力边界而不是制造第二套 lifecycle。只有稳定收益且捕获、编码和远端画面不变时,才把直播后的默认动态预览改为静止缩略图或隐藏并允许按需展开;隐藏预览绝不停止 capture track 或 sender encode。
@@ -363,7 +363,7 @@ Token-free SFU standby prewarm 保持可选且不创建 participant、publicatio
## 当前交付边界
- Production 与 canonical application/runtime tree 使用严格单一 `screener-v12` wire。当前实现包含短 room-lived grant、明确 missing-room 结果、产品诊断下载移除、完整短 URL 邀请 UI、exact-current audio stats 与音轨跨 relay/SFU reconnect 连续性,固定 Browser VP8、video `contentHint = "motion"`、audio `music`,且不存在 codec selector、quality-state/wire 字段、自定义 SFU layer list或分享生命周期 codec 分支。Browser SFU 使用 pinned LiveKit 默认 representations、Dynacast、server send-side BWE、disabled AdaptiveStream 与 RED off;精确执行状态由 `docs/status.md` 持有。
- Production 与 canonical source 使用严格单一 `screener-v12` wire。Canonical source 的每分享代次 actual-sender gate 在证据通过时让未来 direct/relay 和 Host 单一 SFU publication 使用 H.264,否则使用 VP8;production 仍是固定 VP8 artifact。两者都使用 video `contentHint = "motion"`、audio `music`,不存在 codec selector、quality-state/wire 字段、自定义 SFU layer list、backup publication或健康 edge codec 切换。Browser SFU 使用 pinned LiveKit 默认 representations、Dynacast、server send-side BWE、disabled AdaptiveStream 与 RED off;精确执行状态由 `docs/status.md` 持有。
- Pause/Resume 由当前 Host session 和 `shareGeneration` 约束。Host 后台性能和 VP8 hardware acceleration 分别按受控根因证据诊断;标准 ICE/SFU 按真实异构网络验收。
- 当前发布面只包含 Web Host、Viewer 和 browser relay。Native sender、capture helper、shared-encode executable 与两个分发包仍由各自后续决策和物理证据驱动,不属于当前 Browser release。
+3 -3
View File
@@ -27,7 +27,7 @@ SFU 的 publication、subscription、server-resource accounting、临时 overlap
- 端点发送容量:所有非服务器端点使用同一个服务端权威稳态 outbound media-copy capacity,由唯一部署项 `ENDPOINT_MEDIA_COPY_CAPACITY` 配置,默认 `2` 且只接受 `1`、`2` 或 `3`;不读取旧键或保留第二套 tier。peer child 或 Host 的唯一 SFU publication 各消耗一份,上游接收不消耗该预算。客户端只能声明不高于部署值的可用容量,角色、浏览器、UA、设备类型和页面可见性不产生不同上限。服务端在每次 `authenticated` 权威快照中返回最终 `endpointMediaCopyCapacity`;普通非 peer-assisted 模式也只授权稳定加入顺序中的至多 `C` 个 Host child,超额 Viewer 保持入房等待并在槽位释放后提升,信令不能绕过该集合。Browser Host 与 Viewer 严格校验该值并用它约束稳态 assignment;并行产生媒体的 candidate 必须持有唯一 `TransitionOverlapSlot`,active+pending 物理 copy 只可达到 `min(C + 1, 3)` 并在 commit/abort 后释放。`C=3` 满载时必须先收到 active bounded-gap retirement 再 prepare,绝不产生第四份 copy。SFU subscriber egress 由独立 server admission 记账。当前不兼容房间合同以单一 release 原子升级;缺失/非法 cap、旧 wire 与 executable sender都在取得room/route authority前fail closed。
- P2P 优先:每条普通逻辑媒体边先尝试 direct/peer UDP,服务器不得默认中转可直连媒体。更多观看者优先沿满足统一端点容量的无环 peer 图分布;每名 Viewer 同时至多一个 active upstream,失败时只影响需要重选的 child。任何 source-reachable endpoint 在 effective capacity 有空位时都可作为 parent candidate;Peer-fed 与 SFU-fed endpoint 使用同一个 provisional-child transaction,并在 candidate child 解码出第一帧新画面后独立提交该 edge。Host 已满且首次需要 SFU 时,同一 controller 用有限 carrier cursor 建立唯一 publication;carrier 必须能在既有 overlap 上限内完成,不能先切断健康分支。终态归原始等待需求,publication 成功后该需求先取得 SFU 画面,剩余 direct candidate 再在后台收敛;完整事务由 ADR-0005 定义。
- NAT 与传输边界:STUN 必须可用,所有 ordinary peer `RTCPeerConnection` 只接收 STUN ICE servers。Browser LiveKit publisher/subscriber PC 必须显式使用空 `iceServers`,保留 LiveKit signaling 交付的授权 SFU UDP candidates,并由标准 ICE 选择 nominated non-relay pair;不得把 ordinary-peer STUN、LiveKit join response 中的 STUN 或默认公共 STUN混入该 PC,也不得把某一种 candidate type 或数量写成成功条件。部署 STUN 仍供 ordinary peers 和 LiveKit server-side 公网地址发现使用。浏览器 ICE connectivity checks 是 direct UDP 可达性的权威;Screener 不合成 remote candidate、不预测端口、不分类 NAT,也不使用 TCP 探测推断 UDP 媒体可达性。所有恢复和候选授权必须绑定 authenticated room/endpoint session、share generation、base/pending route revision 与 candidate connection identity;成功即停止,失败、超时或绑定失效即撤销,direct 阶段耗尽后在同一 operation deadline 内进入 SFU/UDP,全部可用路径耗尽后明确等待或失败。已接受的完整顺序只有保持或尝试 direct/STUN peers,再复用或建立 direct-ingress Host publication/SFU subscription。Screener 不配置 TURN、ICE/TCP、media TCP 或 TLS-relayed media;当前 LiveKit 也保持 UDP-only。未来若真实严格防火墙矩阵证明需要另一种传输,必须把它作为 LiveKit 内部能力重新决策,不能增加第三种应用 route candidate。LiveKit join JWT 仅按协议进入不得记录的 WebSocket request target;HTTPS/WSS 继续独立使用 TLS/TCP。
- 质量控制:推荐档位始终只有 720p30(3 Mbps)、1080p30(5 Mbps)、1080p60(8 Mbps)三档,默认选择中档 `1080p30`。高级设置的分辨率选项增加 `480p`(854x480),其 FPS 与码率仍由高级控件独立选择;不增加 `480p` 推荐档、preset ID 或固定的 480p FPS/码率组合。三档与高级设置初始值统一采用中档 `balanced`(均衡),`maintain-resolution`(清晰)、`maintain-framerate`(流畅)保留为显式选择。Browser 对每个新分享 source 运行一次有界实际 sender 预检;H.264 源帧与编码帧进度通过时,未来 direct/relay offer 原生排序 H.264 后接 VP8,Host 的唯一 SFU publication 也使用 H.264;不支持、证据不足、超时或表现不达标时只使用 VP8。界面、质量状态和 wire 不提供 codec 选择,结果不持久化,也不切换已连接线路。分享期间可直接换视频档位和 64/128/192 kbps 音频 sender ceiling,音频默认选择中档 128 kbps,不重新选择来源或重建健康连接。Peer-assisted relay 与已配置的 SFU 必须接收同一个严格校验的房间级权威设置,覆盖当前和以后建立的 sender;普通 P2P 信令结构保持不变。任何设置都只是浏览器上限和偏好,具体降级由 WebRTC/LiveKit 决定;必须读回实际 sender 参数并通过 stats 观测结果、显示参数拒绝、改写和持续受限原因。Host 资源不足时由 Host 主动选择较低的现有档位。Screener 不设置自定义 SFU layer list、动态 FPS、综合评分或周期性全房间调档。
- 质量控制:推荐档位始终只有 720p30(3 Mbps)、1080p30(5 Mbps)、1080p60(8 Mbps)三档,默认选择中档 `1080p30`。高级设置的分辨率选项增加 `480p`(854x480),其 FPS 与码率仍由高级控件独立选择;不增加 `480p` 推荐档、preset ID 或固定的 480p FPS/码率组合。三档与高级设置初始值统一采用中档 `balanced`(均衡),`maintain-resolution`(清晰)、`maintain-framerate`(流畅)保留为显式选择。Browser Host 在每个新分享代次运行一次有界实际 sender 预检,Relay 在该代首次取得 upstream source 且尚无 child 时运行同一预检;H.264 源帧与编码帧进度通过时,未来 direct/relay offer 原生排序 H.264 后接 VP8,Host 的唯一 SFU publication 也使用 H.264;不支持、证据不足、超时或表现不达标时只使用 VP8。界面、质量状态和 wire 不提供 codec 选择,结果不持久化,也不因质量设置、换源或换父重复探测或切换已连接线路。分享期间可直接换视频档位和 64/128/192 kbps 音频 sender ceiling,音频默认选择中档 128 kbps,不重新选择来源或重建健康连接。Peer-assisted relay 与已配置的 SFU 必须接收同一个严格校验的房间级权威设置,覆盖当前和以后建立的 sender;普通 P2P 信令结构保持不变。任何设置都只是浏览器上限和偏好,具体降级由 WebRTC/LiveKit 决定;必须读回实际 sender 参数并通过 stats 观测结果、显示参数拒绝、改写和持续受限原因。Host 资源不足时由 Host 主动选择较低的现有档位。Screener 不设置自定义 SFU layer list、动态 FPS、综合评分或周期性全房间调档。
- 路径隔离表示:direct/peer 继续使用每条 PeerConnection 独立的 stock WebRTC GCC,不把 Viewer 反馈合成全房目标。Browser Host 的唯一 SFU publication 不设置 `screenShareSimulcastLayers`,由锁定的 LiveKit client 默认行为拥有当前 H.264 或 VP8 source decision 对应的 screen-share representations、source replacement 与 republish;LiveKit Dynacast 和启用 send-side BWE 的 server stream allocator 负责需求聚合与每个 subscriber 的实际下行层。Screener 不实现质量评分、带宽估计、周期调档或应用媒体层 selector。所有当前 Viewer 都可能承担 peer children,因此 subscriber 保持 `adaptiveStream: false`,不得让本地元素尺寸或可见性控制 relay ingress。表示选择不改变拓扑、端点容量或路由失效权威。
- 路由失效输入:应用不推断 parent-wide 质量,也不计算综合分数。当前 authenticated child 只在 PeerConnection `failed/closed`,或非权威暂停时现有 periodic stats 在统一 deadline 内没有新 decoded frame 时,使自己的 exact edge 失效并唤醒同一个 reconcile。decoded progress 只保存 route-keyed `lastDecodedAt`,route/connection变化或新帧重置、暂停抑制,同一 edge只报告一次,不增加轮询器。Active SFU route 暂时没有 video track、stats report 为空或 stats 读取失败时,现有采样 cadence 仍把该次作为无 decoded progress 推进 deadline;它不得停表、伪造连接指标或立即判定 edge 失效,只有 route 停用或 teardown 才停止该 cadence。该 operation 把失败的 exact direct parent 在当前 fact version 视为已尝试;foreground direct window结束后用SFU取得可用画面,剩余direct candidate只在画面后方收敛,新的外部事实才允许本轮失败tuple重新进入。Track availability、低码率、低 FPS、模糊、sender/viewer stats 与 SFU layer只用于stock WebRTC/LiveKit恢复和诊断。relay自身ingress失效时,它作为child换父并保留subtree;disconnect或effective downstream capacity降低只让direct/overflow children逐个走同一事务。
- 断线恢复:临时网络变化先交给 WebRTC/LiveKit 的 ICE、consent、重连和流状态。Direct edge 可在 exact current parent/child sessions 内采用 framework rebuild 的新 connection ID而不改变topology。应用只在上述exact edge失效后运行一次有界reconcile;任一步成功即停止。候选耗尽时hard-invalid、confirmed-departed或overflow edge被退休并进入明确waiting,healthy bootstrap edge保留。所有异步结果受authenticated session、share generation、base/pending route revision与candidate connection identity约束,未受影响分支保持sticky。权威暂停abort未提交candidate、保留active graph、抑制stall,并在resume后重新reconcile。
@@ -161,7 +161,7 @@ SFU 的 publication、subscription、server-resource accounting、临时 overlap
- Given 同一时刻采集、host outbound 与 viewer inbound 证据,When 进行根因分类,Then 关联派生后的实际协商视频 codec/profile/parameters、适用时的 `scalabilityMode` 与真实解码表现,依次区分 Capture 低、Capture 高但 outbound 因 CPU 低、Capture 高但 outbound 因 bandwidth 低、outbound 正常但 inbound 低、以及 inbound 正常但肉眼模糊五类;不用 UA 或设备型号代替运行时证据,也不上传 raw SDP。
- Given Viewer 上报当前路径质量证据,When 报告未鉴权、过频、重复、过期或没有匹配的 sender/viewer 统计,Then 报告被丢弃;有效报告也只能用于诊断,不能改变 Host 表示或路径状态。
- Given 受控 RTT/丢包测试条件,When 连续分享高动态游戏画面 30 分钟,Then 延迟、帧率与丢帧达到上述目标,且没有不可恢复的冻结。
- Given Windows Chrome/Edge Browser Host 开始或继续分享,When 浏览器返回 display video track,Then 应用设置 `contentHint = "motion"`,并按浏览器 stats 如实显示实际 RTP codec 与实现字段。Given 新 source 的有界 H.264 预检证明实际 `video/H264` sender 的源帧/编码帧进度达标,Then 后续 direct 或 browser-relay offer 按浏览器原生 H.264 capability 顺序列出 H.264,再列 VP8 与修复格式;双方不支持 H.264 时同一次协商选择 VP8。Given 预检不支持、证据不足、超时、失败或不达标,Then offer 只列 VP8 与修复格式。缺少 VP8、`setCodecPreferences()` 或应用失败仍在创建 offer 前明确失败。Given 建立 SFU publication,Then 使用 Host source 的同一 H.264/VP8 决定、禁用 backup codec,且 Screener 不设置自定义 simulcast layer list。
- Given Windows Chrome/Edge Browser Host 开始或继续分享,When 浏览器返回 display video track,Then 应用设置 `contentHint = "motion"`,并按浏览器 stats 如实显示实际 RTP codec 与实现字段。Given 当前分享代次的有界 H.264 预检证明实际 `video/H264` sender 的源帧/编码帧进度达标,Then 后续 direct 或 browser-relay offer 按浏览器原生 H.264 capability 顺序列出 H.264,再列 VP8 与修复格式;双方不支持 H.264 时同一次协商选择 VP8。Given 预检不支持、证据不足、超时、失败或不达标,Then offer 只列 VP8 与修复格式。缺少 VP8、`setCodecPreferences()` 或应用失败仍在创建 offer 前明确失败。Given 建立 SFU publication,Then 使用 Host share generation 的同一 H.264/VP8 决定、禁用 backup codec,且 Screener 不设置自定义 simulcast layer list。
- Given 分享过程中切换 Wi-Fi/有线网络,When ICE 连接失败,Then 客户端触发 ICE restart,并在成功或超时后给出明确状态。
- Given 当前路径仍可播放且控制器准备迁移 logical parent 或替换 transport,When 候选路径拥有模型授权的 overlap 资源,Then 旧路径保持权威,直到 exact candidate child 解码出第一帧新画面后才原子提交;ICE `connected` 单独不能触发切换。
- Given 候选没有准入资源,或 provisional 路径失败、超时、代次过期、参与者变化、房间停止、权限撤销或 session 被替换,When 本次机会结束,Then 原路径和未受影响 branches 继续工作,临时资源被销毁,重试遵守模型定义的有界策略。
@@ -171,7 +171,7 @@ SFU 的 publication、subscription、server-resource accounting、临时 overlap
- Given Web Host 或浏览器 relay 建立或替换分享音轨,When 对实际音频 sender 应用发送参数,Then 从当次 `getParameters()` 派生所选 `encodings[0].maxBitrate = 64000 | 128000 | 192000`、调用 `setParameters()` 并立即读回,档位缺失时按 128000 处理;Given Web Viewer 为含唯一有效 audio/Opus payload 的 offer 生成 answer,Then 用结构化 SDP parser 只 upsert 该 payload 的 `stereo=1` 与 `maxaveragebitrate=192000`,并把同一个结果用于 `setLocalDescription()` 和 signaling。首次连接、ICE restart、同 parent rebuild 和 relay child 都适用;空 SDP、无/歧义 Opus、多个有效 audio、重复目标参数、畸形 fmtp 或 parse/write 异常必须按对象 identity 返回浏览器原 answer 并继续,不能发送部分修改或静默冒充 stereo/实际码率。只有实际 `setLocalDescription()` 失败才进入既有连接错误路径。`sprop-stereo` 与 `usedtx` 不新增,其他 fmtp/FEC/session/media 字段逐项保留。
- Given 通过 SFU 发布分享音轨,When 构建 LiveKit 2.22.0 publish options,Then 64 kbps 使用 `AudioPresets.musicStereo`、128 kbps 使用 `AudioPresets.musicHighQualityStereo`、192 kbps 使用同等结构的显式 sender ceiling,三档都设置 `forceStereo = true`、`dtx = false` 与 `red = false`;验收同时核对 SDK 选项、applied ceiling、当前 RTP codec/fmtp 与 stats 实际码率,不能从 preset 名称或 `opus/48000/2` 单独推断实际声道内容。
- Given 分享正在直播或暂停且 Host 选择新的 64/128/192 kbps 音频档位,When 当前 profile mutation 串行执行,Then 服务端以 last-wins 保存并广播最新 desired profile,每个当前 Host、relay 与 SFU audio sender 从 fresh `getParameters()` 派生 `setParameters()` 并立即 readback,过期 completion 不得覆盖新值;以后创建的 sender 使用最新 desired profile。单个端点应用失败时保留媒体与其旧 applied 值、显示可见告警并允许后续更新或重建重试,不得谎称全房已收敛。当前 applied readback 保持 endpoint-local;没有显式回执 wire 就不宣称跨端点原子提交。该操作不改变 Opus、route revision、media-binding generation 或 publication generation,也不创建第二个音频控制环。当前 screen-audio Host 基线是 Chrome/Edge;其他浏览器只有实际 sender readback 成功时才算支持。
- Given Host 打开已接受的 `screener-v12` 分享设置或更新房间画质,When 查看 UI 或校验消息,Then 不存在 codec selector 或 codec quality-state/wire 字段。source-scoped H.264 预检只产生当前页面内存中的内部发送偏好;结果不持久化、不重新协商健康 edge,实际协商 codec 与 encoder 只作为只读诊断。
- Given Host 打开已接受的 `screener-v12` 分享设置或更新房间画质,When 查看 UI 或校验消息,Then 不存在 codec selector 或 codec quality-state/wire 字段。share-generation-scoped H.264 预检只产生当前页面内存中的内部发送偏好;结果不持久化、不重新协商健康 edge,实际协商 codec 与 encoder 只作为只读诊断。
- Given 浏览器返回屏幕音轨,When 产品展示音频状态或未来连接详情,Then 只能把存在性、浏览器实际返回的 track settings、实际协商 codec/fmtp 和实际流量分别标为捕获、协商或观测事实;字段缺失必须显示为未知,且不得从 Opus 的 `48000/2` RTP 声明推断源采样率或实际 stereo。
- Given Web 或后续 native sender 分享电影/视频音频,When 进行产品验收,Then 必须用独立 A/B/C 证据关联 capture audio track/settings、实际 negotiated codec/fmtp、outbound/inbound 实际码率、loss、jitter、concealment、jitter-buffer 与 A/V sync;P2P、浏览器 relay 与 SFU route switch 保持同一持久 Viewer `MediaStream`/媒体元素和 stereo 协商,不新增独立 audio element 或 Web Audio 时钟边界。native sender 必须通过该音质和同步矩阵,且不得把 audio `maxBitrate` ceiling 当作提质证明。
- Given 分享者同时运行语音软件,When 选择仅分享游戏声音,Then supported Windows 11 native sender 只保留目标游戏及其子进程的渲染音频,观众、通知和其他应用声音不得进入分享;浏览器 system audio 无法满足时必须允许 video-only 或明确告知其可能包含语音/通知,不能用 AEC、降噪或来源标签伪称已隔离。
+7 -2
View File
@@ -48,8 +48,13 @@ function codecName(codec: string): string {
export function codecContractWarnings(metrics: ConnectionMetrics): string[] {
const warnings: string[] = [];
if (metrics.codec && metrics.codec.toLowerCase() !== "video/vp8") {
warnings.push(`视频编码为 ${codecName(metrics.codec)},预期 VP8`);
if (
metrics.codec &&
!["video/h264", "video/vp8"].includes(metrics.codec.toLowerCase())
) {
warnings.push(
`视频编码为 ${codecName(metrics.codec)},预期 H264 或 VP8`,
);
}
if (metrics.audioCodec && metrics.audioCodec.toLowerCase() !== "audio/opus") {
warnings.push(`音频编码为 ${codecName(metrics.audioCodec)},预期 Opus`);
+7 -1
View File
@@ -6,6 +6,7 @@ import type {
} from "../../shared/protocol";
import { countEndpointMediaCopies } from "../../shared/media-copy-accounting";
import { HostPeer } from "../webrtc/host-peer";
import type { BrowserVideoCodec } from "../webrtc/video-codec";
import type { PeerSnapshot } from "../types";
import type { QualityProfile } from "./quality";
@@ -21,6 +22,7 @@ interface HostProvisionalPrepareInput extends HostProvisionalInput {
iceConfig: IceConfig;
stream: MediaStream;
profile: QualityProfile;
videoCodec: BrowserVideoCodec;
}
interface HostProvisionalChildEvents {
@@ -86,7 +88,10 @@ export class HostProvisionalChild {
private startPrepared(
revision: number,
candidate: PreparedRouteCandidate,
input: Pick<HostProvisionalPrepareInput, "iceConfig" | "stream" | "profile">,
input: Pick<
HostProvisionalPrepareInput,
"iceConfig" | "stream" | "profile" | "videoCodec"
>,
): void {
let peer: HostPeer;
peer = new HostPeer(
@@ -113,6 +118,7 @@ export class HostProvisionalChild {
}
},
},
input.videoCodec,
candidate.connectionId,
);
this.signalingPeer = peer;
+9 -2
View File
@@ -10,6 +10,7 @@ import {
type SfuPublisherFailureStage,
} from "../sfu/publisher";
import type { ConnectionMetrics } from "../types";
import type { BrowserVideoCodec } from "../webrtc/video-codec";
import type {
AudioSenderParameterReadback,
QualityProfile,
@@ -25,7 +26,11 @@ import {
interface HostPublisherTransport {
connect(config: SfuConnectionConfig): Promise<boolean>;
activate(stream: MediaStream, profile: QualityProfile): Promise<boolean>;
activate(
stream: MediaStream,
profile: QualityProfile,
videoCodec?: BrowserVideoCodec,
): Promise<boolean>;
deactivate(): Promise<boolean>;
replaceStream(stream: MediaStream): Promise<boolean>;
updateProfile(profile: QualityProfile): Promise<boolean>;
@@ -54,6 +59,7 @@ export interface HostSfuPublisherSnapshot {
interface HostSfuRouteEvents {
getStream: () => MediaStream | null;
getProfile: () => QualityProfile;
getVideoCodec: () => BrowserVideoCodec;
reconcileChildren: (childPeerIds: string[]) => void;
send: (message: ClientMessage) => boolean;
onPublisherUpdate?: (snapshot: HostSfuPublisherSnapshot | null) => void;
@@ -481,12 +487,13 @@ export class HostSfuRoute {
private async preparePublisher(slot: HostPublisherSlot): Promise<boolean> {
const stream = this.events.getStream();
const profile = this.events.getProfile();
const videoCodec = this.events.getVideoCodec();
if (!stream) {
this.handleFailure(slot);
return false;
}
try {
if (!(await slot.publisher.activate(stream, profile))) {
if (!(await slot.publisher.activate(stream, profile, videoCodec))) {
this.handleFailure(slot);
await disconnectPublisher(slot.publisher);
return false;
+40
View File
@@ -120,6 +120,8 @@ import {
MAX_ENDPOINT_MEDIA_CHILDREN,
reconcileBoundedMediaChildren,
} from "../webrtc/media-assignment";
import type { BrowserVideoCodec } from "../webrtc/video-codec";
import { preferredVideoCodecForTrack } from "../webrtc/video-codec-preflight";
import {
hostActionErrorNotice,
hostServerErrorNotice,
@@ -321,6 +323,8 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
const pendingQualityChangeRef = useRef<QualitySettings | null>(null);
const qualitySettingsRef = useRef<QualitySettings>(DEFAULT_QUALITY_SETTINGS);
const advancedQualityRef = useRef<QualitySettings>(advancedQuality);
const videoCodecRef = useRef<BrowserVideoCodec>("vp8");
const codecProbeAbortRef = useRef<AbortController | null>(null);
const sharingPausedRef = useRef(false);
const retiringStreamRef = useRef<MediaStream | null>(null);
const hostSfuRouteRef = useRef<HostSfuRoute | null>(null);
@@ -398,6 +402,8 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
sourceSwitchRef.current = null;
qualityChangeRef.current = null;
pendingQualityChangeRef.current = null;
codecProbeAbortRef.current?.abort();
codecProbeAbortRef.current = null;
stopPreferredRoomRenewal();
signalRef.current?.stop();
peersRef.current.forEach((peer) => peer.dispose());
@@ -432,6 +438,28 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
);
}
async function probeStreamVideoCodec(
stream: MediaStream,
): Promise<BrowserVideoCodec> {
codecProbeAbortRef.current?.abort();
const controller = new AbortController();
codecProbeAbortRef.current = controller;
const track = stream.getVideoTracks()[0];
try {
return track
? await preferredVideoCodecForTrack(
track,
qualitySettingsRef.current,
controller.signal,
)
: "vp8";
} finally {
if (codecProbeAbortRef.current === controller) {
codecProbeAbortRef.current = null;
}
}
}
function stopPreferredRoomRenewal(): void {
if (preferredRoomRenewalTimerRef.current !== null) {
window.clearInterval(preferredRoomRenewalTimerRef.current);
@@ -469,6 +497,7 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
route = new HostSfuRoute({
getStream: () => streamRef.current,
getProfile: () => qualitySettingsRef.current,
getVideoCodec: () => videoCodecRef.current,
reconcileChildren: (childPeerIds) => {
if (
isCurrentGeneration(generation) &&
@@ -522,6 +551,9 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
sourceSwitchRef.current = null;
qualityChangeRef.current = null;
pendingQualityChangeRef.current = null;
codecProbeAbortRef.current?.abort();
codecProbeAbortRef.current = null;
videoCodecRef.current = "vp8";
const signal = signalRef.current;
if (signal) {
if (notifyServer) {
@@ -1002,6 +1034,7 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
iceConfig,
stream,
profile: qualitySettingsRef.current,
videoCodec: videoCodecRef.current,
});
}
@@ -1074,6 +1107,7 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
}
},
},
videoCodecRef.current,
);
peersRef.current.set(peerId, peer);
let started: boolean;
@@ -1448,6 +1482,12 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
setDetails(captureDetails(captured));
watchCaptureEnd(captured, generation);
videoCodecRef.current = await probeStreamVideoCodec(captured);
if (!isCurrentGeneration(generation)) {
captured.getTracks().forEach((track) => track.stop());
return;
}
let createdRoom: HostRoomState | null = null;
let claimedRoom = false;
try {
+19 -5
View File
@@ -23,6 +23,7 @@ import {
type VideoSenderParameterReadback,
} from "../media/quality";
import type { ConnectionMetrics } from "../types";
import type { BrowserVideoCodec } from "../webrtc/video-codec";
import {
captureMetrics,
collectConnectionMetricsFromReport,
@@ -169,7 +170,11 @@ export class SfuPublisher {
}
}
activate(stream: MediaStream, profile: QualityProfile): Promise<boolean> {
activate(
stream: MediaStream,
profile: QualityProfile,
videoCodec: BrowserVideoCodec = "vp8",
): Promise<boolean> {
this.desiredProfile = profile;
this.startupVideoProfilePending = needsStartupVideoProfile(profile);
const effectiveVideoProfile = startupVideoProfile(profile);
@@ -195,7 +200,7 @@ export class SfuPublisher {
room,
videoTrack,
sdk.Track.Source.ScreenShare,
videoPublishOptions(effectiveVideoProfile),
videoPublishOptions(effectiveVideoProfile, videoCodec),
);
if (!this.owns(room, generation)) {
return false;
@@ -1165,7 +1170,7 @@ async function configurePublishedVideo(
const retainedPublishOptions = {
...published.publication.options,
...videoTrack.publishOptions,
...videoPublishOptions(profile),
...videoPublishOptions(profile, publishedVideoCodec(published)),
};
published.publication.options = retainedPublishOptions;
videoTrack.publishOptions = retainedPublishOptions;
@@ -1248,10 +1253,19 @@ function mergeQualityWarnings(...warnings: Array<string | null>): string | null
return present.length > 0 ? present.join(";") : null;
}
function videoPublishOptions(profile: QualityProfile): TrackPublishOptions {
function publishedVideoCodec(published: PublishedTrack): BrowserVideoCodec {
return published.publication.options?.videoCodec === "h264"
? "h264"
: "vp8";
}
function videoPublishOptions(
profile: QualityProfile,
videoCodec: BrowserVideoCodec,
): TrackPublishOptions {
return {
backupCodec: false,
videoCodec: "vp8",
videoCodec,
screenShareEncoding: {
maxBitrate: profile.maxBitrate,
maxFramerate: profile.maxFramerate,
+7 -3
View File
@@ -24,7 +24,10 @@ import {
collectConnectionMetrics,
createStatsAccumulator,
} from "./stats";
import { applyVp8Codec } from "./vp8-codec";
import {
applyVideoCodecPreference,
type BrowserVideoCodec,
} from "./video-codec";
const MAX_PENDING_CANDIDATES = 64;
type PeerIceConfig = Pick<RTCConfiguration, "iceServers">;
@@ -73,6 +76,7 @@ export class HostPeer {
private stream: MediaStream,
private desiredProfile: QualityProfile,
private readonly events: HostPeerEvents,
private readonly videoCodec: BrowserVideoCodec = "vp8",
connectionId = createOpaqueId(),
) {
this.connectionId = connectionId;
@@ -105,8 +109,8 @@ export class HostPeer {
direction: "sendonly",
streams: [this.stream],
});
if (!applyVp8Codec(videoTransceiver)) {
this.setError(null, "当前浏览器无法使用 VP8 视频编码");
if (!applyVideoCodecPreference(videoTransceiver, this.videoCodec)) {
this.setError(null, "当前浏览器无法使用支持的视频编码");
return false;
}
this.videoSender = videoTransceiver.sender;
+276
View File
@@ -0,0 +1,276 @@
import {
configureVideoSender,
startupVideoProfile,
type QualityProfile,
} from "../media/quality";
import {
applyH264ProbeCodec,
type BrowserVideoCodec,
} from "./video-codec";
const PREFLIGHT_DEADLINE_MS = 4_000;
const PREFLIGHT_POLL_MS = 100;
const PREFLIGHT_MIN_SOURCE_FRAMES = 15;
const PREFLIGHT_MAX_FRAME_DEFICIT = 2;
const PREFLIGHT_FPS_WINDOW_MS = 1_000;
export interface H264ProbeSample {
outboundId: string;
sourceId: string;
timestamp: number;
codec: string;
framesEncoded: number | null;
sourceFrames: number | null;
encodedFramesPerSecond: number | null;
sourceFramesPerSecond: number | null;
qualityLimitationReason: string | null;
}
type StatsRecord = RTCStats & Record<string, unknown>;
function finiteNumber(value: unknown): number | null {
return typeof value === "number" && Number.isFinite(value) ? value : null;
}
function stringValue(value: unknown): string | null {
return typeof value === "string" ? value : null;
}
function recordById(
report: RTCStatsReport,
id: unknown,
): StatsRecord | null {
if (typeof id !== "string") {
return null;
}
return (report.get(id) as StatsRecord | undefined) ?? null;
}
function readH264ProbeSample(report: RTCStatsReport): H264ProbeSample | null {
const candidates: StatsRecord[] = [];
report.forEach((raw) => {
const record = raw as StatsRecord;
if (
record.type === "outbound-rtp" &&
record.kind === "video" &&
record.isRemote !== true &&
finiteNumber(record.framesEncoded) !== null
) {
candidates.push(record);
}
});
if (candidates.length !== 1) {
return null;
}
const outbound = candidates[0]!;
const codec = recordById(report, outbound.codecId);
const source = recordById(report, outbound.mediaSourceId);
const mimeType = stringValue(codec?.mimeType);
if (
codec?.type !== "codec" ||
source?.type !== "media-source" ||
mimeType?.toLowerCase() !== "video/h264"
) {
return null;
}
return {
outboundId: outbound.id,
sourceId: source.id,
timestamp: finiteNumber(outbound.timestamp) ?? 0,
codec: mimeType,
framesEncoded: finiteNumber(outbound.framesEncoded),
sourceFrames: finiteNumber(source.frames),
encodedFramesPerSecond: finiteNumber(outbound.framesPerSecond),
sourceFramesPerSecond: finiteNumber(source.framesPerSecond),
qualityLimitationReason: stringValue(outbound.qualityLimitationReason),
};
}
export function h264ProbeSustainsSource(
baseline: H264ProbeSample,
current: H264ProbeSample,
): boolean | null {
if (
baseline.outboundId !== current.outboundId ||
baseline.sourceId !== current.sourceId ||
current.timestamp <= baseline.timestamp ||
current.codec.toLowerCase() !== "video/h264" ||
current.qualityLimitationReason === "cpu"
) {
return false;
}
if (
baseline.framesEncoded !== null &&
current.framesEncoded !== null &&
baseline.sourceFrames !== null &&
current.sourceFrames !== null &&
current.framesEncoded >= baseline.framesEncoded &&
current.sourceFrames >= baseline.sourceFrames
) {
const encoded = current.framesEncoded - baseline.framesEncoded;
const source = current.sourceFrames - baseline.sourceFrames;
if (source < PREFLIGHT_MIN_SOURCE_FRAMES) {
return null;
}
return encoded + PREFLIGHT_MAX_FRAME_DEFICIT >= source;
}
if (
current.timestamp - baseline.timestamp < PREFLIGHT_FPS_WINDOW_MS ||
current.encodedFramesPerSecond === null ||
current.sourceFramesPerSecond === null ||
current.sourceFramesPerSecond <= 0
) {
return null;
}
return (
current.encodedFramesPerSecond + PREFLIGHT_MAX_FRAME_DEFICIT >=
current.sourceFramesPerSecond
);
}
function hasProbeWindow(
baseline: H264ProbeSample,
current: H264ProbeSample,
): boolean {
if (
baseline.outboundId !== current.outboundId ||
baseline.sourceId !== current.sourceId ||
current.timestamp <= baseline.timestamp
) {
return false;
}
if (
baseline.sourceFrames !== null &&
current.sourceFrames !== null &&
current.sourceFrames >= baseline.sourceFrames
) {
return (
current.sourceFrames - baseline.sourceFrames >=
PREFLIGHT_MIN_SOURCE_FRAMES
);
}
return current.timestamp - baseline.timestamp >= PREFLIGHT_FPS_WINDOW_MS;
}
function delay(milliseconds: number, signal?: AbortSignal): Promise<void> {
return new Promise((resolve, reject) => {
if (signal?.aborted) {
reject(signal.reason);
return;
}
const onAbort = () => {
window.clearTimeout(timer);
reject(signal?.reason);
};
const timer = window.setTimeout(() => {
signal?.removeEventListener("abort", onAbort);
resolve();
}, milliseconds);
signal?.addEventListener("abort", onAbort, { once: true });
});
}
async function waitForIceGathering(
connection: RTCPeerConnection,
deadline: number,
signal?: AbortSignal,
): Promise<void> {
while (connection.iceGatheringState !== "complete") {
if (Date.now() >= deadline) {
throw new Error("H.264 preflight timed out");
}
await delay(PREFLIGHT_POLL_MS, signal);
}
}
async function runH264Probe(
track: MediaStreamTrack,
profile: QualityProfile,
signal?: AbortSignal,
): Promise<boolean> {
const senderConnection = new RTCPeerConnection({ iceServers: [] });
const receiverConnection = new RTCPeerConnection({ iceServers: [] });
const abort = () => {
senderConnection.close();
receiverConnection.close();
};
signal?.addEventListener("abort", abort, { once: true });
try {
const transceiver = senderConnection.addTransceiver(track, {
direction: "sendonly",
});
if (!applyH264ProbeCodec(transceiver)) {
return false;
}
await configureVideoSender(
transceiver.sender,
startupVideoProfile(profile),
);
const deadline = Date.now() + PREFLIGHT_DEADLINE_MS;
await senderConnection.setLocalDescription(
await senderConnection.createOffer(),
);
await waitForIceGathering(senderConnection, deadline, signal);
await receiverConnection.setRemoteDescription(
senderConnection.localDescription!,
);
await receiverConnection.setLocalDescription(
await receiverConnection.createAnswer(),
);
await waitForIceGathering(receiverConnection, deadline, signal);
await senderConnection.setRemoteDescription(
receiverConnection.localDescription!,
);
let warmupBaseline: H264ProbeSample | null = null;
let measurementBaseline: H264ProbeSample | null = null;
while (Date.now() < deadline) {
if (signal?.aborted || track.readyState === "ended") {
return false;
}
const sample = readH264ProbeSample(
await transceiver.sender.getStats(),
);
if (sample) {
warmupBaseline ??= sample;
if (
!measurementBaseline &&
sample !== warmupBaseline &&
hasProbeWindow(warmupBaseline, sample)
) {
measurementBaseline = sample;
} else if (measurementBaseline && sample !== measurementBaseline) {
const result = h264ProbeSustainsSource(measurementBaseline, sample);
if (result !== null) {
return result;
}
}
}
await delay(PREFLIGHT_POLL_MS, signal);
}
return false;
} finally {
signal?.removeEventListener("abort", abort);
senderConnection.close();
receiverConnection.close();
}
}
export async function preferredVideoCodecForTrack(
track: MediaStreamTrack,
profile: QualityProfile,
signal?: AbortSignal,
): Promise<BrowserVideoCodec> {
let codec: BrowserVideoCodec = "vp8";
if (track.kind !== "video" || track.readyState === "ended") {
console.debug("[Screener] video codec preflight", codec);
return codec;
}
try {
codec = (await runH264Probe(track, profile, signal)) ? "h264" : "vp8";
} catch {}
if (!signal?.aborted) {
console.debug("[Screener] video codec preflight", codec);
}
return codec;
}
+83
View File
@@ -0,0 +1,83 @@
export type BrowserVideoCodec = "h264" | "vp8";
const REPAIR_CODEC_MIME_TYPES = new Set([
"video/rtx",
"video/red",
"video/ulpfec",
"video/flexfec-03",
]);
function videoCapabilities(): RTCRtpCodec[] {
return RTCRtpSender.getCapabilities?.("video")?.codecs ?? [];
}
function vp8Codecs(codecs: readonly RTCRtpCodec[]): RTCRtpCodec[] {
return codecs.filter(
({ mimeType }) => mimeType.toLowerCase() === "video/vp8",
);
}
function h264ModeOneCodecs(codecs: readonly RTCRtpCodec[]): RTCRtpCodec[] {
return codecs.filter(({ mimeType, sdpFmtpLine }) => {
if (mimeType.toLowerCase() !== "video/h264") {
return false;
}
return (
!sdpFmtpLine ||
/(?:^|;)\s*packetization-mode=1(?:;|$)/i.test(sdpFmtpLine)
);
});
}
function repairCodecs(codecs: readonly RTCRtpCodec[]): RTCRtpCodec[] {
return codecs.filter(({ mimeType }) =>
REPAIR_CODEC_MIME_TYPES.has(mimeType.toLowerCase()),
);
}
function applyCodecs(
transceiver: RTCRtpTransceiver,
codecs: RTCRtpCodec[],
): boolean {
if (typeof transceiver.setCodecPreferences !== "function") {
return false;
}
try {
transceiver.setCodecPreferences(codecs);
return true;
} catch {
return false;
}
}
export function applyVideoCodecPreference(
transceiver: RTCRtpTransceiver,
preference: BrowserVideoCodec,
): boolean {
if (typeof RTCRtpSender === "undefined") {
return false;
}
const codecs = videoCapabilities();
const vp8 = vp8Codecs(codecs);
if (vp8.length === 0) {
return false;
}
const h264 = preference === "h264" ? h264ModeOneCodecs(codecs) : [];
return applyCodecs(transceiver, [
...h264,
...vp8,
...repairCodecs(codecs),
]);
}
export function applyH264ProbeCodec(
transceiver: RTCRtpTransceiver,
): boolean {
if (typeof RTCRtpSender === "undefined") {
return false;
}
const codecs = videoCapabilities();
const h264 = h264ModeOneCodecs(codecs);
return h264.length > 0 &&
applyCodecs(transceiver, [...h264, ...repairCodecs(codecs)]);
}
+102 -1
View File
@@ -7,6 +7,8 @@ import type { QualityProfile } from "../media/quality";
import type { PeerSnapshot } from "../types";
import { HostPeer } from "./host-peer";
import { MAX_ENDPOINT_MEDIA_CHILDREN } from "./media-assignment";
import type { BrowserVideoCodec } from "./video-codec";
import { preferredVideoCodecForTrack } from "./video-codec-preflight";
interface ViewerRelayEvents {
sendSignal: (peerId: string, payload: SignalPayload) => boolean;
@@ -31,6 +33,11 @@ export class ViewerRelay {
private preparedCandidate: PreparedRouteCandidate | null = null;
private plannedChildPeerIds: string[] = [];
private syncQueue = Promise.resolve();
private videoCodec: BrowserVideoCodec = "vp8";
private codecProbeTrack: MediaStreamTrack | null = null;
private codecProbeAbort: AbortController | null = null;
private codecProbePromise: Promise<void> | null = null;
private codecProbeSettled = false;
private disposed = false;
constructor(
@@ -84,7 +91,14 @@ export class ViewerRelay {
this.preparedCandidate = { ...candidate };
this.plannedChildPeerIds = planned;
if (candidate.transport === "direct") {
this.startPreparedChild(revision, candidate, stream);
const pendingProbe = this.codecProbePromise;
if (pendingProbe) {
void pendingProbe.then(() => {
this.startPreparedChildIfCurrent(revision, candidate, stream);
});
} else {
this.startPreparedChildIfCurrent(revision, candidate, stream);
}
}
return true;
}
@@ -126,6 +140,7 @@ export class ViewerRelay {
this.preparedCandidate = null;
this.plannedChildPeerIds = [];
this.discardPreparedPeer();
this.startCodecProbe();
}
private discardPreparedPeer(): void {
@@ -155,6 +170,9 @@ export class ViewerRelay {
}
}
this.childPeerIds = nextChildPeerIds;
if (this.childPeerIds.length > 0) {
this.cancelCodecProbe();
}
for (const childPeerId of this.childPeerIds) {
const peer = this.peers.get(childPeerId);
if (
@@ -166,17 +184,27 @@ export class ViewerRelay {
this.startPeer(childPeerId, this.stream);
}
}
this.startCodecProbe();
}
setStream(stream: MediaStream): void {
if (this.disposed) {
return;
}
const nextVideoTrack = stream.getVideoTracks()[0] ?? null;
if (nextVideoTrack !== this.codecProbeTrack) {
this.cancelCodecProbe();
this.codecProbeTrack = nextVideoTrack;
if (!this.codecProbeSettled) {
this.videoCodec = "vp8";
}
}
const preparedPeer = this.preparedChild?.peer ?? null;
this.stream = stream;
this.syncQueue = this.syncQueue
.then(() => this.syncStream(stream, preparedPeer))
.catch(() => undefined);
this.startCodecProbe();
}
updateProfile(profile: QualityProfile): Promise<boolean> {
@@ -271,6 +299,10 @@ export class ViewerRelay {
}
stop(): void {
this.cancelCodecProbe();
this.codecProbeTrack = null;
this.codecProbeSettled = false;
this.videoCodec = "vp8";
this.stream = null;
this.discardPreparedChild();
this.disposePeers();
@@ -281,6 +313,8 @@ export class ViewerRelay {
return;
}
this.disposed = true;
this.cancelCodecProbe();
this.codecProbeTrack = null;
this.stream = null;
this.discardPreparedChild();
this.childPeerIds = [];
@@ -316,6 +350,23 @@ export class ViewerRelay {
});
}
private startPreparedChildIfCurrent(
revision: number,
candidate: PreparedRouteCandidate,
stream: MediaStream,
): void {
if (
this.disposed ||
this.stream !== stream ||
this.preparedChild ||
this.preparedRevision !== revision ||
!sameCandidate(this.preparedCandidate, candidate)
) {
return;
}
this.startPreparedChild(revision, candidate, stream);
}
private async syncStream(
stream: MediaStream,
preparedPeer: HostPeer | null,
@@ -475,11 +526,61 @@ export class ViewerRelay {
}
},
},
this.videoCodec,
connectionId,
);
return peer;
}
private startCodecProbe(): void {
const track = this.codecProbeTrack;
if (
this.disposed ||
this.codecProbeSettled ||
this.codecProbeAbort ||
this.codecProbePromise ||
!track ||
track.readyState === "ended" ||
this.childPeerIds.length > 0 ||
this.preparedChild
) {
return;
}
const controller = new AbortController();
this.codecProbeAbort = controller;
const promise = preferredVideoCodecForTrack(
track,
this.desiredProfile,
controller.signal,
)
.then((codec) => {
if (
this.codecProbeAbort !== controller ||
controller.signal.aborted ||
this.codecProbeTrack !== track ||
this.disposed
) {
return;
}
this.videoCodec = codec;
this.codecProbeSettled = true;
this.codecProbeAbort = null;
})
.finally(() => {
if (this.codecProbePromise === promise) {
this.codecProbePromise = null;
this.startCodecProbe();
}
});
this.codecProbePromise = promise;
}
private cancelCodecProbe(): void {
const controller = this.codecProbeAbort;
this.codecProbeAbort = null;
controller?.abort();
}
private failPreparedChild(peer: HostPeer): void {
const prepared = this.preparedChild;
if (prepared?.peer !== peer) {
-35
View File
@@ -1,35 +0,0 @@
const REPAIR_CODEC_MIME_TYPES = new Set([
"video/rtx",
"video/red",
"video/ulpfec",
"video/flexfec-03",
]);
export function applyVp8Codec(transceiver: RTCRtpTransceiver): boolean {
if (
typeof RTCRtpSender === "undefined" ||
typeof transceiver.setCodecPreferences !== "function"
) {
return false;
}
try {
const codecs = RTCRtpSender.getCapabilities?.("video")?.codecs ?? [];
const vp8Codecs = codecs.filter(
({ mimeType }) => mimeType.toLowerCase() === "video/vp8",
);
if (vp8Codecs.length === 0) {
return false;
}
transceiver.setCodecPreferences([
...vp8Codecs,
...codecs.filter(({ mimeType }) =>
REPAIR_CODEC_MIME_TYPES.has(mimeType.toLowerCase()),
),
]);
return true;
} catch {
return false;
}
}
+87 -3
View File
@@ -7,12 +7,21 @@ import {
import { HostProvisionalChild } from "../src/client/media/host-provisional-child.ts";
import type { PeerSnapshot } from "../src/client/types.ts";
import { HostPeer } from "../src/client/webrtc/host-peer.ts";
import type { BrowserVideoCodec } from "../src/client/webrtc/video-codec.ts";
import { ViewerRelay } from "../src/client/webrtc/viewer-relay.ts";
import type {
IceConfig,
ParticipantRouteAssignment,
} from "../src/shared/protocol.ts";
const codecPreflight = vi.hoisted(() => ({
probe: vi.fn(async (): Promise<"h264" | "vp8"> => "vp8"),
}));
vi.mock("../src/client/webrtc/video-codec-preflight.ts", () => ({
preferredVideoCodecForTrack: codecPreflight.probe,
}));
const statsCallbacks: Array<() => void> = [];
class FakeSender {
@@ -387,6 +396,7 @@ function createPeer(
onUpdate: (snapshot: PeerSnapshot) => void = () => undefined,
iceConfig: IceConfig = { iceServers: [] },
profile: QualityProfile = QUALITY_PROFILES["720p30"],
videoCodec: BrowserVideoCodec = "vp8",
): HostPeer {
return new HostPeer(
"viewer-peer",
@@ -397,6 +407,7 @@ function createPeer(
sendSignal: () => true,
onUpdate,
},
videoCodec,
);
}
@@ -434,6 +445,8 @@ async function completeVideoStartup(
}
beforeEach(() => {
codecPreflight.probe.mockReset();
codecPreflight.probe.mockResolvedValue("vp8");
FakePeerConnection.latest = null;
FakePeerConnection.instances = [];
FakePeerConnection.activeCount = 0;
@@ -519,6 +532,50 @@ describe("HostPeer source replacement", () => {
expect(connection.createOfferCallCount).toBe(1);
});
it("prefers native H264 mode 1 and retains VP8 fallback", async () => {
vi.stubGlobal("RTCRtpSender", {
getCapabilities: () => ({
codecs: [
{
mimeType: "video/H264",
clockRate: 90_000,
sdpFmtpLine: "packetization-mode=0;profile-level-id=42001f",
},
{
mimeType: "video/H264",
clockRate: 90_000,
sdpFmtpLine: "packetization-mode=1;profile-level-id=42001f",
},
{ mimeType: "video/VP8", clockRate: 90_000 },
{ mimeType: "video/rtx", clockRate: 90_000 },
{ mimeType: "video/RED", clockRate: 90_000 },
],
headerExtensions: [],
}),
});
const peer = createPeer(
createStream(createTrack("video", "video"), null),
() => undefined,
{ iceServers: [] },
QUALITY_PROFILES["720p30"],
"h264",
);
await expect(peer.start()).resolves.toBe(true);
expect(
FakePeerConnection.latest!.codecPreferenceCalls[0]?.map((codec) => [
codec.mimeType.toLowerCase(),
codec.sdpFmtpLine ?? null,
]),
).toEqual([
["video/h264", "packetization-mode=1;profile-level-id=42001f"],
["video/vp8", null],
["video/rtx", null],
["video/red", null],
]);
});
it("fails before creating an offer when codec preferences are unavailable", async () => {
FakePeerConnection.omitCodecPreferenceSetter = true;
const updates: PeerSnapshot[] = [];
@@ -531,7 +588,7 @@ describe("HostPeer source replacement", () => {
expect(FakePeerConnection.latest!.createOfferCallCount).toBe(0);
expect(FakePeerConnection.latest!.localDescription).toBeNull();
expect(updates.at(-1)?.error).toBe("当前浏览器无法使用 VP8 视频编码");
expect(updates.at(-1)?.error).toBe("当前浏览器无法使用支持的视频编码");
});
it("fails before creating an offer when VP8 is unavailable", async () => {
@@ -554,7 +611,7 @@ describe("HostPeer source replacement", () => {
expect(FakePeerConnection.latest!.createOfferCallCount).toBe(0);
expect(FakePeerConnection.latest!.localDescription).toBeNull();
expect(updates.at(-1)?.error).toBe("当前浏览器无法使用 VP8 视频编码");
expect(updates.at(-1)?.error).toBe("当前浏览器无法使用支持的视频编码");
});
it("fails before creating an offer when codec preference setup fails", async () => {
@@ -569,7 +626,7 @@ describe("HostPeer source replacement", () => {
expect(FakePeerConnection.latest!.createOfferCallCount).toBe(0);
expect(FakePeerConnection.latest!.localDescription).toBeNull();
expect(updates.at(-1)?.error).toBe("当前浏览器无法使用 VP8 视频编码");
expect(updates.at(-1)?.error).toBe("当前浏览器无法使用支持的视频编码");
});
it("applies STUN-only ICE configuration at creation and update", () => {
@@ -1384,6 +1441,7 @@ function hostProvisionalInput(
iceConfig: { iceServers: [] },
stream,
profile: QUALITY_PROFILES["720p30"],
videoCodec: "vp8" as const,
};
}
@@ -1516,6 +1574,29 @@ describe("ViewerRelay downstream ownership", () => {
transport: "direct" as const,
});
it("uses a completed H264 probe only for a future child", async () => {
codecPreflight.probe.mockResolvedValueOnce("h264");
const relay = new ViewerRelay(
{ iceServers: [] },
QUALITY_PROFILES["720p30"],
{ sendSignal: () => true },
);
relay.setStream(createStream(createTrack("video", "h264-source"), null));
await vi.waitFor(() => expect(codecPreflight.probe).toHaveBeenCalledOnce());
await Promise.resolve();
relay.setChildren(["h264-child"]);
await vi.waitFor(() =>
expect(FakePeerConnection.latest?.codecPreferenceCalls).toHaveLength(1),
);
expect(
FakePeerConnection.latest!.codecPreferenceCalls[0]?.map(({ mimeType }) =>
mimeType.toLowerCase(),
),
).toEqual(["video/h264", "video/vp8"]);
relay.dispose();
});
it("promotes the exact prepared child connection within the current Viewer cap", async () => {
const signals: Array<{ peerId: string; connectionId: string }> = [];
const relay = new ViewerRelay(
@@ -1593,6 +1674,7 @@ describe("ViewerRelay downstream ownership", () => {
);
relay.setStream(createStream(createTrack("video", "rollback-video"), null));
expect(relay.prepareChild(7, routeCandidate(7, "first-probe"), ["first-probe"])).toBe(true);
await vi.waitFor(() => expect(FakePeerConnection.latest).not.toBeNull());
const firstProbe = FakePeerConnection.latest!;
expect(relay.prepareChild(8, routeCandidate(8, "second-probe"), ["second-probe"])).toBe(true);
const secondProbe = FakePeerConnection.latest!;
@@ -1675,6 +1757,7 @@ describe("ViewerRelay downstream ownership", () => {
["prepared-child"],
),
).toBe(true);
await vi.waitFor(() => expect(FakePeerConnection.latest).not.toBeNull());
const preparedConnection = FakePeerConnection.latest!;
expect(preparedConnection.senders[1]?.track).toBeNull();
@@ -1701,6 +1784,7 @@ describe("ViewerRelay downstream ownership", () => {
);
relay.setStream(createStream(createTrack("video", "cap-video"), null));
expect(relay.prepareChild(1, routeCandidate(1, "child-0"), ["child-0"])).toBe(true);
await vi.waitFor(() => expect(FakePeerConnection.latest).not.toBeNull());
const preparedConnection = FakePeerConnection.latest!;
preparedConnection.connectionState = "connected";
relay.activateChildren(1, ["child-0"]);
+8
View File
@@ -259,6 +259,7 @@ describe("minimal route transition contracts", () => {
const route = new HostSfuRoute({
getStream: () => ({}) as MediaStream,
getProfile: () => QUALITY_PROFILES["720p30"],
getVideoCodec: () => "h264",
reconcileChildren: () => undefined,
send: (message) => {
messages.push(message);
@@ -281,6 +282,11 @@ describe("minimal route transition contracts", () => {
});
await route.acceptConfig(sfuConfig(1));
expect(publishers[0]?.activate).toHaveBeenCalledOnce();
expect(publishers[0]?.activate).toHaveBeenCalledWith(
expect.anything(),
QUALITY_PROFILES["720p30"],
"h264",
);
route.accept({
revision: 2,
@@ -311,6 +317,7 @@ describe("minimal route transition contracts", () => {
const route = new HostSfuRoute({
getStream: () => ({}) as MediaStream,
getProfile: () => QUALITY_PROFILES["720p30"],
getVideoCodec: () => "vp8",
reconcileChildren: (children) => reconciledChildren.push(children),
send: () => true,
createPublisher: () => {
@@ -365,6 +372,7 @@ describe("minimal route transition contracts", () => {
const route = new HostSfuRoute({
getStream: () => ({}) as MediaStream,
getProfile: () => QUALITY_PROFILES["720p30"],
getVideoCodec: () => "vp8",
reconcileChildren: () => undefined,
send: () => true,
createPublisher: () => {
+22
View File
@@ -1,5 +1,6 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { QUALITY_PROFILES } from "../src/client/media/quality.ts";
import { SfuPublisher } from "../src/client/sfu/publisher.ts";
import { SfuSubscriber } from "../src/client/sfu/subscriber.ts";
import type { ConnectionMetrics } from "../src/client/types.ts";
@@ -878,6 +879,27 @@ describe("SfuPublisher", () => {
});
});
it("publishes one H264 source without a backup codec", async () => {
const publisher = new SfuPublisher();
const video = track("video", "video-h264");
await publisher.connect(connection);
await expect(
publisher.activate(stream(video), qualityProfile, "h264"),
).resolves.toBe(true);
const publication = livekit.state.rooms[0].localParticipant.publications[0];
expect(publication.options).toMatchObject({
source: Track.Source.ScreenShare,
backupCodec: false,
videoCodec: "h264",
});
await expect(
publisher.updateProfile(QUALITY_PROFILES["720p30"]),
).resolves.toBe(true);
expect(publication.options.videoCodec).toBe("h264");
});
it("keeps media and prior audio readback when a live update fails", async () => {
const publisher = new SfuPublisher();
await publisher.connect(connection);
+2 -2
View File
@@ -107,14 +107,14 @@ describe("StatsGrid progressive disclosure", () => {
createElement(StatsGrid, {
metrics: {
...metrics,
codec: "video/H264",
codec: "video/VP9",
audioCodec: "audio/PCMU",
},
direction: "receive",
}),
);
expect(html).toContain("视频编码为 H264,预期 VP8");
expect(html).toContain("视频编码为 VP9,预期 H264 或 VP8");
expect(html).toContain("音频编码为 PCMU,预期 Opus");
expect(html).not.toContain("视频 Codec");
expect(html).not.toContain("音频 Codec");
+128
View File
@@ -0,0 +1,128 @@
import { describe, expect, it } from "vitest";
import {
h264ProbeSustainsSource,
type H264ProbeSample,
} from "../src/client/webrtc/video-codec-preflight.ts";
function sample(
overrides: Partial<H264ProbeSample> = {},
): H264ProbeSample {
return {
outboundId: "outbound",
sourceId: "source",
timestamp: 1_000,
codec: "video/H264",
framesEncoded: 10,
sourceFrames: 10,
encodedFramesPerSecond: 30,
sourceFramesPerSecond: 30,
qualityLimitationReason: "none",
...overrides,
};
}
describe("H264 sender preflight", () => {
it("accepts bounded pipeline lag against the same source", () => {
const baseline = sample();
const current = sample({
timestamp: 2_000,
framesEncoded: 39,
sourceFrames: 40,
});
expect(h264ProbeSustainsSource(baseline, current)).toBe(true);
});
it("rejects sustained encoder frame dropping", () => {
const baseline = sample();
const current = sample({
timestamp: 2_000,
framesEncoded: 22,
sourceFrames: 40,
encodedFramesPerSecond: 12,
sourceFramesPerSecond: 30,
});
expect(h264ProbeSustainsSource(baseline, current)).toBe(false);
});
it("waits for enough source progress before deciding", () => {
const baseline = sample();
const current = sample({
timestamp: 1_400,
framesEncoded: 20,
sourceFrames: 20,
});
expect(h264ProbeSustainsSource(baseline, current)).toBeNull();
});
it("rejects CPU-limited or identity-mismatched samples", () => {
const baseline = sample();
expect(
h264ProbeSustainsSource(
baseline,
sample({
timestamp: 2_000,
framesEncoded: 40,
sourceFrames: 40,
qualityLimitationReason: "cpu",
}),
),
).toBe(false);
expect(
h264ProbeSustainsSource(
baseline,
sample({
outboundId: "replacement",
timestamp: 2_000,
framesEncoded: 40,
sourceFrames: 40,
}),
),
).toBe(false);
});
it("uses reported cadence when cumulative source frames are unavailable", () => {
const baseline = sample({ framesEncoded: null, sourceFrames: null });
expect(
h264ProbeSustainsSource(
baseline,
sample({
timestamp: 2_000,
framesEncoded: null,
sourceFrames: null,
encodedFramesPerSecond: 29,
sourceFramesPerSecond: 30,
}),
),
).toBe(true);
expect(
h264ProbeSustainsSource(
baseline,
sample({
timestamp: 2_000,
framesEncoded: null,
sourceFrames: null,
encodedFramesPerSecond: 12,
sourceFramesPerSecond: 30,
}),
),
).toBe(false);
expect(
h264ProbeSustainsSource(
baseline,
sample({
timestamp: 1_500,
framesEncoded: null,
sourceFrames: null,
encodedFramesPerSecond: 30,
sourceFramesPerSecond: 30,
}),
),
).toBeNull();
});
});