diff --git a/.env.example b/.env.example index a06eca9..3b71f8a 100644 --- a/.env.example +++ b/.env.example @@ -4,8 +4,9 @@ LISTEN_HOST=0.0.0.0 PORT=8787 # For LAN viewers, replace localhost with this computer's LAN address. PUBLIC_BASE_URL=http://localhost:8787 -# Include both the host browser origin and PUBLIC_BASE_URL when they differ. -ALLOWED_ORIGINS=http://localhost:8787 +# Leave empty to follow PUBLIC_BASE_URL. For multiple trusted browser addresses, +# list all exact origins (scheme, host and port), including PUBLIC_BASE_URL. +ALLOWED_ORIGINS= # Optional site access password in every environment. Unset or empty allows # entry without a site password; room ownership and invitations still apply. # Non-empty values are matched exactly; spaces and Unicode are supported. diff --git a/deploy/nginx/piik.conf.example b/deploy/nginx/piik.conf.example index a0bfb3f..01ddead 100644 --- a/deploy/nginx/piik.conf.example +++ b/deploy/nginx/piik.conf.example @@ -37,7 +37,7 @@ server { add_header X-Content-Type-Options "nosniff" always; add_header Referrer-Policy "no-referrer" always; add_header X-Frame-Options "DENY" always; - add_header Permissions-Policy "camera=(), microphone=(), geolocation=()" always; + add_header Permissions-Policy "camera=(self), microphone=(self), geolocation=()" always; # Media uses /signal. Reject credential-bearing RTC query endpoints without # persisting their query strings, including requests from already-open pages. diff --git a/docs/README.md b/docs/README.md index 3ef26b1..9b51ca1 100644 --- a/docs/README.md +++ b/docs/README.md @@ -98,7 +98,8 @@ and [website preview and publishing](./operations/website.md). | Host and viewer behavior | [Presentation and lifecycle](./standards/presentation-lifecycle.md) | | Why a design was chosen | [Architecture decisions](./adr/) | | Measurements and platform limits | [Research](./research/) and [verification status](./verification-status.md) | -| Camera and Host microphone candidate | [Capture scope and limits](./research/camera-and-microphone.md) | +| Camera and Host microphone | [Capture scope and limits](./research/camera-and-microphone.md) | +| Room voice proposal and media-fidelity boundaries | [Design assessment](./research/room-voice.md) | ## Shared conventions diff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.md index fc7cc0d..b5ae60e 100644 --- a/docs/guide/troubleshooting.md +++ b/docs/guide/troubleshooting.md @@ -16,6 +16,7 @@ collect a diagnostic report if the problem persists. | Yellow outline around the shared window or screen | This is Windows' capture indicator. See [capture borders](#yellow-capture-border-on-windows) for Windows 11 controls and an optional Windows 10 workaround. | | App startup fails | Read the reason on the page and in the terminal. Reopen the App and enable the chip-shaped **Debug launch** control after the theme button before trying again. A failed startup then exports a report; its path appears in the terminal. | | Local invitation will not open | Check that both devices are on the same network and can reach each other. Guest Wi-Fi or firewall rules can block local access. | +| Cannot create a room (403) | On a self-hosted site, ask the administrator to check the [allowed site address](#room-creation-returns-403). | | Page opens but video will not connect | Use **Reconnect** if available, or refresh the viewing page. See [connection troubleshooting](#when-video-will-not-connect) if it still fails. | | Sharing stops after sleep or suspension | Wake the device and return to the sharing tab; start sharing again if needed. Browser and OS suspension can interrupt capture or playback. | @@ -23,6 +24,24 @@ For a bug report, include the version, OS/browser, what you expected, and how to reproduce it. [Diagnostics and export](../../cmd/piik-app/README.md#diagnostics) explains how to collect a local report and what to review before sharing it. +## Room creation returns 403 + +Piik rejects room creation from a browser address that the server has not allowed, +even when the page opens normally. The site administrator should: + +1. Set `PUBLIC_BASE_URL` to the browser's site address, including the scheme and + any non-default port, without a path. +2. Unset or empty `ALLOWED_ORIGINS` to use that address. If several addresses are + needed, list every trusted origin explicitly, separated by commas. Check for + an old `http://localhost:8787` override after copying a configuration example. +3. Restart Piik (recreate the container with `docker compose up -d` for Compose), + reload the page and retry. + +If the response says `Origin not allowed` with correct settings, check that the +reverse proxy preserves the browser's `Origin` header. A 403 from the proxy or +WAF itself needs its own logs; do not disable origin validation to bypass it. +See [Server deployment](../operations/self-hosting.md) for the configuration. + ## App windows or screens are missing Keep Piik App running on the computer where you are sharing. Open the current diff --git a/docs/guide/troubleshooting.zh-CN.md b/docs/guide/troubleshooting.zh-CN.md index 322ccfd..92d197b 100644 --- a/docs/guide/troubleshooting.zh-CN.md +++ b/docs/guide/troubleshooting.zh-CN.md @@ -16,6 +16,7 @@ | 分享的窗口或屏幕周围出现黄框 | 这是 Windows 的采集提示。查看[采集边框说明](#windows-分享时出现黄框)，了解 Windows 11 的开关和 Windows 10 的可选处理方法。 | | App 启动失败 | 先查看页面和终端中的具体原因。重新打开 App，在主题按钮后的芯片图标开启「诊断启动」再重试。启动失败时会自动导出报告，保存位置见终端。 | | 本地邀请打不开 | 确认两台设备在同一网络且可以互相访问。访客 Wi-Fi 或防火墙规则可能阻止本地连接。 | +| 无法创建房间 (403) | 自建站点请联系管理员检查[允许访问的站点地址](#创建房间返回-403)。 | | 网页能打开，画面连不上 | **重新连接** 按钮可用时先点击，否则刷新观看页。仍然失败时，查看 [画面连接排查](#画面连接不上)。 | | 休眠或挂起后分享中断 | 唤醒设备，回到分享标签页，必要时重新开始分享。浏览器或系统挂起可能中断采集和播放。 | @@ -23,6 +24,20 @@ [诊断与导出](../../cmd/piik-app/README.zh-CN.md#诊断) 说明如何收集本地报告，以及分享前需要检查哪些内容。 +## 创建房间返回 403 + +浏览器访问的网址不在服务端允许的范围内时，页面可能正常打开，但创建房间会被拒绝。 +站点管理员可以按以下步骤检查： + +1. 将 `PUBLIC_BASE_URL` 设为浏览器访问的站点地址，包含协议和非默认端口，不带路径。 +2. `ALLOWED_ORIGINS` 不设置或留空时使用上述地址；需要多个地址时，用英文逗号列出全部 + 可信地址。复制过配置示例的话，检查这里是否仍残留 `http://localhost:8787`。 +3. 重启 Piik；使用 Compose 时运行 `docker compose up -d` 重建容器。刷新网页后重试。 + +配置正确但响应仍是 `Origin not allowed` 时，检查反向代理是否保留浏览器的 `Origin` +请求头。如果 403 来自代理或 WAF 本身，则应查看对应日志，不要通过关闭来源校验来绕过。 +配置方法见 [Server 部署指南](../operations/self-hosting.zh-CN.md)。 + ## App 找不到窗口或屏幕 在分享画面的这台电脑上保持 Piik App 运行，并从 App 打开当前站点。 diff --git a/docs/operations/self-hosting.md b/docs/operations/self-hosting.md index f95dda1..fb73ea0 100644 --- a/docs/operations/self-hosting.md +++ b/docs/operations/self-hosting.md @@ -76,6 +76,11 @@ working directory. A blank `SITE_ACCESS_PASSWORD` allows entry without a site passphrase; enter a passphrase to require one. Room invitations and access settings still apply. +`PUBLIC_BASE_URL` must match the address opened in the browser, including HTTPS +and any non-default port. Leave `ALLOWED_ORIGINS` unset or empty for this single +address. An existing non-empty value overrides that default; a stale value can +cause [403 when creating a room](../guide/troubleshooting.md#room-creation-returns-403). + `MAX_VIEWERS_PER_ROOM` sets the room's Viewer limit, excluding the Host. Choose `1..20` and restart to apply it. More Viewers can require more network and relay resources. See [room capacity](../standards/configuration.md#room-capacity) @@ -96,6 +101,12 @@ share.example.com { Reload Caddy. It obtains and renews the certificate automatically when DNS points to the server and TCP 80/443 are reachable. See [Caddy's HTTPS proxy guide](https://caddyserver.com/docs/quick-starts/reverse-proxy#https). +For nginx, see the [configuration example](../../deploy/nginx/piik.conf.example). +If you copied an older example, change its `Permissions-Policy` to +`camera=(self), microphone=(self), geolocation=()` and reload nginx. Empty +camera/microphone allowlists prevent Browser capture even when the user grants +permission; `self` allows this site to request access. + ### 3. Open the ports and verify Allow **TCP 80/443** for HTTPS and **UDP 3478** for STUN in the server firewall diff --git a/docs/operations/self-hosting.zh-CN.md b/docs/operations/self-hosting.zh-CN.md index f9396bf..9ad9cdd 100644 --- a/docs/operations/self-hosting.zh-CN.md +++ b/docs/operations/self-hosting.zh-CN.md @@ -71,6 +71,10 @@ SITE_ACCESS_PASSWORD= `SITE_ACCESS_PASSWORD` 留空时，进入站点无须口令； 如需设置口令，填写你想使用的口令即可。房间邀请与加入权限仍由房主管理。 +`PUBLIC_BASE_URL` 应与浏览器访问的网址一致，包括 HTTPS 和非默认端口。 +只使用这个地址时，`ALLOWED_ORIGINS` 不设置或留空即可；已有的非空值会覆盖此默认值， +旧地址未更新可能导致[创建房间返回 403](../guide/troubleshooting.zh-CN.md#创建房间返回-403)。 + `MAX_VIEWERS_PER_ROOM` 可设置每房观众上限，不含房主，支持 `1..20`，修改后重启生效。 人数越多，对网络和转发资源的需求也可能增加。默认值及 App 房间的区别见 [人数限制](../standards/configuration.md#room-capacity)。 @@ -89,6 +93,10 @@ share.example.com { 重新加载 Caddy。域名解析正确且 TCP 80/443 可访问时，它会自动申请和续期证书。 具体操作见 [Caddy 的 HTTPS 代理说明](https://caddyserver.com/docs/quick-starts/reverse-proxy#https)。 +nginx 可参考[配置示例](../../deploy/nginx/piik.conf.example)。如果沿用了旧示例，请将 +`Permissions-Policy` 改为 `camera=(self), microphone=(self), geolocation=()`，然后重新加载 nginx。 +摄像头和麦克风的空白允许列表会阻止浏览器采集，即使用户已授权；`self` 允许本站请求访问。 + ### 3. 放行端口并检查 在服务器防火墙和云平台安全组中放行 **TCP 80/443**（HTTPS）和 **UDP 3478**（STUN）。 diff --git a/docs/research/browser-local-encoding-pool.md b/docs/research/browser-local-encoding-pool.md index 7289a6c..70188e0 100644 --- a/docs/research/browser-local-encoding-pool.md +++ b/docs/research/browser-local-encoding-pool.md @@ -93,6 +93,64 @@ maximum queued service; reverse feedback is unshaped. They are not a model of public NAT, geographic RTT or every congestion/loss pattern. Reduced quality and temporary stalls under real constraints remain possible. +### Balanced Startup And Recovery + +Serial Chrome 152 checks on 2026-09-25 reproduced two composition defects in +VP8, 1080p30, motion + balanced, with a 5 Mbps ceiling. A forward-only UDP +shaper limited A to 400 kbps; B remained unshaped. These synthetic-source runs +establish delivered dimensions and recovery, not game perceptual quality or the +cause of reports without diagnostics. + +- A group's local warmup spent its five-frame startup protection before any + child published its frames. Restoring balanced then allowed native initial + downscaling against an untrained allocation. Protection now begins at the + first committed outgoing frame; paused/empty frames cannot spend it and later + children cannot reset it. The frame threshold and polling cadence are unchanged. +- The synthetic carrier used motion content intent. Chromium classifies that + as realtime video, without the default screen-content ALR probing. After a + producer adapted downward, low real output could leave the outgoing native + bandwidth estimate slow to recover. The carrier now uses detail once at + construction, selecting the framework's screen-content behavior. The real + producer retains motion and the Host's degradation preference. + +Pinned WebRTC explains both boundaries: starting the quality scaler +[restarts initial frame dropping](https://webrtc.googlesource.com/src/+/6f37672d358475cd17544121a12494da454d85fb/video/adaptation/video_stream_encoder_resource_manager.cc#230); +[content hints select the screencast option](https://webrtc.googlesource.com/src/+/6f37672d358475cd17544121a12494da454d85fb/pc/rtp_sender.cc#1423), +which selects [ALR probing configuration](https://webrtc.googlesource.com/src/+/6f37672d358475cd17544121a12494da454d85fb/video/video_send_stream_impl.cc#183). +The [screen probing default is enabled](https://webrtc.googlesource.com/src/+/6f37672d358475cd17544121a12494da454d85fb/rtc_base/experiments/alr_experiment.cc#51). +Detail also changes the tiny encoder's content type; it is not an independent +public probing switch or a request to change the real picture's quality. + +| VP8, 14-second constraint / 40-second recovery | Initial A / B | Constrained A / B | Recovered A / B | +| --- | --- | --- | --- | +| Ordinary | 1080p / 1080p | 540p / 1080p | 1080p / 1080p | +| Previous pool | 270p / 270p | 270p / 540p | 720p / 1080p | +| Publication-start correction alone | 1080p / 1080p | 270p / 1080p | 540p / 1080p | +| Publication start + screen-content carrier | 1080p / 1080p | 270p / 1080p | 1080p / 1080p | + +The combined VP8 run returned to 1080p about 17 seconds after shaping ended. +H264 with played audio also returned to 1080p; its unconstrained child retained +1080p throughout. A separate one-second pulse after 45 healthy seconds still +caused native downscaling: the repaired VP8/audio path briefly reached 180p, +then regained 1080p about 13 seconds after pulse start. The previous pool's +no-audio pulse ended the 40-second recovery at 360p. Audio changes the bandwidth +composition, so these pulse runs do not establish an exact speedup. Normal +WebRTC can also temporarily reduce resolution after a pulse; this repair does +not promise blur-free delivery under changing network or device load. + +A single-consumer control, retaining its mature producer, recovered without +the deeper dual-consumer drop. Together with the publication-start control, +this distinguishes cold producer adaptation from repeated producer churn. +The rate-owner repair separately retains existing encoders through output-rate +spikes and shared budget changes; a genuinely weaker child can still need its +own producer. No delayed-budget policy, bitrate floor or manual recovery probe +was added. Probe padding remains framework-owned traffic under native limits. + +Budget attribution remains unchanged: targetBitrate is the encoder's allocated +target, not raw link bandwidth ([upstream stats correction](https://webrtc.googlesource.com/src/+/fe25b0e928ea4e64aa134f5dc8012343320deec5%5E%21/)). +Replacing it with availableOutgoingBitrate would bypass native allocation and +protection. Keep producer, carrier, egress and decoded observations separate. + ## Cost And Accounting Before the CPU-carrier refinement, a matched actual-product VP8 1080p30 comparison's ordinary pair made @@ -133,6 +191,8 @@ npx tsx scripts/browser-local-pool-probe.ts carrier --1080 --background ``` Add `--h264`, `--single` or `--late` for the corresponding case. Set +`--network --auto --short-pulse` for a 45-second warmup, one-second constraint +and 40-second recovery instead of the ordinary 14-second constraint. Set `CHROME_PATH` for another installed Chromium binary. `--auto` extends the recovery observation; product code owns all adaptation. Results go to ignored `build/browser-local-pool`; summarize with diff --git a/docs/research/room-voice.md b/docs/research/room-voice.md new file mode 100644 index 0000000..bfb5c4b --- /dev/null +++ b/docs/research/room-voice.md @@ -0,0 +1,287 @@ +# Room Voice: Design Assessment + +Reviewed 2026-09-24 against public `v1.6.3` (`f423df0a`). This is a proposal, +not an accepted product contract or implementation authorization. The separate +room-interaction experiment supplies candidate lifecycle behavior; it has not +been integrated. [TODO](../todo.md#now) owns the outstanding work and decisions. + +## Recommendation + +Keep existing room authority and the Host-rooted screen route. Model room voice +as participant-owned microphone publications and independent listening. Chat, +screen playback and voice share admission, not a media lifetime. Do not turn the +screen tree into a conference graph or introduce another room/session store. + +There is no equally small solution that simultaneously promises all 21 members +can talk, preserves every P2P-only deployment, and supplies conference-grade +connectivity. Choose the first voice scope before implementing its transport. +For the next technical experiment, assess a small direct group and audio +recapture isolation first. Four total voice participants is a useful test size, +**not an accepted product limit or a demonstrated performance guarantee**. +If full-room voice is required initially, assess a mature conference SFU before +building a mesh intended to grow into one. + +## Current Owners And Actual Gaps + +| Current owner | Verified behavior | Consequence for voice | +| --- | --- | --- | +| Room store and authenticated signaling | Room admission, participant identity, grant revocation and session replacement | Extend this authority; no voice login or second membership store | +| `HostAudio` in `src/client/media/host-audio.ts` | Source and microphone mixed into one share-scoped track | This is commentary; it cannot supply independent conversation | +| `internal/app/nativehost/audio_mix.go` | Native source/mic PCM share one mixer and output clock | Keep source audio here; avoid duplicating the Host mic across outputs | +| `src/client/media/viewer-audio.ts` | Audio follows the picture's play/pause/mute state | Room speech needs its own playback owner | +| Screen controller and media edges | One Host source, one upstream per Viewer, bounded screen-copy capacity | Voice must not borrow screen generations, first-video-frame proof or copy slots | +| Embedded SFU admission/media | One committed room publication, video-required metadata, share-scoped resource identity | Multiple audio-only publishers need a new capability, not another flag | + +The SFU constraints are explicit in `validPublicationMedia` in +`internal/server/sfu/media.go`, `ResourceFence` and `CommitPublication` in +`internal/server/sfu/admission.go`, and the Host/Viewer dispatch in +`internal/server/signal/router_sfu_signal.go`. +[ADR-0013](../adr/0013-embedded-node-local-media.md) embeds Pion/LiveKit components; +it does not embed LiveKit's complete room service. + +The interaction candidate separates a retained Host room session from screen +publication. That is a useful foundation, not permission to allocate rooms on +page visit. Preserve the accepted entry: first share creates the room and an +existing room can resume. Retaining opted-in interaction after stopping media is +the candidate lifecycle, not published behavior. A separate voice-only +room-creation flow would be another product decision. + +## Transport Choice + +| Approach | Benefit | Cost and boundary | +| --- | --- | --- | +| Small direct voice mesh | Keeps media P2P and does not require a reachable central media server | Each sender sends to every listener; needs a total voice-member bound, separate resource accounting and real NAT/device tests | +| Mature multi-publisher SFU | Each speaker uploads once; suitable direction for full-room voice | New publication/subscription integration and central-media availability/privacy decision | +| Host forwarding or mixing hub | Fewer peer relationships | Reintroduces Host dependency, forwarding/mixing and echo ownership; unsuitable for Host-independent conversation | +| Custom voice relay tree or automatic mesh/SFU switching | Potentially combines deployment options | Adds another routing/recovery problem; outside the first scope | + +For a full bidirectional mesh, `N` participants need `N*(N-1)/2` pair connections +and up to `N-1` microphone copies per sender: 21 people means 210 pairs and 20 +copies; four means six pairs and three copies. These are topology counts, not +bandwidth or CPU measurements. Limiting speakers alone does not bound sender +upload: one speaker and 20 listeners still need 20 copies. The screen's existing +`C=1..3` limit must retain its meaning; any independent voice budget needs explicit +acceptance and combined-load measurement. + +Two SFU implementations deserve a bounded comparison: a complete mature room +engine, or a narrow participant-audio adapter around existing media components. +The former adds deployment and potentially competing session ownership; the +latter leaves Piik responsible for admission, subscriptions, recovery and cleanup. +Do not describe the adapter as mature conference behavior merely because its +forwarder uses mature libraries. LiveKit's +[room/participant/track model](https://docs.livekit.io/intro/basics/rooms-participants-tracks/) +and [SFU architecture](https://docs.livekit.io/reference/internals/livekit-sfu/) +show the layers involved. + +Select one topology for the first capability. Jitsi's +[two-party P2P design](https://jitsi.org/blog/p2p4121/) retains its bridge connection +and changes transport when group size changes: useful evidence that a hybrid +mode has real lifecycle cost. It is not a reason to copy that complexity here. + +Deployment is part of this decision. Demo and App Local/public-link modes are +P2P-only today. Public-link tunneling carries HTTP/WebSocket control traffic, +not a public SFU media endpoint. Existing STUN is not TURN, and existing embedded +UDP media does not provide media TCP/TLS. A full conference engine's +[connectivity options](https://docs.livekit.io/transport/self-hosting/ports-firewall/) +are not inherited by importing its library. This proposal enables no new service +or port and does not change Demo configuration. + +## Proposed Responsibility Boundaries + +```text +Existing room authority and authenticated signaling + +-- screen publication -> existing P2P/SFU screen route -> picture/source sound + +-- interaction events -> bounded chat/reactions + +-- voice admission -> participant microphone -> selected voice transport + -> independent voice playback +``` + +These are responsibility boundaries, not a request for a generic media framework +or one new service per line. Keep the first implementation cohesive: + +| Responsibility | Owns | Does not own | +| --- | --- | --- | +| Room authority | Who belongs, who may publish/listen, bounds and retirement | OS microphone permission or local volume | +| Participant voice session | Explicit join intent, current admitted publication/subscriptions and their transport lifetime | Screen routes, share generation or a second reconnect policy | +| Microphone input | One acquisition, device choice, local mute, replacement and disposal | Remote permission to turn capture on | +| Voice playback | Received tracks, local volume/person mute, output capability and autoplay recovery | The video's pause/mute state or remote speaking permission | + +Do not attach room speech to a screen peer merely to reduce connection count: +screen parent replacement and share retirement have different lifetimes. The +selected voice transport owns its connections and counts them explicitly. + +Prefer a Browser-owned room microphone for both roles, including a Host using +App screen capture. This reuses platform speech processing and avoids a second +native voice service. Reuse existing device-selection semantics, not native +device identifiers. Browser microphone acquisition requires a secure context; +plain HTTP LAN pages and unsupported browsers must have truthful capability +presentation. App detection alone does not establish microphone availability. +See the [capture specification](https://www.w3.org/TR/mediacapture-streams/). +Browser voice also does not inherit the native Pion path's port mapping, NAT +assistance or native media fallback just because an App is attached. Measure +voice reachability independently; a working screen path does not prove it. + +Room voice and source sound need different audio treatment. Voice uses the +platform's speech/AEC facilities; music/game sound keeps its fidelity settings. +Do not force both through the existing source-audio Opus profile or write a new +voice codec, AEC, PCM subtraction routine or virtual-audio driver. + +## Consent, Lifecycle And Presentation + +Proposed first permission model: admitted members may explicitly join voice and +then unmute themselves; existing room removal remains the authority boundary. +Start with a closed microphone. Joining may unlock playback, but only explicit +unmute requests input permission. If Host-only speaking approval is required, +specify it before adding a moderation UI. No remote action may start a microphone; +LiveKit also disables [remote unmute by default](https://docs.livekit.io/intro/basics/rooms-participants-tracks/participants/). + +| Event | Proposed behavior | +| --- | --- | +| Share start/stop/pause, source or quality change | Voice stays independent | +| Local mic mute | Silence that publication without leaving voice; retain explicit device intent | +| Local person mute or voice volume | Change local listening only; picture volume remains separate | +| Leave voice | Release mic and voice media, remain in room/chat | +| Host leaves page while room service remains | Retire Host voice; other admitted participants may continue | +| App exits in Local/public-link mode | Room service disappears; independent voice cannot outlive its authority | +| Site-mode App media fails | Browser voice may continue while the Site room authority remains | +| Revocation, replacement or room close | Retire affected resources and reject stale completions/signaling | +| Mic permission/device failure | Preserve picture and other voices; show the actual input failure | +| Hidden page or phone interruption | Visibility alone is not leave/mute; report actual suspension/device loss | + +Transient signaling loss is not automatically a media failure or a grant +revocation. Current `presence.go` and `route.Controller.DisconnectSession` +distinguish disconnected sessions from confirmed departure; Viewers have bounded +membership grace. Voice must consume that authority owner's transitions, with no +parallel grace timer or indefinite stale-session publication. Before implementation, +define and test the Host case, reservation retention, reconnect re-admission and +revocation during an outage. Do not extrapolate Viewer grace to a Host or reuse +an old session ID as current authority. This is a design gate, not a claim that +screen recovery already implements voice recovery. + +One input owner prevents duplicate captures. Retire late permission results after +leave; prepare a replacement device before releasing a healthy old input; keep a +failed explicit selection visible rather than silently choosing another mic. +Repeated mute/join/leave must converge without a second saved effective-state flag. +Ordinary publication mute need not renegotiate every time; mature systems +[distinguish mute, unpublish and unsubscribe](https://docs.livekit.io/intro/basics/rooms-participants-tracks/tracks/). + +Use the existing participants/sofa and room actions, not a conference grid. +First controls: join/leave voice, mic toggle and input choice/check, independent +voice volume, and local person mute in the participant menu. Keep source-sound +settings with the picture. Default to system output; show output selection only +where supported. Do not add deafen, global push-to-talk or a permission-role matrix +until their need and restoration behavior are established. + +Display local intent, authorized membership, transport readiness and actual +playback separately. A speaking pulse follows observed audio activity, not a +connected flag or UUID animation seed. Catch blocked playback, including tracks +arriving after joining, and offer an explicit enable-audio action; compare +[LiveKit's autoplay handling](https://docs.livekit.io/reference/components/react/component/startaudio/). +Mobile lock-screen calling remains unproven; platform suspension is not fixed by +extra timers. See [browser lifecycle guidance](https://developer.chrome.com/docs/web-platform/page-lifecycle-api). + +## Audio Isolation Is A Prerequisite + +Whole-system loopback records received room speech played on the captured output +endpoint. That speech can then be sent back through screen audio. Headphones +remove an acoustic path, not this digital recapture; microphone AEC does not clean +the separate source-audio track. This matters even if only the Host shares a screen. + +Windows exposes narrower process-tree capture and process exclusion. Current +Piik includes the selected process tree for window sound or captures the output +endpoint for system sound; it does not implement process exclusion. Excluding +the browser that plays Piik voice could also exclude intended browser content. +Test actual process ownership and output routing rather than assuming a window +selection isolates sound. See Microsoft's +[loopback description](https://learn.microsoft.com/en-us/windows/win32/coreaudio/loopback-recording) +and [process-capture sample](https://learn.microsoft.com/en-us/samples/microsoft/windows-classic-samples/applicationloopbackaudio-sample/). + +Browser `restrictOwnAudio` is relevant; `suppressLocalAudioPlayback` has a different +purpose. Chrome documents the former as excluding audio originating from the +capturing document when system audio is captured. Check support, applied settings +and decoded output on supported platforms; a hint is not cross-browser proof. +See the [Screen Capture specification](https://www.w3.org/TR/screen-capture/) +and [Chrome's implementation announcement](https://developer.chrome.com/blog/chrome-141-beta). + +Verify source sound and incoming voices using distinguishable signals. Cover +native window/system capture, browser tab/window/system capture, source changes, +same-browser content and output-device changes. Where isolation cannot be +established, a supported combination or explicit feature limitation must be +chosen before release. Never silently discard source sound or promise that a +headphone hint solves it. + +## Compatibility, Trust And Bounds + +The Host microphone must have one distribution path. Sending it in both existing +commentary and independent voice causes duplicates and defeats individual mute. +Moving it exclusively to voice can make older/non-participating Viewers lose +commentary. Choose an explicit room capability/participation boundary or a +coordinated major-version transition; do not ship a silent behavior change. +The [versioning owner](../standards/versioning.md) governs that choice. This +assessment does not select a release number or authorize removal of native mic. +Once that boundary is selected, define the handoff before coding: retire the old +microphone distribution before enabling the new one, and test failure/cancellation +without silently losing commentary for Viewers who cannot receive room voice. + +Browser-first capture can avoid new native voice commands, but does not make +room signaling automatically compatible. Trace strict readers in both directions; +advertise support before new messages. Keep stable peer identity, authenticated +session, grant and media connection identity distinct. Reuse an existing identity +only when its lifetime matches; do not add a room-wide epoch for convenience. + +Bind signaling to current authenticated participants and exact publication/peer +resources; bound memberships, publications, subscriptions, candidates and message +rates before allocation. Leave/revoke must release actual resources before their +slots become reusable. No audio through chat events or durable voice recordings. + +P2P revocation has an enforcement limit: the server can refuse signaling and +conforming endpoints can stop capture/drop tracks, but it cannot intercept media +between two modified peers. An SFU can enforce its own forwarding removal. Do not +promise identical server-enforced muting across these models, or hide the central +server's media access under the existing P2P privacy label. + +## Implementation Sequence And Acceptance + +1. Reconcile the isolated interaction candidate with current main and accept its + room/picture separation on its own. Preserve the existing staged work; do not + merge an old branch wholesale. Voice need not block chat/UI acceptance. +2. Select total voice membership, deployment coverage, speaking policy and the + commentary/compatibility boundary. Record accepted changes in the existing + product owners and an ADR only when architectural decisions are accepted. +3. Run an isolated transport/audio-isolation experiment before shipping UI. + Measure screen plus voice CPU, upload, number of connections, audible latency, + NAT failure, echo and cleanup. Use one topology; stop if the required scope + implies a custom conference engine or unsupported audio isolation. +4. Implement one vertical slice through the owners above: explicit join, muted + input, unmute, remote audio, local mute and leave. Then expand participants and + exact authority teardown. Reuse existing controls, status/comic meanings and + diagnostic export; avoid separate feature-local rules. +5. Accept faults as well as the happy path: permission resolves after leave, + device switch fails, source changes, autoplay is blocked, signaling drops + during healthy audio, reconnect races revoke/replacement, Host leaves, service + exits, mobile suspends, and repeated join/leave leaks no resources. Test quiet + and muted participants without video-frame admission requirements. Include + supported old/new endpoints and Chinese/English/visual presentation. + +Transport experiments must report measured limits rather than treating a +successful localhost call as public-network or mobile acceptance. This planning +round ran no voice, acoustic-echo or device-performance experiment. + +## Independent Fidelity Work + +Stereo acceptance can proceed separately: distinguishable left/right signals +through Browser, Native, relay and SFU, with microphone mixing and source changes. +The released native answer fix negotiates receive-side stereo; SDP alone does not +prove decoded channel separation. Opus negotiation follows +[RFC 7587 section 7](https://www.rfc-editor.org/rfc/rfc7587.html#section-7). +Do not add surround channels as a prerequisite for voice. + +For [HDR issue #420](https://github.com/TNTcraftHIM/Piik/issues/420), investigate +correct SDR output first: preserve HDR range during native capture, then use +platform tone/gamut mapping before the existing SDR encoder. Windows documents +[floating-point capture](https://learn.microsoft.com/en-us/windows/apps/develop/media-authoring-processing/screen-capture) +and a [tone-map effect](https://learn.microsoft.com/en-us/windows/win32/direct2d/hdr-tone-map-effect). +Validate mixed SDR/HDR content, highlights, SDR reference output and display-mode +changes. Already-clipped browser input cannot be restored downstream. Full HDR +transport/metadata, surround sound, recording and transcription remain outside +this proposal. diff --git a/docs/standards/configuration.md b/docs/standards/configuration.md index e0c34c0..5268538 100644 --- a/docs/standards/configuration.md +++ b/docs/standards/configuration.md @@ -24,7 +24,7 @@ service secret store or an untracked access-restricted environment file. | `LISTEN_HOST` | Defaults to `0.0.0.0`; bare-metal production normally uses `127.0.0.1`. | | `PORT` | Positive TCP port, default `8787`; the tracked release wrapper supports only that default. | | `PUBLIC_BASE_URL` | Exact public HTTP(S) origin; production requires HTTPS. | -| `ALLOWED_ORIGINS` | Comma-separated exact HTTP(S) origins; wildcard is invalid. | +| `ALLOWED_ORIGINS` | Unset or empty follows `PUBLIC_BASE_URL`. A non-empty comma-separated list replaces that default; include every trusted browser origin (scheme, host and port). Wildcard is invalid. | | `SITE_ACCESS_PASSWORD` | Optional in every environment. Unset or empty allows entry without a site password. A configured value is matched exactly, including spaces and Unicode; there are no password length or character rules. General HTTP request limits still apply. Room ownership and Viewer admission remain independent. | | `ROOM_DATABASE_PATH` | Hosted defaults to `rooms.sqlite` in its working directory when unset or blank. An explicit absolute file path selects another SQLite file; `:memory:` opts into process-memory room authority. App Local remains in memory. | | `MAX_VIEWERS_PER_ROOM` | `1..20`, default `20`; excludes the Host. | diff --git a/docs/standards/engineering.md b/docs/standards/engineering.md index 029fb0f..617d82c 100644 --- a/docs/standards/engineering.md +++ b/docs/standards/engineering.md @@ -245,6 +245,15 @@ hits from tests and previews, are leads rather than proof of product use or absence. Follow the actual producer-to-consumer path and check the changed boundary; isolated helper tests can miss a mismatch between the two ends. +For a shared-resource failure, review both the failing child and a healthy +sibling through the real producer/consumer boundary. Compare preparation with +live operation, and current implementations with supported compatibility paths. +Exercise optional failure after success and a dependency that remains pending +after cancellation; a timeout constant or a close call alone proves no deadline. +Keep the regression at the owning boundary and verify it rejects the old +behavior. Report which failure sequences were exercised, rather than treating +test counts, file coverage or an earlier audit as proof of lifecycle coverage. + Trace acquisition, use, commit and retirement, including replacement and failure. For async ownership, check A starting, B replacing it, then A completing or cleaning up. Where compatibility is promised or being established for public diff --git a/docs/standards/media-quality.md b/docs/standards/media-quality.md index be2a506..f610714 100644 --- a/docs/standards/media-quality.md +++ b/docs/standards/media-quality.md @@ -64,7 +64,9 @@ production behavior and remaining acceptance. the selected source produces no frames. Setup and encoding failures remain bounded; quiet-source waiting must not block pause, stop or connection control. Applied settings change only after the replacement is ready and installed; - failure preserves the previous capture and profile. Platform quality preference uses the hardware + required-output failure preserves the previous capture and profile. Failed lower + outputs stay unavailable across preparation and installation without rejecting + a healthy original output. Platform quality preference uses the hardware encoder's standard quality-versus-speed hint. Pion owns Native WebRTC transport; the shared media adapter uses LiveKit media components for bandwidth estimation, forwarding allocation, pacing and recovery. @@ -113,7 +115,9 @@ room quality setting. Auto uses one bounded actual-sender probe rather than a capability or device allowlist. A proved sender prefers H.264 with VP8 fallback; unsupported, failed, or inconclusive probes use VP8. Manual VP8 or H264 strictly selects that codec. -The result stays fixed for the current share and is reconsidered for a new share. +The probe deadline covers setup, negotiation and statistics; cancellation retires +its resources even when a Browser operation remains pending. The result stays +fixed for the current share and is reconsidered for a new share. ADR-0007 owns probe and negotiation mechanics. The Browser chooses the concrete encoder implementation. A reported H.264 codec @@ -130,8 +134,10 @@ choice for quality/source replacement. Native relay encoding uses the same selection owner. Auto measures encoding work for synthetic NV12 frames at the selected dimensions and frame rate, with a bounded warmup and sample. If H264 sustains the target it is selected; otherwise -VP8 is measured within the remaining four-second budget. This is a throughput -check, not a perceptual-quality score or a promise under future GPU load. +VP8 is measured within the remaining four-second budget. An unsuccessful or +timed-out comparison retains an already-proved H264 encoder; without a proved +encoder, startup still fails. This is a throughput check, not a perceptual-quality +score or a promise under future GPU load. The returned actual codec owns the shared source, preview, and relay; live quality/source changes retain it. Other native platform encoders remain H264. @@ -143,6 +149,10 @@ it is never attached directly to an outbound sender. Replacing or retiring a sender also stops its clone, so native adaptation state cannot survive by being inherited through the original track. Host pause and live capture constraints are propagated to current Host-owned clones. +Connection retirement also settles its pending sender and negotiation waits; +closing a Browser PeerConnection alone does not guarantee that its promises +settle. Cancellation starts before joining work that depends on it, while a +replacement still waits for the retired owner's cleanup. Eligible Browser P2P parents use the source-owned pool in [ADR-0014](../adr/0014-browser-node-local-encoding-pool.md). Compatible direct @@ -151,6 +161,9 @@ remain separate. Each outgoing connection keeps native transport, allocation and recovery, with a tiny carrier supplying its RTP clock. Producers use the existing native video target under Host ceilings. Actual forwarded-frame and producer observations remain distinct from the tiny carrier's statistics. +Producer startup protection begins at actual outgoing publication, excluding +local warmup and paused frames. The synthetic carrier uses screen-content +transport probing; the real producer keeps the Host's picture adaptation. The first eligible consumer follows the same path; unsupported APIs or failed pooling use ordinary senders. Prepared ordinary quality candidates, Native ingress and Browser SFU remain independent compositions. @@ -186,8 +199,8 @@ starts one shared decode/scale/encode group, while healthy forwarding retains the original input without starting a decoder. The group produces the highest needed output and its lower fallbacks and stops unused upper outputs. Each child receives only its selected output. The forwarding SFU adds no server transcoder. -Downstream attachment failure or retirement ends only that edge, never its -shared source or healthy siblings. +Downstream attachment or sending failure and retirement belong to that edge, +never its shared video/audio source or healthy siblings. Native capture uses the same screen-share output construction within Host ceilings. Its source, source clock and higher outputs remain independent of diff --git a/docs/todo.md b/docs/todo.md index 693910e..232ccf1 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -1,6 +1,6 @@ # Current TODO Ledger -Last reviewed: 2026-09-24 +Last reviewed: 2026-09-25 Only **Now** is executable. Product modules own behavior; Git/PRs own completed history. A parked idea is not implementation authority. @@ -28,20 +28,30 @@ history. A parked idea is not implementation authority. can resume, and stopping media keeps the opted-in interaction session. Do not add visit-triggered creation or a separate pre-share room-creation entry. This experiment does not authorize integration or publication. -- [ ] **Assess room voice and media fidelity alongside interaction.** Compare - mature room-audio designs before selecting a small first scope: explicit mic - consent, mute/device/output controls, speaker authority and disconnect cleanup. - Keep room membership, voice publication and the Host's shared picture distinct; - do not send audio through chat signaling or reverse the current screen tree. +- [ ] **Room voice: design before implementation.** The + [voice assessment](./research/room-voice.md) maps current owners, mature options + and a staged verification plan. Resolve total voice membership/topology, + P2P-only deployment coverage, speaking authority and the Host-commentary + compatibility boundary before implementation. Validate source-audio isolation + from received speech before exposing simultaneous sharing and voice. Keep + room authority, microphone and screen lifetimes distinct; no new room store, + screen-tree reversal or automatic mesh/SFU controller is approved. +- [ ] **Verify media fidelity independently of voice.** Verify actual left/right separation through Browser, Native and SFU paths; receive-side stereo negotiation alone does not establish audible fidelity. Assess HDR capture to correct SDR output for ordinary viewers before full HDR, with issue [#420](https://github.com/TNTcraftHIM/Piik/issues/420) as the reported - case. Multi-speaker routing, full HDR and surround sound remain design decisions, - not additions authorized for this maintenance release. + case. Full HDR and surround sound remain design decisions. These findings do + not establish physical device acceptance or authorize a new release. ## Awaiting Device Or Reporter Evidence +- [ ] **Self-hosted room creation HTTP 403.** Retest affected deployments using + the [address checks](./guide/troubleshooting.md#room-creation-returns-403). + A stale explicit `ALLOWED_ORIGINS` overrides `PUBLIC_BASE_URL`; the sample now + follows the public address and Piik identifies its Origin rejection. Reporter + configuration and response details are still absent; distinguish Piik's + rejection from a proxy/WAF 403 before assigning their cause. - [ ] **No available media route.** A decline is reported since around v1.4; establish comparable attempts on the same endpoints/network before attributing a version regression. Obtain matched Host/Viewer diagnostics from @@ -97,6 +107,9 @@ history. A parked idea is not implementation authority. local permission and recovery checks do not establish the reporters' causes. Manual H264 returning immediately to idle also needs reporter diagnostics; bounded selection past an unusable hardware encoder is locally verified. + Startup failure is reported again on the latest release without a known codec. + Auto selection now retains proved H264 if the optional VP8 comparison times + out; this confirmed selection defect does not establish the reporter's cause. A separate v1.6.1 App/Chrome report lists Window/Screen sources, then waits and returns to idle with a share-start error after selection. Its codec is unknown; similarity to the H264 symptom does not establish the same cause. @@ -110,6 +123,12 @@ history. A parked idea is not implementation authority. fullscreen, display-mode change and device loss. A reproduced downstream retirement race no longer stops the shared source. Controlled-window checks do not reproduce the reporting game/device environment. +- [ ] **Brief blur followed by immediate recovery.** Compare the same source, + codec, peers and network with ordinary WebRTC before attributing the report + to pooling. The [pool checks](./research/browser-local-encoding-pool.md#balanced-startup-and-recovery) + cover rate-owner retention, publication startup, native recovery probing and + ordinary fallback. They do not establish this reporter's capture path or result; + correlate actual receiver dimensions, QP, resource adaptation and packet loss. - [ ] **Windows 32-bit candidate acceptance.** Verify the isolated `spike/windows-x86-capture` candidate's launch, capture/audio, memory pressure, source replacement and update links on a 32-bit Windows device. WOW64 diff --git a/internal/app/mediaedge/audio.go b/internal/app/mediaedge/audio.go index 49e840c..d658dfb 100644 --- a/internal/app/mediaedge/audio.go +++ b/internal/app/mediaedge/audio.go @@ -114,7 +114,10 @@ func (source *AudioSource) WritePCMWithDropped(pcm []byte, duration time.Duratio return err } source.bytes.Add(uint64(len(packet))) - return source.samples.WriteSample(media.Sample{Data: packet, Duration: duration, PrevDroppedPackets: dropped}) + // Pion still delivers to healthy bindings when one connection cannot write. + // Connection lifecycle owns those failures; they do not end the PCM source. + _ = source.samples.WriteSample(media.Sample{Data: packet, Duration: duration, PrevDroppedPackets: dropped}) + return nil } func (source *AudioSource) WriteRTP(packet *rtp.Packet) error { diff --git a/internal/app/mediaedge/audio_test.go b/internal/app/mediaedge/audio_test.go new file mode 100644 index 0000000..9d84121 --- /dev/null +++ b/internal/app/mediaedge/audio_test.go @@ -0,0 +1,75 @@ +package mediaedge + +import ( + "io" + "testing" + "time" + + "github.com/TNTcraftHIM/Piik/internal/app/nativeaudio" + "github.com/pion/interceptor" + "github.com/pion/rtp" + "github.com/pion/webrtc/v4" +) + +// Exercise Pion's actual shared-track fanout without sockets or retirement races. +type audioBinding struct { + id string + err error + packets int +} + +func (binding *audioBinding) CodecParameters() []webrtc.RTPCodecParameters { + return []webrtc.RTPCodecParameters{{RTPCodecCapability: opusCapability, PayloadType: 111}} +} +func (*audioBinding) HeaderExtensions() []webrtc.RTPHeaderExtensionParameter { return nil } +func (*audioBinding) SSRC() webrtc.SSRC { return 1 } +func (*audioBinding) SSRCRetransmission() webrtc.SSRC { return 0 } +func (*audioBinding) SSRCForwardErrorCorrection() webrtc.SSRC { return 0 } +func (binding *audioBinding) WriteStream() webrtc.TrackLocalWriter { return binding } +func (binding *audioBinding) ID() string { return binding.id } +func (*audioBinding) RTCPReader() interceptor.RTCPReader { return nil } +func (binding *audioBinding) WriteRTP(_ *rtp.Header, payload []byte) (int, error) { + binding.packets++ + return len(payload), binding.err +} +func (binding *audioBinding) Write(payload []byte) (int, error) { + return binding.WriteRTP(nil, payload) +} + +func TestAudioSourceIsolatesBindingWriteFailure(t *testing.T) { + engine, err := NewEngine(EngineOptions{BindAddress: "127.0.0.1:0", IncludeLoopback: true}) + if err != nil { + t.Fatal(err) + } + defer engine.Close() + source, err := engine.NewAudioSource(2, 128_000) + if err != nil { + t.Fatal(err) + } + defer source.Close() + failed := &audioBinding{id: "retiring", err: io.ErrClosedPipe} + healthy := &audioBinding{id: "healthy"} + for _, binding := range []*audioBinding{failed, healthy} { + if _, err = source.samples.Bind(binding); err != nil { + t.Fatal(err) + } + } + pcm := make([]byte, nativeaudio.FrameBytes) + for expected := 1; expected <= 2; expected++ { + if err = source.WritePCM(pcm, 20*time.Millisecond); err != nil { + t.Fatalf("one failed binding stopped the shared source: %v", err) + } + if healthy.packets != expected || failed.packets != expected { + t.Fatalf("fanout stopped: healthy=%d failed=%d", healthy.packets, failed.packets) + } + } + if err = source.WritePCM(pcm[:1], 20*time.Millisecond); err == nil { + t.Fatal("invalid PCM was accepted") + } + if err = source.Close(); err != nil { + t.Fatal(err) + } + if err = source.WritePCM(pcm, 20*time.Millisecond); err == nil { + t.Fatal("closed audio source was accepted") + } +} diff --git a/internal/app/mediaedge/publication.go b/internal/app/mediaedge/publication.go index 14d19d4..cd3a98d 100644 --- a/internal/app/mediaedge/publication.go +++ b/internal/app/mediaedge/publication.go @@ -42,8 +42,13 @@ func (engine *Engine) NewPublication(source *Source, options EdgeOptions) (*Publ len(options.ConnectionID) > 256 || options.Audio != nil && options.Audio.engine != engine { return nil, errors.New("native publication input is invalid") } + // A new publication starts on the base source. Independent lower-output + // groups are not part of its descriptor and own their failures separately. + outputsUnavailable := func() bool { + return slices.Contains(source.outputBitrates[:min(len(source.outputBitrates), len(source.formats))], uint32(0)) + } source.mu.Lock() - unavailable := slices.Contains(source.outputBitrates, uint32(0)) + unavailable := outputsUnavailable() source.mu.Unlock() if unavailable { return nil, errors.New("native publication output is unavailable") @@ -89,7 +94,7 @@ func (engine *Engine) NewPublication(source *Source, options EdgeOptions) (*Publ }) engine.mu.Lock() source.mu.Lock() - if engine.closed || source.closed || len(source.publications) >= 2 || slices.Contains(source.outputBitrates, uint32(0)) { + if engine.closed || source.closed || len(source.publications) >= 2 || outputsUnavailable() { source.mu.Unlock() engine.mu.Unlock() _ = publication.Close() diff --git a/internal/app/mediaedge/publication_test.go b/internal/app/mediaedge/publication_test.go index 5b7ad4f..f36fd15 100644 --- a/internal/app/mediaedge/publication_test.go +++ b/internal/app/mediaedge/publication_test.go @@ -97,6 +97,47 @@ func TestPublicationOwnsOneReservationAndSourceLifetime(t *testing.T) { } } +func TestPublicationAdmissionIgnoresUnrelatedFailedOutputs(t *testing.T) { + engine, err := NewEngine(EngineOptions{BindAddress: "127.0.0.1:0", IncludeLoopback: true}) + if err != nil { + t.Fatal(err) + } + defer engine.Close() + source, err := engine.NewSource("vp8", 2, 2, nil) + if err != nil { + t.Fatal(err) + } + defer source.Close() + publication, err := engine.NewPublication(source, EdgeOptions{ConnectionID: "initial"}) + if err != nil { + t.Fatalf("unconfigured outputs prevented initial publication: %v", err) + } + defer publication.Close() + if err = source.ConfigureOutputs([]uint32{90_000, 300_000, 90_000, 90_000}); err != nil { + t.Fatal(err) + } + for _, slot := range []int{2, 3, 0, 1} { + if err = source.DisableLayer(slot); err != nil { + t.Fatal(err) + } + candidate, createErr := engine.NewPublication(source, EdgeOptions{ConnectionID: "replacement"}) + if slot >= 2 { + if createErr != nil { + t.Fatalf("unrelated failed output %d prevented publication: %v", slot, createErr) + } + if len(candidate.Media().Layers) != 2 || len(source.publications) != 2 { + t.Fatal("independent output failure changed the base publication") + } + if err = candidate.Close(); err != nil { + t.Fatal(err) + } + } else if createErr == nil { + _ = candidate.Close() + t.Fatalf("failed base output %d was admitted", slot) + } + } +} + func TestPublicationQualityUsesHighestSentLayerCadence(t *testing.T) { check := func(err error) { t.Helper() diff --git a/internal/app/nativehost/session.go b/internal/app/nativehost/session.go index a17203b..6c792cf 100644 --- a/internal/app/nativehost/session.go +++ b/internal/app/nativehost/session.go @@ -39,6 +39,8 @@ type CaptureState struct { FPS uint32 `json:"fps,omitempty"` RestoreToken string `json:"restoreToken,omitempty"` Outputs []nativecapture.OutputProfile `json:"outputs"` + + unavailableLayers []int } type Event struct { @@ -91,14 +93,17 @@ type Session struct { done chan error ready chan error - mu sync.Mutex - updateMu sync.Mutex - edges map[string]*mediaedge.Edge - publications map[publicationKey]*mediaedge.Publication - captureApplied chan struct{} - audioChanged chan struct{} - paused bool - closed bool + mu sync.Mutex + updateMu sync.Mutex + edges map[string]*mediaedge.Edge + publications map[publicationKey]*mediaedge.Publication + + captureApplied chan struct{} + captureUnavailable []int + + audioChanged chan struct{} + paused bool + closed bool } func Start(parent context.Context, options Options) (*Session, error) { @@ -512,6 +517,7 @@ func (session *Session) commitCapture( session.stream = replacement applied := make(chan struct{}) session.captureApplied = applied + session.captureUnavailable = state.unavailableLayers if replaceAudio { session.audioStream = replacementAudio select { @@ -545,14 +551,25 @@ func (session *Session) commitCapture( // Only the video reader installs source metadata, after leaving the old input. // The update response waits for this exact stream's installation, not its process. func (session *Session) installCapture(next *nativecapture.Stream) error { + session.mu.Lock() + unavailable := session.captureUnavailable + session.mu.Unlock() session.source.BeginGeneration() if err := configureCaptureOutputs(session.source, next.Outputs()); err != nil { return err } + // Preparation has consumed these events. Apply them to the new generation + // before acknowledging it, just as the live reader handles unavailable layers. + for _, layer := range unavailable { + if err := session.source.DisableLayer(layer); err != nil { + return err + } + } session.mu.Lock() if session.stream == next && session.captureApplied != nil { close(session.captureApplied) session.captureApplied = nil + session.captureUnavailable = nil } session.mu.Unlock() session.emit(Event{Type: "capture-state", ShareID: session.shareID, State: "active"}) @@ -915,6 +932,9 @@ func waitForCaptureProfile( } inputSeen := false var starting CaptureState + var unavailable []int + outputs := stream.Outputs() + original := min(1, len(outputs)-1) for { frame, err := stream.Read() if err != nil { @@ -922,8 +942,14 @@ func waitForCaptureProfile( return } if frame.Kind == nativecapture.FrameLayerUnavailable { - send(result{stage: "output-unavailable", err: errors.New("native capture profile has an unavailable output")}) - return + if starting.State != "starting" || frame.Layer < 0 || frame.Layer >= len(outputs) || frame.Layer == original { + send(result{stage: "output-unavailable", err: errors.New("native capture profile has an invalid or unavailable original output")}) + return + } + if !slices.Contains(unavailable, frame.Layer) { + unavailable = append(unavailable, frame.Layer) + } + continue } if frame.Kind == nativecapture.FrameBegin && !inputSeen { inputSeen = true @@ -958,6 +984,7 @@ func waitForCaptureProfile( return } state.AdapterIndex, state.EncoderIndex = starting.AdapterIndex, starting.EncoderIndex + state.unavailableLayers = unavailable send(result{state: state}) return } diff --git a/internal/app/nativehost/session_test.go b/internal/app/nativehost/session_test.go index aef9f17..a581b55 100644 --- a/internal/app/nativehost/session_test.go +++ b/internal/app/nativehost/session_test.go @@ -8,6 +8,8 @@ import ( "io" "log/slog" "os" + "slices" + "strconv" "strings" "testing" "time" @@ -20,6 +22,10 @@ func TestMain(tests *testing.M) { if os.Getenv("PIIK_NATIVEHOST_PIPE_FIXTURE") == "backend" { for _, key := range []string{"PIIK_CAPTURE_STARTING", "PIIK_CAPTURE_ACTIVE"} { writeCaptureFixtureFrame(nativecapture.Frame{Kind: nativecapture.FrameStatus, Data: []byte(os.Getenv(key))}) + if layer := os.Getenv("PIIK_CAPTURE_UNAVAILABLE"); key == "PIIK_CAPTURE_STARTING" && layer != "" { + value, _ := strconv.Atoi(layer) + writeCaptureFixtureFrame(nativecapture.Frame{Kind: nativecapture.FrameLayerUnavailable, Layer: value, Data: []byte("fixture")}) + } } _, _ = io.Copy(io.Discard, os.Stdin) os.Exit(0) @@ -83,7 +89,7 @@ func TestCaptureCommitWaitsForReaderMetadataOrTermination(t *testing.T) { go func() { session.updateMu.Lock() err := session.commitCapture(options, QualityProfile{Video: options.Profile, AudioBitrate: 64_000}, replacement, - CaptureState{AdapterIndex: &adapter, EncoderIndex: &encoder}, nil, false) + CaptureState{AdapterIndex: &adapter, EncoderIndex: &encoder, unavailableLayers: []int{0}}, nil, false) session.updateMu.Unlock() committed <- err }() @@ -116,6 +122,35 @@ func TestCaptureCommitWaitsForReaderMetadataOrTermination(t *testing.T) { if media.Layers[1].Width != 854 || media.Layers[1].Height != 480 || media.Layers[1].Bitrate != 2_000_000 { t.Fatalf("acknowledged stale metadata: %+v", media) } + plan, err := source.BeginFrame(time.Second) + check(err) + if plan.Bitrates[0] != 0 || plan.Bitrates[1] == 0 || session.captureUnavailable != nil { + t.Fatalf("replacement lost or retained pending failures: %+v", plan) + } + // The next healthy generation must not inherit the failed layer. + healthy, err := nativecapture.StartVideo(ctx, executable, options) + check(err) + defer healthy.Close() + go func() { + committed <- session.commitCapture(options, session.profile, healthy, CaptureState{}, nil, false) + }() + select { + case <-replacement.Done(): + case <-time.After(5 * time.Second): + t.Fatal("replacement fixture did not exit") + } + check(session.installCapture(healthy)) + select { + case err = <-committed: + check(err) + case <-time.After(time.Second): + t.Fatal("healthy replacement remained pending") + } + plan, err = source.BeginFrame(time.Second) + check(err) + if plan.Bitrates[0] == 0 || plan.Bitrates[1] == 0 { + t.Fatalf("healthy generation inherited a failed layer: %+v", plan) + } return } go session.run() @@ -165,19 +200,38 @@ func TestCaptureProfileRetainsSelectedBackendFromStarting(t *testing.T) { t.Fatal(err) } profile := nativecapture.VideoProfile{Width: 1280, Height: 720, Framerate: 30, Bitrate: 3_000_000, Preference: "balanced"} - stream, err := nativecapture.StartVideo(t.Context(), executable, nativecapture.VideoOptions{ - Target: nativecapture.CaptureTarget{Kind: "display", SourceID: "1", Title: "Fixture"}, Codec: "h264", Profile: profile, OutputGroups: 4, - }) - if err != nil { - t.Fatal(err) - } - defer stream.Close() - state, err := waitForCaptureProfile(t.Context(), stream, profile, "h264", false, false) - if err != nil { - t.Fatal(err) - } - if state.AdapterIndex == nil || *state.AdapterIndex != 1 || state.EncoderIndex == nil || *state.EncoderIndex != 2 { - t.Fatalf("active profile lost the actual hardware selection: %+v", state) + for _, failed := range []string{"", "0", "1", "5", "6"} { + t.Run("unavailable-"+failed, func(t *testing.T) { + t.Setenv("PIIK_CAPTURE_UNAVAILABLE", failed) + stream, err := nativecapture.StartVideo(t.Context(), executable, nativecapture.VideoOptions{ + Target: nativecapture.CaptureTarget{Kind: "display", SourceID: "1", Title: "Fixture"}, Codec: "h264", Profile: profile, OutputGroups: 4, + }) + if err != nil { + t.Fatal(err) + } + defer stream.Close() + state, err := waitForCaptureProfile(t.Context(), stream, profile, "h264", false, false) + if failed == "1" || failed == "6" { + if err == nil { + t.Fatal("original or invalid output failure was accepted") + } + return + } + if err != nil { + t.Fatal(err) + } + if state.AdapterIndex == nil || *state.AdapterIndex != 1 || state.EncoderIndex == nil || *state.EncoderIndex != 2 { + t.Fatalf("active profile lost the actual hardware selection: %+v", state) + } + var unavailable []int + if failed != "" { + layer, _ := strconv.Atoi(failed) + unavailable = []int{layer} + } + if !slices.Equal(state.unavailableLayers, unavailable) { + t.Fatalf("preparation lost optional output failures: %+v", state.unavailableLayers) + } + }) } } diff --git a/internal/server/app/app_test.go b/internal/server/app/app_test.go index 596de8f..bc398e6 100644 --- a/internal/server/app/app_test.go +++ b/internal/server/app/app_test.go @@ -634,7 +634,7 @@ func TestLocalPasswordUsesConfiguredDestinationCookieAcrossLANAndPublicLink(t *t func TestSiteAccessRequiresAnAllowedOriginAndAValidLoginBody(t *testing.T) { server := start(t, Options{Config: testConfig(t)}) - forbidden := `{"error":"Forbidden"}` + forbidden := `{"error":"Origin not allowed"}` server.do(http.MethodPost, "/api/site-access", withBearer(testAccessPassword)). expect(http.StatusForbidden, forbidden) @@ -1085,7 +1085,7 @@ func TestRoomAccessRequiresSameOriginSiteAccessAndTheExactHostToken(t *testing.T server.updateRoomAccess(accessRequest{ roomID: first.RoomID, hostToken: first.HostToken, body: rotate, cookie: cookie, origin: "https://foreign.test", - }).expect(http.StatusForbidden, `{"error":"Forbidden"}`) + }).expect(http.StatusForbidden, `{"error":"Origin not allowed"}`) server.updateRoomAccess(accessRequest{ roomID: first.RoomID, body: rotate, cookie: cookie, @@ -1122,7 +1122,7 @@ func TestRoomCreationRejectsMalformedRequestsAndForeignBrowserOrigins(t *testing server.do(http.MethodPost, "/api/rooms", withOrigin("https://foreign.test"), withJSON(`{"codeEntryPolicy":"open"}`)). - expect(http.StatusForbidden, `{"error":"Forbidden"}`) + expect(http.StatusForbidden, `{"error":"Origin not allowed"}`) // An oversized or non-JSON body is the same 400. server.do(http.MethodPost, "/api/rooms", withOrigin(allowedOrigin), @@ -1137,6 +1137,46 @@ func TestRoomCreationRejectsMalformedRequestsAndForeignBrowserOrigins(t *testing expectHeader("Allow", "POST") } +func TestRoomCreationBehindProxyUsesConfiguredBrowserOrigin(t *testing.T) { + for _, test := range []struct { + name, publicURL, origins, browserOrigin string + status int + }{ + {"public default", "https://share.example.test", "", "https://share.example.test", http.StatusCreated}, + {"normalized default port", "https://share.example.test:443", "", "https://share.example.test", http.StatusCreated}, + {"stale localhost override", "https://share.example.test", "http://localhost:8787", "https://share.example.test", http.StatusForbidden}, + {"explicit alternate origin", "https://share.example.test", "https://alternate.test", "https://alternate.test", http.StatusCreated}, + {"different public port", "https://share.example.test:8443", "", "https://share.example.test", http.StatusForbidden}, + {"missing origin", "https://share.example.test", "", "", http.StatusForbidden}, + {"foreign origin", "https://share.example.test", "", "https://foreign.test", http.StatusForbidden}, + } { + t.Run(test.name, func(t *testing.T) { + configuration, err := config.Load(map[string]string{ + "PUBLIC_BASE_URL": test.publicURL, "ALLOWED_ORIGINS": test.origins, + "ROOM_DATABASE_PATH": ":memory:", "LISTEN_HOST": "127.0.0.1", + }) + if err != nil { + t.Fatal(err) + } + configuration.Port = 0 + server := start(t, Options{Config: configuration}) + proxy := func(request *http.Request) { + request.Host = "127.0.0.1:8787" + request.Header.Set("X-Forwarded-Host", "share.example.test") + request.Header.Set("X-Forwarded-Proto", "https") + } + server.do(http.MethodGet, "/api/site-access", proxy). + expect(http.StatusOK, `{"required":false,"authenticated":true}`) + result := server.do(http.MethodPost, "/api/rooms", proxy, + withOrigin(test.browserOrigin), withJSON(`{"codeEntryPolicy":"open"}`)) + result.expectStatus(test.status) + if test.status == http.StatusForbidden { + result.expect(test.status, `{"error":"Origin not allowed"}`) + } + }) + } +} + func TestRoomCreationReturnsServiceUnavailableAtTheGlobalRoomBound(t *testing.T) { configuration := testConfig(t) configuration.SiteAccessPassword = "" diff --git a/internal/server/app/routes.go b/internal/server/app/routes.go index bb6a013..6d48157 100644 --- a/internal/server/app/routes.go +++ b/internal/server/app/routes.go @@ -166,7 +166,7 @@ func (s *Server) handleSiteAccess(writer http.ResponseWriter, request *http.Requ } if !s.allowedRequestOrigin(request) { - sendJSON(writer, http.StatusForbidden, errorBody{"Forbidden"}) + sendJSON(writer, http.StatusForbidden, errorBody{"Origin not allowed"}) return } // Released clients send a bodyless Bearer request. JSON preserves passwords @@ -289,7 +289,7 @@ func (s *Server) handleRoomCreation( return } if !s.allowedRequestOrigin(request) { - sendJSON(writer, http.StatusForbidden, errorBody{"Forbidden"}) + sendJSON(writer, http.StatusForbidden, errorBody{"Origin not allowed"}) return } // When site access requires a password, room creation uses its cookie; @@ -337,7 +337,7 @@ func (s *Server) authorizedRoomHostToken( request *http.Request, ) (string, bool) { if !s.allowedRequestOrigin(request) { - sendJSON(writer, http.StatusForbidden, errorBody{"Forbidden"}) + sendJSON(writer, http.StatusForbidden, errorBody{"Origin not allowed"}) return "", false } if !s.roomCreationAuthorized(request) { diff --git a/native/capture/windows/main.cpp b/native/capture/windows/main.cpp index 42ed513..7d4db14 100644 --- a/native/capture/windows/main.cpp +++ b/native/capture/windows/main.cpp @@ -1265,6 +1265,20 @@ struct VideoEncoderSelection final { std::unique_ptr initial; }; +template +VideoEncoderSelection CompareSoftwareEncoder(std::unique_ptr hardware, + std::optional hardware_work, EncoderClock::time_point deadline, Measure measure) { + try { + RequireEncoderTime(deadline); + const double software_work = measure(); + if (!hardware_work || software_work <= *hardware_work) return {}; + } catch (const std::exception&) { + if (!hardware_work) throw; + } + // The deadline bounds new probing, not reuse of an already-proved encoder. + return {OutputKind::h264, std::move(hardware)}; +} + using EncoderCandidate = std::pair; void LogEncoderRejection(EncoderCandidate candidate, const GateFailure& error) { @@ -1353,17 +1367,13 @@ VideoEncoderSelection SelectVideoEncoder( hardware.reset(); device = DeviceContext{}; } - RequireEncoderTime(deadline); - if (!device.device) device = CreateDevice(SelectAdapter(adapters, arguments.adapter_index)); - try { - const double software_work = MeasureEncoderWork(nullptr, device.device.Get(), - arguments.profile, deadline); - if (!hardware_work || software_work <= *hardware_work) return {}; - } catch (const std::exception&) { - if (!hardware_work) throw; + if (!device.device) { + RequireEncoderTime(deadline); + device = CreateDevice(SelectAdapter(adapters, arguments.adapter_index)); } - RequireEncoderTime(deadline); - return {OutputKind::h264, std::move(hardware)}; + return CompareSoftwareEncoder(std::move(hardware), hardware_work, deadline, [&] { + return MeasureEncoderWork(nullptr, device.device.Get(), arguments.profile, deadline); + }); } struct CaptureInput final { diff --git a/native/capture/windows/output_worker.test.cpp b/native/capture/windows/output_worker.test.cpp index 5531b5a..38d0950 100644 --- a/native/capture/windows/output_worker.test.cpp +++ b/native/capture/windows/output_worker.test.cpp @@ -149,8 +149,47 @@ void CheckEncoderCandidates() { assert(tried.empty()); } +void CheckAutoEncoderFallback() { + const auto deadline = EncoderClock::time_point::max(); + const auto expired = EncoderClock::time_point::min(); + auto hardware = [] { return std::make_unique(nullptr, std::shared_future{}); }; + for (const auto probe_deadline : {deadline, expired}) { + auto initial = hardware(); + const auto* proven = initial.get(); + bool probed = false; + const auto selected = CompareSoftwareEncoder(std::move(initial), .04, probe_deadline, [&]() -> double { + probed = true; + // A slow software comparison exhausts its budget after H264 succeeded. + RequireEncoderTime(expired); + return 0; + }); + assert(selected.kind == OutputKind::h264 && selected.initial.get() == proven); + assert(probed == (probe_deadline == deadline)); + } + for (const double software_work : {.02, .04, .06}) { + const auto selected = CompareSoftwareEncoder(hardware(), .04, deadline, [&] { return software_work; }); + assert((selected.kind == OutputKind::vp8) == (software_work <= .04)); + assert(static_cast(selected.initial) == (software_work > .04)); + } + const auto software_only = CompareSoftwareEncoder(nullptr, std::nullopt, deadline, [] { return .03; }); + assert(software_only.kind == OutputKind::vp8 && !software_only.initial); + for (const auto probe_deadline : {deadline, expired}) { + bool failed = false; + try { + CompareSoftwareEncoder(nullptr, std::nullopt, probe_deadline, [&]() -> double { + RequireEncoderTime(expired); + return 0; + }); + } catch (const GateFailure& error) { + failed = error.stage() == "codec-probe-timeout"; + } + assert(failed); // No proved encoder must remain a startup failure. + } +} + int main() { CheckEncoderCandidates(); + CheckAutoEncoderFallback(); CheckPrimaryFailureDetails(); ComPtr device; Check(D3D11CreateDevice(nullptr, D3D_DRIVER_TYPE_WARP, nullptr, 0, nullptr, 0, diff --git a/scripts/app-browser-fanout-gate.ts b/scripts/app-browser-fanout-gate.ts index 44d38ea..e534a31 100644 --- a/scripts/app-browser-fanout-gate.ts +++ b/scripts/app-browser-fanout-gate.ts @@ -208,6 +208,10 @@ async function main(): Promise { await click(host, 'button[aria-label^="H264"]'); stage = "share-picker"; await click(host, 'button.lr-tv-big.is-action'); + // This arm measures Native ingress. Browser capture can start without App + // discovery, so first establish the control connection this gate requires. + await waitForSample(() => read(host), + (sample) => sample.controlOpen && sample.requests.includes("list-sources"), 10000); stage = "share-browser-capture"; await click(host, 'button[data-source-tab="browser"]'); await click(host, '.lr-source-option.is-browser'); @@ -263,6 +267,8 @@ async function main(): Promise { stage = "source-audio-change"; await evaluate(cdp, host, `fanoutGate.audio(${!initialAudio})`, Date.now()+5000); await click(host, 'button[aria-label="Switch source"]'); + await click(host, 'button[data-source-tab="browser"]'); + await click(host, '.lr-source-option.is-browser'); await waitForSample(() => read(host), (s) => s.receiveRequests === 2 && s.captures === 2 && s.sharing, 15000); const beforeSource = await read(viewers[1]!); await waitForSample(() => read(viewers[1]!), (s) => s.width > 0 && s.frames >= beforeSource.frames+20 && s.audio===Number(!initialAudio), 25000); diff --git a/scripts/app-native-host-gate.ts b/scripts/app-native-host-gate.ts index ff117da..33b752c 100644 --- a/scripts/app-native-host-gate.ts +++ b/scripts/app-native-host-gate.ts @@ -78,6 +78,7 @@ interface GateResult { liveQualityChanged: boolean; pausedQualityChanged: boolean; backgroundProfileRecovery: boolean | null; + sourceWindowRecovery: boolean | null; mediaObjectPreserved: boolean; nativeSourceChanged: boolean; nativeQualityEvidence: boolean; @@ -641,6 +642,7 @@ async function main(): Promise { liveQualityChanged: false, pausedQualityChanged: false, backgroundProfileRecovery: mode === "local" && sourceKind === "window" ? false : null, + sourceWindowRecovery: mode === "local" && sourceKind === "window" ? false : null, mediaObjectPreserved: false, nativeSourceChanged: false, nativeQualityEvidence: false, @@ -1242,6 +1244,36 @@ async function main(): Promise { ), Boolean, 5_000); } result.backgroundProfileRecovery = true; + // WGC must keep the share through size changes and a temporarily quiet + // minimized window. This is a window-transition check, not a game test. + for (const bounds of [ + { width: 960, height: 650 }, + { windowState: "fullscreen" }, + { windowState: "normal" }, + { windowState: "minimized" }, + ]) { + stage = "native-window-" + (bounds.windowState ?? "resize"); + await sourceCdp!.call("Browser.setWindowBounds", { + windowId, bounds, + }, undefined, Date.now() + 5_000); + if (bounds.windowState === "minimized") { + await new Promise((resolveWait) => setTimeout(resolveWait, 5_000)); + await sourceCdp!.call("Browser.setWindowBounds", { + windowId, bounds: { windowState: "normal" }, + }, undefined, Date.now() + 5_000); + } + const before = await evaluate(cdp, viewer, + "document.querySelector('video')?.getVideoPlaybackQuality().totalVideoFrames ?? 0", Date.now() + 5_000); + await waitForValue((deadline) => evaluate(cdp!, viewer, + `(() => { + const video = document.querySelector('video'); + return Boolean(video && video.srcObject === window.__piikGateMedia && + video.getVideoPlaybackQuality().totalVideoFrames >= ${before + 30}); + })()`, deadline, + ), Boolean, 20_000); + await assertVideoCodec(cdp, viewer, actualCodec); + } + result.sourceWindowRecovery = true; } stage = "native-source-picker"; await waitForValue( @@ -1481,6 +1513,7 @@ async function main(): Promise { : result.viewerConnected && result.viewerFrames >= 30 && result.qualityControlsEnabled && result.liveQualityChanged && result.backgroundProfileRecovery !== false && + result.sourceWindowRecovery !== false && result.livePresetChanges === 2 && result.codecPreserved && result.pausedQualityChanged && result.mediaObjectPreserved && diff --git a/scripts/browser-local-pool-page.js b/scripts/browser-local-pool-page.js index 51beff9..1de1b00 100644 --- a/scripts/browser-local-pool-page.js +++ b/scripts/browser-local-pool-page.js @@ -12,6 +12,7 @@ const single = settings.get('single') === '1', late = settings.get('late') === ' const nativeSource = settings.get('nativeSource') === '1', background = settings.get('background') === '1'; const lifecycle = settings.get('lifecycle') === '1', shapedRate = Number(settings.get('rate') || 120000); const relay = settings.get('relay') === '1'; +const shortPulse = settings.get('shortPulse') === '1'; const selectedProfile = () => ({ resolution: high ? '1080p' : '480p', maxFramerate: 30, maxBitrate: high ? 5000000 : 1700000, degradationPreference: settings.get('preference') || (high ? 'maintain-resolution' : 'balanced'), @@ -282,7 +283,7 @@ async function run() { extensions: e.send.localDescription?.sdp.match(/^a=extmap:.*$/gm), parameters: e.sender.getParameters().encodings })); const snapshots = [await snapshot()]; - phase = 'healthy'; await wait(6000); snapshots.push(await snapshot()); + phase = 'healthy'; await wait(shortPulse ? 45000 : 6000); snapshots.push(await snapshot()); if (lifecycle) { let appliedProfile = selectedProfile(); const apply = async profile => { @@ -341,7 +342,7 @@ async function run() { } if (network) { await fetch('/shaper?rate=' + shapedRate); - phase = 'limited'; await wait(14000); snapshots.push(await snapshot()); + phase = 'limited'; await wait(shortPulse ? 1000 : 14000); snapshots.push(await snapshot()); await fetch('/shaper?rate=0'); // --auto extends recovery observation; adaptation always belongs to the product. phase = 'released'; await wait(automatic ? 40000 : 10000); snapshots.push(await snapshot()); @@ -350,7 +351,7 @@ async function run() { audioMonitors: audioMonitors.map(({ role, pulses, attached }) => ({ role, pulses, attached })), monitors: monitors.map(({ role, samples }) => ({ role, samples })) }; } -const configuration = { mode, codec, product: true, network, automatic, av, high, single, late, +const configuration = { mode, codec, product: true, network, automatic, av, high, single, late, shortPulse, background, nativeSource, lifecycle, relay, shapedRate, profile: selectedProfile() }; try { window.probeResult = { ...configuration, ...await run(), controls, visibility, productTrace, errors }; diff --git a/scripts/browser-local-pool-probe.ts b/scripts/browser-local-pool-probe.ts index 0bf7874..9374005 100644 --- a/scripts/browser-local-pool-probe.ts +++ b/scripts/browser-local-pool-probe.ts @@ -9,7 +9,7 @@ import { createPoolShaper } from './browser-pool-shaper.mjs'; const mode = process.argv[2] || 'carrier', args = process.argv.slice(3); const flags = new Set(['--product', '--1080', '--h264', '--single', '--av', '--late', '--background', - '--native-source', '--lifecycle', '--network', '--auto', '--relay', '--debug']); + '--native-source', '--lifecycle', '--network', '--auto', '--relay', '--debug', '--short-pulse']); if (!['ordinary', 'carrier'].includes(mode)) throw Error('Expected ordinary|carrier'); for (const arg of args) if (!flags.has(arg) && !arg.startsWith('--rate=') && !arg.startsWith('--preference=')) throw Error('Unsupported option: ' + arg); @@ -18,12 +18,14 @@ const automatic = args.includes('--auto'), high = args.includes('--1080'), av = const single = args.includes('--single'), late = args.includes('--late'), background = args.includes('--background'); const nativeSource = background || args.includes('--native-source'), lifecycle = args.includes('--lifecycle'); const relay = args.includes('--relay'); +const shortPulse = args.includes('--short-pulse'); const rate = Number(args.find(arg => arg.startsWith('--rate='))?.split('=')[1] || 120000); const preference = args.find(arg => arg.startsWith('--preference='))?.split('=')[1] || (high ? 'maintain-resolution' : 'balanced'); if (!['balanced', 'maintain-resolution', 'maintain-framerate'].includes(preference)) throw Error('Unsupported degradation preference'); if (!Number.isInteger(rate) || rate <= 0) throw Error('Rate must be a positive integer in bps'); if (lifecycle && (nativeSource || network || relay)) throw Error('Lifecycle uses direct canvas capture without network shaping'); if (single && late) throw Error('Single-viewer control cannot also late-join B'); +if (shortPulse && (!network || !automatic)) throw Error('Short pulse requires --network --auto'); const root = resolve(import.meta.dirname, '..'); const bundle = await build({ entryPoints: [join(root, 'scripts/browser-local-pool-page.js')], bundle: true, write: false, format: 'esm', platform: 'browser', logLevel: 'error' }); @@ -71,7 +73,7 @@ const chrome = await launchChrome(process.env.CHROME_PATH || 'C:/Program Files/G ...(background ? [] : ['--disable-background-timer-throttling', '--disable-renderer-backgrounding']), ...(nativeSource ? ['--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream'] : [])]); chrome.stdout.resume(); chrome.stderr.resume(); -let result: Record = { mode, codec, product: true, network, automatic, high, av, single, late, +let result: Record = { mode, codec, product: true, network, automatic, high, av, single, late, shortPulse, background, nativeSource, lifecycle, relay, shapedRate: rate, preference }; try { const version = await waitForVersion(debugPort, chrome); @@ -79,7 +81,7 @@ try { page = await createPage(cdp, 'about:blank'); const query = new URLSearchParams({ mode, codec, rate: String(rate), preference }); if (process.argv.includes('--debug')) query.set('debug', '1'); - for (const [key, value] of Object.entries({ network, automatic, high, av, single, late, background, nativeSource, lifecycle, relay })) + for (const [key, value] of Object.entries({ network, automatic, high, av, single, late, background, nativeSource, lifecycle, relay, shortPulse })) query.set(key, value ? '1' : '0'); await cdp.call('Page.navigate', { url: 'http://127.0.0.1:' + port + '/?' + query }, page.sessionId, Date.now() + 5000); await waitForSample(() => evaluate(cdp!, page!, 'window.probeDone===true', Date.now() + 2000), @@ -96,7 +98,7 @@ try { result.probeSha256 = createHash('sha256').update(body).digest('hex'); result.cleanup = clean; await mkdir(join(root, 'build/browser-local-pool'), { recursive: true }); - const name = [mode, codec.toLowerCase(), network && 'network', automatic && 'auto', high && '1080', + const name = [mode, codec.toLowerCase(), network && 'network', automatic && 'auto', shortPulse && 'pulse', high && '1080', single && 'single', late && 'late', background && 'background', lifecycle && 'lifecycle', relay && 'relay', Date.now()].filter(Boolean).join('-'); const output = join(root, 'build/browser-local-pool', name + '.json'); await writeFile(output, JSON.stringify(result, null, 2) + '\n'); diff --git a/scripts/browser-local-pool-summary.mjs b/scripts/browser-local-pool-summary.mjs index edf212e..0ffd893 100644 --- a/scripts/browser-local-pool-summary.mjs +++ b/scripts/browser-local-pool-summary.mjs @@ -5,7 +5,7 @@ for (const filename of process.argv.slice(2)) { result.cleanup && Object.values(result.cleanup).every(Boolean); const configuration = { file: filename, probeSha256: result.probeSha256, mode: result.mode, codec: result.codec, product: result.product, high: result.high, single: result.single, late: result.late, network: result.network, - automatic: result.automatic, av: result.av, background: result.background, nativeSource: result.nativeSource, + automatic: result.automatic, shortPulse: result.shortPulse, av: result.av, background: result.background, nativeSource: result.nativeSource, lifecycle: result.lifecycle, relay: result.relay, shapedRate: result.shapedRate, profile: result.profile, completed: !!completed, error: result.error, errors: result.errors, cleanup: result.cleanup }; if (!completed) process.exitCode = 1; diff --git a/src/client/lib/api.ts b/src/client/lib/api.ts index 55e6f8b..c791777 100644 --- a/src/client/lib/api.ts +++ b/src/client/lib/api.ts @@ -26,11 +26,19 @@ export class ApiError extends Error { } async function responseBody(response: Response): Promise { + let body: unknown; try { - return await response.json(); + body = await response.json(); } catch { throw new ApiError(say("host.err.serverError"), response.status); } + if ( + response.status === 403 && body && typeof body === "object" && + "error" in body && body.error === "Origin not allowed" + ) { + throw new ApiError(say("api.originNotAllowed"), response.status); + } + return body; } function parseSiteAccessStatus(value: unknown): SiteAccessStatus { diff --git a/src/client/locales/en.ts b/src/client/locales/en.ts index 4c7d3b3..6931b6d 100644 --- a/src/client/locales/en.ts +++ b/src/client/locales/en.ts @@ -385,6 +385,7 @@ export const en: Record = { "gate.connectFailed": "Cannot reach the site access service; try again", "gate.expired": "Site access expired; verify again", "gate.serviceUnavailable": "Site access service unavailable ({status})", + "api.originNotAllowed": "This address is not allowed by the server (403). Ask the site administrator to check PUBLIC_BASE_URL and ALLOWED_ORIGINS.", "host.notStarted": "Not started", "host.startCancelled": "Start cancelled", diff --git a/src/client/locales/zh.ts b/src/client/locales/zh.ts index f8b1efd..1eac569 100644 --- a/src/client/locales/zh.ts +++ b/src/client/locales/zh.ts @@ -385,6 +385,7 @@ export const zh = { "gate.connectFailed": "无法连接站点访问服务，请重试", "gate.expired": "站点访问已失效，请重新验证", "gate.serviceUnavailable": "站点验证服务暂时不可用 ({status})", + "api.originNotAllowed": "服务器未允许当前网址 (403)。请联系站点管理员检查 PUBLIC_BASE_URL 和 ALLOWED_ORIGINS。", "host.notStarted": "尚未开始", "host.startCancelled": "启动已取消", diff --git a/src/client/media/browser-encoding-output.ts b/src/client/media/browser-encoding-output.ts index 8562b06..395f36a 100644 --- a/src/client/media/browser-encoding-output.ts +++ b/src/client/media/browser-encoding-output.ts @@ -63,7 +63,9 @@ export class BrowserEncodingOutput { if (!context) throw new Error("Browser carrier canvas unavailable"); this.clockContext = context; this.track = this.clock.captureStream(0).getVideoTracks()[0] as CanvasCaptureMediaStreamTrack; - this.track.contentHint = "motion"; + // This synthetic clock needs WebRTC's screen-content ALR probing when the + // real producer sends little data. The producer owns picture adaptation. + this.track.contentHint = "detail"; const streams = encodedStreams(sender); this.writer = streams.writable.getWriter(); void streams.readable.pipeTo(new WritableStream({ write: (frame) => this.receive(frame) }), diff --git a/src/client/media/browser-encoding-pool.ts b/src/client/media/browser-encoding-pool.ts index aa24c57..ed48803 100644 --- a/src/client/media/browser-encoding-pool.ts +++ b/src/client/media/browser-encoding-pool.ts @@ -217,6 +217,7 @@ export class BrowserEncodingPool { member.encoded.select(pending.group.producer.id, () => pending.group.producer.requestKey(), () => { if (member.disposed || member.pending !== pending) return; member.current = pending.group; + pending.group.producer.beginOutput(); member.pending = undefined; member.encoderStats = createStatsAccumulator(); member.previousOutput = undefined; member.carrier = true; @@ -247,9 +248,10 @@ export class BrowserEncodingPool { } continue; } - // An unshared pipeline already has the right owner: let its native - // encoder adapt to the updated budget instead of recreating it. - if (current && this.compatible(member, current) && !this.references(current).some((other) => other !== member)) continue; + // A short output-rate spike is not new demand. Keep the rate owner and + // let its native encoder adapt unless another child needs a higher rate. + if (current && this.compatible(member, current) && !this.references(current).some((other) => + this.compatible(other, current) && other.budget !== undefined && other.budget > member.budget!)) continue; const candidates = [...this.groups].filter((group) => this.compatible(member, group)); let target = candidates.find((group) => group !== current && fits(group.output, member.budget) && (!current || group.budget === member.budget)); diff --git a/src/client/media/browser-encoding-producer.ts b/src/client/media/browser-encoding-producer.ts index 990f3f1..6aac3e1 100644 --- a/src/client/media/browser-encoding-producer.ts +++ b/src/client/media/browser-encoding-producer.ts @@ -1,7 +1,6 @@ import { createOpaqueId } from "../lib/opaque-id"; import { debugError, debugEvent } from "../lib/debug"; import { debugRtcFailure, debugRtcStats, debugTrack, observeDebugConnection } from "../lib/debug-webrtc"; -import { maxEncodedVideoFrames } from "../webrtc/stats"; import { addRemoteIceCandidate } from "../webrtc/nat-prediction"; import { encodedStreams } from "./browser-encoding-output"; import { @@ -28,6 +27,7 @@ export class BrowserEncodingProducer { private paused: boolean; private budget: number; private startupPending: boolean; + private startupFrames: number | undefined; private startupTimer: ReturnType | undefined; private applied: { capture: QualityProfile; sender: QualityProfile } | null = null; @@ -67,6 +67,9 @@ export class BrowserEncodingProducer { this.streamWriter = writer; void streams.readable.pipeTo(new WritableStream({ write: async (frame) => { if (this.disposed) return; + if (this.startupPending && this.startupFrames !== undefined && !this.paused && frame.data.byteLength > 0) { + this.startupFrames = Math.min(STARTUP_VIDEO_ENCODED_FRAMES, this.startupFrames + 1); + } // Outputs clone synchronously before the local decoder consumes the original. this.onFrame(frame); await writer.write(frame); @@ -142,7 +145,15 @@ export class BrowserEncodingProducer { const request = sender.setParameters as (parameters: RTCRtpSendParameters, options: { encodingOptions: Array<{ keyFrame: boolean }> }) => Promise; await request.call(sender, sender.getParameters(), { encodingOptions: [{ keyFrame: true }] }); - }); + // Producer failure falls back to ordinary encoding on the same connection. + // Its retired keyframe requests must not fail that outgoing connection. + }).catch(this.fail); + } + + beginOutput(): void { + // Local warmup cannot train an outgoing connection's allocation. Start the + // existing protection only after a child actually writes this producer. + this.startupFrames ??= 0; } async report(): Promise { @@ -151,7 +162,7 @@ export class BrowserEncodingProducer { const report = await connection.getStats().catch((error) => { debugRtcFailure(connection, error); throw error; }); debugRtcStats(connection, report); if (!this.disposed && this.startupPending && - (maxEncodedVideoFrames(report, this.input!.id) ?? 0) >= STARTUP_VIDEO_ENCODED_FRAMES) { + (this.startupFrames ?? 0) >= STARTUP_VIDEO_ENCODED_FRAMES) { this.startupPending = false; void this.serialize(() => this.applyCurrent()).catch(this.fail); } diff --git a/src/client/media/host-sfu-route.ts b/src/client/media/host-sfu-route.ts index 35f4092..a8a8b2b 100644 --- a/src/client/media/host-sfu-route.ts +++ b/src/client/media/host-sfu-route.ts @@ -161,30 +161,11 @@ export class HostSfuRoute { this.resyncing = true; this.route.reset(); this.recovery = null; - const pending = this.pending; - const active = this.active; - this.pending = null; - this.active = null; - if (pending) { - pending.failed = true; - } - if (active) { - active.failed = true; - } + const closing = this.retirePublishers(); const accepted = this.route.accept(update); this.events.reconcileChildren([]); - await this.queueTransition(async () => { - if (active?.active) { - await active.publisher.deactivate().catch(() => false); - } - if (active) { - await disconnectPublisher(active.publisher); - } - if (pending && pending !== active) { - await disconnectPublisher(pending.publisher); - } - }); + await this.queueTransition(() => closing); if (this.closed || this.resyncGeneration !== resyncGeneration) { return "stale"; @@ -403,27 +384,23 @@ export class HostSfuRoute { this.resyncing = false; this.route.reset(); this.recovery = null; - await this.queueTransition(async () => { - const pending = this.pending; - const active = this.active; - this.pending = null; - this.active = null; - if (pending) { - pending.failed = true; - } - if (active) { - active.failed = true; - } - if (active?.active) { - await active.publisher.deactivate().catch(() => false); - } - await Promise.all([ - pending ? disconnectPublisher(pending.publisher) : undefined, - active && active !== pending - ? disconnectPublisher(active.publisher) - : undefined, - ]); - }); + const closing = this.retirePublishers(); + await this.queueTransition(() => closing); + } + + private async retirePublishers(): Promise { + const pending = this.pending; + const active = this.active; + this.pending = null; + this.active = null; + if (pending) pending.failed = true; + if (active) active.failed = true; + // Cancel before joining the transition queue: its current operation may + // itself be waiting for one of these publishers to finish preparation. + await Promise.all([ + pending ? disconnectPublisher(pending.publisher) : undefined, + active && active !== pending ? this.disconnectRetiredPublisher(active) : undefined, + ]); } private queueActivation( diff --git a/src/client/pages/HostPage.tsx b/src/client/pages/HostPage.tsx index 2f0721d..719c278 100644 --- a/src/client/pages/HostPage.tsx +++ b/src/client/pages/HostPage.tsx @@ -2485,19 +2485,20 @@ export function HostPage({ ); } else { setDetails(captureDetails(captured)); - videoCodecRef.current = await resolveStreamVideoCodec(captured); + const codec = await resolveStreamVideoCodec(captured); if (isCurrentShare(generation, shareGeneration)) { + videoCodecRef.current = codec; await startBrowserNativeIngress(generation, shareGeneration, captured); nativeStarted = nativeMediaIngressRef.current !== null; } } - setResolvedVideoCodec(videoCodecRef.current.primary); } if (!isCurrentShare(generation, shareGeneration)) { captured?.getTracks().forEach((track) => track.stop()); if (nativeStarted) disposeNativeShare(shareGeneration); return; } + if (captured) setResolvedVideoCodec(videoCodecRef.current.primary); let createdRoom: HostRoomState | null = null; let claimedRoom = false; diff --git a/src/client/sfu/peer.ts b/src/client/sfu/peer.ts index 05f0e54..fe60e3e 100644 --- a/src/client/sfu/peer.ts +++ b/src/client/sfu/peer.ts @@ -1,5 +1,6 @@ import { observeDebugConnection } from "../lib/debug-webrtc"; import { addRemoteIceCandidate } from "../webrtc/nat-prediction"; +import { waitForConnectionOperation } from "../webrtc/connection-operation"; import type { ServerMessage, SfuMedia, @@ -27,7 +28,8 @@ export class SfuPeer { SfuSignalMessage, "kind" | "description" | "media" > | null = null; - private closed = false; + private readonly lifetime = new AbortController(); + private get closed(): boolean { return this.lifetime.signal.aborted; } private signalTail: Promise = Promise.resolve(); constructor( @@ -86,7 +88,7 @@ export class SfuPeer { media?: SfuMedia, ): Promise { this.descriptionSent = false; - await this.pc.setLocalDescription(description); + await this.waitForOperation(() => this.pc.setLocalDescription(description)); if (this.closed) return; const local = this.pc.localDescription; if (!local || (local.type !== "offer" && local.type !== "answer")) { @@ -129,14 +131,14 @@ export class SfuPeer { const work = async (): Promise => { if (this.closed) return; if (message.kind === "layers" && message.activeCount !== undefined) { - await onLayers?.(message.activeCount); + await this.waitForOperation(() => onLayers?.(message.activeCount!) ?? Promise.resolve()); } else if (message.kind === "description" && message.description) { - await onDescription(message.description); + await this.waitForOperation(() => onDescription(message.description!)); for (const candidate of this.remoteCandidates.splice(0)) - await addRemoteIceCandidate(this.pc, candidate); + await this.waitForOperation(() => addRemoteIceCandidate(this.pc, candidate)); } else if (message.kind === "candidate" && message.candidate) { if (this.pc.remoteDescription) - await addRemoteIceCandidate(this.pc, message.candidate); + await this.waitForOperation(() => addRemoteIceCandidate(this.pc, message.candidate!)); else if (this.remoteCandidates.length < 64) this.remoteCandidates.push(message.candidate); else throw new Error("Too many SFU ICE candidates"); @@ -147,9 +149,13 @@ export class SfuPeer { return next; } + waitForOperation(operation: () => Promise): Promise { + return waitForConnectionOperation(this.lifetime.signal, operation); + } + close(): void { if (this.closed) return; - this.closed = true; + this.lifetime.abort(); this.pc.onicecandidate = null; this.pc.onconnectionstatechange = null; this.pc.ontrack = null; diff --git a/src/client/sfu/publisher.ts b/src/client/sfu/publisher.ts index ba8fc8a..92d6798 100644 --- a/src/client/sfu/publisher.ts +++ b/src/client/sfu/publisher.ts @@ -107,7 +107,7 @@ export class SfuPublisher { async (description) => { if (description.type !== "answer") throw new Error("SFU publisher expected an answer"); - await peer.pc.setRemoteDescription(description); + await peer.waitForOperation(() => peer.pc.setRemoteDescription(description)); }, async (activeCount) => { await this.enqueue(async () => { @@ -120,7 +120,7 @@ export class SfuPublisher { parameters.encodings.forEach((encoding, index) => { encoding.active = index < activeCount; }); - await sender.setParameters(parameters); + await peer.waitForOperation(() => sender.setParameters(parameters)); return this.peer === peer; }); }, @@ -147,7 +147,7 @@ export class SfuPublisher { this.failureStage = "source"; const sourceVideo = requiredVideo(stream); video = this.ownTrack(cloneSenderVideoTrack(sourceVideo)); - await applyVideoCaptureProfile(video, profile); + await peer.waitForOperation(() => applyVideoCaptureProfile(video!, profile)); if (this.peer !== peer) { this.releaseTrack(video); return false; @@ -193,7 +193,7 @@ export class SfuPublisher { await this.configure(profile); if (this.peer !== peer) return false; await peer.sendDescription( - await peer.pc.createOffer(), + await peer.waitForOperation(() => peer.pc.createOffer()), this.media(profile), ); if (this.peer !== peer) return false; @@ -201,11 +201,12 @@ export class SfuPublisher { this.startStats(); return true; } catch (error) { + if (video && this.video !== video) this.releaseTrack(video); + if (this.peer !== peer) return false; debugError("webrtc", "sfu-activate-failed", error, { stage: this.failureStage ?? "connect", }); - if (video && this.video !== video) this.releaseTrack(video); - if (this.peer === peer) this.fail(this.failureStage ?? "connect"); + this.fail(this.failureStage ?? "connect"); throw error; } }); @@ -238,15 +239,15 @@ export class SfuPublisher { const previousBaseline = this.startupFramesBaseline; if (videoChanged) this.resetStats(); try { - if (videoChanged) await applyVideoCaptureProfile(nextVideo, profile); + if (videoChanged) await peer.waitForOperation(() => applyVideoCaptureProfile(nextVideo, profile)); if (this.peer !== peer) return false; nextVideo.enabled = !this.paused; if (nextAudio) { nextAudio.enabled = !this.paused; nextAudio.contentHint = "music"; } - if (videoChanged) await videoSender.replaceTrack(nextVideo); - await audioSender.replaceTrack(nextAudio); + if (videoChanged) await peer.waitForOperation(() => videoSender.replaceTrack(nextVideo)); + await peer.waitForOperation(() => audioSender.replaceTrack(nextAudio)); if (this.peer !== peer) return false; this.video = nextVideo; this.audio = nextAudio; @@ -265,8 +266,8 @@ export class SfuPublisher { this.setPaused(this.paused); return true; } catch (error) { - debugError("webrtc", "sfu-source-failed", error); if (this.peer !== peer) return false; + debugError("webrtc", "sfu-source-failed", error); this.video = previousVideo; this.audio = previousAudio; if (videoChanged) { @@ -274,9 +275,10 @@ export class SfuPublisher { this.startupFramesBaseline = previousBaseline; } try { - if (videoChanged) await videoSender.replaceTrack(previousVideo); - await audioSender.replaceTrack(previousAudio); + if (videoChanged) await peer.waitForOperation(() => videoSender.replaceTrack(previousVideo)); + await peer.waitForOperation(() => audioSender.replaceTrack(previousAudio)); await this.configure(profile, videoChanged); + if (this.peer !== peer) return false; this.videoWarning = { key: "host.fail.sfuSwitch" }; return false; } catch { @@ -310,7 +312,7 @@ export class SfuPublisher { .degradationPreference; if (configureVideo) this.resetStats(); try { - if (videoChanged) await applyVideoCaptureProfile(video, profile); + if (videoChanged) await peer.waitForOperation(() => applyVideoCaptureProfile(video, profile)); const result = await this.configure(profile, configureVideo); if (this.peer !== peer) return false; if (!peer.send({ kind: "media", media: this.media(profile) })) @@ -318,11 +320,12 @@ export class SfuPublisher { this.profile = profile; return result; } catch (error) { - debugError("webrtc", "sfu-profile-failed", error, { requested: profile }); if (this.peer !== peer) return false; + debugError("webrtc", "sfu-profile-failed", error, { requested: profile }); try { - if (videoChanged) await applyVideoCaptureProfile(video, previous); + if (videoChanged) await peer.waitForOperation(() => applyVideoCaptureProfile(video, previous)); await this.configure(previous, configureVideo); + if (this.peer !== peer) return false; this.videoWarning = { key: "host.fail.sfuParams" }; return false; } catch { @@ -390,7 +393,7 @@ export class SfuPublisher { return false; try { await peer.sendDescription( - await peer.pc.createOffer({ iceRestart: true }), + await peer.waitForOperation(() => peer.pc.createOffer({ iceRestart: true })), this.media(this.profile), ); return true; @@ -478,29 +481,31 @@ export class SfuPublisher { parameters.encodings.forEach((encoding, index) => Object.assign(encoding, encodings[index]), ); - await sender.setParameters(parameters); + await peer.waitForOperation(() => sender.setParameters(parameters)); if (this.peer !== peer) return false; - const readback = await configureVideoSender( + const readback = await peer.waitForOperation(() => configureVideoSender( sender, this.startupPending ? startupVideoProfile(profile) : profile, - ); + )); if (this.peer !== peer) return false; this.senderParameters = readback; this.videoWarning = senderParameterWarning(readback); } - if (!this.audio || !this.audioSender) { + const audioSender = this.audioSender; + if (!this.audio || !audioSender) { this.audioWarning = null; return true; } try { - const audio = await configureScreenAudioSender( - this.audioSender, + const audio = await peer.waitForOperation(() => configureScreenAudioSender( + audioSender, profile.screenAudioQuality, - ); + )); if (this.peer !== peer) return false; this.audioWarning = audioSenderParameterWarning(audio); return true; } catch { + if (this.peer !== peer) return false; this.audioWarning = { key: "host.fail.sfuAudioParams" }; return false; } @@ -527,7 +532,7 @@ export class SfuPublisher { if (!peer || !video || !sender || this.statsInFlight === stats) return; this.statsInFlight = stats; try { - const report = await peer.pc.getStats(); + const report = await peer.waitForOperation(() => peer.pc.getStats()); debugRtcStats(peer.pc, report); debugTrack(video, { event: "sfu-sample" }); if (this.peer !== peer || this.video !== video || this.stats !== stats) diff --git a/src/client/sfu/subscriber.ts b/src/client/sfu/subscriber.ts index a31a3fa..3e766e4 100644 --- a/src/client/sfu/subscriber.ts +++ b/src/client/sfu/subscriber.ts @@ -73,10 +73,10 @@ export class SfuSubscriber { await peer.acceptSignal(message, async (description) => { if (description.type !== "offer") throw new Error("SFU subscriber expected an offer"); - await peer.pc.setRemoteDescription(description); + await peer.waitForOperation(() => peer.pc.setRemoteDescription(description)); if (this.peer !== peer) return; await peer.sendDescription( - preferScreenAudioStereo(await peer.pc.createAnswer()), + preferScreenAudioStereo(await peer.waitForOperation(() => peer.pc.createAnswer())), ); this.restartPending = false; }); diff --git a/src/client/webrtc/connection-operation.ts b/src/client/webrtc/connection-operation.ts new file mode 100644 index 0000000..e67ad15 --- /dev/null +++ b/src/client/webrtc/connection-operation.ts @@ -0,0 +1,21 @@ +// Closing a PeerConnection may leave its operations-chain promises pending. +// The connection owner cancels its waits; late results cannot resume its work. +// https://www.w3.org/TR/webrtc/#dfn-chain-an-asynchronous-operation +export async function waitForConnectionOperation( + signal: AbortSignal, + operation: () => Promise, +): Promise { + signal.throwIfAborted(); + let abort!: () => void; + const stopped = new Promise((_, reject) => { + abort = () => reject(signal.reason); + signal.addEventListener("abort", abort, { once: true }); + }); + try { + const result = await Promise.race([operation(), stopped]); + signal.throwIfAborted(); + return result; + } finally { + signal.removeEventListener("abort", abort); + } +} diff --git a/src/client/webrtc/host-peer.ts b/src/client/webrtc/host-peer.ts index bd7f674..9e40297 100644 --- a/src/client/webrtc/host-peer.ts +++ b/src/client/webrtc/host-peer.ts @@ -1,6 +1,7 @@ import type { CopyKey } from "../ui/copy"; import type { IceConfig, SignalPayload } from "../../shared/protocol"; import { createOpaqueId } from "../lib/opaque-id"; +import { waitForConnectionOperation } from "./connection-operation"; import type { BrowserEncodingPool, BrowserPooledSender } from "../media/browser-encoding-pool"; import { BrowserEncodingOutput, encodedStreams, supportsBrowserEncoding } from "../media/browser-encoding-output"; import { debugError, debugEvent } from "../lib/debug"; @@ -91,7 +92,8 @@ export class HostPeer { private appliedAudioSenderParameters: | AudioSenderParameterReadback | null = null; - private disposed = false; + private readonly lifetime = new AbortController(); + private get disposed(): boolean { return this.lifetime.signal.aborted; } private profileRevision = 0; private negotiationEpoch = 0; private ordinaryAnswerEpoch: number | null = null; @@ -252,16 +254,18 @@ export class HostPeer { try { try { - if (videoChanged) await videoSender.replaceTrack(nextVideoTrack); - await audioSender.replaceTrack(nextAudioTrack); + if (videoChanged) await this.waitForOperation(() => videoSender.replaceTrack(nextVideoTrack)); + await this.waitForOperation(() => audioSender.replaceTrack(nextAudioTrack)); if (audioDirectionChanged) { audioTransceiver.direction = nextAudioDirection; } } catch (error) { + if (this.disposed) return false; const [videoRollback] = await Promise.allSettled([ - videoChanged ? videoSender.replaceTrack(previousVideoTrack) : Promise.resolve(), - audioSender.replaceTrack(previousAudioTrack), + videoChanged ? this.waitForOperation(() => videoSender.replaceTrack(previousVideoTrack)) : Promise.resolve(), + this.waitForOperation(() => audioSender.replaceTrack(previousAudioTrack)), ]); + if (this.disposed) return false; if (audioDirectionChanged) { audioTransceiver.direction = previousAudioDirection; } @@ -294,6 +298,7 @@ export class HostPeer { video: videoChanged && this.connection.connectionState === "connected", audio: true, }); + if (this.disposed) return false; this.snapshot = { ...this.snapshot, error: null }; this.emit(); return !audioDirectionChanged || (await this.createOffer(false)); @@ -369,7 +374,12 @@ export class HostPeer { return false; } if (updateCaptureConstraints) { - await applyVideoCaptureProfile(videoTrack, profile); + try { + await this.waitForOperation(() => applyVideoCaptureProfile(videoTrack, profile)); + } catch (error) { + if (this.disposed) return false; + throw error; + } if ( this.disposed || this.senderVideoTrack !== videoTrack || @@ -434,7 +444,7 @@ export class HostPeer { this.pendingCandidates.push(payload.candidate); } } else { - await addRemoteIceCandidate(this.connection, payload.candidate); + await this.waitForOperation(() => addRemoteIceCandidate(this.connection, payload.candidate)); } }); } @@ -490,7 +500,7 @@ export class HostPeer { if (this.disposed) { return; } - this.disposed = true; + this.lifetime.abort(); this.nextNegotiationEpoch(); this.ordinaryAnswerEpoch = null; if (this.statsTimer !== null) { @@ -552,7 +562,7 @@ export class HostPeer { if (!owns()) return; const request = sender.setParameters as (parameters: RTCRtpSendParameters, options: { encodingOptions: Array<{ keyFrame: boolean }> }) => Promise; - await request.call(sender, sender.getParameters(), { encodingOptions: [{ keyFrame: true }] }); + await this.waitForOperation(() => request.call(sender, sender.getParameters(), { encodingOptions: [{ keyFrame: true }] })); }); binding = this.videoPool.create(source, sender, this.connection, this.desiredProfile, this.encodedOutput, () => this.enqueueSenderMutation(async () => { @@ -561,12 +571,12 @@ export class HostPeer { const next = binding?.carrierScale() === undefined ? cloneSenderVideoTrack(source) : this.encodedOutput!.track; try { this.applyPausedState(next, null); - await applyVideoCaptureProfile(next, this.desiredProfile); + await this.waitForOperation(() => applyVideoCaptureProfile(next, this.desiredProfile)); if (!owns()) { if (next !== this.encodedOutput?.track) next.stop(); return false; } - if (next !== previous) await sender.replaceTrack(next); - await configureVideoSender(sender, + if (next !== previous) await this.waitForOperation(() => sender.replaceTrack(next)); + await this.waitForOperation(() => configureVideoSender(sender, this.startupVideoProfilePending ? startupVideoProfile(this.desiredProfile) : this.desiredProfile, - binding?.carrierScale()); + binding?.carrierScale())); if (!owns()) throw new DOMException("Retired Browser pool binding", "AbortError"); this.senderVideoTrack = next; if (previous !== next && previous !== this.encodedOutput?.track) previous.stop(); @@ -574,9 +584,10 @@ export class HostPeer { return true; } catch (error) { if (!this.disposed && sender.track !== previous) { - try { await sender.replaceTrack(previous); } catch { this.dispose(); } + try { await this.waitForOperation(() => sender.replaceTrack(previous)); } catch { this.dispose(); } } if (next !== previous && next !== this.encodedOutput?.track) next.stop(); + if (this.disposed) return false; debugError("encoding-pool", "carrier-attachment-failed", error, { connectionId: this.connectionId }); return false; } @@ -596,6 +607,10 @@ export class HostPeer { }); } + private waitForOperation(operation: () => Promise): Promise { + return waitForConnectionOperation(this.lifetime.signal, operation); + } + private enqueueSenderMutation(operation: () => Promise): Promise { const result = this.senderMutationTail.then(operation); this.senderMutationTail = result.then( @@ -634,11 +649,11 @@ export class HostPeer { if (restart) { this.connection.restartIce(); } - const offer = await this.connection.createOffer(); + const offer = await this.waitForOperation(() => this.connection.createOffer()); if (!this.ownsLocalOffer(epoch)) { return false; } - await this.connection.setLocalDescription(offer); + await this.waitForOperation(() => this.connection.setLocalDescription(offer)); if ( !this.ownsLocalOffer(epoch) || !this.connection.localDescription @@ -676,7 +691,7 @@ export class HostPeer { private async flushCandidates(): Promise { const candidates = this.pendingCandidates.splice(0); for (const candidate of candidates) { - await addRemoteIceCandidate(this.connection, candidate); + await this.waitForOperation(() => addRemoteIceCandidate(this.connection, candidate)); } } @@ -688,7 +703,7 @@ export class HostPeer { return; } try { - await this.connection.setRemoteDescription(payload.description); + await this.waitForOperation(() => this.connection.setRemoteDescription(payload.description)); if (!this.ownsAnswer(epoch)) { return; } @@ -744,7 +759,7 @@ export class HostPeer { this.statsInFlight = true; const statsAccumulator = this.statsAccumulator; try { - const metricsPromise = collectConnectionMetrics( + const metricsPromise = this.waitForOperation(() => collectConnectionMetrics( this.connection, "send", statsAccumulator, @@ -752,7 +767,7 @@ export class HostPeer { trackIdentifier: captureTrack.id, audioTrackIdentifier: captureAudioTrack?.id ?? null, }, - ); + )); const capture = captureMetrics(captureTrack); debugTrack(captureTrack, { connectionId: this.connectionId, event: "sample" }); const metrics = { ...(await metricsPromise), ...capture }; @@ -792,6 +807,7 @@ export class HostPeer { } private setError(error: unknown, key: CopyKey): void { + if (this.disposed) return; debugError("webrtc", "sender-failed", error, { connectionId: this.connectionId, reason: key }); this.snapshot = { ...this.snapshot, error: { key } }; this.emit(); @@ -820,8 +836,9 @@ export class HostPeer { if (mutation.video) { try { - senderParameters = await configureVideoSender(sender, profile, this.pooledVideo?.carrierScale()); + senderParameters = await this.waitForOperation(() => configureVideoSender(sender, profile, this.pooledVideo?.carrierScale())); } catch (error) { + if (this.disposed) return false; videoSucceeded = false; debugError("webrtc", "sender-parameters-failed", error, { connectionId: this.connectionId, profileRevision, requested: profile }); } @@ -835,19 +852,21 @@ export class HostPeer { return false; } - if (mutation.video && videoSucceeded && this.pooledVideo) { - try { await this.pooledVideo.updateProfile(this.desiredProfile); } + const pooledVideo = this.pooledVideo; + if (mutation.video && videoSucceeded && pooledVideo) { + try { await this.waitForOperation(() => pooledVideo.updateProfile(this.desiredProfile)); } catch { videoSucceeded = false; } } if (mutation.audio && audioSender) { if (audioSender.track) { try { - audioSenderParameters = await configureScreenAudioSender( + audioSenderParameters = await this.waitForOperation(() => configureScreenAudioSender( audioSender, profile.screenAudioQuality, - ); + )); } catch (error) { + if (this.disposed) return false; audioSucceeded = false; debugError("webrtc", "audio-parameters-failed", error, { connectionId: this.connectionId, profileRevision }); } diff --git a/src/client/webrtc/video-codec-preflight.ts b/src/client/webrtc/video-codec-preflight.ts index 5c695e2..975ac27 100644 --- a/src/client/webrtc/video-codec-preflight.ts +++ b/src/client/webrtc/video-codec-preflight.ts @@ -268,114 +268,111 @@ function hasProbeWindow( return current.timestamp - baseline.timestamp >= PREFLIGHT_WARMUP_MS; } -function delay(milliseconds: number, signal?: AbortSignal): Promise { - return new Promise((resolve, reject) => { - if (signal?.aborted) { - reject(signal.reason); - return; - } - const onAbort = () => { - window.clearTimeout(timer); - reject(signal?.reason); - }; - const timer = window.setTimeout(() => { - signal?.removeEventListener("abort", onAbort); - resolve(); - }, milliseconds); - signal?.addEventListener("abort", onAbort, { once: true }); - }); -} - async function runH264Probe( track: MediaStreamTrack, profile: QualityProfile, signal?: AbortSignal, ): Promise { const senderConnection = new RTCPeerConnection({ iceServers: [] }); - const receiverConnection = new RTCPeerConnection({ iceServers: [] }); - // This in-process probe can start encoding as soon as one local pair works; - // gathering every candidate must not extend the codec decision. - const senderCandidates: RTCIceCandidate[] = []; - const receiverCandidates: RTCIceCandidate[] = []; - let senderRemoteReady = false; - let receiverRemoteReady = false; - let iceFailed = false; - const addCandidate = async ( - target: RTCPeerConnection, - candidate: RTCIceCandidate, - ): Promise => { - try { - await addRemoteIceCandidate(target, candidate); - } catch { - iceFailed = true; - } - }; - senderConnection.addEventListener("icecandidate", (event) => { - if (!event.candidate) return; - if (receiverRemoteReady) { - void addCandidate(receiverConnection, event.candidate); - } else { - senderCandidates.push(event.candidate); - } + let receiver: RTCPeerConnection | undefined; + let pollTimer: number | undefined; + let stopReason: Error | undefined; + let stop!: (reason: Error) => void; + const stopped = new Promise((_, reject) => { + stop = (reason) => { + stopReason ??= reason; + reject(stopReason); + }; }); - receiverConnection.addEventListener("icecandidate", (event) => { - if (!event.candidate) return; - if (senderRemoteReady) { - void addCandidate(senderConnection, event.candidate); - } else { - receiverCandidates.push(event.candidate); - } - }); - const abort = () => { - senderConnection.close(); - receiverConnection.close(); + // Closing a PeerConnection need not settle its pending SDP operations. + // Every await belongs to this one budget, including setup and statistics. + const wait = async (operation: () => Promise): Promise => { + if (stopReason) throw stopReason; + const value = await Promise.race([operation(), stopped]); + if (stopReason) throw stopReason; + return value; }; + const abort = () => stop(new DOMException("Codec probe cancelled", "AbortError")); + const timer = window.setTimeout( + () => stop(new Error("Codec probe timed out")), PREFLIGHT_DEADLINE_MS, + ); signal?.addEventListener("abort", abort, { once: true }); try { + const receiverConnection = receiver = new RTCPeerConnection({ iceServers: [] }); + // This in-process probe can start encoding as soon as one local pair works; + // gathering every candidate must not extend the codec decision. + const senderCandidates: RTCIceCandidate[] = []; + const receiverCandidates: RTCIceCandidate[] = []; + let senderRemoteReady = false; + let receiverRemoteReady = false; + let iceFailed = false; + const addCandidate = async ( + target: RTCPeerConnection, + candidate: RTCIceCandidate, + ): Promise => { + try { + await addRemoteIceCandidate(target, candidate); + } catch { + iceFailed = true; + } + }; + senderConnection.addEventListener("icecandidate", (event) => { + if (!event.candidate) return; + if (receiverRemoteReady) { + void addCandidate(receiverConnection, event.candidate); + } else { + senderCandidates.push(event.candidate); + } + }); + receiverConnection.addEventListener("icecandidate", (event) => { + if (!event.candidate) return; + if (senderRemoteReady) { + void addCandidate(senderConnection, event.candidate); + } else { + receiverCandidates.push(event.candidate); + } + }); const transceiver = senderConnection.addTransceiver(track, { direction: "sendonly", }); if (!applyH264ProbeCodec(transceiver)) { return false; } - await configureVideoSender( + await wait(() => configureVideoSender( transceiver.sender, startupVideoProfile(profile), - ); - const deadline = Date.now() + PREFLIGHT_DEADLINE_MS; - await senderConnection.setLocalDescription( - await senderConnection.createOffer(), - ); - await receiverConnection.setRemoteDescription( + )); + const offer = await wait(() => senderConnection.createOffer()); + await wait(() => senderConnection.setLocalDescription(offer)); + await wait(() => receiverConnection.setRemoteDescription( senderConnection.localDescription!, - ); + )); receiverRemoteReady = true; - await Promise.all( + await wait(() => Promise.all( senderCandidates.splice(0).map((candidate) => addCandidate(receiverConnection, candidate), ), - ); - await receiverConnection.setLocalDescription( - await receiverConnection.createAnswer(), - ); - await senderConnection.setRemoteDescription( + )); + const answer = await wait(() => receiverConnection.createAnswer()); + await wait(() => receiverConnection.setLocalDescription(answer)); + await wait(() => senderConnection.setRemoteDescription( receiverConnection.localDescription!, - ); + )); senderRemoteReady = true; - await Promise.all( + await wait(() => Promise.all( receiverCandidates.splice(0).map((candidate) => addCandidate(senderConnection, candidate), ), - ); + )); let warmupBaseline: H264ProbeSample | null = null; let measurementBaseline: H264ProbeSample | null = null; - while (Date.now() < deadline) { + for (;;) { if (signal?.aborted || track.readyState === "ended" || iceFailed) { return false; } const sample = readH264ProbeSample( - await transceiver.sender.getStats(), + await wait(() => transceiver.sender.getStats()), ); if (sample) { warmupBaseline ??= sample; @@ -396,13 +393,16 @@ async function runH264Probe( } } } - await delay(PREFLIGHT_POLL_MS, signal); + await wait(() => new Promise((resolve) => { + pollTimer = window.setTimeout(resolve, PREFLIGHT_POLL_MS); + })); } - return false; } finally { + window.clearTimeout(timer); + window.clearTimeout(pollTimer); signal?.removeEventListener("abort", abort); senderConnection.close(); - receiverConnection.close(); + receiver?.close(); } } @@ -411,7 +411,7 @@ export async function preferredVideoCodecForTrack( profile: QualityProfile, signal?: AbortSignal, ): Promise { - if (track.kind !== "video" || track.readyState === "ended") { + if (signal?.aborted || track.kind !== "video" || track.readyState === "ended") { return "vp8"; } let probe: H264ProbeTrack | null = null; diff --git a/tests/browser-encoding-pool.test.ts b/tests/browser-encoding-pool.test.ts new file mode 100644 index 0000000..afdca05 --- /dev/null +++ b/tests/browser-encoding-pool.test.ts @@ -0,0 +1,146 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { BrowserEncodingPool } from "../src/client/media/browser-encoding-pool"; +import { BrowserEncodingProducer } from "../src/client/media/browser-encoding-producer"; +import type { BrowserEncodingOutput } from "../src/client/media/browser-encoding-output"; +import { QUALITY_PROFILES } from "../src/client/media/quality"; + +const codec = { mimeType: "video/VP8", clockRate: 90_000 }; +const profile = QUALITY_PROFILES["1080p30"]; +const source = { kind: "video", readyState: "live", enabled: true, id: "source" } as MediaStreamTrack; +const producers = new Map(); +let pool: BrowserEncodingPool; + +function stats(row: Record): RTCStatsReport { + return new Map([ + ["video", { id: "video", timestamp: Date.now(), type: "outbound-rtp", kind: "video", codecId: "codec", ...row }], + ["codec", { id: "codec", timestamp: Date.now(), type: "codec", ...codec }], + ]) as unknown as RTCStatsReport; +} + +function member(budget = 2_000_000) { + const demand = { budget }; + let producerId: string | null = null; + const select = vi.fn((id: string, _requestKey: () => Promise, selected: () => void) => { + producerId = id; + selected(); + }); + const encoded = { + select, push: vi.fn(), passthrough: vi.fn(), setPaused: vi.fn(), + snapshot: () => ({ frames: producerId ? 30 : 0, width: 1920, height: 1080, timestamp: Date.now(), lastProducerId: producerId }), + } as unknown as BrowserEncodingOutput; + const connection = { connectionState: "connected", getStats: async () => stats({ targetBitrate: demand.budget }) }; + const fatal = vi.fn(); + const handle = pool.create(source, {} as RTCRtpSender, connection as RTCPeerConnection, profile, + encoded, async () => true, async () => undefined, fatal)!; + return { demand, select, fatal, handle, producerId: () => producerId }; +} + +async function sample(count = 1) { + await vi.advanceTimersByTimeAsync(500 * count); +} + +beforeEach(() => { + vi.useFakeTimers(); + vi.stubGlobal("RTCRtpSender", { getCapabilities: () => ({ codecs: [codec] }) }); + producers.clear(); + vi.spyOn(BrowserEncodingProducer.prototype, "start").mockImplementation(async function (this: BrowserEncodingProducer, budget) { + producers.set(this, { budget: budget!, bitrate: budget! / 2, bytes: 0, frames: 0 }); + }); + vi.spyOn(BrowserEncodingProducer.prototype, "report").mockImplementation(async function (this: BrowserEncodingProducer) { + const output = producers.get(this)!; + output.bytes += output.bitrate / 16; + output.frames += 15; + return stats({ bytesSent: output.bytes, framesEncoded: output.frames, frameWidth: 1920, frameHeight: 1080, + qualityLimitationReason: "none" }); + }); + vi.spyOn(BrowserEncodingProducer.prototype, "update").mockImplementation(async function (this: BrowserEncodingProducer, _profile, budget) { + producers.get(this)!.budget = budget!; + }); + vi.spyOn(BrowserEncodingProducer.prototype, "setPaused").mockImplementation(() => undefined); + vi.spyOn(BrowserEncodingProducer.prototype, "dispose").mockImplementation(() => undefined); + pool = new BrowserEncodingPool(); +}); + +afterEach(() => { + pool.dispose(); + vi.restoreAllMocks(); + vi.unstubAllGlobals(); + vi.useRealTimers(); +}); + +describe("Browser encoding pool ownership", () => { + it("keeps a shared encoder through a bitrate spike with unchanged native demand", async () => { + const first = member(), second = member(); + await sample(4); + expect(producers.size).toBe(1); + const [producer, output] = [...producers][0]; + expect(first.producerId()).toBe(producer.id); + expect(second.producerId()).toBe(producer.id); + + output.bitrate = output.budget + 16; + await sample(); + output.bitrate = output.budget / 2; + await sample(2); + + expect(producers.size).toBe(1); + expect(first.select).toHaveBeenCalledTimes(1); + expect(second.select).toHaveBeenCalledTimes(1); + expect(producer.dispose).not.toHaveBeenCalled(); + }); + + it("updates shared demand in place when all children need a lower rate", async () => { + const first = member(), second = member(); + await sample(4); + const [producer, output] = [...producers][0]; + first.demand.budget = second.demand.budget = 500_000; + await sample(2); + + expect(producers.size).toBe(1); + expect(output.budget).toBe(500_000); + expect(producer.update).toHaveBeenCalledWith(profile, 500_000); + expect(first.select).toHaveBeenCalledTimes(1); + expect(second.select).toHaveBeenCalledTimes(1); + }); + + it("gives a weaker child its own rate owner and reunites it after recovery", async () => { + const strong = member(), weak = member(); + await sample(4); + const [original, output] = [...producers][0]; + weak.demand.budget = 500_000; + await sample(3); + + expect(producers.size).toBe(2); + expect(strong.producerId()).toBe(original.id); + expect(weak.producerId()).not.toBe(original.id); + expect(output.budget).toBe(2_000_000); + expect(strong.select).toHaveBeenCalledTimes(1); + + weak.demand.budget = 2_000_000; + await sample(3); + expect(weak.producerId()).toBe(original.id); + expect(producers.size).toBe(2); + expect(strong.fatal).not.toHaveBeenCalled(); + expect(weak.fatal).not.toHaveBeenCalled(); + }); + + it("starts protection at committed output, ignoring retired selection callbacks", async () => { + const begin = vi.spyOn(BrowserEncodingProducer.prototype, "beginOutput"); + const first = member(); + let selected: (() => void) | undefined; + first.select.mockImplementation((_id, _requestKey, commit) => { selected = commit; }); + await sample(3); + expect(selected).toBeDefined(); + expect(begin).not.toHaveBeenCalled(); + selected!(); + expect(begin).toHaveBeenCalledOnce(); + + const late = member(); + let retired: (() => void) | undefined; + late.select.mockImplementation((_id, _requestKey, commit) => { retired = commit; }); + await sample(2); + expect(retired).toBeDefined(); + late.handle.dispose(); + retired!(); + expect(begin).toHaveBeenCalledOnce(); + }); +}); diff --git a/tests/browser-encoding-producer.test.ts b/tests/browser-encoding-producer.test.ts index 77175b2..5833f02 100644 --- a/tests/browser-encoding-producer.test.ts +++ b/tests/browser-encoding-producer.test.ts @@ -1,6 +1,6 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { BrowserEncodingProducer } from "../src/client/media/browser-encoding-producer"; -import { QUALITY_PROFILES } from "../src/client/media/quality"; +import { configureVideoSender, QUALITY_PROFILES } from "../src/client/media/quality"; vi.mock("../src/client/media/quality", async (original) => ({ ...await original(), @@ -9,6 +9,7 @@ vi.mock("../src/client/media/quality", async (original) => ({ })); class Track extends EventTarget { + readonly id = "capture"; enabled = true; readyState = "live"; contentHint = "motion"; @@ -25,9 +26,16 @@ class Connection { localDescription: RTCSessionDescriptionInit | null = null; remoteDescription: RTCSessionDescriptionInit | null = null; readonly close = vi.fn(() => { this.connectionState = "closed"; }); + private frames!: ReadableStreamDefaultController; + readonly getStats = vi.fn(async () => new Map([ + ["video", { type: "outbound-rtp", kind: "video", mediaSourceId: "source", framesEncoded: 100 }], + ["source", { type: "media-source", trackIdentifier: "capture" }], + ]) as unknown as RTCStatsReport); readonly sender = { + getParameters: () => ({ encodings: [{}] }), + setParameters: vi.fn(async () => undefined), createEncodedStreams: () => ({ - readable: new ReadableStream(), + readable: new ReadableStream({ start: controller => { this.frames = controller; } }), writable: new WritableStream(), }), }; @@ -38,6 +46,10 @@ class Connection { async setLocalDescription(value: RTCSessionDescriptionInit) { this.localDescription = value; } async setRemoteDescription(value: RTCSessionDescriptionInit) { this.remoteDescription = value; } state(value: string) { this.connectionState = value; this.onconnectionstatechange?.(); } + async frame(bytes = 1) { + this.frames.enqueue({ data: new ArrayBuffer(bytes) } as RTCEncodedVideoFrame); + await vi.advanceTimersByTimeAsync(0); + } } const producers: BrowserEncodingProducer[] = []; @@ -51,6 +63,7 @@ function createProducer() { } beforeEach(() => { + vi.clearAllMocks(); vi.useFakeTimers(); Connection.instances = []; vi.stubGlobal("RTCPeerConnection", Connection); @@ -66,6 +79,37 @@ afterEach(async () => { }); describe("Browser encoding producer startup", () => { + it("protects first publication instead of spending the startup frames on local warmup", async () => { + const { producer } = createProducer(); + await producer.start(); + for (const connection of Connection.instances) connection.state("connected"); + const send = Connection.instances[0]; + const preference = () => vi.mocked(configureVideoSender).mock.lastCall![1].degradationPreference; + const report = async () => { await producer.report(); await vi.advanceTimersByTimeAsync(0); }; + + for (let i = 0; i < 10; i++) await send.frame(); + await report(); + expect(preference()).toBe("maintain-resolution"); + + producer.beginOutput(); + producer.setPaused(true); + for (let i = 0; i < 5; i++) await send.frame(); + producer.setPaused(false); + await send.frame(0); + for (let i = 0; i < 4; i++) await send.frame(); + await report(); + expect(preference()).toBe("maintain-resolution"); + + // Another child's attachment must not restart this producer's protection. + producer.beginOutput(); + await send.frame(); + await report(); + expect(preference()).toBe("balanced"); + producer.beginOutput(); + await report(); + expect(preference()).toBe("balanced"); + }); + it.each(["queued", "active"])("keeps the producer after a %s candidate rejection", async (phase) => { const { producer, failed } = createProducer(); const starting = producer.start(); @@ -123,4 +167,26 @@ describe("Browser encoding producer startup", () => { expect(next.failed).not.toHaveBeenCalled(); expect(next.producer.track?.readyState).toBe("live"); }); + + it("routes keyframe request failure through the producer owner, not the outgoing connection", async () => { + const { producer, source, failed } = createProducer(); + await producer.start(); + for (const connection of Connection.instances) connection.state("connected"); + Connection.instances[0].sender.setParameters.mockRejectedValueOnce(new Error("Encoder failed")); + + await expect(producer.requestKey()).resolves.toBeUndefined(); + expect(failed).toHaveBeenCalledOnce(); + expect(producer.track).toBeNull(); + expect(source.stop).not.toHaveBeenCalled(); + }); + + it("ignores a keyframe request during ordinary recovery after the producer has retired", async () => { + const { producer, failed } = createProducer(); + await producer.start(); + Connection.instances[0].state("failed"); + expect(failed).toHaveBeenCalledOnce(); + + await expect(producer.requestKey()).resolves.toBeUndefined(); + expect(failed).toHaveBeenCalledOnce(); + }); }); diff --git a/tests/client-state.test.ts b/tests/client-state.test.ts index 66b035a..df51b0d 100644 --- a/tests/client-state.test.ts +++ b/tests/client-state.test.ts @@ -14,6 +14,7 @@ import { getSiteAccess, getOptionalRuntimeCapabilities, replaceOwnedRoom, + updateRoomAccess, } from "../src/client/lib/api.ts"; import { defaultHostDisplayName, @@ -739,6 +740,34 @@ describe("client session identity", () => { }); describe("site access API", () => { + it.each(["zh", "en"] as const)("explains rejected origins across write APIs in %s", async (lang) => { + setCopy({ lang, vis: false }); + vi.stubGlobal("fetch", vi.fn().mockImplementation(async () => + new Response(JSON.stringify({ error: "Origin not allowed" }), { status: 403 }), + )); + const message = lang === "zh" + ? "服务器未允许当前网址 (403)。请联系站点管理员检查 PUBLIC_BASE_URL 和 ALLOWED_ORIGINS。" + : "This address is not allowed by the server (403). Ask the site administrator to check PUBLIC_BASE_URL and ALLOWED_ORIGINS."; + for (const request of [ + () => authenticateSiteAccess("password"), + () => createRoom("open", null), + () => replaceOwnedRoom("4321", "host-token", "open", null), + () => updateRoomAccess("4321", "host-token", { action: "rotate-viewer-grant" }), + ]) { + await expect(request()).rejects.toMatchObject({ status: 403, message }); + } + }); + + it("does not diagnose arbitrary proxy rejection bodies as an origin mismatch", async () => { + for (const body of [JSON.stringify({ error: "Forbidden" }), "Forbidden"]) { + vi.stubGlobal("fetch", vi.fn().mockResolvedValue(new Response(body, { status: 403 }))); + await expect(createRoom("open", null)).rejects.toMatchObject({ + status: 403, + message: expect.not.stringContaining("ALLOWED_ORIGINS"), + }); + } + }); + it("checks site access and sends the exact password in a Unicode-capable JSON body", async () => { const fetchMock = vi .fn() diff --git a/tests/host-peer-tracks.test.ts b/tests/host-peer-tracks.test.ts index 7204e48..32a279c 100644 --- a/tests/host-peer-tracks.test.ts +++ b/tests/host-peer-tracks.test.ts @@ -862,6 +862,41 @@ describe("HostPeer source replacement", () => { expect(nextSenderVideo?.stop).toHaveBeenCalledOnce(); }); + it.each(["video", "audio", "rollback"])("retires a replacement blocked in %s without waiting for the browser", async (stage) => { + const video = createTrack("video", "old-video"); + const audio = createTrack("audio", "old-audio"); + const peer = createPeer(createStream(video, audio)); + await peer.start(); + const connection = FakePeerConnection.latest!; + const videoSender = connection.senders[0]!; + const audioSender = connection.senders[1]!; + const previousClone = videoSender.track!; + videoSender.deferReplaceCall = stage === "video" ? 1 : stage === "rollback" ? 2 : null; + audioSender.deferReplaceCall = stage === "audio" ? 1 : null; + audioSender.failNextReplace = stage === "rollback"; + const nextVideo = createTrack("video", "next-video"); + const nextAudio = createTrack("audio", "next-audio"); + const replacing = peer.replaceStream(createStream(nextVideo, nextAudio)); + await vi.waitFor(() => expect( + stage === "audio" ? audioSender.replaceTrack : videoSender.replaceTrack, + ).toHaveBeenCalledTimes(stage === "rollback" ? 2 : 1)); + const replacementClone = vi.mocked(nextVideo.clone).mock.results[0]!.value; + const queuedProfile = peer.updateProfile(QUALITY_PROFILES["1080p30"]); + peer.dispose(); + await expect(replacing).resolves.toBe(false); + await expect(queuedProfile).resolves.toBe(false); + expect(previousClone.stop).toHaveBeenCalledOnce(); + expect(replacementClone.stop).toHaveBeenCalledOnce(); + for (const source of [video, audio, nextVideo, nextAudio]) expect(source.stop).not.toHaveBeenCalled(); + const audioCalls = audioSender.replaceTrack.mock.calls.length; + const offers = connection.createOfferCallCount; + videoSender.releaseDeferredReplaceTrack(); + audioSender.releaseDeferredReplaceTrack(); + await new Promise((resolve) => setImmediate(resolve)); + expect(audioSender.replaceTrack).toHaveBeenCalledTimes(audioCalls); + expect(connection.createOfferCallCount).toBe(offers); + }); + it.each([false, true])("keeps the video sender and quality when only audio changes (failure=%s)", async (fail) => { const video = createTrack("video", "same-video"); const peer = createPeer(createStream(video, null)); diff --git a/tests/host-quality-ownership.test.ts b/tests/host-quality-ownership.test.ts index af0ccb8..00d6ed3 100644 --- a/tests/host-quality-ownership.test.ts +++ b/tests/host-quality-ownership.test.ts @@ -721,6 +721,44 @@ describe("Host quality ownership", () => { expect(replacement.close).toHaveBeenCalledOnce(); }); + it.each(["probe", "ingress"])("does not commit Browser codec state after sharing ends during %s", async (stage) => { + const current = fixture(false); + current.activeGenerationRef.current = null; + current.nativeShareGenerationRef.current = null; + const probe = deferred<{ primary: string }>(); + const ingress = deferred(); + const captured = { getTracks: () => [current.track] }; + const resolveStreamVideoCodec = vi.fn(() => probe.promise); + const startBrowserNativeIngress = vi.fn(() => ingress.promise); + const setResolvedVideoCodec = vi.fn(); + Object.assign(current.context, { + phase: "idle", captureBrowserSource: async () => captured, + HostAudio: class {}, setStream: vi.fn(), watchCaptureEnd: vi.fn(), + resolveStreamVideoCodec, startBrowserNativeIngress, setResolvedVideoCodec, + }); + current.disposeResources.mockImplementation(() => { + current.videoCodecRef.current = { primary: "vp8" }; + setResolvedVideoCodec(null); + }); + const starting = current.startBrowser(); + await vi.waitFor(() => expect(resolveStreamVideoCodec).toHaveBeenCalledOnce()); + if (stage === "ingress") { + probe.resolve({ primary: "h264" }); + await vi.waitFor(() => expect(startBrowserNativeIngress).toHaveBeenCalledOnce()); + } + current.context.endSharing({ key: "host.notice.stopped" }); + setResolvedVideoCodec.mockClear(); + probe.resolve({ primary: "h264" }); + ingress.resolve(); + await starting; + expect(current.videoCodecRef.current.primary).toBe("vp8"); + expect(setResolvedVideoCodec).not.toHaveBeenCalled(); + expect(startBrowserNativeIngress).toHaveBeenCalledTimes(stage === "ingress" ? 1 : 0); + expect(current.track.stop).toHaveBeenCalled(); + expect(current.createRoom).not.toHaveBeenCalled(); + expect(current.roomMutationRef.current).toBeNull(); + }); + it("retires a cancelled start ACK before publishing audio, codec or bridge state", async () => { const current = fixture(); current.nativeShareGenerationRef.current = null; diff --git a/tests/route-transition.test.ts b/tests/route-transition.test.ts index b1bf89f..c7dbe3c 100644 --- a/tests/route-transition.test.ts +++ b/tests/route-transition.test.ts @@ -906,6 +906,34 @@ describe("minimal route transition contracts", () => { expect(publishers[1]?.disconnect).toHaveBeenCalled(); }); + it.each(["disconnect", "resync"])("cancels pending Host preparation before joining its queue during %s", async (reason) => { + let cancel!: (result: boolean) => void; + const activating = new Promise(resolve => { cancel = resolve; }); + const publisher = createFakePublisher([], "pending"); + publisher.activate.mockImplementation(() => activating); + publisher.disconnect.mockImplementation(async () => { cancel(false); }); + const route = new HostSfuRoute({ + getStream: () => ({}) as MediaStream, + getProfile: () => QUALITY_PROFILES["720p30"], + getVideoCodec: () => "vp8", + reconcileChildren: () => undefined, + send: () => true, + createPublisher: () => publisher, + }); + await route.acceptAndWait({ + revision: 2, phase: "active", assignment: hostAssignment("pending"), + }); + const preparing = route.acceptConfig(sfuConfig(2, "pending")); + await vi.waitFor(() => expect(publisher.activate).toHaveBeenCalledOnce()); + const retiring = reason === "disconnect" ? route.disconnect() : route.resyncAuthoritative({ + revision: 1, phase: "active", assignment: hostAssignment(null), + }); + expect(publisher.disconnect).toHaveBeenCalled(); + await Promise.all([preparing, retiring]); + expect(publisher.setPaused).not.toHaveBeenCalled(); + await route.disconnect(); + }); + it("ignores a retired publisher failure after same-generation resync", async () => { let releaseOldConnect!: () => void; const oldConnectGate = new Promise((resolve) => { diff --git a/tests/sfu-client.test.ts b/tests/sfu-client.test.ts index f3bd49c..479de3b 100644 --- a/tests/sfu-client.test.ts +++ b/tests/sfu-client.test.ts @@ -612,13 +612,13 @@ describe("embedded SFU browser transport", () => { expect(pc.transceivers[1]!.sender.parameters.encodings[0]!.maxBitrate).toBe(64_000); }); - it("closes every owned clone during in-flight replacement without stopping the sources", async () => { + it.each(["replaceTrack", "setParameters"] as const)("closes every owned clone while %s remains pending without stopping the sources", async (operation) => { const { publisher: host, pc, video, sound } = await publisher(); let release!: () => void; const blocked = new Promise((resolve) => { release = resolve; }); - pc.transceivers[0]!.sender.setParameters.mockImplementationOnce( + pc.transceivers[0]!.sender[operation].mockImplementationOnce( async () => blocked, ); const replacement = new FakeTrack("video"); @@ -627,7 +627,6 @@ describe("embedded SFU browser transport", () => { host.setPaused(true); expect(replacement.clones[0]!.enabled).toBe(false); await host.disconnect(); - release(); expect(await replacing).toBe(false); for (const track of [ video.clones[0]!, @@ -638,6 +637,10 @@ describe("embedded SFU browser transport", () => { for (const source of [video, sound, replacement]) expect(source.stop).not.toHaveBeenCalled(); expect(pc.close).toHaveBeenCalledOnce(); + const audioCalls = pc.transceivers[1]!.sender.replaceTrack.mock.calls.length; + release(); + await vi.advanceTimersByTimeAsync(0); + expect(pc.transceivers[1]!.sender.replaceTrack).toHaveBeenCalledTimes(audioCalls); }); it("fails closed when restoring a replaced sender fails", async () => { diff --git a/tests/video-codec-preflight.test.ts b/tests/video-codec-preflight.test.ts index a846733..2498e52 100644 --- a/tests/video-codec-preflight.test.ts +++ b/tests/video-codec-preflight.test.ts @@ -1,8 +1,9 @@ -import { describe, expect, it } from "vitest"; +import { afterEach, describe, expect, it, vi } from "vitest"; import { h264ProbeTarget, h264ProbeSustainsTarget, + preferredVideoCodecForTrack, type H264ProbeSample, } from "../src/client/webrtc/video-codec-preflight.ts"; import { QUALITY_PROFILES } from "../src/client/media/quality.ts"; @@ -158,3 +159,132 @@ describe("H264 sender preflight", () => { ).toBe(false); }); }); + +describe("codec probe lifetime", () => { + afterEach(() => { + vi.unstubAllGlobals(); + vi.useRealTimers(); + }); + + function fixture(blocked = "", failReceiver = false) { + vi.useFakeTimers(); + vi.stubGlobal("window", globalThis); + let resolve!: () => void; + const pending = { promise: new Promise((ready) => { resolve = ready; }), resolve: () => resolve() }; + const wait = (stage: string) => blocked === stage ? pending.promise : Promise.resolve(); + const source = { kind: "video", readyState: "live", getSettings: () => ({}), stop: vi.fn() }; + const track = { ...source, stop: vi.fn() }; + const createElement = vi.fn(() => ({ + getContext: () => ({ fillRect() {} }), + captureStream: () => ({ getVideoTracks: () => [track] }), + })); + vi.stubGlobal("document", { createElement }); + vi.stubGlobal("RTCRtpSender", { + getCapabilities: () => ({ codecs: [{ mimeType: "video/H264", clockRate: 90_000 }] }), + }); + const connections: Connection[] = []; + class Connection { + constructor() { + if (failReceiver && connections.length === 1) throw new Error("no receiver"); + connections.push(this); + } + close = vi.fn(); + addEventListener() {} + localDescription = { type: "offer", sdp: "fixture" }; + setLocalDescription = vi.fn(async () => {}); + async setRemoteDescription() {} + async createOffer() { await wait("offer"); return this.localDescription; } + async createAnswer() { return { type: "answer", sdp: "fixture" }; } + addTransceiver() { + return { + setCodecPreferences() {}, + sender: { + track, + getParameters: () => ({ encodings: [{}] }), + setParameters: () => wait("configure"), + getStats: async () => { + await wait("stats"); + const timestamp = Date.now(); + const frames = timestamp * 30 / 1_000; + return new Map([ + ["out", { id: "out", type: "outbound-rtp", kind: "video", timestamp, + codecId: "codec", mediaSourceId: "source", framesEncoded: frames }], + ["codec", { type: "codec", mimeType: "video/H264" }], + ["source", { id: "source", type: "media-source", frames }], + ]); + }, + }, + }; + } + } + vi.stubGlobal("RTCPeerConnection", Connection); + return { + source: source as unknown as MediaStreamTrack, + track, connections, createElement, pending, + }; + } + + it.each(["configure", "offer", "stats"])("bounds a pending %s operation and cleans up", async (stage) => { + const f = fixture(stage); + let result: string | undefined; + const run = preferredVideoCodecForTrack(f.source, QUALITY_PROFILES["1080p30"]) + .then((codec) => { result = codec; }); + await vi.advanceTimersByTimeAsync(3_999); + expect(result).toBeUndefined(); + await vi.advanceTimersByTimeAsync(1); + expect(result).toBe("vp8"); + await run; + expect(f.track.stop).toHaveBeenCalledOnce(); + expect(f.source.stop).not.toHaveBeenCalled(); + expect(f.connections.every((connection) => connection.close.mock.calls.length === 1)).toBe(true); + expect(vi.getTimerCount()).toBe(0); + f.pending.resolve(); + await vi.advanceTimersByTimeAsync(0); + if (stage !== "stats") expect(f.connections[0]!.setLocalDescription).not.toHaveBeenCalled(); + }); + + it("cancels pending SDP without waiting for the browser promise", async () => { + const f = fixture("offer"); + const controller = new AbortController(); + let result: string | undefined; + const run = preferredVideoCodecForTrack(f.source, QUALITY_PROFILES["1080p30"], controller.signal) + .then((codec) => { result = codec; }); + await vi.advanceTimersByTimeAsync(0); + controller.abort(); + await vi.advanceTimersByTimeAsync(0); + expect(result).toBe("vp8"); + await run; + expect(f.track.stop).toHaveBeenCalledOnce(); + expect(vi.getTimerCount()).toBe(0); + f.pending.resolve(); + await vi.advanceTimersByTimeAsync(0); + expect(f.connections[0]!.setLocalDescription).not.toHaveBeenCalled(); + }); + + it("does not allocate an already-cancelled probe", async () => { + const f = fixture(); + await expect(preferredVideoCodecForTrack(f.source, QUALITY_PROFILES["1080p30"], AbortSignal.abort())) + .resolves.toBe("vp8"); + expect(f.createElement).not.toHaveBeenCalled(); + expect(f.connections).toHaveLength(0); + }); + + it("closes the sender if receiver construction fails", async () => { + const f = fixture("", true); + await expect(preferredVideoCodecForTrack(f.source, QUALITY_PROFILES["1080p30"])) + .resolves.toBe("vp8"); + expect(f.connections[0]!.close).toHaveBeenCalledOnce(); + expect(f.track.stop).toHaveBeenCalledOnce(); + expect(vi.getTimerCount()).toBe(0); + }); + + it("keeps a proved H264 result and retires the budget", async () => { + const f = fixture(); + const run = preferredVideoCodecForTrack(f.source, QUALITY_PROFILES["1080p30"]); + await vi.advanceTimersByTimeAsync(2_000); + await expect(run).resolves.toBe("h264"); + expect(f.track.stop).toHaveBeenCalledOnce(); + expect(f.source.stop).not.toHaveBeenCalled(); + expect(vi.getTimerCount()).toBe(0); + }); +});