diff --git a/.env.example b/.env.example index 3b71f8a..20ba2eb 100644 --- a/.env.example +++ b/.env.example @@ -35,6 +35,8 @@ SFU_UDP_PORT= SFU_LISTEN_HOST=0.0.0.0 # Optional advertised IPv4 override when the listener's address is behind NAT. SFU_PUBLIC_IP= +# Require server media for every room instead of P2P-first. Requires SFU_UDP_PORT. +SFU_ONLY=false # Ordinary peer ICE discovery. Production requires at least one STUN URL. # These are advertised addresses, not local listener addresses. They may use diff --git a/AGENTS.md b/AGENTS.md index 171e7d1..131e228 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,7 +30,7 @@ [routing/transport](./docs/standards/routing-transport.md), [media quality](./docs/standards/media-quality.md), and [presentation/lifecycle](./docs/standards/presentation-lifecycle.md). -- Media stays automatic and P2P-first; central services provide room authority, +- Media stays automatic and defaults to P2P-first; central services provide room authority, signaling, STUN, observability, and bounded embedded SFU/UDP fallback using mature LiveKit media components under [ADR-0013](./docs/adr/0013-embedded-node-local-media.md). Route changes must preserve the single-graph, single-operation model in diff --git a/README.md b/README.md index 58e85d9..a48d317 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,7 @@ watch in their browsers. - **Share from Piik App or an existing Piik site.** Choose a screen, window, browser tab or camera; available sources and audio depend on the platform. - **Direct connections first.** Media travels between participants where possible (P2P). A self-hosted server can provide automatic media forwarding (SFU) as a fallback. - **Rooms you control.** Invitations, room codes and optional room passwords, for one host and up to 20 viewers. +- **Chat while watching.** Send messages and reactions to friends, with optional on-screen chat. - **Flexible viewing.** Light and dark themes, playback controls, picture-in-picture and a connection topology view. - **One program to host a site.** Web UI, room management and optional media forwarding are packaged together. diff --git a/README.zh-CN.md b/README.zh-CN.md index 511d839..11131a2 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -33,6 +33,7 @@ Piik 是一个免费开源的屏幕共享工具,适合游戏围观、一起看 - **用 Piik App 或已有 Piik 站点分享。** 支持屏幕、窗口、浏览器标签页和摄像头;可选来源与音频能力取决于平台。 - **优先直接连接。** 画面优先在设备之间传输(P2P),自建站点可启用媒体转发(SFU)自动兜底。 - **房间由你管理。** 支持邀请链接、房间号和可选的房间密码,一位房主、最多 20 位观众。 +- **边看边聊。** 发送文字、和朋友互送表情,也可开启弹幕。 - **灵活的观看方式。** 支持深浅主题、播放控制、画中画和连接拓扑查看。 - **一个程序就能建站。** 网页、房间管理和可选的媒体转发打包在同一服务端程序中。 diff --git a/deploy/container/.env.example b/deploy/container/.env.example index c83e7a5..89c06ac 100644 --- a/deploy/container/.env.example +++ b/deploy/container/.env.example @@ -8,6 +8,8 @@ SITE_ACCESS_PASSWORD= # Optional automatic media fallback: set BOTH values and allow UDP 7882. # SFU_UDP_PORT=7882 # SFU_PUBLIC_IP=203.0.113.10 +# Optional: require SFU for all rooms instead of P2P-first (requires SFU_UDP_PORT). +# SFU_ONLY=true # Optional NAT survey: also allow UDP 3479 and 3480. # NAT_PREDICTION_ENABLED=true diff --git a/docs/README.md b/docs/README.md index 9b51ca1..4666b81 100644 --- a/docs/README.md +++ b/docs/README.md @@ -99,7 +99,8 @@ and [website preview and publishing](./operations/website.md). | Why a design was chosen | [Architecture decisions](./adr/) | | Measurements and platform limits | [Research](./research/) and [verification status](./verification-status.md) | | Camera and Host microphone | [Capture scope and limits](./research/camera-and-microphone.md) | -| Room voice proposal and media-fidelity boundaries | [Design assessment](./research/room-voice.md) | +| Unreleased text, reactions and floating room tools | [Room-interaction candidate](./research/room-interactions.md) | +| Stereo verification and HDR-to-SDR assessment | [Media fidelity](./research/media-fidelity.md) | ## Shared conventions diff --git a/docs/adr/0005-automatic-hybrid-media-routing.md b/docs/adr/0005-automatic-hybrid-media-routing.md index 4b3b970..2fc2de6 100644 --- a/docs/adr/0005-automatic-hybrid-media-routing.md +++ b/docs/adr/0005-automatic-hybrid-media-routing.md @@ -28,6 +28,14 @@ Hosted and Local always use this controller. The former rollout switch and separate Host-star authority are removed; SFU availability is still selected by configuration and the room's privacy policy, not by another routing mode. +The optional Hosted `SFU_ONLY` policy restricts this controller's candidate +eligibility to SFU. It reuses the publication/subscription ledger, deadlines, +first-frame commit and recovery. It does not create another graph or scheduler. +It disables Peer relay and background Peer work; admission failure does not +relax the restriction. The service rejects an explicit peer-only request instead +of treating it as permission to forward media. Default P2P-first and App Local +behavior are unchanged. + ### One Capacity Rule Every non-server endpoint uses one server-authoritative steady outbound media- diff --git a/docs/adr/0008-window-scoped-audio-capture.md b/docs/adr/0008-window-scoped-audio-capture.md index 9283c79..820837d 100644 --- a/docs/adr/0008-window-scoped-audio-capture.md +++ b/docs/adr/0008-window-scoped-audio-capture.md @@ -2,7 +2,7 @@ - Status: accepted for Browser capture and the Windows App audio paths - Date: 2026-08-21 -- Last updated: 2026-08-27 +- Last updated: 2026-10-01 ## Context @@ -26,8 +26,8 @@ platform probe and physical gate report it. Track presence does not prove its source or isolation. 3. Returned screen-audio tracks use `contentHint = "music"`. Current bounded sender ceilings and narrow Opus stereo answer normalization are owned by - [media quality](../standards/media-quality.md). There is no Web Audio mixer, - resampler, second representation, or independent audio clock. + [media quality](../standards/media-quality.md). Audio scope adds no second + representation or independent clock; Host commentary uses the existing mixer. 4. Browser source switching replaces the current screen-audio track in the same persistent media stream. Local Viewer play, mute, and volume stay inside the native media element. @@ -38,6 +38,13 @@ platform probe and physical gate report it. probe and physical gate pass. WGC/MF video and the App media boundary are owned by [ADR-0010](./0010-cross-platform-client-runtime.md); measurements are in [Native App media](../research/native-client-media.md). +7. Windows App screen sharing can optionally exclude one selected process tree + through WASAPI's exclusion mode, on build 20348+ with a successful capability + probe. This mode spans all render endpoints; it is not restricted to the + default playback device. Reuse source enumeration and the existing PCM/mixed + output. Generic Browser capture and arbitrary unrelated process sets are not + covered. [Media quality](../standards/media-quality.md#screen-audio) owns + selection lifetime and failure behavior. ## Consequences diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index 776a68a..983962e 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -7,6 +7,7 @@ watching, using Piik online, using Piik App and sharing through an existing site [Join a room](#join-a-friends-room) · [Use online](#use-piik-online) · [Piik App](#share-with-piik-app) · [Existing site](#share-from-an-existing-site) · +[Chat and reactions](#chat-and-reactions) · [Troubleshooting](#when-something-gets-in-the-way) Viewers need a browser and an invitation. To start sharing, download Piik App @@ -154,6 +155,36 @@ then select a picture card to start sharing. Device names and previews depend on browser permissions; while a camera is shared, other cameras are listed by name. Use headphones to keep speaker sound from feeding back into the microphone. +### Exclude a voice app from screen audio + +On supported Windows Apps, open the **Screens** tab and enable **System sound**. +Under **Exclude application audio**, select a window belonging to the voice app, +then choose the screen to share. Open the app's window and refresh if it is absent. +This excludes one process and its children. Other apps remain audible, including +those playing through other output devices. +It does not mute your Piik microphone or prevent speaker sound reaching a microphone. + +If the excluded app closes or restarts, source audio stops. Use **Switch source** +and select it again. Turning source audio off retains the exclusion choice; +select **None** to restore unfiltered system audio. This control requires an updated page and a Windows App +whose capability probe supports process exclusion; it is not a Browser-capture filter. + +## Chat and reactions + +Once the host starts the first share, participants can select **Chat** below the +sofa to send messages. Select a person on the sofa to open their reaction menu: +your own reactions appear above you; reactions sent to a friend fly towards them. +The chat window can be moved and resized without rearranging the page. + +**On screen**, beside the message box or in the viewer's playback bar, shows new +messages over the picture. It is off by default and only changes your own view. +System video fullscreen and picture in picture cannot show these overlays. + +Stopping a share keeps the room's chat available. App Local and Public invite +rooms require the App to stay open. Each page retains only its latest received +messages: refreshing clears them, and late joiners receive no earlier history. +Chat travels through the room's server and is **not end-to-end encrypted**. + ## When something gets in the way For sound, capture, startup or playback issues, see [Troubleshooting](./troubleshooting.md). diff --git a/docs/guide/getting-started.zh-CN.md b/docs/guide/getting-started.zh-CN.md index ef3ef23..147e12f 100644 --- a/docs/guide/getting-started.zh-CN.md +++ b/docs/guide/getting-started.zh-CN.md @@ -6,7 +6,8 @@ Piik 支持一位房主分享屏幕,最多 20 位受邀观众通过浏览器 本指南介绍加入房间、使用在线版、使用 Piik App,以及通过已有站点分享的方法。 [加入房间](#加入朋友的房间) · [在线版](#使用在线版) · -[Piik App](#用-piik-app-分享) · [已有站点](#使用已有站点分享) · [问题排查](#遇到问题了) +[Piik App](#用-piik-app-分享) · [已有站点](#使用已有站点分享) · +[聊天与表情](#聊天与表情) · [问题排查](#遇到问题了) 观众只需要浏览器和邀请链接。 分享画面推荐下载 Piik App,选择 **公网邀请**。 @@ -133,6 +134,31 @@ Linux 原生采集还需要[系统组件](../../native/capture/linux/README.md) 设备名称和可用预览取决于浏览器权限;正在分享摄像头时,其他摄像头只显示名称。 建议佩戴耳机,避免扬声器声音被麦克风再次收进去。 +### 分享屏幕时排除语音软件的声音 + +在支持此功能的 Windows App 中,打开 **屏幕** 页签并开启 **系统声音**, +在 **排除应用声音** 中选择语音软件的窗口,再选择要分享的屏幕。 +未列出的应用请先打开窗口,再刷新。此功能排除所选进程及其子进程, +其他应用的声音仍会被分享,包括输出到其他播放设备的声音。 +它不影响 Piik 麦克风,也不能消除扬声器被麦克风收音产生的回声。 + +所选应用关闭或重启后,来源声音会停止;请通过 **切换来源** 重新选择。 +暂时关闭来源声音会保留排除选择;需要恢复完整的系统声音时,选择 **不排除**。 +此控件需要新版页面和通过进程排除能力检测的 Windows App,不适用于浏览器自带的分享选择器。 + +## 聊天与表情 + +房主首次开始分享、创建房间后,参与者可以点击沙发下方的 **聊天** 发送消息。 +点击沙发上的人物可打开表情菜单:给自己发的表情出现在头顶,送给朋友的表情会飞向对方。 +聊天窗口可以移动和缩放,不会挤动页面。 + +输入框旁和观众播放控制栏中的 **弹幕** 可以把新消息显示在画面上,默认关闭,只影响自己的观看。 +系统视频全屏和画中画无法显示弹幕。 + +停止分享后仍可继续聊天;App 的本地房间和公网邀请需要保持 App 运行。 +每个页面只保留本页最近收到的消息,刷新后清空,后来加入的人看不到此前的记录。 +聊天消息经过房间所在的服务器,**不提供端到端加密**。 + ## 遇到问题了 声音、采集、启动或播放遇到问题时,先看[问题排查](./troubleshooting.zh-CN.md)。 diff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.md index b5ae60e..fed063f 100644 --- a/docs/guide/troubleshooting.md +++ b/docs/guide/troubleshooting.md @@ -147,8 +147,9 @@ Try these in order, stopping when the picture arrives: 6. **Invite a friend on another network.** A Viewer who can receive the picture and has spare forwarding capacity may give Piik another path to you. Piik chooses this automatically; extra people help only when those connections work. -7. **Use a site with media fallback.** Its operator must enable SFU forwarding, - and the Host must turn off **Privacy mode** before sharing. The public +7. **Use a site with media fallback.** Its operator must enable SFU forwarding. + If the site offers **Privacy mode**, the Host must turn it off before sharing. + Sites configured for **Server media** already require SFU. The public `demo.piik.tv` site and the App's **Public invite** mode do not provide SFU. [Self-hosting](../operations/self-hosting.md#optional-media-fallback) is an advanced option. Both P2P and SFU media currently use UDP; a network that blocks diff --git a/docs/guide/troubleshooting.zh-CN.md b/docs/guide/troubleshooting.zh-CN.md index 92d197b..6bae1e0 100644 --- a/docs/guide/troubleshooting.zh-CN.md +++ b/docs/guide/troubleshooting.zh-CN.md @@ -123,7 +123,8 @@ NAT(网络地址转换)让多台设备共用一个上网地址。不同路 修改路由器工作模式前,先备份配置。 6. **请另一网络的朋友加入。** 如果对方能正常看到画面,且有余力转发,Piik 可能经由这位观众 把画面送到你这里。线路由 Piik 自动选择,能否帮上忙取决于这些连接是否可用。 -7. **使用有媒体兜底的站点。** 管理员需要先启用 SFU 转发,房主在分享前关闭 **隐私模式**。 +7. **使用有媒体兜底的站点。** 管理员需要先启用 SFU 转发。如果页面提供 **隐私模式**, + 房主应在分享前关闭;配置为 **服务器转发** 的站点已经强制使用 SFU。 公开站点 `demo.piik.tv` 和 App 的 **公网邀请** 模式均没有 SFU 兜底。 有服务器的用户也可以按 [部署指南](../operations/self-hosting.zh-CN.md#可选启用媒体兜底) 配置,属于进阶操作。 目前 P2P 和 SFU 媒体都使用 UDP;网络完全禁止 UDP 时,仍需换网络或由管理员放行。 diff --git a/docs/operations/self-hosting.md b/docs/operations/self-hosting.md index fb73ea0..a8c1108 100644 --- a/docs/operations/self-hosting.md +++ b/docs/operations/self-hosting.md @@ -112,6 +112,9 @@ permission; `self` allows this site to request access. Allow **TCP 80/443** for HTTPS and **UDP 3478** for STUN in the server firewall and cloud security group. Keep TCP 8787 private. The STUN hostname must resolve directly to the server; a CDN HTTP proxy does not forward its UDP traffic. +See the [complete port table](../standards/configuration.md#public-and-private-ports) +for optional services. Embedded SFU media shares the single UDP port set by +`SFU_UDP_PORT`. Open `https://share.example.com/healthz`; it should return `{"status":"ok"}`. Then open the site, share a screen and join from another device. This initial @@ -122,9 +125,15 @@ configuration uses P2P media, so participants need a usable UDP path. Add `SFU_UDP_PORT=7882` to `.env`, allow UDP 7882, and restart Piik to enable automatic SFU fallback. When the server is behind NAT, also set `SFU_PUBLIC_IP` to its reachable public IPv4 address. The same binary provides the fallback. +Allow server forwarding capacity and outbound bandwidth for Viewers actually +using the SFU. The Host must turn off **Privacy mode** before sharing to allow this route. Both direct and SFU media need a usable UDP connection. +To send every room through your server, also set `SFU_ONLY=true` and restart. +The sharing settings will show **Server media**. Plan server bandwidth for every +Viewer; this mode does not fall back to P2P if the server route fails. + ## Keep it running and update For automatic startup, use the [systemd or container guide](./service-management.md). diff --git a/docs/operations/self-hosting.zh-CN.md b/docs/operations/self-hosting.zh-CN.md index 9ad9cdd..13501a2 100644 --- a/docs/operations/self-hosting.zh-CN.md +++ b/docs/operations/self-hosting.zh-CN.md @@ -101,6 +101,8 @@ nginx 可参考[配置示例](../../deploy/nginx/piik.conf.example)。如果沿 在服务器防火墙和云平台安全组中放行 **TCP 80/443**(HTTPS)和 **UDP 3478**(STUN)。 TCP 8787 仅供本机反向代理访问。STUN 域名需要直接解析到服务器,不能只经过 CDN 的 HTTP 代理。 +可选服务的端口见[完整端口表](../standards/configuration.md#public-and-private-ports)。 +内置 SFU 的媒体流复用 `SFU_UDP_PORT` 指定的单个 UDP 端口。 打开 `https://share.example.com/healthz`,应返回 `{"status":"ok"}`。 随后打开站点,分享一个画面,并用另一台设备加入验证。 @@ -111,9 +113,14 @@ TCP 8787 仅供本机反向代理访问。STUN 域名需要直接解析到服务 在 `.env` 中添加 `SFU_UDP_PORT=7882`,放行 UDP 7882,再重启 Piik,即可启用自动 SFU 兜底。 如果服务器处于 NAT 后方,还需将 `SFU_PUBLIC_IP` 设置为外部可达的公网 IPv4 地址。 这项功能由同一个服务端程序提供。 +服务器需要为实际经由 SFU 观看的观众承担转发负载和出口带宽。 房主需要在开始分享前关闭 **隐私模式**,才会允许使用这条线路。 直连和 SFU 媒体都需要可用的 UDP 通路。 +如需所有房间都通过服务器转发,再设置 `SFU_ONLY=true` 并重启。 +分享设置中会显示 **服务器转发**。请为每位观众预留服务器带宽; +服务器线路失败时,此模式不会改走 P2P。 + ## 长期运行与更新 需要开机启动时,参阅 [systemd 与容器配置](./service-management.zh-CN.md)。 diff --git a/docs/project-memory.md b/docs/project-memory.md index 8b4f721..e257a60 100644 --- a/docs/project-memory.md +++ b/docs/project-memory.md @@ -1,6 +1,6 @@ # Project Memory -Last updated: 2026-09-29 +Last updated: 2026-10-01 Piik is private, low-latency screen sharing for one Host and up to 20 authenticated friends. The current product surface is Web Host, Web Viewer, and @@ -43,7 +43,8 @@ blank keeps the self-contained site open. ## Product Map - [Rooms and access](./standards/rooms-access.md) owns room codes, - invitations, code entry, credentials, and lightweight/SQLite persistence. + invitations, code entry, credentials, room interactions and lightweight/SQLite + persistence. - [Routing and transport](./standards/routing-transport.md) owns P2P-first distribution, endpoint capacity, SFU fallback, recovery, and privacy limits. - [Capture and media quality](./standards/media-quality.md) owns Browser capture, @@ -70,8 +71,9 @@ implementation and routine UI detail. available, and anonymous provenance distinguishes ordinary from predicted selected paths. The configuration default remains stock ICE. The only application fallback is one bounded embedded SFU/UDP - publication; Piik configures no TURN or TCP media route. A pre-share - peer-only policy can exclude that SFU suffix for one share generation. + publication; Piik configures no TURN or TCP media route. Hosted operators may + require SFU-only media through the same controller. On P2P-first sites, a + pre-share peer-only policy can exclude that SFU suffix for one share generation. - Every endpoint shares one steady outbound-copy cap, default `2` and limited to `1..3`. One room controller owns one committed graph, one reconcile loop, and one serial child operation. First decoded frame commits availability work; diff --git a/docs/research/browser-local-encoding-pool.md b/docs/research/browser-local-encoding-pool.md index 74a2b9d..873c448 100644 --- a/docs/research/browser-local-encoding-pool.md +++ b/docs/research/browser-local-encoding-pool.md @@ -93,6 +93,14 @@ maximum queued service; reverse feedback is unshaped. They are not a model of public NAT, geographic RTT or every congestion/loss pattern. Reduced quality and temporary stalls under real constraints remain possible. +A 2026-09-30 actual-tab fault check separated A at 400 kbps from healthy B, +then failed only A's producer. Both H264 and VP8 retained the outgoing +connections, returned to ordinary encoding and retired all pooled groups. +B retained 1904x1048 at about 30 fps; its largest observed frame-callback gap +around fallback was 271 ms for H264 and 80 ms for VP8. Source-wide fallback +therefore has a measurable handoff cost, even without a resolution drop or +connection failure. This does not establish low-end hardware capacity. + ### Balanced Startup And Recovery Serial Chrome 152 checks on 2026-09-25 reproduced two composition defects in @@ -316,12 +324,19 @@ complexity: changing to simple bars restored 1080p, retained after restoring the textured scene. WebRTC's [quality scaler](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/modules/video_coding/utility/quality_scaler.cc) uses QP thresholds and sampling history as well as bandwidth-related adaptation. Its default two-second checks and longer sampling after down-adaptation can -outlast a short disturbance. Native Chrome 154 logging also confirmed a stable -QP deadband with the NVIDIA H264 MFT: after one 1048p-to-786p reduction, smoothed -QP stayed between the recorded 24/37 thresholds for the remaining 90-second -observation. Capture remained full size, but no low-QP upscale was requested. -This is not merely old samples taking time to expire; restored bandwidth alone -does not require this scaler to restore resolution. +outlast a short disturbance. A Chrome 154 ordinary-sender tab control on +2026-09-30 remained at 524p / about 30 fps after 120 recovery seconds following two +one-second rate/loss pulses. Actual capture stayed 1904x1048 / 30 fps; the native +log's slow QP filter settled around 27, above its upscale threshold of 24. +Changing only the scene to simple moving bars restored full size in about +12 seconds. Restoring the textured scene then retained full size for the +remaining 30 seconds. The sender made only its two startup parameter writes; +no pool, reconnect, preference reset or frame reset caused that recovery. +This confirms a content-dependent stable state, not merely stale samples or +a blocked recovery operation. Restored bandwidth alone does not require this +scaler to restore resolution. The scaler and default-threshold source matched +inspected upstream main on that date; this does not establish every device's +behavior or the field report's cause. A matched three-arm comparison used fixed 5 Mbps, alternating 5.0/4.9 Mbps every 500 ms, and repeated identical 5 Mbps writes. Native reconfiguration @@ -335,6 +350,149 @@ perceptual blur measurement. They do not justify delaying real demand, bitrate compensation or periodic resets; equal product updates are already deduplicated. The field's repeated brief blur still lacks matching Host histories. +**Hardware encoder startup can block sibling encoding.** On 2026-10-01, +Windows Chrome 154.0.8037.58 tests added three independent Canvas H264 producers, +coinciding with 174/221/225 ms gaps in the existing producer; captured frames +continued within 47 ms. The new producers never joined the pool or replaced its +output, and the existing +producer's parameters did not change. Their NVIDIA initialization took +161/163/162 ms. Substituting software VP8, confirmed by RTCStats as libvpx, gave +60/48/109 ms existing-producer gaps in the same startup windows. +A subsequent trace recorded 148 ms of `CreateAndInitializeVEA` on the shared +Renderer Media thread, matching GPU-side initialization. Frames timestamped +during that interval entered encoding together on the same thread after it +returned. This establishes blocking in the local reproduction, consistent with +[Chromium's synchronous initialization IPC](https://github.com/chromium/chromium/blob/154.0.8037.58/media/mojo/clients/mojo_video_encode_accelerator.cc#L117). +The trace buffer lost events elsewhere; this conclusion uses complete task and +frame events, not missing events. It does not identify the driver's internal +cost, explain earlier downscaling or attribute unmatched field reports. No +codec-default change, spare prewarmed encoder or pool reset follows from it. +The Windows [VEA provider](https://github.com/chromium/chromium/blob/154.0.8037.58/media/mojo/services/mojo_video_encode_accelerator_provider.cc) +already assigns each service a dedicated COM STA runner; the trace likewise +places the old service's encoding and new initialization on different threads. +Asynchronous Renderer initialization is therefore a plausible upstream repair, +not merely moving the wait onto another shared queue. It still needs a patched +Browser comparison, including initialization failure and callback ordering; +shared GPU/driver cost remains unproved. + +**Optional reuse must preserve the existing rate owner.** A two-Viewer, +textured-tab H264 comparison exposed a Piik handoff defect. During recovery +from a one-second rate/loss pulse, a healthy Viewer moved from a 5 Mbps group +to a lower-budget group because its sampled output was 29 rather than 27 fps +at the same 1048p. Both reported `qualityLimitationReason:none`, but the same +NVIDIA encoder implementation's interval-average QP was about 51 on the +candidate versus 37 on the current output; the candidate then fell to 786p. The +[stats definition](https://www.w3.org/TR/webrtc-stats/#dom-rtcqualitylimitationreason-none) +describes resolution/FPS limitation, not equal quantization quality. + +Pending membership raised the candidate's budget before the existing +lower-budget rejection ran. The candidate repair checks this constraint before +membership and reuses one non-regression predicate before submitting a pending +selection. FPS comparisons also respect the Host ceiling; a sampled 32 fps +under a 30 fps ceiling is not an improvement over 30 fps. Initial output and +necessary downgrades retain their path, without a QP policy, timer or encoder reset. + +Selection also needs proof at the actual write boundary: while waiting for a key, +a candidate can downscale after passing the sampled comparison. The output now +asks the pool to revalidate current authority, budget and actual key dimensions +before replacing its queue or writing bytes. Rejection clears both pending owners, +invalidates a contradicted sample and retains prior output and recovery requests. +H264/VP8 display-capture checks preserve live/paused settings, source replacement +and ordinary fallback; played audio remains present. Brief video stalls remain. + +The bounded H264 and VP8 two-Viewer controls for this admission repair retained +the healthy sibling at 1048p and about 29 fps through both recovery intervals. +The constrained Viewer still adapted and recovered. This establishes one +avoidable sibling degradation path, not the field room's cause or elimination +of all brief blur. The affected admission rules predate the display-isolation +repair. Raw observations stay in ignored fixtures; comparisons read native stats +without calling the producer's startup-transition helper. + +Current demand also matters at group admission. Two-Viewer traces showed a +newcomer raising an incumbent group's allocation because matching used the +previous applied budget. The incumbent then needed another encoder, while the +newcomer's former encoder retired. Matching current native demand prevents that +exchange; a pending newcomer cannot establish compatibility with its own vote. +Cancellation must stop at accepted output writing, including a pause that delays +its completion. These are ownership repairs, not suppression of real demand +changes or removal of the measured hardware initialization cost. + +**Quiet content is not rate compatibility.** A separate fixed-allocation +comparison kept the two outgoing H264 ceilings at 3 and 5 Mbps, with no network +shaping, while alternating simple and textured moving content. Actual native +allocations stayed at those ceilings during the later scene cycles. Reuse based +on a quiet 500 ms byte-rate sample repeatedly moved the weaker child into the +5 Mbps producer; returning motion split it out again. Five producers were created +over the run. Requiring the candidate's rate budget to fit the child's allocation +at initial admission, optional reuse and pending revalidation retained two +producers with no scene-driven handoffs. The healthy H264 sibling retained full +1048p; the weaker output continued native adaptation. VP8 also retained two +producers through the same scene changes. Three controlled regressions fail +when these checks are removed, including a paused pending child whose candidate +budget rises. Unequal demands may retain an additional producer even during a +quiet scene; bounded membership and native recovery remain unchanged. These +checks remove unnecessary encoder churn, not all content-dependent adaptation. + +**A known allocation should initialize the local encoder.** A replacement local +PeerConnection otherwise starts its own estimate at WebRTC's default 300 kbps, +even when the outgoing connection already allocates 4 Mbps. Chrome 154's native +logs confirm this second cold start. On the same isolated 1904x1048 textured tab, +seeding only the local answer with that 4 Mbps allocation increased VP8 output +in the first second from 6 to 27 frames; its first sampled target was 3.33 Mbps +rather than 2.70 Mbps. The 400 kbps control retained its native 333 kbps initial +encoder target and two first-second frames with either version. These bounded +results do not promise faster first frames or remove weak-network adaptation. + +The hint uses the existing Host-bounded allocation, not a multiplier or floor. +WebRTC reads the send configuration from the +[remote description](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/pc/channel.cc) +and accepts a positive `x-google-start-bitrate` in its +[codec bitrate configuration](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/media/engine/webrtc_media_engine.cc). +An offer-only control lost the hint in Chrome's answer and did not improve +startup. Piik therefore changes only its owned local answer; external SDP, +codec identity and subsequent sender-budget updates stay unchanged. The +application writes this hint once per producer, not on each budget change. +The full two-Viewer VP8 follow-up retained the healthy child at 1048p through +two one-second 400 kbps / 4% loss pulses at 40 ms configured RTT. The constrained +child still down-adapted and froze during recovery, but regained 1048p within +each 40-second recovery window. This is not a freeze-free claim. A separate +audio/loss check delivered all 97 scheduled audio pulses to both played outputs; +H264 quiet-start, live/paused settings, source replacement and ordinary-fallback +checks passed. All owned test connections, processes and profiles retired. + +**Residual VP8 stalls do not require an output wait.** The ordinary two-Viewer +control with the same source and rate/loss pulses also froze and downscaled, +ending the recovery windows at 786p and 524p. It entered the second pulse at a +lower resolution than pooling, so their recovery FPS is not a fair cost comparison. +Replaying the pooled child's budget sequence on a warmed local producer without +a shaper, carrier, queue or handoff produced nine frames in the same eight-second +low-budget interval; adding the two original keyframe requests yielded eight, +matching the original interval. Startup protection had already ended. Native +logs show frame dropping and the scaler's spaced downscales. This establishes +stalls under that budget sequence, not the cause of the sequence: keyframe bursts +can still affect downstream estimation, and parameter reconfiguration remains +in the replay. These results do not support removing dependency recovery or +startup protection, growing queues or repeatedly resetting encoders. Matched +field evidence and broader transport/allocation effects remain open. +The rate/loss run with audio retained all 65 pulses after startup at both +played outputs, despite video stalls; this does not establish gap-free video. +A separate six-second complete RTP-media drop on one child, preserving RTCP +and signaling, resumed its played video 157 ms after release for VP8 and 128 ms +for H264. Both healthy siblings retained full dimensions; H264 still recorded +brief sibling stalls. All post-release audio pulses arrived. This bounded local +check does not establish recovery for an interrupted relay or reporter's network. + +A matched single-Viewer control separates pool handoffs from ordinary adaptation. +Pooled sending used one producer with no handoffs; ordinary sending used no pool +and made only two parameter writes. After the second one-second rate/loss pulse, +both reached 262p before ending their 40-second recovery window at 390p / 15 fps. +Actual source frames stayed 1904x1048 at about 30 fps. Native logs showed quality +scaler down-adaptation, Chromium's below-360p software fallback and subsequent +recovery to NVIDIA H264 at 390p; this was not a hardware failure. Pool reuse and +frequent application-level budget writes are therefore not necessary for this +bounded low-output state. Their effect on the intervening trajectory and the +unmatched field reports remains distinct. + The tab case exposed a more specific capture-feedback problem. A Chromium trace kept the source geometry at 1904x1049 while capture supplied a 268x148 content rectangle. Feedback rose to 66,600 pixels, but capture did not grow. diff --git a/docs/research/media-fidelity.md b/docs/research/media-fidelity.md new file mode 100644 index 0000000..eaa7ec2 --- /dev/null +++ b/docs/research/media-fidelity.md @@ -0,0 +1,162 @@ +# Media Fidelity + +Reviewed 2026-10-01. [Media quality](../standards/media-quality.md) owns behavior; +[TODO](../todo.md) owns remaining work. These synthetic checks do not establish +all-device acceptance or full HDR support. + +## Stereo + +Browser display capture requests two channels without speech processing. The +Host mixer uses a stereo Web Audio destination; Native capture and mixing use +48 kHz, two-channel PCM16. Native Opus uses the audio application mode. Browser, +Native and SFU receivers request stereo. The existing 64/128/192 kbps choices +are ceilings, not a lossless-audio promise. + +`opus/48000/2` alone does not prove stereo content: Opus permits mono packets +under that declaration. [RFC 7587](https://www.rfc-editor.org/rfc/rfc7587.html#section-7) +distinguishes receive preferences from packet content. Test distinct signals +after decoding instead of inferring fidelity from SDP or a two-channel mixer. + +The embedded SFU browser gate sends 500 Hz left and 1500 Hz right, and measures +decoded channel separation. On local Chrome 154, both the default fallback and +SFU-only arms retained more than 77 dB separation in both channels at the 64 kbps +ceiling, including normal video reception. Source changes and subscription +recovery retain the existing audio gate. Run `npm run gate:embedded-sfu` with `PIIK_EMBEDDED_SFU_GATE=true`, +`CHROME_PATH`, and optional `PIIK_EMBEDDED_SFU_ONLY=true`; it cleans up its own +processes and ports. This is synthetic local evidence, not an acoustic or +public-network measurement. + +A local Chrome 154 check through the production Host/Viewer peers retained more +than 57 dB separation after direct reception and after one decoding/re-encoding +relay, with a third tone mixed as Host microphone input. The receiver used its +normal video playback sink; a receiver-only Web Audio probe without playback +returned silence and was not a fidelity result. The Native Opus regression test +independently requires more than 20 dB decoded separation at all three bitrates. +These checks support keeping the existing stereo path. They do not cover every +physical source device, browser or acoustic setup, and do not establish surround +sound support. + +## SDR Conversion + +[#445](https://github.com/TNTcraftHIM/Piik/issues/445) reports washed-out Native +capture on an SDR Windows 10 system. Its proposed explanation is not established: +Microsoft documents full-range RGB as the +[video processor's default input range](https://learn.microsoft.com/en-us/windows/win32/api/d3d11/ns-d3d11-d3d11_video_processor_color_space). +The confirmed gap was that `FrameConverter` left its input/output range and +matrix implicit. The same converter handles BGRA capture and decoded NV12 +relay input; correcting only screen capture would leave the relay inconsistent. + +A local D3D11 check through the previous converter produced full-range NV12 from +BGRA color bars (black Y=0, white Y=255). The production `AdaptiveEncoder` VP8 +output then reproduced crushed shadows and clipped highlights in Chrome 154: + +| Source gray RGB | Previous conversion, WebRTC playback | Explicit limited-range BT.601 | +| --- | --- | --- | +| 0 | 0 | 0 | +| 32 | 19 | 33 | +| 128 | 130 | 128 | +| 235 | 255 | 235 | +| 255 | 255 | 255 | + +The fixture encoded 256x128 color bars with the production converter/encoder, +injected those keyframes into a local VP8 PeerConnection without the color-space +RTP extension, and sampled the playing video into a canvas. The experimental +conversion also restored red/green/blue bars to within one channel value of the +source. This establishes a bounded SDR defect, not the cause of every color +report. + +WebCodecs and WebRTC are not interchangeable acceptance paths: explicit BT.709 +looked correct through WebCodecs but shifted primary colors through this WebRTC +receiver. Chromium's +[remote-frame conversion](https://chromium.googlesource.com/chromium/src/+/master/third_party/blink/renderer/modules/peerconnection/media_stream_remote_video_source.cc) +can use BT.601 when color metadata is unspecified. White Y=235 is normal for +limited-range YUV; it must become display RGB=255, not be treated as a defect by +itself. + +The candidate now declares RGB capture as full range and reads decoded H.264 +color metadata from Media Foundation. Unspecified SDR and VP8 use the existing +WebRTC BT.601 convention. The converter produces limited-range BT.601 NV12; +the H.264 media types declare matching range, matrix, primaries and transfer. +Setting only matrix/range did not produce matching H.264 VUI on the tested +NVIDIA encoder. No wire field or user setting is added. + +Use the Windows 10 +[explicit video color-space API](https://learn.microsoft.com/en-us/windows/win32/api/d3d11_1/nf-d3d11_1-id3d11videocontext1-videoprocessorsetstreamcolorspace1). +The legacy nominal-range API squeezed decoded NV12 a second time in local tests. +Even with the explicit API and a positive format-conversion capability result, +NV12 BT.709-to-BT.601 conversion changed levels on NVIDIA when scaling and left +the matrix unchanged on AMD. Noncanonical NV12 therefore passes through RGB on +the GPU before the shared output conversion. Ordinary RGB capture and already +canonical NV12 keep one pass; no CPU readback, vendor branch or custom shader +is added to the product path. + +The opt-in `-CheckColor` probe in the [Windows capture guide](../../native/capture/windows/README.md) +checks full/limited-range BT.601/BT.709 input, scaling and input replacement +through the production converter. It passed on local RTX 4070 SUPER and AMD +Radeon adapters. RTX H.264/VP8 encoding followed by Chrome 154 WebRTC playback +also passed direct reception, decode/scale/re-encode and explicitly tagged +BT.709 H.264 input to both output codecs; sampled RGB errors were at most 5/255. +AMD hardware encoding was not accepted locally: MFT activation returned +`0x8007000e` before color configuration. These local checks do not settle the +reporter's Windows 10 environment, all hardware encoders, HDR content or missing +upstream color declarations. + +## HDR To SDR + +[#420](https://github.com/TNTcraftHIM/Piik/issues/420) reports overexposure with +both App and Browser capture on one Windows HDR machine. The report does not +locate the failing stage. + +The previous native WGC frame pool and thumbnails requested BGRA8, discarding +HDR range before the existing 8-bit H.264/VP8 encoder. Microsoft's +[capture guidance](https://learn.microsoft.com/en-us/windows/apps/develop/media-authoring-processing/screen-capture) +warns that HDR needs floating-point capture throughout the input pipeline to +avoid clipping. Reducing brightness after encoding cannot restore clipped +highlights. This is a confirmed implementation gap, not proof that every +reported Browser failure has the same cause. + +The candidate uses one `CaptureSdrConverter` before native output fanout and in +thumbnails. HDR displays use an scRGB FP16 pool, then Windows Direct2D's +[HDR tone-map effect](https://learn.microsoft.com/en-us/windows/win32/direct2d/hdr-tone-map-effect) +compresses luminance, the +[white-level effect](https://learn.microsoft.com/en-us/windows/win32/direct2d/white-level-adjustment-effect) +accounts for the source display's SDR white, and color management converts to +sRGB BGRA8. Ordinary SDR retains its texture-copy path. No custom shader, +CPU readback, new packaged dependency or per-output tone mapper is added. + +Source-monitor movement and DXGI factory invalidation refresh color metadata, +following [GetDesc1's contract](https://learn.microsoft.com/en-us/windows/win32/api/dxgi1_6/nf-dxgi1_6-idxgioutput6-getdesc1). +A format change recreates the WGC pool through its existing resize boundary; +it does not restart routes or encoders. WGC supplies no per-frame MaxCLL, so +conversion uses the display peak (at least the configured SDR white), or the +HDR10 10,000-nit range when usable peak metadata is absent. This stable platform +policy avoids a second image-content adaptation loop; real-display appearance +remains an acceptance requirement. + +The retained `-CheckColor` probe passed on RTX 4070 SUPER and AMD Radeon: +0–1000-nit neutral bars preserved distinguishable highlights, source white +80/240/80 nits and SDR/HDR size changes preserved ownership and ordinary SDR +colors. Production H.264/VP8 output decoded over Chrome 154 WebRTC reproduced the +converted SDR gray bars within 1/255. These are synthetic input checks, not +observations of an HDR game or the reporter's machine. + +Warmed GPU timestamp measurements for conversion, including the owned input +copy, had the following local medians. Dimensions are captured input sizes, +not newly supported sharing presets. + +| Adapter | 1920x1080 | 3840x2160 | +| --- | --- | --- | +| RTX 4070 SUPER | 0.045 ms | 0.240 ms | +| AMD Radeon | 2.39 ms | 11.17 ms | + +HDR conversion has a real GPU cost on weaker devices. These bounded measurements +exclude encoding and do not establish gaming/endurance performance. Current +monitors were SDR: physical HDR highlights, SDR content on HDR displays, +thumbnails, mixed displays and mode changes under load remain unaccepted. +Browser capture owns its own conversion; inspect the acquired frame before +assigning a downstream bug. + +Full HDR requires a separate end-to-end decision for bit depth, codec support, +color metadata, decoding and mixed Viewer capabilities. It is not a quality +dropdown addition. Preserve ordinary SDR routing and existing compatibility +with HDR-to-SDR; no full-HDR or surround implementation is claimed. diff --git a/docs/research/native-client-media.md b/docs/research/native-client-media.md index bb6a810..f5f9112 100644 --- a/docs/research/native-client-media.md +++ b/docs/research/native-client-media.md @@ -1,11 +1,47 @@ # Native App Media Evidence -- Reviewed: 2026-09-29 +- Reviewed: 2026-10-01 - Scope: platform capture, shared encoded sources, Pion transport and Browser decode; current behavior belongs to [media quality](../standards/media-quality.md) - Status: Windows physical native Host and Viewer gates passed; macOS and Linux adapters compile and package but still require physical media gates +## Windows Audio Exclusion + +A Windows 11 build 26200 check used two independent WinMM processes emitting +500 Hz and 1500 Hz tones, through the production capture helper and Go PCM +reader. Default-device loopback captured both at approximately 1799 amplitude; +process inclusion retained the first with under 0.3 amplitude of the other, +and exclusion retained the second with under 0.04 of the excluded tone. Target +exit retired capture in about 42 ms; attempting the stale identity failed. +No raw audio was saved. The extended capability probe completed in 300 ms. + +A parent/child follow-up used a silent selected parent with a 1900 Hz child, +plus an independent 2900 Hz process. Process-scoped reference amplitudes were +about 1799; exclusion retained the independent tone at 1800 and reduced the +selected child's tone to 9.3 (under 0.6%). Parent exit retired capture in 64 ms; +its stale identity was rejected. The reference uses each signal's own process +capture: default-device loopback also includes unrelated playback and endpoint +processing, so its amplitude is not a process-capture gain reference. + +The session fixture separately verifies failed exclusion replacement mutes +source audio while preserving video, microphone and encoded audio output; pause +and audio-profile changes retain the selected input. These checks establish a +local Windows implementation and one direct child, not coverage of every voice +application's process tree or simultaneous playback across several output devices. Those remain +device acceptance boundaries. [ADR-0008](../adr/0008-window-scoped-audio-capture.md) +owns the platform choice. + +The [Windows activation contract](https://learn.microsoft.com/en-us/windows/win32/api/audioclientactivationparams/ns-audioclientactivationparams-audioclient_process_loopback_params) +accepts one PID and its process tree, not a list of unrelated processes. +The [win-capture-audio implementation](https://github.com/bozbez/win-capture-audio/releases/tag/v2.2.0-beta) +supports broader exclusion by tracking audio sessions, capturing non-matching +processes and mixing them. Applying that design here would require session +discovery, process-tree deduplication and bounded multi-input mixing; Piik's +current source-plus-microphone mixer does not own that inventory. Multiple +exclusions therefore require a capture design change, not just a multi-select +control. Combining exclusion streams does not itself exclude their combined set. + ## Initial Physical Evidence (2026-09-05) These checks established the original native media paths. The Browser-mediated @@ -271,6 +307,33 @@ that result nor the allocation timings establish the reported system-wide lag's cause. Retain the existing implementation; reopen optimization from a matched bottleneck. +## Native H.264 Motion Quality + +Issue [#432](https://github.com/TNTcraftHIM/Piik/issues/432) reports more visible +blocking with native H.264 than VP8 on v1.6.7. Review of its attached Browser +report confirms 720p30 in both comparison windows, roughly 11 Mbps received, +and no reported packet loss; the 1080p60 H.264 window receives roughly 10 Mbps. +Those samples establish delivered format and transport observations, not equal +source content or objective picture quality. No encoded stream, reference +frames or QP measurements accompany that report. + +The reporter retracted the initial CBR bitrate-ceiling diagnosis after finding +that the experimental fixture changed bitrate during the measured interval. +That switching belongs to `PIIK_H264_FIXTURE` in `windows/main.cpp`, not live +capture. The corrected experiment reports CBR reaching its requested bitrate. +Production `h264_encoder.cpp` explicitly requests and validates Baseline; the +Browser's negotiated `profile-level-id` alone is not an inspection of emitted +SPS. The later experimental CABAC result cannot establish the production +Baseline bitstream's behavior. Microsoft's [encoder reference](https://learn.microsoft.com/en-us/windows/win32/medfound/h-264-video-encoder) +distinguishes profile, rate control and optional quality/speed settings. + +The original 720p comparison remains unresolved. An equal-content comparison +must preserve actual bitrate, frame rate, latency and production codec settings, +inspect the resulting bitstream and retain reference/decoded pictures. Raising +the bitrate ceiling, lowering the default frame rate or changing profile is not +justified by these logs alone; those changes have bandwidth, motion and receiver +compatibility consequences. + ## Implementation Boundary - `nativecapture` owns the child process, source identity, and bounded frame protocol. diff --git a/docs/research/room-interactions.md b/docs/research/room-interactions.md new file mode 100644 index 0000000..a114cd9 --- /dev/null +++ b/docs/research/room-interactions.md @@ -0,0 +1,260 @@ +# Room Interaction Design + +The owner accepted the text/reaction UI and model with bounded local acceptance. +[Status](../status.md#accepted-interaction-phase) owns release readiness and +[TODO](../todo.md#now) owns remaining work. Reviewed 2026-10-02. +This document records the implementation design and its tradeoffs; +[rooms and access](../standards/rooms-access.md#room-interactions) owns authority +and retention, while [presentation and lifecycle](../standards/presentation-lifecycle.md#product-surface) +owns room/publication lifetime. + +## Model And Owners + +| Concern | Owner | Lifetime | +| --- | --- | --- | +| Identity, room membership and access | Existing room store and authenticated signaling session | Current admitted participant | +| Screen/camera plus Host commentary | Existing capture and media owners | Current share and source | +| Text and reactions | `signal/interactions.go`, shared protocol and `RoomInteractionSession` | Current authenticated room session; bounded page-local presentation | +| Chat panel, optional chat overlay and reaction placement | Shared `RoomInteractions` and `RoomChatOverlay`; `RoomInteractionSession` owns bounded events | Current room UI | + +The existing room authority admits interactions through the same authenticated +WebSocket; there is no second admission or identity system. + +Messages do not need a media route, decoded frame, or SFU. They do not mutate the +media graph, add a port, create an extra connection, or restart capture. Pause, +source replacement and media recovery keep the current conversation. In this +candidate, a capable Host explicitly opts into a persistent room session. +Restoring an existing room connects that session without requesting capture or +starting media. The capture-ready owner requests publication; merely entering a +capture picker or awaiting permission cannot publish. Stopping the publication +retires only its media generation, while the authenticated Host and its +interaction session remain available. A later `start-sharing` claims a +fresh generation on that same signaling connection. Room close, access loss, +session replacement and page teardown still retire the whole session. A Host +without the capability keeps the existing stop-and-leave behavior. + +Room restoration and authority recovery share one awaited owner; connection reuse +checks the actual room/token binding. Publication acknowledgment precedes media +effects. A rejected start names its attempted generation, retires only that local +attempt and keeps room interactions available; it cannot stop another generation. +Opening the Host page alone still does not create an unused room. + +The server treats the current Host session's non-empty generation, matched to the +room's reconnect fence, as the only publishing truth. Room membership and Host +presence are separate, so a late Viewer in an idle room receives no media route +even when the Host remains available for room interactions. +Connected Viewers can continue interacting when the Host disconnects, while the +room server and their admitted sessions remain available. A departed participant +cannot receive a targeted reaction. Closing an App that also runs the room server +ends that service; merely leaving the Host page does not. + +## Delivery, Compatibility And Bounds + +- `/api/capabilities` advertises optional `roomInteractions` and + `hostRoomSession`; absence means the corresponding behavior is unavailable. + The page explicitly subscribes after authentication. Only subscribed current + sessions receive interaction events. The Host room-session behavior also + requires the `roomSession: true` authentication opt-in; older clients retain + the original stop-and-leave contract. Existing v23 media messages remain + valid, and publication start/result messages are sent only to an opted-in + session. +- The viewer discovers this optional capability independently of first-picture + startup. Late capability discovery enables data on the existing connection. +- A strict payload union admits plain chat text or a registered reaction. Chat is + at most 280 Unicode code points, normalized to NFC, without control/bidi + characters. It renders as text, with no markup, links, attachments or commands. +- The server accepts at most one interaction per 800 ms per connection. It + validates identity, subscription and target before fan-out. A slow recipient's + data is skipped at 16 KiB queued output, reserving the existing bounded queue + for signaling. The browser also declines sends under output backpressure. +- The sender waits for its server echo, not a guessed success or a read receipt. + A rejection or a 5-second confirmation timeout preserves the draft. There is + no offline outbox or automatic retry; manually retrying an unconfirmed send + may duplicate a message that arrived before the confirmation was lost. Only + the matching pending request may confirm a send or clear its draft/error; + late echoes may enter history without confirming a newer attempt. +- The page retains the latest 80 received chat messages and at most eight + simultaneous reaction effects. Effects expire after 2.4 seconds. Reconnection + clears pending effects and re-subscribes, without replaying messages or actions. + Same-room re-admission retains history, self attribution and the local draft, + even when its wire peer ID changes. Refresh, leaving, authority loss or room + session replacement clears the page conversation; late joiners receive only + new messages. Nothing persists in Browser storage. +- There is no server history, database table or message body in diagnostic + exports. Transport is the site's HTTPS/WSS connection; **chat is not end-to-end + encrypted**, and the server operator can access live message contents. + +## Interaction Design + +Chat uses the shared `FloatingPanel` shell for its nonmodal window. The shell +owns native visibility, placement, movement, resizing and focus return; the +conversation retains its session, draft, reading position and unread state. +It opens initially at the lower right, without +resizing the picture or moving the couch. Drag its title bar, use arrow keys on +the focused bar, or explicitly choose a corner under Position. Its summary also +allows dragging; a drag never activates the menu. Resize from either diagonal +corner, use arrow keys on a resize handle, or choose a preset under Position. +Resizing holds the opposite corner steady. Size and position belong to this room +UI only; viewport and keyboard changes bound them without erasing preferences. +The native manual +popover keeps the conversation visible while operating the picture or reactions. +Close, the entry toggle, or Escape hides it; this preserves the room conversation. Inner +menus consume Escape first, then the most recently opened utility window, then +theater mode. Input-method +composition is not a dismissal request. Opening focuses the panel without opening +the keyboard; explicit close returns focus to the entry. Nested disclosures +cannot change panel visibility. +The visual viewport bounds it above an onscreen keyboard. +Hidden messages are not marked read. A bounded scrolling log keeps arrivals from +growing the panel. Consecutive messages from the same sender share visual identity; +each message retains its accessible author. Hiding and reopening preserves the +reader's position; a latest-message action resumes following. Sending waits for +confirmation without locking draft editing or introducing a message queue. +The conversation entry sits below the couch, separate from +the Host's sharing controls; theater retains its compact conversation entry. +Selecting a person opens the compact reaction palette in the shared floating +shell, initially beside the selected person (below when space permits). Each +opening anchors there before allowing free movement. Its portrait and name identify the +target even after moving the window. Selecting a different participant keeps its +placement and any manually chosen size; selecting the same person toggles it. +The default palette height follows its content, bounded by the visible viewport. +One's own palette expresses a reaction above oneself and holds the local effects switch; other +people's palettes offer targeted expressions and two labelled throwing props. +Connection details remain secondary when that participant has inspectable data. +Close or Escape dismisses the palette; it also closes if its participant leaves. +Sending keeps it open for repeated interaction, with the existing short cooldown. +Confirmation never changes menu visibility; failure stays visible in the card. +Opening or closing a reaction card preserves the chat draft and unread state. +Self expressions follow the sender's head, including idle and interaction motion. +Only that sender's newest self expression replaces their earlier one. A targeted +emoji is a delivery: it travels from sender to recipient, retains a compact +source label above the centered glyph, and settles separately above the +recipient's own expression. Late mounts and reduced motion retain that attribution without +replaying elapsed motion. A gift never replaces the recipient's self expression; +all delivered events remain in the accessible log. +Tomato/poop throws require choosing a current participant, land with a small +character response and settle above the head, keeping the face visible. A tomato +squashes with a few droplets; poop wobbles with a small puff. Other expressions +use short gestures and sparse accents. All decorative descendants share the +event's remaining phase and expiry; event ID and sender identity seed their +variation. No particles, stains or animation timers outlive the event. +A departed participant is never silently replaced as target. +Everyone sees the same server-issued event; placement follows their own couch +layout. Effects have a couch-only coordinate owner, independent of panel layout, +and never become a video overlay or a media status. Impact transforms leave the +UUID-owned idle animation, names and status indicators alone. Hiding effects or +retiring the event cancels its own animation; showing a still-live event uses its +remaining lifetime. Users can hide effects; reduced motion uses a static stamp. +Theater mode hides the couch and retains a chat entry, using the same floating +panel and draft without changing the picture's size. HTML player fullscreen +keeps the existing danmaku control; it does not include the conversation panel. +Unread messages advance only while the log is onscreen, the document is visible +and the reader is following its latest messages. A hidden or scrolled-back log +does not consume arrivals. These are local unread indicators, not read receipts. + +The optional chat overlay (danmaku) projects newly received chat over the picture; +it adds no message kind, transport or delivery promise. It defaults off, with the +same local switch beside the chat composer and Viewer playback bar, including HTML +fullscreen. At most three single-line comments appear for up to six seconds each. +Font size follows the picture width, with a fixed readable contrast and duration +rather than separate typography, opacity and speed preferences. +Each event's sender and ID choose its lane, and a newer event replaces that lane's +previous flight. All received text remains in the full chat log, without a playback queue. Text is clipped +to the picture space above playback controls; smaller spaces show fewer complete +lanes without restarting their timelines. It does not intercept playback gestures. Showing the overlay +does not mark the chat log as read. Hidden documents, unavailable pictures, +source changes and disconnection clear flights; showing or reconnecting never +replays earlier comments. Reduced motion uses temporary stationary text. Native +video-only fullscreen and picture in picture cannot include HTML overlays. + +Server confirmation means acceptance, not an acknowledgment from every recipient. +Delivered chat follows server order; congestion can leave gaps without reordering +the received subset. The same event retains its lane and time basis even when +preceding events are missed. Visibility, local switches and missing messages may +change which effects are shown; geometry and font size follow the local viewport. + +Reaction animations are decorative. The same bounded events also provide +localized sender/action/target descriptions in a polite accessible log, including +when visual effects are hidden. Reduced motion removes decorative movement from +both the flight and the picker controls. + +Chinese, English and visual modes share the component and behavior. Visual mode +retains user-entered text and participant names, with icon controls and localized +accessible names. Host identity reuses the existing crown; neither chat colour nor +reaction motion claims media connectivity, capture permission or delivery status. + +## Presentation Synchronization + +The server stamps acceptance time once and broadcasts the same event in its +existing serialized order. A subscription response provides server time; the +room session estimates its offset from the local round-trip midpoint and keeps +that estimate until reconnection. Effect expiry and both renderers share that +session's clock. Late effects start at their remaining phase; expired chat still +enters the log and confirms a matching send, without replaying an effect. +Returning from reduced motion also respects the original expiry. + +This is approximate alignment, limited by subscription round-trip asymmetry, +queueing and clock drift; it does not promise frame-exact synchronization. There +is no continuous clock exchange, sorting by timestamp, or animation-frame stream. +Changing a device's clock mid-connection can disturb effect timing until the +next subscription; message order and delivery do not depend on that clock. +Participant motion and event lanes reuse the same identity hash, with an event's +ID and actor identity determining its lane independently of prior arrivals. +A shared mutable random-number stream would drift whenever a client misses or +hides an event. Responsive layout remains local. +This follows the same-input principle described in +[Deterministic Lockstep](https://gafferongames.com/post/deterministic_lockstep/), +without introducing lockstep waiting or frame-by-frame simulation. + +## History Direction (Not Yet Implemented) + +History ownership remains undecided. A Host cache can also miss live messages and +is unavailable while the Host is absent; treating its copy as server-confirmed +history would change message authority. Compare that cost with a bounded room-server +memory buffer before adding replay. Either design needs explicit access and +retention boundaries. Recovered text would enter the log, never replay old +danmaku or reaction effects. Current behavior remains the no-history contract above. + +## Audio Boundary + +The Host microphone remains commentary mixed into the shared source. +Bidirectional room voice is not part of the product plan. +Text, reactions and danmaku introduce no microphone capture, voice +admission or additional media peers. + +## Primary References And Rationale + +- [LiveKit media and data](https://docs.livekit.io/frontends/build/media-data/) + separates publications from finite data. Piik already has an authenticated + signaling channel; adding the LiveKit room service solely for chat is unnecessary. +- [LiveKit text streams](https://docs.livekit.io/transport/data/text-streams/) + distinguishes current-participant delivery from persistent history. The + candidate likewise promises transient room messages, with explicit local bounds. +- [WebSocket buffering](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/bufferedAmount) + and [WebSocket API limitations](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) + motivate bounded queues and explicit send failure rather than an accumulating outbox. +- [WebRTC data channels](https://www.w3.org/TR/webrtc/#rtcdatachannel) + depend on peer connections. Using them here would couple chat readiness and + fan-out to a screen-sharing route; the existing room WebSocket avoids that. +- [React state identity](https://react.dev/learn/preserving-and-resetting-state) + supports resetting the panel with its room session rather than a reconnecting + transport identity. [ARIA log semantics](https://www.w3.org/TR/wai-aria-1.2/#log) + provide sequential, non-interrupting announcements alongside visual effects. +- [Meet reactions](https://support.google.com/meet/answer/13151720?hl=en) + attach expressions to participants and let viewers reduce visual distraction. + Piik likewise keeps reactions outside the shared picture, with explicit local + display preference and a separate accessible event description. +- [Microsoft participant cards](https://support.microsoft.com/en-us/outlook/profile-cards-in-microsoft-365) + place person-specific actions on the selected person. Piik uses that direct + entry on its existing couch rather than requiring a second participant list. +- [Native popovers](https://developer.mozilla.org/en-US/docs/Web/API/Popover_API/Using) + provide top-layer placement, light dismissal and focus return without adding + a second overlay framework. Piik owns the bounded card's anchor and contents. +- [Animation time](https://developer.mozilla.org/en-US/docs/Web/API/Animation/currentTime) + lets a newly mounted or resumed effect seek to its remaining phase. The + [clock comparison](https://developer.mozilla.org/en-US/docs/Web/API/Performance/now#ticking_during_sleep) + explains why replacing wall time with a monotonic clock alone would not solve + cross-platform sleep and resume. +- [niconico playback controls](https://help.nicochannel.jp/hc/ja/articles/4414625162777-%E5%8B%95%E7%94%BB%E3%81%AE%E8%A6%96%E8%81%B4%E6%96%B9%E6%B3%95) + keep comment display optional alongside playback. Piik likewise separates + each viewer's overlay preference from message delivery and the conversation. diff --git a/docs/research/room-voice.md b/docs/research/room-voice.md deleted file mode 100644 index bfb5c4b..0000000 --- a/docs/research/room-voice.md +++ /dev/null @@ -1,287 +0,0 @@ -# Room Voice: Design Assessment - -Reviewed 2026-09-24 against public `v1.6.3` (`f423df0a`). This is a proposal, -not an accepted product contract or implementation authorization. The separate -room-interaction experiment supplies candidate lifecycle behavior; it has not -been integrated. [TODO](../todo.md#now) owns the outstanding work and decisions. - -## Recommendation - -Keep existing room authority and the Host-rooted screen route. Model room voice -as participant-owned microphone publications and independent listening. Chat, -screen playback and voice share admission, not a media lifetime. Do not turn the -screen tree into a conference graph or introduce another room/session store. - -There is no equally small solution that simultaneously promises all 21 members -can talk, preserves every P2P-only deployment, and supplies conference-grade -connectivity. Choose the first voice scope before implementing its transport. -For the next technical experiment, assess a small direct group and audio -recapture isolation first. Four total voice participants is a useful test size, -**not an accepted product limit or a demonstrated performance guarantee**. -If full-room voice is required initially, assess a mature conference SFU before -building a mesh intended to grow into one. - -## Current Owners And Actual Gaps - -| Current owner | Verified behavior | Consequence for voice | -| --- | --- | --- | -| Room store and authenticated signaling | Room admission, participant identity, grant revocation and session replacement | Extend this authority; no voice login or second membership store | -| `HostAudio` in `src/client/media/host-audio.ts` | Source and microphone mixed into one share-scoped track | This is commentary; it cannot supply independent conversation | -| `internal/app/nativehost/audio_mix.go` | Native source/mic PCM share one mixer and output clock | Keep source audio here; avoid duplicating the Host mic across outputs | -| `src/client/media/viewer-audio.ts` | Audio follows the picture's play/pause/mute state | Room speech needs its own playback owner | -| Screen controller and media edges | One Host source, one upstream per Viewer, bounded screen-copy capacity | Voice must not borrow screen generations, first-video-frame proof or copy slots | -| Embedded SFU admission/media | One committed room publication, video-required metadata, share-scoped resource identity | Multiple audio-only publishers need a new capability, not another flag | - -The SFU constraints are explicit in `validPublicationMedia` in -`internal/server/sfu/media.go`, `ResourceFence` and `CommitPublication` in -`internal/server/sfu/admission.go`, and the Host/Viewer dispatch in -`internal/server/signal/router_sfu_signal.go`. -[ADR-0013](../adr/0013-embedded-node-local-media.md) embeds Pion/LiveKit components; -it does not embed LiveKit's complete room service. - -The interaction candidate separates a retained Host room session from screen -publication. That is a useful foundation, not permission to allocate rooms on -page visit. Preserve the accepted entry: first share creates the room and an -existing room can resume. Retaining opted-in interaction after stopping media is -the candidate lifecycle, not published behavior. A separate voice-only -room-creation flow would be another product decision. - -## Transport Choice - -| Approach | Benefit | Cost and boundary | -| --- | --- | --- | -| Small direct voice mesh | Keeps media P2P and does not require a reachable central media server | Each sender sends to every listener; needs a total voice-member bound, separate resource accounting and real NAT/device tests | -| Mature multi-publisher SFU | Each speaker uploads once; suitable direction for full-room voice | New publication/subscription integration and central-media availability/privacy decision | -| Host forwarding or mixing hub | Fewer peer relationships | Reintroduces Host dependency, forwarding/mixing and echo ownership; unsuitable for Host-independent conversation | -| Custom voice relay tree or automatic mesh/SFU switching | Potentially combines deployment options | Adds another routing/recovery problem; outside the first scope | - -For a full bidirectional mesh, `N` participants need `N*(N-1)/2` pair connections -and up to `N-1` microphone copies per sender: 21 people means 210 pairs and 20 -copies; four means six pairs and three copies. These are topology counts, not -bandwidth or CPU measurements. Limiting speakers alone does not bound sender -upload: one speaker and 20 listeners still need 20 copies. The screen's existing -`C=1..3` limit must retain its meaning; any independent voice budget needs explicit -acceptance and combined-load measurement. - -Two SFU implementations deserve a bounded comparison: a complete mature room -engine, or a narrow participant-audio adapter around existing media components. -The former adds deployment and potentially competing session ownership; the -latter leaves Piik responsible for admission, subscriptions, recovery and cleanup. -Do not describe the adapter as mature conference behavior merely because its -forwarder uses mature libraries. LiveKit's -[room/participant/track model](https://docs.livekit.io/intro/basics/rooms-participants-tracks/) -and [SFU architecture](https://docs.livekit.io/reference/internals/livekit-sfu/) -show the layers involved. - -Select one topology for the first capability. Jitsi's -[two-party P2P design](https://jitsi.org/blog/p2p4121/) retains its bridge connection -and changes transport when group size changes: useful evidence that a hybrid -mode has real lifecycle cost. It is not a reason to copy that complexity here. - -Deployment is part of this decision. Demo and App Local/public-link modes are -P2P-only today. Public-link tunneling carries HTTP/WebSocket control traffic, -not a public SFU media endpoint. Existing STUN is not TURN, and existing embedded -UDP media does not provide media TCP/TLS. A full conference engine's -[connectivity options](https://docs.livekit.io/transport/self-hosting/ports-firewall/) -are not inherited by importing its library. This proposal enables no new service -or port and does not change Demo configuration. - -## Proposed Responsibility Boundaries - -```text -Existing room authority and authenticated signaling - +-- screen publication -> existing P2P/SFU screen route -> picture/source sound - +-- interaction events -> bounded chat/reactions - +-- voice admission -> participant microphone -> selected voice transport - -> independent voice playback -``` - -These are responsibility boundaries, not a request for a generic media framework -or one new service per line. Keep the first implementation cohesive: - -| Responsibility | Owns | Does not own | -| --- | --- | --- | -| Room authority | Who belongs, who may publish/listen, bounds and retirement | OS microphone permission or local volume | -| Participant voice session | Explicit join intent, current admitted publication/subscriptions and their transport lifetime | Screen routes, share generation or a second reconnect policy | -| Microphone input | One acquisition, device choice, local mute, replacement and disposal | Remote permission to turn capture on | -| Voice playback | Received tracks, local volume/person mute, output capability and autoplay recovery | The video's pause/mute state or remote speaking permission | - -Do not attach room speech to a screen peer merely to reduce connection count: -screen parent replacement and share retirement have different lifetimes. The -selected voice transport owns its connections and counts them explicitly. - -Prefer a Browser-owned room microphone for both roles, including a Host using -App screen capture. This reuses platform speech processing and avoids a second -native voice service. Reuse existing device-selection semantics, not native -device identifiers. Browser microphone acquisition requires a secure context; -plain HTTP LAN pages and unsupported browsers must have truthful capability -presentation. App detection alone does not establish microphone availability. -See the [capture specification](https://www.w3.org/TR/mediacapture-streams/). -Browser voice also does not inherit the native Pion path's port mapping, NAT -assistance or native media fallback just because an App is attached. Measure -voice reachability independently; a working screen path does not prove it. - -Room voice and source sound need different audio treatment. Voice uses the -platform's speech/AEC facilities; music/game sound keeps its fidelity settings. -Do not force both through the existing source-audio Opus profile or write a new -voice codec, AEC, PCM subtraction routine or virtual-audio driver. - -## Consent, Lifecycle And Presentation - -Proposed first permission model: admitted members may explicitly join voice and -then unmute themselves; existing room removal remains the authority boundary. -Start with a closed microphone. Joining may unlock playback, but only explicit -unmute requests input permission. If Host-only speaking approval is required, -specify it before adding a moderation UI. No remote action may start a microphone; -LiveKit also disables [remote unmute by default](https://docs.livekit.io/intro/basics/rooms-participants-tracks/participants/). - -| Event | Proposed behavior | -| --- | --- | -| Share start/stop/pause, source or quality change | Voice stays independent | -| Local mic mute | Silence that publication without leaving voice; retain explicit device intent | -| Local person mute or voice volume | Change local listening only; picture volume remains separate | -| Leave voice | Release mic and voice media, remain in room/chat | -| Host leaves page while room service remains | Retire Host voice; other admitted participants may continue | -| App exits in Local/public-link mode | Room service disappears; independent voice cannot outlive its authority | -| Site-mode App media fails | Browser voice may continue while the Site room authority remains | -| Revocation, replacement or room close | Retire affected resources and reject stale completions/signaling | -| Mic permission/device failure | Preserve picture and other voices; show the actual input failure | -| Hidden page or phone interruption | Visibility alone is not leave/mute; report actual suspension/device loss | - -Transient signaling loss is not automatically a media failure or a grant -revocation. Current `presence.go` and `route.Controller.DisconnectSession` -distinguish disconnected sessions from confirmed departure; Viewers have bounded -membership grace. Voice must consume that authority owner's transitions, with no -parallel grace timer or indefinite stale-session publication. Before implementation, -define and test the Host case, reservation retention, reconnect re-admission and -revocation during an outage. Do not extrapolate Viewer grace to a Host or reuse -an old session ID as current authority. This is a design gate, not a claim that -screen recovery already implements voice recovery. - -One input owner prevents duplicate captures. Retire late permission results after -leave; prepare a replacement device before releasing a healthy old input; keep a -failed explicit selection visible rather than silently choosing another mic. -Repeated mute/join/leave must converge without a second saved effective-state flag. -Ordinary publication mute need not renegotiate every time; mature systems -[distinguish mute, unpublish and unsubscribe](https://docs.livekit.io/intro/basics/rooms-participants-tracks/tracks/). - -Use the existing participants/sofa and room actions, not a conference grid. -First controls: join/leave voice, mic toggle and input choice/check, independent -voice volume, and local person mute in the participant menu. Keep source-sound -settings with the picture. Default to system output; show output selection only -where supported. Do not add deafen, global push-to-talk or a permission-role matrix -until their need and restoration behavior are established. - -Display local intent, authorized membership, transport readiness and actual -playback separately. A speaking pulse follows observed audio activity, not a -connected flag or UUID animation seed. Catch blocked playback, including tracks -arriving after joining, and offer an explicit enable-audio action; compare -[LiveKit's autoplay handling](https://docs.livekit.io/reference/components/react/component/startaudio/). -Mobile lock-screen calling remains unproven; platform suspension is not fixed by -extra timers. See [browser lifecycle guidance](https://developer.chrome.com/docs/web-platform/page-lifecycle-api). - -## Audio Isolation Is A Prerequisite - -Whole-system loopback records received room speech played on the captured output -endpoint. That speech can then be sent back through screen audio. Headphones -remove an acoustic path, not this digital recapture; microphone AEC does not clean -the separate source-audio track. This matters even if only the Host shares a screen. - -Windows exposes narrower process-tree capture and process exclusion. Current -Piik includes the selected process tree for window sound or captures the output -endpoint for system sound; it does not implement process exclusion. Excluding -the browser that plays Piik voice could also exclude intended browser content. -Test actual process ownership and output routing rather than assuming a window -selection isolates sound. See Microsoft's -[loopback description](https://learn.microsoft.com/en-us/windows/win32/coreaudio/loopback-recording) -and [process-capture sample](https://learn.microsoft.com/en-us/samples/microsoft/windows-classic-samples/applicationloopbackaudio-sample/). - -Browser `restrictOwnAudio` is relevant; `suppressLocalAudioPlayback` has a different -purpose. Chrome documents the former as excluding audio originating from the -capturing document when system audio is captured. Check support, applied settings -and decoded output on supported platforms; a hint is not cross-browser proof. -See the [Screen Capture specification](https://www.w3.org/TR/screen-capture/) -and [Chrome's implementation announcement](https://developer.chrome.com/blog/chrome-141-beta). - -Verify source sound and incoming voices using distinguishable signals. Cover -native window/system capture, browser tab/window/system capture, source changes, -same-browser content and output-device changes. Where isolation cannot be -established, a supported combination or explicit feature limitation must be -chosen before release. Never silently discard source sound or promise that a -headphone hint solves it. - -## Compatibility, Trust And Bounds - -The Host microphone must have one distribution path. Sending it in both existing -commentary and independent voice causes duplicates and defeats individual mute. -Moving it exclusively to voice can make older/non-participating Viewers lose -commentary. Choose an explicit room capability/participation boundary or a -coordinated major-version transition; do not ship a silent behavior change. -The [versioning owner](../standards/versioning.md) governs that choice. This -assessment does not select a release number or authorize removal of native mic. -Once that boundary is selected, define the handoff before coding: retire the old -microphone distribution before enabling the new one, and test failure/cancellation -without silently losing commentary for Viewers who cannot receive room voice. - -Browser-first capture can avoid new native voice commands, but does not make -room signaling automatically compatible. Trace strict readers in both directions; -advertise support before new messages. Keep stable peer identity, authenticated -session, grant and media connection identity distinct. Reuse an existing identity -only when its lifetime matches; do not add a room-wide epoch for convenience. - -Bind signaling to current authenticated participants and exact publication/peer -resources; bound memberships, publications, subscriptions, candidates and message -rates before allocation. Leave/revoke must release actual resources before their -slots become reusable. No audio through chat events or durable voice recordings. - -P2P revocation has an enforcement limit: the server can refuse signaling and -conforming endpoints can stop capture/drop tracks, but it cannot intercept media -between two modified peers. An SFU can enforce its own forwarding removal. Do not -promise identical server-enforced muting across these models, or hide the central -server's media access under the existing P2P privacy label. - -## Implementation Sequence And Acceptance - -1. Reconcile the isolated interaction candidate with current main and accept its - room/picture separation on its own. Preserve the existing staged work; do not - merge an old branch wholesale. Voice need not block chat/UI acceptance. -2. Select total voice membership, deployment coverage, speaking policy and the - commentary/compatibility boundary. Record accepted changes in the existing - product owners and an ADR only when architectural decisions are accepted. -3. Run an isolated transport/audio-isolation experiment before shipping UI. - Measure screen plus voice CPU, upload, number of connections, audible latency, - NAT failure, echo and cleanup. Use one topology; stop if the required scope - implies a custom conference engine or unsupported audio isolation. -4. Implement one vertical slice through the owners above: explicit join, muted - input, unmute, remote audio, local mute and leave. Then expand participants and - exact authority teardown. Reuse existing controls, status/comic meanings and - diagnostic export; avoid separate feature-local rules. -5. Accept faults as well as the happy path: permission resolves after leave, - device switch fails, source changes, autoplay is blocked, signaling drops - during healthy audio, reconnect races revoke/replacement, Host leaves, service - exits, mobile suspends, and repeated join/leave leaks no resources. Test quiet - and muted participants without video-frame admission requirements. Include - supported old/new endpoints and Chinese/English/visual presentation. - -Transport experiments must report measured limits rather than treating a -successful localhost call as public-network or mobile acceptance. This planning -round ran no voice, acoustic-echo or device-performance experiment. - -## Independent Fidelity Work - -Stereo acceptance can proceed separately: distinguishable left/right signals -through Browser, Native, relay and SFU, with microphone mixing and source changes. -The released native answer fix negotiates receive-side stereo; SDP alone does not -prove decoded channel separation. Opus negotiation follows -[RFC 7587 section 7](https://www.rfc-editor.org/rfc/rfc7587.html#section-7). -Do not add surround channels as a prerequisite for voice. - -For [HDR issue #420](https://github.com/TNTcraftHIM/Piik/issues/420), investigate -correct SDR output first: preserve HDR range during native capture, then use -platform tone/gamut mapping before the existing SDR encoder. Windows documents -[floating-point capture](https://learn.microsoft.com/en-us/windows/apps/develop/media-authoring-processing/screen-capture) -and a [tone-map effect](https://learn.microsoft.com/en-us/windows/win32/direct2d/hdr-tone-map-effect). -Validate mixed SDR/HDR content, highlights, SDR reference output and display-mode -changes. Already-clipped browser input cannot be restored downstream. Full HDR -transport/metadata, surround sound, recording and transcription remain outside -this proposal. diff --git a/docs/standards/configuration.md b/docs/standards/configuration.md index fac3d95..4288653 100644 --- a/docs/standards/configuration.md +++ b/docs/standards/configuration.md @@ -56,6 +56,7 @@ Automatic SFU fallback runs inside the Hosted process when `SFU_UDP_PORT` is set | `SFU_UDP_PORT` | Optional UDP media port `1..65535`; unset or blank disables SFU and fixes Privacy mode on for every room. Set `7882` for the standard public listener and allow the Host to choose Privacy mode. | | `SFU_LISTEN_HOST` | IPv4 bind address, default `0.0.0.0`; independent of HTTP `LISTEN_HOST`. Read only when SFU is enabled. | | `SFU_PUBLIC_IP` | Optional explicit IPv4 advertised-address override for a host behind NAT. Read only when SFU is enabled. | +| `SFU_ONLY` | `false` by default. `true` requires `SFU_UDP_PORT` and sends every room through SFU, without Peer candidates or background P2P convergence. Applies after Server restart. | SFU control uses the application's authenticated signaling connection. No separate control origin or infrastructure credentials are configured. Local and @@ -67,6 +68,13 @@ configured STUN and peer relays remain; no media-server fallback is possible. The server enforces this even if a Host requests hybrid mode. There is no separate `SFU_ENABLED` flag to conflict with the listener configuration. +With `SFU_ONLY=true`, the page shows a locked **Server media** policy in place of +P2P settings. An older page requesting Privacy mode is refused before sharing; +refresh it or turn Privacy mode off. Capacity exhaustion and connection failure +use the existing bounded SFU admission/recovery path, never a P2P escape. +This policy needs sufficient server bandwidth and a working UDP route; it is not +a TURN/TCP fallback. App Local/public-link and the public Demo remain P2P-only. + The SQLite parent directory must exist and be writable. The systemd template sets `/var/lib/piik/rooms.sqlite` under its managed state directory; the container image sets `/home/nonroot/rooms.sqlite` under its writable data diff --git a/docs/standards/engineering.md b/docs/standards/engineering.md index 109232d..dcbf258 100644 --- a/docs/standards/engineering.md +++ b/docs/standards/engineering.md @@ -148,7 +148,10 @@ above rather than searching the tree. Closing a Browser connection may leave its API promises pending. Its owner cancels the corresponding waits so a serialized successor never depends on completion from a retired connection; late results still need identity checks. - Later cancellation must not erase a failure already observed before cleanup. + Failure effects need the same final identity check as successful results, + serialized with replacement; an earlier input check alone is insufficient. + Later cancellation must not erase a failure already accepted by its current + owner. Perform blocking retirement after releasing that owner's state lock. A caller deadline does not cancel a dependency that ignores it. Keep late work bounded and serialize its cleanup; verify the dependency's actual lifetime. Construction is part of acquisition: if it throws, release resources created diff --git a/docs/standards/media-quality.md b/docs/standards/media-quality.md index 99db584..cace60c 100644 --- a/docs/standards/media-quality.md +++ b/docs/standards/media-quality.md @@ -39,6 +39,17 @@ production behavior and remaining acceptance. frame matches its desktop source dimensions. Other frames retain their own aspect ratio. This changes only the encoded presentation, never the game or display settings; vendor-private scaling is not inferred. +- Windows SDR conversion owns one output color contract: limited-range BT.601 + NV12 with matching H.264 metadata. RGB capture and decoded video retain their + input range/matrix through conversion and relay scaling; dimensions do not + determine color space. Unspecified SDR follows the WebRTC convention. +- Windows native HDR sources retain scRGB FP16 until platform tone mapping, + source-display white-level adjustment and sRGB conversion produce SDR. Live + capture and source thumbnails share this owner; conversion precedes output + fanout and does not add a media route or per-encoder tone curve. Ordinary SDR + keeps its copy path. Source-display color changes refresh capture input, + not room authority or transport. [Fidelity evidence](../research/media-fidelity.md) + owns platform checks; this is HDR-to-SDR, not end-to-end HDR delivery. - Where Windows supports border control, the native source picker offers **Show capture border**, off by default. The choice stays in the Host page and follows native source and quality changes. Source previews request borderless @@ -177,12 +188,32 @@ closing a Browser PeerConnection alone does not guarantee that its promises settle. Cancellation starts before joining work that depends on it, while a replacement still waits for the retired owner's cleanup. +Browser and Native encoder-group admission uses current native demand, not the +previously applied rate. A new or pending join must not displace an existing +rate owner by raising or retaining its budget against that owner's demand. +Revalidate pending admission before planning other memberships. + Eligible Browser P2P parents use the source-owned pool in [ADR-0014](../adr/0014-browser-node-local-encoding-pool.md). Compatible direct children share one independent local WebRTC producer; incompatible demands -remain separate. Each outgoing connection keeps native transport, allocation -and recovery, with a tiny carrier supplying its RTP clock. Producers use -the existing native video target under Host ceilings. Actual forwarded-frame +remain separate. Joining another producer must fit the child's native allocation; +a quiet scene's low byte rate does not prove that producer's rate budget fits. +Pending separation remains necessary while another compatible member needs a +higher native budget; quiet output alone cannot cancel it. When that demand +leaves or the child's budget recovers, the current producer can adapt in place. +Optional reuse also preserves the healthy current output's rate budget and +observed quality within Host ceilings through recovery-frame commitment, using +the actual candidate frame and current output rather than preparation alone. +Rejected membership retains current delivery and its recovery requests. +A recovery frame already being written retains its owner until completion, +including across pause/resume; cancellation belongs to the output that knows +whether writing has started. +Each outgoing connection keeps native transport, allocation and recovery, with +a tiny carrier supplying its RTP clock. Producers use the existing native video +target under Host ceilings. A new producer seeds its +local WebRTC start rate from that allocation, without a minimum or a change to +the outgoing connection's estimate. Subsequent budgets use the existing sender +parameters; initialization does not add a recovery controller. Actual forwarded-frame and producer observations remain distinct from the tiny carrier's statistics. Producer startup protection begins at actual outgoing publication, excluding local warmup and paused frames. The synthetic carrier uses screen-content @@ -294,6 +325,17 @@ A Native audio-process failure does not end healthy video. A later explicit source replacement resumes audio through the existing track and encoder owner; it does not trigger an automatic capture retry or change room audio topology. +Optional Windows screen-audio exclusion binds one selected process tree to its +PID and creation time under [ADR-0008](../adr/0008-window-scoped-audio-capture.md). +The selection lasts for the share, survives source-audio off/on and refresh, +and is never saved across App runs or rebound by executable name. Changing to a +window source or explicitly selecting no exclusion clears it. A requested +exclusion retires previous source audio before replacement preparation; failure +or target exit leaves source audio silent while healthy video and the Host +microphone continue. Only an explicit source selection resumes it. A missing +capability must reject the request, never discard the exclusion. Old pages keep +their existing audio behavior. + ## Observable Truth Requested resolution, frame rate, bitrate, content intent, codec, and audio diff --git a/docs/standards/naming.md b/docs/standards/naming.md index 8e31015..0a7f6d3 100644 --- a/docs/standards/naming.md +++ b/docs/standards/naming.md @@ -40,6 +40,7 @@ Set the register by the reader's task: | Surface | Register | | --- | --- | | Website | Light, everyday language in introductions and illustration captions; keep the current restraint | +| App and Web controls | Direct feature and action names, including panel titles and settings; “聊天” / “Chat” rather than an invitation or joke | | README | A friendly introduction, followed by direct explanations of features, limits and first steps | | Documentation index | Clear topic names and short descriptions that help readers choose a guide | | Deployment, configuration and maintenance guides | Formal, precise instructions: prerequisites, actions, expected results and recovery steps | @@ -102,6 +103,8 @@ in the welcome or illustration caption. Do not soften a failure into a joke, invent connection progress or promise guaranteed connectivity or absolute privacy. Screen sharing includes games, creative work and showing a useful discovery. +Playfulness belongs in optional welcome/waiting lines, empty-state captions and +illustration reactions; it does not rename the control that performs the action. Operational pages omit introductory sentences that repeat their controls. Helper text should explain a meaningful choice or prevent an error. diff --git a/docs/standards/presentation-lifecycle.md b/docs/standards/presentation-lifecycle.md index 4fe86bc..5191363 100644 --- a/docs/standards/presentation-lifecycle.md +++ b/docs/standards/presentation-lifecycle.md @@ -24,6 +24,15 @@ room identity, current state, roster, and one manual media-reconnect action. Controls and text must remain usable without overlap or horizontal overflow on desktop and mobile viewports. +A capable, opted-in authenticated room session +outlives its current publication. Stopping capture retires media, not chat; +restoring an existing room joins it without starting capture. Entering the page +alone does not create a room. Room close, lost authority and page teardown still +retire the session. Connected Viewers may keep interacting while the Host page +is absent, provided the room service and their admitted sessions remain available. +[Rooms and access](./rooms-access.md#room-interactions) owns interaction authority +and retention; [status](../status.md#accepted-interaction-phase) owns release readiness. + Piik App opens this same application in the system Browser. Its small startup surface selects Local, temporary public invitation, or a configured Site before entering the Host workspace. Viewer links opened at that activated diff --git a/docs/standards/rooms-access.md b/docs/standards/rooms-access.md index c908780..26fb6b6 100644 --- a/docs/standards/rooms-access.md +++ b/docs/standards/rooms-access.md @@ -1,6 +1,6 @@ # Rooms And Access -This file owns the current room, admission, invitation, and persistence +This file owns the current room, admission, invitation, interaction, and persistence contract. [ADR-0002](../adr/0002-memory-resident-protected-rooms.md) explains the storage decision; implementation detail belongs in code and tests. @@ -91,6 +91,26 @@ recreate a revoked subscription; no separate media token survives revocation. Changing code-entry policy or password affects later code-only attempts and does not silently revoke invitations. +## Room Interactions + +Text and reactions reuse the authenticated room signaling connection. The server +derives sender identity from that session and accepts targeted reactions only +for participants still connected to the same room. Interaction subscriptions +inherit existing membership, replacement and revocation checks; they grant no +additional authority and do not modify media routes. + +Delivery is transient and bounded. The server keeps no message history; each +page keeps only a bounded set of received messages and effects, without writing +them to storage or diagnostics. Re-admission to the same room may retain that +page's conversation, but refresh, leaving or authority loss clears it. There is +no history replay or offline outbox. HTTPS/WSS protects transport; chat is not +end-to-end encrypted, and the site operator can access live message contents. + +[Presentation and lifecycle](./presentation-lifecycle.md#product-surface) owns +the distinction between a room session and its current media publication. +[Interaction research](../research/room-interactions.md) records delivery bounds, +compatibility and design rationale. + ## Persistence Modes - **Stable mode (Hosted default):** one exact-schema SQLite owner persists room diff --git a/docs/standards/routing-transport.md b/docs/standards/routing-transport.md index c5fb992..19bfe17 100644 --- a/docs/standards/routing-transport.md +++ b/docs/standards/routing-transport.md @@ -7,7 +7,7 @@ owns ports and [self-hosting](../operations/self-hosting.md) owns service setup. ## Topology -- Media is automatic and P2P-first. Users do not select parents, route types, or +- Media is automatic and defaults to P2P-first. Users do not select parents, route types, or network transports. - The Host is the only source. Every Viewer has at most one active upstream, and the committed media graph is acyclic and source-reachable. @@ -21,9 +21,17 @@ same bounded P2P graph without its SFU fallback and fixes Privacy mode on for al rooms, including App Local and public-link rooms. The server enforces that policy for waiting and active participants; there is no separate Host-star mode. +A Hosted operator may require SFU-only media through `SFU_ONLY=true`. The same +controller then admits only SFU candidates: no Peer parents, Viewer relays or +background P2P convergence. The Host still publishes once, and every Viewer +uses the existing SFU subscription ledger and first-frame commit. Capacity or +connection failure never permits a Peer route. An explicitly peer-only share +is rejected before acquiring media authority; it is never silently forwarded. +The page advertises this site policy and hides inapplicable P2P preferences. + ## Per-Share Route Policy -The Host chooses route policy before sharing and it remains fixed for that +On a P2P-first site, the Host chooses route policy before sharing and it remains fixed for that share generation: - when SFU is available, default hybrid mode keeps P2P first and permits the diff --git a/docs/standards/visual-language.md b/docs/standards/visual-language.md index 39d5f74..857a117 100644 --- a/docs/standards/visual-language.md +++ b/docs/standards/visual-language.md @@ -56,6 +56,15 @@ do not stretch short values to fill a phone row. Check actual control bounds around breakpoints, in both orientations and all presentation modes; page-level overflow clipping is not evidence that the content fits. +Persistent room utilities use the shared floating panel without reflowing the +living room. The shell owns visibility, focus return, movement, resizing and +viewport/keyboard bounds; contents retain their domain state. Support title-bar +dragging and keyboard placement. An interaction palette starts beside its chosen +participant, stays open after sending and keeps its position when retargeted. +One-shot settings and guidance retain their existing disclosures. Escape closes +inner guidance first, then the most recently opened floating panel, then theater +mode; an IME composition must not dismiss these surfaces. + Keep the application language control's segmented Chinese, English and visual shortcuts. When additional catalogs exist, insert one language menu after the visual shortcut; all extra languages share that slot. Show the @@ -172,6 +181,8 @@ Participant characters should leave room for that distinctive presence. | Browser application | Browser chrome around content; not a second design of television | | A captured window or display | Window title bar or display stand, with the shared media metaphor kept distinct | | A camera source or Host commentary | `camera` identifies the alternative picture source; `microphone` identifies the enabled Host voice input and `microphoneOff` its muted state. Neither replaces the speaker metaphor for source/playback sound. | +| Room text and reactions | `chat` opens the conversation, `danmaku` controls its optional picture overlay, `smile` identifies reactions and `send` submits the chosen content. These do not represent media transport or voice. | +| Floating utility placement | `grip` marks a draggable title bar; `move` opens placement controls. Closing a panel hides its presentation, not its room session. | | Server forwarding media | A server on the media path; opening a Site alone does not imply SFU | These are **identity/object colours**, not status. A green pawn remains green diff --git a/docs/status.md b/docs/status.md index 63f8ebf..1a086f7 100644 --- a/docs/status.md +++ b/docs/status.md @@ -1,6 +1,6 @@ # Current Status -Last updated: 2026-09-29 +Last updated: 2026-10-02 This is the compact execution/deployment index. Product modules own behavior, [verification status](./verification-status.md) owns unresolved physical limits, @@ -35,6 +35,24 @@ Website delivery includes a bilingual reader documentation center generated from the repository's guides alongside the homepage, with search and detailed troubleshooting separate from the homepage FAQ. +## Accepted Interaction Phase + +The accepted [room-interaction design](./research/room-interactions.md) includes +the verified media repairs. It adds authenticated text and reactions +over existing signaling, with optional local chat overlay and a shared floating +panel. Its opted-in room session can outlive a publication; capture, quality and +route owners remain unchanged. Optional Windows screen-audio exclusion uses the +existing source/mixer boundary. Hosted Server +also accepts an optional SFU-only policy through the same route controller. +Windows native capture and thumbnails now share automatic HDR-to-SDR conversion; +synthetic GPU and Browser reception checks pass, with physical HDR display +acceptance still pending in [verification status](./verification-status.md). +Room voice is outside the product plan; Host microphone commentary remains. +4K remains deferred. The owner accepted the UI/model and bounded local acceptance +and authorized v1.7.0 publication. Physical device limits remain open. +Release artifacts and operator deployment records own delivery completion; +[TODO](./todo.md#now) owns remaining work. + ## Deployment Source, package identity, GitHub repository and local worktree paths use Piik. diff --git a/docs/todo.md b/docs/todo.md index c5dbee7..18c47af 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -1,70 +1,58 @@ # Current TODO Ledger -Last reviewed: 2026-09-30 +Last reviewed: 2026-10-02 Only **Now** is executable. Product modules own behavior; Git/PRs own completed history. A parked idea is not implementation authority. ## Now -- [ ] **Published-product bug and experience acceptance.** Prioritize share - startup, audio/source changes, sustained viewing, recovery and retirement, - admission, and narrow/touch UI. Verify confirmed fixes through the existing - owners before resuming feature polish. Keep field reports without matched - evidence separate from locally reproduced defects. -- [ ] **Sustained Browser degradation.** The H264 field cause remains open. +- [ ] **Authorized release delivery.** The owner accepted the room-interaction + phase, complete bilingual release notes and bounded acceptance, and authorized + v1.7.0 publication. Follow + [the integration workflow](../CONTRIBUTING.md#full-integration-and-release-workflow) + for one squash merge, packaging, publication and scoped postflight. Release + artifacts and operator records own completion; do not add a release-record + commit. No implementation work remains open for this phase. + +## Held By Owner + +- [ ] **Browser-internal encoder work.** Paused by the owner on 2026-10-01. [Comparisons and Chromium traces](./research/browser-local-encoding-pool.md#sustained-h264-recovery) - distinguish encoder recovery hysteresis from a tab-capture feedback/size-step - lock reproduced in bare WebRTC. Verify the upstream soft-target/hard-maximum - repair candidate against a patched Browser, which remains unbuilt. The Piik - sender boundary now isolates display frames while preserving native idle refresh, - capture intent and operation retirement. Real hidden-tab SFU checks pass; - broader device/background coverage remains. Native logs confirm content-driven - QP hysteresis and a cost from bitrate reconfiguration, but not repeated brief - blur. Match field Host/Viewer evidence - before assigning the field cause or declaring repeated brief blur fixed. - Retain existing clone/startup protections and candidate proof; no periodic - reset or quality floor. A recent Piik regression is not established. + own the evidence for synchronous H264 initialization, native adaptation and + the unbuilt upstream repair candidate. Keep the verified Piik budget and + handoff repairs; do not add periodic resets, a quality floor or a Browser fork. + The field room's cause and all brief blur are not established as resolved. ## Deferred Feature Work -Feature design and interaction polish are paused while the bug/experience phase -is active. Retain these decisions for resumption, not as parallel work. +These proposals remain deferred beyond the accepted interaction phase. - [ ] **Passive App attachment: design hold.** Site mode authorizes one selected origin and supplies native media without starting a local room server. A passive replacement needs an accepted site-consent/discovery flow; it must not admit arbitrary sites or add another runtime owner. Keep Site mode until that decision; removal of the Demo prefill does not authorize changing site trust. -- [ ] **Refine the isolated room-interaction prototype.** Review chat, optional - danmaku and participant-targeted reactions for usability, placement and intended - cross-view synchronization. Preserve room authorization and media-route owners. - Accept its combined layout: sharing settings with the picture, floating chat - without page reflow and room actions before diagnostics. Check narrow-screen - keyboard avoidance, its theater entry and participant menu on real devices; - keep chat available independently of sharing. - Retain current room creation: the first share creates a room, an existing room - can resume, and stopping media keeps the opted-in interaction session. Do not - add visit-triggered creation or a separate pre-share room-creation entry. - This experiment does not authorize integration or publication. -- [ ] **Room voice: design before implementation.** The - [voice assessment](./research/room-voice.md) maps current owners, mature options - and a staged verification plan. Resolve total voice membership/topology, - P2P-only deployment coverage, speaking authority and the Host-commentary - compatibility boundary before implementation. Validate source-audio isolation - from received speech before exposing simultaneous sharing and voice. Keep - room authority, microphone and screen lifetimes distinct; no new room store, - screen-tree reversal or automatic mesh/SFU controller is approved. -- [ ] **Verify media fidelity independently of voice.** - Verify actual left/right separation through Browser, Native and SFU paths; - receive-side stereo negotiation alone does not establish audible fidelity. - Assess HDR capture to correct SDR output for ordinary viewers before full HDR, - with issue [#420](https://github.com/TNTcraftHIM/Piik/issues/420) as the reported - case. Full HDR and surround sound remain design decisions. These findings do - not establish physical device acceptance or authorize a new release. ## Awaiting Device Or Reporter Evidence +- [ ] **HDR, audio-exclusion and mobile interaction device coverage.** Complete + the remaining physical checks in + [verification status](./verification-status.md#candidate-evidence-boundary): + real HDR/mixed displays under load, real voice-app process trees across playback + devices, and phone keyboard/theater/participant-menu interaction. Local GPU, + decoded-video, synthetic audio and responsive-browser evidence do not cover + those environments. Issues [#420](https://github.com/TNTcraftHIM/Piik/issues/420) + (Browser HDR) and [#445](https://github.com/TNTcraftHIM/Piik/issues/445) + (Windows 10 colors) still need matched capture evidence; do not assign them + the locally reproduced Native conversion defects. +- [ ] **Native H.264 motion quality (#432).** The original 720p30 H.264/VP8 + visual comparison remains unresolved. Browser evidence confirms similar + delivered bitrate/format and zero reported loss, but contains no matched + pictures or bitstream. The reporter withdrew the fixture-contaminated CBR + ceiling claim. [Evidence review](./research/native-client-media.md#native-h264-motion-quality) + owns that distinction. Reproduce with identical content and unchanged + production settings before proposing profile, default-FPS or bitrate changes. - [ ] **Self-hosted room creation HTTP 403.** Retest affected deployments using the [address checks](./guide/troubleshooting.md#room-creation-returns-403). Obtain the configured public address/origin and response details; distinguish @@ -78,6 +66,9 @@ is active. Retain these decisions for resumption, not as parallel work. evidence before calling that an ICE/NAT failure. HTTP 1033 belongs to the public-link item below, before media routing. Local fixes and STUN checks do not establish these reporters' causes. + [#443](https://github.com/TNTcraftHIM/Piik/issues/443)'s Host report reaches Native + prepare/answer/candidate processing without a connected remote edge; obtain + the same attempt's Viewer and App transport reports to locate the failure. - [ ] **Camera and Host microphone device coverage.** The owner accepted the sharing layout and authorized release with these physical limits recorded. Verify real audio levels/echo, multiple-device replacement and native mixing on @@ -92,6 +83,9 @@ is active. Retain these decisions for resumption, not as parallel work. [runtime evidence](./research/cross-platform-client-runtime.md#public-invitation-startup). Demo access failures and public-link HTTP 1033 also remain unconfirmed; check connector, DNS/provider and remote access separately from WebRTC availability. + [#434](https://github.com/TNTcraftHIM/Piik/issues/434)'s attached log records DNS + refusal during Cloudflare edge discovery, before readiness; retrying media + cannot repair that resolver failure. Do not assign it to every startup report. - [ ] **Interruption during established viewing.** A Viewer reportedly returns to P2P connecting after watching for a while. Include [#429](https://github.com/TNTcraftHIM/Piik/issues/429)'s reported SFU-to-P2P @@ -126,6 +120,10 @@ is active. Retain these decisions for resumption, not as parallel work. also needs paired evidence; distinguish activation, codec configuration and actual output before changing the encoder contract. Local NVIDIA success does not settle AMD activation. + In [#437](https://github.com/TNTcraftHIM/Piik/issues/437), both manually selected + codecs reportedly work; its Browser report records Auto startup returning + `operation-failed` after 6.85 seconds, before the request timeout. Obtain the + same attempt's App diagnostics to locate selection/activation failure. - [ ] **Share ends after entering a game.** Screen sharing reportedly works until entering a game freezes the picture, followed seconds later by share termination. Version, capture path, codec and matched diagnostics are unknown. @@ -249,3 +247,7 @@ not establish better connection success or speed; this note adds no retry policy 11. **Release-operation policy.** Protected release environments, immutable draft assets and changing mirror-failure policy remain unaccepted proposals; evaluate their benefit before adding release machinery. +12. **Linux ARM64 Server distribution (#439).** Evaluate archives and container + publishing together with deployment names, update links and runtime checks. + A cross-compile alone does not establish a supported package. This is a + community proposal, not part of the current accepted candidate. diff --git a/docs/verification-status.md b/docs/verification-status.md index 470100a..70aaa27 100644 --- a/docs/verification-status.md +++ b/docs/verification-status.md @@ -1,6 +1,6 @@ # Verification Status -Last updated: 2026-09-17 +Last updated: 2026-10-01 This file owns physical evidence limits that change how the product may be described. [Status](./status.md) owns the execution/deployment index. @@ -47,6 +47,26 @@ or a measured improvement worth its implementation and maintenance cost. ## Candidate Evidence Boundary +The Windows HDR-to-SDR candidate has synthetic FP16 conversion and decoded +H.264/VP8 Browser reception evidence. Local displays were in SDR mode; real HDR +content, SDR white on HDR displays, mixed-display movement, display-mode changes +and sustained GPU contention still require physical acceptance. The +[fidelity assessment](./research/media-fidelity.md#hdr-to-sdr) owns measurements. +These checks do not establish Browser-capture HDR behavior or end-to-end HDR. + +Windows source-audio exclusion has bounded parent/child tone-isolation and +target-exit evidence. It does not establish coverage of arbitrary process trees +or real voice applications across playback devices. The +[audio assessment](./research/native-client-media.md#windows-audio-exclusion) +owns that distinction. + +Room interactions have multi-page Browser evidence for stop/resume, idle-room +join, Host departure/return, reconnect, invitation revocation and event timing. +Responsive Browser checks cover narrow portrait and landscape layouts, but not +real phone keyboards, Safari behavior or every touch/device combination. The +owner accepted this bounded UI/model walkthrough for phase closure; these +physical limits remain open and are not release authorization. + The 2026-09-11 functional test acceptance used two genuine Windows App packages and matching Server source/Web builds under the same private contract. Both App/Site version directions delivered decoded 1080p H264 from an isolated native test window. diff --git a/internal/app/app.go b/internal/app/app.go index b645a17..f65036a 100644 --- a/internal/app/app.go +++ b/internal/app/app.go @@ -621,14 +621,15 @@ func discoverNativeMedia(ctx context.Context, configuredPath string) nativeRunti captureProcess: path, capture: capabilities, capabilities: loopback.NativeMediaCapabilities{ - ReceiverReuse: true, - Video: summary.Video, - CaptureBorderControl: summary.CaptureBorderControl, - Microphone: summary.Microphone, - ProcessAudio: summary.ProcessAudio, - SystemAudio: summary.SystemAudio, - HardwareH264: summary.HardwareH264, - SoftwareVP8: summary.SoftwareVP8, + ReceiverReuse: true, + Video: summary.Video, + CaptureBorderControl: summary.CaptureBorderControl, + Microphone: summary.Microphone, + ProcessAudio: summary.ProcessAudio, + ProcessAudioExclusion: summary.ProcessAudioExclusion, + SystemAudio: summary.SystemAudio, + HardwareH264: summary.HardwareH264, + SoftwareVP8: summary.SoftwareVP8, }, } } diff --git a/internal/app/app_test.go b/internal/app/app_test.go index 40ff38d..2b17851 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -121,6 +121,12 @@ func TestPublicLinkStartupDistinguishesCancellationFromFailure(t *testing.T) { } cancel() } + waitForResult := 5 * time.Second + if mode == "failed" { + // Startup includes three retry pauses and four real child launches; + // this is not the cancellation/ready-tunnel retirement deadline. + waitForResult = 15 * time.Second + } select { case err = <-done: joined = true @@ -139,7 +145,7 @@ func TestPublicLinkStartupDistinguishesCancellationFromFailure(t *testing.T) { if ready != (mode == "ready" || mode == "ready-failed") { t.Fatalf("startup readiness = %v for %s", ready, mode) } - case <-time.After(5 * time.Second): + case <-time.After(waitForResult): t.Fatal("App did not retire its local authority and tunnel") } content, err := os.ReadFile(filepath.Join(directory, "logs", "client.log")) diff --git a/internal/app/loopback/server.go b/internal/app/loopback/server.go index 798b931..a2f4e66 100644 --- a/internal/app/loopback/server.go +++ b/internal/app/loopback/server.go @@ -69,14 +69,15 @@ type Health struct { } type NativeMediaCapabilities struct { - ReceiverReuse bool `json:"receiverReuse,omitempty"` - Video bool `json:"video"` - CaptureBorderControl bool `json:"captureBorderControl,omitempty"` - Microphone bool `json:"microphone,omitempty"` - ProcessAudio bool `json:"processAudio"` - SystemAudio bool `json:"systemAudio"` - HardwareH264 bool `json:"hardwareH264"` - SoftwareVP8 bool `json:"softwareVP8"` + ReceiverReuse bool `json:"receiverReuse,omitempty"` + Video bool `json:"video"` + CaptureBorderControl bool `json:"captureBorderControl,omitempty"` + Microphone bool `json:"microphone,omitempty"` + ProcessAudio bool `json:"processAudio"` + ProcessAudioExclusion bool `json:"processAudioExclusion,omitempty"` + SystemAudio bool `json:"systemAudio"` + HardwareH264 bool `json:"hardwareH264"` + SoftwareVP8 bool `json:"softwareVP8"` } type Server struct { diff --git a/internal/app/mediaedge/edge_test.go b/internal/app/mediaedge/edge_test.go index 6b5a495..700c8de 100644 --- a/internal/app/mediaedge/edge_test.go +++ b/internal/app/mediaedge/edge_test.go @@ -65,9 +65,11 @@ func TestOutputPlanRetiresOnlyFailedLayerConsumers(t *testing.T) { lowerTracker, higherTracker := trackers.AddTracker(0), trackers.AddTracker(1) lowerTracker.Observe(0, 1200, 1188, true, 90000, nil) higherTracker.Observe(0, 1200, 1188, true, 90000, nil) - if err = source.DisableLayer(0); err != nil { + retire, err := source.MarkLayerUnavailable(0) + if err != nil { t.Fatal(err) } + retire() if low.State() != webrtc.PeerConnectionStateClosed || high.State() != webrtc.PeerConnectionStateConnected { t.Fatalf("failed/sibling state = %s/%s", low.State(), high.State()) } @@ -174,6 +176,59 @@ func TestRelayFailureCannotRetireReconfiguredProfile(t *testing.T) { } } +func TestRetiredRelayFailureCannotDisableLaterDemand(t *testing.T) { + engine, err := NewEngine(EngineOptions{BindAddress: "127.0.0.1:0", IncludeLoopback: true}) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = engine.Close() }) + source, err := engine.NewSource("vp8", 2, 2, nil) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = source.Close() }) + source.relay = &relayDerivation{source: source} + if err = source.SetFormat(1, 640, 360); err != nil { + t.Fatal(err) + } + if err = source.SetRelayProfile(nativecapture.VideoProfile{ + Width: 1280, Height: 720, Framerate: 30, Bitrate: 3_000_000, Preference: "balanced", + }); err != nil { + t.Fatal(err) + } + high, _, _ := connectedReceiver(t, engine, source, "retiring-relay-high") + low, _, _ := connectedReceiver(t, engine, source, "retiring-relay-low") + if err = low.SetTargetLayer(0); err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithCancelCause(t.Context()) + defer cancel(nil) + done := make(chan struct{}) + t.Cleanup(func() { close(done) }) + run := &relayRun{owner: source.relay, ctx: ctx, cancel: cancel, done: done, + plan: relayPlan{profile: source.relayProfile, format: source.formats[1].Load()}} + source.relay.run = run + if err = low.Close(); err != nil { + t.Fatal(err) + } + if !errors.Is(context.Cause(ctx), context.Canceled) { + t.Fatal("last lower consumer did not retire the derivation normally") + } + // A failure already read from the old process can reach the source lock + // after cancellation, before the process exits and the next run starts. + retire, err := source.markLayerUnavailable(0, run) + if err != nil || retire != nil || source.outputBitrates[0] == 0 { + t.Fatal("normal retirement disabled a reusable output slot") + } + rejoined, _, _ := connectedReceiver(t, engine, source, "retiring-relay-rejoined") + if err = rejoined.SetTargetLayer(0); err != nil { + t.Fatalf("later lower demand was rejected: %v", err) + } + if rejoined.State() != webrtc.PeerConnectionStateConnected || high.State() != webrtc.PeerConnectionStateConnected { + t.Fatal("late failure retired a current consumer or healthy sibling") + } +} + func TestShutdownClosesSiblingTransportsBeforeWaitingForWrites(t *testing.T) { for _, owner := range []string{"source", "engine"} { t.Run(owner, func(t *testing.T) { diff --git a/internal/app/mediaedge/groups.go b/internal/app/mediaedge/groups.go index 15f3c81..b0d8dbe 100644 --- a/internal/app/mediaedge/groups.go +++ b/internal/app/mediaedge/groups.go @@ -95,6 +95,23 @@ func (source *Source) planGroups(demands []outputDemand) (OutputPlan, error) { delete(source.memberships, consumer) } } + compatible := func(group *outputGroup, demand outputDemand) bool { + for _, other := range demands { + if other.active && other.lower && other.budget != demand.budget && + (other.consumer.CurrentSource() == group.media.Source || source.memberships[other.consumer] == group) { + return false + } + } + return true + } + // Revalidate pending joins before they can prevent an incumbent from + // adjusting in place. Membership order must not give stale demand priority. + for _, demand := range demands { + group := source.memberships[demand.consumer] + if demand.active && demand.lower && group != nil && demand.consumer.CurrentSource() != group.media.Source && !compatible(group, demand) { + delete(source.memberships, demand.consumer) + } + } // The previous frame's active bit alone may outlive its last handoff. inUse := func(group *outputGroup) bool { if group == nil || !group.active { @@ -119,6 +136,9 @@ func (source *Source) planGroups(demands []outputDemand) (OutputPlan, error) { for slot := 0; slot < len(source.outputBitrates); slot++ { group := source.groups[slot] if group != nil && source.outputBitrates[slot] > 0 && group.budget == demand.budget { + if demand.consumer.CurrentSource() != group.media.Source && !compatible(group, demand) { + continue + } if selected == nil || inUse(group) { selected = group } @@ -152,15 +172,7 @@ func (source *Source) planGroups(demands []outputDemand) (OutputPlan, error) { if source.outputBitrates[group.slot] == 0 { continue } - compatible := true - for _, other := range demands { - if other.active && other.lower && other.budget != demand.budget && - (other.consumer.CurrentSource() == group.media.Source || source.memberships[other.consumer] == group) { - compatible = false - break - } - } - if compatible { + if compatible(group, demand) { selected = group break } diff --git a/internal/app/mediaedge/groups_test.go b/internal/app/mediaedge/groups_test.go index d0ebf6b..7b23156 100644 --- a/internal/app/mediaedge/groups_test.go +++ b/internal/app/mediaedge/groups_test.go @@ -1,6 +1,7 @@ package mediaedge import ( + "fmt" "testing" "time" @@ -253,6 +254,64 @@ func newGroupLifecycleFixture(t *testing.T) (*Source, *groupTestConsumer, *outpu return source, b, group, demands } +func TestOutputGroupsPreserveCurrentDemandDuringAdmission(t *testing.T) { + for _, pending := range []bool{false, true} { + for _, reverse := range []bool{false, true} { + t.Run(fmt.Sprintf("pending=%v/reverse=%v", pending, reverse), func(t *testing.T) { + source, b, gb, demands := newGroupLifecycleFixture(t) + a := demands[0].consumer + ga := source.memberships[a] + source.installGroup(ga, false) + source.installGroup(gb, false) + groupCount := len(source.groups) + demands[1].budget = 100_000 + if pending { + if _, err := source.planGroups(demands); err != nil { + t.Fatal(err) + } + if source.memberships[b] != ga || b.CurrentSource() != gb.media.Source { + t.Fatal("fixture did not prepare a compatible rejoin") + } + } + source.groupRecovery.Store(0) + demands[0].budget = 60_000 + if reverse { + demands[0], demands[1] = demands[1], demands[0] + } + plan, err := source.planGroups(demands) + if err != nil { + t.Fatal(err) + } + if source.memberships[a] != ga || source.memberships[b] != gb || len(source.groups) != groupCount || + plan.Bitrates[ga.slot] != 60_000 || plan.Bitrates[gb.slot] != 100_000 || source.groupRecovery.Load() != 0 { + t.Fatalf("admission displaced an incumbent: a=%v b=%v groups=%d/%d rates=%v recovery=%d", + source.memberships[a] == ga, source.memberships[b] == gb, len(source.groups), groupCount, + plan.Bitrates, source.groupRecovery.Load()) + } + }) + } + } +} + +func TestOutputGroupsRetainOriginalRecoveryAfterPendingJoin(t *testing.T) { + source, b, gb, demands := newGroupLifecycleFixture(t) + source.installGroup(source.memberships[demands[0].consumer], false) + source.installGroup(gb, false) + demands[1].budget = demands[0].budget + if _, err := source.planGroups(demands); err != nil { + t.Fatal(err) + } + source.groupRecovery.Store(0) + demands[0].budget = 60_000 + demands[1].lower, demands[1].original = false, true + if _, err := source.planGroups(demands); err != nil { + t.Fatal(err) + } + if source.memberships[b] != nil || source.groupRecovery.Load()&2 == 0 || b.CurrentSource() != gb.media.Source { + t.Fatal("return to original lost recovery or replaced the current source early") + } +} + func TestOutputGroupRejectsRecoveryBeforeChangingAttachment(t *testing.T) { for _, path := range []string{"derived", "original"} { for _, stamp := range []string{"future", "duplicate", "before-generation-start"} { diff --git a/internal/app/mediaedge/publication_test.go b/internal/app/mediaedge/publication_test.go index f36fd15..4b94b37 100644 --- a/internal/app/mediaedge/publication_test.go +++ b/internal/app/mediaedge/publication_test.go @@ -73,9 +73,11 @@ func TestPublicationOwnsOneReservationAndSourceLifetime(t *testing.T) { if err != nil { t.Fatal(err) } - if err = source.DisableLayer(0); err != nil { + retire, err := source.MarkLayerUnavailable(0) + if err != nil { t.Fatal(err) } + retire() if _, err = engine.NewPublication(source, EdgeOptions{ConnectionID: "failed-output"}); err == nil { t.Fatal("failed output was admitted into publication metadata") } @@ -117,9 +119,11 @@ func TestPublicationAdmissionIgnoresUnrelatedFailedOutputs(t *testing.T) { t.Fatal(err) } for _, slot := range []int{2, 3, 0, 1} { - if err = source.DisableLayer(slot); err != nil { + retire, err := source.MarkLayerUnavailable(slot) + if err != nil { t.Fatal(err) } + retire() candidate, createErr := engine.NewPublication(source, EdgeOptions{ConnectionID: "replacement"}) if slot >= 2 { if createErr != nil { diff --git a/internal/app/mediaedge/source.go b/internal/app/mediaedge/source.go index de2ef3d..77a781f 100644 --- a/internal/app/mediaedge/source.go +++ b/internal/app/mediaedge/source.go @@ -1,6 +1,7 @@ package mediaedge import ( + "context" "errors" "io" "sync" @@ -211,12 +212,10 @@ func (source *Source) ConfigureOutputs(ceilings []uint32) error { return nil } -func (source *Source) DisableLayer(layer int) error { - retire, err := source.markLayerUnavailable(layer, nil) - if retire != nil { - retire() - } - return err +// MarkLayerUnavailable commits output failure without closing transports. The +// caller must run the returned retirement after releasing its capture-owner lock. +func (source *Source) MarkLayerUnavailable(layer int) (func(), error) { + return source.markLayerUnavailable(layer, nil) } func (source *Source) markLayerUnavailable(layer int, run *relayRun) (func(), error) { @@ -227,8 +226,10 @@ func (source *Source) markLayerUnavailable(layer int, run *relayRun) (func(), er source.writeMu.Lock() defer source.writeMu.Unlock() source.mu.Lock() - if run != nil && (run.owner.run != run || source.relayProfile != run.plan.profile || - source.formats[len(source.formats)-1].Load() != run.plan.format) { + // Normal retirement cannot disable a later run's slots. A real cancellation + // cause still owns failure isolation until that run has finished retiring. + if run != nil && (run.owner.run != run || errors.Is(context.Cause(run.ctx), context.Canceled) || + source.relayProfile != run.plan.profile || source.formats[len(source.formats)-1].Load() != run.plan.format) { source.mu.Unlock() return nil, nil } diff --git a/internal/app/nativeaudio/encoder_test.go b/internal/app/nativeaudio/encoder_test.go index 82ab344..0744054 100644 --- a/internal/app/nativeaudio/encoder_test.go +++ b/internal/app/nativeaudio/encoder_test.go @@ -85,6 +85,58 @@ func TestEncodePreservesAudiblePCM(t *testing.T) { } } +func TestEncodePreservesStereoSeparation(t *testing.T) { + for _, bitrate := range []int{64_000, 128_000, 192_000} { + encoder, err := NewEncoder(bitrate) + if err != nil { + t.Fatal(err) + } + decoder, err := gopus.NewDecoder(gopus.DefaultDecoderConfig(SampleRate, Channels)) + if err != nil { + t.Fatal(err) + } + pcm := make([]byte, FrameBytes) + decoded := make([]int16, FrameSamples*Channels) + frequencies := [2]float64{500, 1500} + var tones [2][2]complex128 + for block := 0; block < 30; block++ { + for frame := 0; frame < FrameSamples; frame++ { + for channel, frequency := range frequencies { + phase := 2 * math.Pi * frequency * float64(block*FrameSamples+frame) / SampleRate + binary.LittleEndian.PutUint16(pcm[(frame*Channels+channel)*2:], uint16(int16(8_000*math.Sin(phase)))) + } + } + packet, err := encoder.Encode(pcm) + if err != nil { + t.Fatal(err) + } + count, err := decoder.DecodeInt16(packet, decoded) + if err != nil || count != FrameSamples { + t.Fatalf("decode: %d samples, %v", count, err) + } + if block < 5 { + continue // Exclude codec startup; compare frequency energy, not phase delay. + } + for frame := 0; frame < count; frame++ { + for tone, frequency := range frequencies { + phase := 2 * math.Pi * frequency * float64(block*FrameSamples+frame) / SampleRate + for channel := range Channels { + tones[channel][tone] += complex(float64(decoded[frame*Channels+channel]), 0) * complex(math.Cos(phase), math.Sin(phase)) + } + } + } + } + for channel := range Channels { + wanted, other := tones[channel][channel], tones[channel][1-channel] + level := math.Hypot(real(wanted), imag(wanted)) + leak := math.Hypot(real(other), imag(other)) + if level < 1 || level < 10*leak { + t.Fatalf("%d bps channel %d lost separation: signal %f, leakage %f", bitrate, channel, level, leak) + } + } + } +} + func TestNewEncoderRejectsOutOfRangeBitrate(t *testing.T) { for _, bitrate := range []int{0, 5_999, 510_001} { if _, err := NewEncoder(bitrate); err == nil { diff --git a/internal/app/nativecapture/probe.go b/internal/app/nativecapture/probe.go index cc30fcb..8f4a0a1 100644 --- a/internal/app/nativecapture/probe.go +++ b/internal/app/nativecapture/probe.go @@ -42,36 +42,39 @@ type Adapter struct { } type Capabilities struct { - Protocol int `json:"protocol"` - Platform string `json:"platform"` - PlatformBuild string `json:"platformBuild"` - VideoCapture bool `json:"videoCapture"` - CaptureBorderControl bool `json:"captureBorderControl,omitempty"` - Microphone bool `json:"microphone,omitempty"` - ProcessAudio bool `json:"processAudio"` - SystemAudio bool `json:"systemAudio"` - SoftwareVP8 bool `json:"softwareVP8"` - Adapters []Adapter `json:"adapters"` + Protocol int `json:"protocol"` + Platform string `json:"platform"` + PlatformBuild string `json:"platformBuild"` + VideoCapture bool `json:"videoCapture"` + CaptureBorderControl bool `json:"captureBorderControl,omitempty"` + Microphone bool `json:"microphone,omitempty"` + ProcessAudio bool `json:"processAudio"` + ProcessAudioExclusion bool `json:"processAudioExclusion,omitempty"` + SystemAudio bool `json:"systemAudio"` + SoftwareVP8 bool `json:"softwareVP8"` + Adapters []Adapter `json:"adapters"` } type Summary struct { - Video bool - CaptureBorderControl bool - Microphone bool - ProcessAudio bool - SystemAudio bool - HardwareH264 bool - SoftwareVP8 bool + Video bool + CaptureBorderControl bool + Microphone bool + ProcessAudio bool + ProcessAudioExclusion bool + SystemAudio bool + HardwareH264 bool + SoftwareVP8 bool } func (capabilities Capabilities) Summary() Summary { summary := Summary{ - Video: capabilities.VideoCapture, - CaptureBorderControl: capabilities.VideoCapture && capabilities.CaptureBorderControl, - Microphone: capabilities.Microphone, - ProcessAudio: capabilities.ProcessAudio, - SystemAudio: capabilities.SystemAudio, - SoftwareVP8: capabilities.SoftwareVP8, + Video: capabilities.VideoCapture, + CaptureBorderControl: capabilities.VideoCapture && capabilities.CaptureBorderControl, + Microphone: capabilities.Microphone, + ProcessAudio: capabilities.ProcessAudio, + ProcessAudioExclusion: capabilities.ProcessAudioExclusion && capabilities.Microphone, + SystemAudio: capabilities.SystemAudio, + SoftwareVP8: capabilities.SoftwareVP8, } for _, adapter := range capabilities.Adapters { if len(adapter.HardwareH264) > 0 { diff --git a/internal/app/nativecapture/probe_test.go b/internal/app/nativecapture/probe_test.go index 6f2d1ef..893b955 100644 --- a/internal/app/nativecapture/probe_test.go +++ b/internal/app/nativecapture/probe_test.go @@ -33,6 +33,22 @@ func TestDecodeProbeRejectsUnknownTrailingAndDuplicateData(t *testing.T) { } } +func TestAudioExclusionCapabilityRequiresExplicitSupportAndMixing(t *testing.T) { + capabilities, err := decodeProbe([]byte(validProbe)) + if err != nil || capabilities.Summary().ProcessAudioExclusion { + t.Fatal("old capture probe advertised audio exclusion", err) + } + capabilities, err = decodeProbe([]byte(strings.Replace(validProbe, `"processAudio":true`, + `"processAudio":true,"processAudioExclusion":true,"microphone":true`, 1))) + if err != nil || !capabilities.Summary().ProcessAudioExclusion { + t.Fatal("explicit audio exclusion capability was lost", err) + } + capabilities.Microphone = false + if capabilities.Summary().ProcessAudioExclusion { + t.Fatal("audio exclusion requires the replaceable mixed source") + } +} + func TestSummaryDoesNotConflateVideoAudioAndEncoder(t *testing.T) { capabilities, err := decodeProbe([]byte( `{"protocol":7,"platform":"windows","platformBuild":"19045","videoCapture":true,"processAudio":false,"systemAudio":true,"adapters":[{"index":0,"name":"GPU","identity":"0x0:0x1","hardwareH264":[]}]}`, diff --git a/internal/app/nativecapture/stream.go b/internal/app/nativecapture/stream.go index b8caf53..2b31d66 100644 --- a/internal/app/nativecapture/stream.go +++ b/internal/app/nativecapture/stream.go @@ -281,13 +281,17 @@ func appendOutputArguments(arguments []string, outputs []OutputProfile) ([]strin return arguments, nil } -func StartAudio(parent context.Context, executable string, target CaptureTarget) (*Stream, error) { - if !target.Valid() { +func StartAudio(parent context.Context, executable string, target CaptureTarget, exclude bool) (*Stream, error) { + if !target.Valid() || (exclude && target.Kind != "window") { return nil, errors.New("native audio target is invalid") } + kind := target.Kind + if exclude { + kind = "exclude" + } return startAudioStream(parent, executable, []string{ "--capture-audio", - target.Kind, + kind, strconv.FormatUint(uint64(target.PID), 10), zeroWhenEmpty(target.CreationTime), }) diff --git a/internal/app/nativecontrol/session.go b/internal/app/nativecontrol/session.go index 19d80f4..40a59a7 100644 --- a/internal/app/nativecontrol/session.go +++ b/internal/app/nativecontrol/session.go @@ -182,6 +182,12 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any, !validQualitySettings(request.Profile) { return nil, protocolViolation("native start-share request is invalid") } + if err := session.validateAudioExclusion(request.Source, request.Audio, request.ExcludeAudio); err != nil { + return nil, err + } + if request.ExcludeAudio != nil && !request.MicrophoneMixing { + return nil, errors.New("native audio exclusion requires mixed audio") + } slog.Debug("piik-client", "event", "native-profile-requested", "requestId", envelope.ID, "share", diagnostics.ID(request.ShareID), "profile", nativeQualityProfile(request.Profile), "codec", request.Codec, "audio", request.Audio, "sourceKind", request.Source.Kind, "adapterIndex", request.AdapterIndex, "encoderIndex", request.EncoderIndex) @@ -245,6 +251,9 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any, !validIdentities(request.ShareID) || !request.Source.Valid() { return nil, protocolViolation("native replace-share-source request is invalid") } + if err := session.validateAudioExclusion(request.Source, request.Audio, request.ExcludeAudio); err != nil { + return nil, err + } host := session.current(request.ShareID) if host == nil { return nil, errors.New("native share does not exist") @@ -261,7 +270,7 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any, ShowCaptureBorder: request.ShowCaptureBorder && session.capabilities.CaptureBorderControl, AdapterIndex: request.AdapterIndex, EncoderIndex: request.EncoderIndex, - }, audio) + }, audio, request.ExcludeAudio) if err != nil { return nil, err } @@ -534,6 +543,19 @@ func (session *Session) finishHostOperation(done chan struct{}, result any) { session.mu.Unlock() } +func (session *Session) validateAudioExclusion(source nativecapture.CaptureTarget, audio bool, excluded *nativecapture.CaptureTarget) error { + if excluded == nil { + return nil + } + if !audio || source.Kind != "display" || excluded.Kind != "window" || !excluded.Valid() { + return protocolViolation("native audio exclusion target is invalid") + } + if !session.capabilities.Summary().ProcessAudioExclusion || !session.capabilities.SystemAudio { + return errors.New("native audio exclusion is unavailable") + } + return nil +} + func (session *Session) shareUpdateResult(request requestEnvelope, shareID string, profile nativehost.QualityProfile, err error) any { if err != nil { return operationFailure(request, err) @@ -571,6 +593,7 @@ func (session *Session) startShare( Profile: profile.Video, }, MicrophoneMixing: request.MicrophoneMixing && session.capabilities.Microphone, + ExcludeAudio: request.ExcludeAudio, Profile: profile, EdgeCapacity: request.EdgeCapacity, AudioEnabled: request.Audio && session.capabilities.Summary().AudioFor( diff --git a/internal/app/nativecontrol/session_test.go b/internal/app/nativecontrol/session_test.go index acd5d15..cad4b95 100644 --- a/internal/app/nativecontrol/session_test.go +++ b/internal/app/nativecontrol/session_test.go @@ -641,3 +641,32 @@ func TestReplaceShareSourceKeepsOneStrictTargetShape(t *testing.T) { t.Fatalf("malformed target was treated as an operational failure: %v", err) } } + +func TestAudioExclusionValidatesScopeWithoutDroppingRequestedPolicy(t *testing.T) { + session := &Session{capabilities: nativecapture.Capabilities{Microphone: true, SystemAudio: true, ProcessAudioExclusion: true}} + display := nativecapture.CaptureTarget{Kind: "display", SourceID: "1", Title: "Screen"} + window := nativecapture.CaptureTarget{Kind: "window", SourceID: "2", PID: 123, CreationTime: "456", Title: "Voice fixture"} + if err := session.validateAudioExclusion(display, true, &window); err != nil { + t.Fatal(err) + } + for _, invalid := range []struct { + source nativecapture.CaptureTarget + audio bool + excluded nativecapture.CaptureTarget + }{ + {display, false, window}, {window, true, window}, {display, true, display}, + {display, true, nativecapture.CaptureTarget{Kind: "window", SourceID: "2", PID: 123, Title: "Missing lifetime"}}, + } { + if _, ok := session.validateAudioExclusion(invalid.source, invalid.audio, &invalid.excluded).(protocolViolation); !ok { + t.Fatal("malformed audio exclusion was not rejected") + } + } + session.capabilities.ProcessAudioExclusion = false + err := session.validateAudioExclusion(display, true, &window) + if _, malformed := err.(protocolViolation); err == nil || malformed { + t.Fatal("unsupported audio exclusion must fail the operation, not silently widen capture or close control") + } + if err := session.validateAudioExclusion(display, true, nil); err != nil { + t.Fatal("legacy capture became dependent on exclusion support") + } +} diff --git a/internal/app/nativecontrol/wire.go b/internal/app/nativecontrol/wire.go index a68404d..b69ab1c 100644 --- a/internal/app/nativecontrol/wire.go +++ b/internal/app/nativecontrol/wire.go @@ -42,19 +42,20 @@ type qualitySettings struct { } type startShareRequest struct { - Version int `json:"version"` - ID string `json:"id"` - Type string `json:"type"` - ShareID string `json:"shareId"` - Source nativecapture.CaptureTarget `json:"source"` - Audio bool `json:"audio"` - MicrophoneMixing bool `json:"microphoneMixing,omitempty"` - ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"` - AdapterIndex uint32 `json:"adapterIndex"` - EncoderIndex uint32 `json:"encoderIndex"` - EdgeCapacity int `json:"edgeCapacity"` - Profile qualitySettings `json:"profile"` - Codec string `json:"codec"` + Version int `json:"version"` + ID string `json:"id"` + Type string `json:"type"` + ShareID string `json:"shareId"` + Source nativecapture.CaptureTarget `json:"source"` + Audio bool `json:"audio"` + MicrophoneMixing bool `json:"microphoneMixing,omitempty"` + ExcludeAudio *nativecapture.CaptureTarget `json:"excludeAudio,omitempty"` + ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"` + AdapterIndex uint32 `json:"adapterIndex"` + EncoderIndex uint32 `json:"encoderIndex"` + EdgeCapacity int `json:"edgeCapacity"` + Profile qualitySettings `json:"profile"` + Codec string `json:"codec"` } type microphoneRequest struct { @@ -81,15 +82,16 @@ type updateShareRequest struct { } type replaceShareSourceRequest struct { - Version int `json:"version"` - ID string `json:"id"` - Type string `json:"type"` - ShareID string `json:"shareId"` - Source nativecapture.CaptureTarget `json:"source"` - Audio bool `json:"audio"` - ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"` - AdapterIndex uint32 `json:"adapterIndex"` - EncoderIndex uint32 `json:"encoderIndex"` + Version int `json:"version"` + ID string `json:"id"` + Type string `json:"type"` + ShareID string `json:"shareId"` + Source nativecapture.CaptureTarget `json:"source"` + Audio bool `json:"audio"` + ExcludeAudio *nativecapture.CaptureTarget `json:"excludeAudio,omitempty"` + ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"` + AdapterIndex uint32 `json:"adapterIndex"` + EncoderIndex uint32 `json:"encoderIndex"` } type stopShareRequest struct { diff --git a/internal/app/nativehost/audio_mix_test.go b/internal/app/nativehost/audio_mix_test.go index 00302c5..bbfb7e3 100644 --- a/internal/app/nativehost/audio_mix_test.go +++ b/internal/app/nativehost/audio_mix_test.go @@ -88,6 +88,83 @@ func startMixedFixture(t *testing.T) (*Session, nativecapture.VideoOptions, <-ch return session, options, events } +func TestAudioExclusionFailureMutesOnlySourceAndNeverFallsBack(t *testing.T) { + session, options, events := startMixedFixture(t) + enabled := true + if err := session.SetMicrophone(&enabled, nil, nil); err != nil { + t.Fatal(err) + } + output := session.audioSource + session.mixer.mu.Lock() + microphone := session.mixer.microphone + session.mixer.mu.Unlock() + excluded := &nativecapture.CaptureTarget{Kind: "window", SourceID: "123", PID: 456, CreationTime: "789", Title: "Voice fixture"} + for _, fault := range []string{"PIIK_VIDEO_FAIL", "PIIK_EXCLUSION_FAIL"} { + if err := session.ReplaceSource(options, true, nil); err != nil { + t.Fatal(err) + } + video := session.stream + t.Setenv(fault, "1") + if err := session.ReplaceSource(options, true, excluded); err == nil { + t.Fatal("failed exclusion was accepted", fault) + } + if session.SourceAudio() == nil || *session.SourceAudio() || session.audioStream != nil || session.stream != video { + t.Fatal("failed exclusion retained unfiltered audio or retired video", fault) + } + t.Setenv(fault, "") + } + marker := filepath.Join(t.TempDir(), "excluded") + t.Setenv("PIIK_EXCLUSION_STARTED", marker) + if err := session.ReplaceSource(options, true, excluded); err != nil { + t.Fatal(err) + } + identity, err := os.ReadFile(marker) + if err != nil || string(identity) != "456:789" || !*session.SourceAudio() { + t.Fatal("replacement did not bind exclusion to the selected process lifetime", err) + } + sourceAudio := session.audioStream + profile := session.profile + profile.AudioBitrate = 128000 + if err := session.UpdateProfile(profile); err != nil { + t.Fatal(err) + } + session.SetPaused(true) + session.SetPaused(false) + if session.audioStream != sourceAudio { + t.Fatal("profile/pause replaced exclusion") + } + for len(events) > 0 { + <-events + } + if err := sourceAudio.Close(); err != nil { + t.Fatal(err) + } + timer := time.NewTimer(time.Second) + defer timer.Stop() + waiting := true + for waiting { + select { + case event := <-events: + if event.Type != "audio-state" { + continue + } + if event.SourceAudio || !event.Microphone || event.Failed { + t.Fatalf("source loss was misreported: %+v", event) + } + waiting = false + case <-timer.C: + t.Fatal("source loss was not reported") + } + } + session.mixer.mu.Lock() + sameMicrophone := session.mixer.microphone == microphone + sourceEnded := session.mixer.source == nil + session.mixer.mu.Unlock() + if !sourceEnded || !sameMicrophone || session.audioSource != output || !session.HasAudio() { + t.Fatal("exclusion loss disturbed microphone or output ownership") + } +} + func TestNativeMicrophonePreservesMediaOwnersAcrossInputChanges(t *testing.T) { session, options, events := startMixedFixture(t) check := func(err error) { @@ -130,7 +207,7 @@ func TestNativeMicrophonePreservesMediaOwnersAcrossInputChanges(t *testing.T) { } microphone = replaced options.Target.SourceID = "2" - check(session.ReplaceSource(options, true)) + check(session.ReplaceSource(options, true, nil)) session.mixer.mu.Lock() sameMicrophone := session.mixer.microphone == microphone session.mixer.mu.Unlock() diff --git a/internal/app/nativehost/audio_test.go b/internal/app/nativehost/audio_test.go index 0018fab..f2fc435 100644 --- a/internal/app/nativehost/audio_test.go +++ b/internal/app/nativehost/audio_test.go @@ -56,6 +56,17 @@ func runAudioRecoveryCapture() { } } audio := microphone || (len(os.Args) > 1 && os.Args[1] == "--capture-audio") + if !audio && os.Getenv("PIIK_VIDEO_FAIL") == "1" { + os.Exit(2) + } + if !microphone && audio && os.Args[2] == "exclude" { + if os.Getenv("PIIK_EXCLUSION_FAIL") == "1" { + os.Exit(2) + } + if marker := os.Getenv("PIIK_EXCLUSION_STARTED"); marker != "" { + _ = os.WriteFile(marker, []byte(os.Args[3]+":"+os.Args[4]), 0600) + } + } period := time.Second / 30 status := []byte(`{"state":"active","audio":true}`) if audio { @@ -179,7 +190,7 @@ func TestAudioEOFWaitsForExplicitSourceReplacement(t *testing.T) { waitPackets(&videoPackets, videoPackets.Load()+10, "video after audio EOF") before := audioPackets.Load() options.Target.SourceID = "2" - check(session.ReplaceSource(options, true)) + check(session.ReplaceSource(options, true, nil)) if session.currentStream() == previousVideo || session.currentAudioStream() == previousAudio || session.edge("audio-edge") != edge || session.audioSource != audioSource { t.Fatal("explicit replacement did not preserve the media owners") diff --git a/internal/app/nativehost/session.go b/internal/app/nativehost/session.go index 6d528d7..174b284 100644 --- a/internal/app/nativehost/session.go +++ b/internal/app/nativehost/session.go @@ -67,6 +67,7 @@ type Options struct { Video nativecapture.VideoOptions Profile QualityProfile AudioEnabled bool + ExcludeAudio *nativecapture.CaptureTarget MicrophoneMixing bool EdgeCapacity int BindAddress string @@ -138,6 +139,7 @@ func Start(parent context.Context, options Options) (*Session, error) { parent, options.CaptureProcess, options.Video.Target, + options.ExcludeAudio, ) if audioErr != nil { slog.DebugContext(parent, "piik-client", "event", "capture-audio-unavailable", "share", diagnostics.ID(options.ShareID), diagnostics.Error(audioErr)) @@ -401,6 +403,7 @@ func (session *Session) UpdateProfile(profile QualityProfile) error { func (session *Session) ReplaceSource( options nativecapture.VideoOptions, audioEnabled bool, + excludeAudio *nativecapture.CaptureTarget, ) error { session.updateMu.Lock() defer session.updateMu.Unlock() @@ -420,6 +423,21 @@ func (session *Session) ReplaceSource( if session.mixer == nil && audioEnabled != hasAudio { return errors.New("native source audio availability cannot change while sharing") } + if excludeAudio != nil { + if session.mixer == nil { + return errors.New("native audio exclusion requires mixed audio") + } + // A privacy change retires the old source before preparation. Failure must + // leave it silent rather than retaining unfiltered audio; video/mic stay live. + session.mu.Lock() + previousAudio := session.audioStream + session.audioStream = nil + session.mu.Unlock() + session.mixer.setSource(nil) + if previousAudio != nil { + _ = previousAudio.Close() + } + } options.Profile = profile.Video options.RestoreToken = "" replacement, state, err := session.prepareVideo(session.ctx, options, options.Target.Kind == "picker", false) @@ -432,6 +450,7 @@ func (session *Session) ReplaceSource( session.ctx, session.captureProcess, options.Target, + excludeAudio, ) if err != nil { _ = replacement.Close() @@ -452,11 +471,15 @@ func startAudioCapture( ctx context.Context, captureProcess string, target nativecapture.CaptureTarget, + excludeAudio *nativecapture.CaptureTarget, ) (*nativecapture.Stream, error) { + if excludeAudio != nil { + return nativecapture.StartAudio(ctx, captureProcess, *excludeAudio, true) + } if target.Kind == "display" || target.Kind == "picker" { return nativecapture.StartSystemAudio(ctx, captureProcess) } - return nativecapture.StartAudio(ctx, captureProcess, target) + return nativecapture.StartAudio(ctx, captureProcess, target, false) } func (session *Session) prepareVideo( @@ -560,7 +583,7 @@ func (session *Session) installCapture(next *nativecapture.Stream) error { // Preparation has consumed these events. Apply them to the new generation // before acknowledging it, just as the live reader handles unavailable layers. for _, layer := range unavailable { - if err := session.source.DisableLayer(layer); err != nil { + if err := session.disableCaptureLayer(next, layer); err != nil { return err } } @@ -575,6 +598,22 @@ func (session *Session) installCapture(next *nativecapture.Stream) error { return nil } +func (session *Session) disableCaptureLayer(stream *nativecapture.Stream, layer int) error { + session.mu.Lock() + if session.stream != stream || session.closed || session.ctx.Err() != nil { + session.mu.Unlock() + return nil + } + // Accept failure atomically with capture selection. Closing transports may + // wait for callbacks, so only the source-state change belongs under this lock. + retire, err := session.source.MarkLayerUnavailable(layer) + session.mu.Unlock() + if retire != nil { + retire() + } + return err +} + func (session *Session) CloseEdge(connectionID string) { session.mu.Lock() edge := session.edges[connectionID] @@ -831,7 +870,7 @@ func (session *Session) runVideo() error { if session.source == nil { return fail(errors.New("native output ended before its source state")) } - if err = session.source.DisableLayer(frame.Layer); err != nil { + if err = session.disableCaptureLayer(current, frame.Layer); err != nil { return fail(err) } case nativecapture.FramePCM: diff --git a/internal/app/nativehost/session_test.go b/internal/app/nativehost/session_test.go index a581b55..8f8003c 100644 --- a/internal/app/nativehost/session_test.go +++ b/internal/app/nativehost/session_test.go @@ -11,6 +11,7 @@ import ( "slices" "strconv" "strings" + "sync" "testing" "time" @@ -189,6 +190,122 @@ func TestCaptureCommitWaitsForReaderMetadataOrTermination(t *testing.T) { } } +func TestCaptureFailureBelongsToSelectedStream(t *testing.T) { + for _, replace := range []bool{false, true} { + t.Run(strconv.FormatBool(replace), func(t *testing.T) { + check := func(err error) { + t.Helper() + if err != nil { + t.Fatal(err) + } + } + t.Setenv("PIIK_NATIVEHOST_PIPE_FIXTURE", "backend") + t.Setenv("PIIK_CAPTURE_STARTING", string(captureStatePayload(t, + `{"state":"starting","codec":"vp8","adapterIndex":0,"adapterName":"Fixture","adapterIdentity":"0:0","encoderName":"VP8","encoderIdentity":"vp8"}`))) + t.Setenv("PIIK_CAPTURE_ACTIVE", string(captureStatePayload(t, `{"state":"active","codec":"vp8","width":1280,"height":720,"fps":30}`))) + t.Setenv("PIIK_CAPTURE_UNAVAILABLE", "0") + executable, err := os.Executable() + check(err) + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + options := nativecapture.VideoOptions{Target: nativecapture.CaptureTarget{Kind: "display", SourceID: "1", Title: "Fixture"}, Codec: "vp8", + Profile: nativecapture.VideoProfile{Width: 1280, Height: 720, Framerate: 30, Bitrate: 3_000_000, Preference: "balanced"}, OutputGroups: 4} + previous, err := nativecapture.StartVideo(ctx, executable, options) + check(err) + defer previous.Close() + t.Setenv("PIIK_CAPTURE_UNAVAILABLE", "") + replacement, err := nativecapture.StartVideo(ctx, executable, options) + check(err) + defer replacement.Close() + engine, err := mediaedge.NewEngine(mediaedge.EngineOptions{BindAddress: "127.0.0.1:0", IncludeLoopback: true}) + check(err) + defer engine.Close() + source, err := engine.NewSource("vp8", 4, 2, nil) + check(err) + defer source.Close() + check(configureCaptureOutputs(source, previous.Outputs())) + publication, err := engine.NewPublication(source, mediaedge.EdgeOptions{ConnectionID: "capture-failure-owner"}) + check(err) + active := make(chan struct{}, 1) + session := &Session{ctx: ctx, cancel: cancel, engine: engine, source: source, stream: previous, + videoOptions: options, ready: make(chan error, 1), events: func(_ context.Context, event Event) { + if event.State == "active" { + select { + case active <- struct{}{}: + default: + } + } + }} + // Delay the actual reader after its first identity check, without a + // production hook. Log I/O and ordinary scheduling can delay it here. + entered, release := make(chan struct{}), make(chan struct{}) + unblock := sync.OnceFunc(func() { close(release) }) + previousLog := slog.Default() + defer slog.SetDefault(previousLog) + slog.SetDefault(slog.New(slog.NewTextHandler(io.Discard, &slog.HandlerOptions{ + Level: slog.LevelDebug, ReplaceAttr: func(_ []string, attr slog.Attr) slog.Attr { + if attr.Key == "event" && attr.Value.String() == "capture-output-unavailable" { + close(entered) + <-release + } + return attr + }, + }))) + readerDone := make(chan error, 1) + go func() { readerDone <- session.runVideo() }() + defer func() { + unblock() + cancel() + _ = previous.Close() + _ = replacement.Close() + select { + case <-readerDone: + case <-time.After(5 * time.Second): + t.Error("capture reader did not retire") + } + }() + select { + case <-entered: + case err = <-readerDone: + readerDone <- err + t.Fatalf("reader stopped before output failure: %v", err) + case <-time.After(5 * time.Second): + t.Fatal("old output failure did not reach the reader") + } + committed := make(chan error, 1) + if replace { + go func() { + committed <- session.commitCapture(options, QualityProfile{Video: options.Profile}, replacement, CaptureState{}, nil, false) + }() + deadline := time.Now().Add(5 * time.Second) + for session.currentStream() != replacement { + if time.Now().After(deadline) { + t.Fatal("replacement was not selected") + } + time.Sleep(time.Millisecond) + } + } + unblock() + select { + case <-active: + case <-time.After(5 * time.Second): + t.Fatal("capture reader did not finish the failure or handoff") + } + if replace { + select { + case err = <-committed: + check(err) + case <-time.After(5 * time.Second): + t.Fatal("replacement installation was not acknowledged") + } + } + if err = publication.SetActiveCount(1); (err == nil) != replace { + t.Fatalf("publication availability after replacement=%v: %v", replace, err) + } + }) + } +} + func TestCaptureProfileRetainsSelectedBackendFromStarting(t *testing.T) { t.Setenv("PIIK_NATIVEHOST_PIPE_FIXTURE", "backend") t.Setenv("PIIK_CAPTURE_STARTING", string(captureStatePayload(t, diff --git a/internal/server/app/app.go b/internal/server/app/app.go index acf53d1..f58605b 100644 --- a/internal/server/app/app.go +++ b/internal/server/app/app.go @@ -154,6 +154,7 @@ func New(options Options) (*Server, error) { server.signalOptions = signal.Options{ Store: store, EndpointMediaCopyCapacity: configuration.EndpointMediaCopyCapacity, + SfuOnly: configuration.SFU != nil && configuration.SFU.Only, Ice: config.IceConfig(configuration), // The TS signaling server read natPredictionEnabled off the same // `ice` options object it forwarded; the Go IceConfig is already the diff --git a/internal/server/app/app_test.go b/internal/server/app/app_test.go index 8134f50..c0aee53 100644 --- a/internal/server/app/app_test.go +++ b/internal/server/app/app_test.go @@ -366,7 +366,17 @@ func TestCapabilitiesReportServicesDisabledByDefault(t *testing.T) { server := start(t, Options{Config: testConfig(t)}) server.do(http.MethodGet, "/api/capabilities"). - expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":false}`) + expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":false,"roomInteractions":true,"hostRoomSession":true}`) + +} + +func TestCapabilitiesReportSFUOnlyPolicy(t *testing.T) { + configuration := testConfig(t) + configuration.SFU = &config.SFUConfig{ListenHost: "127.0.0.1", Port: 0, Only: true} + configuration.NATPredictionEnabled = true + server := start(t, Options{Config: configuration}) + server.do(http.MethodGet, "/api/capabilities"). + expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":true,"sfuOnly":true,"natPrediction":false,"roomInteractions":true,"hostRoomSession":true}`) } func TestCapabilitiesReportOptionalNATPredictionWithoutExposingConfiguration(t *testing.T) { @@ -377,7 +387,7 @@ func TestCapabilitiesReportOptionalNATPredictionWithoutExposingConfiguration(t * server := start(t, Options{Config: configuration}) server.do(http.MethodGet, "/api/capabilities"). - expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":true}`). + expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":true,"roomInteractions":true,"hostRoomSession":true}`). expectHeader("Cache-Control", "no-store"). expectHeader("X-Content-Type-Options", "nosniff") @@ -420,7 +430,8 @@ func TestSFUListenerFollowsApplicationCloseAndEnd(t *testing.T) { t.Fatal("missing runtime owner") } server.do(http.MethodGet, "/api/capabilities"). - expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":true,"natPrediction":false}`) + expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":true,"natPrediction":false,"roomInteractions":true,"hostRoomSession":true}`) + var err error if ending { err = server.End(context.Background()) diff --git a/internal/server/app/routes.go b/internal/server/app/routes.go index 7a35806..278f518 100644 --- a/internal/server/app/routes.go +++ b/internal/server/app/routes.go @@ -80,7 +80,10 @@ func (s *Server) route( sendJSON(writer, http.StatusOK, protocol.RuntimeCapabilities{ ConnectionAttemptProgress4: true, Sfu: s.config.SFU != nil, - NatPrediction: s.config.NATPredictionEnabled, + SfuOnly: s.config.SFU != nil && s.config.SFU.Only, + NatPrediction: s.config.NATPredictionEnabled && !(s.config.SFU != nil && s.config.SFU.Only), + RoomInteractions: true, + HostRoomSession: true, }) return } diff --git a/internal/server/config/config.go b/internal/server/config/config.go index 481c0da..2a5bcd0 100644 --- a/internal/server/config/config.go +++ b/internal/server/config/config.go @@ -68,6 +68,7 @@ type SFUConfig struct { ListenHost string Port int PublicIP string + Only bool } // Config describes runtime configuration. Zero values mean open site @@ -273,8 +274,15 @@ func parseRoomDatabasePath(value string) (string, error) { } func parseSFU(env map[string]string) (*SFUConfig, error) { + only, err := parseBoolean(env["SFU_ONLY"], "SFU_ONLY") + if err != nil { + return nil, err + } rawPort := strings.TrimSpace(env["SFU_UDP_PORT"]) if rawPort == "" { + if only { + return nil, errors.New("SFU_ONLY requires SFU_UDP_PORT") + } return nil, nil } port, err := parseBoundedInteger(rawPort, 0, "SFU_UDP_PORT", 1, maxPort) @@ -298,6 +306,7 @@ func parseSFU(env map[string]string) (*SFUConfig, error) { ListenHost: listenHost, Port: int(port), PublicIP: publicIP, + Only: only, }, nil } diff --git a/internal/server/config/config_test.go b/internal/server/config/config_test.go index c6a5ed3..4d17164 100644 --- a/internal/server/config/config_test.go +++ b/internal/server/config/config_test.go @@ -187,6 +187,21 @@ func TestLoadSFUOptional(t *testing.T) { } } +func TestSFUOnlyRequiresAnEnabledListener(t *testing.T) { + configuration := mustLoad(t, map[string]string{"SFU_UDP_PORT": "7882", "SFU_ONLY": "true"}) + if !configuration.SFU.Only { + t.Fatal("SFU_ONLY was ignored") + } + for _, values := range []map[string]string{ + {"SFU_ONLY": "true"}, + {"SFU_ONLY": "yes", "SFU_UDP_PORT": "7882"}, + } { + if _, err := Load(values); err == nil { + t.Fatalf("accepted invalid SFU policy: %v", values) + } + } +} + func TestLoadAccepts(t *testing.T) { developmentPath, err := filepath.Abs(filepath.Join("state", "rooms.sqlite")) if err != nil { diff --git a/internal/server/protocol/client.go b/internal/server/protocol/client.go index 8bc3e54..ae318ab 100644 --- a/internal/server/protocol/client.go +++ b/internal/server/protocol/client.go @@ -26,6 +26,8 @@ type AuthenticateHostMessage struct { ViewerPresence bool `json:"viewerPresence,omitempty"` ConnectionAttemptProgress4 bool `json:"connectionAttemptProgress4,omitempty"` DisplayName *DisplayName `json:"displayName,omitempty"` + RoomOnly bool `json:"roomOnly,omitempty"` + RoomSession bool `json:"roomSession,omitempty"` } // AuthenticateViewerMessage is the viewer member of authenticateMessageSchema. @@ -172,6 +174,15 @@ type StopSharingMessage struct { ShareGeneration string `json:"shareGeneration,omitempty"` } +// StartSharingMessage begins a new media publication on an existing Host room session. +type StartSharingMessage struct { + Type string `json:"type"` + ShareGeneration string `json:"shareGeneration"` + SharingPaused bool `json:"sharingPaused,omitempty"` + QualitySettings *QualitySettings `json:"qualitySettings,omitempty"` + RoutePolicy RoutePolicy `json:"routePolicy"` +} + // AbandonRoomMessage is { type: "abandon-room" }. type AbandonRoomMessage struct { Type string `json:"type"` @@ -197,6 +208,7 @@ func (ResetSenderQualityMessage) isClientMessage() {} func (SetDisplayNameMessage) isClientMessage() {} func (SetSharingPausedMessage) isClientMessage() {} func (StopSharingMessage) isClientMessage() {} +func (StartSharingMessage) isClientMessage() {} func (AbandonRoomMessage) isClientMessage() {} // DecodeClientMessage parses JSON and validates the message schema. @@ -207,6 +219,10 @@ func DecodeClientMessage(data []byte) (ClientMessage, error) { return nil, err } switch messageType { + case "subscribe-room-interactions": + return decodeEmptyClientMessage(data, SubscribeRoomInteractionsMessage{Type: messageType}) + case "send-room-interaction": + return decodeSendRoomInteraction(data) case "authenticate": return decodeAuthenticate(data) case "signaling-challenge": @@ -247,6 +263,8 @@ func DecodeClientMessage(data []byte) (ClientMessage, error) { return decodeSetSharingPaused(data) case "stop-sharing": return decodeStopSharing(data) + case "start-sharing": + return decodeStartSharing(data) case "abandon-room": return decodeEmptyClientMessage(data, AbandonRoomMessage{Type: messageType}) } @@ -282,8 +300,9 @@ func decodeAuthenticate(data []byte) (ClientMessage, error) { if err := present.require("type", "protocol", "roomId", "role", "token", "clientId"); err != nil { return nil, err } - if err := present.optional("shareGeneration", "sharingPaused", "qualitySettings", + if err := present.optional("shareGeneration", "roomOnly", "roomSession", "sharingPaused", "qualitySettings", "routePolicy", "viewerPresence", "displayName", "connectionAttemptProgress4"); err != nil { + return nil, err } if message.Protocol != SignalingProtocol { @@ -301,6 +320,15 @@ func decodeAuthenticate(data []byte) (ClientMessage, error) { if present.has("shareGeneration") && !ValidOpaqueID(message.ShareGeneration) { return nil, errors.New("shareGeneration is not an opaque id") } + if present.has("roomOnly") && !message.RoomOnly { + return nil, errors.New("roomOnly must be true when present") + } + if present.has("roomSession") && !message.RoomSession { + return nil, errors.New("roomSession must be true when present") + } + if message.RoomOnly && !message.RoomSession { + return nil, errors.New("roomOnly requires roomSession") + } if present.has("viewerPresence") && !message.ViewerPresence { return nil, errors.New("viewerPresence must be true when present") } @@ -705,3 +733,21 @@ func decodeStopSharing(data []byte) (ClientMessage, error) { } return message, nil } + +func decodeStartSharing(data []byte) (ClientMessage, error) { + var message StartSharingMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err := present.require("type", "shareGeneration", "routePolicy"); err != nil { + return nil, err + } + if err := present.optional("sharingPaused", "qualitySettings"); err != nil { + return nil, err + } + if !ValidOpaqueID(message.ShareGeneration) { + return nil, errors.New("shareGeneration is not an opaque id") + } + return message, nil +} diff --git a/internal/server/protocol/contract_test.go b/internal/server/protocol/contract_test.go index 4931b9e..a3d74d3 100644 --- a/internal/server/protocol/contract_test.go +++ b/internal/server/protocol/contract_test.go @@ -16,6 +16,7 @@ func TestRuntimeCapabilities(t *testing.T) { {`{}`, RuntimeCapabilities{}}, {`{"connectionAttemptProgress4":true}`, RuntimeCapabilities{ConnectionAttemptProgress4: true}}, {`{"sfu":true,"extra":{"enabled":true}}`, RuntimeCapabilities{Sfu: true}}, + {`{"sfu":true,"sfuOnly":true}`, RuntimeCapabilities{Sfu: true, SfuOnly: true}}, {`{"natPrediction":true,"SFU":true}`, RuntimeCapabilities{NatPrediction: true}}, {`{"sfu":true,"natPrediction":true}`, RuntimeCapabilities{Sfu: true, NatPrediction: true}}, {`{"sfu":"ignored duplicate","sfu":true}`, RuntimeCapabilities{Sfu: true}}, @@ -26,7 +27,7 @@ func TestRuntimeCapabilities(t *testing.T) { } } for _, data := range []string{ - `null`, `[]`, `true`, `{"sfu":null}`, `{"sfu":"true"}`, + `null`, `[]`, `true`, `{"sfu":null}`, `{"sfu":"true"}`, `{"sfuOnly":null}`, `{"sfuOnly":"true"}`, `{"natPrediction":null}`, `{"natPrediction":1}`, `{"connectionAttemptProgress4":null}`, `{"connectionAttemptProgress4":4}`, } { @@ -698,6 +699,44 @@ func TestAuthenticateAppliesTheRoutePolicyDefault(t *testing.T) { } } +func TestAuthenticateAcceptsPersistentRoomSessionOptIn(t *testing.T) { + message, err := DecodeClientMessage([]byte( + `{"type":"authenticate","protocol":"piik-v23","roomId":"1234","role":"host",` + + `"token":"` + repeat("a", 43) + `","clientId":"client_12345678","roomSession":true,"roomOnly":true}`)) + if err != nil { + t.Fatal(err) + } + host, ok := message.(AuthenticateHostMessage) + if !ok || !host.RoomSession || !host.RoomOnly { + t.Fatalf("decoded %#v, want room-session opt-in and room-only state", message) + } + if _, err := DecodeClientMessage([]byte( + `{"type":"authenticate","protocol":"piik-v23","roomId":"1234","role":"host",` + + `"token":"` + repeat("a", 43) + `","clientId":"client_12345678","roomOnly":true}`)); err == nil { + t.Fatal("accepted roomOnly without roomSession opt-in") + } +} + +func TestStartSharingRequiresAnOpaqueGenerationAndRoutePolicy(t *testing.T) { + message, err := DecodeClientMessage([]byte( + `{"type":"start-sharing","shareGeneration":"share_12345678",` + + `"routePolicy":{"peerOnly":true,"topologyOptimization":false,"natPrediction":false}}`)) + if err != nil { + t.Fatal(err) + } + if _, ok := message.(StartSharingMessage); !ok { + t.Fatalf("decoded %#v, want StartSharingMessage", message) + } + for _, raw := range []string{ + `{"type":"start-sharing","shareGeneration":"short","routePolicy":{"peerOnly":true,"topologyOptimization":false,"natPrediction":false}}`, + `{"type":"start-sharing","shareGeneration":"share_12345678"}`, + } { + if _, err := DecodeClientMessage([]byte(raw)); err == nil { + t.Errorf("accepted invalid start-sharing message %s", raw) + } + } +} + // The sealed unions accept nothing outside their own message structs. func TestDecodeRejectsUnknownEnvelopes(t *testing.T) { for _, raw := range []string{`{}`, `[]`, `null`, `"authenticate"`, `{"type":42}`, diff --git a/internal/server/protocol/fixture_test.go b/internal/server/protocol/fixture_test.go index f02d054..7171630 100644 --- a/internal/server/protocol/fixture_test.go +++ b/internal/server/protocol/fixture_test.go @@ -55,6 +55,8 @@ func TestFixtureConstants(t *testing.T) { "viewerQualityEvidenceExpiryMs": float64(ViewerQualityEvidenceExpiryMs), "persistentNativeEdgeDegradedWindows": float64(PersistentNativeEdgeDegradedWindows), "maxDisplayNameCodePoints": float64(MaxDisplayNameCodePoints), + "maxChatCodePoints": float64(MaxChatCodePoints), + "interactionIntervalMs": float64(InteractionIntervalMs), "defaultViewerDisplayName": DefaultViewerDisplayName, "defaultHostDisplayNamePrefix": DefaultHostDisplayNamePrefix, "minViewerPasswordLength": float64(MinViewerPasswordLength), diff --git a/internal/server/protocol/interactions.go b/internal/server/protocol/interactions.go new file mode 100644 index 0000000..51f987f --- /dev/null +++ b/internal/server/protocol/interactions.go @@ -0,0 +1,194 @@ +package protocol + +import ( + "errors" + "strings" + "unicode/utf8" + + "golang.org/x/text/unicode/norm" +) + +const MaxChatCodePoints = 280 +const InteractionIntervalMs = 800 + +type InteractionPayload struct { + Kind string `json:"kind"` + Text string `json:"text,omitempty"` + Reaction string `json:"reaction,omitempty"` + TargetPeerID string `json:"targetPeerId,omitempty"` +} + +func (payload *InteractionPayload) UnmarshalJSON(data []byte) error { + type plain InteractionPayload + var value plain + fields, err := decodeObject(data, &value) + if err != nil { + return err + } + if err = fields.require("kind"); err != nil { + return err + } + switch value.Kind { + case "chat": + if err = fields.require("text"); err != nil { + return err + } + if fields.has("reaction") || fields.has("targetPeerId") || + hasUnpairedSurrogateEscape(fields["text"]) || !validChatText(value.Text) { + return errors.New("invalid chat payload") + } + case "reaction": + if err = fields.require("reaction"); err != nil { + return err + } + if err = fields.optional("targetPeerId"); err != nil { + return err + } + if fields.has("text") || (fields.has("targetPeerId") && !ValidOpaqueID(value.TargetPeerID)) { + return errors.New("invalid reaction payload") + } + switch value.Reaction { + case "tomato", "poop": + if !ValidOpaqueID(value.TargetPeerID) { + return errors.New("throwing a prop requires a participant") + } + case "wave", "heart", "clap", "laugh", "wow", "party", "fire", "eyes", "star", "sleep": + default: + return errors.New("unknown reaction") + } + default: + return errors.New("unknown interaction kind") + } + *payload = InteractionPayload(value) + return nil +} + +func validChatText(value string) bool { + return utf8.ValidString(value) && utf8.RuneCountInString(value) >= 1 && + utf8.RuneCountInString(value) <= MaxChatCodePoints && !forbiddenDisplayNameCharacters.MatchString(value) && + norm.NFC.IsNormalString(value) && strings.TrimFunc(value, IsJSWhitespace) == value +} + +type SubscribeRoomInteractionsMessage struct { + Type string `json:"type"` +} +type SendRoomInteractionMessage struct { + Type string `json:"type"` + RequestID string `json:"requestId"` + Payload InteractionPayload `json:"payload"` +} + +func (SubscribeRoomInteractionsMessage) isClientMessage() {} +func (SendRoomInteractionMessage) isClientMessage() {} + +func decodeSendRoomInteraction(data []byte) (ClientMessage, error) { + var message SendRoomInteractionMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err = present.require("type", "requestId", "payload"); err != nil { + return nil, err + } + if !ValidOpaqueID(message.RequestID) { + return nil, errors.New("invalid interaction requestId") + } + return message, nil +} + +type InteractionSender struct { + PeerID string `json:"peerId"` + Role Role `json:"role"` + DisplayName DisplayName `json:"displayName"` +} + +func (sender *InteractionSender) UnmarshalJSON(data []byte) error { + type plain InteractionSender + var value plain + present, err := decodeObject(data, &value) + if err != nil { + return err + } + if err = present.require("peerId", "role", "displayName"); err != nil { + return err + } + if !ValidOpaqueID(value.PeerID) { + return errors.New("invalid sender peerId") + } + *sender = InteractionSender(value) + return nil +} + +type RoomInteractionsReadyMessage struct { + Type string `json:"type"` + ServerTime Int `json:"serverTime"` +} +type RoomInteractionMessage struct { + Type string `json:"type"` + ID string `json:"id"` + RequestID string `json:"requestId"` + OccurredAt Int `json:"occurredAt"` + Sender InteractionSender `json:"sender"` + Payload InteractionPayload `json:"payload"` +} +type RoomInteractionRejectedMessage struct { + Type string `json:"type"` + RequestID string `json:"requestId"` + Reason string `json:"reason"` +} + +func (RoomInteractionsReadyMessage) isServerMessage() {} +func (RoomInteractionMessage) isServerMessage() {} +func (RoomInteractionRejectedMessage) isServerMessage() {} + +func decodeRoomInteractionsReady(data []byte) (ServerMessage, error) { + var message RoomInteractionsReadyMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err = present.require("type", "serverTime"); err != nil { + return nil, err + } + if !inRangeInt(message.ServerTime, 0, MaxSafeInteger) { + return nil, errors.New("invalid interaction server time") + } + return message, nil +} + +func decodeRoomInteraction(data []byte) (ServerMessage, error) { + var message RoomInteractionMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err = present.require("type", "id", "requestId", "occurredAt", "sender", "payload"); err != nil { + return nil, err + } + if !ValidOpaqueID(message.ID) || !ValidOpaqueID(message.RequestID) { + return nil, errors.New("invalid interaction identity") + } + if !inRangeInt(message.OccurredAt, 0, MaxSafeInteger) { + return nil, errors.New("invalid interaction time") + } + return message, nil +} +func decodeRoomInteractionRejected(data []byte) (ServerMessage, error) { + var message RoomInteractionRejectedMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err = present.require("type", "requestId", "reason"); err != nil { + return nil, err + } + if !ValidOpaqueID(message.RequestID) { + return nil, errors.New("invalid interaction requestId") + } + switch message.Reason { + case "rate-limited", "target-unavailable", "not-subscribed", "busy": + default: + return nil, errors.New("invalid rejection reason") + } + return message, nil +} diff --git a/internal/server/protocol/server.go b/internal/server/protocol/server.go index f25aa0a..3e1f9d1 100644 --- a/internal/server/protocol/server.go +++ b/internal/server/protocol/server.go @@ -57,6 +57,23 @@ type AuthenticatedViewerMessage struct { Role Role `json:"role"` } +// SharingStartedMessage acknowledges a new media publication on an existing Host session. +type SharingStartedMessage struct { + Type string `json:"type"` + ShareGeneration string `json:"shareGeneration"` + RoutePolicy RoutePolicy `json:"routePolicy"` + RouteRevision Int `json:"routeRevision"` + RouteAssignment ParticipantRouteAssignment `json:"routeAssignment"` + QualitySettings QualitySettings `json:"qualitySettings"` + Paused bool `json:"paused"` +} + +type SharingStartFailedMessage struct { + Type string `json:"type"` + ShareGeneration string `json:"shareGeneration"` + Code string `json:"code"` +} + // SignalingChallengeResponseMessage is the challenge echo. type SignalingChallengeResponseMessage struct { Type string `json:"type"` @@ -197,6 +214,8 @@ type ErrorMessage struct { func (AuthenticatedHostMessage) isServerMessage() {} func (AuthenticatedViewerMessage) isServerMessage() {} +func (SharingStartedMessage) isServerMessage() {} +func (SharingStartFailedMessage) isServerMessage() {} func (SignalingChallengeResponseMessage) isServerMessage() {} func (ServerSignalMessage) isServerMessage() {} func (ServerRestartRequestMessage) isServerMessage() {} @@ -230,8 +249,18 @@ func DecodeServerMessage(data []byte) (ServerMessage, error) { return nil, err } switch messageType { + case "room-interactions-ready": + return decodeRoomInteractionsReady(data) + case "room-interaction": + return decodeRoomInteraction(data) + case "room-interaction-rejected": + return decodeRoomInteractionRejected(data) case "authenticated": return decodeAuthenticated(data) + case "sharing-started": + return decodeSharingStarted(data) + case "sharing-start-failed": + return decodeSharingStartFailed(data) case "signaling-challenge-response": return decodeChallengeResponse(data) case "signal": @@ -386,6 +415,40 @@ func decodeAuthenticated(data []byte) (ServerMessage, error) { return nil, fmt.Errorf("unknown authenticated role %q", role) } +func decodeSharingStarted(data []byte) (ServerMessage, error) { + var message SharingStartedMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err := present.require("type", "shareGeneration", "routePolicy", "routeRevision", + "routeAssignment", "qualitySettings", "paused"); err != nil { + return nil, err + } + if !ValidOpaqueID(message.ShareGeneration) { + return nil, errors.New("shareGeneration is not an opaque id") + } + if !validRevision(message.RouteRevision) { + return nil, errors.New("routeRevision is out of range") + } + return message, nil +} + +func decodeSharingStartFailed(data []byte) (ServerMessage, error) { + var message SharingStartFailedMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err := present.require("type", "shareGeneration", "code"); err != nil { + return nil, err + } + if !ValidOpaqueID(message.ShareGeneration) || (message.Code != "FORBIDDEN" && message.Code != "SERVER_ERROR") { + return nil, errors.New("invalid sharing-start-failed response") + } + return message, nil +} + func decodeChallengeResponse(data []byte) (ServerMessage, error) { var message SignalingChallengeResponseMessage present, err := decodeObject(data, &message) diff --git a/internal/server/protocol/types.go b/internal/server/protocol/types.go index 4365aed..4bfd186 100644 --- a/internal/server/protocol/types.go +++ b/internal/server/protocol/types.go @@ -170,7 +170,10 @@ func (v *RoutePolicy) UnmarshalJSON(data []byte) error { type RuntimeCapabilities struct { ConnectionAttemptProgress4 bool `json:"connectionAttemptProgress4,omitempty"` Sfu bool `json:"sfu"` + SfuOnly bool `json:"sfuOnly,omitempty"` NatPrediction bool `json:"natPrediction"` + RoomInteractions bool `json:"roomInteractions,omitempty"` + HostRoomSession bool `json:"hostRoomSession,omitempty"` } // UnmarshalJSON implements json.Unmarshaler. @@ -183,12 +186,14 @@ func (v *RuntimeCapabilities) UnmarshalJSON(data []byte) error { if present == nil { return errors.New("runtime capabilities must be an object") } - if err := present.optional("sfu", "natPrediction", "connectionAttemptProgress4"); err != nil { + if err := present.optional("sfu", "sfuOnly", "roomInteractions", "hostRoomSession", "natPrediction", "connectionAttemptProgress4"); err != nil { + return err } for key, target := range map[string]*bool{ - "sfu": &v.Sfu, "natPrediction": &v.NatPrediction, + "sfu": &v.Sfu, "sfuOnly": &v.SfuOnly, "natPrediction": &v.NatPrediction, "connectionAttemptProgress4": &v.ConnectionAttemptProgress4, + "roomInteractions": &v.RoomInteractions, "hostRoomSession": &v.HostRoomSession, } { if value, ok := present[key]; ok { if err := json.Unmarshal(value, target); err != nil { diff --git a/internal/server/route/api.go b/internal/server/route/api.go index eaf052e..9c41d2d 100644 --- a/internal/server/route/api.go +++ b/internal/server/route/api.go @@ -455,6 +455,7 @@ type Options struct { EndpointMediaCopyCapacity int OperationTimeoutMs int64 SfuEnabled bool + SfuOnly bool QualityConvergenceEnabled bool NatPredictionEnabled bool } @@ -470,6 +471,7 @@ type Controller struct { endpointMediaCopyCapacity int operationTimeoutMs int64 sfuEnabled bool + sfuOnly bool qualityConvergenceEnabled bool natPredictionEnabled bool diff --git a/internal/server/route/candidates.go b/internal/server/route/candidates.go index 6b1cd35..931e3d8 100644 --- a/internal/server/route/candidates.go +++ b/internal/server/route/candidates.go @@ -70,7 +70,7 @@ func (c *Controller) buildCandidatePlans(childPeerID string, sfuOnly, preferUntr } var candidates []CandidateTuple switch { - case sfuOnly: + case sfuOnly || c.sfuOnly: if sfuCandidate != nil { candidates = []CandidateTuple{*sfuCandidate} } @@ -166,6 +166,9 @@ func (c *Controller) candidateValid(childPeerID string, plan CandidatePlan, att return false } if tuple.Kind == UpstreamPeer { + if c.sfuOnly { + return false + } parent, _ := c.participants.Get(tuple.ParentPeerID) if parent == nil || parent.sessionID == "" || parent.departureConfirmed || (att != nil && parent.sessionID != att.parentSessionID) || slices.Contains(c.descendantsOf(childPeerID), parent.peerID) || !c.sourceUsable(parent.peerID) || diff --git a/internal/server/route/controller.go b/internal/server/route/controller.go index bde5567..6463d73 100644 --- a/internal/server/route/controller.go +++ b/internal/server/route/controller.go @@ -271,6 +271,9 @@ func New(options Options) *Controller { if !isSafeInteger(options.OperationTimeoutMs) || options.OperationTimeoutMs <= 0 { panic(errors.New("Route operation timeout must be a positive integer")) } + if options.SfuOnly && !options.SfuEnabled { + panic(errors.New("SFU-only routing requires SFU availability")) + } c := &Controller{ hostPeerID: options.HostPeerID, debugRoomID: options.DebugRoomID, @@ -278,8 +281,9 @@ func New(options Options) *Controller { endpointMediaCopyCapacity: options.EndpointMediaCopyCapacity, operationTimeoutMs: options.OperationTimeoutMs, sfuEnabled: options.SfuEnabled, - qualityConvergenceEnabled: options.QualityConvergenceEnabled, - natPredictionEnabled: options.NatPredictionEnabled, + sfuOnly: options.SfuOnly, + qualityConvergenceEnabled: options.QualityConvergenceEnabled && !options.SfuOnly, + natPredictionEnabled: options.NatPredictionEnabled && !options.SfuOnly, routeTimings: make(map[string]*routeTimingRecord), qualityObservations: make(map[string]*routeQualityObservation), senderQualityObservations: make(map[string]*senderQualityObservation), @@ -291,6 +295,7 @@ func New(options Options) *Controller { "endpointCapacity", options.EndpointMediaCopyCapacity, "operationTimeoutMs", options.OperationTimeoutMs, "sfuEnabled", options.SfuEnabled, + "sfuOnly", options.SfuOnly, "qualityConvergenceEnabled", options.QualityConvergenceEnabled) return c } diff --git a/internal/server/route/sfu_only_test.go b/internal/server/route/sfu_only_test.go new file mode 100644 index 0000000..aafe397 --- /dev/null +++ b/internal/server/route/sfu_only_test.go @@ -0,0 +1,64 @@ +package route + +import "testing" + +func TestSFUOnlyUsesOnePublicationWithoutPeerConvergence(t *testing.T) { + routes := newController(1, Options{SfuEnabled: true, SfuOnly: true, + QualityConvergenceEnabled: true, NatPredictionEnabled: true}) + for index, viewer := range []string{A, B, C} { + now := int64(index * 10) + addViewer(routes, viewer, 1, &now) + op := must(t, routes.Reconcile(now).Operation) + eq(t, len(op.Candidates), 1) + publication := PublicationReuse + input := BeginInput{NowMs: now + 1, ConnectionID: viewer + "_sfu", + Reservation: sfuReuse(viewer)} + if index == 0 { + publication = PublicationCreate + input.PublicationGeneration = "publication" + input.PublicationConnectionID = "publication_connection" + input.Reservation = sfuCreate(viewer, "publication_resource") + } + eq(t, op.Candidates[0].Tuple, sfuTuple(publication)) + prepared := beginOperation(t, routes, input) + // Availability still commits only on the exact child's media-ready proof. + eq(t, hasEdge(routes, viewer), false) + eq(t, routes.CandidateReady(guardFor(viewer, prepared.Current.Revision, input.ConnectionID), + now+2, nil, CandidateProof{}).Accepted, true) + eq(t, edgeOf(t, routes, viewer).Kind, UpstreamSfu) + noOperation(t, routes.Reconcile(now+3).Operation) + } + eq(t, routes.physicalCopies(HOST), 1) + eq(t, routes.directContinuations.Len(), 0) + // A failed publication is replaced through the same bounded operation. + publication := routes.Publication() + eq(t, routes.InvalidateHostPublication(PublicationGuard{HostSessionID: "host_session", + RouteRevision: routes.Revision(), Generation: publication.Generation, + ConnectionID: publication.ConnectionID}, ms(40)), true) + op := must(t, routes.Reconcile(41).Operation) + eq(t, len(op.Candidates), 1) + eq(t, op.Candidates[0].Tuple, sfuTuple(PublicationReplace)) +} + +func TestSFUOnlyFailureDoesNotEscapeToPeers(t *testing.T) { + for _, admissionDenied := range []bool{false, true} { + routes := newController(2, Options{SfuEnabled: true, SfuOnly: true}) + addViewer(routes, A, 2, nil) + op := must(t, routes.Reconcile(0).Operation) + if admissionDenied { + routes.SkipCurrentCandidate(cursorGuard(op), 1, RejectionSfuAdmission) + } else { + prepared := beginOperation(t, routes, BeginInput{NowMs: 1, + ConnectionID: "sfu_connection", PublicationGeneration: "publication", + PublicationConnectionID: "publication_connection", Reservation: sfuCreate("edge", "publication")}) + routes.CandidateFailed(guardFor(A, prepared.Current.Revision, "sfu_connection"), 2) + } + noOperation(t, routes.Reconcile(3).Operation) + eq(t, hasEdge(routes, A), false) + // Resource release may retry SFU; it must not reopen a Peer opportunity. + routes.TouchExternalFacts() + retry := must(t, routes.Reconcile(4).Operation) + eq(t, len(retry.Candidates), 1) + eq(t, retry.Candidates[0].Tuple.Kind, UpstreamSfu) + } +} diff --git a/internal/server/signal/interactions.go b/internal/server/signal/interactions.go new file mode 100644 index 0000000..f3aac14 --- /dev/null +++ b/internal/server/signal/interactions.go @@ -0,0 +1,62 @@ +package signal + +import "github.com/TNTcraftHIM/Piik/internal/server/protocol" + +// Room authority remains in the authenticated session/store. Interaction state +// ends with that connection; no message history, routing state or retry worker +// lives on the server. Server.mu serializes membership checks and broadcast. +func (s *Server) handleRoomInteraction(sess *session, actor *authenticatedSession, message protocol.SendRoomInteractionMessage) { + reject := func(reason string) { + s.send(sess, protocol.RoomInteractionRejectedMessage{ + Type: "room-interaction-rejected", RequestID: message.RequestID, Reason: reason, + }) + } + if !sess.roomInteractions || actor.displayName == nil { + reject("not-subscribed") + return + } + now := s.now() + if now < sess.nextInteractionAtMs { + reject("rate-limited") + return + } + sess.nextInteractionAtMs = now + protocol.InteractionIntervalMs + if targetID := message.Payload.TargetPeerID; targetID != "" { + target, exists := s.connectedPeer(actor.roomID, targetID) + targetSession := s.sessionsByID[target.SessionID] + if !exists || targetSession == nil || targetSession.revoked || !targetSession.open() || !s.isCurrentSession(targetSession) { + reject("target-unavailable") + return + } + } + // Reserve the existing signaling queue for media/authority work when a + // reader stalls. Room data must not fill it and terminate a healthy share. + const maxInteractionQueueBytes = 16 * 1024 + if sess.queuedBytes >= maxInteractionQueueBytes { + reject("busy") + return + } + event := protocol.RoomInteractionMessage{ + Type: "room-interaction", ID: opaqueID(), RequestID: message.RequestID, + OccurredAt: protocol.Int(now), + Sender: protocol.InteractionSender{PeerID: actor.peerID, Role: actor.role, DisplayName: protocol.DisplayName(*actor.displayName)}, + Payload: message.Payload, + } + encoded, err := protocol.EncodeServerMessage(event) + if err != nil { + panic(err) + } + deliver := func(sessionID string) { + target := s.sessionsByID[sessionID] + if target != nil && target.roomInteractions && !target.revoked && target.open() && + s.isCurrentSession(target) && target.queuedBytes < maxInteractionQueueBytes { + s.sendEncoded(target, encoded) + } + } + if host, ok := s.store.GetConnectedHost(actor.roomID); ok { + deliver(host.SessionID) + } + for _, viewer := range s.store.GetConnectedViewers(actor.roomID) { + deliver(viewer.SessionID) + } +} diff --git a/internal/server/signal/interactions_test.go b/internal/server/signal/interactions_test.go new file mode 100644 index 0000000..48832f9 --- /dev/null +++ b/internal/server/signal/interactions_test.go @@ -0,0 +1,144 @@ +package signal + +import ( + "sync/atomic" + "testing" + "time" + + "github.com/TNTcraftHIM/Piik/internal/server/protocol" +) + +func subscribeInteractions(t *testing.T, client *testClient) { + t.Helper() + client.sendJSON(map[string]any{"type": "subscribe-room-interactions"}) + ready := client.next("room-interactions-ready") + if _, err := protocol.DecodeServerMessage(ready.raw); err != nil { + t.Fatal(err) + } +} + +func TestRoomInteractionsStayInsideAuthorizedOptedInMembership(t *testing.T) { + var now atomic.Int64 + now.Store(100_000) + h := startHarness(t, harnessOptions{maxViewersPerRoom: 4, now: now.Load}) + host, viewer, old := openClient(t, h), openClient(t, h), openClient(t, h) + authenticate(t, host, h.room, protocol.RoleHost, "host_interaction", -1, "", presenceOptions{displayName: "房主", viewerPresence: true}) + actor := authenticate(t, viewer, h.room, protocol.RoleViewer, "viewer_interaction", -1, "", presenceOptions{displayName: "小明", viewerPresence: true}) + authenticate(t, old, h.room, protocol.RoleViewer, "old_page_interaction", -1, "", presenceOptions{}) + otherRoom, err := h.server.CreateRoom(protocol.CodeEntryOpen, nil, "") + if err != nil { + t.Fatal(err) + } + outsider := openClient(t, h) + other := authenticate(t, outsider, otherRoom, protocol.RoleViewer, "outsider_interaction", -1, "", presenceOptions{displayName: "Elsewhere", viewerPresence: true}) + for _, client := range []*testClient{host, viewer, outsider} { + subscribeInteractions(t, client) + } + viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_1", "payload": map[string]any{"kind": "chat", "text": "你好 👋"}}) + hostEvent := host.next("room-interaction") + viewerEvent := viewer.next("room-interaction") + if string(hostEvent.raw) != string(viewerEvent.raw) { + t.Fatal("recipients received different events") + } + decoded, err := protocol.DecodeServerMessage(hostEvent.raw) + if err != nil { + t.Fatal(err) + } + event := decoded.(protocol.RoomInteractionMessage) + if event.OccurredAt != protocol.Int(now.Load()) { + t.Fatal("event time was not stamped at server acceptance") + } + if event.Sender.PeerID != actor.PeerID || event.Sender.DisplayName != "小明" { + t.Fatal("sender identity was not derived from authentication") + } + for _, client := range []*testClient{old, outsider} { + if _, ok := client.tryNext("room-interaction", 30*time.Millisecond); ok { + t.Fatal("interaction escaped its negotiated room boundary") + } + } + viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_2", "payload": map[string]any{"kind": "reaction", "reaction": "wave"}}) + expectMatch(t, viewer.next("room-interaction-rejected").raw, `{"reason":"rate-limited"}`) + now.Add(protocol.InteractionIntervalMs) + viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_3", "payload": map[string]any{"kind": "reaction", "reaction": "tomato", "targetPeerId": other.PeerID}}) + expectMatch(t, viewer.next("room-interaction-rejected").raw, `{"reason":"target-unavailable"}`) + // Room data and its rejections never terminate signaling or require media. + viewer.sendJSON(map[string]any{"type": "signaling-challenge", "sequence": 123}) + viewer.next("signaling-challenge-response") +} + +func TestRoomInteractionRequiresAuthenticationAndSubscription(t *testing.T) { + h := startHarness(t, harnessOptions{}) + guest := openClient(t, h) + guest.sendJSON(map[string]any{"type": "subscribe-room-interactions"}) + expectMatch(t, guest.next("error").raw, `{"code":"AUTH_REQUIRED"}`) + viewer := openClient(t, h) + authenticate(t, viewer, h.room, protocol.RoleViewer, "unsubscribed_viewer", -1, "", presenceOptions{displayName: "Friend", viewerPresence: true}) + viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_1", "payload": map[string]any{"kind": "chat", "text": "hello"}}) + expectMatch(t, viewer.next("room-interaction-rejected").raw, `{"reason":"not-subscribed"}`) +} + +func TestViewersCanInteractAfterHostDisconnects(t *testing.T) { + var now atomic.Int64 + now.Store(100_000) + h := startHarness(t, harnessOptions{now: now.Load}) + host, first, second := openClient(t, h), openClient(t, h), openClient(t, h) + hostActor := authenticate(t, host, h.room, protocol.RoleHost, "leaving_host", -1, "", + presenceOptions{displayName: "Host", roomSession: true}) + authenticate(t, first, h.room, protocol.RoleViewer, "remaining_first", -1, "", presenceOptions{displayName: "First"}) + secondActor := authenticate(t, second, h.room, protocol.RoleViewer, "remaining_second", -1, "", presenceOptions{displayName: "Second"}) + subscribeInteractions(t, first) + subscribeInteractions(t, second) + h.closeClient(host) + for _, payload := range []map[string]any{ + {"kind": "chat", "text": "still here"}, + {"kind": "reaction", "reaction": "wave"}, + {"kind": "reaction", "reaction": "tomato", "targetPeerId": secondActor.PeerID}, + } { + first.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "after_host_left", "payload": payload}) + if a, b := first.next("room-interaction"), second.next("room-interaction"); string(a.raw) != string(b.raw) { + t.Fatal("remaining viewers did not receive the same interaction") + } + now.Add(protocol.InteractionIntervalMs) + } + first.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "departed_target", + "payload": map[string]any{"kind": "reaction", "reaction": "tomato", "targetPeerId": hostActor.PeerID}}) + expectMatch(t, first.next("room-interaction-rejected").raw, `{"reason":"target-unavailable"}`) +} + +func TestRoomInteractionBackpressureDoesNotRetireMediaSignaling(t *testing.T) { + h := startHarness(t, harnessOptions{}) + host, viewer := openClient(t, h), openClient(t, h) + authenticate(t, host, h.room, protocol.RoleHost, "data_host_1234", -1, "", presenceOptions{displayName: "Host", viewerPresence: true}) + actor := authenticate(t, viewer, h.room, protocol.RoleViewer, "slow_viewer_1234", -1, "", presenceOptions{displayName: "Viewer", viewerPresence: true}) + subscribeInteractions(t, host) + subscribeInteractions(t, viewer) + h.locked(func() { + peer, ok := h.server.store.GetConnectedViewer(h.room.RoomID, actor.PeerID) + if !ok { + t.Fatal("missing viewer") + } + slow := h.server.sessionsByID[peer.SessionID] + // Inject a stalled writer under the same lock as a real broadcast, so + // the test does not depend on OS TCP buffer sizes or the reader's timing. + previous := slow.queuedBytes + slow.queuedBytes = 20 * 1024 + hostPeer, _ := h.server.store.GetConnectedHost(h.room.RoomID) + hostSession := h.server.sessionsByID[hostPeer.SessionID] + h.server.handleRoomInteraction(hostSession, hostSession.authenticated, protocol.SendRoomInteractionMessage{ + Type: "send-room-interaction", RequestID: "busy_request_1234", Payload: protocol.InteractionPayload{Kind: "chat", Text: "hello"}, + }) + if slow.terminated || slow.closeQueued { + t.Error("room data closed the signaling session") + } + if slow.queuedBytes != 20*1024 { + t.Error("room data entered a stalled signaling queue") + } + slow.queuedBytes = previous + }) + host.next("room-interaction") + if _, ok := viewer.tryNext("room-interaction", 30*time.Millisecond); ok { + t.Fatal("stalled recipient got room data") + } + viewer.sendJSON(map[string]any{"type": "signaling-challenge", "sequence": 456}) + viewer.next("signaling-challenge-response") +} diff --git a/internal/server/signal/router.go b/internal/server/signal/router.go index 60c6780..f6cda6b 100644 --- a/internal/server/signal/router.go +++ b/internal/server/signal/router.go @@ -75,6 +75,7 @@ type routerOptions struct { store *room.Store endpointMediaCopyCapacity int sfu *sfuFallback + sfuOnly bool hooks routerHooks now func() int64 afterFunc func(time.Duration, func()) func() bool @@ -160,6 +161,7 @@ type router struct { store *room.Store capacity int sfu *sfuFallback + sfuOnly bool hooks routerHooks now func() int64 afterFunc func(time.Duration, func()) func() bool @@ -187,6 +189,7 @@ func newRouter(options routerOptions) *router { store: options.store, capacity: options.endpointMediaCopyCapacity, sfu: options.sfu, + sfuOnly: options.sfuOnly, hooks: options.hooks, now: options.now, afterFunc: options.afterFunc, @@ -995,6 +998,7 @@ func (r *router) createController(roomID string, rm *roomRuntime, host authentic EndpointMediaCopyCapacity: r.capacity, OperationTimeoutMs: operationTimeoutMs, SfuEnabled: r.sfu != nil && !(routePolicy != nil && routePolicy.PeerOnly), + SfuOnly: r.sfuOnly, QualityConvergenceEnabled: routePolicy != nil && routePolicy.TopologyOptimization, NatPredictionEnabled: routePolicy != nil && routePolicy.NatPrediction, }) diff --git a/internal/server/signal/server.go b/internal/server/signal/server.go index bbf7561..4a351a0 100644 --- a/internal/server/signal/server.go +++ b/internal/server/signal/server.go @@ -72,6 +72,8 @@ type Options struct { EndpointMediaCopyCapacity int // SfuFallback is nil when embedded SFU media is disabled. SfuFallback *SfuFallback + // SfuOnly restricts the existing controller to server media candidates. + SfuOnly bool // Ice is the derived wire ICE configuration (config.IceConfig). Ice protocol.IceConfig // NATPredictionEnabled is the server capability that clamps a Host's @@ -215,6 +217,9 @@ func New(options Options) (*Server, error) { if options.SfuFallback != nil && (options.SfuFallback.Media == nil || options.SfuFallback.Admission == nil) { return nil, errors.New("SFU fallback requires media and resource admission") } + if options.SfuOnly && options.SfuFallback == nil { + return nil, errors.New("SFU-only routing requires SFU media and resource admission") + } s := &Server{ store: options.Store, endpointMediaCopyCapacity: options.EndpointMediaCopyCapacity, @@ -271,6 +276,7 @@ func New(options Options) (*Server, error) { store: s.store, endpointMediaCopyCapacity: s.endpointMediaCopyCapacity, sfu: fallback, + sfuOnly: options.SfuOnly, hooks: routerHooks{ sendToSession: s.sendToSession, shareGeneration: func(roomID string) string { return s.shares[roomID].generation }, @@ -713,6 +719,8 @@ type authRequest struct { viewerPresence bool connectionAttemptProgress4 bool displayName *string + roomOnly bool + roomSession bool } func authRequestOf(message protocol.ClientMessage) (authRequest, bool) { @@ -730,6 +738,8 @@ func authRequestOf(message protocol.ClientMessage) (authRequest, bool) { viewerPresence: m.ViewerPresence, connectionAttemptProgress4: m.ConnectionAttemptProgress4, displayName: displayNameString(m.DisplayName), + roomOnly: m.RoomOnly, + roomSession: m.RoomSession, }, true case protocol.AuthenticateViewerMessage: return authRequest{ @@ -758,6 +768,14 @@ func displayNameString(name *protocol.DisplayName) *string { // section. Password checks run in a goroutine so the reader can keep reading // while the KDF runs. func (s *Server) authenticate(sess *session, request authRequest) { + // Reject the conflicting preference before acquiring room or media authority, + // including reconnects from older pages. Never turn peer-only into SFU consent. + if request.role == protocol.RoleHost && !request.roomOnly && s.router.sfuOnly && request.routePolicy.PeerOnly { + s.sendError(sess, "FORBIDDEN", "This site requires server media; turn off Privacy mode or use another site") + sess.close(websocket.StatusCode(protocol.SignalCloseAuthenticationFailed), "Route policy conflict") + s.finishAuthenticating(sess) + return + } if request.role == protocol.RoleHost && !sess.siteAccessAuthenticated { s.sendError(sess, "AUTH_REQUIRED", "Site access is required") sess.close(websocket.StatusCode(protocol.SignalCloseAuthenticationFailed), "Authentication failed") @@ -889,28 +907,32 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part displayName: displayName, viewerPresence: request.viewerPresence, connectionAttemptProgress4: request.connectionAttemptProgress4, + roomSession: request.roomSession, } sess.authenticated = authenticated if participant.Role == protocol.RoleViewer { s.clearViewerEvidence(roomID, participant.PeerID) } s.clearViewerGrace(roomID, participant.PeerID) - routeParticipant := authenticatedRouteParticipant{ - roomID: roomID, - role: participant.Role, - peerID: participant.PeerID, - sessionID: sess.sessionID, + mediaActive := participant.Role == protocol.RoleHost && shareGeneration != "" + if participant.Role == protocol.RoleViewer { + mediaActive = s.activeHostShareGeneration(roomID) != "" } - if participant.Role == protocol.RoleHost { + routeParticipant := routeParticipant(sess, authenticated) + if participant.Role == protocol.RoleHost && mediaActive { policy := s.routePolicyOf(roomID) routeParticipant.routePolicy = &policy } - hybridState, assigned := s.connectRouteParticipant(routeParticipant) - if !assigned { - s.logger.Error("Media router did not assign an authenticated participant") - s.sendError(sess, "SERVER_ERROR", "Media assignment failed") - sess.close(websocket.StatusInternalError, "Media assignment failed") - return + hybridState := hybridAuthenticationState{routeAssignment: emptyAssignment()} + if participant.Role == protocol.RoleViewer || mediaActive { + var assigned bool + hybridState, assigned = s.connectRouteParticipant(routeParticipant) + if !assigned { + s.logger.Error("Media router did not assign an authenticated participant") + s.sendError(sess, "SERVER_ERROR", "Media assignment failed") + sess.close(websocket.StatusInternalError, "Media assignment failed") + return + } } // The viewer's media identity is derived from the committed edge, never // stored beside it; connectParticipant has just settled the graph. @@ -934,8 +956,11 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part qualitySettings = *share.qualitySettings } var roomShareGeneration *string - if share.generation != "" { - generation := share.generation + if participant.Role == protocol.RoleHost && mediaActive { + generation := shareGeneration + roomShareGeneration = &generation + } else if participant.Role == protocol.RoleViewer && mediaActive { + generation := s.activeHostShareGeneration(roomID) roomShareGeneration = &generation } if participant.Role == protocol.RoleHost { @@ -960,15 +985,15 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part ViewerPasswordEnabled: participant.ViewerPasswordEnabled, }) } else { - hostPaused := participant.HostOnline && - share.pausedGeneration == share.generation + hostOnline := participant.HostOnline && mediaActive + hostPaused := hostOnline && share.pausedGeneration == share.generation s.send(sess, protocol.AuthenticatedViewerMessage{ Type: "authenticated", Protocol: protocol.SignalingProtocol, PeerID: participant.PeerID, MaxViewers: protocol.Int(s.store.MaxViewersPerRoom()), EndpointMediaCopyCapacity: protocol.Int(s.endpointMediaCopyCapacity), - HostOnline: participant.HostOnline, + HostOnline: hostOnline, HostPaused: &hostPaused, ConnectionID: connectionID, IceConfig: s.ice, @@ -989,12 +1014,14 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part } } - if participant.Role == protocol.RoleHost { + if participant.Role == protocol.RoleHost && mediaActive { s.router.setPaused(roomID, s.shares[roomID].pausedGeneration == shareGeneration) } - s.router.completeAuthentication(routeParticipant, hybridState) + if participant.Role == protocol.RoleViewer || mediaActive { + s.router.completeAuthentication(routeParticipant, hybridState) + } - if participant.Role == protocol.RoleHost && shareGeneration != "" && + if participant.Role == protocol.RoleHost && mediaActive && s.shares[roomID].pausedGeneration == shareGeneration { s.sendToSession(sess.sessionID, protocol.PauseSharingSourceMessage{ Type: "pause-sharing-source", @@ -1004,7 +1031,7 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part s.sendViewerPresence(roomID) - if participant.Role == protocol.RoleHost { + if participant.Role == protocol.RoleHost && mediaActive { for _, viewer := range connectedViewers { s.sendToSession(viewer.SessionID, protocol.HostStatusMessage{ Type: "host-status", @@ -1021,6 +1048,15 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part // with. It returns the settled generation and must run with mu held, before the // `authenticated` message is built from the record it writes. func (s *Server) settleHostShare(roomID, sessionID string, request authRequest) string { + if request.roomOnly { + // A room-only reconnect must not resurrect a stale publication. The + // stop path normally already cleared it; this guard covers a socket + // loss between the client cleanup and the server's stop message. + if s.shares[roomID].generation != "" { + s.stopSharing(roomID) + } + return "" + } current := s.shares[roomID] currentGeneration := current.generation shareGeneration := request.shareGeneration @@ -1111,10 +1147,33 @@ func (s *Server) routePolicyOf(roomID string) protocol.RoutePolicy { return s.configuredRoutePolicy(protocol.DefaultRoutePolicy) } +// activeHostShareGeneration is the media truth for a room. The room share +// record intentionally keeps its generation as a stale reconnect fence after +// stopSharing, so membership or that fence alone cannot mean "publishing". +// Callers hold Server.mu. +func (s *Server) activeHostShareGeneration(roomID string) string { + host, ok := s.store.GetConnectedHost(roomID) + if !ok { + return "" + } + hostSession := s.sessionsByID[host.SessionID] + if hostSession == nil || hostSession.authenticated == nil || + hostSession.authenticated.role != protocol.RoleHost || + hostSession.authenticated.roomID != roomID { + return "" + } + generation := hostSession.authenticated.shareGeneration + if generation == "" || s.shares[roomID].generation != generation { + return "" + } + return generation +} + // configuredRoutePolicy restricts share preferences to the services this runtime owns. func (s *Server) configuredRoutePolicy(policy protocol.RoutePolicy) protocol.RoutePolicy { policy.PeerOnly = s.router.sfu == nil || policy.PeerOnly - policy.NatPrediction = s.natPredictionEnabled && policy.NatPrediction + policy.NatPrediction = s.natPredictionEnabled && !s.router.sfuOnly && policy.NatPrediction + policy.TopologyOptimization = !s.router.sfuOnly && policy.TopologyOptimization return policy } @@ -1131,6 +1190,12 @@ func routeParticipant(sess *session, authenticated *authenticatedSession) authen func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authenticatedSession, message protocol.ClientMessage) { switch m := message.(type) { + case protocol.SubscribeRoomInteractionsMessage: + // Explicit opt-in protects older v23 readers from unknown events. + sess.roomInteractions = true + s.send(sess, protocol.RoomInteractionsReadyMessage{Type: "room-interactions-ready", ServerTime: protocol.Int(s.now())}) + case protocol.SendRoomInteractionMessage: + s.handleRoomInteraction(sess, authenticated, m) case protocol.SignalingChallengeMessage: if !s.sessions.Has(sess) || sess.authenticated != authenticated { return @@ -1165,6 +1230,10 @@ func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authen s.sendError(sess, "FORBIDDEN", "Only the host may set quality settings") return } + if authenticated.shareGeneration == "" || + s.shares[authenticated.roomID].generation != authenticated.shareGeneration { + return + } share := s.shares[authenticated.roomID] share.qualitySettings = &m.QualitySettings s.shares[authenticated.roomID] = share @@ -1246,28 +1315,39 @@ func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authen s.shares[authenticated.roomID] = share s.router.setPaused(authenticated.roomID, m.Paused) s.broadcastHostStatus(authenticated.roomID, true, m.Paused) + case protocol.StartSharingMessage: + s.startSharing(sess, authenticated, m) case protocol.StopSharingMessage: if authenticated.role != protocol.RoleHost { s.sendError(sess, "FORBIDDEN", "只有当前房主可以停止分享") return } - if authenticated.shareGeneration == "" || - s.shares[authenticated.roomID].generation != authenticated.shareGeneration || + if authenticated.shareGeneration == "" { + return + } + if s.shares[authenticated.roomID].generation != authenticated.shareGeneration || (m.ShareGeneration != "" && m.ShareGeneration != authenticated.shareGeneration) { sess.close(websocket.StatusCode(protocol.SignalCloseSessionReplaced), "Sharing generation replaced") return } - // The participant leaves the store before stopSharing() - // so the sharing-stopped / host-status pair is what viewers see. - if s.sessions.Has(sess) && sess.authenticated == authenticated { - if _, err := s.store.DisconnectParticipant(authenticated.roomID, authenticated.peerID, sess.sessionID); err != nil { - // The TS threw out of the message handler; nothing caught it. - panic(fmt.Errorf("room store failed while stopping sharing: %w", err)) + if !authenticated.roomSession { + // Older clients treat stopping media as leaving the Host session. + // Keep that contract until the client explicitly opts into the + // persistent room-session lifecycle. + if s.sessions.Has(sess) && sess.authenticated == authenticated { + if _, err := s.store.DisconnectParticipant(authenticated.roomID, authenticated.peerID, sess.sessionID); err != nil { + panic(fmt.Errorf("room store failed while stopping sharing: %w", err)) + } } + s.stopSharing(authenticated.roomID) + sess.close(websocket.StatusNormalClosure, "Sharing stopped") + return } s.stopSharing(authenticated.roomID) + // Room membership and its interaction stream outlive the media + // publication. The next start-sharing message claims a fresh generation. + authenticated.shareGeneration = "" s.sendViewerPresence(authenticated.roomID) - sess.close(websocket.StatusNormalClosure, "Sharing stopped") case protocol.AbandonRoomMessage: if authenticated.role != protocol.RoleHost { s.sendError(sess, "FORBIDDEN", "Only the host may abandon the room") @@ -1280,6 +1360,76 @@ func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authen } } +// startSharing claims a new publication generation without replacing the +// authenticated Host participant. Room interactions therefore keep the same +// session while the media graph is rebuilt for the new source. +func (s *Server) startSharing(sess *session, authenticated *authenticatedSession, message protocol.StartSharingMessage) { + if authenticated.role != protocol.RoleHost { + s.sendError(sess, "FORBIDDEN", "Only the host may start sharing") + return + } + if !authenticated.roomSession { + s.sendError(sess, "FORBIDDEN", "Persistent room session is not enabled") + return + } + if !s.sessions.Has(sess) || sess.authenticated != authenticated || + authenticated.shareGeneration != "" || message.ShareGeneration == s.shares[authenticated.roomID].generation || + (s.router.sfuOnly && message.RoutePolicy.PeerOnly) { + s.send(sess, protocol.SharingStartFailedMessage{Type: "sharing-start-failed", + ShareGeneration: message.ShareGeneration, Code: "FORBIDDEN"}) + return + } + + // stop-sharing deliberately leaves the previous generation as a stale + // fence for reconnect validation. A fresh start supersedes that fence + // without emitting a second sharing-stopped event. + share := s.shares[authenticated.roomID] + share.generation = "" + share.pausedGeneration = "" + s.shares[authenticated.roomID] = share + request := authRequest{ + role: protocol.RoleHost, + roomID: authenticated.roomID, + shareGeneration: message.ShareGeneration, + sharingPaused: message.SharingPaused, + qualitySettings: message.QualitySettings, + routePolicy: message.RoutePolicy, + } + shareGeneration := s.settleHostShare(authenticated.roomID, sess.sessionID, request) + authenticated.shareGeneration = shareGeneration + routeParticipant := routeParticipant(sess, authenticated) + policy := s.routePolicyOf(authenticated.roomID) + routeParticipant.routePolicy = &policy + routeState, assigned := s.connectRouteParticipant(routeParticipant) + if !assigned { + authenticated.shareGeneration = "" + s.stopSharing(authenticated.roomID) + s.send(sess, protocol.SharingStartFailedMessage{Type: "sharing-start-failed", + ShareGeneration: message.ShareGeneration, Code: "SERVER_ERROR"}) + return + } + + share = s.shares[authenticated.roomID] + qualitySettings := protocol.DefaultQualitySettings + if share.qualitySettings != nil { + qualitySettings = *share.qualitySettings + } + s.send(sess, protocol.SharingStartedMessage{ + Type: "sharing-started", + ShareGeneration: shareGeneration, + RoutePolicy: s.routePolicyOf(authenticated.roomID), + RouteRevision: protocol.Int(routeState.routeRevision), + RouteAssignment: routeState.routeAssignment, + QualitySettings: qualitySettings, + Paused: share.pausedGeneration == shareGeneration, + }) + // Acknowledge publication authority before any media effects use it. + s.router.setPaused(authenticated.roomID, share.pausedGeneration == shareGeneration) + s.router.completeAuthentication(routeParticipant, routeState) + s.sendViewerPresence(authenticated.roomID) + s.broadcastHostStatus(authenticated.roomID, true, share.pausedGeneration == shareGeneration) +} + // --------------------------------------------------------------------------- // quality evidence // --------------------------------------------------------------------------- diff --git a/internal/server/signal/server_test.go b/internal/server/signal/server_test.go index ae9bd39..a1328e0 100644 --- a/internal/server/signal/server_test.go +++ b/internal/server/signal/server_test.go @@ -9,6 +9,7 @@ package signal // TS went through HTTP; the site-access cookie is a stub predicate. import ( + "bytes" "context" "encoding/json" "errors" @@ -80,6 +81,7 @@ func TestStopSharingDoesNotWriteRoomAuthority(t *testing.T) { h := startHarness(t, harnessOptions{store: store}) host := openClient(t, h) authenticated := authenticate(t, host, h.room, protocol.RoleHost, "storage-free-host", 1, "", presenceOptions{}) + host.ignore("route-update") h.locked(func() { if err := database.Close(); err != nil { t.Fatal(err) @@ -94,6 +96,151 @@ func TestStopSharingDoesNotWriteRoomAuthority(t *testing.T) { authenticate(t, resumed, h.room, protocol.RoleHost, "storage-free-host", 1, "", presenceOptions{}) } +func TestHostKeepsRoomSessionAndInteractionsAfterSharingStops(t *testing.T) { + h := startHarness(t, harnessOptions{maxViewersPerRoom: 2}) + host := openClient(t, h) + hostAuth := authenticate(t, host, h.room, protocol.RoleHost, "persistent-host", 1, + "first_share_generation_12345678", presenceOptions{displayName: "Host", viewerPresence: true, roomSession: true}) + viewer := openClient(t, h) + authenticate(t, viewer, h.room, protocol.RoleViewer, "persistent-viewer", 1, "", + presenceOptions{displayName: "Viewer", viewerPresence: true}) + host.ignore("route-update") + host.ignore("viewer-presence") + subscribeInteractions(t, host) + subscribeInteractions(t, viewer) + + host.sendJSON(map[string]any{"type": "stop-sharing", "shareGeneration": hostAuth.ShareGeneration}) + viewer.next("sharing-stopped") + viewer.next("host-status") + host.expectNone(80 * time.Millisecond) + h.locked(func() { + connected, ok := h.server.store.GetConnectedHost(h.room.RoomID) + if !ok || connected.SessionID == "" { + t.Fatal("stopping media must retain the Host room session") + } + }) + + host.sendJSON(map[string]any{ + "type": "send-room-interaction", + "requestId": "persistent_chat_request", + "payload": map[string]any{"kind": "chat", "text": "still here"}, + }) + host.next("room-interaction") + viewer.next("room-interaction") + + host.sendJSON(map[string]any{ + "type": "start-sharing", + "shareGeneration": "second_share_generation_12345678", + "routePolicy": map[string]any{"peerOnly": true, "topologyOptimization": true, "natPrediction": false}, + }) + started := host.next("sharing-started") + if !bytes.Contains(started.raw, []byte(`"shareGeneration":"second_share_generation_12345678"`)) { + t.Fatalf("unexpected sharing-started: %s", started.raw) + } + viewer.next("host-status") + h.closeClient(viewer) + h.closeClient(host) +} + +func TestIdleHostChatAndPublicationAcknowledgmentOrder(t *testing.T) { + h := startHarness(t, harnessOptions{}) + host := openClient(t, h) + authenticate(t, host, h.room, protocol.RoleHost, "room-only-host", 1, "", presenceOptions{roomSession: true, roomOnly: true, viewerPresence: true}) + host.ignore("viewer-presence") + viewer := openClient(t, h) + authenticate(t, viewer, h.room, protocol.RoleViewer, "room-only-viewer", 1, "", presenceOptions{}) + subscribeInteractions(t, host) + subscribeInteractions(t, viewer) + host.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "chat_before_capture", + "payload": map[string]any{"kind": "chat", "text": "before capture"}}) + host.next("room-interaction") + viewer.next("room-interaction") + host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "new_publication_generation", + "routePolicy": protocol.DefaultRoutePolicy}) + // No ignored route messages: acknowledgment must precede media effects. + host.sendJSON(map[string]any{"type": "signaling-challenge", "sequence": 1}) + host.next("signaling-challenge-response") + host.mu.Lock() + first := host.messages[0].typ + host.mu.Unlock() + if first != "sharing-started" { + t.Fatalf("first publication message = %s", first) + } + host.next("sharing-started") + host.next("route-update") +} + +func TestRejectedPublicationRetainsHostRoomSession(t *testing.T) { + h := startHarness(t, harnessOptions{}) + host := openClient(t, h) + authenticate(t, host, h.room, protocol.RoleHost, "rejected-start-host", 1, "", presenceOptions{roomSession: true, roomOnly: true, viewerPresence: true}) + host.ignore("viewer-presence") + subscribeInteractions(t, host) + // Inject the existing route-construction failure, after room admission. + h.locked(func() { h.server.router.capacity = 0 }) + host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "rejected_publication_generation", + "routePolicy": protocol.DefaultRoutePolicy}) + expectMatch(t, host.next("sharing-start-failed").raw, + `{"shareGeneration":"rejected_publication_generation","code":"SERVER_ERROR"}`) + h.locked(func() { + h.server.router.capacity = 2 + if h.server.activeHostShareGeneration(h.room.RoomID) != "" { + t.Fatal("failed publication remained active") + } + }) + host.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "chat_after_rejected_start", + "payload": map[string]any{"kind": "chat", "text": "still in room"}}) + host.next("room-interaction") + host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "next_publication_generation", + "routePolicy": protocol.DefaultRoutePolicy}) + host.next("sharing-started") + host.next("route-update") + host.ignore("route-update") + host.sendJSON(map[string]any{"type": "stop-sharing", "shareGeneration": "next_publication_generation"}) + host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "next_publication_generation", + "routePolicy": protocol.DefaultRoutePolicy}) + expectMatch(t, host.next("sharing-start-failed").raw, + `{"shareGeneration":"next_publication_generation","code":"FORBIDDEN"}`) +} + +func TestPersistentRoomSessionReconnectStaysMediaIdle(t *testing.T) { + h := startHarness(t, harnessOptions{maxViewersPerRoom: 2}) + host := openClient(t, h) + hostAuth := authenticate(t, host, h.room, protocol.RoleHost, "idle-reconnect-host", 1, + "idle_reconnect_generation_12345678", presenceOptions{roomSession: true}) + viewer := openClient(t, h) + authenticate(t, viewer, h.room, protocol.RoleViewer, "idle-reconnect-viewer", 1, "", presenceOptions{}) + host.ignore("route-update") + viewer.ignore("route-update") + host.sendJSON(map[string]any{"type": "stop-sharing", "shareGeneration": hostAuth.ShareGeneration}) + viewer.next("sharing-stopped") + viewer.next("host-status") + h.closeClient(viewer) + h.closeClient(host) + + resumed := openClient(t, h) + resumedAuth := authenticate(t, resumed, h.room, protocol.RoleHost, "idle-reconnect-host", 1, "", + presenceOptions{roomSession: true, roomOnly: true}) + if resumedAuth.ShareGeneration != nil { + t.Fatalf("room-only Host must not receive a media generation: %v", *resumedAuth.ShareGeneration) + } + h.locked(func() { + if runtime, ok := h.server.router.rooms.Get(h.room.RoomID); ok && runtime.controller != nil { + t.Fatal("room-only Host reconnect created a media controller") + } + }) + lateViewer := openClient(t, h) + lateAuth := authenticate(t, lateViewer, h.room, protocol.RoleViewer, "idle-reconnect-late-viewer", 1, "", presenceOptions{}) + if lateAuth.HostOnline || lateAuth.HostPaused == nil || *lateAuth.HostPaused { + t.Fatalf("idle room advertised active Host media: online=%v paused=%v", lateAuth.HostOnline, lateAuth.HostPaused) + } + if lateAuth.ShareGeneration != nil || lateAuth.RouteAssignment.Upstream.Kind != "none" { + t.Fatalf("idle room advertised a media route: generation=%v assignment=%+v", lateAuth.ShareGeneration, lateAuth.RouteAssignment) + } + h.closeClient(lateViewer) + h.closeClient(resumed) +} + // --------------------------------------------------------------------------- // harness // --------------------------------------------------------------------------- @@ -113,6 +260,7 @@ type harnessOptions struct { room *room.CreatedRoom afterFunc func(time.Duration, func()) func() bool sfu *SfuFallback + sfuOnly bool } type harness struct { @@ -189,6 +337,7 @@ func startHarness(t *testing.T, options harnessOptions) *harness { Store: store, EndpointMediaCopyCapacity: orValue(options.endpointMediaCopyCapacity, 2), SfuFallback: options.sfu, + SfuOnly: options.sfuOnly, Ice: config.IceConfig(config.Config{ STUNURLs: options.stunURLs, NATPredictionEnabled: options.natPredictionEnabled, @@ -803,6 +952,8 @@ type presenceOptions struct { sharingPaused *bool qualitySettings map[string]any routePolicy *protocol.RoutePolicy + roomSession bool + roomOnly bool } // authenticate ports the authenticate() helper. relayCapacity -1 is the TS @@ -833,6 +984,12 @@ func authenticate( if shareGeneration != "" { message["shareGeneration"] = shareGeneration } + if presence.roomSession { + message["roomSession"] = true + } + if presence.roomOnly { + message["roomOnly"] = true + } if presence.sharingPaused != nil { message["sharingPaused"] = *presence.sharingPaused } diff --git a/internal/server/signal/session.go b/internal/server/signal/session.go index 46dd9fa..e6c169c 100644 --- a/internal/server/signal/session.go +++ b/internal/server/signal/session.go @@ -23,6 +23,7 @@ type authenticatedSession struct { displayName *string viewerPresence bool connectionAttemptProgress4 bool + roomSession bool } // outbound is one queue item: a text frame or the close frame that ends the @@ -54,6 +55,8 @@ type session struct { challenged bool lastSignalingChallengeAtMs int64 authenticated *authenticatedSession + roomInteractions bool + nextInteractionAtMs int64 // authStop stops the authentication deadline; authGeneration is // bumped whenever the TS cleared the timer so a callback that lost the diff --git a/internal/server/signal/sfu_only_test.go b/internal/server/signal/sfu_only_test.go new file mode 100644 index 0000000..357e0ec --- /dev/null +++ b/internal/server/signal/sfu_only_test.go @@ -0,0 +1,34 @@ +package signal + +import ( + "testing" + + "github.com/TNTcraftHIM/Piik/internal/server/protocol" + "github.com/TNTcraftHIM/Piik/internal/server/sfu" + "github.com/TNTcraftHIM/Piik/internal/server/sfu/sfutest" +) + +func TestSFUOnlyRejectsPeerOnlyBeforeAcquiringShareAuthority(t *testing.T) { + h := startHarness(t, harnessOptions{sfuOnly: true, natPredictionEnabled: true, + sfu: &SfuFallback{Media: sfutest.New(), Admission: sfu.NewAdmission(sfu.AdmissionOptions{IngressCapacity: 2, EgressCapacity: 20})}}) + policy := protocol.RoutePolicy{PeerOnly: true, TopologyOptimization: true, NatPrediction: true} + rejected := openClient(t, h) + rejected.sendJSON(map[string]any{"type": "authenticate", "protocol": protocol.SignalingProtocol, + "roomId": h.room.RoomID, "role": "host", "clientId": "old-peer-only-host", + "token": h.room.HostToken, "routePolicy": policy}) + expectMatch(t, rejected.next("error").raw, `{"code":"FORBIDDEN"}`) + expectCloseCode(t, rejected, int(protocol.SignalCloseAuthenticationFailed)) + + // An idle room remains available. Refusing a conflicting share preserves it. + host := openClient(t, h) + authenticate(t, host, h.room, protocol.RoleHost, "idle-host", -1, "", + presenceOptions{roomSession: true, roomOnly: true, routePolicy: &policy}) + host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "sfu_only_share_generation", + "routePolicy": policy}) + expectMatch(t, host.next("sharing-start-failed").raw, `{"code":"FORBIDDEN"}`) + policy.PeerOnly = false + host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "sfu_only_share_generation", + "routePolicy": policy}) + expectMatch(t, host.next("sharing-started").raw, + `{"routePolicy":{"peerOnly":false,"topologyOptimization":false,"natPrediction":false}}`) +} diff --git a/native/capture/windows/README.md b/native/capture/windows/README.md index 2e781c3..83fea64 100644 --- a/native/capture/windows/README.md +++ b/native/capture/windows/README.md @@ -5,7 +5,8 @@ enumerates local displays and visible top-level windows, binds each window to its PID and process creation time, and has independent modes for: - capability discovery (`--probe`); -- process-tree or default-device loopback PCM (`--capture-audio`); or +- process-tree inclusion/exclusion or default-device loopback PCM + (`--capture-audio`); or - WGC/D3D11 screen/window video with adapter-bound Media Foundation H.264 or libvpx VP8 output (`--capture-video --codec auto|h264|vp8`). - bounded 320x180 BMP source previews (`--preview`), delivered once per target @@ -30,7 +31,12 @@ Video and audio run as separate bounded child processes. A source whose audio loopback cannot be initialized keeps video available and reports audio unavailable instead of failing the whole source. Process loopback is probed by activation rather than inferred from a Windows build number; display sources -use the standard render-device loopback available on Windows 10 and later. +normally use the standard render-device loopback available on Windows 10 and +later. The optional `--capture-audio exclude ` mode requires +build 20348+ and a successful exclusion probe. It captures all output devices +except the selected process tree, and retires when that process exits. The +[screen-audio contract](../../../docs/standards/media-quality.md#screen-audio) +owns selection and failure behavior. Auto measures delivered frame cadence through the target-profile encoding pipeline within a four-second selection budget; H.264 that sustains the target @@ -50,6 +56,19 @@ The checks include synthetic output-worker replacement and failure cases using WARP and a test codec, plus preview cleanup when WGC rejects a hidden test window. They do not capture a user screen or validate a physical encoder. +On a Windows machine with a physical D3D11 adapter, check SDR and HDR-to-SDR conversion: + +```powershell +powershell -NoProfile -File native/capture/windows/build.ps1 -OutputDirectory build/color-check -CheckColor +``` + +This sends synthetic RGB, full/limited-range BT.601/BT.709 NV12 and FP16 HDR bars +through the production converters. It checks range/matrix, highlight detail, +SDR/HDR replacement, white-level changes and retained-frame ownership without +capturing the desktop. Decoded H.264/VP8, Browser reception and real HDR displays +remain separate media acceptance; [fidelity research](../../../docs/research/media-fidelity.md) +records current evidence and limits. + The implementation follows Microsoft's MIT-licensed reference samples and official API contracts without copying their WIL framework. The retained MF fixture compiles the same encoder source with `PIIK_H264_FIXTURE`; there is diff --git a/native/capture/windows/build.ps1 b/native/capture/windows/build.ps1 index 6c33630..2a39a40 100644 --- a/native/capture/windows/build.ps1 +++ b/native/capture/windows/build.ps1 @@ -2,7 +2,8 @@ param( [Parameter(Mandatory = $true)] [string]$OutputDirectory, - [switch]$Check + [switch]$Check, + [switch]$CheckColor ) Set-StrictMode -Version Latest @@ -34,7 +35,7 @@ $includeFlags = @('', 'third_party\abseil-cpp', 'third_party\boringssl\src\inclu '/external:I "{0}"' -f (Join-Path $webrtc.Include $_).TrimEnd('\') } $compileFlags = '/nologo /c /std:c++20 /EHsc /GR /O2 /W4 /WX /MT /D_ITERATOR_DEBUG_LEVEL=0 /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 /DNTDDI_VERSION=0x0A00000A /external:W0 ' + ($includeFlags -join ' ') -$systemLibraries = 'ole32.lib mmdevapi.lib runtimeobject.lib user32.lib gdi32.lib dwmapi.lib shell32.lib mfplat.lib mf.lib mfuuid.lib d3d11.lib dxgi.lib dxguid.lib evr.lib oleaut32.lib windowsapp.lib winmm.lib ws2_32.lib strmiids.lib crypt32.lib dmoguids.lib iphlpapi.lib msdmo.lib secur32.lib wmcodecdspuuid.lib' +$systemLibraries = 'ole32.lib mmdevapi.lib runtimeobject.lib user32.lib gdi32.lib dwmapi.lib shell32.lib mfplat.lib mf.lib mfuuid.lib d3d11.lib d2d1.lib dxgi.lib dxguid.lib evr.lib oleaut32.lib windowsapp.lib winmm.lib ws2_32.lib strmiids.lib crypt32.lib dmoguids.lib iphlpapi.lib msdmo.lib secur32.lib wmcodecdspuuid.lib' $linkCommand = '"{0}" /nologo /libpath:"{1}"' -f $webrtc.Linker,$webrtc.RuntimeLibraries function Invoke-CaptureBuild([string]$command) { @@ -76,4 +77,16 @@ if ($Check) { if ($LASTEXITCODE -ne 0) { throw "Capture check failed: $name" } } } +if ($CheckColor) { + $source = Join-Path $helperDirectory 'frame_converter.probe.cpp' + $object = Join-Path $outputPath 'frame_converter.probe.obj' + $executable = Join-Path $outputPath 'frame_converter.probe.exe' + Invoke-CaptureBuild ('cl.exe {0} "{1}" /Fo:"{2}"' -f $compileFlags,$source,$object) + $dependencies = @('process_audio', 'capture_target', 'h264_encoder', 'adaptive_encoder') | ForEach-Object { + '"{0}"' -f (Join-Path $outputPath ($_ + '.obj')) + } + Invoke-CaptureBuild ('{0} /out:"{1}" "{2}" {3} "{4}" {5}' -f $linkCommand,$executable,$object,($dependencies -join ' '),$webrtc.Library,$systemLibraries) + & $executable + if ($LASTEXITCODE -ne 0) { throw 'Capture color check failed.' } +} Write-Output $executablePath diff --git a/native/capture/windows/capture_color.h b/native/capture/windows/capture_color.h new file mode 100644 index 0000000..b712528 --- /dev/null +++ b/native/capture/windows/capture_color.h @@ -0,0 +1,34 @@ +#pragma once + +#include "capture_error.h" + +#include +#include +#include + +namespace piik::capture::windows { + +// Untagged WebRTC SDR uses BT.601. Keep the converter's pixels and the H264 +// encoder's range/matrix declaration in agreement; VP8 has no equivalent VUI. +inline constexpr auto kSdrVideoColor = DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P601; +inline constexpr auto kDesktopColor = DXGI_COLOR_SPACE_RGB_FULL_G22_NONE_P709; + +inline void SetSdrVideoColor(IMFAttributes* type) { + Check(type->SetUINT32(MF_MT_YUV_MATRIX, MFVideoTransferMatrix_BT601), "video-type-matrix"); + Check(type->SetUINT32(MF_MT_VIDEO_NOMINAL_RANGE, MFNominalRange_16_235), "video-type-range"); + Check(type->SetUINT32(MF_MT_VIDEO_PRIMARIES, MFVideoPrimaries_SMPTE170M), "video-type-primaries"); + Check(type->SetUINT32(MF_MT_TRANSFER_FUNCTION, MFVideoTransFunc_709), "video-type-transfer"); +} + +// The decoder owns the incoming stream's declaration. Preserve it through the +// relay's local scale/re-encode boundary instead of guessing from resolution. +inline DXGI_COLOR_SPACE_TYPE DecodedVideoColor(IMFAttributes* type) { + const auto matrix = MFGetAttributeUINT32(type, MF_MT_YUV_MATRIX, MFVideoTransferMatrix_BT601); + const bool full = MFGetAttributeUINT32(type, MF_MT_VIDEO_NOMINAL_RANGE, + MFNominalRange_16_235) == MFNominalRange_0_255; + if (matrix == MFVideoTransferMatrix_BT709) + return full ? DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P709 : DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P709; + return full ? DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P601 : kSdrVideoColor; +} + +} // namespace piik::capture::windows diff --git a/native/capture/windows/capture_sdr.h b/native/capture/windows/capture_sdr.h new file mode 100644 index 0000000..7aa1d8d --- /dev/null +++ b/native/capture/windows/capture_sdr.h @@ -0,0 +1,209 @@ +#pragma once + +#include "capture_error.h" +#include "capture_target.h" + +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +namespace piik::capture::windows { + +struct CaptureColor final { + bool hdr = false; + float white_nits = 80.0f; + float peak_nits = 80.0f; + + DXGI_FORMAT Format() const { + return hdr ? DXGI_FORMAT_R16G16B16A16_FLOAT : DXGI_FORMAT_B8G8R8A8_UNORM; + } +}; + +// Windows invalidates the DXGI factory when display color characteristics +// change. Window movement additionally changes the source monitor; neither +// event belongs to encoder or media-route recovery. +class CaptureDisplayColor final { + public: + CaptureDisplayColor(TargetKind kind, UINT64 source_id) + : kind_(kind), source_id_(source_id) {} + + CaptureColor Resolve() { + const auto handle = static_cast(source_id_); + const HMONITOR monitor = kind_ == TargetKind::window + ? MonitorFromWindow(reinterpret_cast(handle), MONITOR_DEFAULTTONEAREST) + : reinterpret_cast(handle); + if (factory_ && factory_->IsCurrent() && monitor == monitor_) return color_; + + Microsoft::WRL::ComPtr factory; + if (FAILED(CreateDXGIFactory1(IID_PPV_ARGS(&factory)))) + return monitor == monitor_ ? color_ : CaptureColor{}; + CaptureColor next; + for (UINT adapter_index = 0;; ++adapter_index) { + Microsoft::WRL::ComPtr adapter; + if (FAILED(factory->EnumAdapters1(adapter_index, &adapter))) break; + for (UINT output_index = 0;; ++output_index) { + Microsoft::WRL::ComPtr output; + if (FAILED(adapter->EnumOutputs(output_index, &output))) break; + Microsoft::WRL::ComPtr extended; + DXGI_OUTPUT_DESC1 description{}; + if (FAILED(output.As(&extended)) || FAILED(extended->GetDesc1(&description)) || + description.Monitor != monitor) continue; + next.hdr = description.ColorSpace == DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020; + if (next.hdr) { + next.white_nits = ReadWhiteLevel(description.DeviceName); + // WGC has no per-frame MaxCLL. Use the source display's stable peak; + // absent usable metadata, retain the HDR10 luminance range. + const float peak = std::isfinite(description.MaxLuminance) && + description.MaxLuminance > 0.0f + ? description.MaxLuminance : 10'000.0f; + next.peak_nits = std::max(next.white_nits, peak); + } + factory_ = std::move(factory); + monitor_ = monitor; + color_ = next; + return color_; + } + } + return next; + } + + private: + static float ReadWhiteLevel(const wchar_t* device_name) { + UINT32 path_count = 0, mode_count = 0; + if (GetDisplayConfigBufferSizes(QDC_ONLY_ACTIVE_PATHS, &path_count, &mode_count) != + ERROR_SUCCESS) return 80.0f; + std::vector paths(path_count); + std::vector modes(mode_count); + if (QueryDisplayConfig(QDC_ONLY_ACTIVE_PATHS, &path_count, paths.data(), &mode_count, + modes.data(), nullptr) != ERROR_SUCCESS) return 80.0f; + for (UINT32 index = 0; index < path_count; ++index) { + const auto& path = paths[index]; + DISPLAYCONFIG_SOURCE_DEVICE_NAME name{}; + name.header = {DISPLAYCONFIG_DEVICE_INFO_GET_SOURCE_NAME, sizeof(name), + path.sourceInfo.adapterId, path.sourceInfo.id}; + if (DisplayConfigGetDeviceInfo(&name.header) != ERROR_SUCCESS || + wcscmp(name.viewGdiDeviceName, device_name) != 0) continue; + DISPLAYCONFIG_SDR_WHITE_LEVEL white{}; + white.header = {DISPLAYCONFIG_DEVICE_INFO_GET_SDR_WHITE_LEVEL, sizeof(white), + path.targetInfo.adapterId, path.targetInfo.id}; + if (DisplayConfigGetDeviceInfo(&white.header) == ERROR_SUCCESS && white.SDRWhiteLevel > 0) + return static_cast(white.SDRWhiteLevel) * 80.0f / 1000.0f; + } + return 80.0f; + } + + const TargetKind kind_; + const UINT64 source_id_; + Microsoft::WRL::ComPtr factory_; + HMONITOR monitor_ = nullptr; + CaptureColor color_; +}; + +// Produces an owned SDR texture before preview or output fanout. Ordinary SDR +// keeps its exact copy path. HDR uses Windows effects on the capture GPU once, +// rather than duplicating tone mapping in each encoder or reading pixels to CPU. +class CaptureSdrConverter final { + public: + explicit CaptureSdrConverter(ID3D11Device* device) : device_(device) { + device_->GetImmediateContext(&context_); + } + + Microsoft::WRL::ComPtr Convert(ID3D11Texture2D* source, + const CaptureColor& color) { + D3D11_TEXTURE2D_DESC description{}; + source->GetDesc(&description); + if (description.Format != color.Format()) + Fail("capture-color-format", "Captured frame and display color generation differ"); + description.Usage = D3D11_USAGE_DEFAULT; + description.CPUAccessFlags = description.MiscFlags = 0; + description.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET; + Microsoft::WRL::ComPtr owned; + Check(device_->CreateTexture2D(&description, nullptr, &owned), "capture-owned-input"); + context_->CopyResource(owned.Get(), source); + if (!color.hdr) return owned; + if (!drawing_) Initialize(); + + Microsoft::WRL::ComPtr surface; + Check(owned.As(&surface), "capture-hdr-surface"); + const auto input_properties = D2D1::BitmapProperties1(D2D1_BITMAP_OPTIONS_NONE, + D2D1::PixelFormat(description.Format, D2D1_ALPHA_MODE_IGNORE)); + Microsoft::WRL::ComPtr input; + Check(drawing_->CreateBitmapFromDxgiSurface(surface.Get(), &input_properties, &input), + "capture-hdr-bitmap"); + Check(tone_map_->SetValue(D2D1_HDRTONEMAP_PROP_INPUT_MAX_LUMINANCE, color.peak_nits), + "capture-hdr-peak"); + Check(tone_map_->SetValue(D2D1_HDRTONEMAP_PROP_OUTPUT_MAX_LUMINANCE, color.white_nits), + "capture-sdr-peak"); + Check(white_level_->SetValue(D2D1_WHITELEVELADJUSTMENT_PROP_OUTPUT_WHITE_LEVEL, color.white_nits), + "capture-sdr-white"); + + description.Format = DXGI_FORMAT_B8G8R8A8_UNORM; + Microsoft::WRL::ComPtr output; + Check(device_->CreateTexture2D(&description, nullptr, &output), "capture-sdr-texture"); + surface.Reset(); + Check(output.As(&surface), "capture-sdr-surface"); + const auto output_properties = D2D1::BitmapProperties1( + D2D1_BITMAP_OPTIONS_TARGET | D2D1_BITMAP_OPTIONS_CANNOT_DRAW, + D2D1::PixelFormat(description.Format, D2D1_ALPHA_MODE_IGNORE)); + Microsoft::WRL::ComPtr target; + Check(drawing_->CreateBitmapFromDxgiSurface(surface.Get(), &output_properties, &target), + "capture-sdr-bitmap"); + tone_map_->SetInput(0, input.Get()); + drawing_->SetTarget(target.Get()); + Microsoft::WRL::ComPtr image; + srgb_->GetOutput(&image); + drawing_->BeginDraw(); + drawing_->DrawImage(image.Get(), nullptr, nullptr, + D2D1_INTERPOLATION_MODE_NEAREST_NEIGHBOR, D2D1_COMPOSITE_MODE_SOURCE_COPY); + const HRESULT result = drawing_->EndDraw(); + drawing_->SetTarget(nullptr); + tone_map_->SetInput(0, nullptr); + Check(result, "capture-hdr-tone-map"); + return output; + } + + private: + void Initialize() { + Microsoft::WRL::ComPtr factory; + Check(D2D1CreateFactory(D2D1_FACTORY_TYPE_SINGLE_THREADED, IID_PPV_ARGS(&factory)), + "capture-d2d-factory"); + Microsoft::WRL::ComPtr dxgi; + Check(device_.As(&dxgi), "capture-d2d-dxgi"); + Microsoft::WRL::ComPtr device; + Check(factory->CreateDevice(dxgi.Get(), &device), "capture-d2d-device"); + Check(device->CreateDeviceContext(D2D1_DEVICE_CONTEXT_OPTIONS_NONE, &drawing_), + "capture-d2d-context"); + const D2D1_RENDERING_CONTROLS controls{D2D1_BUFFER_PRECISION_16BPC_FLOAT, {0, 0}}; + drawing_->SetRenderingControls(&controls); + Check(drawing_->CreateEffect(CLSID_D2D1HdrToneMap, &tone_map_), "capture-tone-map-effect"); + Check(tone_map_->SetValue(D2D1_HDRTONEMAP_PROP_DISPLAY_MODE, D2D1_HDRTONEMAP_DISPLAY_MODE_SDR), + "capture-tone-map-mode"); + Check(drawing_->CreateEffect(CLSID_D2D1WhiteLevelAdjustment, &white_level_), "capture-white-effect"); + Check(white_level_->SetValue(D2D1_WHITELEVELADJUSTMENT_PROP_INPUT_WHITE_LEVEL, 80.0f), + "capture-input-white"); + white_level_->SetInputEffect(0, tone_map_.Get()); + Check(drawing_->CreateEffect(CLSID_D2D1ColorManagement, &srgb_), "capture-srgb-effect"); + Microsoft::WRL::ComPtr linear, srgb; + Check(drawing_->CreateColorContext(D2D1_COLOR_SPACE_SCRGB, nullptr, 0, &linear), "capture-linear-color"); + Check(drawing_->CreateColorContext(D2D1_COLOR_SPACE_SRGB, nullptr, 0, &srgb), "capture-srgb-color"); + Check(srgb_->SetValue(D2D1_COLORMANAGEMENT_PROP_SOURCE_COLOR_CONTEXT, linear.Get()), "capture-input-color"); + Check(srgb_->SetValue(D2D1_COLORMANAGEMENT_PROP_DESTINATION_COLOR_CONTEXT, srgb.Get()), "capture-output-color"); + Check(srgb_->SetValue(D2D1_COLORMANAGEMENT_PROP_QUALITY, D2D1_COLORMANAGEMENT_QUALITY_BEST), + "capture-color-quality"); + srgb_->SetInputEffect(0, white_level_.Get()); + } + + Microsoft::WRL::ComPtr device_; + Microsoft::WRL::ComPtr context_; + Microsoft::WRL::ComPtr drawing_; + Microsoft::WRL::ComPtr tone_map_, white_level_, srgb_; +}; + +} // namespace piik::capture::windows diff --git a/native/capture/windows/capture_target.cpp b/native/capture/windows/capture_target.cpp index 8393792..4675cb3 100644 --- a/native/capture/windows/capture_target.cpp +++ b/native/capture/windows/capture_target.cpp @@ -19,6 +19,7 @@ #include "capture_target.h" #include "capture_border.h" #include "capture_geometry.h" +#include "capture_sdr.h" #include #include @@ -497,8 +498,11 @@ HRESULT CaptureWithWgc(TargetKind kind, UINT64 source_id, if (size.Width <= 0 || size.Height <= 0 || frame_ready == nullptr) { result = HRESULT_FROM_WIN32(ERROR_INVALID_DATA); } else { + windows::CaptureDisplayColor display_color(kind, source_id); + const auto color = display_color.Resolve(); + windows::CaptureSdrConverter sdr(device.Get()); pool = Direct3D11CaptureFramePool::CreateFreeThreaded( - capture_device, DirectXPixelFormat::B8G8R8A8UIntNormalized, 1, + capture_device, static_cast(color.Format()), 1, size); session = pool.CreateCaptureSession(item); frame_token = pool.FrameArrived( @@ -530,6 +534,7 @@ HRESULT CaptureWithWgc(TargetKind kind, UINT64 source_id, ::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>(); ComPtr texture; result = access->GetInterface(IID_PPV_ARGS(&texture)); + if (SUCCEEDED(result)) texture = sdr.Convert(texture.Get(), color); D3D11_TEXTURE2D_DESC description{}; if (SUCCEEDED(result)) texture->GetDesc(&description); if (SUCCEEDED(result) && @@ -1016,9 +1021,14 @@ HRESULT ValidateProcessTarget(DWORD pid, UINT64 expected_creation_time) { } HANDLE process = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid); if (process == nullptr) return HRESULT_FROM_WIN32(GetLastError()); + const HRESULT result = ValidateProcessTarget(process, expected_creation_time); + CloseHandle(process); + return result; +} + +HRESULT ValidateProcessTarget(HANDLE process, UINT64 expected_creation_time) { UINT64 actual_creation_time = 0; HRESULT result = ReadProcessCreationTime(process, &actual_creation_time); - CloseHandle(process); if (SUCCEEDED(result) && actual_creation_time != expected_creation_time) { result = HRESULT_FROM_WIN32(ERROR_INVALID_DATA); } diff --git a/native/capture/windows/capture_target.h b/native/capture/windows/capture_target.h index e1ccf2c..b18c072 100644 --- a/native/capture/windows/capture_target.h +++ b/native/capture/windows/capture_target.h @@ -13,6 +13,7 @@ int WriteMicrophoneList(); HRESULT WriteSourcePreview(TargetKind kind, UINT64 source_id, DWORD pid, UINT64 expected_creation_time); HRESULT ValidateProcessTarget(DWORD pid, UINT64 expected_creation_time); +HRESULT ValidateProcessTarget(HANDLE process, UINT64 expected_creation_time); HRESULT ValidateWindowTarget(UINT64 source_id, DWORD pid, UINT64 expected_creation_time); HRESULT ValidateDisplayTarget(UINT64 source_id); diff --git a/native/capture/windows/capture_target.test.cpp b/native/capture/windows/capture_target.test.cpp index 296a53b..4f02628 100644 --- a/native/capture/windows/capture_target.test.cpp +++ b/native/capture/windows/capture_target.test.cpp @@ -15,6 +15,8 @@ int main() { assert(GetProcessTimes(GetCurrentProcess(), &created, &exited, &kernel, &user)); const UINT64 creation_time = (static_cast(created.dwHighDateTime) << 32) | created.dwLowDateTime; + assert(SUCCEEDED(piik::capture::ValidateProcessTarget(GetCurrentProcess(), creation_time))); + assert(FAILED(piik::capture::ValidateProcessTarget(GetCurrentProcess(), creation_time + 1))); HANDLE previous_output = GetStdHandle(STD_OUTPUT_HANDLE); HANDLE sink = CreateFileW(L"NUL", GENERIC_WRITE, FILE_SHARE_WRITE, nullptr, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr); diff --git a/native/capture/windows/frame_converter.probe.cpp b/native/capture/windows/frame_converter.probe.cpp new file mode 100644 index 0000000..10da309 --- /dev/null +++ b/native/capture/windows/frame_converter.probe.cpp @@ -0,0 +1,218 @@ +// Physical D3D11 check; opt in with build.ps1 -CheckColor. No desktop capture. +#define wmain capture_main +#include "main.cpp" +#undef wmain +#include + +namespace { + +constexpr std::array kRgb{ + 0xff000000, 0xff202020, 0xff808080, 0xffebebeb, + 0xffffffff, 0xffff0000, 0xff00ff00, 0xff0000ff}; +constexpr std::array, 8> kYuv{{ + {16, 128, 128}, {44, 128, 128}, {126, 128, 128}, {218, 128, 128}, + {235, 128, 128}, {82, 90, 240}, {145, 54, 34}, {41, 240, 110}}}; +constexpr std::array, 8> kYuv709{{ + {16, 128, 128}, {44, 128, 128}, {126, 128, 128}, {218, 128, 128}, + {235, 128, 128}, {63, 102, 240}, {173, 42, 26}, {32, 240, 118}}}; +constexpr std::array, 8> kYuvFull{{ + {0, 128, 128}, {32, 128, 128}, {128, 128, 128}, {235, 128, 128}, + {255, 128, 128}, {76, 85, 255}, {150, 44, 21}, {29, 255, 107}}}; +constexpr std::array, 8> kYuv709Full{{ + {0, 128, 128}, {32, 128, 128}, {128, 128, 128}, {235, 128, 128}, + {255, 128, 128}, {54, 99, 255}, {182, 30, 12}, {18, 255, 116}}}; + +ComPtr ColorBars(const DeviceContext& device, UINT width, + UINT height, DXGI_COLOR_SPACE_TYPE color) { + const bool rgb = color == kDesktopColor; + const auto& yuv = color == DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P709 ? kYuv709 + : color == DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P709 ? kYuv709Full + : color == DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P601 ? kYuvFull : kYuv; + std::vector pixels(static_cast(width) * height * (rgb ? 4 : 2)); + for (UINT row = 0; row < height; ++row) { + for (UINT column = 0; column < width; ++column) { + const size_t bar = column * kRgb.size() / width; + const size_t index = static_cast(row) * width + column; + if (rgb) { + for (UINT channel = 0; channel < 4; ++channel) + pixels[index * 4 + channel] = static_cast(kRgb[bar] >> (channel * 8)); + } else { + pixels[index] = static_cast(yuv[bar][0]); + if (row < height / 2) + pixels[static_cast(width) * height + index] = + static_cast(yuv[bar][1 + column % 2]); + } + } + } + D3D11_TEXTURE2D_DESC description{}; + description.Width = width; + description.Height = height; + description.MipLevels = description.ArraySize = description.SampleDesc.Count = 1; + description.Format = rgb ? DXGI_FORMAT_B8G8R8A8_UNORM : DXGI_FORMAT_NV12; + description.BindFlags = D3D11_BIND_RENDER_TARGET | D3D11_BIND_SHADER_RESOURCE; + D3D11_SUBRESOURCE_DATA data{}; + data.pSysMem = pixels.data(); + data.SysMemPitch = width * (rgb ? 4 : 1); + ComPtr texture; + Check(device.device->CreateTexture2D(&description, &data, &texture), "color-source"); + return texture; +} + +void CheckColors(const DeviceContext& device, ID3D11Texture2D* texture) { + D3D11_TEXTURE2D_DESC description{}; + texture->GetDesc(&description); + description.Usage = D3D11_USAGE_STAGING; + description.BindFlags = 0; + description.CPUAccessFlags = D3D11_CPU_ACCESS_READ; + ComPtr staging; + Check(device.device->CreateTexture2D(&description, nullptr, &staging), "color-readback"); + device.context->CopyResource(staging.Get(), texture); + D3D11_MAPPED_SUBRESOURCE mapped{}; + Check(device.context->Map(staging.Get(), 0, D3D11_MAP_READ, 0, &mapped), "color-map"); + const auto* bytes = static_cast(mapped.pData); + bool correct = true; + for (size_t bar = 0; bar < kYuv.size(); ++bar) { + const auto x = description.Width * (2 * bar + 1) / (2 * kYuv.size()); + const size_t y = description.Height / 2; + const size_t uv = (description.Height + y / 2) * mapped.RowPitch + (x & ~size_t{1}); + const std::array actual{bytes[y * mapped.RowPitch + x], bytes[uv], bytes[uv + 1]}; + for (size_t channel = 0; channel < 3; ++channel) { + if (std::abs(actual[channel] - kYuv[bar][channel]) > 2) { + std::cerr << "bar=" << bar << " channel=" << channel + << " expected=" << kYuv[bar][channel] << " actual=" << actual[channel] << '\n'; + correct = false; + } + } + } + device.context->Unmap(staging.Get(), 0); + if (!correct) throw std::runtime_error("SDR range or matrix changed"); +} + +ComPtr LinearBars(const DeviceContext& device, UINT width, + UINT height, const std::array& nits) { + std::vector pixels(static_cast(width) * height * 4); + for (size_t pixel = 0; pixel < static_cast(width) * height; ++pixel) { + const auto value = DirectX::PackedVector::XMConvertFloatToHalf(nits[(pixel % width) * 8 / width] / 80.0f); + for (size_t channel = 0; channel < 3; ++channel) pixels[pixel * 4 + channel] = value; + pixels[pixel * 4 + 3] = DirectX::PackedVector::XMConvertFloatToHalf(1.0f); + } + D3D11_TEXTURE2D_DESC description{}; + description.Width = width; + description.Height = height; + description.MipLevels = description.ArraySize = description.SampleDesc.Count = 1; + description.Format = DXGI_FORMAT_R16G16B16A16_FLOAT; + description.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET; + D3D11_SUBRESOURCE_DATA data{}; + data.pSysMem = pixels.data(); + data.SysMemPitch = width * 8; + ComPtr texture; + Check(device.device->CreateTexture2D(&description, &data, &texture), "hdr-source"); + return texture; +} + +std::array ReadGrayBars(const DeviceContext& device, ID3D11Texture2D* texture) { + D3D11_TEXTURE2D_DESC description{}; + texture->GetDesc(&description); + description.Usage = D3D11_USAGE_STAGING; + description.BindFlags = 0; + description.CPUAccessFlags = D3D11_CPU_ACCESS_READ; + ComPtr staging; + Check(device.device->CreateTexture2D(&description, nullptr, &staging), "hdr-readback"); + device.context->CopyResource(staging.Get(), texture); + D3D11_MAPPED_SUBRESOURCE mapped{}; + Check(device.context->Map(staging.Get(), 0, D3D11_MAP_READ, 0, &mapped), "hdr-map"); + const auto* bytes = static_cast(mapped.pData); + std::array result{}; + bool neutral = true; + for (size_t bar = 0; bar < result.size(); ++bar) { + const size_t pixel = description.Height / 2 * mapped.RowPitch + + description.Width * (2 * bar + 1) / 16 * 4; + result[bar] = bytes[pixel]; + neutral = neutral && std::abs(result[bar] - bytes[pixel + 1]) <= 1 && + std::abs(result[bar] - bytes[pixel + 2]) <= 1; + } + device.context->Unmap(staging.Get(), 0); + if (!neutral) throw std::runtime_error("HDR gray acquired a color cast"); + return result; +} + +void CheckHdr(const DeviceContext& device) { + CaptureSdrConverter capture(device.device.Get()); + VideoProfile profile; + profile.width = 256; + profile.height = 128; + FrameConverter converter(device.device.Get(), profile); + for (const UINT width : {256u, 512u, 256u}) { + // SDR must retain the old pixels after an HDR frame, with the same owner. + auto sdr = ColorBars(device, width, width / 2, kDesktopColor); + auto owned = capture.Convert(sdr.Get(), {}); + CheckColors(device, converter.Convert(owned.Get(), width, width / 2, + {static_cast(width), static_cast(width / 2)}, kDesktopColor).Get()); + auto hdr = LinearBars(device, width, width / 2, {0, 5, 20, 80, 200, 400, 700, 1000}); + for (const float white : {80.0f, 240.0f, 80.0f}) { + auto mapped = capture.Convert(hdr.Get(), {true, white, 1000.0f}); + const auto levels = ReadGrayBars(device, mapped.Get()); + std::cout << "HDR white=" << white << " levels="; + for (const auto level : levels) std::cout << level << ','; + std::cout << '\n'; + if (levels.front() > 1 || levels.back() < 245) + throw std::runtime_error("HDR black/peak mapping is incorrect"); + for (size_t index = 1; index < levels.size(); ++index) + if (levels[index] <= levels[index - 1]) + throw std::runtime_error("HDR luminance detail was clipped before tone mapping"); + auto next = capture.Convert(sdr.Get(), {}); + if (ReadGrayBars(device, mapped.Get()) != levels) + throw std::runtime_error("Capture replacement modified a retained frame"); + CheckColors(device, converter.Convert(next.Get(), width, width / 2, + {static_cast(width), static_cast(width / 2)}, kDesktopColor).Get()); + } + } +} + +} // namespace + +int main() { + try { + Runtime runtime; + const auto adapters = EnumerateAdapters(); + if (adapters.empty()) throw std::runtime_error("A physical D3D11 adapter is required"); + for (const auto& adapter : adapters) { + auto device = CreateDevice(adapter); + for (const UINT output_width : {256u, 128u}) { + VideoProfile profile; + profile.width = output_width; + profile.height = output_width / 2; + FrameConverter converter(device.device.Get(), profile); + // Reuse the same converter across RGB/NV12 and size replacements. + for (const UINT input_width : {256u, 512u, 256u}) { + for (const auto color : {kDesktopColor, kSdrVideoColor, + DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P709, + DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P601, + DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P709, kDesktopColor}) { + const UINT input_height = input_width / 2; + auto source = ColorBars(device, input_width, input_height, color); + auto output = converter.Convert(source.Get(), input_width, input_height, + {static_cast(input_width), static_cast(input_height)}, + color); + try { + CheckColors(device, output.Get()); + } catch (...) { + std::cerr << "adapter=" << adapter.index << " color=" << color + << " input-width=" << input_width << " output-width=" << output_width << '\n'; + throw; + } + } + } + } + std::cout << "SDR conversion passed: " << NarrowAscii(adapter.description.Description) << '\n'; + CheckHdr(device); + std::cout << "HDR conversion passed: " << NarrowAscii(adapter.description.Description) << '\n'; + } + return 0; + } catch (const std::exception& error) { + std::cerr << error.what() << '\n'; + if (const auto* failure = dynamic_cast(&error)) + std::cerr << "stage=" << failure->stage() << " result=" << std::hex << failure->result() << '\n'; + return 1; + } +} diff --git a/native/capture/windows/h264_decoder.h b/native/capture/windows/h264_decoder.h index 0afad8c..f7e0bd5 100644 --- a/native/capture/windows/h264_decoder.h +++ b/native/capture/windows/h264_decoder.h @@ -1,6 +1,7 @@ #pragma once #include "capture_error.h" +#include "capture_color.h" #include "capture_input.h" @@ -56,6 +57,12 @@ class H264Decoder final { H264Decoder(const H264Decoder&) = delete; H264Decoder& operator=(const H264Decoder&) = delete; + DXGI_COLOR_SPACE_TYPE ColorSpace() const { + Microsoft::WRL::ComPtr type; + Check(decoder_->GetOutputCurrentType(0, &type), "current-output-type"); + return windows::DecodedVideoColor(type.Get()); + } + void Decode(const InputEnvelope& input, const Consumer& consume) { if (input.kind != 2 || input.data.empty() || input.data.size() > 4 * 1024 * 1024) throw std::runtime_error("Invalid H264 decoder input"); diff --git a/native/capture/windows/h264_encoder.cpp b/native/capture/windows/h264_encoder.cpp index 3d52b55..a9c01cb 100644 --- a/native/capture/windows/h264_encoder.cpp +++ b/native/capture/windows/h264_encoder.cpp @@ -1,6 +1,7 @@ #include "h264_encoder.h" #include "capture_output.h" +#include "capture_color.h" #include #include @@ -327,6 +328,7 @@ ComPtr CreateOutputType( "output-type-major"); Check(type->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_H264), "output-type-subtype"); + SetSdrVideoColor(type.Get()); Check(type->SetUINT32(MF_MT_AVG_BITRATE, profile.bit_rate), "output-type-bitrate"); Check(MFSetAttributeSize(type.Get(), MF_MT_FRAME_SIZE, profile.width, @@ -354,6 +356,7 @@ ComPtr CreateInputType( "input-type-major"); Check(type->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_NV12), "input-type-subtype"); + SetSdrVideoColor(type.Get()); Check(MFSetAttributeSize(type.Get(), MF_MT_FRAME_SIZE, profile.width, profile.height), "input-type-size"); diff --git a/native/capture/windows/main.cpp b/native/capture/windows/main.cpp index c957ab2..8f34976 100644 --- a/native/capture/windows/main.cpp +++ b/native/capture/windows/main.cpp @@ -3,7 +3,7 @@ #include #include -#include +#include #include #include #include @@ -32,6 +32,8 @@ #include "capture_target.h" #include "capture_border.h" #include "capture_geometry.h" +#include "capture_color.h" +#include "capture_sdr.h" #include "process_audio.h" #include "h264_encoder.h" #ifndef PIIK_H264_FIXTURE @@ -663,8 +665,9 @@ class FrameConverter final { public: explicit FrameConverter( ID3D11Device* device, - VideoProfile profile = kDefaultVideoProfile) - : device_(device), profile_(profile) { + VideoProfile profile = kDefaultVideoProfile, + DXGI_COLOR_SPACE_TYPE output_color = kSdrVideoColor) + : device_(device), profile_(profile), output_color_(output_color) { Check(device_->QueryInterface(IID_PPV_ARGS(&video_device_)), "video-processor-device"); ComPtr context; @@ -673,13 +676,30 @@ class FrameConverter final { } ComPtr Convert(ID3D11Texture2D* source, UINT32 width, - UINT32 height, SIZE presentation) { + UINT32 height, SIZE presentation, + DXGI_COLOR_SPACE_TYPE input_color) { if (source == nullptr || width == 0 || height == 0 || width > 16'384 || height > 16'384) { Fail("capture-size", "captured window dimensions are invalid"); } - if (!enumerator_ || width != input_width_ || height != input_height_) { - Configure(width, height); + ComPtr normalized; + if (output_color_ == kSdrVideoColor && input_color != kDesktopColor && + input_color != kSdrVideoColor) { + // YUV-to-YUV matrix/range changes are not reliable across tested drivers, + // even when reported as supported. Normalize through RGB on the GPU; + // ordinary RGB capture and canonical NV12 scaling stay single-pass. + if (!rgb_converter_) + rgb_converter_ = std::make_unique(device_.Get(), profile_, kDesktopColor); + normalized = rgb_converter_->Convert(source, width, height, presentation, input_color); + source = normalized.Get(); + width = profile_.width; + height = profile_.height; + presentation = {static_cast(width), static_cast(height)}; + input_color = kDesktopColor; + } + if (!enumerator_ || width != input_width_ || height != input_height_ || + input_color != input_color_) { + Configure(width, height, input_color); } D3D11_TEXTURE2D_DESC output_description = {}; @@ -687,7 +707,8 @@ class FrameConverter final { output_description.Height = profile_.height; output_description.MipLevels = 1; output_description.ArraySize = 1; - output_description.Format = DXGI_FORMAT_NV12; + output_description.Format = output_color_ == kDesktopColor + ? DXGI_FORMAT_B8G8R8A8_UNORM : DXGI_FORMAT_NV12; output_description.SampleDesc.Count = 1; output_description.Usage = D3D11_USAGE_DEFAULT; output_description.BindFlags = D3D11_BIND_RENDER_TARGET; @@ -745,7 +766,7 @@ class FrameConverter final { } private: - void Configure(UINT32 width, UINT32 height) { + void Configure(UINT32 width, UINT32 height, DXGI_COLOR_SPACE_TYPE input_color) { D3D11_VIDEO_PROCESSOR_CONTENT_DESC description = {}; description.InputFrameFormat = D3D11_VIDEO_FRAME_FORMAT_PROGRESSIVE; description.InputFrameRate = {profile_.frame_rate, 1}; @@ -762,20 +783,27 @@ class FrameConverter final { ComPtr processor; Check(video_device_->CreateVideoProcessor(enumerator.Get(), 0, &processor), "video-processor-create"); + // The incoming frame owns its range/matrix. Never infer these from size. + video_context_->VideoProcessorSetStreamColorSpace1(processor.Get(), 0, input_color); + video_context_->VideoProcessorSetOutputColorSpace1(processor.Get(), output_color_); enumerator_ = std::move(enumerator); processor_ = std::move(processor); input_width_ = width; input_height_ = height; + input_color_ = input_color; } ComPtr device_; VideoProfile profile_; + const DXGI_COLOR_SPACE_TYPE output_color_; + std::unique_ptr rgb_converter_; ComPtr video_device_; - ComPtr video_context_; + ComPtr video_context_; ComPtr enumerator_; ComPtr processor_; UINT32 input_width_ = 0; UINT32 input_height_ = 0; + DXGI_COLOR_SPACE_TYPE input_color_ = kDesktopColor; }; winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice @@ -933,6 +961,7 @@ struct ProductArguments final { VideoProfile profile; std::vector outputs; bool show_capture_border = false; + bool exclude_audio = false; }; DegradationPreference ParseDegradationPreference(const wchar_t* value) { @@ -1020,7 +1049,8 @@ ProductArguments ParseProductArguments(int count, wchar_t** values) { } if (count == 5 && std::wstring(values[1]) == L"--capture-audio") { arguments.mode = ProductArguments::Mode::audio; - arguments.target_kind = ParseTargetKind(values[2]); + arguments.exclude_audio = std::wstring(values[2]) == L"exclude"; + arguments.target_kind = arguments.exclude_audio ? piik::capture::TargetKind::window : ParseTargetKind(values[2]); } else if (count >= 16 && count <= 11 + 5 * piik::capture::kMaxOutputs && (count - 11) % 5 == 0 && std::wstring(values[1]) == L"--encoded-video" && std::wstring(values[2]) == L"--codec" && @@ -1126,7 +1156,7 @@ HRESULT RunAudioCapture(const ProductArguments& arguments) { stop.get(), should_stop, ready, pcm); } return piik::capture::CaptureProcessAudio( - arguments.pid, arguments.creation_time, stop.get(), + arguments.pid, arguments.creation_time, arguments.exclude_audio, stop.get(), should_stop, ready, pcm); } @@ -1153,6 +1183,7 @@ UINT32 WindowsBuild() { void WriteCapabilityProbe() { constexpr UINT32 kCreateForWindowMinimumBuild = 18'362; constexpr UINT32 kProcessLoopbackMinimumBuild = 19'041; + constexpr UINT32 kProcessExclusionMinimumBuild = 20'348; const UINT32 build = WindowsBuild(); bool window_capture = false; if (build >= kCreateForWindowMinimumBuild) { @@ -1167,6 +1198,8 @@ void WriteCapabilityProbe() { build >= kProcessLoopbackMinimumBuild && piik::capture::ProcessAudioAvailable(); const bool system_audio = piik::capture::SystemAudioAvailable(); + const bool process_audio_exclusion = build >= kProcessExclusionMinimumBuild && + piik::capture::ProcessAudioAvailable(true); std::vector adapters = EnumerateAdapters(); std::ostringstream output; @@ -1178,6 +1211,7 @@ void WriteCapabilityProbe() { << ",\"microphone\":true,\"softwareVP8\":true" << ",\"processAudio\":" << (process_audio ? "true" : "false") + << ",\"processAudioExclusion\":" << (process_audio_exclusion ? "true" : "false") << ",\"systemAudio\":" << (system_audio ? "true" : "false") << ",\"adapters\":["; for (size_t adapter_index = 0; adapter_index < adapters.size(); @@ -1361,6 +1395,7 @@ VideoEncoderSelection SelectVideoEncoder( struct CaptureInput final { ComPtr texture; + DXGI_COLOR_SPACE_TYPE color_space = kDesktopColor; UINT32 width = 0; UINT32 height = 0; SIZE presentation{}; @@ -1448,7 +1483,7 @@ class OutputWorker final { converted_height = height; } return converter->Convert(input->texture.Get(), input->width, - input->height, input->presentation); + input->height, input->presentation, input->color_space); }, profile_.width, profile_.height, input->timestamp, work.recovery, work.bitrate); } catch (...) { // Device loss is current shared evidence; an old codec failure only @@ -1703,12 +1738,13 @@ void RunEncodedVideo(ProductArguments arguments) { bool began = false; UINT64 last_timestamp = 0; auto submit = [&](ComPtr texture, UINT32 width, UINT32 height, - UINT64 timestamp, UINT64 duration) { + UINT64 timestamp, UINT64 duration, DXGI_COLOR_SPACE_TYPE color) { if ((began && timestamp <= last_timestamp) || width < arguments.profile.width || height < arguments.profile.height) { Fail("encoded-input-output", "decoded frame changed the source timeline or output bounds"); } auto input = std::make_shared(); input->texture = std::move(texture); + input->color_space = color; input->width = width; input->height = height; input->presentation = {static_cast(width), static_cast(height)}; @@ -1735,12 +1771,12 @@ void RunEncodedVideo(ProductArguments arguments) { Check(sample->GetSampleDuration(&duration), "decoded-duration"); if (timestamp < 0 || duration <= 0) Fail("decoded-timing", "decoder did not preserve source timing"); submit(OwnDecodedTexture(device, sample, width, height, stride), width, height, - static_cast(timestamp), static_cast(duration)); + static_cast(timestamp), static_cast(duration), h264->ColorSpace()); }); } else { const auto decoded = vp8->Decode(input.data); if (!decoded.nv12.empty()) submit(UploadDecodedNV12(device, decoded.nv12.data(), decoded.width, decoded.height), - decoded.width, decoded.height, input.timestamp, input.duration); + decoded.width, decoded.height, input.timestamp, input.duration, kSdrVideoColor); } } } @@ -1801,8 +1837,11 @@ void RunVideoCapture(ProductArguments arguments) { if (initial_size.Width <= 0 || initial_size.Height <= 0) { Fail("capture-size", "selected window has no capturable content"); } + CaptureDisplayColor display_color(arguments.target_kind, arguments.source_id); + CaptureSdrConverter sdr(device.device.Get()); + auto pool_format = display_color.Resolve().Format(); Direct3D11CaptureFramePool pool = Direct3D11CaptureFramePool::CreateFreeThreaded( - capture_device, DirectXPixelFormat::B8G8R8A8UIntNormalized, 2, + capture_device, static_cast(pool_format), 2, initial_size); GraphicsCaptureSession capture_session = pool.CreateCaptureSession(item); EnableFastCaptureUpdates(capture_session); @@ -1964,15 +2003,17 @@ void RunVideoCapture(ProductArguments arguments) { } auto content_size = latest.ContentSize(); if (content_size.Width <= 0 || content_size.Height <= 0) continue; + const auto color = display_color.Resolve(); if (content_size.Width != pool_size.Width || - content_size.Height != pool_size.Height) { + content_size.Height != pool_size.Height || color.Format() != pool_format) { latest.Close(); latest = nullptr; latest_input.reset(); pool.Recreate(capture_device, - DirectXPixelFormat::B8G8R8A8UIntNormalized, 2, + static_cast(color.Format()), 2, content_size); pool_size = content_size; + pool_format = color.Format(); continue; } ComPtr source = CaptureTexture(latest); @@ -1983,13 +2024,7 @@ void RunVideoCapture(ProductArguments arguments) { UINT32 content_height = std::min( source_description.Height, static_cast(content_size.Height)); auto input = std::make_shared(); - auto owned_description = source_description; - owned_description.Usage = D3D11_USAGE_DEFAULT; - owned_description.CPUAccessFlags = 0; - owned_description.MiscFlags = 0; - owned_description.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET; - Check(device.device->CreateTexture2D(&owned_description, nullptr, &input->texture), "capture-owned-input"); - device.context->CopyResource(input->texture.Get(), source.Get()); + input->texture = sdr.Convert(source.Get(), color); input->width = content_width; input->height = content_height; input->timestamp = timestamp; diff --git a/native/capture/windows/process_audio.cpp b/native/capture/windows/process_audio.cpp index ca84706..f439697 100644 --- a/native/capture/windows/process_audio.cpp +++ b/native/capture/windows/process_audio.cpp @@ -31,7 +31,8 @@ namespace { class ActivationHandler final : public RuntimeClass< RuntimeClassFlags, FtmBase, IActivateAudioInterfaceCompletionHandler> { public: - explicit ActivationHandler(HANDLE completed) : completed_(completed) {} + ActivationHandler() : completed_(CreateEventW(nullptr, FALSE, FALSE, nullptr)) {} + ~ActivationHandler() { if (completed_) CloseHandle(completed_); } STDMETHODIMP ActivateCompleted(IActivateAudioInterfaceAsyncOperation* operation) override { ComPtr activated; @@ -45,6 +46,7 @@ class ActivationHandler final : public RuntimeClass< HRESULT Result() const { return result_; } ComPtr Client() const { return client_; } + HANDLE Completed() const { return completed_; } private: HANDLE completed_ = nullptr; @@ -52,26 +54,29 @@ class ActivationHandler final : public RuntimeClass< ComPtr client_; }; -HRESULT ActivateProcessLoopback(DWORD pid, HANDLE completed, +HRESULT ActivateProcessLoopback(DWORD pid, bool exclude, ComPtr* client) { AUDIOCLIENT_ACTIVATION_PARAMS parameters{}; parameters.ActivationType = AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK; parameters.ProcessLoopbackParams.TargetProcessId = pid; parameters.ProcessLoopbackParams.ProcessLoopbackMode = - PROCESS_LOOPBACK_MODE_INCLUDE_TARGET_PROCESS_TREE; + exclude ? PROCESS_LOOPBACK_MODE_EXCLUDE_TARGET_PROCESS_TREE + : PROCESS_LOOPBACK_MODE_INCLUDE_TARGET_PROCESS_TREE; PROPVARIANT variant{}; variant.vt = VT_BLOB; variant.blob.cbSize = sizeof(parameters); variant.blob.pBlobData = reinterpret_cast(¶meters); - auto handler = Make(completed); + // The completion can outlive a timed-out caller; it owns its event handle. + auto handler = Make(); if (!handler) return E_OUTOFMEMORY; + if (!handler->Completed()) return HRESULT_FROM_WIN32(GetLastError()); ComPtr operation; HRESULT result = ActivateAudioInterfaceAsync(VIRTUAL_AUDIO_DEVICE_PROCESS_LOOPBACK, __uuidof(IAudioClient), &variant, handler.Get(), &operation); if (FAILED(result)) return result; - if (WaitForSingleObject(completed, 10'000) != WAIT_OBJECT_0) return HRESULT_FROM_WIN32(WAIT_TIMEOUT); + if (WaitForSingleObject(handler->Completed(), 10'000) != WAIT_OBJECT_0) return HRESULT_FROM_WIN32(WAIT_TIMEOUT); result = handler->Result(); if (SUCCEEDED(result)) *client = handler->Client(); return result; @@ -114,6 +119,14 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo return result; } HRESULT result = S_OK; + // Bind process audio to the selected lifetime, including while draining PCM. + // Exclusion must never become unfiltered system capture after its target dies. + const auto processAlive = [&]() -> HRESULT { + if (!process) return S_OK; + const DWORD wait = WaitForSingleObject(process, 0); + if (wait == WAIT_TIMEOUT) return S_OK; + return HRESULT_FROM_WIN32(wait == WAIT_OBJECT_0 ? ERROR_PROCESS_ABORTED : GetLastError()); + }; ComPtr capture; WAVEFORMATEX format{}; format.wFormatTag = WAVE_FORMAT_PCM; @@ -131,6 +144,7 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo if (SUCCEEDED(result)) result = client->GetService(IID_PPV_ARGS(&capture)); if (SUCCEEDED(result)) result = client->SetEventHandle(sampleReady); if (SUCCEEDED(result)) result = client->Start(); + if (SUCCEEDED(result)) result = processAlive(); if (SUCCEEDED(result)) result = ready_writer(); if (SUCCEEDED(result)) { std::array silence{}; @@ -171,6 +185,7 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo } UINT32 frames = 0; while (SUCCEEDED(result = capture->GetNextPacketSize(&frames)) && frames > 0) { + if (FAILED(result = processAlive())) break; BYTE* data = nullptr; DWORD flags = 0; UINT64 devicePosition = 0; @@ -198,6 +213,7 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo } capture->ReleaseBuffer(frames); while (pending.size() - consumed >= kAudioBytesPerChunk) { + if (FAILED(result = processAlive())) break; result = writer(nextTimestamp, pending.data() + consumed, kAudioBytesPerChunk); if (FAILED(result)) break; @@ -218,13 +234,10 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo } // namespace -bool ProcessAudioAvailable() { - HANDLE completed = CreateEventW(nullptr, FALSE, FALSE, nullptr); - if (completed == nullptr) return false; +bool ProcessAudioAvailable(bool exclude) { ComPtr client; const HRESULT result = - ActivateProcessLoopback(GetCurrentProcessId(), completed, &client); - CloseHandle(completed); + ActivateProcessLoopback(GetCurrentProcessId(), exclude, &client); return SUCCEEDED(result) && client != nullptr; } @@ -233,32 +246,24 @@ bool SystemAudioAvailable() { return SUCCEEDED(ActivateEndpoint(&client)) && client != nullptr; } -HRESULT CaptureProcessAudio(DWORD pid, UINT64 expectedCreationTime, +HRESULT CaptureProcessAudio(DWORD pid, UINT64 expectedCreationTime, bool exclude, HANDLE stop_event, const StopProbe& stop_probe, const ReadyWriter& ready_writer, const PCMWriter& writer) { HRESULT com_result = CoInitializeEx(nullptr, COINIT_MULTITHREADED); if (FAILED(com_result)) return com_result; - HRESULT result = ValidateProcessTarget(pid, expectedCreationTime); - HANDLE process = nullptr; - HANDLE completed = nullptr; + // Validate and watch the same handle so PID reuse cannot change the target. + HANDLE process = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION | SYNCHRONIZE, FALSE, pid); + HRESULT result = process ? ValidateProcessTarget(process, expectedCreationTime) + : HRESULT_FROM_WIN32(GetLastError()); ComPtr client; if (SUCCEEDED(result)) { - process = OpenProcess(SYNCHRONIZE, FALSE, pid); - if (process == nullptr) result = HRESULT_FROM_WIN32(GetLastError()); - } - if (SUCCEEDED(result)) { - completed = CreateEventW(nullptr, FALSE, FALSE, nullptr); - if (completed == nullptr) result = HRESULT_FROM_WIN32(GetLastError()); - } - if (SUCCEEDED(result)) { - result = ActivateProcessLoopback(pid, completed, &client); + result = ActivateProcessLoopback(pid, exclude, &client); } if (SUCCEEDED(result)) { result = CaptureAudioFrames(client, process, true, stop_event, stop_probe, ready_writer, writer); } - if (completed != nullptr) CloseHandle(completed); if (process != nullptr) CloseHandle(process); client.Reset(); CoUninitialize(); diff --git a/native/capture/windows/process_audio.h b/native/capture/windows/process_audio.h index 55e0215..1090149 100644 --- a/native/capture/windows/process_audio.h +++ b/native/capture/windows/process_audio.h @@ -21,11 +21,11 @@ using PCMWriter = using ReadyWriter = std::function; using StopProbe = std::function; -bool ProcessAudioAvailable(); +bool ProcessAudioAvailable(bool exclude = false); bool SystemAudioAvailable(); HRESULT CaptureMicrophone(const std::wstring& device_id, HANDLE stop_event, const StopProbe& stop_probe, const ReadyWriter& ready_writer, const PCMWriter& writer); -HRESULT CaptureProcessAudio(DWORD pid, UINT64 expected_creation_time, +HRESULT CaptureProcessAudio(DWORD pid, UINT64 expected_creation_time, bool exclude, HANDLE stop_event, const StopProbe& stop_probe, const ReadyWriter& ready_writer, const PCMWriter& writer); diff --git a/scripts/embedded-sfu-gate.ts b/scripts/embedded-sfu-gate.ts index 007ade2..b18e8bd 100644 --- a/scripts/embedded-sfu-gate.ts +++ b/scripts/embedded-sfu-gate.ts @@ -65,6 +65,7 @@ async function main(): Promise { const chromePath = process.env.CHROME_PATH?.trim(); if (!chromePath) throw new Error("CHROME_PATH is required"); const nativeArm = process.env.PIIK_EMBEDDED_SFU_NATIVE === "true"; + const sfuOnly = process.env.PIIK_EMBEDDED_SFU_ONLY === "true"; const displayArm = process.env.PIIK_EMBEDDED_SFU_DISPLAY === "true"; if (displayArm && nativeArm) throw new Error("Display and Native capture are separate arms"); const twoRooms = process.env.PIIK_EMBEDDED_SFU_TWO_ROOMS === "true"; @@ -93,6 +94,7 @@ async function main(): Promise { const sourceProfile = nativeArm ? await mkdtemp(join(tmpdir(), "piik-client-media-")) : null; const [sourcePort, sourceDebugPort] = nativeArm ? [await reservePort(), await reservePort()] : [0, 0]; const result = { passed: false, arm: `${nativeArm ? "native" : displayArm ? "display" : "browser"}-${codec}-simulcast${twoRooms ? "-two-rooms" : ""}`, stage: "start", + sfuOnly, stereoSeparationDb: null as number[] | null, startedAt: new Date().toISOString(), finishedAt: "", processes: [] as Array<{ role: string; pid: number | null }>, high: null as { frames: number; width: number; height: number } | null, low: null as { frames: number; width: number; height: number } | null, @@ -144,7 +146,8 @@ async function main(): Promise { PIIK_ENV: "development", PIIK_DEBUG: "route", LISTEN_HOST: "127.0.0.1", PORT: String(serverPort), PIIK_LOG_DIR: join(BUILD_ROOT, "logs"), PUBLIC_BASE_URL: origin, ALLOWED_ORIGINS: origin, - SFU_LISTEN_HOST: "127.0.0.1", SFU_UDP_PORT: String(mediaPort), SFU_PUBLIC_IP: "127.0.0.1" }, + SFU_LISTEN_HOST: "127.0.0.1", SFU_UDP_PORT: String(mediaPort), SFU_PUBLIC_IP: "127.0.0.1", + SFU_ONLY: String(sfuOnly) }, }); processStarted("server", server); server.stdout.resume(); @@ -257,6 +260,7 @@ async function main(): Promise { result.stage = "decoded-audio-energy"; await waitForSample(() => call(viewer!, "gate.audioEnergy()", 3_000), (energy) => { result.audioEnergy.high = energy; return energy > 0; }, 5_000); + if (!nativeArm) result.stereoSeparationDb = await call(viewer, "gate.audioSeparation()", 5_000); result.host = await call(host, "gate.snapshot()"); if (displayArm) { result.stage = "display-background"; @@ -357,7 +361,9 @@ async function main(): Promise { } result.finishedAt = new Date().toISOString(); result.passed = !result.error && result.host?.published === true && result.host.simulcast && - result.viewer?.decoded === true && result.viewer.committed && result.viewer.peerFailures > 0 && + result.viewer?.decoded === true && result.viewer.committed && + (sfuOnly ? result.viewer.peerFailures === 0 : result.viewer.peerFailures > 0) && + (nativeArm || result.stereoSeparationDb?.every(db => db > 20) === true) && result.viewer.audioKbps > 0 && result.audioEnergy.high > 0 && result.audioEnergy.low > result.audioEnergy.high && result.high !== null && result.low !== null && result.audioEnergy.recovered > 0 && result.publicationRetired && result.subscriptionRecovered && result.cleanup && result.nativeShareStopped && diff --git a/scripts/embedded-sfu-page.ts b/scripts/embedded-sfu-page.ts index 83246c3..207de8e 100644 --- a/scripts/embedded-sfu-page.ts +++ b/scripts/embedded-sfu-page.ts @@ -239,18 +239,58 @@ export async function startHost(nativeSource?: { title: string; port: number }, video: { width: 1280, height: 720, frameRate: 15 }, audio: false, }) : canvas.captureStream(15); audio = new AudioContext(); - const tone = audio.createOscillator(); const gain = audio.createGain(); gain.gain.value = 0.05; const destination = audio.createMediaStreamDestination(); - tone.connect(gain).connect(destination); - tone.start(); + const channels = audio.createChannelMerger(2); + [500, 1500].forEach((frequency, channel) => { + const tone = audio!.createOscillator(); + tone.frequency.value = frequency; + tone.connect(channels, 0, channel); + tone.start(); + }); + channels.connect(gain).connect(destination); await audio.resume(); source.addTrack(destination.stream.getAudioTracks()[0]!); await connect(room, true); return room; } +// Check decoded channels, not just /2 in SDP (Opus also uses that for mono). +export async function audioSeparation(stream = video?.srcObject): Promise { + if (!(stream instanceof MediaStream) || !stream.getAudioTracks().length) throw new Error("No received audio"); + const context = new AudioContext(); + try { + const input = context.createMediaStreamSource(stream); + const splitter = context.createChannelSplitter(2); + const sink = context.createGain(); + sink.gain.value = 0; + sink.connect(context.destination); + input.connect(splitter); + const analyzers = [0, 1].map(channel => { + const analyzer = context.createAnalyser(); + analyzer.fftSize = 8192; + analyzer.smoothingTimeConstant = 0; + splitter.connect(analyzer, channel); + analyzer.connect(sink); + return analyzer; + }); + await context.resume(); + await new Promise(resolve => setTimeout(resolve, 1000)); + return analyzers.map((analyzer, channel) => { + const bins = new Float32Array(analyzer.frequencyBinCount); + analyzer.getFloatFrequencyData(bins); + const level = (frequency: number) => { + const bin = Math.round(frequency * analyzer.fftSize / context.sampleRate); + return Math.max(-120, ...bins.slice(bin - 1, bin + 2)); + }; + return level(channel === 0 ? 500 : 1500) - level(channel === 0 ? 1500 : 500); + }); + } finally { + await context.close(); + } +} + export async function startViewer(room: CreateRoomResponse): Promise { await connect(room, false); } diff --git a/src/client/App.tsx b/src/client/App.tsx index 7ce6a31..254f64b 100644 --- a/src/client/App.tsx +++ b/src/client/App.tsx @@ -397,8 +397,11 @@ function SiteAccessGate({ } return ( diff --git a/src/client/components/living/CaptureSourcePicker.tsx b/src/client/components/living/CaptureSourcePicker.tsx index be4e1c6..bd6cba3 100644 --- a/src/client/components/living/CaptureSourcePicker.tsx +++ b/src/client/components/living/CaptureSourcePicker.tsx @@ -32,6 +32,7 @@ export type NativeSourceList = sources: NativeCaptureTarget[]; processAudio: boolean; systemAudio: boolean; + processAudioExclusion?: boolean; captureBorderControl?: boolean; }; @@ -51,6 +52,7 @@ export function CaptureSourcePicker({ loadCameras, initialAudio = true, initialShowCaptureBorder = false, + initialExcludeAudio, audioLocked = false, selectionDisabled = false, }: { @@ -60,7 +62,7 @@ export function CaptureSourcePicker({ initialCamera?: string; activeCameraVideo?: HTMLVideoElement | null; loadCameras?: typeof loadCameraPreviews; - onNative: (target: NativeCaptureTarget, audio: boolean, showCaptureBorder: boolean) => void; + onNative: (target: NativeCaptureTarget, audio: boolean, showCaptureBorder: boolean, excludeAudio?: NativeCaptureTarget) => void; onPreview: ( target: NativeCaptureTarget, signal?: AbortSignal, @@ -72,6 +74,7 @@ export function CaptureSourcePicker({ initialTab?: SourceTab; initialAudio?: boolean; initialShowCaptureBorder?: boolean; + initialExcludeAudio?: NativeCaptureTarget; audioLocked?: boolean; selectionDisabled?: boolean; }) { @@ -81,6 +84,7 @@ export function CaptureSourcePicker({ const [tab, setTab] = useState(null); const [shareAudio, setShareAudio] = useState(initialAudio); const [showCaptureBorder, setShowCaptureBorder] = useState(initialShowCaptureBorder); + const [excludeAudio, setExcludeAudio] = useState(initialExcludeAudio); const appDetected = nativeSources.kind === "ready" || nativeSources.kind === "failed" || nativeSources.kind === "unsupported" || nativeSources.kind === "incompatible"; const tabs = SOURCE_TABS.filter((value) => appDetected || value === "browser" || value === "camera"); @@ -122,6 +126,14 @@ export function CaptureSourcePicker({ ? nativeSources.processAudio : nativeSources.systemAudio); const anyNativeAudio = sources.some(supportsAudio); + // Source enumeration already owns process identity; do not create a second app scanner. + const audioApplications = nativeSources.kind === "ready" + ? [...new Map(nativeSources.sources.filter(target => target.kind === "window") + .map(target => [`${target.pid}:${target.creationTime}`, target])).values()] : []; + const exclusionKey = excludeAudio?.kind === "window" ? `${excludeAudio.pid}:${excludeAudio.creationTime}` : ""; + const canExcludeAudio = activeTab === "display" && nativeSources.kind === "ready" && nativeSources.processAudioExclusion; + const missingExclusion = !!excludeAudio && !audioApplications.some(target => `${target.pid}:${target.creationTime}` === exclusionKey); + const exclusionUnavailable = activeTab === "display" && shareAudio && !!excludeAudio && (!canExcludeAudio || missingExclusion); const refreshing = activeTab === "camera" ? cameraSources.busy : nativeSources.kind === "loading"; const refreshLabel = t(refreshing ? "host.sourcePicker.loading" : "host.sourcePicker.refresh"); const audioAction = t(!anyNativeAudio ? "host.noAudio" : audioLocked @@ -300,12 +312,12 @@ export function CaptureSourcePicker({ key={nativeCaptureTargetKey(target)} target={target} disabled={ - selectionDisabled || (audioLocked && shareAudio && !supportsAudio(target)) + selectionDisabled || exclusionUnavailable || (audioLocked && shareAudio && !supportsAudio(target)) } onPreview={onPreview} onSelect={() => onNative(target, shareAudio && supportsAudio(target), - supportsCaptureBorder && showCaptureBorder) + supportsCaptureBorder && showCaptureBorder, target.kind === "display" ? excludeAudio : undefined) } /> )) @@ -352,6 +364,32 @@ export function CaptureSourcePicker({ ) : null} + {canExcludeAudio && shareAudio && anyNativeAudio ? ( +
{ + if (event.currentTarget.open) event.currentTarget.scrollIntoView({ block: "nearest" }); + }}> + + + {!vis && {t("host.sourcePicker.excludeAudio")}{excludeAudio ? ` · ${excludeAudio.title}` : ""}} + + +
+ + + {t(excludeAudio ? "host.sourcePicker.excludeAudioHint" : "host.sourcePicker.excludeAudioChoose")} + +
+
+ ) : null} ) : null} diff --git a/src/client/components/living/Couch.tsx b/src/client/components/living/Couch.tsx index e6fe09d..1ef91ca 100644 --- a/src/client/components/living/Couch.tsx +++ b/src/client/components/living/Couch.tsx @@ -25,19 +25,27 @@ export interface CouchHostEntry { onSelect?: () => void; } +export interface CouchProps { + view: "host" | "viewer"; + host?: CouchHostEntry | null; + entries: CouchEntry[]; + selectedKey?: string | null; + onSelect?: (key: string) => void; + participantAction?: { + open: (key: string, trigger: HTMLButtonElement) => void; + target: string | null; + controls: string; + }; +} + export function Couch({ view, host, entries, selectedKey, onSelect, -}: { - view: "host" | "viewer"; - host?: CouchHostEntry | null; - entries: CouchEntry[]; - selectedKey?: string | null; - onSelect?: (key: string) => void; -}) { + participantAction, +}: CouchProps) { const { t } = useCopy(); const [couchRef, containerWidth] = useElementWidth(); const count = entries.length + (host ? 1 : 0); @@ -76,15 +84,16 @@ export function Couch({ aria-label={`${t("common.host")} · ${t("common.viewers")}`} > {host ? ( - - {host.onSelect ? ( + + {host.onSelect || participantAction ? ( ); return ( - + {pawn} ); diff --git a/src/client/components/living/FloatingPanel.tsx b/src/client/components/living/FloatingPanel.tsx new file mode 100644 index 0000000..29c215e --- /dev/null +++ b/src/client/components/living/FloatingPanel.tsx @@ -0,0 +1,222 @@ +import { useLayoutEffect, useRef, useState, type PointerEvent, type ReactNode, type RefObject } from "react"; +import { useCopy } from "../../ui/copy"; +import { Glyph, type GlyphName } from "../../ui/icons"; +import "./floating-panel.css"; + +const PANEL_SIZES = { default: { width: 420, height: 560 }, large: { width: 560, height: 680 } }; +const COMPACT_SIZES = { default: { width: 304, height: undefined }, large: { width: 380, height: 460 } }; +type PanelGesture = "move" | "nw" | "se"; +type PanelChange = { x: number; y: number } | { rect: DOMRect; corner: "nw" | "se"; dx: number; dy: number }; + +/** Native visibility owns the shell; content owns its session, draft and reading position. */ +export function FloatingPanel({ id, trigger, title, icon, compact = false, anchorOnOpen = false, className = "", children, onOpenChange }: { + id: string; trigger: string | RefObject; title: string; icon: GlyphName; compact?: boolean; className?: string; + anchorOnOpen?: boolean; + children: ReactNode; onOpenChange?: (open: boolean) => void; +}) { + const { t, vis } = useCopy(); + const [open, setOpen] = useState(false); + const panelRef = useRef(null); + const positionMenu = useRef(null); + const sizes = compact ? COMPACT_SIZES : PANEL_SIZES; + const position = useRef({ x: 1, y: 1 }); + const preferredSize = useRef(sizes.default); + const placePanel = useRef<((change?: PanelChange) => void) | null>(null); + const gesture = useRef<{ pointerId: number; x: number; y: number; rect: DOMRect; kind: PanelGesture } | null>(null); + const suppressDragClick = useRef(false); + const getTrigger = () => typeof trigger === "string" ? document.getElementById(trigger) : trigger.current; + const focusTrigger = () => { + const element = getTrigger(); + if (element?.isConnected) element.focus({ preventScroll: true }); + }; + + function startGesture(event: PointerEvent, kind: PanelGesture) { + if (!event.isPrimary || event.button !== 0 || !panelRef.current) return; + if (kind === "move" && (event.target as Element).closest("button, .lr-floating-positions")) return; + panelRef.current.getAnimations().forEach(animation => animation.finish()); + gesture.current = { pointerId: event.pointerId, x: event.clientX, y: event.clientY, + rect: panelRef.current.getBoundingClientRect(), kind }; + // Capture on the summary preserves its native click/keyboard activation. + const handle = (event.target as Element).closest("summary") ?? event.currentTarget; + handle.focus({ preventScroll: true }); + handle.setPointerCapture(event.pointerId); + } + useLayoutEffect(() => { + const panel = panelRef.current; + if (!open || !panel) return; + // The visual viewport keeps the composer above a phone's keyboard without + // resizing the living room or taking ownership of scrolling/focus. + const viewport = window.visualViewport; + const place = (change?: PanelChange) => { + const layout = document.documentElement; + const width = viewport?.width ?? layout.clientWidth, height = viewport?.height ?? layout.clientHeight; + const maxWidth = Math.max(0, width - 24), maxHeight = Math.max(0, height - 24); + const left = (viewport?.offsetLeft ?? 0) + 12, top = (viewport?.offsetTop ?? 0) + 12; + let point = change && "x" in change ? change : undefined; + if (change && "corner" in change) { + const { rect, corner, dx, dy } = change; + const nw = corner === "nw"; + // Hold the opposite corner steady; viewport changes only constrain the + // rendered size, while deliberate resizing writes the room UI preference. + const availableWidth = nw ? rect.right - left : left + maxWidth - rect.left; + const availableHeight = nw ? rect.bottom - top : top + maxHeight - rect.top; + const w = Math.min(availableWidth, Math.max(Math.min(compact ? 264 : 320, maxWidth), rect.width + (nw ? -dx : dx))); + const h = Math.min(availableHeight, Math.max(Math.min(compact ? 240 : 360, maxHeight), rect.height + (nw ? -dy : dy))); + preferredSize.current = { width: w, height: h }; + point = { x: nw ? rect.right - w : rect.left, y: nw ? rect.bottom - h : rect.top }; + } + panel.style.width = `${Math.min(preferredSize.current.width, maxWidth)}px`; + panel.style.maxHeight = `${maxHeight}px`; + const preferredHeight = preferredSize.current.height; + panel.style.height = preferredHeight === undefined ? "auto" : `${Math.min(preferredHeight, maxHeight)}px`; + const freeX = Math.max(0, maxWidth - panel.offsetWidth), freeY = Math.max(0, maxHeight - panel.offsetHeight); + // Keep the chosen relative position when the keyboard or viewport changes. + // Only deliberate movement writes it; a narrow viewport must not erase it. + if (point) { + if (freeX) position.current.x = Math.max(0, Math.min(1, (point.x - left) / freeX)); + if (freeY) position.current.y = Math.max(0, Math.min(1, (point.y - top) / freeY)); + } + panel.style.inset = `${top + freeY * position.current.y}px auto auto ${left + freeX * position.current.x}px`; + }; + const resize = () => place(); + const hideOnEscape = (event: KeyboardEvent) => { + // Inner native surfaces consume Escape first, even after focus moves out. + if (event.key !== "Escape" || event.defaultPrevented || event.isComposing || + document.querySelector('[popover="auto"]:popover-open, dialog:modal')) return; + // Native top-layer order follows opening, not DOM order. Consume one + // close request even when chat and the reaction palette are both open. + const panels = [...document.querySelectorAll(".lr-floating-panel:popover-open")]; + if (!panels.length) return; + const topmost = panels.reduce((top, next) => Number(next.dataset.openedAt) >= Number(top.dataset.openedAt) ? next : top); + if (topmost !== panel) return; + event.preventDefault(); + const restoreFocus = panel.contains(document.activeElement); + panel.hidePopover(); + if (restoreFocus) focusTrigger(); + }; + placePanel.current = place; + place(); + // Contextual palettes start beside their invoker, then use the same free + // movement owner. Retargeting an open palette must not undo a user's drag. + const anchor = anchorOnOpen && getTrigger()?.getBoundingClientRect(); + if (anchor) { + const top = (viewport?.offsetTop ?? 0) + 12; + const bottom = top + (viewport?.height ?? document.documentElement.clientHeight) - 24; + const below = anchor.bottom + 8; + const headroom = anchor.top - top >= panel.offsetHeight + 136 ? 128 : 0; + place({ x: anchor.x + anchor.width / 2 - panel.offsetWidth / 2, + y: below + panel.offsetHeight <= bottom ? below : anchor.top - panel.offsetHeight - 8 - headroom }); + } + viewport?.addEventListener("resize", resize); + viewport?.addEventListener("scroll", resize); + window.addEventListener("resize", resize); + // Content-sized palettes still fit after a target, language or status change. + const observer = new ResizeObserver(resize); + observer.observe(panel); + document.addEventListener("keydown", hideOnEscape); + return () => { + placePanel.current = null; + gesture.current = null; + viewport?.removeEventListener("resize", resize); + viewport?.removeEventListener("scroll", resize); + window.removeEventListener("resize", resize); + observer.disconnect(); + document.removeEventListener("keydown", hideOnEscape); + }; + }, [open, trigger, compact, anchorOnOpen]); + + return (); +} diff --git a/src/client/components/living/LauncherForm.tsx b/src/client/components/living/LauncherForm.tsx index 06af218..7ba7b5e 100644 --- a/src/client/components/living/LauncherForm.tsx +++ b/src/client/components/living/LauncherForm.tsx @@ -112,7 +112,7 @@ export function LauncherForm({ key={choice.mode} type="button" role="radio" - className={`lr-client-mode${mode === choice.mode ? " is-selected" : ""}`} + className={`lr-tile lr-client-mode${mode === choice.mode ? " is-selected" : ""}`} aria-checked={mode === choice.mode} tabIndex={mode === choice.mode ? 0 : -1} aria-label={t(choice.label)} diff --git a/src/client/components/living/MicrophoneDeviceSelect.tsx b/src/client/components/living/MicrophoneDeviceSelect.tsx index 28760d9..cd07f41 100644 --- a/src/client/components/living/MicrophoneDeviceSelect.tsx +++ b/src/client/components/living/MicrophoneDeviceSelect.tsx @@ -3,7 +3,7 @@ import type { CaptureDevice } from "../../media/capture-devices"; import { useCopy } from "../../ui/copy"; import { Glyph } from "../../ui/icons"; -// Shared presentation only: capture and replacement stay with the Host owner. +// Shared presentation only: the caller owns capture and device replacement. export function MicrophoneDeviceSelect({ value, load, onChange, disabled, revision, browser = true }: { value: string; load: () => Promise; onChange: (id: string) => void; disabled?: boolean; revision?: unknown; browser?: boolean; diff --git a/src/client/components/living/Pawn.tsx b/src/client/components/living/Pawn.tsx index 75c8f7f..edb5341 100644 --- a/src/client/components/living/Pawn.tsx +++ b/src/client/components/living/Pawn.tsx @@ -7,7 +7,7 @@ export function PersonShape({ eyes = true, gaze = 0, eyeClassName, host }: { eyes?: boolean | "closed"; gaze?: number; eyeClassName?: string; host?: boolean; }) { return <> - + {eyes ? {eyes === "closed" ? - + ; } diff --git a/src/client/components/living/PlaybackControls.tsx b/src/client/components/living/PlaybackControls.tsx index 38cadac..c368a4d 100644 --- a/src/client/components/living/PlaybackControls.tsx +++ b/src/client/components/living/PlaybackControls.tsx @@ -1,4 +1,4 @@ -import { useEffect, useRef, useState, type RefObject } from "react"; +import { useEffect, useRef, useState, type RefObject, type ReactNode } from "react"; import { ViewerAudio, type ViewerAudioSnapshot } from "../../media/viewer-audio"; import { useCopy } from "../../ui/copy"; import { debugError } from "../../lib/debug"; @@ -15,6 +15,7 @@ import "./playback-controls.css"; export function PlaybackControls({ videoRef, stream, audioTrackKey, canPlay, theaterMode, onPlay, onToggleTheater, onReconnect, reconnectAvailable, + extraActions, }: { videoRef: RefObject; stream: MediaStream | null; @@ -25,6 +26,7 @@ export function PlaybackControls({ onToggleTheater: () => void; onReconnect: () => void; reconnectAvailable: boolean; + extraActions?: ReactNode; }) { const { t, vis } = useCopy(); const audioRef = useRef(null); @@ -146,6 +148,7 @@ export function PlaybackControls({ + {extraActions} : null; +} + +function ConnectedToggle({ session, caption }: { session: RoomInteractionSession; caption: boolean }) { + const { overlayEnabled } = useSyncExternalStore(session.subscribe, session.getSnapshot); + return session.setOverlayEnabled(!overlayEnabled)} />; +} + +export function RoomChatOverlay({ session, visible }: { session: RoomInteractionSession | null; visible: boolean }) { + return session ? : null; +} + +function ConnectedOverlay({ session, visible }: { session: RoomInteractionSession; visible: boolean }) { + const { overlayMessages } = useSyncExternalStore(session.subscribe, session.getSnapshot); + useEffect(() => { + const sync = () => session.setOverlayVisible(visible && !document.hidden); + sync(); + document.addEventListener("visibilitychange", sync); + return () => { + document.removeEventListener("visibilitychange", sync); + session.setOverlayVisible(false); + }; + }, [session, visible]); + return visible ? : null; +} + +function ChatFlight({ message, now }: { + message: ReturnType["overlayMessages"][number]; now: () => number; +}) { + return { + // Mounting or returning from reduced motion uses the remaining lifetime. + for (const animation of event.currentTarget.getAnimations()) { + if (animation instanceof CSSAnimation && animation.animationName === "lr-chat-flight") { + animation.currentTime = Math.min(CHAT_OVERLAY_DURATION_MS, Math.max(0, CHAT_OVERLAY_DURATION_MS - (message.expiresAt - now()))); + } + } + }}> + {message.payload.kind === "chat" && message.payload.text} + ; +} diff --git a/src/client/components/living/RoomInteractions.tsx b/src/client/components/living/RoomInteractions.tsx new file mode 100644 index 0000000..efe3c68 --- /dev/null +++ b/src/client/components/living/RoomInteractions.tsx @@ -0,0 +1,228 @@ +import { useEffect, useId, useLayoutEffect, useRef, useState, useSyncExternalStore } from "react"; +import { normalizeChatText, MAX_CHAT_CODE_POINTS, REACTION_IDS, isThrow } from "../../../shared/room-interactions"; +import { RoomInteractionSession } from "../../lib/room-interactions"; +import { useCopy } from "../../ui/copy"; +import { Glyph } from "../../ui/icons"; +import { Btn, SwitchItem } from "./primitives"; +import { participantColor } from "./participant-color"; +import { FloatingPanel } from "./FloatingPanel"; +import { PawnSvg } from "./Pawn"; +import { Couch, type CouchProps } from "./Couch"; +import { RoomReaction, ReactionIcon } from "./RoomReaction"; +import { RoomChatToggle } from "./RoomChatOverlay"; +import "./room-interactions.css"; + +interface Participant { peerId: string; displayName: string } +type Props = CouchProps & { session: RoomInteractionSession | null }; + +export function RoomInteractions(props: Props) { + return props.session ? : ; +} + +type ConnectedProps = Props & { session: RoomInteractionSession }; +function ConnectedRoomInteractions(props: ConnectedProps) { + const state = useSyncExternalStore(props.session.subscribe, props.session.getSnapshot); + return ; +} + +function InteractionPanel({ session, state, ...couch }: ConnectedProps & { + state: ReturnType; +}) { + const { t, vis } = useCopy(); + const [chatOpen, setChatOpen] = useState(false); + const [effects, setEffects] = useState(true); + const [target, setTarget] = useState(null); + const [draft, setDraft] = useState(""); + const [lastSeen, setLastSeen] = useState(null); + const [visible, setVisible] = useState(false); + const [following, setFollowing] = useState(true); + const area = useRef(null); + const anchor = useRef(null); + const log = useRef(null); + const id = useId(); + const menuId = `${id}-person`; + const participants: Participant[] = [ + ...(couch.host ? [{ peerId: couch.host.key, displayName: couch.host.name }] : []), + ...couch.entries.map(entry => ({ peerId: entry.key, displayName: entry.name })), + ]; + const latest = state.messages.at(-1)?.id ?? null; + const unseenIndex = state.messages.findIndex(message => message.id === lastSeen); + const unread = latest === lastSeen ? 0 : state.messages.length - unseenIndex - 1; + const canSend = state.ready && !state.pending && !state.coolingDown; + const person = participants.find(person => person.peerId === target); + const rosterKey = participants.map(person => person.peerId).join(","); + const inspect = target === couch.host?.key && couch.host.onSelect ? () => { if (!couch.host?.selected) couch.host?.onSelect?.(); } + : couch.entries.some(entry => entry.key === target && entry.selectable !== false) && couch.onSelect + ? () => { if (target !== couch.selectedKey) couch.onSelect?.(target!); } : undefined; + + useEffect(() => { + const confirmed = state.confirmed; + if (confirmed?.payload.kind === "chat") { + const text = confirmed.payload.text; + setDraft(current => normalizeChatText(current) === text ? "" : current); + } + }, [state.confirmed]); + useLayoutEffect(() => { + const popup = document.getElementById(menuId); + if (!popup) return; + if (person && anchor.current?.isConnected) popup.showPopover({ source: anchor.current }); + else popup.hidePopover(); + }, [target, rosterKey, menuId]); + + function openReactions(key: string, trigger: HTMLButtonElement) { + const popup = document.getElementById(menuId); + if (target === key && popup?.matches(":popover-open")) { popup.hidePopover(); return; } + anchor.current = trigger; + setTarget(key); + } + useEffect(() => { + if (!chatOpen || !log.current) return; + let active = true; + let inView = false; + const sync = () => { if (active) setVisible(inView && document.visibilityState === "visible"); }; + const observer = new IntersectionObserver(entries => { + const entry = entries.at(-1); + inView = Boolean(entry?.isIntersecting && entry.intersectionRatio > 0); + sync(); + }); + observer.observe(log.current); + document.addEventListener("visibilitychange", sync); + return () => { + active = false; + observer.disconnect(); + document.removeEventListener("visibilitychange", sync); + setVisible(false); + }; + }, [chatOpen]); + useEffect(() => { + if (!chatOpen || !visible || !following || document.visibilityState !== "visible" || + !log.current?.getClientRects().length) return; + setLastSeen(latest); + log.current.scrollTop = log.current.scrollHeight; + }, [latest, chatOpen, visible, following]); + + return
+
+ {state.reactions.map(({ id, sender, payload }) => payload.kind === "reaction" &&

+ {t(payload.targetPeerId ? "interaction.receivedReaction" : "interaction.expressedReaction", { + sender: sender.displayName, + reaction: t(`interaction.reaction.${payload.reaction}`), + target: participants.find(person => person.peerId === payload.targetPeerId)?.displayName ?? t("interaction.participant"), + })} +

)} +
+ +
+
{ const node = event.currentTarget; setFollowing(node.scrollHeight - node.scrollTop - node.clientHeight < 40); }}> + {state.messages.length === 0 ?
+

{vis ? "···" : t("interaction.empty")}

+
: state.messages.map((message, index) => { + const previous = state.messages[index - 1]; + const continuation = previous?.sender.peerId === message.sender.peerId && previous.sender.displayName === message.sender.displayName; + return
+ {!continuation && } +
+ {message.sender.displayName} + {message.sender.role === "host" && · {t("common.host")}} +

{message.payload.kind === "chat" && message.payload.text}

+
+
; + })} +
+ {!following && state.messages.length > 0 && } +
+
{ + event.preventDefault(); + const text = normalizeChatText(draft); + if (text && canSend) { setFollowing(true); session.send({ kind: "chat", text }); } + }}> + setDraft(event.target.value)} maxLength={MAX_CHAT_CODE_POINTS * 2} + aria-label={t("interaction.placeholder")} placeholder={vis ? "···" : t("interaction.placeholder")} + autoComplete="off" + onKeyDown={event => { if (event.key === "Enter" && event.nativeEvent.isComposing) event.preventDefault(); }} /> + + +
+ +
+ {state.error ? <>{t(`interaction.error.${state.error}`)} + : !state.ready ? <>{t("interaction.reconnecting")} + : !vis && {t("interaction.ephemeral")}} +
+ {draft && } +
+
+
+ + {effects && state.reactions.map((reaction, index) => { + const payload = reaction.payload; + if (payload.kind !== "reaction") return null; + // Only one's own expression replaces one's previous expression. + // Receiving a gift never speaks for that person or hides their reaction. + const covered = !payload.targetPeerId && !isThrow(payload.reaction) && state.reactions.slice(index + 1).some(next => + next.payload.kind === "reaction" && !next.payload.targetPeerId && !isThrow(next.payload.reaction) && + next.sender.peerId === reaction.sender.peerId); + if (covered) return null; + return participants.some(person => person.peerId === reaction.sender.peerId) && + (!payload.targetPeerId || participants.some(person => person.peerId === payload.targetPeerId)) && + ; + })} +
+
+ + + {unread > 0 && {unread}} + +
+ { if (!open) setTarget(null); }}> +
+
+ + {person + ? + : } + {!vis && {t(target === state.peerId ? "interaction.express" : "interaction.toPerson")}} + {person?.displayName ?? t("interaction.participant")} + +
+ {[false, true].filter(throws => !throws || (person && target !== state.peerId)).map(throws =>
+ {REACTION_IDS.filter(reaction => isThrow(reaction) === throws).map(reaction => )} +
)} + {state.error || !state.ready ?
+ + {t(state.error ? `interaction.error.${state.error}` : "interaction.reconnecting")} +
: null} + {inspect && } + {target === state.peerId &&
+ + {vis && } +
} +
+
+
; +} diff --git a/src/client/components/living/RoomReaction.tsx b/src/client/components/living/RoomReaction.tsx new file mode 100644 index 0000000..51d1838 --- /dev/null +++ b/src/client/components/living/RoomReaction.tsx @@ -0,0 +1,119 @@ +import { useLayoutEffect, useRef, type CSSProperties, type RefObject } from "react"; +import { REACTION_DURATION_MS, REACTION_GLYPHS, isThrow, type ReactionId } from "../../../shared/room-interactions"; +import type { RoomInteraction } from "../../lib/room-interactions"; +import { identityHash } from "../../lib/identity-hash"; +import { Glyph } from "../../ui/icons"; + +/** Props use the same small silhouette in the picker and on the couch. */ +export function ReactionIcon({ reaction }: { reaction: ReactionId }) { + if (!isThrow(reaction)) return ; + return ; +} + +/** Fixed, event-seeded accents: no independent timer or mutable random stream. */ +function ReactionAccents({ reaction, seed }: { reaction: ReactionId; seed: number }) { + return + {reaction === "tomato" && } + {Array.from({ length: reaction === "sleep" ? 3 : 5 }, (_, index) => { + const angle = (-160 + index * 35 + (seed % 13)) * Math.PI / 180; + const radius = 27 + ((seed >>> (index * 4)) % 12); + return + {reaction === "heart" ? + : reaction === "sleep" ? z + : reaction === "poop" ? + : reaction === "tomato" ? + : reaction === "party" ? + : } + ; + })} + ; +} + +/** Event expiry owns lifetime. This component owns only placement and its own impact. */ +export function RoomReaction({ reaction, area, now }: { + reaction: RoomInteraction & { expiresAt: number }; area: RefObject; now: () => number; +}) { + const flight = useRef(null); + const payload = reaction.payload; + useLayoutEffect(() => { + const root = area.current; + const element = flight.current; + if (!root || !element || payload.kind !== "reaction") return; + const pawn = (peer: string) => root.querySelector(`[data-room-peer="${CSS.escape(peer)}"] .lr-person`); + const sender = pawn(reaction.sender.peerId); + const recipient = pawn(payload.targetPeerId ?? reaction.sender.peerId); + if (!sender || !recipient) return; + let frame = 0; + let impact: Animation | undefined; + const place = () => { + if (now() >= reaction.expiresAt || !sender.isConnected || !recipient.isConnected) { + element.style.visibility = "hidden"; + return; + } + frame = requestAnimationFrame(place); + if (document.hidden || !root.getClientRects().length) { + element.style.visibility = "hidden"; + impact?.cancel(); + return; + } + const rect = element.getBoundingClientRect(); + // Transforms do not trigger ResizeObserver. Follow the actual head for + // this event's remaining lifetime, including hover, lean and impact. + const from = sender.querySelector(".lr-person-head")?.getBoundingClientRect(); + const to = recipient.querySelector(".lr-person-head")?.getBoundingClientRect(); + if (!from || !to) { element.style.visibility = "hidden"; return; } + element.style.setProperty("--from-x", `${from.x - rect.x + from.width / 2}px`); + element.style.setProperty("--from-y", `${from.y - rect.y + from.height / 2}px`); + element.style.setProperty("--to-x", `${to.x - rect.x + to.width / 2}px`); + element.style.setProperty("--to-y", `${to.y - rect.y + to.height / 2}px`); + element.style.setProperty("--throw-direction", from.x > to.x ? "-1" : "1"); + element.style.visibility = "visible"; + }; + place(); + const body = recipient.querySelector(".lr-reaction-body"); + if (isThrow(payload.reaction) && payload.targetPeerId && body && + !window.matchMedia("(prefers-reduced-motion: reduce)").matches && !root.closest("[data-comic-reduced-motion]")) { + const elapsed = Math.max(0, REACTION_DURATION_MS - (reaction.expiresAt - now())); + const delay = REACTION_DURATION_MS * .32 - elapsed; + if (delay > -560) { + const direction = sender.getBoundingClientRect().x > recipient.getBoundingClientRect().x ? -1 : 1; + impact = body.animate([ + { transform: "none" }, + { transform: payload.reaction === "tomato" ? `rotate(${direction * 6}deg) scale(1.05,.95)` : `translateX(${direction * 4}px) rotate(${direction * 8}deg)` }, + { transform: `rotate(${-direction * 3}deg)` }, + { transform: "none" }, + ], { duration: 560, delay, easing: "ease-out" }); + } + } + return () => { cancelAnimationFrame(frame); impact?.cancel(); }; + }, [area, reaction, payload, now]); + if (payload.kind !== "reaction") return null; + const gift = Boolean(payload.targetPeerId && !isThrow(payload.reaction)); + return ; +} diff --git a/src/client/components/living/Tooltip.tsx b/src/client/components/living/Tooltip.tsx index 75a14fb..27ae5e0 100644 --- a/src/client/components/living/Tooltip.tsx +++ b/src/client/components/living/Tooltip.tsx @@ -294,15 +294,17 @@ export function Tooltip({ if (!interactionOpen) return; // Hover does not move keyboard focus into the trigger. const dismiss = (event: KeyboardEvent) => { - if (event.key !== "Escape") return; + if (event.key !== "Escape" || event.isComposing) return; // Close every hint channel, including a hover panel suppressed by a // focused neighbour. Only visible guidance consumes the outer action. const style = tipRef.current && getComputedStyle(tipRef.current); if (style?.visibility === "visible" && style.pointerEvents !== "none") event.preventDefault(); dismissPanel(); }; - document.addEventListener("keydown", dismiss); - return () => document.removeEventListener("keydown", dismiss); + // Guidance consumes Escape before its containing window, regardless of + // which surface mounted first. Outer handlers respect defaultPrevented. + document.addEventListener("keydown", dismiss, true); + return () => document.removeEventListener("keydown", dismiss, true); }, [interactionOpen]); useEffect(() => { diff --git a/src/client/components/living/floating-panel.css b/src/client/components/living/floating-panel.css new file mode 100644 index 0000000..b17fa80 --- /dev/null +++ b/src/client/components/living/floating-panel.css @@ -0,0 +1,33 @@ +.lr-floating-surface { color: var(--ink); background: var(--paper); border: 1px solid var(--control-edge); border-radius: 24px; box-shadow: 0 12px 36px rgb(16 26 44 / .2), var(--control-shadow); } +.lr-floating-panel { position: fixed; inset: auto 12px 12px auto; max-width: calc(100vw - 24px); max-height: calc(100dvh - 24px); margin: 0; padding: 16px; overflow: auto; overscroll-behavior: contain; transform-origin: right bottom; } +.lr-floating-panel:popover-open { display: flex; flex-direction: column; animation: lr-floating-enter 180ms var(--spring); } +.lr-floating-panel:focus { outline: none; } +.lr-floating-panel:focus-visible { outline: 2px solid var(--action); outline-offset: -2px; } +.lr-floating-header { display: flex; align-items: center; gap: 8px; min-height: 56px; flex-shrink: 0; margin: -16px -16px 0; padding: 6px 10px 6px 24px; border-bottom: 1px solid var(--wall-2); border-radius: 24px 24px 0 0; cursor: grab; user-select: none; touch-action: none; } +.lr-floating-header:active { cursor: grabbing; } +.lr-floating-header:focus-visible { outline: 2px solid var(--action); outline-offset: -3px; } +.lr-floating-title { display: flex; flex: 1; min-width: 0; align-items: center; gap: 7px; font-size: 14px; } +.lr-floating-title svg { flex: none; color: var(--muted); } +.lr-floating-title b { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } +.lr-floating-position { position: relative; flex: none; } +.lr-floating-position > summary { display: flex; align-items: center; gap: 5px; min-height: 40px; padding: 0 9px; border-radius: 12px; font-size: 12px; color: var(--muted); cursor: pointer; list-style: none; } +.lr-floating-position > summary::-webkit-details-marker { display: none; } +.lr-floating-position > summary:is(:hover, :focus-visible), .lr-floating-position[open] > summary { color: var(--ink); background: var(--control-hover); } +.lr-floating-positions { position: absolute; z-index: 2; top: calc(100% + 6px); right: 0; display: grid; grid-template-columns: 1fr 1fr; gap: 6px; width: 180px; padding: 8px; border: 1px solid var(--control-edge); border-radius: 18px; background: var(--paper); box-shadow: var(--control-shadow), 0 8px 24px #0002; } +.lr-floating-positions button { display: grid; justify-items: center; gap: 6px; min-height: 60px; padding: 8px 4px; border: 0; border-radius: 10px; color: var(--ink); background: transparent; font-size: 11px; } +.lr-floating-positions button:is(:hover, :focus-visible) { color: var(--action); background: var(--control-hover); } +.lr-floating-positions .lr-floating-size { border-top: 1px solid var(--wall-2); } +.lr-floating-resize { position: absolute; z-index: 3; display: grid; place-items: center; width: 28px; height: 28px; color: var(--muted); cursor: nwse-resize; touch-action: none; user-select: none; } +.lr-floating-resize.is-nw { top: 1px; left: 1px; transform: rotate(180deg); } +.lr-floating-resize.is-se { right: 1px; bottom: 1px; } +.lr-floating-resize:is(:hover, :focus-visible) { color: var(--action); } +.lr-floating-resize:focus-visible { outline: 2px solid var(--action); outline-offset: -4px; border-radius: 10px; } +@keyframes lr-floating-enter { from { opacity: 0; transform: translateY(8px) scale(.97); } to { opacity: 1; transform: none; } } +.lr-popover-close { display: grid; place-items: center; width: 36px; height: 36px; flex: none; background: transparent; color: var(--muted); border: 0; border-radius: 12px; cursor: pointer; } +.lr-popover-close:is(:hover, :focus-visible) { background: var(--control-hover); color: var(--ink); } +[data-comic-reduced-motion] .lr-floating-panel:popover-open { animation: none; } +@media (prefers-reduced-motion: reduce) { .lr-floating-panel:popover-open { animation: none; } } +@media (max-width: 420px) { + .lr-floating-panel { padding: 12px; } + .lr-floating-header { margin: -12px -12px 0; padding: 6px 10px 6px 24px; } +} diff --git a/src/client/components/living/hints/index.tsx b/src/client/components/living/hints/index.tsx index 85474a5..5cc31be 100644 --- a/src/client/components/living/hints/index.tsx +++ b/src/client/components/living/hints/index.tsx @@ -1,13 +1,13 @@ // Control hint scenes (操作提示漫画): small explanatory scenes used // as hover/focus tooltips in every mode, with captions added by Tooltip. // Same cast, palette, 320x96 canvas, and motion -// constitution as the state comics. Sets live in set1..set4.tsx; -// each must define exactly its SetNKind keys — the merge below then proves +// constitution as the state comics. Each scene group defines exactly its +// own HintKind subset — the merge below then proves // completeness at compile time. JSX-free shell: this module is also loaded // by the tsx preview harness, which uses the classic JSX runtime. import { createElement, memo, type CSSProperties } from "react"; -import type { HintKind, HintScene, MetricHintKind } from "../../../ui/visual-kinds"; +import type { HintKind, HintScene, MetricHintKind, InteractionHintKind } from "../../../ui/visual-kinds"; import { SET1_SCENES } from "./set1"; import { SET2_SCENES } from "./set2"; import { SET3_SCENES } from "./set3"; @@ -16,6 +16,7 @@ import { PLAYBACK_SCENES } from "./playback"; import { CONTROL_SCENES } from "./controls"; import { METRIC_SCENES } from "./metrics"; import { ADMISSION_SCENES } from "./admission"; +import { INTERACTION_SCENES } from "./interactions"; import { comicStyle, getComicPresentation, type ComicTone, type ComicMotion } from "../comic-presentation"; export type { HintKind, HintScene }; @@ -99,6 +100,7 @@ export const HINT_KINDS: readonly HintKind[] = [ "hint-pip-exit", "hint-pip-unavailable", ...Object.keys(METRIC_SCENES) as MetricHintKind[], + ...Object.keys(INTERACTION_SCENES) as InteractionHintKind[], ]; export const HINT_SCENES: Record = { @@ -110,6 +112,7 @@ export const HINT_SCENES: Record = { ...CONTROL_SCENES, ...METRIC_SCENES, ...ADMISSION_SCENES, + ...INTERACTION_SCENES, }; const HINT_KIND_SET: ReadonlySet = new Set(HINT_KINDS); diff --git a/src/client/components/living/hints/interactions.tsx b/src/client/components/living/hints/interactions.tsx new file mode 100644 index 0000000..c8f330b --- /dev/null +++ b/src/client/components/living/hints/interactions.tsx @@ -0,0 +1,59 @@ +import { FAINT, Frame, MiniTv, Pawn, SKY, YOU, rmBlock } from "../Comic"; +import { Glyph } from "../../../ui/icons"; +import type { ComicTheme, HintScene, InteractionHintKind } from "../../../ui/visual-kinds"; + +function InteractionMotion() { + return ; +} + +function Bubble({ x, y, color = YOU }: { x: number; y: number; color?: string }) { + return + + + ; +} + +function ChatHint({ theme, send = false }: { theme: ComicTheme; send?: boolean }) { + return <>{[0, 160].map((x, index) => + + + + {send || index ? : } + {Boolean(index) && } + )}; +} + +// The picture and chat stay present in both panels; only on-picture text changes. +function OverlayHint({ theme, show }: { theme: ComicTheme; show: boolean }) { + return <>{[0, 160].map((x, index) => + + + + + + + + + + )}; +} + +export const INTERACTION_SCENES: Record = { + "hint-chat-open": props => , + "hint-chat-send": props => , + "hint-chat-overlay-show": props => , + "hint-chat-overlay-hide": props => , +}; diff --git a/src/client/components/living/participant-motion.ts b/src/client/components/living/participant-motion.ts index c3e4679..57d3464 100644 --- a/src/client/components/living/participant-motion.ts +++ b/src/client/components/living/participant-motion.ts @@ -1,8 +1,8 @@ +import { identityHash } from "../../lib/identity-hash"; + /** UUID selects a quiet cadence; wall time keeps independently mounted views in phase. */ export function participantMotion(identity: string) { - let seed = 2166136261; - for (const char of identity) seed = Math.imul(seed ^ char.charCodeAt(0), 16777619); - seed >>>= 0; + const seed = identityHash(identity); return { duration: 42_000 + (seed % 17_003), offset: (seed >>> 8) % 59_003, diff --git a/src/client/components/living/primitives.tsx b/src/client/components/living/primitives.tsx index 40f5289..153bb5b 100644 --- a/src/client/components/living/primitives.tsx +++ b/src/client/components/living/primitives.tsx @@ -49,6 +49,7 @@ export function Btn({ onClick, type = "button", hint, + hintText, hintTone, hintMotion, draw, @@ -68,6 +69,8 @@ export function Btn({ type?: "button" | "submit"; /** Shared hint comic; text modes add the localized caption. */ hint?: ComicKind | HintKind; + /** Optional explanation beyond the control's accessible name. */ + hintText?: CopyKey; hintTone?: ComicTone; hintMotion?: ComicMotion; /** Draw-in spot id for state-beat toggle icons (see Glyph). */ @@ -114,7 +117,7 @@ export function Btn({ ); return hint ? ( - {button} + {button} ) : ( button ); diff --git a/src/client/components/living/room-chat-overlay.css b/src/client/components/living/room-chat-overlay.css new file mode 100644 index 0000000..b1a7e25 --- /dev/null +++ b/src/client/components/living/room-chat-overlay.css @@ -0,0 +1,25 @@ +.lr-tv-screen:has(> .lr-chat-overlay) { container-type: inline-size; } +.lr-chat-overlay { position: absolute; inset: 0; overflow: hidden; pointer-events: none; container-type: size; } +.lr-tv-screen:has(> .lr-playback) > .lr-chat-overlay { grid-area: 1 / 1 / 2 / 2; } +.lr-chat-flight { + position: absolute; left: 100%; top: calc(12px + var(--chat-lane) * 30px); + max-width: 80%; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; + padding: 2px 10px; border-radius: 999px; background: rgb(15 25 35 / .68); + color: #fff; font-size: clamp(14px, 2cqw, 18px); font-weight: 600; line-height: 24px; + text-shadow: 0 1px 2px rgb(0 0 0 / .4); + animation: lr-chat-flight linear both; +} +@keyframes lr-chat-flight { to { transform: translateX(calc(-100cqw - 100%)); } } +/* Fit complete 28px lines above playback controls. Visibility preserves the + running flight when the available picture space changes. */ +@container (height < 100px) { .lr-chat-flight[data-lane="2"] { visibility: hidden; } } +@container (height < 70px) { .lr-chat-flight[data-lane="1"] { visibility: hidden; } } +@container (height < 40px) { .lr-chat-flight { visibility: hidden; } } +.lr-chat-overlay-toggle { display: inline-flex; } +@container (max-width: 440px) { + .lr-playback:has(.lr-chat-overlay-toggle) .lr-btn { width: clamp(28px, 9cqw, 40px); } +} +[data-comic-reduced-motion] .lr-chat-flight { animation: none; left: 12px; } +@media (prefers-reduced-motion: reduce) { + .lr-chat-flight { animation: none; left: 12px; } +} diff --git a/src/client/components/living/room-interactions.css b/src/client/components/living/room-interactions.css new file mode 100644 index 0000000..4251464 --- /dev/null +++ b/src/client/components/living/room-interactions.css @@ -0,0 +1,195 @@ +.lr-room-interactions { position: relative; width: min(960px, 100%); margin: 0 auto; } +.lr-interaction-couch { position: relative; } +.lr-interaction-toggle { display: flex; justify-content: center; align-items: center; gap: 2px; width: fit-content; margin: 12px auto 4px; padding: 3px; background: var(--paper); border: 1px solid var(--control-edge); border-radius: 18px; } +.lr-interaction-chat-toggle { display: flex; position: relative; } +.lr-interaction-toggle .lr-btn { min-height: 44px; } +.lr-interaction-unread { align-self: flex-start; margin: -4px 0 0 -8px; z-index: 1; min-width: 22px; padding: 2px 6px; border-radius: 20px; color: var(--paper); background: var(--action); font: 700 11px var(--mono); text-align: center; } +.lr-chat-compose, .lr-chat-footer { flex-shrink: 0; } +.lr-interaction-effects { display: flex; align-items: center; gap: 8px; min-height: 40px; width: fit-content; margin-top: 8px; font-size: 12px; font-weight: 400; color: var(--muted); } +.lr-chat-history { position: relative; flex: 1; min-height: 96px; } +.lr-chat-log { height: 100%; overflow: auto; overscroll-behavior: contain; scrollbar-width: thin; scrollbar-gutter: stable; padding: 18px 2px; } +.lr-chat-latest { position: absolute; bottom: 8px; left: 50%; translate: -50% 0; display: flex; align-items: center; gap: 6px; min-height: 36px; max-width: calc(100% - 16px); padding: 6px 12px; border: 1px solid var(--control-edge); border-radius: 18px; background: var(--paper); color: var(--ink); box-shadow: var(--control-shadow); font-size: 12px; white-space: nowrap; } +.lr-chat-latest:is(:hover, :focus-visible) { background: var(--control-hover); } +.lr-chat-empty { display: grid; place-content: center; justify-items: center; gap: 10px; height: 100%; color: var(--muted); font-size: 13px; } +.lr-chat-empty p { margin: 0; } +.lr-chat-welcome { position: relative; display: flex; gap: 4px; padding-top: 20px; } +.lr-chat-welcome > svg:last-child { position: absolute; top: 0; right: 0; color: var(--action); } +.lr-chat-message { position: relative; display: flex; padding: 0 20px 0 34px; margin: 12px 0 0; } +.lr-chat-message:first-child { margin-top: 0; } +.lr-chat-message.is-self { justify-content: flex-end; padding: 0 34px 0 20px; } +.lr-chat-message.is-continuation { margin-top: 4px; } +.lr-chat-avatar { position: absolute; top: 1px; left: 0; } +.lr-chat-avatar .lr-person { width: 28px; height: 34px; } +.lr-chat-message.is-self .lr-chat-avatar { left: auto; right: 0; } +.lr-chat-bubble { min-width: 0; } +.lr-chat-author { display: block; margin: 0 4px 5px; font-size: 11px; color: var(--muted); overflow-wrap: anywhere; } +.lr-chat-message.is-self .lr-chat-author { text-align: right; } +.lr-chat-message p { padding: 9px 13px; margin: 0; border-radius: 4px 16px 16px; background: var(--wall); overflow-wrap: anywhere; white-space: pre-wrap; font-size: 14px; line-height: 1.6; } +.lr-chat-message.is-self p { background: color-mix(in srgb, var(--action) 11%, var(--paper)); border-radius: 16px 4px 16px 16px; } +.lr-chat-message.is-continuation p { border-radius: 16px; } +.lr-chat-compose { display: flex; align-items: center; gap: 4px; margin-top: 8px; padding: 4px; border: 1px solid var(--control-edge); border-radius: 20px; background: var(--wall); } +.lr-chat-compose:has(input:focus-visible) { outline: 3px solid var(--action); outline-offset: 2px; } +.lr-chat-compose > input { min-width: 0; flex: 1; width: 0; height: 44px; border: 0; outline: none; border-radius: 14px; padding: 0 10px; background: transparent; color: var(--ink); font-size: 16px; } +.lr-chat-compose .lr-btn { flex: none; width: 44px; height: 44px; min-width: 44px; min-height: 44px; padding: 0; border-radius: 15px; } +.lr-chat-footer { display: flex; align-items: center; gap: 10px; min-height: 40px; margin-top: 8px; padding-right: 12px; color: var(--muted); font-size: 11px; line-height: 1.5; } +.lr-chat-footer .lr-chat-overlay-toggle { flex: none; } +.lr-chat-footer .lr-btn { min-height: 36px; padding: 0 8px; } +.lr-chat-footer .lr-interaction-status { flex: 1; min-width: 0; min-height: 0; margin: 0; } +.lr-chat-count { flex: none; margin-left: auto; font-variant-numeric: tabular-nums; } +.lr-interaction-status { display: flex; align-items: center; gap: 6px; min-height: 24px; margin-top: 6px; font-size: 11px; color: var(--muted); } +.lr-person-menu { transform-origin: center bottom; } +.lr-person-menu-body { flex: 1; min-height: 0; padding-top: 12px; overflow: auto; overscroll-behavior: contain; scrollbar-width: thin; } +.lr-person-menu-target { margin-bottom: 10px; } +.lr-person-menu-name { display: flex; align-items: center; gap: 10px; min-width: 0; } +.lr-person-menu-name > span:last-child { min-width: 0; } +.lr-person-menu-name small { display: block; color: var(--muted); font-size: 11px; line-height: 1.6; } +.lr-person-menu-name b { display: block; font-size: 14px; font-weight: 650; overflow-wrap: anywhere; } +.lr-person-menu-portrait { display: grid; place-items: center; width: 34px; height: 40px; flex: none; } +.lr-person-menu-portrait .lr-person { width: 30px; height: 36px; } +.lr-reaction-grid { display: grid; grid-template-columns: repeat(5, minmax(0, 1fr)); gap: 4px; } +.lr-reaction-props { display: grid; grid-template-columns: 1fr 1fr; gap: 8px; margin-top: 10px; } +:is(.lr-reaction-grid, .lr-reaction-props) button { display: flex; align-items: center; justify-content: center; gap: 6px; min-height: 44px; padding: 4px; border: 1px solid transparent; background: transparent; color: inherit; border-radius: 14px; font-size: 26px; cursor: pointer; touch-action: manipulation; transition: background-color 140ms ease, border-color 140ms ease; } +.lr-reaction-props button { background: var(--wall); } +.lr-reaction-props button > span { font-size: 12px; } +:is(.lr-reaction-grid, .lr-reaction-props) button[aria-disabled="true"] { opacity: .4; cursor: default; } +:is(.lr-reaction-grid, .lr-reaction-props) button > :first-child { transition: transform 180ms var(--spring); } +:is(.lr-reaction-grid, .lr-reaction-props) button:is(:hover, :focus-visible):not([aria-disabled="true"]) { background: var(--control-hover); border-color: var(--action); } +:is(.lr-reaction-grid, .lr-reaction-props) button:is(:hover, :focus-visible):not([aria-disabled="true"]) > :first-child { transform: translateY(-2px); } +:is(.lr-reaction-grid, .lr-reaction-props) button:active:not([aria-disabled="true"]) > :first-child { transform: scale(.9); } +.lr-person-menu-details { display: flex; align-items: center; justify-content: center; gap: 8px; width: 100%; min-height: 40px; padding: 8px; margin-top: 8px; border: 0; border-radius: 12px; background: transparent; color: var(--muted); font-size: 12px; cursor: pointer; } +.lr-person-menu-details:is(:hover, :focus-visible) { background: var(--control-hover); color: var(--ink); } +.lr-person-menu .lr-interaction-effects { font-size: 11px; } +/* Theater keeps the same top-layer chat; the picture never changes size. */ +.lr-room.is-theater .lr-room-interactions { position: absolute; top: 12px; right: 12px; z-index: 5; width: auto; } +.lr-room.is-theater .lr-interaction-toggle { margin: 0 0 0 auto; } +.lr-room.is-theater .lr-interaction-couch { display: none; } +.lr-pawn[aria-expanded="true"] { background: color-mix(in srgb, var(--paper) 72%, transparent); border-radius: 18px; } +.lr-reaction-flight { position: absolute; inset: 0; pointer-events: none; z-index: 5; overflow: visible; } +/* The path positions the center; percentage translation centers any glyph size. */ +.lr-reaction-projectile { position: absolute; left: 0; top: 0; display: flex; width: max-content; translate: -50% -50%; font-size: 30px; line-height: 1; animation: lr-reaction-flight var(--reaction-duration) ease-out both; } +.lr-reaction-glyph { display: flex; animation: lr-reaction-wiggle var(--reaction-duration) ease-out both; } +.lr-reaction-glyph > svg { width: 34px; height: 34px; } +.is-tomato .lr-reaction-glyph { animation-name: lr-reaction-squash; } +.is-poop .lr-reaction-glyph { animation-name: lr-reaction-plop; } +:is(.is-clap, .is-laugh, .is-party) .lr-reaction-glyph { animation-name: lr-reaction-bounce; } +:is(.is-heart, .is-wow, .is-fire, .is-star) .lr-reaction-glyph { animation-name: lr-reaction-pulse; } +.is-eyes .lr-reaction-glyph { animation-name: lr-reaction-peek; } +.is-sleep .lr-reaction-glyph { animation-name: lr-reaction-doze; } +.lr-reaction-accents { position: absolute; left: 50%; top: 50%; translate: -50% -50%; overflow: visible; fill: #efb94e; } +.lr-reaction-accents.is-heart { fill: #e77585; } +.lr-reaction-accents.is-tomato { fill: #e96c55; } +.lr-reaction-accents.is-poop { fill: #b09b82; opacity: .65; } +.lr-reaction-accents.is-fire { fill: #ed9b58; } +.lr-reaction-accents:is(.is-eyes, .is-sleep) { fill: var(--muted); } +.lr-reaction-accents.is-sleep { font: 700 12px var(--mono); } +.lr-reaction-accents.is-party > :nth-child(2n) { fill: #65b7aa; } +.lr-reaction-accents.is-party > :nth-child(3n) { fill: #e77585; } +.lr-reaction-spark { animation: lr-reaction-spark-self var(--reaction-duration) ease-out both; } +.is-gift .lr-reaction-spark { animation-name: lr-reaction-spark-gift; } +.is-throw .lr-reaction-spark { animation-name: lr-reaction-spark-impact; } +.is-expression .lr-reaction-projectile { animation-name: lr-reaction-expression; } +.is-gift .lr-reaction-projectile { animation-name: lr-reaction-gift; } +.is-expression .is-sleep .lr-reaction-spark { animation-name: lr-reaction-drift; } +.lr-reaction-impact { fill: none; stroke: #e96c55; stroke-width: 1.5; transform-box: fill-box; transform-origin: center; animation: lr-reaction-impact var(--reaction-duration) ease-out both; } +/* Attribution follows the glyph without shifting its center off the head. */ +.lr-reaction-from { position: absolute; bottom: calc(100% + 4px); left: 50%; translate: -50% 0; display: flex; align-items: center; gap: 4px; width: max-content; padding: 4px 6px; border: 1px solid var(--control-edge); border-radius: 10px; background: var(--paper); color: var(--ink); box-shadow: var(--control-shadow); font-size: 11px; font-weight: 600; } +.lr-reaction-from > span { max-width: 80px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } +.lr-reaction-from > svg { flex: none; color: var(--muted); } +.lr-reaction-body { transform-box: fill-box; transform-origin: center bottom; } +@keyframes lr-reaction-flight { + 0% { opacity: 0; transform: translate(var(--from-x), var(--from-y)) scale(.3); } + 10% { opacity: 1; } + 22% { transform: translate(calc((var(--from-x) + var(--to-x)) / 2), calc(min(var(--from-y), var(--to-y)) - 48px)) rotate(calc(-15deg * var(--throw-direction))) scale(1); } + 32% { transform: translate(var(--to-x), var(--to-y)) rotate(calc(8deg * var(--throw-direction))) scale(1.12, .85); } + 44% { transform: translate(var(--to-x), calc(var(--to-y) - 31px)) scale(.85); } + 56%, 74% { opacity: 1; transform: translate(var(--to-x), calc(var(--to-y) - 23px)) scale(.75); } + 100% { opacity: 0; transform: translate(var(--to-x), calc(var(--to-y) - 29px)) scale(.65); } +} +@keyframes lr-reaction-expression { + 0% { opacity: 0; transform: translate(var(--to-x), calc(var(--to-y) - 13px)) scale(.75); } + 14%, 70% { opacity: 1; transform: translate(var(--to-x), calc(var(--to-y) - 33px)) scale(1); } + 100% { opacity: 0; transform: translate(var(--to-x), calc(var(--to-y) - 45px)); } +} +@keyframes lr-reaction-gift { + 0% { opacity: 0; transform: translate(var(--from-x), calc(var(--from-y) - 28px)) scale(.7); } + 12% { opacity: 1; transform: translate(var(--from-x), calc(var(--from-y) - 40px)) scale(.9); } + 35% { transform: translate(calc((var(--from-x) + var(--to-x)) / 2), calc(min(var(--from-y), var(--to-y)) - 100px)) scale(1); } + 52%, 80% { opacity: 1; transform: translate(var(--to-x), calc(var(--to-y) - 76px)) scale(1); } + 100% { opacity: 0; transform: translate(var(--to-x), calc(var(--to-y) - 84px)) scale(.94); } +} +@keyframes lr-reaction-wiggle { + 0%, 52%, 86%, 100% { transform: none; } + 60%, 76% { transform: rotate(-12deg); } + 68% { transform: rotate(12deg); } +} +@keyframes lr-reaction-bounce { + 0%, 52%, 68%, 84%, 100% { transform: none; } + 60%, 76% { transform: translateY(-5px) scale(1.06); } +} +@keyframes lr-reaction-pulse { + 0%, 52%, 72%, 100% { transform: none; } + 62% { transform: scale(1.15); } +} +@keyframes lr-reaction-peek { + 0%, 52%, 88%, 100% { transform: none; } + 60%, 64% { transform: translateX(-3px) rotate(-4deg); } + 74%, 78% { transform: translateX(3px) rotate(4deg); } +} +@keyframes lr-reaction-doze { + 0%, 52%, 100% { transform: none; } + 68% { transform: translateY(3px) rotate(-7deg); } + 84% { transform: translateY(1px) rotate(3deg); } +} +@keyframes lr-reaction-drift { + 0%, 14% { opacity: 0; transform: translateY(6px) scale(.5); } + 30% { opacity: .7; } + 90%, 100% { opacity: 0; transform: translate(var(--spark-x), var(--spark-y)) scale(1.1); } +} +@keyframes lr-reaction-impact { + 0%, 31% { opacity: 0; transform: scale(.4); } + 34% { opacity: .55; } + 48%, 100% { opacity: 0; transform: scale(1.6); } +} +@keyframes lr-reaction-squash { + 0%, 28%, 58%, 100% { transform: none; } + 34% { transform: scale(1.35, .65) rotate(-8deg); } + 44% { transform: scale(.9, 1.1) rotate(6deg); } +} +@keyframes lr-reaction-plop { + 0%, 28%, 64%, 100% { transform: none; } + 34% { transform: scale(1.2, .8); } + 43% { transform: rotate(-14deg); } + 53% { transform: rotate(9deg); } +} +@keyframes lr-reaction-spark-self { + 0%, 13% { opacity: 0; transform: scale(0); } + 22% { opacity: .85; } + 60%, 100% { opacity: 0; transform: translate(var(--spark-x), var(--spark-y)) rotate(var(--spark-turn)); } +} +@keyframes lr-reaction-spark-gift { + 0%, 51% { opacity: 0; transform: scale(0); } + 60% { opacity: .85; } + 95%, 100% { opacity: 0; transform: translate(var(--spark-x), var(--spark-y)) rotate(var(--spark-turn)); } +} +@keyframes lr-reaction-spark-impact { + 0%, 31% { opacity: 0; transform: scale(0); } + 38% { opacity: .85; } + 72%, 100% { opacity: 0; transform: translate(var(--spark-x), var(--spark-y)) rotate(var(--spark-turn)) scale(.6); } +} +[data-comic-reduced-motion] .lr-reaction-projectile { animation: none; transform: translate(var(--to-x), calc(var(--to-y) - 33px)); } +[data-comic-reduced-motion] .is-gift .lr-reaction-projectile { transform: translate(var(--to-x), calc(var(--to-y) - 76px)); } +[data-comic-reduced-motion] .lr-reaction-body { transform: none !important; } +[data-comic-reduced-motion] .lr-reaction-glyph { animation: none; } +[data-comic-reduced-motion] .lr-reaction-accents { display: none; } +[data-comic-reduced-motion] :is(.lr-reaction-grid, .lr-reaction-props) button > :first-child { transition: none; transform: none !important; } +@media (prefers-reduced-motion: reduce) { + .lr-reaction-projectile { animation: none; transform: translate(var(--to-x), calc(var(--to-y) - 33px)); } + .is-gift .lr-reaction-projectile { transform: translate(var(--to-x), calc(var(--to-y) - 76px)); } + .lr-reaction-body { transform: none !important; } + .lr-reaction-glyph { animation: none; } + .lr-reaction-accents { display: none; } + :is(.lr-reaction-grid, .lr-reaction-props) button > :first-child { transition: none; transform: none !important; } +} +@media (max-width: 420px) { + .lr-chat-compose { gap: 6px; } +} diff --git a/src/client/components/living/use-theater-mode.ts b/src/client/components/living/use-theater-mode.ts new file mode 100644 index 0000000..e7c6e2f --- /dev/null +++ b/src/client/components/living/use-theater-mode.ts @@ -0,0 +1,22 @@ +import { useEffect, useState } from "react"; + +/** The stage owns theater presentation; room membership and drafts stay mounted. */ +export function useTheaterMode() { + const [theaterMode, setTheaterMode] = useState(false); + useEffect(() => { + if (!theaterMode) return; + const exitOnEscape = (event: KeyboardEvent) => { + // A nonmodal popover can remain open after Tab moves focus outside it; + // native Escape must dismiss that surface before leaving theater mode. + if (event.key === "Escape" && !event.defaultPrevented && !event.isComposing && + !document.querySelector('[popover="auto"]:popover-open')) setTheaterMode(false); + }; + document.body.classList.add("lr-theater-open"); + window.addEventListener("keydown", exitOnEscape); + return () => { + document.body.classList.remove("lr-theater-open"); + window.removeEventListener("keydown", exitOnEscape); + }; + }, [theaterMode]); + return [theaterMode, setTheaterMode] as const; +} diff --git a/src/client/lib/debug.ts b/src/client/lib/debug.ts index 170b50f..18e27b3 100644 --- a/src/client/lib/debug.ts +++ b/src/client/lib/debug.ts @@ -30,7 +30,7 @@ const events: Array<{ record: BrowserDebugEvent; bytes: number }> = []; const startedAt = new Date().toISOString(); const started = performance.now(); let sequence = 0, retainedBytes = 0, evictedEvents = 0, truncatedEvents = 0; -const secretKey = /(?:token|password|passwd|secret|credential|authorization|cookie|private.?key|ice.?pwd|ufrag|usernamefragment|sdp|grant)|^(?:candidate|title|displayName)$/i; +const secretKey = /(?:token|password|passwd|secret|credential|authorization|cookie|private.?key|ice.?pwd|ufrag|usernamefragment|sdp|grant)|^(?:candidate|title|displayName|text)$/i; function safeText(value: string): string { return value diff --git a/src/client/lib/identity-hash.ts b/src/client/lib/identity-hash.ts new file mode 100644 index 0000000..65acd39 --- /dev/null +++ b/src/client/lib/identity-hash.ts @@ -0,0 +1,6 @@ +/** Stable decorative choices only; not a cryptographic or authority identifier. */ +export function identityHash(identity: string): number { + let seed = 2166136261; + for (const char of identity) seed = Math.imul(seed ^ char.charCodeAt(0), 16777619); + return seed >>> 0; +} diff --git a/src/client/lib/room-interactions.ts b/src/client/lib/room-interactions.ts new file mode 100644 index 0000000..fe29185 --- /dev/null +++ b/src/client/lib/room-interactions.ts @@ -0,0 +1,169 @@ +import { interactionPayloadSchema, type ClientMessage, type ServerMessage, type InteractionPayload } from "../../shared/protocol"; +import { INTERACTION_INTERVAL_MS, REACTION_DURATION_MS } from "../../shared/room-interactions"; +import { createOpaqueId } from "./opaque-id"; +import { identityHash } from "./identity-hash"; + +export type RoomInteraction = Extract; +type Rejection = Extract["reason"]; +export type InteractionError = Rejection | "offline" | "unconfirmed"; +export const CHAT_OVERLAY_DURATION_MS = 6000; +const CHAT_OVERLAY_LANES = 3; +interface InteractionState { + ready: boolean; + peerId: string | null; + messages: (RoomInteraction & { isSelf: boolean })[]; + reactions: (RoomInteraction & { expiresAt: number })[]; + overlayEnabled: boolean; + overlayMessages: (RoomInteraction & { expiresAt: number; lane: number })[]; + pending: { requestId: string; payload: InteractionPayload } | null; + confirmed: RoomInteraction | null; + coolingDown: boolean; + error: InteractionError | null; +} +const initialState = (): InteractionState => ({ ready: false, peerId: null, messages: [], reactions: [], + overlayEnabled: false, overlayMessages: [], + pending: null, confirmed: null, coolingDown: false, error: null }); + +// One owner per signaling client. Authentication admits room data; media +// generations and route changes deliberately do not reset this state. +export class RoomInteractionSession { + // Component state follows this room session, not reconnecting wire peer IDs. + readonly key = createOpaqueId(); + private state = initialState(); + private listeners = new Set<() => void>(); + private pendingTimer: ReturnType | undefined; + private cooldownTimer: ReturnType | undefined; + private effectsTimer: ReturnType | undefined; + private overlayVisible = false; + private closed = false; + private subscriptionStartedAt: number | null = null; + private serverOffset = 0; + + constructor(private readonly sendMessage: (message: ClientMessage) => boolean, readonly now = Date.now) {} + getSnapshot = () => this.state; + subscribe = (listener: () => void) => { + this.listeners.add(listener); + return () => { this.listeners.delete(listener); }; + }; + private update(patch: Partial) { + this.state = { ...this.state, ...patch }; + this.listeners.forEach(listener => listener()); + } + + authenticated(peerId: string) { + if (this.closed) return; + this.disconnected(); + this.update({ peerId, error: this.state.error === "unconfirmed" ? "unconfirmed" : null }); + this.subscriptionStartedAt = this.now(); + this.sendMessage({ type: "subscribe-room-interactions" }); + } + + receive(message: ServerMessage): boolean { + if (this.closed) return false; + if (message.type === "room-interactions-ready") { + if (this.subscriptionStartedAt === null) return true; + // One connection-scoped midpoint estimate; arrivals never move the clock. + this.serverOffset = message.serverTime - (this.subscriptionStartedAt + this.now()) / 2; + this.subscriptionStartedAt = null; + this.update({ ready: true }); + return true; + } + if (message.type === "room-interaction-rejected") { + if (message.requestId === this.state.pending?.requestId) { + clearTimeout(this.pendingTimer); + this.update({ pending: null, error: message.reason }); + } + return true; + } + if (message.type !== "room-interaction") return false; + if (!this.state.ready) return true; + const now = this.now(); + // Clock uncertainty may put a fresh event slightly ahead. Never extend its + // lifetime beyond one full effect, and never replay an expired effect. + const beganAt = Math.min(now, message.occurredAt - this.serverOffset); + const own = message.sender.peerId === this.state.peerId; + const confirmsPending = own && message.requestId === this.state.pending?.requestId; + const confirmed = confirmsPending ? message : this.state.confirmed; + const pending = confirmsPending ? null : this.state.pending; + if (!pending) clearTimeout(this.pendingTimer); + if (message.payload.kind === "chat") { + if (this.state.messages.some(item => item.id === message.id)) return true; + let overlayMessages = this.state.overlayMessages.filter(item => item.expiresAt > now); + const lane = identityHash(`${message.sender.peerId}:${message.id}`) % CHAT_OVERLAY_LANES; + const expiresAt = beganAt + CHAT_OVERLAY_DURATION_MS; + if (this.state.overlayEnabled && this.overlayVisible && expiresAt > now) { + // Newest on this fixed lane wins; missing messages cannot shift a later + // event to another lane, and bursts cannot accumulate a playback queue. + overlayMessages = [...overlayMessages.filter(item => item.lane !== lane), { ...message, lane, expiresAt }]; + } + this.update({ messages: [...this.state.messages.slice(-79), { ...message, isSelf: own }], + overlayMessages, pending, confirmed, error: confirmsPending ? null : this.state.error }); + } else { + if (this.state.reactions.some(item => item.id === message.id)) return true; + const expiresAt = beganAt + REACTION_DURATION_MS; + const reactions = this.state.reactions.filter(item => item.expiresAt > now); + this.update({ reactions: expiresAt > now ? [...reactions.slice(-7), { ...message, expiresAt }] : reactions, + pending, confirmed, error: confirmsPending ? null : this.state.error }); + } + this.expireEffects(); + return true; + } + + send(payload: InteractionPayload): boolean { + if (this.closed || !this.state.ready || this.state.pending || this.state.coolingDown || + !interactionPayloadSchema.safeParse(payload).success) return false; + const requestId = createOpaqueId(); + try { + if (!this.sendMessage({ type: "send-room-interaction", requestId, payload })) { + this.update({ error: "offline" }); + return false; + } + } catch { + this.update({ error: "offline" }); + return false; + } + this.update({ pending: { requestId, payload }, coolingDown: true, error: null }); + this.pendingTimer = setTimeout(() => this.update({ pending: null, error: "unconfirmed" }), 5000); + this.cooldownTimer = setTimeout(() => this.update({ coolingDown: false }), INTERACTION_INTERVAL_MS); + return true; + } + + setOverlayEnabled(enabled: boolean) { + if (this.closed || this.state.overlayEnabled === enabled) return; + this.update({ overlayEnabled: enabled, overlayMessages: [] }); + this.expireEffects(); + } + + setOverlayVisible(visible: boolean) { + if (this.closed || this.overlayVisible === visible) return; + this.overlayVisible = visible; + if (!visible) this.update({ overlayMessages: [] }); + this.expireEffects(); + } + + private expireEffects() { + clearTimeout(this.effectsTimer); + const next = Math.min(...[...this.state.reactions, ...this.state.overlayMessages].map(item => item.expiresAt)); + if (!Number.isFinite(next)) return; + this.effectsTimer = setTimeout(() => { + this.update({ reactions: this.state.reactions.filter(item => item.expiresAt > this.now()), + overlayMessages: this.state.overlayMessages.filter(item => item.expiresAt > this.now()) }); + this.expireEffects(); + }, Math.max(1, next - this.now())); + } + + disconnected() { + this.subscriptionStartedAt = null; + this.serverOffset = 0; + clearTimeout(this.pendingTimer); + clearTimeout(this.cooldownTimer); + clearTimeout(this.effectsTimer); + this.update({ ready: false, reactions: [], overlayMessages: [], coolingDown: false, pending: null, confirmed: null, + error: this.state.pending ? "unconfirmed" : this.state.error }); + } + close() { + this.closed = true; + this.disconnected(); + this.update(initialState()); + } +} diff --git a/src/client/lib/signaling.ts b/src/client/lib/signaling.ts index 158e66d..d34acdb 100644 --- a/src/client/lib/signaling.ts +++ b/src/client/lib/signaling.ts @@ -5,11 +5,13 @@ import { type ClientMessage, type DisplayName, type QualitySettings, + type RoutePolicy, type ServerMessage, } from "../../shared/protocol"; import type { SignalConnectionState } from "../types"; import { qualitySettingsEqual } from "../media/quality"; import { debugEvent } from "./debug"; +import { RoomInteractionSession } from "./room-interactions"; type WithoutProtocolEnvelope = T extends { type: string; @@ -74,9 +76,10 @@ function signalUrl(): string { } export class SignalingClient { + interactions: RoomInteractionSession | null = null; private socket: WebSocket | null = null; private stopped = false; - private authenticated = false; + private authenticatedPeerId: string | null = null; private reconnectAttempt = 0; private reconnectTimer: number | null = null; private authenticationTimer: number | null = null; @@ -90,12 +93,27 @@ export class SignalingClient { private previousChallenge: PendingSignalingChallenge | null = null; private visibilityListenerAttached = false; private hostQualityIntent: HostQualityIntent | null = null; + private pendingShareGeneration: string | null = null; constructor( private readonly identity: SignalingIdentity, private readonly events: SignalingEvents, private readonly now: () => number = Date.now, - ) {} + capabilities: { roomInteractions?: boolean } = {}, + ) { + if (capabilities.roomInteractions) this.enableRoomInteractions(); + } + + // Capability discovery may finish after media authentication. Enabling room + // data must not replace the signaling connection or its healthy media. + enableRoomInteractions(): RoomInteractionSession | null { + if (this.stopped) return null; + if (!this.interactions) { + this.interactions = new RoomInteractionSession(message => this.send(message), this.now); + if (this.authenticatedPeerId) this.interactions.authenticated(this.authenticatedPeerId); + } + return this.interactions; + } start(): void { if (this.socket || this.stopped) { @@ -108,12 +126,14 @@ export class SignalingClient { } stop(): void { + this.interactions?.close(); this.stopped = true; - this.authenticated = false; + this.authenticatedPeerId = null; this.clearTimers(); this.detachVisibilityListener(); this.terminalMessage = null; this.hostQualityIntent = null; + this.pendingShareGeneration = null; const socket = this.socket; this.socket = null; if (socket && socket.readyState < WebSocket.CLOSING) { @@ -125,12 +145,14 @@ export class SignalingClient { send(message: ClientMessage): boolean { if ( - !this.authenticated || + !this.authenticatedPeerId || !this.socket || this.socket.readyState !== WebSocket.OPEN ) { return false; } + if (message.type === "send-room-interaction" && + this.socket.bufferedAmount > 16 * 1024) return false; this.socket.send(JSON.stringify(message)); debugEvent("signal", "sent", { generation: this.socketGeneration, message }); return true; @@ -166,6 +188,50 @@ export class SignalingClient { } } + ownsHostRoom(roomId: string, hostToken: string): boolean { + return this.identity.role === "host" && this.identity.roomId === roomId && this.identity.token === hostToken; + } + + wantsHostPublication(shareGeneration: string): boolean { + return this.identity.role === "host" && this.identity.roomOnly !== true && + this.identity.shareGeneration === shareGeneration; + } + + /** Keep the authenticated room session and claim a fresh media generation. */ + startSharing( + shareGeneration: string, + qualitySettings: QualitySettings, + routePolicy: RoutePolicy, + sharingPaused = false, + ): boolean { + if (this.identity.role !== "host" || this.stopped || !shareGeneration) { + return false; + } + this.identity.roomOnly = undefined; + this.identity.shareGeneration = shareGeneration; + this.identity.sharingPaused = sharingPaused; + this.identity.qualitySettings = { ...qualitySettings }; + this.identity.routePolicy = { ...routePolicy }; + this.pendingShareGeneration = shareGeneration; + const sent = this.flushHostShareIntent(); + return sent || this.authenticatedPeerId === null; + } + + /** Stop only the media publication; the room and interaction session stay alive. */ + stopSharing(shareGeneration: string): boolean { + if (this.identity.role !== "host" || this.stopped || !shareGeneration || + this.identity.shareGeneration !== shareGeneration) { + return false; + } + const sent = this.send({ type: "stop-sharing", shareGeneration }); + this.identity.shareGeneration = undefined; + this.identity.sharingPaused = false; + this.identity.roomOnly = true; + this.pendingShareGeneration = null; + this.hostQualityIntent = null; + return sent; + } + confirmSharingPaused(): void { if (this.identity.role === "host") { this.identity.sharingPaused = true; @@ -237,7 +303,7 @@ export class SignalingClient { }; socket.send(JSON.stringify(authenticate)); this.authenticationTimer = window.setTimeout(() => { - if (!this.authenticated && this.socket === socket) { + if (!this.authenticatedPeerId && this.socket === socket) { socket.close(4000, "authentication timeout"); } }, 8_000); @@ -275,7 +341,7 @@ export class SignalingClient { if ( message.type === "error" && message.code === "INVALID_MESSAGE" && - !this.authenticated + !this.authenticatedPeerId ) { this.terminateForProtocolMismatch(); return; @@ -291,7 +357,7 @@ export class SignalingClient { } if (message.type === "authenticated") { - this.authenticated = true; + this.authenticatedPeerId = message.peerId; this.reconnectAttempt = 0; this.clearAuthenticationTimer(); if (this.terminalMessage) { @@ -305,7 +371,21 @@ export class SignalingClient { this.events.onStatus("connected"); debugEvent("signal", "authenticated", { role: this.identity.role, generation }); this.refreshSignalingWatchdog(); + this.interactions?.authenticated(message.peerId); + this.reconcileHostShareState(message); } + if ((message.type === "sharing-started" || message.type === "sharing-start-failed") && + this.pendingShareGeneration === message.shareGeneration) { + this.pendingShareGeneration = null; + } + if (message.type === "sharing-start-failed" && this.identity.role === "host" && + this.identity.shareGeneration === message.shareGeneration) { + this.identity.shareGeneration = undefined; + this.identity.sharingPaused = false; + this.identity.roomOnly = true; + this.hostQualityIntent = null; + } + if (this.interactions?.receive(message)) return; if ( message.type === "host-status" && this.identity.role === "host" && @@ -328,7 +408,8 @@ export class SignalingClient { return; } this.socket = null; - this.authenticated = false; + this.authenticatedPeerId = null; + this.interactions?.disconnected(); debugEvent("signal", "closed", { role: this.identity.role, code: event.code, reconnecting: !this.stopped && shouldReconnectSignaling(event.code) }); this.clearSignalingWatchdog(); @@ -392,6 +473,84 @@ export class SignalingClient { return true; } + private flushHostShareIntent(): boolean { + const shareGeneration = this.pendingShareGeneration; + if ( + !shareGeneration || + this.identity.role !== "host" || + this.identity.shareGeneration !== shareGeneration + ) { + return false; + } + try { + if (!this.send({ + type: "start-sharing", + shareGeneration, + sharingPaused: this.identity.sharingPaused, + qualitySettings: this.identity.qualitySettings, + routePolicy: this.identity.routePolicy, + })) { + return false; + } + return true; + } catch { + return false; + } + } + + private reconcileHostShareState( + message: Extract, + ): void { + if (this.identity.role !== "host" || this.identity.roomSession !== true) { + return; + } + const desiredGeneration = this.identity.shareGeneration; + if (this.identity.roomOnly === true) { + this.pendingShareGeneration = null; + if (message.shareGeneration !== null) { + // stopSharing may have been requested while authentication was still + // in flight. The authenticated reply is the first point where the + // server can accept the idempotent stop for that session. + this.send({ + type: "stop-sharing", + shareGeneration: message.shareGeneration, + }); + } + return; + } + if (!desiredGeneration) { + this.pendingShareGeneration = null; + return; + } + if (message.shareGeneration === desiredGeneration) { + // Authentication already claimed this generation; sending a second + // start-sharing would be rejected as a duplicate publication. + this.pendingShareGeneration = null; + return; + } + if (message.shareGeneration === null) { + this.flushHostShareIntent(); + return; + } + if (this.pendingShareGeneration === desiredGeneration) { + // A stop/start can cross the authentication response: the socket was + // authenticated with the old generation, while the UI already owns a + // fresh one. WebSocket ordering makes this pair one serialized server + // transition; keep the new intent and retire the stale publication. + if (this.send({ + type: "stop-sharing", + shareGeneration: message.shareGeneration, + })) { + this.flushHostShareIntent(); + } + return; + } + // Without a pending local intent, the server's active generation is + // authoritative. Do not retry an intent that no longer describes this + // authenticated session. + this.pendingShareGeneration = null; + } + private reconcileHostQualityIntent( message: Extract, ): void { @@ -521,7 +680,8 @@ export class SignalingClient { return; } this.socket = null; - this.authenticated = false; + this.authenticatedPeerId = null; + this.interactions?.disconnected(); this.clearAuthenticationTimer(); this.clearSignalingWatchdog(); this.events.onStatus("reconnecting"); @@ -573,7 +733,7 @@ export class SignalingClient { private isWatchdogEligible(): boolean { return ( - this.authenticated && + this.authenticatedPeerId !== null && this.isDocumentVisible() && this.socket?.readyState === WebSocket.OPEN ); diff --git a/src/client/locales/en.ts b/src/client/locales/en.ts index 211b668..e400d2e 100644 --- a/src/client/locales/en.ts +++ b/src/client/locales/en.ts @@ -118,6 +118,60 @@ export const en: Record = { "host.device.listFailed": "Could not load devices. Try refreshing.", "host.device.permission.microphone": "Turn on the microphone and allow access to see the available devices here.", "host.microphone.volume": "Microphone volume", + "interaction.open": "Chat", + "interaction.title": "Chat", + "interaction.close": "Hide chat window", + "floating.move": "Move window", + "floating.moveHint": "Drag the title bar, use arrow keys to adjust, or choose a corner under Position.", + "floating.position": "Position", + "floating.resize": "Resize window", + "floating.resizeHint": "Drag a corner or use arrow keys; preset sizes are under Position.", + "floating.size.default": "Default size", + "floating.size.large": "Large size", + "floating.position.topLeft": "Top left", + "floating.position.topRight": "Top right", + "floating.position.bottomLeft": "Bottom left", + "floating.position.bottomRight": "Bottom right", + "interaction.overlay.label": "On screen", + "interaction.overlay.show": "Show chat on screen", + "interaction.overlay.hide": "Hide chat on screen", + "interaction.overlay.showHint": "Show new messages over the picture, just for you.", + "interaction.overlay.hideHint": "Hide on-screen messages. They remain in the chat window.", + "interaction.effects": "Show reaction effects", + "interaction.ephemeral": "Messages stay on this page and clear when you refresh.", + "interaction.empty": "Make yourself at home. Say hello.", + "interaction.prop.tomato": "Toss tomato", + "interaction.prop.poop": "Toss poop", + "interaction.placeholder": "Say something…", + "interaction.send": "Send", + "interaction.reactions": "Reactions", + "interaction.express": "React", + "interaction.toPerson": "To", + "interaction.withPerson": "Interact with {name}", + "interaction.participant": "a room participant", + "interaction.receivedReaction": "{sender} to {target}: {reaction}.", + "interaction.expressedReaction": "{sender}: {reaction}.", + "interaction.unread": "{count} new messages", + "interaction.latest": "Back to latest", + "interaction.reconnecting": "Connect to send messages and reactions", + "interaction.error.offline": "Could not send. Try again shortly.", + "interaction.error.unconfirmed": "No confirmation from the server yet.", + "interaction.error.rate-limited": "A little too fast. Try again in a moment.", + "interaction.error.target-unavailable": "They just left. Try another person.", + "interaction.error.not-subscribed": "Reactions and chat are unavailable. Rejoin the room.", + "interaction.error.busy": "The connection is busy. Try again shortly.", + "interaction.reaction.wave": "Wave", + "interaction.reaction.heart": "Love it", + "interaction.reaction.clap": "Applause", + "interaction.reaction.laugh": "Laugh", + "interaction.reaction.wow": "Mind blown", + "interaction.reaction.party": "Celebrate", + "interaction.reaction.fire": "On fire", + "interaction.reaction.eyes": "Watching", + "interaction.reaction.star": "Gold star", + "interaction.reaction.sleep": "Sleepy", + "interaction.reaction.tomato": "Throw a tomato", + "interaction.reaction.poop": "Throw some poop", "host.microphone.mute": "Mute microphone", "host.microphone.pending": "Opening microphone", "host.microphone.browserOnly": "This capture method does not support a microphone yet. Choose Browser or Camera.", @@ -133,6 +187,11 @@ export const en: Record = { "host.sourcePicker.tab.display": "Screens", "host.sourcePicker.windowAudio": "Application sound", "host.sourcePicker.systemAudio": "System sound", + "host.sourcePicker.excludeAudio": "Exclude application audio", + "host.sourcePicker.excludeNone": "None", + "host.sourcePicker.excludeMissing": "Not listed; select again", + "host.sourcePicker.excludeAudioChoose": "Choose an application window to exclude. If it is missing, open its window and refresh.", + "host.sourcePicker.excludeAudioHint": "Share audio from all output devices except the selected process and its children. If it exits, source audio stops until you select again.", "host.sourcePicker.showCaptureBorder": "Show capture border", "host.sourcePicker.showCaptureBorderHint": "Turn off to hide the border when Windows allows it. Other captures may keep it visible.", "host.sourcePicker.empty": "No sources available", @@ -149,8 +208,8 @@ export const en: Record = { "host.sourcePicker.system": "System screen or window picker", "host.sourcePicker.systemAction": "Choose a screen or window with the system picker", "host.sourcePicker.audioHint": "Application sound for windows; system sound for screens", - "host.sourcePicker.refresh": "Refresh screens and windows", - "host.sourcePicker.loading": "Finding local screens and windows", + "host.sourcePicker.refresh": "Refresh sharing sources", + "host.sourcePicker.loading": "Finding sharing sources", "host.join": "Join a room", "host.starting": "Connecting", "host.cancelStart": "Cancel", @@ -241,6 +300,7 @@ export const en: Record = { "host.quality.1080p60": "1080p\u00a0· 60\u00a0fps", "host.advanced": "Sharing settings", "host.settings.button": "Settings", + "host.settings.picture": "Picture", "host.settings.sound": "Sound", "host.settings.technical": "Connection & encoding", @@ -259,6 +319,8 @@ export const en: Record = { "host.advanced.audio.music": "Music", "host.advanced.audio.veryHigh": "Studio", "host.advanced.route": "Route policy", + "host.advanced.route.sfuOnly": "Server media", + "host.advanced.route.sfuOnlyHint": "This site sends all shared video and audio through its media server", "host.advanced.route.topo": "Auto topology tuning", "host.advanced.route.topoHint": "Gradually pick healthier paths while live", "host.advanced.route.peerOnly": "Privacy mode", diff --git a/src/client/locales/zh.ts b/src/client/locales/zh.ts index dd0c4f0..3346dfe 100644 --- a/src/client/locales/zh.ts +++ b/src/client/locales/zh.ts @@ -118,6 +118,60 @@ export const zh = { "host.device.listFailed": "暂时无法获取设备,请刷新重试。", "host.device.permission.microphone": "开启麦克风并允许访问后,可以在这里选择设备。", "host.microphone.volume": "麦克风音量", + "interaction.open": "聊天", + "interaction.title": "聊天", + "interaction.close": "收起聊天窗口", + "floating.move": "移动窗口", + "floating.moveHint": "拖动标题栏移动窗口;方向键可微调,也可以从「位置」选择角落。", + "floating.position": "位置", + "floating.resize": "调整窗口大小", + "floating.resizeHint": "拖动角落或按方向键调整大小,也可以从「位置」选择预设尺寸。", + "floating.size.default": "默认大小", + "floating.size.large": "大尺寸", + "floating.position.topLeft": "左上角", + "floating.position.topRight": "右上角", + "floating.position.bottomLeft": "左下角", + "floating.position.bottomRight": "右下角", + "interaction.overlay.label": "弹幕", + "interaction.overlay.show": "显示弹幕", + "interaction.overlay.hide": "隐藏弹幕", + "interaction.overlay.showHint": "在画面上显示新消息,仅对自己生效。", + "interaction.overlay.hideHint": "隐藏画面上的消息,聊天窗口仍可查看。", + "interaction.effects": "显示互动动效", + "interaction.ephemeral": "仅保留本页收到的消息,刷新后清空。", + "interaction.empty": "坐这儿,聊两句。", + "interaction.prop.tomato": "丢番茄", + "interaction.prop.poop": "丢便便", + "interaction.placeholder": "说点什么…", + "interaction.send": "发送", + "interaction.reactions": "表情与小道具", + "interaction.express": "发表情", + "interaction.toPerson": "送给", + "interaction.withPerson": "与 {name} 互动", + "interaction.participant": "房间成员", + "interaction.receivedReaction": "{sender} 对 {target} 使用了「{reaction}」。", + "interaction.expressedReaction": "{sender}:{reaction}。", + "interaction.unread": "{count} 条新消息", + "interaction.latest": "回到最新消息", + "interaction.reconnecting": "连接后即可发送", + "interaction.error.offline": "暂时未能发送,请稍后重试。", + "interaction.error.unconfirmed": "暂未收到发送确认。", + "interaction.error.rate-limited": "发得有点快,稍等一下再试。", + "interaction.error.target-unavailable": "对方刚刚离开了,换个人试试。", + "interaction.error.not-subscribed": "互动暂不可用,请重新进入房间。", + "interaction.error.busy": "连接正忙,稍后再试。", + "interaction.reaction.wave": "打个招呼", + "interaction.reaction.heart": "比个心", + "interaction.reaction.clap": "鼓掌", + "interaction.reaction.laugh": "笑出声", + "interaction.reaction.wow": "震惊", + "interaction.reaction.party": "撒花", + "interaction.reaction.fire": "太燃了", + "interaction.reaction.eyes": "围观", + "interaction.reaction.star": "亮个星", + "interaction.reaction.sleep": "困了", + "interaction.reaction.tomato": "丢番茄", + "interaction.reaction.poop": "丢便便", "host.microphone.mute": "关闭麦克风", "host.microphone.pending": "正在开启麦克风", "host.microphone.browserOnly": "此采集方式暂不支持麦克风,可改用浏览器或摄像头。", @@ -133,6 +187,11 @@ export const zh = { "host.sourcePicker.tab.display": "屏幕", "host.sourcePicker.windowAudio": "应用声音", "host.sourcePicker.systemAudio": "系统声音", + "host.sourcePicker.excludeAudio": "排除应用声音", + "host.sourcePicker.excludeNone": "不排除", + "host.sourcePicker.excludeMissing": "不在列表中,请重新选择", + "host.sourcePicker.excludeAudioChoose": "选择要排除的应用窗口;未列出的应用请先打开窗口,再刷新。", + "host.sourcePicker.excludeAudioHint": "分享所有输出设备的声音,但不包含所选进程及其子进程。应用退出后来源声音会停止,需重新选择。", "host.sourcePicker.showCaptureBorder": "显示采集边框", "host.sourcePicker.showCaptureBorderHint": "关闭后尝试隐藏黄框;Windows 或其他采集程序可能仍要求显示。", "host.sourcePicker.empty": "暂无可用来源", @@ -149,8 +208,8 @@ export const zh = { "host.sourcePicker.system": "系统屏幕与窗口选择器", "host.sourcePicker.systemAction": "使用系统选择器选择屏幕或窗口", "host.sourcePicker.audioHint": "窗口分享应用声音,整屏分享系统声音", - "host.sourcePicker.refresh": "刷新屏幕和窗口", - "host.sourcePicker.loading": "正在查找本机屏幕和窗口", + "host.sourcePicker.refresh": "刷新分享来源", + "host.sourcePicker.loading": "正在查找分享来源", "host.join": "加入房间", "host.starting": "正在连接", "host.cancelStart": "取消", @@ -241,6 +300,7 @@ export const zh = { "host.quality.1080p60": "1080p\u00a0· 60\u00a0帧", "host.advanced": "分享设置", "host.settings.button": "设置", + "host.settings.picture": "画面", "host.settings.sound": "声音", "host.settings.technical": "连接与编码", @@ -259,6 +319,8 @@ export const zh = { "host.advanced.audio.music": "音乐", "host.advanced.audio.veryHigh": "保真", "host.advanced.route": "路由策略", + "host.advanced.route.sfuOnly": "服务器转发", + "host.advanced.route.sfuOnlyHint": "此站点的共享画面与声音统一由媒体服务器转发", "host.advanced.route.topo": "自动优化拓扑", "host.advanced.route.topoHint": "观看中逐步选择更健康的连接", "host.advanced.route.peerOnly": "隐私模式", diff --git a/src/client/media/browser-encoding-output.ts b/src/client/media/browser-encoding-output.ts index de03a0d..40a7490 100644 --- a/src/client/media/browser-encoding-output.ts +++ b/src/client/media/browser-encoding-output.ts @@ -26,7 +26,7 @@ export interface BrowserEncodingOutputSnapshot { } type Queue = { producerId: string; frames: RTCEncodedVideoFrame[]; needKey: boolean; requestKey: () => Promise }; -type Selection = { queue: Queue; onSelected: () => void }; +type Selection = { queue: Queue; onSelected: () => void; accept: (metadata: RTCEncodedVideoFrameMetadata) => boolean }; // Bound pending encoded data; a lost dependency chain resumes on a fresh key. const MAX_QUEUED_FRAMES = 4; @@ -46,7 +46,7 @@ export class BrowserEncodingOutput { private readonly writer: WritableStreamDefaultWriter; private current: Queue | undefined; private pending: Selection | undefined; - private carrier: RTCEncodedVideoFrame | undefined; + private carrier: { frame: RTCEncodedVideoFrame; recovery: boolean } | undefined; private writing: Promise | undefined; private ownKey: (() => Promise) | undefined; private raw: false | "key" | true = false; @@ -80,15 +80,25 @@ export class BrowserEncodingOutput { }); } - select(producerId: string, requestKey: () => Promise, onSelected: () => void): void { + select(producerId: string, requestKey: () => Promise, onSelected: () => void, + accept: Selection["accept"]): void { if (this.abort.signal.aborted) return; this.epoch++; this.carrier = undefined; - this.pending = { queue: { producerId, frames: [], needKey: true, requestKey }, onSelected }; + this.pending = { queue: { producerId, frames: [], needKey: true, requestKey }, onSelected, accept }; debugEvent("encoding-pool", "selection-requested", { ...this.identity, producerId }); if (!this.paused) this.request(this.pending.queue); } + cancelSelection(producerId: string): boolean { + if (this.pending?.queue.producerId !== producerId) return true; + // Once an accepted key starts writing, that queue owns the wire. Keep its + // pool reference until completion, including across pause/resume. + if (this.current === this.pending.queue) return false; + this.pending = undefined; + return true; + } + push(producerId: string, frame: RTCEncodedVideoFrame): void { if (this.abort.signal.aborted || this.paused || frame.data.byteLength === 0) return; try { @@ -152,30 +162,37 @@ export class BrowserEncodingOutput { const epoch = this.epoch; await this.writing; if (this.abort.signal.aborted || this.paused || this.epoch !== epoch) return; - if (this.raw && !this.pending?.queue.frames[0]) { - if (this.raw === "key" && frame.type !== "key") return; - await this.write(frame, null); - return; - } } - // A native carrier key can be a remote PLI. Recover the selected real - // stream, without requiring its frame type to match this clock frame. - if (frame.type === "key" && this.current && this.pending?.queue.frames[0]?.type !== "key" && - this.current.frames[0]?.type !== "key") this.recover(this.current); - this.carrier = frame; + // Coalescing keeps the newest clock, but must not overwrite a recovery + // request received while the previous output write is still pending. + this.carrier = { frame, recovery: frame.type === "key" || this.carrier?.recovery === true }; const writing = this.drain(); if (this.raw) await writing; } private drain(): Promise | undefined { if (this.abort.signal.aborted || this.paused || this.writing || !this.carrier) return; - const selection = this.pending?.queue.frames[0]?.type === "key" ? this.pending : undefined; - if (this.raw && !selection) return; - const queue = selection?.queue ?? this.current, frame = queue?.frames.shift(); - if (!queue || !frame) return; - const own = this.carrier; - this.carrier = undefined; try { + let selection = this.pending?.queue.frames[0]?.type === "key" ? this.pending : undefined; + // Preparation is not commitment: validate the actual key while the old + // stream is still intact, before consuming either queue or writing bytes. + if (selection && selection.queue !== this.current && !selection.accept(selection.queue.frames[0]!.getMetadata())) { + debugEvent("encoding-pool", "selection-rejected", { ...this.identity, producerId: selection.queue.producerId }); + this.pending = undefined; + selection = undefined; + } + const { frame: own, recovery } = this.carrier; + // A carrier key can be a remote PLI. Only an accepted candidate key can + // replace the current stream's recovery; a rejected one proves nothing. + if (recovery && !selection && this.current && this.current.frames[0]?.type !== "key") this.recover(this.current); + if (this.raw && !selection) { + this.carrier = undefined; + if (this.raw === "key" && own.type !== "key") return; + return this.write(own, null); + } + const queue = selection?.queue ?? this.current, frame = queue?.frames.shift(); + if (!queue || !frame) return; + this.carrier = undefined; const copy = copyFrame(frame, own.timestamp); this.current = queue; this.raw = false; diff --git a/src/client/media/browser-encoding-pool.ts b/src/client/media/browser-encoding-pool.ts index 7fa1c06..97a92c1 100644 --- a/src/client/media/browser-encoding-pool.ts +++ b/src/client/media/browser-encoding-pool.ts @@ -98,6 +98,54 @@ export class BrowserEncodingPool { return [...this.members].filter((member) => member.current === group || member.pending?.group === group); } + private demand(group: Group, except?: Member): number | undefined { + const budgets = this.references(group).flatMap((member) => member !== except && + this.compatible(member, group) && member.budget !== undefined ? [member.budget] : []); + return budgets.length ? Math.max(...budgets) : undefined; + } + + private hasHigherDemand(member: Member, group: Group): boolean { + return (this.demand(group) ?? 0) > member.budget!; + } + + private canJoin(member: Member, group: Group): boolean { + // A pending newcomer cannot prove compatibility with its own demand or + // raise the incumbent owner's allocation merely to reuse its encoder. + const demand = this.demand(group, member); + return demand === undefined || demand === member.budget; + } + + private canReuse(member: Member, group: Group, current: Group): boolean { + const output = group.output, previous = current.output; + // A quiet scene does not make a higher rate owner compatible. Check both + // allocations before pending membership can raise the candidate's budget. + return this.compatible(member, group) && this.canJoin(member, group) && + group.budget <= member.budget! && group.budget >= current.budget && + fits(output, member.budget) && output?.reason === "none" && !!previous && + noRegression(output, previous.width, previous.height, previous.fps, member.profile); + } + + private acceptsFrame(member: Member, group: Group, metadata: RTCEncodedVideoFrameMetadata): boolean { + if (member.disabled || member.paused || !this.compatible(member, group) || group.updating || + member.budget === undefined || group.budget > member.budget || !this.canJoin(member, group)) return false; + const current = member.current; + const optional = current && this.compatible(member, current) && + !this.hasHigherDemand(member, current); + // Initial output and necessary downgrades have no healthy picture to beat. + if (current ? !optional : member.carrier) return true; + const actual = member.encoded.snapshot(); + if (actual.lastProducerId !== (current?.producer.id ?? null)) return false; + if (!metadata.width || !metadata.height || actual.width === null || actual.height === null || + metadata.width < actual.width || metadata.height < actual.height) { + // A real key can disprove the last statistics sample. Do not let another + // reconciliation reuse that sample before the normal poll replaces it. + group.output = undefined; + return false; + } + return current ? this.canReuse(member, group, current) : !!group.output && + noRegression(group.output, actual.width, actual.height, member.raw?.framesPerSecond ?? null, member.profile); + } + private async poll(): Promise { if (this.polling || this.disposed) return; this.polling = true; @@ -155,6 +203,12 @@ export class BrowserEncodingPool { } private reconcile(): void { + // Remove obsolete requests before they can displace a current member in + // this pass. Applied rates may still belong to the previous stats sample. + for (const member of this.members) { + if (member.pending && member.budget !== undefined && !this.canJoin(member, member.pending.group) && + member.encoded.cancelSelection(member.pending.group.producer.id)) member.pending = undefined; + } const changing = new Set(); for (const group of this.groups) { const references = this.references(group); @@ -186,12 +240,12 @@ export class BrowserEncodingPool { // A shared producer can outgrow a waiting child's native allocation. // Replan that membership; never wait forever for its budget to catch up. if (!pending.selecting && pending.group.budget > member.budget && - !fits(pending.group.output, member.budget) && this.references(pending.group).some((other) => other !== member)) { + this.references(pending.group).some((other) => other !== member)) { member.pending = undefined; continue; } if (!pending.selecting && member.current && this.compatible(member, member.current) && - fits(member.current.output, member.budget) && pending.group.budget < member.current.budget) { + !this.hasHigherDemand(member, member.current) && !this.canReuse(member, pending.group, member.current)) { member.pending = undefined; continue; } @@ -223,6 +277,12 @@ export class BrowserEncodingPool { if (!member.disposed && !applied) this.failSource(member.source); }).catch(() => this.failSource(member.source)).finally(() => { member.replacement = undefined; }); this.prune(); + }, (metadata) => { + if (member.disposed || member.pending !== pending) return false; + if (this.acceptsFrame(member, pending.group, metadata)) return true; + member.pending = undefined; + this.prune(); + return false; }); } } @@ -237,10 +297,10 @@ export class BrowserEncodingPool { let earlier = true; for (const group of this.groups) { if (group === current) { earlier = false; continue; } - const improves = group.output && current.output && (group.output.width > current.output.width || - group.output.height > current.output.height || Math.round(group.output.fps) > Math.round(current.output.fps)); - if ((earlier || improves) && this.compatible(member, group) && fits(group.output, member.budget) && group.output?.reason === "none" && - noRegression(group.output!, current.output.width, current.output.height, current.output.fps, member.profile)) { + // A newer output must improve within the same Host ceilings used below. + const improves = group.output && current.output && !noRegression(current.output, + group.output.width, group.output.height, group.output.fps, member.profile); + if ((earlier || improves) && this.canReuse(member, group, current)) { member.pending = { group }; break; } } @@ -248,22 +308,18 @@ export class BrowserEncodingPool { } // A short output-rate spike is not new demand. Keep the rate owner and // let its native encoder adapt unless another child needs a higher rate. - if (current && this.compatible(member, current) && !this.references(current).some((other) => - this.compatible(other, current) && other.budget !== undefined && other.budget > member.budget!)) continue; - const candidates = [...this.groups].filter((group) => this.compatible(member, group)); - let target = candidates.find((group) => group !== current && fits(group.output, member.budget) && - (!current || group.budget === member.budget)); - target ??= candidates.find((group) => group !== current && group.budget === member.budget); + if (current && this.compatible(member, current) && !this.hasHigherDemand(member, current)) continue; + const candidates = [...this.groups].filter((group) => group !== current && this.compatible(member, group) && + this.demand(group) === member.budget); + let target = candidates.find((group) => fits(group.output, member.budget)) ?? candidates[0]; if (!target) { target = this.startGroup(member, member.budget); } member.pending = { group: target }; } for (const group of this.groups) { - const references = this.references(group); - const budgets = references.flatMap((member) => this.compatible(member, group) && member.budget !== undefined ? [member.budget] : []); - if (group.ready && budgets.length) { - const budget = Math.max(...budgets); + const budget = this.demand(group); + if (group.ready && budget !== undefined) { if (budget !== group.budget && !group.updating) { group.budget = budget; group.updating = group.producer.update(group.profile, budget).catch(() => { diff --git a/src/client/media/browser-encoding-producer.ts b/src/client/media/browser-encoding-producer.ts index 5af13c9..799bc7a 100644 --- a/src/client/media/browser-encoding-producer.ts +++ b/src/client/media/browser-encoding-producer.ts @@ -1,3 +1,4 @@ +import { parse, write } from "sdp-transform"; import { createOpaqueId } from "../lib/opaque-id"; import { debugError, debugEvent } from "../lib/debug"; import { debugRtcFailure, debugRtcStats, debugTrack, observeDebugConnection } from "../lib/debug-webrtc"; @@ -119,7 +120,7 @@ export class BrowserEncodingProducer { await receive.setRemoteDescription(send.localDescription!); await flushReceive(); this.checkAlive(); - await receive.setLocalDescription(await receive.createAnswer()); + await receive.setLocalDescription(this.seedLocalAnswer(await receive.createAnswer())); this.checkAlive(); await send.setRemoteDescription(receive.localDescription!); await flushSend(); @@ -217,6 +218,25 @@ export class BrowserEncodingProducer { return this.videoSender; } + private seedLocalAnswer(answer: RTCSessionDescriptionInit): RTCSessionDescriptionInit { + const kbps = Math.floor(Math.min(this.budget, this.desiredProfile.maxBitrate) / 1_000); + if (!Number.isFinite(kbps) || kbps <= 0 || !answer.sdp) return answer; + const session = parse(answer.sdp); + const media = session.media.find((section) => section.type === "video" && section.port !== 0); + const codec = media?.rtp.find((entry) => `video/${entry.codec}`.toLowerCase() === this.codec.mimeType.toLowerCase()); + if (!media || !codec) return answer; + const format = media.fmtp.find((entry) => entry.payload === codec.payload); + const parameters = (format?.config ?? "").split(";").map((part) => part.trim()) + .filter((part) => part && !/^x-google-start-bitrate\s*=/i.test(part)); + parameters.push(`x-google-start-bitrate=${kbps}`); + if (format) format.config = parameters.join(";"); + else media.fmtp.push({ payload: codec.payload, config: parameters.join(";") }); + // This same-Browser encoder already has a downstream native allocation. + // WebRTC reads send rates from the remote answer; an offer hint is dropped. + // Seed once, without a minimum or an external SDP/codec-identity change. + return { ...answer, sdp: write(session) }; + } + private async applyCurrent(): Promise { const sender = this.requireSender(), track = this.input!; const profile = this.desiredProfile; diff --git a/src/client/native/client.ts b/src/client/native/client.ts index f9c50e6..4a12986 100644 --- a/src/client/native/client.ts +++ b/src/client/native/client.ts @@ -74,6 +74,7 @@ export interface NativeShareInput { shareId: string; source: NativeCaptureTarget; audio: boolean; + excludeAudio?: NativeCaptureTarget; showCaptureBorder?: boolean; adapterIndex: number; encoderIndex: number; @@ -281,11 +282,13 @@ export class NativeClient { input: NativeShareInput, ): Promise<{ audio: boolean; codec: NativeVideoCodec; sourceAudio?: boolean }> { // Older Apps reject unknown command fields. - const { showCaptureBorder = false, ...shareInput } = input; + const { showCaptureBorder = false, excludeAudio, ...shareInput } = input; + this.validateAudioExclusion(excludeAudio); const response = await this.request( "start-share", { ...shareInput, + ...(excludeAudio ? { excludeAudio } : {}), ...(this.health.nativeMedia.microphone ? { microphoneMixing: true } : {}), ...(this.health.nativeMedia.captureBorderControl ? { showCaptureBorder } : {}), }, @@ -310,19 +313,28 @@ export class NativeClient { } } + private validateAudioExclusion(target: NativeCaptureTarget | undefined): void { + if (target && !this.health.nativeMedia.processAudioExclusion) { + throw new Error("Piik App does not support audio exclusion"); + } + } + async replaceShareSource( shareId: string, source: NativeCaptureTarget, audio: boolean, path: NativeCapturePath, showCaptureBorder = false, + excludeAudio?: NativeCaptureTarget, ): Promise { + this.validateAudioExclusion(excludeAudio); const response = await this.request( "replace-share-source", { shareId, source, audio, + ...(excludeAudio ? { excludeAudio } : {}), adapterIndex: path.adapterIndex, encoderIndex: path.encoderIndex, ...(this.health.nativeMedia.captureBorderControl ? { showCaptureBorder } : {}), diff --git a/src/client/native/wire.ts b/src/client/native/wire.ts index 4326d5d..8f4542a 100644 --- a/src/client/native/wire.ts +++ b/src/client/native/wire.ts @@ -36,6 +36,7 @@ export const nativeHealthSchema = nativeDiscoveryIdentitySchema.extend({ video: z.boolean().default(false), processAudio: z.boolean().default(false), systemAudio: z.boolean().default(false), + processAudioExclusion: z.boolean().default(false), microphone: z.boolean().default(false), // API availability, not Windows approval or observed border visibility. captureBorderControl: z.boolean().default(false), diff --git a/src/client/pages/AppLauncherPage.tsx b/src/client/pages/AppLauncherPage.tsx index 142498b..d227b00 100644 --- a/src/client/pages/AppLauncherPage.tsx +++ b/src/client/pages/AppLauncherPage.tsx @@ -131,7 +131,7 @@ export function AppLauncherPage() { const updateText = update ? `${t(updateKey)} · ${update.version}` : ""; const updateLink = update ? ( ')}`; -const SOURCES = { kind: "ready", processAudio: true, systemAudio: true, captureBorderControl: true, sources: [ +const SOURCES = { kind: "ready", processAudio: true, systemAudio: true, processAudioExclusion: true, captureBorderControl: true, sources: [ { kind: "window", sourceId: "101", pid: 1, creationTime: "1", title: "Sketchbook" }, { kind: "window", sourceId: "102", pid: 2, creationTime: "2", title: "A short film" }, + { kind: "window", sourceId: "104", pid: 3, creationTime: "3", title: "Voice call" }, { kind: "display", sourceId: "103", title: "Display 1" }, ] } satisfies NativeSourceList; const previewSource = async () => POSTER; @@ -36,6 +45,7 @@ const previewMicrophones = async () => [{ id: "headset", label: "USB Headset" }, export function ControlsPreview() { const { t, lang, vis } = useCopy(); const en = lang === "en"; + const displayPreview = new URLSearchParams(window.location.search).get("source") === "display"; const [sound, setSound] = useState(true); const [microphone, setMicrophone] = useState(false); const [microphoneVolume, setMicrophoneVolume] = useState(1); @@ -54,7 +64,7 @@ export function ControlsPreview() { const [passwordVisible, setPasswordVisible] = useState(false); const [name, setName] = useState("Piik friend"); const [invalid, setInvalid] = useState(false); - const [sourceOpen, setSourceOpen] = useState(false); + const [sourceOpen, setSourceOpen] = useState(displayPreview); const [sourceState, setSourceState] = useState("ready"); const [captureBorderAvailable, setCaptureBorderAvailable] = useState(true); const [showCaptureBorder, setShowCaptureBorder] = useState(false); @@ -92,6 +102,7 @@ export function ControlsPreview() { hint={paused ? "hint-resume" : "hint-pause"} onClick={() => setPaused(!paused)} /> setSharingSettings(value => !value)} /> @@ -108,6 +119,8 @@ export function ControlsPreview() { undefined} label={t("host.advanced.route.peerOnly")} note={t("host.advanced.route.peerOnlyRequired")} hint="hint-route-p2p-required" /> + undefined} label={t("host.advanced.route.sfuOnly")} + note={t("host.advanced.route.sfuOnlyHint")} hint="hint-route-sfu" />
{sourceOpen ? { setSourceOpen(false); notify(); }} onCamera={() => { setSourceOpen(false); notify(); }} loadCameras={previewCameras} @@ -217,5 +234,70 @@ export function ControlsPreview() { onSubmit={event => event.preventDefault()} />
+
+
09

{en ? "A little company" : "聊两句,丢个番茄。"}

+

{en ? "Click a person for reactions. Drag or resize either window." : "点小人发表情;聊天与表情窗都能拖动、缩放。"}

+ +
; } + +function InteractionControlsPreview() { + const [session, setSession] = useState(null); + const [viewer, setViewer] = useState(false); + const [crowded, setCrowded] = useState(false); + const [reaction, setReaction] = useState("heart"); + const [theater, setTheater] = useTheaterMode(); + const video = useRef(null); + const { t, lang } = useCopy(); + useEffect(() => { + const sender = viewer + ? { peerId: "preview-friend", role: "viewer" as const, displayName: "Piik friend" } + : { peerId: "preview-host", role: "host" as const, displayName: "Piik" }; + const model = new RoomInteractionSession(message => { + queueMicrotask(() => { + if (message.type === "subscribe-room-interactions") model.receive({ type: "room-interactions-ready", serverTime: Date.now() }); + if (message.type === "send-room-interaction") model.receive({ type: "room-interaction", id: createOpaqueId(), + requestId: message.requestId, occurredAt: Date.now(), sender, payload: message.payload }); + }); + return true; + }); + model.authenticated(sender.peerId); + setSession(model); + return () => model.close(); + }, [viewer]); + const receiveSample = (payload: InteractionPayload) => session?.receive({ + type: "room-interaction", id: createOpaqueId(), requestId: createOpaqueId(), occurredAt: Date.now(), + sender: viewer ? { peerId: "preview-host", role: "host", displayName: "Piik" } + : { peerId: "preview-friend", role: "viewer", displayName: "Piik friend" }, payload, + }); + return <>
+ + + + + +
+ + ({ + key: index ? `preview-friend-${index}` : "preview-friend", name: index ? `Friend ${index + 1}` : "Piik friend", + you: viewer && index === 0, + status: deriveParticipantStatus({ upstream: { kind: "peer", peerId: "preview-host" }, mediaReady: true }, true), + })) + } />
; +} diff --git a/src/client/pages/HostPage.tsx b/src/client/pages/HostPage.tsx index 9c6add6..a705395 100644 --- a/src/client/pages/HostPage.tsx +++ b/src/client/pages/HostPage.tsx @@ -25,9 +25,12 @@ import { import { browserCaptureDevices } from "../media/capture-devices"; import { AppHeader, LedStrip } from "../components/living/Header"; import { WelcomeLine } from "../components/living/WelcomeLine"; -import { Couch, type CouchEntry } from "../components/living/Couch"; +import type { CouchEntry } from "../components/living/Couch"; import { SharingSettings } from "../components/living/SharingSettings"; import { HostMicrophone, HostMicrophoneSettings } from "../components/living/HostMicrophone"; +import { RoomInteractions } from "../components/living/RoomInteractions"; +import { RoomChatOverlay } from "../components/living/RoomChatOverlay"; +import type { RoomInteractionSession } from "../lib/room-interactions"; import { HostAudio } from "../media/host-audio"; import { CaptureSourcePicker, @@ -330,7 +333,10 @@ function hostTerminationKey(reason: SignalingTerminationReason): CopyKey { } interface HostPageProps { + roomInteractionsAvailable?: boolean; + hostRoomSessionAvailable?: boolean; sfuAvailable?: boolean; + sfuOnly?: boolean; natPredictionAvailable?: boolean; connectionAttemptProgress4?: boolean; launchedByClient?: boolean; @@ -344,12 +350,16 @@ type ShareSourceSelection = client: NativeClient; target: NativeCaptureTarget; audio: boolean; + excludeAudio?: NativeCaptureTarget; showCaptureBorder: boolean; path: NativeCapturePath; }; export function HostPage({ + roomInteractionsAvailable = false, + hostRoomSessionAvailable = false, sfuAvailable = false, + sfuOnly = false, natPredictionAvailable = false, connectionAttemptProgress4 = false, launchedByClient = false, @@ -367,7 +377,8 @@ export function HostPage({ () => ({ ...DEFAULT_ROUTE_POLICY, peerOnly: !sfuAvailable, - natPrediction: natPredictionAvailable, + topologyOptimization: !sfuOnly, + natPrediction: natPredictionAvailable && !sfuOnly, }), ); const [videoCodecMode, setVideoCodecMode] = @@ -381,6 +392,7 @@ export function HostPage({ const [stream, setStream] = useState(null); const hostAudioRef = useRef(null); const [microphoneEnabled, setMicrophoneEnabled] = useState(false); + const [interactionSession, setInteractionSession] = useState(null); const [microphoneVolume, setMicrophoneVolume] = useState(1); const [microphonePending, setMicrophonePending] = useState(false); const [microphoneDevices, setMicrophoneDevices] = useState({ browser: "", native: "" }); @@ -444,6 +456,13 @@ export function HostPage({ useEffect(() => { roomRef.current = room; }, [room]); + useEffect(() => { + let active = true; + if (hostRoomSessionAvailable && room) void roomInitializationRef.current.then(() => { + if (active && isCurrentRoomAuthority(room)) connectRoomSignal(room).start(); + }); + return () => { active = false; }; + }, [hostRoomSessionAvailable, room?.roomId, room?.hostToken]); const [creationProfile, setCreationProfile] = useState(readCreationProfile); const creationProfileRef = useRef(creationProfile); @@ -580,12 +599,14 @@ export function HostPage({ const nativeClientRef = useRef(null); const nativeClientConnectRef = useRef | null>(null); const nativeShareGenerationRef = useRef(null); + const serverShareGenerationRef = useRef(null); const nativeMediaBridgeRef = useRef(null); const nativeMediaIngressRef = useRef(null); const nativeEventCleanupRef = useRef<(() => void) | null>(null); const nativeClientCloseCleanupRef = useRef<(() => void) | null>(null); const nativeModeRef = useRef(false); const nativeSourceAudioRef = useRef(undefined); + const nativeAudioSelectionRef = useRef<{ enabled: boolean; exclude?: NativeCaptureTarget } | null>(null); const nativeSourceRequestRef = useRef(null); const nativePreviewTailRef = useRef>(Promise.resolve()); const nativeSourcePathRef = useRef(null); @@ -688,6 +709,7 @@ export function HostPage({ browserVideoPoolRef.current = null; activeHostChildPeerIdsRef.current = []; endpointMediaCopyCapacityRef.current = MAX_ENDPOINT_MEDIA_CHILDREN; + serverShareGenerationRef.current = null; hostPeerIdRef.current = null; if (copiedResetTimerRef.current !== null) { window.clearTimeout(copiedResetTimerRef.current); @@ -836,7 +858,7 @@ export function HostPage({ return warning; } - function disposeResources(notifyServer: boolean): void { + function disposeResources(notifyServer: boolean, keepRoomSession = false): void { sourceSwitchRef.current = null; qualityChangeRef.current = null; pendingQualityChangeRef.current = null; @@ -851,14 +873,20 @@ export function HostPage({ if (notifyServer) { const shareGeneration = shareGenerationRef.current; if (shareGeneration) { - signal.send({ type: "stop-sharing", shareGeneration }); + if (keepRoomSession) signal.stopSharing(shareGeneration); + else signal.send({ type: "stop-sharing", shareGeneration }); } } - // Never carry a terminal message into a later authentication: the next - // sharing generation may already be reusing this room. - signal.stop(); + if (!keepRoomSession) { + // Never carry a terminal message into a later authentication: the next + // sharing generation may already be reusing this room. + signal.stop(); + } + } + if (!keepRoomSession) { + signalRef.current = null; + setInteractionSession(null); } - signalRef.current = null; shareGenerationRef.current = null; disposeNativeShare(); peersRef.current.forEach((peer) => peer.dispose()); @@ -868,8 +896,11 @@ export function HostPage({ browserVideoPoolRef.current?.dispose(); browserVideoPoolRef.current = null; activeHostChildPeerIdsRef.current = []; - endpointMediaCopyCapacityRef.current = MAX_ENDPOINT_MEDIA_CHILDREN; - hostPeerIdRef.current = null; + if (!keepRoomSession) { + endpointMediaCopyCapacityRef.current = MAX_ENDPOINT_MEDIA_CHILDREN; + hostPeerIdRef.current = null; + } + serverShareGenerationRef.current = null; void hostSfuRouteRef.current?.disconnect(); hostSfuRouteRef.current = null; hostAudioRef.current?.dispose(); @@ -880,16 +911,16 @@ export function HostPage({ retiringStreamRef.current?.getTracks().forEach((track) => track.stop()); streamRef.current = null; retiringStreamRef.current = null; - iceConfigRef.current = null; + if (!keepRoomSession) iceConfigRef.current = null; setStream(null); setDetails(null); setPeerSnapshots(new Map()); - setParticipantPresence([]); + if (!keepRoomSession) setParticipantPresence([]); viewerQualityEvidenceStore.clear(); cancelViewerQualityEvidenceRender(); setViewerQualityEvidence(viewerQualityEvidenceStore.getSnapshot()); activeRouteRevisionRef.current = 0; - setSignalStatus("offline"); + if (!keepRoomSession) setSignalStatus("offline"); setSwitchingSource(false); setChangingQuality(false); sharingPausedRef.current = false; @@ -943,6 +974,7 @@ export function HostPage({ notifyServer = true, comic: ComicKind = "share-ended", tone: ComicTone = "off", + keepRoomSession = hostRoomSessionAvailable, ): void { const generation = activeGenerationRef.current; if (generation === null || generationRef.current !== generation) { @@ -955,7 +987,7 @@ export function HostPage({ if (notifyServer && currentRoom) { writePreferredRoom(currentRoom.roomId); } - disposeResources(notifyServer); + disposeResources(notifyServer, keepRoomSession); setNoticeValue({ ...(typeof message === "string" ? { kind: "text", text: message } @@ -987,7 +1019,9 @@ export function HostPage({ ); const replacement = hostRoomFromCreated(response); if (wasSharing) { - endSharing({ key: "host.roomReplaced" }, false); + endSharing({ key: "host.roomReplaced" }, false, "share-ended", "off", false); + } else if (signalRef.current) { + disposeResources(false); } replaceViewerInvite(activeRoom.roomId, null); if (!(await writeHostRoom( @@ -1054,7 +1088,7 @@ export function HostPage({ if (forgetRoom(activeRoom)) { // endSharing already stated the recovery path; the raw room error // would only overwrite it with the same 404 in server wording. - endSharing({ key: "host.roomInvalid" }, false, "room-not-found", "bad"); + endSharing({ key: "host.roomInvalid" }, false, "room-not-found", "bad", false); return; } } @@ -1156,7 +1190,7 @@ export function HostPage({ shareGeneration: string, selection: Extract, ): Promise { - const { client, target, audio, showCaptureBorder, path } = selection; + const { client, target, audio, excludeAudio, showCaptureBorder, path } = selection; let bridge: NativeMediaBridge | null = null; let shareStarted = false; let nativeEventCleanup: (() => void) | null = null; @@ -1181,6 +1215,7 @@ export function HostPage({ ); nativeMediaBridgeRef.current = bridge; nativeSourceAudioRef.current = undefined; + nativeAudioSelectionRef.current = { enabled: audio, exclude: excludeAudio }; nativeEventCleanup = client.onEvent((event) => { if (event.shareId !== shareGeneration || !isCurrentShare(generation, shareGeneration) || nativeClientRef.current !== client) return; if (event.type === "audio-state") { @@ -1203,6 +1238,7 @@ export function HostPage({ shareId: shareGeneration, source: target, audio, + excludeAudio: audio ? excludeAudio : undefined, showCaptureBorder, adapterIndex: path.adapterIndex, encoderIndex: path.encoderIndex, @@ -1392,6 +1428,7 @@ export function HostPage({ sources, processAudio: client.health.nativeMedia.processAudio, systemAudio: client.health.nativeMedia.systemAudio, + processAudioExclusion: client.health.nativeMedia.processAudioExclusion, captureBorderControl: client.health.nativeMedia.captureBorderControl, }); } catch (error) { @@ -1484,6 +1521,7 @@ export function HostPage({ target: NativeCaptureTarget, audio: boolean, showCaptureBorder: boolean, + excludeAudio?: NativeCaptureTarget, ): void { if (nativeSources?.kind !== "ready") return; const client = nativeClientRef.current; @@ -1491,10 +1529,10 @@ export function HostPage({ if (!client || !path) return; setShowCaptureBorder(showCaptureBorder); if (phase === "live" && nativeModeRef.current) { - void switchNativeSource(client, target, audio, path, showCaptureBorder); + void switchNativeSource(client, target, audio, path, showCaptureBorder, excludeAudio); return; } - void startSharing({ kind: "native", client, target, audio, showCaptureBorder, path }); + void startSharing({ kind: "native", client, target, audio, excludeAudio, showCaptureBorder, path }); } function disposeNativeShare(expectedShare = nativeShareGenerationRef.current): void { @@ -1511,6 +1549,7 @@ export function HostPage({ nativeShareGenerationRef.current = null; nativeModeRef.current = false; nativeSourceAudioRef.current = undefined; + nativeAudioSelectionRef.current = null; setNativeActive(false); if (!client || !shareGeneration) { releaseUnusedNativeClient(); @@ -2214,7 +2253,6 @@ export function HostPage({ function handleSignalMessage( message: ServerMessage, generation: number, - activeRoom: HostRoomState, reauthenticated: boolean, pendingQualitySettings: QualitySettings | null, ): void { @@ -2223,29 +2261,8 @@ export function HostPage({ } if (message.type === "authenticated" && message.role === "host") { discardPreparedHostChild(); - hostPeerIdRef.current = message.peerId; - endpointMediaCopyCapacityRef.current = message.endpointMediaCopyCapacity; - const authenticatedProfile = { - codeEntryPolicy: message.codeEntryPolicy, - roomPassword: message.viewerPasswordEnabled - ? creationProfileRef.current.roomPassword - : null, - }; - saveCreationProfile(authenticatedProfile); - creationProfileRef.current = authenticatedProfile; - setCreationProfile(authenticatedProfile); - setViewerPasswordEnabled(message.viewerPasswordEnabled); - setViewerPasswordDraft(authenticatedProfile.roomPassword ?? ""); - setViewerPasswordVisible(false); routePolicyRef.current = { ...message.routePolicy }; setRoutePolicy({ ...message.routePolicy }); - setRoom((current) => - mergeAuthenticatedHostRoom( - current, - activeRoom.roomId, - message.codeEntryPolicy, - ), - ); const currentQualitySettings = pendingQualitySettings ?? message.qualitySettings; activeRouteRevisionRef.current = message.routeRevision; @@ -2289,9 +2306,24 @@ export function HostPage({ }); return; } - if (message.type === "viewer-presence") { - retainViewerQualityEvidenceForPresence(message.viewers); - setParticipantPresence(message.viewers); + if (message.type === "sharing-started") { + if (message.shareGeneration !== shareGenerationRef.current) return; + serverShareGenerationRef.current = message.shareGeneration; + routePolicyRef.current = { ...message.routePolicy }; + setRoutePolicy({ ...message.routePolicy }); + commitQuality(message.qualitySettings); + activeRouteRevisionRef.current = message.routeRevision; + sharingPausedRef.current = message.paused; + setSharingPaused(message.paused); + setPhase("live"); + const route = ensureHostSfuRoute(generation); + void route + .resyncAuthoritative({ + revision: message.routeRevision, + phase: "active", + assignment: message.routeAssignment, + }) + .then(() => syncHostSfuQualityWarning(route, generation)); return; } if (message.type === "pause-sharing-source") { @@ -2395,29 +2427,237 @@ export function HostPage({ }); return; } - if (message.type === "room-closed") { - if (!forgetRoom(activeRoom)) { + } + + function connectRoomSignal(activeRoom: HostRoomState, shareGeneration?: string, allowReplacement = true): SignalingClient { + const previous = signalRef.current; + if (hostRoomSessionAvailable && previous?.ownsHostRoom(activeRoom.roomId, activeRoom.hostToken)) return previous; + previous?.stop(); + let authenticated = false; + const hostClientId = getStableClientId("host", activeRoom.roomId); + hostClientIdRef.current = hostClientId; + const hostFallback = defaultHostDisplayName(visRef.current); + const initialDisplayName = readDisplayName(hostFallback); + displayNameRef.current = initialDisplayName; + setDisplayName(initialDisplayName); + setDisplayNameDraft(initialDisplayName); + setDisplayNameError(null); + const signal = new SignalingClient( + { + roomId: activeRoom.roomId, + role: "host", + token: activeRoom.hostToken, + clientId: hostClientId, + ...(hostRoomSessionAvailable ? { roomSession: true as const, roomOnly: true as const } : { shareGeneration }), + sharingPaused: false, + qualitySettings: qualitySettingsRef.current, + routePolicy: routePolicyRef.current, + viewerPresence: true, + ...(connectionAttemptProgress4 ? { connectionAttemptProgress4: true } : {}), + displayName: initialDisplayName, + }, + { + onStatus: (status) => { + if (signalRef.current === signal && isCurrentRoomAuthority(activeRoom)) { + setSignalStatus(status); + } + }, + onTerminated: (reason) => { + if (signalRef.current !== signal || !isCurrentRoomAuthority(activeRoom)) return; + if (activeGenerationRef.current !== null) { + endSharing({ key: hostTerminationKey(reason) }, false, "signal-failed", "bad", false); + } else { + disposeResources(false); + setNoticeKey(hostTerminationKey(reason), "signal-failed", "bad"); + setPhase("error"); + } + if (reason === "SESSION_REPLACED") forgetRoom(activeRoom, true); + }, + onAccessRequired: () => { + if (signalRef.current !== signal || !isCurrentRoomAuthority(activeRoom)) return; + if (activeGenerationRef.current !== null) { + endSharing({ key: "gate.expired" }, false, "access-denied", "bad", false); + } else { + disposeResources(false); + setNoticeKey("gate.expired", "access-denied", "bad"); + setPhase("error"); + } + onAuthorizationRequired?.(); + }, + onMessage: (message) => { + if (signalRef.current !== signal || !isCurrentRoomAuthority(activeRoom)) return; + const currentGeneration = activeGenerationRef.current; + const currentShareGeneration = shareGenerationRef.current; + const localShareActive = + currentGeneration !== null && currentShareGeneration !== null; + if ( + !authenticated && + message.type === "error" && + message.code === "INVALID_TOKEN" && + allowReplacement + ) { + if (!localShareActive || !signal.wantsHostPublication(currentShareGeneration!)) { + signalRef.current = null; + signal.stop(); + setInteractionSession(null); + setSignalStatus("offline"); + forgetRoom(activeRoom); + // Capture may still be awaiting permission. Its ready path owns + // creation/publication; room recovery must not start it early. + if (!localShareActive) { + setNoticeKey("host.roomInvalid", "room-not-found", "bad"); + setPhase("error"); + } + return; + } + signalRef.current = null; + signal.stop(); + setSignalStatus("offline"); + roomInitializationRef.current = createReplacementRoom(activeRoom, currentGeneration!, currentShareGeneration!); + return; + } + if (message.type === "error" && message.code === "INVALID_TOKEN") { + forgetRoom(activeRoom); + if (localShareActive) { + endSharing({ key: "host.roomInvalid" }, false, "room-not-found", "bad", false); + } else { + disposeResources(false); + forgetRoom(activeRoom); + setNoticeKey("host.roomInvalid", "room-not-found", "bad"); + setPhase("error"); + } + return; + } + if (message.type === "room-closed") { + if (!forgetRoom(activeRoom)) return; + if (localShareActive) { + endSharing(say("host.roomClosed"), false, "room-closed", "off", false); + } else { + disposeResources(false); + setNoticeKey("host.roomClosed", "room-closed", "off"); + setPhase("ended"); + } + return; + } + if (message.type === "error" && message.code === "HOST_ALREADY_CONNECTED") { + if (localShareActive) { + endSharing(hostServerErrorNotice(message.code), false, "signal-failed", "bad", false); + } else { + disposeResources(false); + setNotice(hostServerErrorNotice(message.code), "signal-failed", "bad"); + setPhase("error"); + } + return; + } + const reauthenticated = + authenticated && + message.type === "authenticated" && + message.role === "host"; + if (message.type === "authenticated" && message.role === "host") { + authenticated = true; + iceConfigRef.current = message.iceConfig; + hostPeerIdRef.current = message.peerId; + endpointMediaCopyCapacityRef.current = message.endpointMediaCopyCapacity; + serverShareGenerationRef.current = message.shareGeneration; + const profile = { + codeEntryPolicy: message.codeEntryPolicy, + roomPassword: message.viewerPasswordEnabled ? creationProfileRef.current.roomPassword : null, + }; + saveCreationProfile(profile); + creationProfileRef.current = profile; + setCreationProfile(profile); + setViewerPasswordEnabled(message.viewerPasswordEnabled); + setViewerPasswordDraft(profile.roomPassword ?? ""); + setViewerPasswordVisible(false); + setRoom(current => mergeAuthenticatedHostRoom(current, activeRoom.roomId, message.codeEntryPolicy)); + peersRef.current.forEach((peer) => + peer.updateIceConfig(message.iceConfig), + ); + void writeHostRoom( + activeRoom, + () => signalRef.current === signal && isCurrentRoomAuthority(activeRoom), + ); + writePreferredRoom(activeRoom.roomId); + } + if ( + message.type === "sharing-started" && + currentShareGeneration !== null && + message.shareGeneration === currentShareGeneration + ) { + serverShareGenerationRef.current = message.shareGeneration; + } + if (message.type === "sharing-start-failed") { + if (localShareActive && message.shareGeneration === currentShareGeneration) { + endSharing(hostServerErrorNotice(message.code), false, "signal-failed", "bad"); + } + return; + } + if (message.type === "viewer-presence") { + retainViewerQualityEvidenceForPresence(message.viewers); + setParticipantPresence(message.viewers); + return; + } + if (message.type === "error") { + if (message.code !== "AUTH_REQUIRED") setNotice(hostServerErrorNotice(message.code), "signal-failed", "bad"); + return; + } + const mediaActive = + currentGeneration !== null && + currentShareGeneration !== null && + serverShareGenerationRef.current === currentShareGeneration; + if (message.type === "authenticated" && message.role === "host" && mediaActive) { + setPhase("live"); + } + if (!mediaActive) return; + handleSignalMessage( + message, + currentGeneration!, + reauthenticated, + reauthenticated + ? signal.pendingHostQualitySettings(currentShareGeneration!) + : null, + ); + }, + }, + undefined, + { roomInteractions: roomInteractionsAvailable }, + ); + signalRef.current = signal; + setInteractionSession(signal.interactions); + return signal; + } + + async function createReplacementRoom(activeRoom: HostRoomState, generation: number, shareGeneration: string): Promise { + try { + const response = await createRoom( + creationProfileRef.current.codeEntryPolicy, + creationProfileRef.current.roomPassword, + readPreferredRoomId(), + ); + const replacement = hostRoomFromCreated(response); + if (!isCurrentShare(generation, shareGeneration) || !isCurrentRoomAuthority(activeRoom)) { + closeAbandonedRoom(replacement); return; } - endSharing(say("host.roomClosed"), false, "room-closed", "off"); - return; - } - if (message.type === "error") { - if (message.code === "INVALID_TOKEN") { - if (!forgetRoom(activeRoom)) { - return; - } - endSharing({ key: "host.roomInvalid" }, false, "room-not-found", "bad"); + forgetRoom(activeRoom); + if (!(await writeHostRoom(replacement, () => isCurrentShare(generation, shareGeneration)))) { + closeAbandonedRoom(replacement); + throw new Error("Host room is already open in another tab"); + } + roomRef.current = replacement; + setRoom(replacement); + const signal = connectRoomSignal(replacement, shareGeneration, false); + if (hostRoomSessionAvailable) signal.startSharing(shareGeneration, qualitySettingsRef.current, routePolicyRef.current); + signal.start(); + } catch (error) { + if (!isCurrentShare(generation, shareGeneration)) return; + endSharing({ key: "host.roomInvalid" }, false, "room-not-found", "bad", false); + if (error instanceof ApiError && error.status === 401 && onAuthorizationRequired) { + onAuthorizationRequired(); return; } - if (message.code === "AUTH_REQUIRED") { - return; - } - if (message.code === "HOST_ALREADY_CONNECTED") { - endSharing(hostServerErrorNotice(message.code), false, "signal-failed", "bad"); - return; - } - setNotice(hostServerErrorNotice(message.code), "signal-failed", "bad"); + setNoticeError(error, "room", "television"); + setPhase("error"); } } @@ -2538,158 +2778,9 @@ export function HostPage({ setRoom(createdRoom); claimedRoom = true; } - let replacementAttempted = false; - const connectSignal = (activeRoom: HostRoomState): SignalingClient => { - let authenticated = false; - const hostClientId = getStableClientId("host", activeRoom.roomId); - hostClientIdRef.current = hostClientId; - const hostFallback = defaultHostDisplayName(visRef.current); - const initialDisplayName = readDisplayName(hostFallback); - displayNameRef.current = initialDisplayName; - setDisplayName(initialDisplayName); - setDisplayNameDraft(initialDisplayName); - setDisplayNameError(null); - const signal = new SignalingClient( - { - roomId: activeRoom.roomId, - role: "host", - token: activeRoom.hostToken, - clientId: hostClientId, - shareGeneration, - sharingPaused: false, - qualitySettings: qualitySettingsRef.current, - routePolicy: routePolicyRef.current, - viewerPresence: true, - ...(connectionAttemptProgress4 ? { connectionAttemptProgress4: true } : {}), - displayName: initialDisplayName, - }, - { - onStatus: (status) => { - if ( - isCurrentShare(generation, shareGeneration) && - signalRef.current === signal - ) { - setSignalStatus(status); - } - }, - onTerminated: (reason) => { - if ( - isCurrentShare(generation, shareGeneration) && - signalRef.current === signal - ) { - endSharing({ key: hostTerminationKey(reason) }, false, "signal-failed", "bad"); - if (reason === "SESSION_REPLACED") forgetRoom(activeRoom, true); - } - }, - onAccessRequired: () => { - if ( - isCurrentShare(generation, shareGeneration) && - signalRef.current === signal - ) { - endSharing({ key: "gate.expired" }, false, "access-denied", "bad"); - onAuthorizationRequired?.(); - } - }, - onMessage: (message) => { - if ( - !isCurrentShare(generation, shareGeneration) || - signalRef.current !== signal - ) { - return; - } - if ( - !authenticated && - message.type === "error" && - message.code === "INVALID_TOKEN" && - !replacementAttempted - ) { - replacementAttempted = true; - signalRef.current = null; - signal.stop(); - setSignalStatus("offline"); - forgetRoom(activeRoom); - void createReplacementRoom(); - return; - } - const reauthenticated = - authenticated && - message.type === "authenticated" && - message.role === "host"; - if (message.type === "authenticated" && message.role === "host") { - authenticated = true; - iceConfigRef.current = message.iceConfig; - peersRef.current.forEach((peer) => - peer.updateIceConfig(message.iceConfig), - ); - void writeHostRoom( - activeRoom, - () => isCurrentShare(generation, shareGeneration) && - signalRef.current === signal, - ); - writePreferredRoom(activeRoom.roomId); - setPhase("live"); - } - handleSignalMessage( - message, - generation, - activeRoom, - reauthenticated, - reauthenticated - ? signal.pendingHostQualitySettings(shareGeneration) - : null, - ); - }, - }, - ); - return signal; - }; - const createReplacementRoom = async (): Promise => { - try { - const response = await createRoom( - creationProfileRef.current.codeEntryPolicy, - creationProfileRef.current.roomPassword, - readPreferredRoomId(), - ); - const replacement = hostRoomFromCreated(response); - if (!isCurrentShare(generation, shareGeneration)) { - captured?.getTracks().forEach((track) => track.stop()); - disposeNativeShare(shareGeneration); - closeAbandonedRoom(replacement); - return; - } - if (!(await writeHostRoom( - replacement, - () => isCurrentShare(generation, shareGeneration), - ))) { - closeAbandonedRoom(replacement); - throw new Error("Host room is already open in another tab"); - } - roomRef.current = replacement; - setRoom(replacement); - const replacementSignal = connectSignal(replacement); - signalRef.current = replacementSignal; - replacementSignal.start(); - } catch (error) { - if (!isCurrentShare(generation, shareGeneration)) { - if (nativeStarted) disposeNativeShare(shareGeneration); - return; - } - activeGenerationRef.current = null; - disposeResources(false); - if ( - error instanceof ApiError && - error.status === 401 && - onAuthorizationRequired - ) { - onAuthorizationRequired(); - return; - } - setNoticeError(error, "room", "television"); - setPhase("error"); - } - }; - const signal = connectSignal(createdRoom); - signalRef.current = signal; + const signal = connectRoomSignal(createdRoom, shareGeneration); + if (hostRoomSessionAvailable) signal.startSharing(shareGeneration, qualitySettingsRef.current, routePolicyRef.current); + signal.start(); } catch (error) { if (!isCurrentShare(generation, shareGeneration)) { @@ -2724,6 +2815,7 @@ export function HostPage({ audio: boolean, path: NativeCapturePath, showCaptureBorder: boolean, + excludeAudio?: NativeCaptureTarget, ): Promise { const generation = activeGenerationRef.current; const shareGeneration = nativeShareGenerationRef.current; @@ -2750,12 +2842,16 @@ export function HostPage({ sourceSwitchRef.current !== token || nativeClientRef.current !== client ) return; + // Preserve an exclusion request even on failure: the App mutes source + // audio before replacing it, and the next picker must not default to all. + if (audio && excludeAudio) nativeAudioSelectionRef.current = { enabled: audio, exclude: excludeAudio }; await client.replaceShareSource( shareGeneration, target, audio, path, showCaptureBorder, + audio ? excludeAudio : undefined, ); if ( !isCurrentGeneration(generation) || @@ -2765,6 +2861,7 @@ export function HostPage({ return; } invalidateSenderQualityEvidence(); + nativeAudioSelectionRef.current = { enabled: audio, exclude: excludeAudio }; if (routePolicyRef.current.topologyOptimization) { signalRef.current?.send({ type: "reset-sender-quality" }); } @@ -3444,6 +3541,8 @@ export function HostPage({ {stream ? (