From 1b9f5bd2a27eb32d35e81e8f2c5d8d6e952bbd0a Mon Sep 17 00:00:00 2001 From: AAA <35992542+TNTcraftHIM@users.noreply.github.com> Date: Fri, 2 Oct 2026 01:36:45 +0800 Subject: [PATCH] feat: add room interactions and media sharing improvements MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Problem and outcome / 问题与结果 This phase adds lightweight room interaction without making media depend on chat, and adds Windows source-audio exclusion and optional SFU-only hosting at their existing owners. It also corrects confirmed encoder/capture ownership and native color-conversion defects. Host microphone commentary remains on the shared media path; room voice and 4K are outside this phase. ## Release notes ### 中文 这是一次功能更新:Piik 加入了文字聊天与表情互动,并完善了 Windows 声音采集、画面颜色和自部署选项。 #### 新功能 - **边看边聊。** 在可移动、可缩放的聊天窗口中发送消息,或开启弹幕。停止分享后,房间中的朋友仍可继续聊天。 - **表情互动。** 点击沙发上的人物,给自己发表情,或把番茄等表情送给朋友;各端按同一事件显示动画。 - **排除应用声音。** Windows App 分享整个屏幕时,可排除一个应用及其子进程的声音,适合同时使用语音软件。所选应用退出后,来源声音停止,避免恢复为未过滤的系统声音。 - **可选的纯 SFU 模式。** 自部署管理员可让共享画面和声音统一通过媒体服务器转发。默认仍为优先 P2P;App 公网邀请保持纯 P2P。 - **Windows HDR 转 SDR。** 原生采集与缩略图自动将 HDR 来源转换为 SDR,沿用现有观看链路。 #### 修复与改进 - 修复原生颜色转换的范围与矩阵处理,避免已复现的暗部丢失、高光裁切和颜色偏差。 - 修复共享编码器切换时使用过期需求或过期帧信息的问题,保护正在观看的画面;修复换源后旧采集事件可能影响新来源的问题。 - 统一新控件、浮窗与来源选择文案,补充中英文使用说明。 #### 使用与升级 聊天与表情需要新版站点页面和 Server;声音排除需要新版页面与支持此能力的 Windows App。聊天只保留本页最近收到的消息,刷新后清空,不补发历史,且不提供端到端加密。 HDR 转换输出为 SDR,不代表端到端 HDR 或多声道支持。既有房主麦克风与立体声能力保留。 ### English This feature update brings text chat and reactions to Piik, alongside improvements to Windows audio capture, picture colors and self-hosting options. #### New features - **Chat while watching.** Send messages in a movable, resizable window, or enable on-screen chat. Friends can keep chatting after the host stops sharing. - **Reactions.** Select a person on the sofa to react above yourself or send a tomato and other reactions to a friend. Participants animate the same shared event. - **Exclude application audio.** When sharing a whole screen with Windows App, exclude one application and its child processes, useful when using a separate voice app. Source audio stops if the selected app exits, preventing a return to unfiltered system sound. - **Optional SFU-only hosting.** Site operators can require shared video and audio to pass through the media server. P2P-first remains the default; App Public invite remains P2P-only. - **Windows HDR-to-SDR conversion.** Native capture and source thumbnails automatically convert HDR sources to SDR for the existing viewing path. #### Fixes and improvements - Correct native color-conversion ranges and matrices, addressing reproduced crushed shadows, clipped highlights and color shifts. - Correct stale demand and frame evidence during shared-encoder handoffs to protect the current picture. Prevent events from a replaced capture from affecting its successor. - Align new controls, floating panels and source-picker copy, with updated Chinese and English guides. #### Using this update Chat and reactions require updated site pages and Server. Audio exclusion requires updated pages and a Windows App that supports the capability. Chat retains only recent messages received by the current page, clears on refresh and does not replay history. It is not end-to-end encrypted. HDR conversion produces SDR; it does not add end-to-end HDR or surround sound. Existing host microphone commentary and stereo support remain available. ## Verification / 验证 - Web type checks, 78 test files / 1275 tests and production build: PASS. - Go core tests, formatting/vet and target builds: PASS; Darwin cgo remains platform-limited on this Windows runner. - Native build/check/color probes and synthetic HDR-to-SDR H.264/VP8 browser reception: PASS. - Access/privacy gate, Native H.264 SFU-only media gate and local Auto/H.264 Host gate: PASS. - Three real production pages: chat/reactions, stop/resume, idle-room join, host departure/return, offline draft recovery and invitation revocation: PASS. - Synthetic parent/child process-audio exclusion and target lifetime: PASS; no raw audio retained. - Windows candidate archive extraction and packaged Local startup/shutdown: PASS. - Documentation and website build: PASS. - [Candidate CI](https://github.com/TNTcraftHIM/Piik/actions/runs/36873766646) at `f26cd208d03bc0dcbe353f8cac217f993216b0a3`: Go core and race checks, Web tests, Server/container checks and Windows/macOS/Linux App packaging all PASS. Publication was skipped as intended. The owner accepted the UI/model and bounded local walkthrough for phase closure. Physical HDR/mixed displays, mobile keyboard, real voice applications across output devices and macOS/Linux capture remain unverified. Existing field reports without matched evidence are not claimed resolved. The owner authorized v1.7.0 publication on 2026-10-02. The final documentation-only commit records that authorization; product code matches the verified candidate above. ## Impact / 影响 Authorized version: **v1.7.0**, matching the local release planner. Current wire contracts remain Browser/server 23, Native control 9 and capture 7. Optional capabilities gate new functionality; existing room/media authority remains owned by the existing services. No new ports, message persistence, room voice or automatic room creation on page visit. Updated README, guides, standards, research evidence and TODO stay with the candidate phase. --- .env.example | 2 + AGENTS.md | 2 +- README.md | 1 + README.zh-CN.md | 1 + deploy/container/.env.example | 2 + docs/README.md | 3 +- .../0005-automatic-hybrid-media-routing.md | 8 + docs/adr/0008-window-scoped-audio-capture.md | 13 +- docs/guide/getting-started.md | 31 + docs/guide/getting-started.zh-CN.md | 28 +- docs/guide/troubleshooting.md | 5 +- docs/guide/troubleshooting.zh-CN.md | 3 +- docs/operations/self-hosting.md | 9 + docs/operations/self-hosting.zh-CN.md | 7 + docs/project-memory.md | 10 +- docs/research/browser-local-encoding-pool.md | 170 ++++- docs/research/media-fidelity.md | 162 +++++ docs/research/native-client-media.md | 65 +- docs/research/room-interactions.md | 260 +++++++ docs/research/room-voice.md | 287 -------- docs/standards/configuration.md | 8 + docs/standards/engineering.md | 5 +- docs/standards/media-quality.md | 48 +- docs/standards/naming.md | 3 + docs/standards/presentation-lifecycle.md | 9 + docs/standards/rooms-access.md | 22 +- docs/standards/routing-transport.md | 12 +- docs/standards/visual-language.md | 11 + docs/status.md | 20 +- docs/todo.md | 94 +-- docs/verification-status.md | 22 +- internal/app/app.go | 17 +- internal/app/app_test.go | 8 +- internal/app/loopback/server.go | 17 +- internal/app/mediaedge/edge_test.go | 57 +- internal/app/mediaedge/groups.go | 30 +- internal/app/mediaedge/groups_test.go | 59 ++ internal/app/mediaedge/publication_test.go | 8 +- internal/app/mediaedge/source.go | 17 +- internal/app/nativeaudio/encoder_test.go | 52 ++ internal/app/nativecapture/probe.go | 49 +- internal/app/nativecapture/probe_test.go | 16 + internal/app/nativecapture/stream.go | 10 +- internal/app/nativecontrol/session.go | 25 +- internal/app/nativecontrol/session_test.go | 29 + internal/app/nativecontrol/wire.go | 46 +- internal/app/nativehost/audio_mix_test.go | 79 ++- internal/app/nativehost/audio_test.go | 13 +- internal/app/nativehost/session.go | 45 +- internal/app/nativehost/session_test.go | 117 ++++ internal/server/app/app.go | 1 + internal/server/app/app_test.go | 17 +- internal/server/app/routes.go | 5 +- internal/server/config/config.go | 9 + internal/server/config/config_test.go | 15 + internal/server/protocol/client.go | 48 +- internal/server/protocol/contract_test.go | 41 +- internal/server/protocol/fixture_test.go | 2 + internal/server/protocol/interactions.go | 194 ++++++ internal/server/protocol/server.go | 63 ++ internal/server/protocol/types.go | 9 +- internal/server/route/api.go | 2 + internal/server/route/candidates.go | 5 +- internal/server/route/controller.go | 9 +- internal/server/route/sfu_only_test.go | 64 ++ internal/server/signal/interactions.go | 62 ++ internal/server/signal/interactions_test.go | 144 ++++ internal/server/signal/router.go | 4 + internal/server/signal/server.go | 212 +++++- internal/server/signal/server_test.go | 157 +++++ internal/server/signal/session.go | 3 + internal/server/signal/sfu_only_test.go | 34 + native/capture/windows/README.md | 23 +- native/capture/windows/build.ps1 | 17 +- native/capture/windows/capture_color.h | 34 + native/capture/windows/capture_sdr.h | 209 ++++++ native/capture/windows/capture_target.cpp | 14 +- native/capture/windows/capture_target.h | 1 + .../capture/windows/capture_target.test.cpp | 2 + .../capture/windows/frame_converter.probe.cpp | 218 ++++++ native/capture/windows/h264_decoder.h | 7 + native/capture/windows/h264_encoder.cpp | 3 + native/capture/windows/main.cpp | 85 ++- native/capture/windows/process_audio.cpp | 53 +- native/capture/windows/process_audio.h | 4 +- scripts/embedded-sfu-gate.ts | 10 +- scripts/embedded-sfu-page.ts | 46 +- src/client/App.tsx | 3 + .../components/living/CaptureSourcePicker.tsx | 44 +- src/client/components/living/Couch.tsx | 45 +- .../components/living/FloatingPanel.tsx | 222 ++++++ src/client/components/living/LauncherForm.tsx | 2 +- .../living/MicrophoneDeviceSelect.tsx | 2 +- src/client/components/living/Pawn.tsx | 4 +- .../components/living/PlaybackControls.tsx | 5 +- .../components/living/RoomChatOverlay.tsx | 55 ++ .../components/living/RoomInteractions.tsx | 228 +++++++ src/client/components/living/RoomReaction.tsx | 119 ++++ src/client/components/living/Tooltip.tsx | 8 +- .../components/living/floating-panel.css | 33 + src/client/components/living/hints/index.tsx | 9 +- .../components/living/hints/interactions.tsx | 59 ++ .../components/living/participant-motion.ts | 6 +- src/client/components/living/primitives.tsx | 5 +- .../components/living/room-chat-overlay.css | 25 + .../components/living/room-interactions.css | 195 ++++++ .../components/living/use-theater-mode.ts | 22 + src/client/lib/debug.ts | 2 +- src/client/lib/identity-hash.ts | 6 + src/client/lib/room-interactions.ts | 169 +++++ src/client/lib/signaling.ts | 180 ++++- src/client/locales/en.ts | 66 +- src/client/locales/zh.ts | 66 +- src/client/media/browser-encoding-output.ts | 57 +- src/client/media/browser-encoding-pool.ts | 88 ++- src/client/media/browser-encoding-producer.ts | 22 +- src/client/native/client.ts | 14 +- src/client/native/wire.ts | 1 + src/client/pages/AppLauncherPage.tsx | 2 +- src/client/pages/ControlsPreview.tsx | 88 ++- src/client/pages/HostPage.tsx | 636 ++++++++++-------- src/client/pages/TooltipPreviewPage.tsx | 2 + src/client/pages/ViewerPage.tsx | 31 +- src/client/pages/controls-preview.css | 2 + src/client/styles.css | 56 +- src/client/ui/icons.tsx | 6 + src/client/ui/visual-kinds.ts | 7 + src/shared/protocol.ts | 61 +- src/shared/room-interactions.ts | 18 + tests/browser-debug.test.ts | 3 +- tests/browser-encoding-output.test.ts | 181 ++++- tests/browser-encoding-pool.test.ts | 336 ++++++++- tests/browser-encoding-producer.test.ts | 50 +- tests/client-state.test.ts | 265 ++++++++ tests/fixtures/wire-samples.json | 136 +++- tests/host-quality-ownership.test.ts | 109 ++- tests/native-client.test.ts | 52 +- tests/protocol-fixture.test.ts | 8 + tests/protocol.test.ts | 61 +- tests/release-automation.test.ts | 2 +- tests/room-interactions.test.ts | 273 ++++++++ 141 files changed, 7004 insertions(+), 986 deletions(-) create mode 100644 docs/research/media-fidelity.md create mode 100644 docs/research/room-interactions.md delete mode 100644 docs/research/room-voice.md create mode 100644 internal/server/protocol/interactions.go create mode 100644 internal/server/route/sfu_only_test.go create mode 100644 internal/server/signal/interactions.go create mode 100644 internal/server/signal/interactions_test.go create mode 100644 internal/server/signal/sfu_only_test.go create mode 100644 native/capture/windows/capture_color.h create mode 100644 native/capture/windows/capture_sdr.h create mode 100644 native/capture/windows/frame_converter.probe.cpp create mode 100644 src/client/components/living/FloatingPanel.tsx create mode 100644 src/client/components/living/RoomChatOverlay.tsx create mode 100644 src/client/components/living/RoomInteractions.tsx create mode 100644 src/client/components/living/RoomReaction.tsx create mode 100644 src/client/components/living/floating-panel.css create mode 100644 src/client/components/living/hints/interactions.tsx create mode 100644 src/client/components/living/room-chat-overlay.css create mode 100644 src/client/components/living/room-interactions.css create mode 100644 src/client/components/living/use-theater-mode.ts create mode 100644 src/client/lib/identity-hash.ts create mode 100644 src/client/lib/room-interactions.ts create mode 100644 src/shared/room-interactions.ts create mode 100644 tests/room-interactions.test.ts diff --git a/.env.example b/.env.example index 3b71f8a..20ba2eb 100644 --- a/.env.example +++ b/.env.example @@ -35,6 +35,8 @@ SFU_UDP_PORT= SFU_LISTEN_HOST=0.0.0.0 # Optional advertised IPv4 override when the listener's address is behind NAT. SFU_PUBLIC_IP= +# Require server media for every room instead of P2P-first. Requires SFU_UDP_PORT. +SFU_ONLY=false # Ordinary peer ICE discovery. Production requires at least one STUN URL. # These are advertised addresses, not local listener addresses. They may use diff --git a/AGENTS.md b/AGENTS.md index 171e7d1..131e228 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,7 +30,7 @@ [routing/transport](./docs/standards/routing-transport.md), [media quality](./docs/standards/media-quality.md), and [presentation/lifecycle](./docs/standards/presentation-lifecycle.md). -- Media stays automatic and P2P-first; central services provide room authority, +- Media stays automatic and defaults to P2P-first; central services provide room authority, signaling, STUN, observability, and bounded embedded SFU/UDP fallback using mature LiveKit media components under [ADR-0013](./docs/adr/0013-embedded-node-local-media.md). Route changes must preserve the single-graph, single-operation model in diff --git a/README.md b/README.md index 58e85d9..a48d317 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,7 @@ watch in their browsers. - **Share from Piik App or an existing Piik site.** Choose a screen, window, browser tab or camera; available sources and audio depend on the platform. - **Direct connections first.** Media travels between participants where possible (P2P). A self-hosted server can provide automatic media forwarding (SFU) as a fallback. - **Rooms you control.** Invitations, room codes and optional room passwords, for one host and up to 20 viewers. +- **Chat while watching.** Send messages and reactions to friends, with optional on-screen chat. - **Flexible viewing.** Light and dark themes, playback controls, picture-in-picture and a connection topology view. - **One program to host a site.** Web UI, room management and optional media forwarding are packaged together. diff --git a/README.zh-CN.md b/README.zh-CN.md index 511d839..11131a2 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -33,6 +33,7 @@ Piik 是一个免费开源的屏幕共享工具,适合游戏围观、一起看 - **用 Piik App 或已有 Piik 站点分享。** 支持屏幕、窗口、浏览器标签页和摄像头;可选来源与音频能力取决于平台。 - **优先直接连接。** 画面优先在设备之间传输(P2P),自建站点可启用媒体转发(SFU)自动兜底。 - **房间由你管理。** 支持邀请链接、房间号和可选的房间密码,一位房主、最多 20 位观众。 +- **边看边聊。** 发送文字、和朋友互送表情,也可开启弹幕。 - **灵活的观看方式。** 支持深浅主题、播放控制、画中画和连接拓扑查看。 - **一个程序就能建站。** 网页、房间管理和可选的媒体转发打包在同一服务端程序中。 diff --git a/deploy/container/.env.example b/deploy/container/.env.example index c83e7a5..89c06ac 100644 --- a/deploy/container/.env.example +++ b/deploy/container/.env.example @@ -8,6 +8,8 @@ SITE_ACCESS_PASSWORD= # Optional automatic media fallback: set BOTH values and allow UDP 7882. # SFU_UDP_PORT=7882 # SFU_PUBLIC_IP=203.0.113.10 +# Optional: require SFU for all rooms instead of P2P-first (requires SFU_UDP_PORT). +# SFU_ONLY=true # Optional NAT survey: also allow UDP 3479 and 3480. # NAT_PREDICTION_ENABLED=true diff --git a/docs/README.md b/docs/README.md index 9b51ca1..4666b81 100644 --- a/docs/README.md +++ b/docs/README.md @@ -99,7 +99,8 @@ and [website preview and publishing](./operations/website.md). | Why a design was chosen | [Architecture decisions](./adr/) | | Measurements and platform limits | [Research](./research/) and [verification status](./verification-status.md) | | Camera and Host microphone | [Capture scope and limits](./research/camera-and-microphone.md) | -| Room voice proposal and media-fidelity boundaries | [Design assessment](./research/room-voice.md) | +| Unreleased text, reactions and floating room tools | [Room-interaction candidate](./research/room-interactions.md) | +| Stereo verification and HDR-to-SDR assessment | [Media fidelity](./research/media-fidelity.md) | ## Shared conventions diff --git a/docs/adr/0005-automatic-hybrid-media-routing.md b/docs/adr/0005-automatic-hybrid-media-routing.md index 4b3b970..2fc2de6 100644 --- a/docs/adr/0005-automatic-hybrid-media-routing.md +++ b/docs/adr/0005-automatic-hybrid-media-routing.md @@ -28,6 +28,14 @@ Hosted and Local always use this controller. The former rollout switch and separate Host-star authority are removed; SFU availability is still selected by configuration and the room's privacy policy, not by another routing mode. +The optional Hosted `SFU_ONLY` policy restricts this controller's candidate +eligibility to SFU. It reuses the publication/subscription ledger, deadlines, +first-frame commit and recovery. It does not create another graph or scheduler. +It disables Peer relay and background Peer work; admission failure does not +relax the restriction. The service rejects an explicit peer-only request instead +of treating it as permission to forward media. Default P2P-first and App Local +behavior are unchanged. + ### One Capacity Rule Every non-server endpoint uses one server-authoritative steady outbound media- diff --git a/docs/adr/0008-window-scoped-audio-capture.md b/docs/adr/0008-window-scoped-audio-capture.md index 9283c79..820837d 100644 --- a/docs/adr/0008-window-scoped-audio-capture.md +++ b/docs/adr/0008-window-scoped-audio-capture.md @@ -2,7 +2,7 @@ - Status: accepted for Browser capture and the Windows App audio paths - Date: 2026-08-21 -- Last updated: 2026-08-27 +- Last updated: 2026-10-01 ## Context @@ -26,8 +26,8 @@ platform probe and physical gate report it. Track presence does not prove its source or isolation. 3. Returned screen-audio tracks use `contentHint = "music"`. Current bounded sender ceilings and narrow Opus stereo answer normalization are owned by - [media quality](../standards/media-quality.md). There is no Web Audio mixer, - resampler, second representation, or independent audio clock. + [media quality](../standards/media-quality.md). Audio scope adds no second + representation or independent clock; Host commentary uses the existing mixer. 4. Browser source switching replaces the current screen-audio track in the same persistent media stream. Local Viewer play, mute, and volume stay inside the native media element. @@ -38,6 +38,13 @@ platform probe and physical gate report it. probe and physical gate pass. WGC/MF video and the App media boundary are owned by [ADR-0010](./0010-cross-platform-client-runtime.md); measurements are in [Native App media](../research/native-client-media.md). +7. Windows App screen sharing can optionally exclude one selected process tree + through WASAPI's exclusion mode, on build 20348+ with a successful capability + probe. This mode spans all render endpoints; it is not restricted to the + default playback device. Reuse source enumeration and the existing PCM/mixed + output. Generic Browser capture and arbitrary unrelated process sets are not + covered. [Media quality](../standards/media-quality.md#screen-audio) owns + selection lifetime and failure behavior. ## Consequences diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index 776a68a..983962e 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -7,6 +7,7 @@ watching, using Piik online, using Piik App and sharing through an existing site [Join a room](#join-a-friends-room) · [Use online](#use-piik-online) · [Piik App](#share-with-piik-app) · [Existing site](#share-from-an-existing-site) · +[Chat and reactions](#chat-and-reactions) · [Troubleshooting](#when-something-gets-in-the-way) Viewers need a browser and an invitation. To start sharing, download Piik App @@ -154,6 +155,36 @@ then select a picture card to start sharing. Device names and previews depend on browser permissions; while a camera is shared, other cameras are listed by name. Use headphones to keep speaker sound from feeding back into the microphone. +### Exclude a voice app from screen audio + +On supported Windows Apps, open the **Screens** tab and enable **System sound**. +Under **Exclude application audio**, select a window belonging to the voice app, +then choose the screen to share. Open the app's window and refresh if it is absent. +This excludes one process and its children. Other apps remain audible, including +those playing through other output devices. +It does not mute your Piik microphone or prevent speaker sound reaching a microphone. + +If the excluded app closes or restarts, source audio stops. Use **Switch source** +and select it again. Turning source audio off retains the exclusion choice; +select **None** to restore unfiltered system audio. This control requires an updated page and a Windows App +whose capability probe supports process exclusion; it is not a Browser-capture filter. + +## Chat and reactions + +Once the host starts the first share, participants can select **Chat** below the +sofa to send messages. Select a person on the sofa to open their reaction menu: +your own reactions appear above you; reactions sent to a friend fly towards them. +The chat window can be moved and resized without rearranging the page. + +**On screen**, beside the message box or in the viewer's playback bar, shows new +messages over the picture. It is off by default and only changes your own view. +System video fullscreen and picture in picture cannot show these overlays. + +Stopping a share keeps the room's chat available. App Local and Public invite +rooms require the App to stay open. Each page retains only its latest received +messages: refreshing clears them, and late joiners receive no earlier history. +Chat travels through the room's server and is **not end-to-end encrypted**. + ## When something gets in the way For sound, capture, startup or playback issues, see [Troubleshooting](./troubleshooting.md). diff --git a/docs/guide/getting-started.zh-CN.md b/docs/guide/getting-started.zh-CN.md index ef3ef23..147e12f 100644 --- a/docs/guide/getting-started.zh-CN.md +++ b/docs/guide/getting-started.zh-CN.md @@ -6,7 +6,8 @@ Piik 支持一位房主分享屏幕,最多 20 位受邀观众通过浏览器 本指南介绍加入房间、使用在线版、使用 Piik App,以及通过已有站点分享的方法。 [加入房间](#加入朋友的房间) · [在线版](#使用在线版) · -[Piik App](#用-piik-app-分享) · [已有站点](#使用已有站点分享) · [问题排查](#遇到问题了) +[Piik App](#用-piik-app-分享) · [已有站点](#使用已有站点分享) · +[聊天与表情](#聊天与表情) · [问题排查](#遇到问题了) 观众只需要浏览器和邀请链接。 分享画面推荐下载 Piik App,选择 **公网邀请**。 @@ -133,6 +134,31 @@ Linux 原生采集还需要[系统组件](../../native/capture/linux/README.md) 设备名称和可用预览取决于浏览器权限;正在分享摄像头时,其他摄像头只显示名称。 建议佩戴耳机,避免扬声器声音被麦克风再次收进去。 +### 分享屏幕时排除语音软件的声音 + +在支持此功能的 Windows App 中,打开 **屏幕** 页签并开启 **系统声音**, +在 **排除应用声音** 中选择语音软件的窗口,再选择要分享的屏幕。 +未列出的应用请先打开窗口,再刷新。此功能排除所选进程及其子进程, +其他应用的声音仍会被分享,包括输出到其他播放设备的声音。 +它不影响 Piik 麦克风,也不能消除扬声器被麦克风收音产生的回声。 + +所选应用关闭或重启后,来源声音会停止;请通过 **切换来源** 重新选择。 +暂时关闭来源声音会保留排除选择;需要恢复完整的系统声音时,选择 **不排除**。 +此控件需要新版页面和通过进程排除能力检测的 Windows App,不适用于浏览器自带的分享选择器。 + +## 聊天与表情 + +房主首次开始分享、创建房间后,参与者可以点击沙发下方的 **聊天** 发送消息。 +点击沙发上的人物可打开表情菜单:给自己发的表情出现在头顶,送给朋友的表情会飞向对方。 +聊天窗口可以移动和缩放,不会挤动页面。 + +输入框旁和观众播放控制栏中的 **弹幕** 可以把新消息显示在画面上,默认关闭,只影响自己的观看。 +系统视频全屏和画中画无法显示弹幕。 + +停止分享后仍可继续聊天;App 的本地房间和公网邀请需要保持 App 运行。 +每个页面只保留本页最近收到的消息,刷新后清空,后来加入的人看不到此前的记录。 +聊天消息经过房间所在的服务器,**不提供端到端加密**。 + ## 遇到问题了 声音、采集、启动或播放遇到问题时,先看[问题排查](./troubleshooting.zh-CN.md)。 diff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.md index b5ae60e..fed063f 100644 --- a/docs/guide/troubleshooting.md +++ b/docs/guide/troubleshooting.md @@ -147,8 +147,9 @@ Try these in order, stopping when the picture arrives: 6. **Invite a friend on another network.** A Viewer who can receive the picture and has spare forwarding capacity may give Piik another path to you. Piik chooses this automatically; extra people help only when those connections work. -7. **Use a site with media fallback.** Its operator must enable SFU forwarding, - and the Host must turn off **Privacy mode** before sharing. The public +7. **Use a site with media fallback.** Its operator must enable SFU forwarding. + If the site offers **Privacy mode**, the Host must turn it off before sharing. + Sites configured for **Server media** already require SFU. The public `demo.piik.tv` site and the App's **Public invite** mode do not provide SFU. [Self-hosting](../operations/self-hosting.md#optional-media-fallback) is an advanced option. Both P2P and SFU media currently use UDP; a network that blocks diff --git a/docs/guide/troubleshooting.zh-CN.md b/docs/guide/troubleshooting.zh-CN.md index 92d197b..6bae1e0 100644 --- a/docs/guide/troubleshooting.zh-CN.md +++ b/docs/guide/troubleshooting.zh-CN.md @@ -123,7 +123,8 @@ NAT(网络地址转换)让多台设备共用一个上网地址。不同路 修改路由器工作模式前,先备份配置。 6. **请另一网络的朋友加入。** 如果对方能正常看到画面,且有余力转发,Piik 可能经由这位观众 把画面送到你这里。线路由 Piik 自动选择,能否帮上忙取决于这些连接是否可用。 -7. **使用有媒体兜底的站点。** 管理员需要先启用 SFU 转发,房主在分享前关闭 **隐私模式**。 +7. **使用有媒体兜底的站点。** 管理员需要先启用 SFU 转发。如果页面提供 **隐私模式**, + 房主应在分享前关闭;配置为 **服务器转发** 的站点已经强制使用 SFU。 公开站点 `demo.piik.tv` 和 App 的 **公网邀请** 模式均没有 SFU 兜底。 有服务器的用户也可以按 [部署指南](../operations/self-hosting.zh-CN.md#可选启用媒体兜底) 配置,属于进阶操作。 目前 P2P 和 SFU 媒体都使用 UDP;网络完全禁止 UDP 时,仍需换网络或由管理员放行。 diff --git a/docs/operations/self-hosting.md b/docs/operations/self-hosting.md index fb73ea0..a8c1108 100644 --- a/docs/operations/self-hosting.md +++ b/docs/operations/self-hosting.md @@ -112,6 +112,9 @@ permission; `self` allows this site to request access. Allow **TCP 80/443** for HTTPS and **UDP 3478** for STUN in the server firewall and cloud security group. Keep TCP 8787 private. The STUN hostname must resolve directly to the server; a CDN HTTP proxy does not forward its UDP traffic. +See the [complete port table](../standards/configuration.md#public-and-private-ports) +for optional services. Embedded SFU media shares the single UDP port set by +`SFU_UDP_PORT`. Open `https://share.example.com/healthz`; it should return `{"status":"ok"}`. Then open the site, share a screen and join from another device. This initial @@ -122,9 +125,15 @@ configuration uses P2P media, so participants need a usable UDP path. Add `SFU_UDP_PORT=7882` to `.env`, allow UDP 7882, and restart Piik to enable automatic SFU fallback. When the server is behind NAT, also set `SFU_PUBLIC_IP` to its reachable public IPv4 address. The same binary provides the fallback. +Allow server forwarding capacity and outbound bandwidth for Viewers actually +using the SFU. The Host must turn off **Privacy mode** before sharing to allow this route. Both direct and SFU media need a usable UDP connection. +To send every room through your server, also set `SFU_ONLY=true` and restart. +The sharing settings will show **Server media**. Plan server bandwidth for every +Viewer; this mode does not fall back to P2P if the server route fails. + ## Keep it running and update For automatic startup, use the [systemd or container guide](./service-management.md). diff --git a/docs/operations/self-hosting.zh-CN.md b/docs/operations/self-hosting.zh-CN.md index 9ad9cdd..13501a2 100644 --- a/docs/operations/self-hosting.zh-CN.md +++ b/docs/operations/self-hosting.zh-CN.md @@ -101,6 +101,8 @@ nginx 可参考[配置示例](../../deploy/nginx/piik.conf.example)。如果沿 在服务器防火墙和云平台安全组中放行 **TCP 80/443**(HTTPS)和 **UDP 3478**(STUN)。 TCP 8787 仅供本机反向代理访问。STUN 域名需要直接解析到服务器,不能只经过 CDN 的 HTTP 代理。 +可选服务的端口见[完整端口表](../standards/configuration.md#public-and-private-ports)。 +内置 SFU 的媒体流复用 `SFU_UDP_PORT` 指定的单个 UDP 端口。 打开 `https://share.example.com/healthz`,应返回 `{"status":"ok"}`。 随后打开站点,分享一个画面,并用另一台设备加入验证。 @@ -111,9 +113,14 @@ TCP 8787 仅供本机反向代理访问。STUN 域名需要直接解析到服务 在 `.env` 中添加 `SFU_UDP_PORT=7882`,放行 UDP 7882,再重启 Piik,即可启用自动 SFU 兜底。 如果服务器处于 NAT 后方,还需将 `SFU_PUBLIC_IP` 设置为外部可达的公网 IPv4 地址。 这项功能由同一个服务端程序提供。 +服务器需要为实际经由 SFU 观看的观众承担转发负载和出口带宽。 房主需要在开始分享前关闭 **隐私模式**,才会允许使用这条线路。 直连和 SFU 媒体都需要可用的 UDP 通路。 +如需所有房间都通过服务器转发,再设置 `SFU_ONLY=true` 并重启。 +分享设置中会显示 **服务器转发**。请为每位观众预留服务器带宽; +服务器线路失败时,此模式不会改走 P2P。 + ## 长期运行与更新 需要开机启动时,参阅 [systemd 与容器配置](./service-management.zh-CN.md)。 diff --git a/docs/project-memory.md b/docs/project-memory.md index 8b4f721..e257a60 100644 --- a/docs/project-memory.md +++ b/docs/project-memory.md @@ -1,6 +1,6 @@ # Project Memory -Last updated: 2026-09-29 +Last updated: 2026-10-01 Piik is private, low-latency screen sharing for one Host and up to 20 authenticated friends. The current product surface is Web Host, Web Viewer, and @@ -43,7 +43,8 @@ blank keeps the self-contained site open. ## Product Map - [Rooms and access](./standards/rooms-access.md) owns room codes, - invitations, code entry, credentials, and lightweight/SQLite persistence. + invitations, code entry, credentials, room interactions and lightweight/SQLite + persistence. - [Routing and transport](./standards/routing-transport.md) owns P2P-first distribution, endpoint capacity, SFU fallback, recovery, and privacy limits. - [Capture and media quality](./standards/media-quality.md) owns Browser capture, @@ -70,8 +71,9 @@ implementation and routine UI detail. available, and anonymous provenance distinguishes ordinary from predicted selected paths. The configuration default remains stock ICE. The only application fallback is one bounded embedded SFU/UDP - publication; Piik configures no TURN or TCP media route. A pre-share - peer-only policy can exclude that SFU suffix for one share generation. + publication; Piik configures no TURN or TCP media route. Hosted operators may + require SFU-only media through the same controller. On P2P-first sites, a + pre-share peer-only policy can exclude that SFU suffix for one share generation. - Every endpoint shares one steady outbound-copy cap, default `2` and limited to `1..3`. One room controller owns one committed graph, one reconcile loop, and one serial child operation. First decoded frame commits availability work; diff --git a/docs/research/browser-local-encoding-pool.md b/docs/research/browser-local-encoding-pool.md index 74a2b9d..873c448 100644 --- a/docs/research/browser-local-encoding-pool.md +++ b/docs/research/browser-local-encoding-pool.md @@ -93,6 +93,14 @@ maximum queued service; reverse feedback is unshaped. They are not a model of public NAT, geographic RTT or every congestion/loss pattern. Reduced quality and temporary stalls under real constraints remain possible. +A 2026-09-30 actual-tab fault check separated A at 400 kbps from healthy B, +then failed only A's producer. Both H264 and VP8 retained the outgoing +connections, returned to ordinary encoding and retired all pooled groups. +B retained 1904x1048 at about 30 fps; its largest observed frame-callback gap +around fallback was 271 ms for H264 and 80 ms for VP8. Source-wide fallback +therefore has a measurable handoff cost, even without a resolution drop or +connection failure. This does not establish low-end hardware capacity. + ### Balanced Startup And Recovery Serial Chrome 152 checks on 2026-09-25 reproduced two composition defects in @@ -316,12 +324,19 @@ complexity: changing to simple bars restored 1080p, retained after restoring the textured scene. WebRTC's [quality scaler](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/modules/video_coding/utility/quality_scaler.cc) uses QP thresholds and sampling history as well as bandwidth-related adaptation. Its default two-second checks and longer sampling after down-adaptation can -outlast a short disturbance. Native Chrome 154 logging also confirmed a stable -QP deadband with the NVIDIA H264 MFT: after one 1048p-to-786p reduction, smoothed -QP stayed between the recorded 24/37 thresholds for the remaining 90-second -observation. Capture remained full size, but no low-QP upscale was requested. -This is not merely old samples taking time to expire; restored bandwidth alone -does not require this scaler to restore resolution. +outlast a short disturbance. A Chrome 154 ordinary-sender tab control on +2026-09-30 remained at 524p / about 30 fps after 120 recovery seconds following two +one-second rate/loss pulses. Actual capture stayed 1904x1048 / 30 fps; the native +log's slow QP filter settled around 27, above its upscale threshold of 24. +Changing only the scene to simple moving bars restored full size in about +12 seconds. Restoring the textured scene then retained full size for the +remaining 30 seconds. The sender made only its two startup parameter writes; +no pool, reconnect, preference reset or frame reset caused that recovery. +This confirms a content-dependent stable state, not merely stale samples or +a blocked recovery operation. Restored bandwidth alone does not require this +scaler to restore resolution. The scaler and default-threshold source matched +inspected upstream main on that date; this does not establish every device's +behavior or the field report's cause. A matched three-arm comparison used fixed 5 Mbps, alternating 5.0/4.9 Mbps every 500 ms, and repeated identical 5 Mbps writes. Native reconfiguration @@ -335,6 +350,149 @@ perceptual blur measurement. They do not justify delaying real demand, bitrate compensation or periodic resets; equal product updates are already deduplicated. The field's repeated brief blur still lacks matching Host histories. +**Hardware encoder startup can block sibling encoding.** On 2026-10-01, +Windows Chrome 154.0.8037.58 tests added three independent Canvas H264 producers, +coinciding with 174/221/225 ms gaps in the existing producer; captured frames +continued within 47 ms. The new producers never joined the pool or replaced its +output, and the existing +producer's parameters did not change. Their NVIDIA initialization took +161/163/162 ms. Substituting software VP8, confirmed by RTCStats as libvpx, gave +60/48/109 ms existing-producer gaps in the same startup windows. +A subsequent trace recorded 148 ms of `CreateAndInitializeVEA` on the shared +Renderer Media thread, matching GPU-side initialization. Frames timestamped +during that interval entered encoding together on the same thread after it +returned. This establishes blocking in the local reproduction, consistent with +[Chromium's synchronous initialization IPC](https://github.com/chromium/chromium/blob/154.0.8037.58/media/mojo/clients/mojo_video_encode_accelerator.cc#L117). +The trace buffer lost events elsewhere; this conclusion uses complete task and +frame events, not missing events. It does not identify the driver's internal +cost, explain earlier downscaling or attribute unmatched field reports. No +codec-default change, spare prewarmed encoder or pool reset follows from it. +The Windows [VEA provider](https://github.com/chromium/chromium/blob/154.0.8037.58/media/mojo/services/mojo_video_encode_accelerator_provider.cc) +already assigns each service a dedicated COM STA runner; the trace likewise +places the old service's encoding and new initialization on different threads. +Asynchronous Renderer initialization is therefore a plausible upstream repair, +not merely moving the wait onto another shared queue. It still needs a patched +Browser comparison, including initialization failure and callback ordering; +shared GPU/driver cost remains unproved. + +**Optional reuse must preserve the existing rate owner.** A two-Viewer, +textured-tab H264 comparison exposed a Piik handoff defect. During recovery +from a one-second rate/loss pulse, a healthy Viewer moved from a 5 Mbps group +to a lower-budget group because its sampled output was 29 rather than 27 fps +at the same 1048p. Both reported `qualityLimitationReason:none`, but the same +NVIDIA encoder implementation's interval-average QP was about 51 on the +candidate versus 37 on the current output; the candidate then fell to 786p. The +[stats definition](https://www.w3.org/TR/webrtc-stats/#dom-rtcqualitylimitationreason-none) +describes resolution/FPS limitation, not equal quantization quality. + +Pending membership raised the candidate's budget before the existing +lower-budget rejection ran. The candidate repair checks this constraint before +membership and reuses one non-regression predicate before submitting a pending +selection. FPS comparisons also respect the Host ceiling; a sampled 32 fps +under a 30 fps ceiling is not an improvement over 30 fps. Initial output and +necessary downgrades retain their path, without a QP policy, timer or encoder reset. + +Selection also needs proof at the actual write boundary: while waiting for a key, +a candidate can downscale after passing the sampled comparison. The output now +asks the pool to revalidate current authority, budget and actual key dimensions +before replacing its queue or writing bytes. Rejection clears both pending owners, +invalidates a contradicted sample and retains prior output and recovery requests. +H264/VP8 display-capture checks preserve live/paused settings, source replacement +and ordinary fallback; played audio remains present. Brief video stalls remain. + +The bounded H264 and VP8 two-Viewer controls for this admission repair retained +the healthy sibling at 1048p and about 29 fps through both recovery intervals. +The constrained Viewer still adapted and recovered. This establishes one +avoidable sibling degradation path, not the field room's cause or elimination +of all brief blur. The affected admission rules predate the display-isolation +repair. Raw observations stay in ignored fixtures; comparisons read native stats +without calling the producer's startup-transition helper. + +Current demand also matters at group admission. Two-Viewer traces showed a +newcomer raising an incumbent group's allocation because matching used the +previous applied budget. The incumbent then needed another encoder, while the +newcomer's former encoder retired. Matching current native demand prevents that +exchange; a pending newcomer cannot establish compatibility with its own vote. +Cancellation must stop at accepted output writing, including a pause that delays +its completion. These are ownership repairs, not suppression of real demand +changes or removal of the measured hardware initialization cost. + +**Quiet content is not rate compatibility.** A separate fixed-allocation +comparison kept the two outgoing H264 ceilings at 3 and 5 Mbps, with no network +shaping, while alternating simple and textured moving content. Actual native +allocations stayed at those ceilings during the later scene cycles. Reuse based +on a quiet 500 ms byte-rate sample repeatedly moved the weaker child into the +5 Mbps producer; returning motion split it out again. Five producers were created +over the run. Requiring the candidate's rate budget to fit the child's allocation +at initial admission, optional reuse and pending revalidation retained two +producers with no scene-driven handoffs. The healthy H264 sibling retained full +1048p; the weaker output continued native adaptation. VP8 also retained two +producers through the same scene changes. Three controlled regressions fail +when these checks are removed, including a paused pending child whose candidate +budget rises. Unequal demands may retain an additional producer even during a +quiet scene; bounded membership and native recovery remain unchanged. These +checks remove unnecessary encoder churn, not all content-dependent adaptation. + +**A known allocation should initialize the local encoder.** A replacement local +PeerConnection otherwise starts its own estimate at WebRTC's default 300 kbps, +even when the outgoing connection already allocates 4 Mbps. Chrome 154's native +logs confirm this second cold start. On the same isolated 1904x1048 textured tab, +seeding only the local answer with that 4 Mbps allocation increased VP8 output +in the first second from 6 to 27 frames; its first sampled target was 3.33 Mbps +rather than 2.70 Mbps. The 400 kbps control retained its native 333 kbps initial +encoder target and two first-second frames with either version. These bounded +results do not promise faster first frames or remove weak-network adaptation. + +The hint uses the existing Host-bounded allocation, not a multiplier or floor. +WebRTC reads the send configuration from the +[remote description](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/pc/channel.cc) +and accepts a positive `x-google-start-bitrate` in its +[codec bitrate configuration](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/media/engine/webrtc_media_engine.cc). +An offer-only control lost the hint in Chrome's answer and did not improve +startup. Piik therefore changes only its owned local answer; external SDP, +codec identity and subsequent sender-budget updates stay unchanged. The +application writes this hint once per producer, not on each budget change. +The full two-Viewer VP8 follow-up retained the healthy child at 1048p through +two one-second 400 kbps / 4% loss pulses at 40 ms configured RTT. The constrained +child still down-adapted and froze during recovery, but regained 1048p within +each 40-second recovery window. This is not a freeze-free claim. A separate +audio/loss check delivered all 97 scheduled audio pulses to both played outputs; +H264 quiet-start, live/paused settings, source replacement and ordinary-fallback +checks passed. All owned test connections, processes and profiles retired. + +**Residual VP8 stalls do not require an output wait.** The ordinary two-Viewer +control with the same source and rate/loss pulses also froze and downscaled, +ending the recovery windows at 786p and 524p. It entered the second pulse at a +lower resolution than pooling, so their recovery FPS is not a fair cost comparison. +Replaying the pooled child's budget sequence on a warmed local producer without +a shaper, carrier, queue or handoff produced nine frames in the same eight-second +low-budget interval; adding the two original keyframe requests yielded eight, +matching the original interval. Startup protection had already ended. Native +logs show frame dropping and the scaler's spaced downscales. This establishes +stalls under that budget sequence, not the cause of the sequence: keyframe bursts +can still affect downstream estimation, and parameter reconfiguration remains +in the replay. These results do not support removing dependency recovery or +startup protection, growing queues or repeatedly resetting encoders. Matched +field evidence and broader transport/allocation effects remain open. +The rate/loss run with audio retained all 65 pulses after startup at both +played outputs, despite video stalls; this does not establish gap-free video. +A separate six-second complete RTP-media drop on one child, preserving RTCP +and signaling, resumed its played video 157 ms after release for VP8 and 128 ms +for H264. Both healthy siblings retained full dimensions; H264 still recorded +brief sibling stalls. All post-release audio pulses arrived. This bounded local +check does not establish recovery for an interrupted relay or reporter's network. + +A matched single-Viewer control separates pool handoffs from ordinary adaptation. +Pooled sending used one producer with no handoffs; ordinary sending used no pool +and made only two parameter writes. After the second one-second rate/loss pulse, +both reached 262p before ending their 40-second recovery window at 390p / 15 fps. +Actual source frames stayed 1904x1048 at about 30 fps. Native logs showed quality +scaler down-adaptation, Chromium's below-360p software fallback and subsequent +recovery to NVIDIA H264 at 390p; this was not a hardware failure. Pool reuse and +frequent application-level budget writes are therefore not necessary for this +bounded low-output state. Their effect on the intervening trajectory and the +unmatched field reports remains distinct. + The tab case exposed a more specific capture-feedback problem. A Chromium trace kept the source geometry at 1904x1049 while capture supplied a 268x148 content rectangle. Feedback rose to 66,600 pixels, but capture did not grow. diff --git a/docs/research/media-fidelity.md b/docs/research/media-fidelity.md new file mode 100644 index 0000000..eaa7ec2 --- /dev/null +++ b/docs/research/media-fidelity.md @@ -0,0 +1,162 @@ +# Media Fidelity + +Reviewed 2026-10-01. [Media quality](../standards/media-quality.md) owns behavior; +[TODO](../todo.md) owns remaining work. These synthetic checks do not establish +all-device acceptance or full HDR support. + +## Stereo + +Browser display capture requests two channels without speech processing. The +Host mixer uses a stereo Web Audio destination; Native capture and mixing use +48 kHz, two-channel PCM16. Native Opus uses the audio application mode. Browser, +Native and SFU receivers request stereo. The existing 64/128/192 kbps choices +are ceilings, not a lossless-audio promise. + +`opus/48000/2` alone does not prove stereo content: Opus permits mono packets +under that declaration. [RFC 7587](https://www.rfc-editor.org/rfc/rfc7587.html#section-7) +distinguishes receive preferences from packet content. Test distinct signals +after decoding instead of inferring fidelity from SDP or a two-channel mixer. + +The embedded SFU browser gate sends 500 Hz left and 1500 Hz right, and measures +decoded channel separation. On local Chrome 154, both the default fallback and +SFU-only arms retained more than 77 dB separation in both channels at the 64 kbps +ceiling, including normal video reception. Source changes and subscription +recovery retain the existing audio gate. Run `npm run gate:embedded-sfu` with `PIIK_EMBEDDED_SFU_GATE=true`, +`CHROME_PATH`, and optional `PIIK_EMBEDDED_SFU_ONLY=true`; it cleans up its own +processes and ports. This is synthetic local evidence, not an acoustic or +public-network measurement. + +A local Chrome 154 check through the production Host/Viewer peers retained more +than 57 dB separation after direct reception and after one decoding/re-encoding +relay, with a third tone mixed as Host microphone input. The receiver used its +normal video playback sink; a receiver-only Web Audio probe without playback +returned silence and was not a fidelity result. The Native Opus regression test +independently requires more than 20 dB decoded separation at all three bitrates. +These checks support keeping the existing stereo path. They do not cover every +physical source device, browser or acoustic setup, and do not establish surround +sound support. + +## SDR Conversion + +[#445](https://github.com/TNTcraftHIM/Piik/issues/445) reports washed-out Native +capture on an SDR Windows 10 system. Its proposed explanation is not established: +Microsoft documents full-range RGB as the +[video processor's default input range](https://learn.microsoft.com/en-us/windows/win32/api/d3d11/ns-d3d11-d3d11_video_processor_color_space). +The confirmed gap was that `FrameConverter` left its input/output range and +matrix implicit. The same converter handles BGRA capture and decoded NV12 +relay input; correcting only screen capture would leave the relay inconsistent. + +A local D3D11 check through the previous converter produced full-range NV12 from +BGRA color bars (black Y=0, white Y=255). The production `AdaptiveEncoder` VP8 +output then reproduced crushed shadows and clipped highlights in Chrome 154: + +| Source gray RGB | Previous conversion, WebRTC playback | Explicit limited-range BT.601 | +| --- | --- | --- | +| 0 | 0 | 0 | +| 32 | 19 | 33 | +| 128 | 130 | 128 | +| 235 | 255 | 235 | +| 255 | 255 | 255 | + +The fixture encoded 256x128 color bars with the production converter/encoder, +injected those keyframes into a local VP8 PeerConnection without the color-space +RTP extension, and sampled the playing video into a canvas. The experimental +conversion also restored red/green/blue bars to within one channel value of the +source. This establishes a bounded SDR defect, not the cause of every color +report. + +WebCodecs and WebRTC are not interchangeable acceptance paths: explicit BT.709 +looked correct through WebCodecs but shifted primary colors through this WebRTC +receiver. Chromium's +[remote-frame conversion](https://chromium.googlesource.com/chromium/src/+/master/third_party/blink/renderer/modules/peerconnection/media_stream_remote_video_source.cc) +can use BT.601 when color metadata is unspecified. White Y=235 is normal for +limited-range YUV; it must become display RGB=255, not be treated as a defect by +itself. + +The candidate now declares RGB capture as full range and reads decoded H.264 +color metadata from Media Foundation. Unspecified SDR and VP8 use the existing +WebRTC BT.601 convention. The converter produces limited-range BT.601 NV12; +the H.264 media types declare matching range, matrix, primaries and transfer. +Setting only matrix/range did not produce matching H.264 VUI on the tested +NVIDIA encoder. No wire field or user setting is added. + +Use the Windows 10 +[explicit video color-space API](https://learn.microsoft.com/en-us/windows/win32/api/d3d11_1/nf-d3d11_1-id3d11videocontext1-videoprocessorsetstreamcolorspace1). +The legacy nominal-range API squeezed decoded NV12 a second time in local tests. +Even with the explicit API and a positive format-conversion capability result, +NV12 BT.709-to-BT.601 conversion changed levels on NVIDIA when scaling and left +the matrix unchanged on AMD. Noncanonical NV12 therefore passes through RGB on +the GPU before the shared output conversion. Ordinary RGB capture and already +canonical NV12 keep one pass; no CPU readback, vendor branch or custom shader +is added to the product path. + +The opt-in `-CheckColor` probe in the [Windows capture guide](../../native/capture/windows/README.md) +checks full/limited-range BT.601/BT.709 input, scaling and input replacement +through the production converter. It passed on local RTX 4070 SUPER and AMD +Radeon adapters. RTX H.264/VP8 encoding followed by Chrome 154 WebRTC playback +also passed direct reception, decode/scale/re-encode and explicitly tagged +BT.709 H.264 input to both output codecs; sampled RGB errors were at most 5/255. +AMD hardware encoding was not accepted locally: MFT activation returned +`0x8007000e` before color configuration. These local checks do not settle the +reporter's Windows 10 environment, all hardware encoders, HDR content or missing +upstream color declarations. + +## HDR To SDR + +[#420](https://github.com/TNTcraftHIM/Piik/issues/420) reports overexposure with +both App and Browser capture on one Windows HDR machine. The report does not +locate the failing stage. + +The previous native WGC frame pool and thumbnails requested BGRA8, discarding +HDR range before the existing 8-bit H.264/VP8 encoder. Microsoft's +[capture guidance](https://learn.microsoft.com/en-us/windows/apps/develop/media-authoring-processing/screen-capture) +warns that HDR needs floating-point capture throughout the input pipeline to +avoid clipping. Reducing brightness after encoding cannot restore clipped +highlights. This is a confirmed implementation gap, not proof that every +reported Browser failure has the same cause. + +The candidate uses one `CaptureSdrConverter` before native output fanout and in +thumbnails. HDR displays use an scRGB FP16 pool, then Windows Direct2D's +[HDR tone-map effect](https://learn.microsoft.com/en-us/windows/win32/direct2d/hdr-tone-map-effect) +compresses luminance, the +[white-level effect](https://learn.microsoft.com/en-us/windows/win32/direct2d/white-level-adjustment-effect) +accounts for the source display's SDR white, and color management converts to +sRGB BGRA8. Ordinary SDR retains its texture-copy path. No custom shader, +CPU readback, new packaged dependency or per-output tone mapper is added. + +Source-monitor movement and DXGI factory invalidation refresh color metadata, +following [GetDesc1's contract](https://learn.microsoft.com/en-us/windows/win32/api/dxgi1_6/nf-dxgi1_6-idxgioutput6-getdesc1). +A format change recreates the WGC pool through its existing resize boundary; +it does not restart routes or encoders. WGC supplies no per-frame MaxCLL, so +conversion uses the display peak (at least the configured SDR white), or the +HDR10 10,000-nit range when usable peak metadata is absent. This stable platform +policy avoids a second image-content adaptation loop; real-display appearance +remains an acceptance requirement. + +The retained `-CheckColor` probe passed on RTX 4070 SUPER and AMD Radeon: +0–1000-nit neutral bars preserved distinguishable highlights, source white +80/240/80 nits and SDR/HDR size changes preserved ownership and ordinary SDR +colors. Production H.264/VP8 output decoded over Chrome 154 WebRTC reproduced the +converted SDR gray bars within 1/255. These are synthetic input checks, not +observations of an HDR game or the reporter's machine. + +Warmed GPU timestamp measurements for conversion, including the owned input +copy, had the following local medians. Dimensions are captured input sizes, +not newly supported sharing presets. + +| Adapter | 1920x1080 | 3840x2160 | +| --- | --- | --- | +| RTX 4070 SUPER | 0.045 ms | 0.240 ms | +| AMD Radeon | 2.39 ms | 11.17 ms | + +HDR conversion has a real GPU cost on weaker devices. These bounded measurements +exclude encoding and do not establish gaming/endurance performance. Current +monitors were SDR: physical HDR highlights, SDR content on HDR displays, +thumbnails, mixed displays and mode changes under load remain unaccepted. +Browser capture owns its own conversion; inspect the acquired frame before +assigning a downstream bug. + +Full HDR requires a separate end-to-end decision for bit depth, codec support, +color metadata, decoding and mixed Viewer capabilities. It is not a quality +dropdown addition. Preserve ordinary SDR routing and existing compatibility +with HDR-to-SDR; no full-HDR or surround implementation is claimed. diff --git a/docs/research/native-client-media.md b/docs/research/native-client-media.md index bb6a810..f5f9112 100644 --- a/docs/research/native-client-media.md +++ b/docs/research/native-client-media.md @@ -1,11 +1,47 @@ # Native App Media Evidence -- Reviewed: 2026-09-29 +- Reviewed: 2026-10-01 - Scope: platform capture, shared encoded sources, Pion transport and Browser decode; current behavior belongs to [media quality](../standards/media-quality.md) - Status: Windows physical native Host and Viewer gates passed; macOS and Linux adapters compile and package but still require physical media gates +## Windows Audio Exclusion + +A Windows 11 build 26200 check used two independent WinMM processes emitting +500 Hz and 1500 Hz tones, through the production capture helper and Go PCM +reader. Default-device loopback captured both at approximately 1799 amplitude; +process inclusion retained the first with under 0.3 amplitude of the other, +and exclusion retained the second with under 0.04 of the excluded tone. Target +exit retired capture in about 42 ms; attempting the stale identity failed. +No raw audio was saved. The extended capability probe completed in 300 ms. + +A parent/child follow-up used a silent selected parent with a 1900 Hz child, +plus an independent 2900 Hz process. Process-scoped reference amplitudes were +about 1799; exclusion retained the independent tone at 1800 and reduced the +selected child's tone to 9.3 (under 0.6%). Parent exit retired capture in 64 ms; +its stale identity was rejected. The reference uses each signal's own process +capture: default-device loopback also includes unrelated playback and endpoint +processing, so its amplitude is not a process-capture gain reference. + +The session fixture separately verifies failed exclusion replacement mutes +source audio while preserving video, microphone and encoded audio output; pause +and audio-profile changes retain the selected input. These checks establish a +local Windows implementation and one direct child, not coverage of every voice +application's process tree or simultaneous playback across several output devices. Those remain +device acceptance boundaries. [ADR-0008](../adr/0008-window-scoped-audio-capture.md) +owns the platform choice. + +The [Windows activation contract](https://learn.microsoft.com/en-us/windows/win32/api/audioclientactivationparams/ns-audioclientactivationparams-audioclient_process_loopback_params) +accepts one PID and its process tree, not a list of unrelated processes. +The [win-capture-audio implementation](https://github.com/bozbez/win-capture-audio/releases/tag/v2.2.0-beta) +supports broader exclusion by tracking audio sessions, capturing non-matching +processes and mixing them. Applying that design here would require session +discovery, process-tree deduplication and bounded multi-input mixing; Piik's +current source-plus-microphone mixer does not own that inventory. Multiple +exclusions therefore require a capture design change, not just a multi-select +control. Combining exclusion streams does not itself exclude their combined set. + ## Initial Physical Evidence (2026-09-05) These checks established the original native media paths. The Browser-mediated @@ -271,6 +307,33 @@ that result nor the allocation timings establish the reported system-wide lag's cause. Retain the existing implementation; reopen optimization from a matched bottleneck. +## Native H.264 Motion Quality + +Issue [#432](https://github.com/TNTcraftHIM/Piik/issues/432) reports more visible +blocking with native H.264 than VP8 on v1.6.7. Review of its attached Browser +report confirms 720p30 in both comparison windows, roughly 11 Mbps received, +and no reported packet loss; the 1080p60 H.264 window receives roughly 10 Mbps. +Those samples establish delivered format and transport observations, not equal +source content or objective picture quality. No encoded stream, reference +frames or QP measurements accompany that report. + +The reporter retracted the initial CBR bitrate-ceiling diagnosis after finding +that the experimental fixture changed bitrate during the measured interval. +That switching belongs to `PIIK_H264_FIXTURE` in `windows/main.cpp`, not live +capture. The corrected experiment reports CBR reaching its requested bitrate. +Production `h264_encoder.cpp` explicitly requests and validates Baseline; the +Browser's negotiated `profile-level-id` alone is not an inspection of emitted +SPS. The later experimental CABAC result cannot establish the production +Baseline bitstream's behavior. Microsoft's [encoder reference](https://learn.microsoft.com/en-us/windows/win32/medfound/h-264-video-encoder) +distinguishes profile, rate control and optional quality/speed settings. + +The original 720p comparison remains unresolved. An equal-content comparison +must preserve actual bitrate, frame rate, latency and production codec settings, +inspect the resulting bitstream and retain reference/decoded pictures. Raising +the bitrate ceiling, lowering the default frame rate or changing profile is not +justified by these logs alone; those changes have bandwidth, motion and receiver +compatibility consequences. + ## Implementation Boundary - `nativecapture` owns the child process, source identity, and bounded frame protocol. diff --git a/docs/research/room-interactions.md b/docs/research/room-interactions.md new file mode 100644 index 0000000..a114cd9 --- /dev/null +++ b/docs/research/room-interactions.md @@ -0,0 +1,260 @@ +# Room Interaction Design + +The owner accepted the text/reaction UI and model with bounded local acceptance. +[Status](../status.md#accepted-interaction-phase) owns release readiness and +[TODO](../todo.md#now) owns remaining work. Reviewed 2026-10-02. +This document records the implementation design and its tradeoffs; +[rooms and access](../standards/rooms-access.md#room-interactions) owns authority +and retention, while [presentation and lifecycle](../standards/presentation-lifecycle.md#product-surface) +owns room/publication lifetime. + +## Model And Owners + +| Concern | Owner | Lifetime | +| --- | --- | --- | +| Identity, room membership and access | Existing room store and authenticated signaling session | Current admitted participant | +| Screen/camera plus Host commentary | Existing capture and media owners | Current share and source | +| Text and reactions | `signal/interactions.go`, shared protocol and `RoomInteractionSession` | Current authenticated room session; bounded page-local presentation | +| Chat panel, optional chat overlay and reaction placement | Shared `RoomInteractions` and `RoomChatOverlay`; `RoomInteractionSession` owns bounded events | Current room UI | + +The existing room authority admits interactions through the same authenticated +WebSocket; there is no second admission or identity system. + +Messages do not need a media route, decoded frame, or SFU. They do not mutate the +media graph, add a port, create an extra connection, or restart capture. Pause, +source replacement and media recovery keep the current conversation. In this +candidate, a capable Host explicitly opts into a persistent room session. +Restoring an existing room connects that session without requesting capture or +starting media. The capture-ready owner requests publication; merely entering a +capture picker or awaiting permission cannot publish. Stopping the publication +retires only its media generation, while the authenticated Host and its +interaction session remain available. A later `start-sharing` claims a +fresh generation on that same signaling connection. Room close, access loss, +session replacement and page teardown still retire the whole session. A Host +without the capability keeps the existing stop-and-leave behavior. + +Room restoration and authority recovery share one awaited owner; connection reuse +checks the actual room/token binding. Publication acknowledgment precedes media +effects. A rejected start names its attempted generation, retires only that local +attempt and keeps room interactions available; it cannot stop another generation. +Opening the Host page alone still does not create an unused room. + +The server treats the current Host session's non-empty generation, matched to the +room's reconnect fence, as the only publishing truth. Room membership and Host +presence are separate, so a late Viewer in an idle room receives no media route +even when the Host remains available for room interactions. +Connected Viewers can continue interacting when the Host disconnects, while the +room server and their admitted sessions remain available. A departed participant +cannot receive a targeted reaction. Closing an App that also runs the room server +ends that service; merely leaving the Host page does not. + +## Delivery, Compatibility And Bounds + +- `/api/capabilities` advertises optional `roomInteractions` and + `hostRoomSession`; absence means the corresponding behavior is unavailable. + The page explicitly subscribes after authentication. Only subscribed current + sessions receive interaction events. The Host room-session behavior also + requires the `roomSession: true` authentication opt-in; older clients retain + the original stop-and-leave contract. Existing v23 media messages remain + valid, and publication start/result messages are sent only to an opted-in + session. +- The viewer discovers this optional capability independently of first-picture + startup. Late capability discovery enables data on the existing connection. +- A strict payload union admits plain chat text or a registered reaction. Chat is + at most 280 Unicode code points, normalized to NFC, without control/bidi + characters. It renders as text, with no markup, links, attachments or commands. +- The server accepts at most one interaction per 800 ms per connection. It + validates identity, subscription and target before fan-out. A slow recipient's + data is skipped at 16 KiB queued output, reserving the existing bounded queue + for signaling. The browser also declines sends under output backpressure. +- The sender waits for its server echo, not a guessed success or a read receipt. + A rejection or a 5-second confirmation timeout preserves the draft. There is + no offline outbox or automatic retry; manually retrying an unconfirmed send + may duplicate a message that arrived before the confirmation was lost. Only + the matching pending request may confirm a send or clear its draft/error; + late echoes may enter history without confirming a newer attempt. +- The page retains the latest 80 received chat messages and at most eight + simultaneous reaction effects. Effects expire after 2.4 seconds. Reconnection + clears pending effects and re-subscribes, without replaying messages or actions. + Same-room re-admission retains history, self attribution and the local draft, + even when its wire peer ID changes. Refresh, leaving, authority loss or room + session replacement clears the page conversation; late joiners receive only + new messages. Nothing persists in Browser storage. +- There is no server history, database table or message body in diagnostic + exports. Transport is the site's HTTPS/WSS connection; **chat is not end-to-end + encrypted**, and the server operator can access live message contents. + +## Interaction Design + +Chat uses the shared `FloatingPanel` shell for its nonmodal window. The shell +owns native visibility, placement, movement, resizing and focus return; the +conversation retains its session, draft, reading position and unread state. +It opens initially at the lower right, without +resizing the picture or moving the couch. Drag its title bar, use arrow keys on +the focused bar, or explicitly choose a corner under Position. Its summary also +allows dragging; a drag never activates the menu. Resize from either diagonal +corner, use arrow keys on a resize handle, or choose a preset under Position. +Resizing holds the opposite corner steady. Size and position belong to this room +UI only; viewport and keyboard changes bound them without erasing preferences. +The native manual +popover keeps the conversation visible while operating the picture or reactions. +Close, the entry toggle, or Escape hides it; this preserves the room conversation. Inner +menus consume Escape first, then the most recently opened utility window, then +theater mode. Input-method +composition is not a dismissal request. Opening focuses the panel without opening +the keyboard; explicit close returns focus to the entry. Nested disclosures +cannot change panel visibility. +The visual viewport bounds it above an onscreen keyboard. +Hidden messages are not marked read. A bounded scrolling log keeps arrivals from +growing the panel. Consecutive messages from the same sender share visual identity; +each message retains its accessible author. Hiding and reopening preserves the +reader's position; a latest-message action resumes following. Sending waits for +confirmation without locking draft editing or introducing a message queue. +The conversation entry sits below the couch, separate from +the Host's sharing controls; theater retains its compact conversation entry. +Selecting a person opens the compact reaction palette in the shared floating +shell, initially beside the selected person (below when space permits). Each +opening anchors there before allowing free movement. Its portrait and name identify the +target even after moving the window. Selecting a different participant keeps its +placement and any manually chosen size; selecting the same person toggles it. +The default palette height follows its content, bounded by the visible viewport. +One's own palette expresses a reaction above oneself and holds the local effects switch; other +people's palettes offer targeted expressions and two labelled throwing props. +Connection details remain secondary when that participant has inspectable data. +Close or Escape dismisses the palette; it also closes if its participant leaves. +Sending keeps it open for repeated interaction, with the existing short cooldown. +Confirmation never changes menu visibility; failure stays visible in the card. +Opening or closing a reaction card preserves the chat draft and unread state. +Self expressions follow the sender's head, including idle and interaction motion. +Only that sender's newest self expression replaces their earlier one. A targeted +emoji is a delivery: it travels from sender to recipient, retains a compact +source label above the centered glyph, and settles separately above the +recipient's own expression. Late mounts and reduced motion retain that attribution without +replaying elapsed motion. A gift never replaces the recipient's self expression; +all delivered events remain in the accessible log. +Tomato/poop throws require choosing a current participant, land with a small +character response and settle above the head, keeping the face visible. A tomato +squashes with a few droplets; poop wobbles with a small puff. Other expressions +use short gestures and sparse accents. All decorative descendants share the +event's remaining phase and expiry; event ID and sender identity seed their +variation. No particles, stains or animation timers outlive the event. +A departed participant is never silently replaced as target. +Everyone sees the same server-issued event; placement follows their own couch +layout. Effects have a couch-only coordinate owner, independent of panel layout, +and never become a video overlay or a media status. Impact transforms leave the +UUID-owned idle animation, names and status indicators alone. Hiding effects or +retiring the event cancels its own animation; showing a still-live event uses its +remaining lifetime. Users can hide effects; reduced motion uses a static stamp. +Theater mode hides the couch and retains a chat entry, using the same floating +panel and draft without changing the picture's size. HTML player fullscreen +keeps the existing danmaku control; it does not include the conversation panel. +Unread messages advance only while the log is onscreen, the document is visible +and the reader is following its latest messages. A hidden or scrolled-back log +does not consume arrivals. These are local unread indicators, not read receipts. + +The optional chat overlay (danmaku) projects newly received chat over the picture; +it adds no message kind, transport or delivery promise. It defaults off, with the +same local switch beside the chat composer and Viewer playback bar, including HTML +fullscreen. At most three single-line comments appear for up to six seconds each. +Font size follows the picture width, with a fixed readable contrast and duration +rather than separate typography, opacity and speed preferences. +Each event's sender and ID choose its lane, and a newer event replaces that lane's +previous flight. All received text remains in the full chat log, without a playback queue. Text is clipped +to the picture space above playback controls; smaller spaces show fewer complete +lanes without restarting their timelines. It does not intercept playback gestures. Showing the overlay +does not mark the chat log as read. Hidden documents, unavailable pictures, +source changes and disconnection clear flights; showing or reconnecting never +replays earlier comments. Reduced motion uses temporary stationary text. Native +video-only fullscreen and picture in picture cannot include HTML overlays. + +Server confirmation means acceptance, not an acknowledgment from every recipient. +Delivered chat follows server order; congestion can leave gaps without reordering +the received subset. The same event retains its lane and time basis even when +preceding events are missed. Visibility, local switches and missing messages may +change which effects are shown; geometry and font size follow the local viewport. + +Reaction animations are decorative. The same bounded events also provide +localized sender/action/target descriptions in a polite accessible log, including +when visual effects are hidden. Reduced motion removes decorative movement from +both the flight and the picker controls. + +Chinese, English and visual modes share the component and behavior. Visual mode +retains user-entered text and participant names, with icon controls and localized +accessible names. Host identity reuses the existing crown; neither chat colour nor +reaction motion claims media connectivity, capture permission or delivery status. + +## Presentation Synchronization + +The server stamps acceptance time once and broadcasts the same event in its +existing serialized order. A subscription response provides server time; the +room session estimates its offset from the local round-trip midpoint and keeps +that estimate until reconnection. Effect expiry and both renderers share that +session's clock. Late effects start at their remaining phase; expired chat still +enters the log and confirms a matching send, without replaying an effect. +Returning from reduced motion also respects the original expiry. + +This is approximate alignment, limited by subscription round-trip asymmetry, +queueing and clock drift; it does not promise frame-exact synchronization. There +is no continuous clock exchange, sorting by timestamp, or animation-frame stream. +Changing a device's clock mid-connection can disturb effect timing until the +next subscription; message order and delivery do not depend on that clock. +Participant motion and event lanes reuse the same identity hash, with an event's +ID and actor identity determining its lane independently of prior arrivals. +A shared mutable random-number stream would drift whenever a client misses or +hides an event. Responsive layout remains local. +This follows the same-input principle described in +[Deterministic Lockstep](https://gafferongames.com/post/deterministic_lockstep/), +without introducing lockstep waiting or frame-by-frame simulation. + +## History Direction (Not Yet Implemented) + +History ownership remains undecided. A Host cache can also miss live messages and +is unavailable while the Host is absent; treating its copy as server-confirmed +history would change message authority. Compare that cost with a bounded room-server +memory buffer before adding replay. Either design needs explicit access and +retention boundaries. Recovered text would enter the log, never replay old +danmaku or reaction effects. Current behavior remains the no-history contract above. + +## Audio Boundary + +The Host microphone remains commentary mixed into the shared source. +Bidirectional room voice is not part of the product plan. +Text, reactions and danmaku introduce no microphone capture, voice +admission or additional media peers. + +## Primary References And Rationale + +- [LiveKit media and data](https://docs.livekit.io/frontends/build/media-data/) + separates publications from finite data. Piik already has an authenticated + signaling channel; adding the LiveKit room service solely for chat is unnecessary. +- [LiveKit text streams](https://docs.livekit.io/transport/data/text-streams/) + distinguishes current-participant delivery from persistent history. The + candidate likewise promises transient room messages, with explicit local bounds. +- [WebSocket buffering](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/bufferedAmount) + and [WebSocket API limitations](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) + motivate bounded queues and explicit send failure rather than an accumulating outbox. +- [WebRTC data channels](https://www.w3.org/TR/webrtc/#rtcdatachannel) + depend on peer connections. Using them here would couple chat readiness and + fan-out to a screen-sharing route; the existing room WebSocket avoids that. +- [React state identity](https://react.dev/learn/preserving-and-resetting-state) + supports resetting the panel with its room session rather than a reconnecting + transport identity. [ARIA log semantics](https://www.w3.org/TR/wai-aria-1.2/#log) + provide sequential, non-interrupting announcements alongside visual effects. +- [Meet reactions](https://support.google.com/meet/answer/13151720?hl=en) + attach expressions to participants and let viewers reduce visual distraction. + Piik likewise keeps reactions outside the shared picture, with explicit local + display preference and a separate accessible event description. +- [Microsoft participant cards](https://support.microsoft.com/en-us/outlook/profile-cards-in-microsoft-365) + place person-specific actions on the selected person. Piik uses that direct + entry on its existing couch rather than requiring a second participant list. +- [Native popovers](https://developer.mozilla.org/en-US/docs/Web/API/Popover_API/Using) + provide top-layer placement, light dismissal and focus return without adding + a second overlay framework. Piik owns the bounded card's anchor and contents. +- [Animation time](https://developer.mozilla.org/en-US/docs/Web/API/Animation/currentTime) + lets a newly mounted or resumed effect seek to its remaining phase. The + [clock comparison](https://developer.mozilla.org/en-US/docs/Web/API/Performance/now#ticking_during_sleep) + explains why replacing wall time with a monotonic clock alone would not solve + cross-platform sleep and resume. +- [niconico playback controls](https://help.nicochannel.jp/hc/ja/articles/4414625162777-%E5%8B%95%E7%94%BB%E3%81%AE%E8%A6%96%E8%81%B4%E6%96%B9%E6%B3%95) + keep comment display optional alongside playback. Piik likewise separates + each viewer's overlay preference from message delivery and the conversation. diff --git a/docs/research/room-voice.md b/docs/research/room-voice.md deleted file mode 100644 index bfb5c4b..0000000 --- a/docs/research/room-voice.md +++ /dev/null @@ -1,287 +0,0 @@ -# Room Voice: Design Assessment - -Reviewed 2026-09-24 against public `v1.6.3` (`f423df0a`). This is a proposal, -not an accepted product contract or implementation authorization. The separate -room-interaction experiment supplies candidate lifecycle behavior; it has not -been integrated. [TODO](../todo.md#now) owns the outstanding work and decisions. - -## Recommendation - -Keep existing room authority and the Host-rooted screen route. Model room voice -as participant-owned microphone publications and independent listening. Chat, -screen playback and voice share admission, not a media lifetime. Do not turn the -screen tree into a conference graph or introduce another room/session store. - -There is no equally small solution that simultaneously promises all 21 members -can talk, preserves every P2P-only deployment, and supplies conference-grade -connectivity. Choose the first voice scope before implementing its transport. -For the next technical experiment, assess a small direct group and audio -recapture isolation first. Four total voice participants is a useful test size, -**not an accepted product limit or a demonstrated performance guarantee**. -If full-room voice is required initially, assess a mature conference SFU before -building a mesh intended to grow into one. - -## Current Owners And Actual Gaps - -| Current owner | Verified behavior | Consequence for voice | -| --- | --- | --- | -| Room store and authenticated signaling | Room admission, participant identity, grant revocation and session replacement | Extend this authority; no voice login or second membership store | -| `HostAudio` in `src/client/media/host-audio.ts` | Source and microphone mixed into one share-scoped track | This is commentary; it cannot supply independent conversation | -| `internal/app/nativehost/audio_mix.go` | Native source/mic PCM share one mixer and output clock | Keep source audio here; avoid duplicating the Host mic across outputs | -| `src/client/media/viewer-audio.ts` | Audio follows the picture's play/pause/mute state | Room speech needs its own playback owner | -| Screen controller and media edges | One Host source, one upstream per Viewer, bounded screen-copy capacity | Voice must not borrow screen generations, first-video-frame proof or copy slots | -| Embedded SFU admission/media | One committed room publication, video-required metadata, share-scoped resource identity | Multiple audio-only publishers need a new capability, not another flag | - -The SFU constraints are explicit in `validPublicationMedia` in -`internal/server/sfu/media.go`, `ResourceFence` and `CommitPublication` in -`internal/server/sfu/admission.go`, and the Host/Viewer dispatch in -`internal/server/signal/router_sfu_signal.go`. -[ADR-0013](../adr/0013-embedded-node-local-media.md) embeds Pion/LiveKit components; -it does not embed LiveKit's complete room service. - -The interaction candidate separates a retained Host room session from screen -publication. That is a useful foundation, not permission to allocate rooms on -page visit. Preserve the accepted entry: first share creates the room and an -existing room can resume. Retaining opted-in interaction after stopping media is -the candidate lifecycle, not published behavior. A separate voice-only -room-creation flow would be another product decision. - -## Transport Choice - -| Approach | Benefit | Cost and boundary | -| --- | --- | --- | -| Small direct voice mesh | Keeps media P2P and does not require a reachable central media server | Each sender sends to every listener; needs a total voice-member bound, separate resource accounting and real NAT/device tests | -| Mature multi-publisher SFU | Each speaker uploads once; suitable direction for full-room voice | New publication/subscription integration and central-media availability/privacy decision | -| Host forwarding or mixing hub | Fewer peer relationships | Reintroduces Host dependency, forwarding/mixing and echo ownership; unsuitable for Host-independent conversation | -| Custom voice relay tree or automatic mesh/SFU switching | Potentially combines deployment options | Adds another routing/recovery problem; outside the first scope | - -For a full bidirectional mesh, `N` participants need `N*(N-1)/2` pair connections -and up to `N-1` microphone copies per sender: 21 people means 210 pairs and 20 -copies; four means six pairs and three copies. These are topology counts, not -bandwidth or CPU measurements. Limiting speakers alone does not bound sender -upload: one speaker and 20 listeners still need 20 copies. The screen's existing -`C=1..3` limit must retain its meaning; any independent voice budget needs explicit -acceptance and combined-load measurement. - -Two SFU implementations deserve a bounded comparison: a complete mature room -engine, or a narrow participant-audio adapter around existing media components. -The former adds deployment and potentially competing session ownership; the -latter leaves Piik responsible for admission, subscriptions, recovery and cleanup. -Do not describe the adapter as mature conference behavior merely because its -forwarder uses mature libraries. LiveKit's -[room/participant/track model](https://docs.livekit.io/intro/basics/rooms-participants-tracks/) -and [SFU architecture](https://docs.livekit.io/reference/internals/livekit-sfu/) -show the layers involved. - -Select one topology for the first capability. Jitsi's -[two-party P2P design](https://jitsi.org/blog/p2p4121/) retains its bridge connection -and changes transport when group size changes: useful evidence that a hybrid -mode has real lifecycle cost. It is not a reason to copy that complexity here. - -Deployment is part of this decision. Demo and App Local/public-link modes are -P2P-only today. Public-link tunneling carries HTTP/WebSocket control traffic, -not a public SFU media endpoint. Existing STUN is not TURN, and existing embedded -UDP media does not provide media TCP/TLS. A full conference engine's -[connectivity options](https://docs.livekit.io/transport/self-hosting/ports-firewall/) -are not inherited by importing its library. This proposal enables no new service -or port and does not change Demo configuration. - -## Proposed Responsibility Boundaries - -```text -Existing room authority and authenticated signaling - +-- screen publication -> existing P2P/SFU screen route -> picture/source sound - +-- interaction events -> bounded chat/reactions - +-- voice admission -> participant microphone -> selected voice transport - -> independent voice playback -``` - -These are responsibility boundaries, not a request for a generic media framework -or one new service per line. Keep the first implementation cohesive: - -| Responsibility | Owns | Does not own | -| --- | --- | --- | -| Room authority | Who belongs, who may publish/listen, bounds and retirement | OS microphone permission or local volume | -| Participant voice session | Explicit join intent, current admitted publication/subscriptions and their transport lifetime | Screen routes, share generation or a second reconnect policy | -| Microphone input | One acquisition, device choice, local mute, replacement and disposal | Remote permission to turn capture on | -| Voice playback | Received tracks, local volume/person mute, output capability and autoplay recovery | The video's pause/mute state or remote speaking permission | - -Do not attach room speech to a screen peer merely to reduce connection count: -screen parent replacement and share retirement have different lifetimes. The -selected voice transport owns its connections and counts them explicitly. - -Prefer a Browser-owned room microphone for both roles, including a Host using -App screen capture. This reuses platform speech processing and avoids a second -native voice service. Reuse existing device-selection semantics, not native -device identifiers. Browser microphone acquisition requires a secure context; -plain HTTP LAN pages and unsupported browsers must have truthful capability -presentation. App detection alone does not establish microphone availability. -See the [capture specification](https://www.w3.org/TR/mediacapture-streams/). -Browser voice also does not inherit the native Pion path's port mapping, NAT -assistance or native media fallback just because an App is attached. Measure -voice reachability independently; a working screen path does not prove it. - -Room voice and source sound need different audio treatment. Voice uses the -platform's speech/AEC facilities; music/game sound keeps its fidelity settings. -Do not force both through the existing source-audio Opus profile or write a new -voice codec, AEC, PCM subtraction routine or virtual-audio driver. - -## Consent, Lifecycle And Presentation - -Proposed first permission model: admitted members may explicitly join voice and -then unmute themselves; existing room removal remains the authority boundary. -Start with a closed microphone. Joining may unlock playback, but only explicit -unmute requests input permission. If Host-only speaking approval is required, -specify it before adding a moderation UI. No remote action may start a microphone; -LiveKit also disables [remote unmute by default](https://docs.livekit.io/intro/basics/rooms-participants-tracks/participants/). - -| Event | Proposed behavior | -| --- | --- | -| Share start/stop/pause, source or quality change | Voice stays independent | -| Local mic mute | Silence that publication without leaving voice; retain explicit device intent | -| Local person mute or voice volume | Change local listening only; picture volume remains separate | -| Leave voice | Release mic and voice media, remain in room/chat | -| Host leaves page while room service remains | Retire Host voice; other admitted participants may continue | -| App exits in Local/public-link mode | Room service disappears; independent voice cannot outlive its authority | -| Site-mode App media fails | Browser voice may continue while the Site room authority remains | -| Revocation, replacement or room close | Retire affected resources and reject stale completions/signaling | -| Mic permission/device failure | Preserve picture and other voices; show the actual input failure | -| Hidden page or phone interruption | Visibility alone is not leave/mute; report actual suspension/device loss | - -Transient signaling loss is not automatically a media failure or a grant -revocation. Current `presence.go` and `route.Controller.DisconnectSession` -distinguish disconnected sessions from confirmed departure; Viewers have bounded -membership grace. Voice must consume that authority owner's transitions, with no -parallel grace timer or indefinite stale-session publication. Before implementation, -define and test the Host case, reservation retention, reconnect re-admission and -revocation during an outage. Do not extrapolate Viewer grace to a Host or reuse -an old session ID as current authority. This is a design gate, not a claim that -screen recovery already implements voice recovery. - -One input owner prevents duplicate captures. Retire late permission results after -leave; prepare a replacement device before releasing a healthy old input; keep a -failed explicit selection visible rather than silently choosing another mic. -Repeated mute/join/leave must converge without a second saved effective-state flag. -Ordinary publication mute need not renegotiate every time; mature systems -[distinguish mute, unpublish and unsubscribe](https://docs.livekit.io/intro/basics/rooms-participants-tracks/tracks/). - -Use the existing participants/sofa and room actions, not a conference grid. -First controls: join/leave voice, mic toggle and input choice/check, independent -voice volume, and local person mute in the participant menu. Keep source-sound -settings with the picture. Default to system output; show output selection only -where supported. Do not add deafen, global push-to-talk or a permission-role matrix -until their need and restoration behavior are established. - -Display local intent, authorized membership, transport readiness and actual -playback separately. A speaking pulse follows observed audio activity, not a -connected flag or UUID animation seed. Catch blocked playback, including tracks -arriving after joining, and offer an explicit enable-audio action; compare -[LiveKit's autoplay handling](https://docs.livekit.io/reference/components/react/component/startaudio/). -Mobile lock-screen calling remains unproven; platform suspension is not fixed by -extra timers. See [browser lifecycle guidance](https://developer.chrome.com/docs/web-platform/page-lifecycle-api). - -## Audio Isolation Is A Prerequisite - -Whole-system loopback records received room speech played on the captured output -endpoint. That speech can then be sent back through screen audio. Headphones -remove an acoustic path, not this digital recapture; microphone AEC does not clean -the separate source-audio track. This matters even if only the Host shares a screen. - -Windows exposes narrower process-tree capture and process exclusion. Current -Piik includes the selected process tree for window sound or captures the output -endpoint for system sound; it does not implement process exclusion. Excluding -the browser that plays Piik voice could also exclude intended browser content. -Test actual process ownership and output routing rather than assuming a window -selection isolates sound. See Microsoft's -[loopback description](https://learn.microsoft.com/en-us/windows/win32/coreaudio/loopback-recording) -and [process-capture sample](https://learn.microsoft.com/en-us/samples/microsoft/windows-classic-samples/applicationloopbackaudio-sample/). - -Browser `restrictOwnAudio` is relevant; `suppressLocalAudioPlayback` has a different -purpose. Chrome documents the former as excluding audio originating from the -capturing document when system audio is captured. Check support, applied settings -and decoded output on supported platforms; a hint is not cross-browser proof. -See the [Screen Capture specification](https://www.w3.org/TR/screen-capture/) -and [Chrome's implementation announcement](https://developer.chrome.com/blog/chrome-141-beta). - -Verify source sound and incoming voices using distinguishable signals. Cover -native window/system capture, browser tab/window/system capture, source changes, -same-browser content and output-device changes. Where isolation cannot be -established, a supported combination or explicit feature limitation must be -chosen before release. Never silently discard source sound or promise that a -headphone hint solves it. - -## Compatibility, Trust And Bounds - -The Host microphone must have one distribution path. Sending it in both existing -commentary and independent voice causes duplicates and defeats individual mute. -Moving it exclusively to voice can make older/non-participating Viewers lose -commentary. Choose an explicit room capability/participation boundary or a -coordinated major-version transition; do not ship a silent behavior change. -The [versioning owner](../standards/versioning.md) governs that choice. This -assessment does not select a release number or authorize removal of native mic. -Once that boundary is selected, define the handoff before coding: retire the old -microphone distribution before enabling the new one, and test failure/cancellation -without silently losing commentary for Viewers who cannot receive room voice. - -Browser-first capture can avoid new native voice commands, but does not make -room signaling automatically compatible. Trace strict readers in both directions; -advertise support before new messages. Keep stable peer identity, authenticated -session, grant and media connection identity distinct. Reuse an existing identity -only when its lifetime matches; do not add a room-wide epoch for convenience. - -Bind signaling to current authenticated participants and exact publication/peer -resources; bound memberships, publications, subscriptions, candidates and message -rates before allocation. Leave/revoke must release actual resources before their -slots become reusable. No audio through chat events or durable voice recordings. - -P2P revocation has an enforcement limit: the server can refuse signaling and -conforming endpoints can stop capture/drop tracks, but it cannot intercept media -between two modified peers. An SFU can enforce its own forwarding removal. Do not -promise identical server-enforced muting across these models, or hide the central -server's media access under the existing P2P privacy label. - -## Implementation Sequence And Acceptance - -1. Reconcile the isolated interaction candidate with current main and accept its - room/picture separation on its own. Preserve the existing staged work; do not - merge an old branch wholesale. Voice need not block chat/UI acceptance. -2. Select total voice membership, deployment coverage, speaking policy and the - commentary/compatibility boundary. Record accepted changes in the existing - product owners and an ADR only when architectural decisions are accepted. -3. Run an isolated transport/audio-isolation experiment before shipping UI. - Measure screen plus voice CPU, upload, number of connections, audible latency, - NAT failure, echo and cleanup. Use one topology; stop if the required scope - implies a custom conference engine or unsupported audio isolation. -4. Implement one vertical slice through the owners above: explicit join, muted - input, unmute, remote audio, local mute and leave. Then expand participants and - exact authority teardown. Reuse existing controls, status/comic meanings and - diagnostic export; avoid separate feature-local rules. -5. Accept faults as well as the happy path: permission resolves after leave, - device switch fails, source changes, autoplay is blocked, signaling drops - during healthy audio, reconnect races revoke/replacement, Host leaves, service - exits, mobile suspends, and repeated join/leave leaks no resources. Test quiet - and muted participants without video-frame admission requirements. Include - supported old/new endpoints and Chinese/English/visual presentation. - -Transport experiments must report measured limits rather than treating a -successful localhost call as public-network or mobile acceptance. This planning -round ran no voice, acoustic-echo or device-performance experiment. - -## Independent Fidelity Work - -Stereo acceptance can proceed separately: distinguishable left/right signals -through Browser, Native, relay and SFU, with microphone mixing and source changes. -The released native answer fix negotiates receive-side stereo; SDP alone does not -prove decoded channel separation. Opus negotiation follows -[RFC 7587 section 7](https://www.rfc-editor.org/rfc/rfc7587.html#section-7). -Do not add surround channels as a prerequisite for voice. - -For [HDR issue #420](https://github.com/TNTcraftHIM/Piik/issues/420), investigate -correct SDR output first: preserve HDR range during native capture, then use -platform tone/gamut mapping before the existing SDR encoder. Windows documents -[floating-point capture](https://learn.microsoft.com/en-us/windows/apps/develop/media-authoring-processing/screen-capture) -and a [tone-map effect](https://learn.microsoft.com/en-us/windows/win32/direct2d/hdr-tone-map-effect). -Validate mixed SDR/HDR content, highlights, SDR reference output and display-mode -changes. Already-clipped browser input cannot be restored downstream. Full HDR -transport/metadata, surround sound, recording and transcription remain outside -this proposal. diff --git a/docs/standards/configuration.md b/docs/standards/configuration.md index fac3d95..4288653 100644 --- a/docs/standards/configuration.md +++ b/docs/standards/configuration.md @@ -56,6 +56,7 @@ Automatic SFU fallback runs inside the Hosted process when `SFU_UDP_PORT` is set | `SFU_UDP_PORT` | Optional UDP media port `1..65535`; unset or blank disables SFU and fixes Privacy mode on for every room. Set `7882` for the standard public listener and allow the Host to choose Privacy mode. | | `SFU_LISTEN_HOST` | IPv4 bind address, default `0.0.0.0`; independent of HTTP `LISTEN_HOST`. Read only when SFU is enabled. | | `SFU_PUBLIC_IP` | Optional explicit IPv4 advertised-address override for a host behind NAT. Read only when SFU is enabled. | +| `SFU_ONLY` | `false` by default. `true` requires `SFU_UDP_PORT` and sends every room through SFU, without Peer candidates or background P2P convergence. Applies after Server restart. | SFU control uses the application's authenticated signaling connection. No separate control origin or infrastructure credentials are configured. Local and @@ -67,6 +68,13 @@ configured STUN and peer relays remain; no media-server fallback is possible. The server enforces this even if a Host requests hybrid mode. There is no separate `SFU_ENABLED` flag to conflict with the listener configuration. +With `SFU_ONLY=true`, the page shows a locked **Server media** policy in place of +P2P settings. An older page requesting Privacy mode is refused before sharing; +refresh it or turn Privacy mode off. Capacity exhaustion and connection failure +use the existing bounded SFU admission/recovery path, never a P2P escape. +This policy needs sufficient server bandwidth and a working UDP route; it is not +a TURN/TCP fallback. App Local/public-link and the public Demo remain P2P-only. + The SQLite parent directory must exist and be writable. The systemd template sets `/var/lib/piik/rooms.sqlite` under its managed state directory; the container image sets `/home/nonroot/rooms.sqlite` under its writable data diff --git a/docs/standards/engineering.md b/docs/standards/engineering.md index 109232d..dcbf258 100644 --- a/docs/standards/engineering.md +++ b/docs/standards/engineering.md @@ -148,7 +148,10 @@ above rather than searching the tree. Closing a Browser connection may leave its API promises pending. Its owner cancels the corresponding waits so a serialized successor never depends on completion from a retired connection; late results still need identity checks. - Later cancellation must not erase a failure already observed before cleanup. + Failure effects need the same final identity check as successful results, + serialized with replacement; an earlier input check alone is insufficient. + Later cancellation must not erase a failure already accepted by its current + owner. Perform blocking retirement after releasing that owner's state lock. A caller deadline does not cancel a dependency that ignores it. Keep late work bounded and serialize its cleanup; verify the dependency's actual lifetime. Construction is part of acquisition: if it throws, release resources created diff --git a/docs/standards/media-quality.md b/docs/standards/media-quality.md index 99db584..cace60c 100644 --- a/docs/standards/media-quality.md +++ b/docs/standards/media-quality.md @@ -39,6 +39,17 @@ production behavior and remaining acceptance. frame matches its desktop source dimensions. Other frames retain their own aspect ratio. This changes only the encoded presentation, never the game or display settings; vendor-private scaling is not inferred. +- Windows SDR conversion owns one output color contract: limited-range BT.601 + NV12 with matching H.264 metadata. RGB capture and decoded video retain their + input range/matrix through conversion and relay scaling; dimensions do not + determine color space. Unspecified SDR follows the WebRTC convention. +- Windows native HDR sources retain scRGB FP16 until platform tone mapping, + source-display white-level adjustment and sRGB conversion produce SDR. Live + capture and source thumbnails share this owner; conversion precedes output + fanout and does not add a media route or per-encoder tone curve. Ordinary SDR + keeps its copy path. Source-display color changes refresh capture input, + not room authority or transport. [Fidelity evidence](../research/media-fidelity.md) + owns platform checks; this is HDR-to-SDR, not end-to-end HDR delivery. - Where Windows supports border control, the native source picker offers **Show capture border**, off by default. The choice stays in the Host page and follows native source and quality changes. Source previews request borderless @@ -177,12 +188,32 @@ closing a Browser PeerConnection alone does not guarantee that its promises settle. Cancellation starts before joining work that depends on it, while a replacement still waits for the retired owner's cleanup. +Browser and Native encoder-group admission uses current native demand, not the +previously applied rate. A new or pending join must not displace an existing +rate owner by raising or retaining its budget against that owner's demand. +Revalidate pending admission before planning other memberships. + Eligible Browser P2P parents use the source-owned pool in [ADR-0014](../adr/0014-browser-node-local-encoding-pool.md). Compatible direct children share one independent local WebRTC producer; incompatible demands -remain separate. Each outgoing connection keeps native transport, allocation -and recovery, with a tiny carrier supplying its RTP clock. Producers use -the existing native video target under Host ceilings. Actual forwarded-frame +remain separate. Joining another producer must fit the child's native allocation; +a quiet scene's low byte rate does not prove that producer's rate budget fits. +Pending separation remains necessary while another compatible member needs a +higher native budget; quiet output alone cannot cancel it. When that demand +leaves or the child's budget recovers, the current producer can adapt in place. +Optional reuse also preserves the healthy current output's rate budget and +observed quality within Host ceilings through recovery-frame commitment, using +the actual candidate frame and current output rather than preparation alone. +Rejected membership retains current delivery and its recovery requests. +A recovery frame already being written retains its owner until completion, +including across pause/resume; cancellation belongs to the output that knows +whether writing has started. +Each outgoing connection keeps native transport, allocation and recovery, with +a tiny carrier supplying its RTP clock. Producers use the existing native video +target under Host ceilings. A new producer seeds its +local WebRTC start rate from that allocation, without a minimum or a change to +the outgoing connection's estimate. Subsequent budgets use the existing sender +parameters; initialization does not add a recovery controller. Actual forwarded-frame and producer observations remain distinct from the tiny carrier's statistics. Producer startup protection begins at actual outgoing publication, excluding local warmup and paused frames. The synthetic carrier uses screen-content @@ -294,6 +325,17 @@ A Native audio-process failure does not end healthy video. A later explicit source replacement resumes audio through the existing track and encoder owner; it does not trigger an automatic capture retry or change room audio topology. +Optional Windows screen-audio exclusion binds one selected process tree to its +PID and creation time under [ADR-0008](../adr/0008-window-scoped-audio-capture.md). +The selection lasts for the share, survives source-audio off/on and refresh, +and is never saved across App runs or rebound by executable name. Changing to a +window source or explicitly selecting no exclusion clears it. A requested +exclusion retires previous source audio before replacement preparation; failure +or target exit leaves source audio silent while healthy video and the Host +microphone continue. Only an explicit source selection resumes it. A missing +capability must reject the request, never discard the exclusion. Old pages keep +their existing audio behavior. + ## Observable Truth Requested resolution, frame rate, bitrate, content intent, codec, and audio diff --git a/docs/standards/naming.md b/docs/standards/naming.md index 8e31015..0a7f6d3 100644 --- a/docs/standards/naming.md +++ b/docs/standards/naming.md @@ -40,6 +40,7 @@ Set the register by the reader's task: | Surface | Register | | --- | --- | | Website | Light, everyday language in introductions and illustration captions; keep the current restraint | +| App and Web controls | Direct feature and action names, including panel titles and settings; “聊天” / “Chat” rather than an invitation or joke | | README | A friendly introduction, followed by direct explanations of features, limits and first steps | | Documentation index | Clear topic names and short descriptions that help readers choose a guide | | Deployment, configuration and maintenance guides | Formal, precise instructions: prerequisites, actions, expected results and recovery steps | @@ -102,6 +103,8 @@ in the welcome or illustration caption. Do not soften a failure into a joke, invent connection progress or promise guaranteed connectivity or absolute privacy. Screen sharing includes games, creative work and showing a useful discovery. +Playfulness belongs in optional welcome/waiting lines, empty-state captions and +illustration reactions; it does not rename the control that performs the action. Operational pages omit introductory sentences that repeat their controls. Helper text should explain a meaningful choice or prevent an error. diff --git a/docs/standards/presentation-lifecycle.md b/docs/standards/presentation-lifecycle.md index 4fe86bc..5191363 100644 --- a/docs/standards/presentation-lifecycle.md +++ b/docs/standards/presentation-lifecycle.md @@ -24,6 +24,15 @@ room identity, current state, roster, and one manual media-reconnect action. Controls and text must remain usable without overlap or horizontal overflow on desktop and mobile viewports. +A capable, opted-in authenticated room session +outlives its current publication. Stopping capture retires media, not chat; +restoring an existing room joins it without starting capture. Entering the page +alone does not create a room. Room close, lost authority and page teardown still +retire the session. Connected Viewers may keep interacting while the Host page +is absent, provided the room service and their admitted sessions remain available. +[Rooms and access](./rooms-access.md#room-interactions) owns interaction authority +and retention; [status](../status.md#accepted-interaction-phase) owns release readiness. + Piik App opens this same application in the system Browser. Its small startup surface selects Local, temporary public invitation, or a configured Site before entering the Host workspace. Viewer links opened at that activated diff --git a/docs/standards/rooms-access.md b/docs/standards/rooms-access.md index c908780..26fb6b6 100644 --- a/docs/standards/rooms-access.md +++ b/docs/standards/rooms-access.md @@ -1,6 +1,6 @@ # Rooms And Access -This file owns the current room, admission, invitation, and persistence +This file owns the current room, admission, invitation, interaction, and persistence contract. [ADR-0002](../adr/0002-memory-resident-protected-rooms.md) explains the storage decision; implementation detail belongs in code and tests. @@ -91,6 +91,26 @@ recreate a revoked subscription; no separate media token survives revocation. Changing code-entry policy or password affects later code-only attempts and does not silently revoke invitations. +## Room Interactions + +Text and reactions reuse the authenticated room signaling connection. The server +derives sender identity from that session and accepts targeted reactions only +for participants still connected to the same room. Interaction subscriptions +inherit existing membership, replacement and revocation checks; they grant no +additional authority and do not modify media routes. + +Delivery is transient and bounded. The server keeps no message history; each +page keeps only a bounded set of received messages and effects, without writing +them to storage or diagnostics. Re-admission to the same room may retain that +page's conversation, but refresh, leaving or authority loss clears it. There is +no history replay or offline outbox. HTTPS/WSS protects transport; chat is not +end-to-end encrypted, and the site operator can access live message contents. + +[Presentation and lifecycle](./presentation-lifecycle.md#product-surface) owns +the distinction between a room session and its current media publication. +[Interaction research](../research/room-interactions.md) records delivery bounds, +compatibility and design rationale. + ## Persistence Modes - **Stable mode (Hosted default):** one exact-schema SQLite owner persists room diff --git a/docs/standards/routing-transport.md b/docs/standards/routing-transport.md index c5fb992..19bfe17 100644 --- a/docs/standards/routing-transport.md +++ b/docs/standards/routing-transport.md @@ -7,7 +7,7 @@ owns ports and [self-hosting](../operations/self-hosting.md) owns service setup. ## Topology -- Media is automatic and P2P-first. Users do not select parents, route types, or +- Media is automatic and defaults to P2P-first. Users do not select parents, route types, or network transports. - The Host is the only source. Every Viewer has at most one active upstream, and the committed media graph is acyclic and source-reachable. @@ -21,9 +21,17 @@ same bounded P2P graph without its SFU fallback and fixes Privacy mode on for al rooms, including App Local and public-link rooms. The server enforces that policy for waiting and active participants; there is no separate Host-star mode. +A Hosted operator may require SFU-only media through `SFU_ONLY=true`. The same +controller then admits only SFU candidates: no Peer parents, Viewer relays or +background P2P convergence. The Host still publishes once, and every Viewer +uses the existing SFU subscription ledger and first-frame commit. Capacity or +connection failure never permits a Peer route. An explicitly peer-only share +is rejected before acquiring media authority; it is never silently forwarded. +The page advertises this site policy and hides inapplicable P2P preferences. + ## Per-Share Route Policy -The Host chooses route policy before sharing and it remains fixed for that +On a P2P-first site, the Host chooses route policy before sharing and it remains fixed for that share generation: - when SFU is available, default hybrid mode keeps P2P first and permits the diff --git a/docs/standards/visual-language.md b/docs/standards/visual-language.md index 39d5f74..857a117 100644 --- a/docs/standards/visual-language.md +++ b/docs/standards/visual-language.md @@ -56,6 +56,15 @@ do not stretch short values to fill a phone row. Check actual control bounds around breakpoints, in both orientations and all presentation modes; page-level overflow clipping is not evidence that the content fits. +Persistent room utilities use the shared floating panel without reflowing the +living room. The shell owns visibility, focus return, movement, resizing and +viewport/keyboard bounds; contents retain their domain state. Support title-bar +dragging and keyboard placement. An interaction palette starts beside its chosen +participant, stays open after sending and keeps its position when retargeted. +One-shot settings and guidance retain their existing disclosures. Escape closes +inner guidance first, then the most recently opened floating panel, then theater +mode; an IME composition must not dismiss these surfaces. + Keep the application language control's segmented Chinese, English and visual shortcuts. When additional catalogs exist, insert one language menu after the visual shortcut; all extra languages share that slot. Show the @@ -172,6 +181,8 @@ Participant characters should leave room for that distinctive presence. | Browser application | Browser chrome around content; not a second design of television | | A captured window or display | Window title bar or display stand, with the shared media metaphor kept distinct | | A camera source or Host commentary | `camera` identifies the alternative picture source; `microphone` identifies the enabled Host voice input and `microphoneOff` its muted state. Neither replaces the speaker metaphor for source/playback sound. | +| Room text and reactions | `chat` opens the conversation, `danmaku` controls its optional picture overlay, `smile` identifies reactions and `send` submits the chosen content. These do not represent media transport or voice. | +| Floating utility placement | `grip` marks a draggable title bar; `move` opens placement controls. Closing a panel hides its presentation, not its room session. | | Server forwarding media | A server on the media path; opening a Site alone does not imply SFU | These are **identity/object colours**, not status. A green pawn remains green diff --git a/docs/status.md b/docs/status.md index 63f8ebf..1a086f7 100644 --- a/docs/status.md +++ b/docs/status.md @@ -1,6 +1,6 @@ # Current Status -Last updated: 2026-09-29 +Last updated: 2026-10-02 This is the compact execution/deployment index. Product modules own behavior, [verification status](./verification-status.md) owns unresolved physical limits, @@ -35,6 +35,24 @@ Website delivery includes a bilingual reader documentation center generated from the repository's guides alongside the homepage, with search and detailed troubleshooting separate from the homepage FAQ. +## Accepted Interaction Phase + +The accepted [room-interaction design](./research/room-interactions.md) includes +the verified media repairs. It adds authenticated text and reactions +over existing signaling, with optional local chat overlay and a shared floating +panel. Its opted-in room session can outlive a publication; capture, quality and +route owners remain unchanged. Optional Windows screen-audio exclusion uses the +existing source/mixer boundary. Hosted Server +also accepts an optional SFU-only policy through the same route controller. +Windows native capture and thumbnails now share automatic HDR-to-SDR conversion; +synthetic GPU and Browser reception checks pass, with physical HDR display +acceptance still pending in [verification status](./verification-status.md). +Room voice is outside the product plan; Host microphone commentary remains. +4K remains deferred. The owner accepted the UI/model and bounded local acceptance +and authorized v1.7.0 publication. Physical device limits remain open. +Release artifacts and operator deployment records own delivery completion; +[TODO](./todo.md#now) owns remaining work. + ## Deployment Source, package identity, GitHub repository and local worktree paths use Piik. diff --git a/docs/todo.md b/docs/todo.md index c5dbee7..18c47af 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -1,70 +1,58 @@ # Current TODO Ledger -Last reviewed: 2026-09-30 +Last reviewed: 2026-10-02 Only **Now** is executable. Product modules own behavior; Git/PRs own completed history. A parked idea is not implementation authority. ## Now -- [ ] **Published-product bug and experience acceptance.** Prioritize share - startup, audio/source changes, sustained viewing, recovery and retirement, - admission, and narrow/touch UI. Verify confirmed fixes through the existing - owners before resuming feature polish. Keep field reports without matched - evidence separate from locally reproduced defects. -- [ ] **Sustained Browser degradation.** The H264 field cause remains open. +- [ ] **Authorized release delivery.** The owner accepted the room-interaction + phase, complete bilingual release notes and bounded acceptance, and authorized + v1.7.0 publication. Follow + [the integration workflow](../CONTRIBUTING.md#full-integration-and-release-workflow) + for one squash merge, packaging, publication and scoped postflight. Release + artifacts and operator records own completion; do not add a release-record + commit. No implementation work remains open for this phase. + +## Held By Owner + +- [ ] **Browser-internal encoder work.** Paused by the owner on 2026-10-01. [Comparisons and Chromium traces](./research/browser-local-encoding-pool.md#sustained-h264-recovery) - distinguish encoder recovery hysteresis from a tab-capture feedback/size-step - lock reproduced in bare WebRTC. Verify the upstream soft-target/hard-maximum - repair candidate against a patched Browser, which remains unbuilt. The Piik - sender boundary now isolates display frames while preserving native idle refresh, - capture intent and operation retirement. Real hidden-tab SFU checks pass; - broader device/background coverage remains. Native logs confirm content-driven - QP hysteresis and a cost from bitrate reconfiguration, but not repeated brief - blur. Match field Host/Viewer evidence - before assigning the field cause or declaring repeated brief blur fixed. - Retain existing clone/startup protections and candidate proof; no periodic - reset or quality floor. A recent Piik regression is not established. + own the evidence for synchronous H264 initialization, native adaptation and + the unbuilt upstream repair candidate. Keep the verified Piik budget and + handoff repairs; do not add periodic resets, a quality floor or a Browser fork. + The field room's cause and all brief blur are not established as resolved. ## Deferred Feature Work -Feature design and interaction polish are paused while the bug/experience phase -is active. Retain these decisions for resumption, not as parallel work. +These proposals remain deferred beyond the accepted interaction phase. - [ ] **Passive App attachment: design hold.** Site mode authorizes one selected origin and supplies native media without starting a local room server. A passive replacement needs an accepted site-consent/discovery flow; it must not admit arbitrary sites or add another runtime owner. Keep Site mode until that decision; removal of the Demo prefill does not authorize changing site trust. -- [ ] **Refine the isolated room-interaction prototype.** Review chat, optional - danmaku and participant-targeted reactions for usability, placement and intended - cross-view synchronization. Preserve room authorization and media-route owners. - Accept its combined layout: sharing settings with the picture, floating chat - without page reflow and room actions before diagnostics. Check narrow-screen - keyboard avoidance, its theater entry and participant menu on real devices; - keep chat available independently of sharing. - Retain current room creation: the first share creates a room, an existing room - can resume, and stopping media keeps the opted-in interaction session. Do not - add visit-triggered creation or a separate pre-share room-creation entry. - This experiment does not authorize integration or publication. -- [ ] **Room voice: design before implementation.** The - [voice assessment](./research/room-voice.md) maps current owners, mature options - and a staged verification plan. Resolve total voice membership/topology, - P2P-only deployment coverage, speaking authority and the Host-commentary - compatibility boundary before implementation. Validate source-audio isolation - from received speech before exposing simultaneous sharing and voice. Keep - room authority, microphone and screen lifetimes distinct; no new room store, - screen-tree reversal or automatic mesh/SFU controller is approved. -- [ ] **Verify media fidelity independently of voice.** - Verify actual left/right separation through Browser, Native and SFU paths; - receive-side stereo negotiation alone does not establish audible fidelity. - Assess HDR capture to correct SDR output for ordinary viewers before full HDR, - with issue [#420](https://github.com/TNTcraftHIM/Piik/issues/420) as the reported - case. Full HDR and surround sound remain design decisions. These findings do - not establish physical device acceptance or authorize a new release. ## Awaiting Device Or Reporter Evidence +- [ ] **HDR, audio-exclusion and mobile interaction device coverage.** Complete + the remaining physical checks in + [verification status](./verification-status.md#candidate-evidence-boundary): + real HDR/mixed displays under load, real voice-app process trees across playback + devices, and phone keyboard/theater/participant-menu interaction. Local GPU, + decoded-video, synthetic audio and responsive-browser evidence do not cover + those environments. Issues [#420](https://github.com/TNTcraftHIM/Piik/issues/420) + (Browser HDR) and [#445](https://github.com/TNTcraftHIM/Piik/issues/445) + (Windows 10 colors) still need matched capture evidence; do not assign them + the locally reproduced Native conversion defects. +- [ ] **Native H.264 motion quality (#432).** The original 720p30 H.264/VP8 + visual comparison remains unresolved. Browser evidence confirms similar + delivered bitrate/format and zero reported loss, but contains no matched + pictures or bitstream. The reporter withdrew the fixture-contaminated CBR + ceiling claim. [Evidence review](./research/native-client-media.md#native-h264-motion-quality) + owns that distinction. Reproduce with identical content and unchanged + production settings before proposing profile, default-FPS or bitrate changes. - [ ] **Self-hosted room creation HTTP 403.** Retest affected deployments using the [address checks](./guide/troubleshooting.md#room-creation-returns-403). Obtain the configured public address/origin and response details; distinguish @@ -78,6 +66,9 @@ is active. Retain these decisions for resumption, not as parallel work. evidence before calling that an ICE/NAT failure. HTTP 1033 belongs to the public-link item below, before media routing. Local fixes and STUN checks do not establish these reporters' causes. + [#443](https://github.com/TNTcraftHIM/Piik/issues/443)'s Host report reaches Native + prepare/answer/candidate processing without a connected remote edge; obtain + the same attempt's Viewer and App transport reports to locate the failure. - [ ] **Camera and Host microphone device coverage.** The owner accepted the sharing layout and authorized release with these physical limits recorded. Verify real audio levels/echo, multiple-device replacement and native mixing on @@ -92,6 +83,9 @@ is active. Retain these decisions for resumption, not as parallel work. [runtime evidence](./research/cross-platform-client-runtime.md#public-invitation-startup). Demo access failures and public-link HTTP 1033 also remain unconfirmed; check connector, DNS/provider and remote access separately from WebRTC availability. + [#434](https://github.com/TNTcraftHIM/Piik/issues/434)'s attached log records DNS + refusal during Cloudflare edge discovery, before readiness; retrying media + cannot repair that resolver failure. Do not assign it to every startup report. - [ ] **Interruption during established viewing.** A Viewer reportedly returns to P2P connecting after watching for a while. Include [#429](https://github.com/TNTcraftHIM/Piik/issues/429)'s reported SFU-to-P2P @@ -126,6 +120,10 @@ is active. Retain these decisions for resumption, not as parallel work. also needs paired evidence; distinguish activation, codec configuration and actual output before changing the encoder contract. Local NVIDIA success does not settle AMD activation. + In [#437](https://github.com/TNTcraftHIM/Piik/issues/437), both manually selected + codecs reportedly work; its Browser report records Auto startup returning + `operation-failed` after 6.85 seconds, before the request timeout. Obtain the + same attempt's App diagnostics to locate selection/activation failure. - [ ] **Share ends after entering a game.** Screen sharing reportedly works until entering a game freezes the picture, followed seconds later by share termination. Version, capture path, codec and matched diagnostics are unknown. @@ -249,3 +247,7 @@ not establish better connection success or speed; this note adds no retry policy 11. **Release-operation policy.** Protected release environments, immutable draft assets and changing mirror-failure policy remain unaccepted proposals; evaluate their benefit before adding release machinery. +12. **Linux ARM64 Server distribution (#439).** Evaluate archives and container + publishing together with deployment names, update links and runtime checks. + A cross-compile alone does not establish a supported package. This is a + community proposal, not part of the current accepted candidate. diff --git a/docs/verification-status.md b/docs/verification-status.md index 470100a..70aaa27 100644 --- a/docs/verification-status.md +++ b/docs/verification-status.md @@ -1,6 +1,6 @@ # Verification Status -Last updated: 2026-09-17 +Last updated: 2026-10-01 This file owns physical evidence limits that change how the product may be described. [Status](./status.md) owns the execution/deployment index. @@ -47,6 +47,26 @@ or a measured improvement worth its implementation and maintenance cost. ## Candidate Evidence Boundary +The Windows HDR-to-SDR candidate has synthetic FP16 conversion and decoded +H.264/VP8 Browser reception evidence. Local displays were in SDR mode; real HDR +content, SDR white on HDR displays, mixed-display movement, display-mode changes +and sustained GPU contention still require physical acceptance. The +[fidelity assessment](./research/media-fidelity.md#hdr-to-sdr) owns measurements. +These checks do not establish Browser-capture HDR behavior or end-to-end HDR. + +Windows source-audio exclusion has bounded parent/child tone-isolation and +target-exit evidence. It does not establish coverage of arbitrary process trees +or real voice applications across playback devices. The +[audio assessment](./research/native-client-media.md#windows-audio-exclusion) +owns that distinction. + +Room interactions have multi-page Browser evidence for stop/resume, idle-room +join, Host departure/return, reconnect, invitation revocation and event timing. +Responsive Browser checks cover narrow portrait and landscape layouts, but not +real phone keyboards, Safari behavior or every touch/device combination. The +owner accepted this bounded UI/model walkthrough for phase closure; these +physical limits remain open and are not release authorization. + The 2026-09-11 functional test acceptance used two genuine Windows App packages and matching Server source/Web builds under the same private contract. Both App/Site version directions delivered decoded 1080p H264 from an isolated native test window. diff --git a/internal/app/app.go b/internal/app/app.go index b645a17..f65036a 100644 --- a/internal/app/app.go +++ b/internal/app/app.go @@ -621,14 +621,15 @@ func discoverNativeMedia(ctx context.Context, configuredPath string) nativeRunti captureProcess: path, capture: capabilities, capabilities: loopback.NativeMediaCapabilities{ - ReceiverReuse: true, - Video: summary.Video, - CaptureBorderControl: summary.CaptureBorderControl, - Microphone: summary.Microphone, - ProcessAudio: summary.ProcessAudio, - SystemAudio: summary.SystemAudio, - HardwareH264: summary.HardwareH264, - SoftwareVP8: summary.SoftwareVP8, + ReceiverReuse: true, + Video: summary.Video, + CaptureBorderControl: summary.CaptureBorderControl, + Microphone: summary.Microphone, + ProcessAudio: summary.ProcessAudio, + ProcessAudioExclusion: summary.ProcessAudioExclusion, + SystemAudio: summary.SystemAudio, + HardwareH264: summary.HardwareH264, + SoftwareVP8: summary.SoftwareVP8, }, } } diff --git a/internal/app/app_test.go b/internal/app/app_test.go index 40ff38d..2b17851 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -121,6 +121,12 @@ func TestPublicLinkStartupDistinguishesCancellationFromFailure(t *testing.T) { } cancel() } + waitForResult := 5 * time.Second + if mode == "failed" { + // Startup includes three retry pauses and four real child launches; + // this is not the cancellation/ready-tunnel retirement deadline. + waitForResult = 15 * time.Second + } select { case err = <-done: joined = true @@ -139,7 +145,7 @@ func TestPublicLinkStartupDistinguishesCancellationFromFailure(t *testing.T) { if ready != (mode == "ready" || mode == "ready-failed") { t.Fatalf("startup readiness = %v for %s", ready, mode) } - case <-time.After(5 * time.Second): + case <-time.After(waitForResult): t.Fatal("App did not retire its local authority and tunnel") } content, err := os.ReadFile(filepath.Join(directory, "logs", "client.log")) diff --git a/internal/app/loopback/server.go b/internal/app/loopback/server.go index 798b931..a2f4e66 100644 --- a/internal/app/loopback/server.go +++ b/internal/app/loopback/server.go @@ -69,14 +69,15 @@ type Health struct { } type NativeMediaCapabilities struct { - ReceiverReuse bool `json:"receiverReuse,omitempty"` - Video bool `json:"video"` - CaptureBorderControl bool `json:"captureBorderControl,omitempty"` - Microphone bool `json:"microphone,omitempty"` - ProcessAudio bool `json:"processAudio"` - SystemAudio bool `json:"systemAudio"` - HardwareH264 bool `json:"hardwareH264"` - SoftwareVP8 bool `json:"softwareVP8"` + ReceiverReuse bool `json:"receiverReuse,omitempty"` + Video bool `json:"video"` + CaptureBorderControl bool `json:"captureBorderControl,omitempty"` + Microphone bool `json:"microphone,omitempty"` + ProcessAudio bool `json:"processAudio"` + ProcessAudioExclusion bool `json:"processAudioExclusion,omitempty"` + SystemAudio bool `json:"systemAudio"` + HardwareH264 bool `json:"hardwareH264"` + SoftwareVP8 bool `json:"softwareVP8"` } type Server struct { diff --git a/internal/app/mediaedge/edge_test.go b/internal/app/mediaedge/edge_test.go index 6b5a495..700c8de 100644 --- a/internal/app/mediaedge/edge_test.go +++ b/internal/app/mediaedge/edge_test.go @@ -65,9 +65,11 @@ func TestOutputPlanRetiresOnlyFailedLayerConsumers(t *testing.T) { lowerTracker, higherTracker := trackers.AddTracker(0), trackers.AddTracker(1) lowerTracker.Observe(0, 1200, 1188, true, 90000, nil) higherTracker.Observe(0, 1200, 1188, true, 90000, nil) - if err = source.DisableLayer(0); err != nil { + retire, err := source.MarkLayerUnavailable(0) + if err != nil { t.Fatal(err) } + retire() if low.State() != webrtc.PeerConnectionStateClosed || high.State() != webrtc.PeerConnectionStateConnected { t.Fatalf("failed/sibling state = %s/%s", low.State(), high.State()) } @@ -174,6 +176,59 @@ func TestRelayFailureCannotRetireReconfiguredProfile(t *testing.T) { } } +func TestRetiredRelayFailureCannotDisableLaterDemand(t *testing.T) { + engine, err := NewEngine(EngineOptions{BindAddress: "127.0.0.1:0", IncludeLoopback: true}) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = engine.Close() }) + source, err := engine.NewSource("vp8", 2, 2, nil) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = source.Close() }) + source.relay = &relayDerivation{source: source} + if err = source.SetFormat(1, 640, 360); err != nil { + t.Fatal(err) + } + if err = source.SetRelayProfile(nativecapture.VideoProfile{ + Width: 1280, Height: 720, Framerate: 30, Bitrate: 3_000_000, Preference: "balanced", + }); err != nil { + t.Fatal(err) + } + high, _, _ := connectedReceiver(t, engine, source, "retiring-relay-high") + low, _, _ := connectedReceiver(t, engine, source, "retiring-relay-low") + if err = low.SetTargetLayer(0); err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithCancelCause(t.Context()) + defer cancel(nil) + done := make(chan struct{}) + t.Cleanup(func() { close(done) }) + run := &relayRun{owner: source.relay, ctx: ctx, cancel: cancel, done: done, + plan: relayPlan{profile: source.relayProfile, format: source.formats[1].Load()}} + source.relay.run = run + if err = low.Close(); err != nil { + t.Fatal(err) + } + if !errors.Is(context.Cause(ctx), context.Canceled) { + t.Fatal("last lower consumer did not retire the derivation normally") + } + // A failure already read from the old process can reach the source lock + // after cancellation, before the process exits and the next run starts. + retire, err := source.markLayerUnavailable(0, run) + if err != nil || retire != nil || source.outputBitrates[0] == 0 { + t.Fatal("normal retirement disabled a reusable output slot") + } + rejoined, _, _ := connectedReceiver(t, engine, source, "retiring-relay-rejoined") + if err = rejoined.SetTargetLayer(0); err != nil { + t.Fatalf("later lower demand was rejected: %v", err) + } + if rejoined.State() != webrtc.PeerConnectionStateConnected || high.State() != webrtc.PeerConnectionStateConnected { + t.Fatal("late failure retired a current consumer or healthy sibling") + } +} + func TestShutdownClosesSiblingTransportsBeforeWaitingForWrites(t *testing.T) { for _, owner := range []string{"source", "engine"} { t.Run(owner, func(t *testing.T) { diff --git a/internal/app/mediaedge/groups.go b/internal/app/mediaedge/groups.go index 15f3c81..b0d8dbe 100644 --- a/internal/app/mediaedge/groups.go +++ b/internal/app/mediaedge/groups.go @@ -95,6 +95,23 @@ func (source *Source) planGroups(demands []outputDemand) (OutputPlan, error) { delete(source.memberships, consumer) } } + compatible := func(group *outputGroup, demand outputDemand) bool { + for _, other := range demands { + if other.active && other.lower && other.budget != demand.budget && + (other.consumer.CurrentSource() == group.media.Source || source.memberships[other.consumer] == group) { + return false + } + } + return true + } + // Revalidate pending joins before they can prevent an incumbent from + // adjusting in place. Membership order must not give stale demand priority. + for _, demand := range demands { + group := source.memberships[demand.consumer] + if demand.active && demand.lower && group != nil && demand.consumer.CurrentSource() != group.media.Source && !compatible(group, demand) { + delete(source.memberships, demand.consumer) + } + } // The previous frame's active bit alone may outlive its last handoff. inUse := func(group *outputGroup) bool { if group == nil || !group.active { @@ -119,6 +136,9 @@ func (source *Source) planGroups(demands []outputDemand) (OutputPlan, error) { for slot := 0; slot < len(source.outputBitrates); slot++ { group := source.groups[slot] if group != nil && source.outputBitrates[slot] > 0 && group.budget == demand.budget { + if demand.consumer.CurrentSource() != group.media.Source && !compatible(group, demand) { + continue + } if selected == nil || inUse(group) { selected = group } @@ -152,15 +172,7 @@ func (source *Source) planGroups(demands []outputDemand) (OutputPlan, error) { if source.outputBitrates[group.slot] == 0 { continue } - compatible := true - for _, other := range demands { - if other.active && other.lower && other.budget != demand.budget && - (other.consumer.CurrentSource() == group.media.Source || source.memberships[other.consumer] == group) { - compatible = false - break - } - } - if compatible { + if compatible(group, demand) { selected = group break } diff --git a/internal/app/mediaedge/groups_test.go b/internal/app/mediaedge/groups_test.go index d0ebf6b..7b23156 100644 --- a/internal/app/mediaedge/groups_test.go +++ b/internal/app/mediaedge/groups_test.go @@ -1,6 +1,7 @@ package mediaedge import ( + "fmt" "testing" "time" @@ -253,6 +254,64 @@ func newGroupLifecycleFixture(t *testing.T) (*Source, *groupTestConsumer, *outpu return source, b, group, demands } +func TestOutputGroupsPreserveCurrentDemandDuringAdmission(t *testing.T) { + for _, pending := range []bool{false, true} { + for _, reverse := range []bool{false, true} { + t.Run(fmt.Sprintf("pending=%v/reverse=%v", pending, reverse), func(t *testing.T) { + source, b, gb, demands := newGroupLifecycleFixture(t) + a := demands[0].consumer + ga := source.memberships[a] + source.installGroup(ga, false) + source.installGroup(gb, false) + groupCount := len(source.groups) + demands[1].budget = 100_000 + if pending { + if _, err := source.planGroups(demands); err != nil { + t.Fatal(err) + } + if source.memberships[b] != ga || b.CurrentSource() != gb.media.Source { + t.Fatal("fixture did not prepare a compatible rejoin") + } + } + source.groupRecovery.Store(0) + demands[0].budget = 60_000 + if reverse { + demands[0], demands[1] = demands[1], demands[0] + } + plan, err := source.planGroups(demands) + if err != nil { + t.Fatal(err) + } + if source.memberships[a] != ga || source.memberships[b] != gb || len(source.groups) != groupCount || + plan.Bitrates[ga.slot] != 60_000 || plan.Bitrates[gb.slot] != 100_000 || source.groupRecovery.Load() != 0 { + t.Fatalf("admission displaced an incumbent: a=%v b=%v groups=%d/%d rates=%v recovery=%d", + source.memberships[a] == ga, source.memberships[b] == gb, len(source.groups), groupCount, + plan.Bitrates, source.groupRecovery.Load()) + } + }) + } + } +} + +func TestOutputGroupsRetainOriginalRecoveryAfterPendingJoin(t *testing.T) { + source, b, gb, demands := newGroupLifecycleFixture(t) + source.installGroup(source.memberships[demands[0].consumer], false) + source.installGroup(gb, false) + demands[1].budget = demands[0].budget + if _, err := source.planGroups(demands); err != nil { + t.Fatal(err) + } + source.groupRecovery.Store(0) + demands[0].budget = 60_000 + demands[1].lower, demands[1].original = false, true + if _, err := source.planGroups(demands); err != nil { + t.Fatal(err) + } + if source.memberships[b] != nil || source.groupRecovery.Load()&2 == 0 || b.CurrentSource() != gb.media.Source { + t.Fatal("return to original lost recovery or replaced the current source early") + } +} + func TestOutputGroupRejectsRecoveryBeforeChangingAttachment(t *testing.T) { for _, path := range []string{"derived", "original"} { for _, stamp := range []string{"future", "duplicate", "before-generation-start"} { diff --git a/internal/app/mediaedge/publication_test.go b/internal/app/mediaedge/publication_test.go index f36fd15..4b94b37 100644 --- a/internal/app/mediaedge/publication_test.go +++ b/internal/app/mediaedge/publication_test.go @@ -73,9 +73,11 @@ func TestPublicationOwnsOneReservationAndSourceLifetime(t *testing.T) { if err != nil { t.Fatal(err) } - if err = source.DisableLayer(0); err != nil { + retire, err := source.MarkLayerUnavailable(0) + if err != nil { t.Fatal(err) } + retire() if _, err = engine.NewPublication(source, EdgeOptions{ConnectionID: "failed-output"}); err == nil { t.Fatal("failed output was admitted into publication metadata") } @@ -117,9 +119,11 @@ func TestPublicationAdmissionIgnoresUnrelatedFailedOutputs(t *testing.T) { t.Fatal(err) } for _, slot := range []int{2, 3, 0, 1} { - if err = source.DisableLayer(slot); err != nil { + retire, err := source.MarkLayerUnavailable(slot) + if err != nil { t.Fatal(err) } + retire() candidate, createErr := engine.NewPublication(source, EdgeOptions{ConnectionID: "replacement"}) if slot >= 2 { if createErr != nil { diff --git a/internal/app/mediaedge/source.go b/internal/app/mediaedge/source.go index de2ef3d..77a781f 100644 --- a/internal/app/mediaedge/source.go +++ b/internal/app/mediaedge/source.go @@ -1,6 +1,7 @@ package mediaedge import ( + "context" "errors" "io" "sync" @@ -211,12 +212,10 @@ func (source *Source) ConfigureOutputs(ceilings []uint32) error { return nil } -func (source *Source) DisableLayer(layer int) error { - retire, err := source.markLayerUnavailable(layer, nil) - if retire != nil { - retire() - } - return err +// MarkLayerUnavailable commits output failure without closing transports. The +// caller must run the returned retirement after releasing its capture-owner lock. +func (source *Source) MarkLayerUnavailable(layer int) (func(), error) { + return source.markLayerUnavailable(layer, nil) } func (source *Source) markLayerUnavailable(layer int, run *relayRun) (func(), error) { @@ -227,8 +226,10 @@ func (source *Source) markLayerUnavailable(layer int, run *relayRun) (func(), er source.writeMu.Lock() defer source.writeMu.Unlock() source.mu.Lock() - if run != nil && (run.owner.run != run || source.relayProfile != run.plan.profile || - source.formats[len(source.formats)-1].Load() != run.plan.format) { + // Normal retirement cannot disable a later run's slots. A real cancellation + // cause still owns failure isolation until that run has finished retiring. + if run != nil && (run.owner.run != run || errors.Is(context.Cause(run.ctx), context.Canceled) || + source.relayProfile != run.plan.profile || source.formats[len(source.formats)-1].Load() != run.plan.format) { source.mu.Unlock() return nil, nil } diff --git a/internal/app/nativeaudio/encoder_test.go b/internal/app/nativeaudio/encoder_test.go index 82ab344..0744054 100644 --- a/internal/app/nativeaudio/encoder_test.go +++ b/internal/app/nativeaudio/encoder_test.go @@ -85,6 +85,58 @@ func TestEncodePreservesAudiblePCM(t *testing.T) { } } +func TestEncodePreservesStereoSeparation(t *testing.T) { + for _, bitrate := range []int{64_000, 128_000, 192_000} { + encoder, err := NewEncoder(bitrate) + if err != nil { + t.Fatal(err) + } + decoder, err := gopus.NewDecoder(gopus.DefaultDecoderConfig(SampleRate, Channels)) + if err != nil { + t.Fatal(err) + } + pcm := make([]byte, FrameBytes) + decoded := make([]int16, FrameSamples*Channels) + frequencies := [2]float64{500, 1500} + var tones [2][2]complex128 + for block := 0; block < 30; block++ { + for frame := 0; frame < FrameSamples; frame++ { + for channel, frequency := range frequencies { + phase := 2 * math.Pi * frequency * float64(block*FrameSamples+frame) / SampleRate + binary.LittleEndian.PutUint16(pcm[(frame*Channels+channel)*2:], uint16(int16(8_000*math.Sin(phase)))) + } + } + packet, err := encoder.Encode(pcm) + if err != nil { + t.Fatal(err) + } + count, err := decoder.DecodeInt16(packet, decoded) + if err != nil || count != FrameSamples { + t.Fatalf("decode: %d samples, %v", count, err) + } + if block < 5 { + continue // Exclude codec startup; compare frequency energy, not phase delay. + } + for frame := 0; frame < count; frame++ { + for tone, frequency := range frequencies { + phase := 2 * math.Pi * frequency * float64(block*FrameSamples+frame) / SampleRate + for channel := range Channels { + tones[channel][tone] += complex(float64(decoded[frame*Channels+channel]), 0) * complex(math.Cos(phase), math.Sin(phase)) + } + } + } + } + for channel := range Channels { + wanted, other := tones[channel][channel], tones[channel][1-channel] + level := math.Hypot(real(wanted), imag(wanted)) + leak := math.Hypot(real(other), imag(other)) + if level < 1 || level < 10*leak { + t.Fatalf("%d bps channel %d lost separation: signal %f, leakage %f", bitrate, channel, level, leak) + } + } + } +} + func TestNewEncoderRejectsOutOfRangeBitrate(t *testing.T) { for _, bitrate := range []int{0, 5_999, 510_001} { if _, err := NewEncoder(bitrate); err == nil { diff --git a/internal/app/nativecapture/probe.go b/internal/app/nativecapture/probe.go index cc30fcb..8f4a0a1 100644 --- a/internal/app/nativecapture/probe.go +++ b/internal/app/nativecapture/probe.go @@ -42,36 +42,39 @@ type Adapter struct { } type Capabilities struct { - Protocol int `json:"protocol"` - Platform string `json:"platform"` - PlatformBuild string `json:"platformBuild"` - VideoCapture bool `json:"videoCapture"` - CaptureBorderControl bool `json:"captureBorderControl,omitempty"` - Microphone bool `json:"microphone,omitempty"` - ProcessAudio bool `json:"processAudio"` - SystemAudio bool `json:"systemAudio"` - SoftwareVP8 bool `json:"softwareVP8"` - Adapters []Adapter `json:"adapters"` + Protocol int `json:"protocol"` + Platform string `json:"platform"` + PlatformBuild string `json:"platformBuild"` + VideoCapture bool `json:"videoCapture"` + CaptureBorderControl bool `json:"captureBorderControl,omitempty"` + Microphone bool `json:"microphone,omitempty"` + ProcessAudio bool `json:"processAudio"` + ProcessAudioExclusion bool `json:"processAudioExclusion,omitempty"` + SystemAudio bool `json:"systemAudio"` + SoftwareVP8 bool `json:"softwareVP8"` + Adapters []Adapter `json:"adapters"` } type Summary struct { - Video bool - CaptureBorderControl bool - Microphone bool - ProcessAudio bool - SystemAudio bool - HardwareH264 bool - SoftwareVP8 bool + Video bool + CaptureBorderControl bool + Microphone bool + ProcessAudio bool + ProcessAudioExclusion bool + SystemAudio bool + HardwareH264 bool + SoftwareVP8 bool } func (capabilities Capabilities) Summary() Summary { summary := Summary{ - Video: capabilities.VideoCapture, - CaptureBorderControl: capabilities.VideoCapture && capabilities.CaptureBorderControl, - Microphone: capabilities.Microphone, - ProcessAudio: capabilities.ProcessAudio, - SystemAudio: capabilities.SystemAudio, - SoftwareVP8: capabilities.SoftwareVP8, + Video: capabilities.VideoCapture, + CaptureBorderControl: capabilities.VideoCapture && capabilities.CaptureBorderControl, + Microphone: capabilities.Microphone, + ProcessAudio: capabilities.ProcessAudio, + ProcessAudioExclusion: capabilities.ProcessAudioExclusion && capabilities.Microphone, + SystemAudio: capabilities.SystemAudio, + SoftwareVP8: capabilities.SoftwareVP8, } for _, adapter := range capabilities.Adapters { if len(adapter.HardwareH264) > 0 { diff --git a/internal/app/nativecapture/probe_test.go b/internal/app/nativecapture/probe_test.go index 6f2d1ef..893b955 100644 --- a/internal/app/nativecapture/probe_test.go +++ b/internal/app/nativecapture/probe_test.go @@ -33,6 +33,22 @@ func TestDecodeProbeRejectsUnknownTrailingAndDuplicateData(t *testing.T) { } } +func TestAudioExclusionCapabilityRequiresExplicitSupportAndMixing(t *testing.T) { + capabilities, err := decodeProbe([]byte(validProbe)) + if err != nil || capabilities.Summary().ProcessAudioExclusion { + t.Fatal("old capture probe advertised audio exclusion", err) + } + capabilities, err = decodeProbe([]byte(strings.Replace(validProbe, `"processAudio":true`, + `"processAudio":true,"processAudioExclusion":true,"microphone":true`, 1))) + if err != nil || !capabilities.Summary().ProcessAudioExclusion { + t.Fatal("explicit audio exclusion capability was lost", err) + } + capabilities.Microphone = false + if capabilities.Summary().ProcessAudioExclusion { + t.Fatal("audio exclusion requires the replaceable mixed source") + } +} + func TestSummaryDoesNotConflateVideoAudioAndEncoder(t *testing.T) { capabilities, err := decodeProbe([]byte( `{"protocol":7,"platform":"windows","platformBuild":"19045","videoCapture":true,"processAudio":false,"systemAudio":true,"adapters":[{"index":0,"name":"GPU","identity":"0x0:0x1","hardwareH264":[]}]}`, diff --git a/internal/app/nativecapture/stream.go b/internal/app/nativecapture/stream.go index b8caf53..2b31d66 100644 --- a/internal/app/nativecapture/stream.go +++ b/internal/app/nativecapture/stream.go @@ -281,13 +281,17 @@ func appendOutputArguments(arguments []string, outputs []OutputProfile) ([]strin return arguments, nil } -func StartAudio(parent context.Context, executable string, target CaptureTarget) (*Stream, error) { - if !target.Valid() { +func StartAudio(parent context.Context, executable string, target CaptureTarget, exclude bool) (*Stream, error) { + if !target.Valid() || (exclude && target.Kind != "window") { return nil, errors.New("native audio target is invalid") } + kind := target.Kind + if exclude { + kind = "exclude" + } return startAudioStream(parent, executable, []string{ "--capture-audio", - target.Kind, + kind, strconv.FormatUint(uint64(target.PID), 10), zeroWhenEmpty(target.CreationTime), }) diff --git a/internal/app/nativecontrol/session.go b/internal/app/nativecontrol/session.go index 19d80f4..40a59a7 100644 --- a/internal/app/nativecontrol/session.go +++ b/internal/app/nativecontrol/session.go @@ -182,6 +182,12 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any, !validQualitySettings(request.Profile) { return nil, protocolViolation("native start-share request is invalid") } + if err := session.validateAudioExclusion(request.Source, request.Audio, request.ExcludeAudio); err != nil { + return nil, err + } + if request.ExcludeAudio != nil && !request.MicrophoneMixing { + return nil, errors.New("native audio exclusion requires mixed audio") + } slog.Debug("piik-client", "event", "native-profile-requested", "requestId", envelope.ID, "share", diagnostics.ID(request.ShareID), "profile", nativeQualityProfile(request.Profile), "codec", request.Codec, "audio", request.Audio, "sourceKind", request.Source.Kind, "adapterIndex", request.AdapterIndex, "encoderIndex", request.EncoderIndex) @@ -245,6 +251,9 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any, !validIdentities(request.ShareID) || !request.Source.Valid() { return nil, protocolViolation("native replace-share-source request is invalid") } + if err := session.validateAudioExclusion(request.Source, request.Audio, request.ExcludeAudio); err != nil { + return nil, err + } host := session.current(request.ShareID) if host == nil { return nil, errors.New("native share does not exist") @@ -261,7 +270,7 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any, ShowCaptureBorder: request.ShowCaptureBorder && session.capabilities.CaptureBorderControl, AdapterIndex: request.AdapterIndex, EncoderIndex: request.EncoderIndex, - }, audio) + }, audio, request.ExcludeAudio) if err != nil { return nil, err } @@ -534,6 +543,19 @@ func (session *Session) finishHostOperation(done chan struct{}, result any) { session.mu.Unlock() } +func (session *Session) validateAudioExclusion(source nativecapture.CaptureTarget, audio bool, excluded *nativecapture.CaptureTarget) error { + if excluded == nil { + return nil + } + if !audio || source.Kind != "display" || excluded.Kind != "window" || !excluded.Valid() { + return protocolViolation("native audio exclusion target is invalid") + } + if !session.capabilities.Summary().ProcessAudioExclusion || !session.capabilities.SystemAudio { + return errors.New("native audio exclusion is unavailable") + } + return nil +} + func (session *Session) shareUpdateResult(request requestEnvelope, shareID string, profile nativehost.QualityProfile, err error) any { if err != nil { return operationFailure(request, err) @@ -571,6 +593,7 @@ func (session *Session) startShare( Profile: profile.Video, }, MicrophoneMixing: request.MicrophoneMixing && session.capabilities.Microphone, + ExcludeAudio: request.ExcludeAudio, Profile: profile, EdgeCapacity: request.EdgeCapacity, AudioEnabled: request.Audio && session.capabilities.Summary().AudioFor( diff --git a/internal/app/nativecontrol/session_test.go b/internal/app/nativecontrol/session_test.go index acd5d15..cad4b95 100644 --- a/internal/app/nativecontrol/session_test.go +++ b/internal/app/nativecontrol/session_test.go @@ -641,3 +641,32 @@ func TestReplaceShareSourceKeepsOneStrictTargetShape(t *testing.T) { t.Fatalf("malformed target was treated as an operational failure: %v", err) } } + +func TestAudioExclusionValidatesScopeWithoutDroppingRequestedPolicy(t *testing.T) { + session := &Session{capabilities: nativecapture.Capabilities{Microphone: true, SystemAudio: true, ProcessAudioExclusion: true}} + display := nativecapture.CaptureTarget{Kind: "display", SourceID: "1", Title: "Screen"} + window := nativecapture.CaptureTarget{Kind: "window", SourceID: "2", PID: 123, CreationTime: "456", Title: "Voice fixture"} + if err := session.validateAudioExclusion(display, true, &window); err != nil { + t.Fatal(err) + } + for _, invalid := range []struct { + source nativecapture.CaptureTarget + audio bool + excluded nativecapture.CaptureTarget + }{ + {display, false, window}, {window, true, window}, {display, true, display}, + {display, true, nativecapture.CaptureTarget{Kind: "window", SourceID: "2", PID: 123, Title: "Missing lifetime"}}, + } { + if _, ok := session.validateAudioExclusion(invalid.source, invalid.audio, &invalid.excluded).(protocolViolation); !ok { + t.Fatal("malformed audio exclusion was not rejected") + } + } + session.capabilities.ProcessAudioExclusion = false + err := session.validateAudioExclusion(display, true, &window) + if _, malformed := err.(protocolViolation); err == nil || malformed { + t.Fatal("unsupported audio exclusion must fail the operation, not silently widen capture or close control") + } + if err := session.validateAudioExclusion(display, true, nil); err != nil { + t.Fatal("legacy capture became dependent on exclusion support") + } +} diff --git a/internal/app/nativecontrol/wire.go b/internal/app/nativecontrol/wire.go index a68404d..b69ab1c 100644 --- a/internal/app/nativecontrol/wire.go +++ b/internal/app/nativecontrol/wire.go @@ -42,19 +42,20 @@ type qualitySettings struct { } type startShareRequest struct { - Version int `json:"version"` - ID string `json:"id"` - Type string `json:"type"` - ShareID string `json:"shareId"` - Source nativecapture.CaptureTarget `json:"source"` - Audio bool `json:"audio"` - MicrophoneMixing bool `json:"microphoneMixing,omitempty"` - ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"` - AdapterIndex uint32 `json:"adapterIndex"` - EncoderIndex uint32 `json:"encoderIndex"` - EdgeCapacity int `json:"edgeCapacity"` - Profile qualitySettings `json:"profile"` - Codec string `json:"codec"` + Version int `json:"version"` + ID string `json:"id"` + Type string `json:"type"` + ShareID string `json:"shareId"` + Source nativecapture.CaptureTarget `json:"source"` + Audio bool `json:"audio"` + MicrophoneMixing bool `json:"microphoneMixing,omitempty"` + ExcludeAudio *nativecapture.CaptureTarget `json:"excludeAudio,omitempty"` + ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"` + AdapterIndex uint32 `json:"adapterIndex"` + EncoderIndex uint32 `json:"encoderIndex"` + EdgeCapacity int `json:"edgeCapacity"` + Profile qualitySettings `json:"profile"` + Codec string `json:"codec"` } type microphoneRequest struct { @@ -81,15 +82,16 @@ type updateShareRequest struct { } type replaceShareSourceRequest struct { - Version int `json:"version"` - ID string `json:"id"` - Type string `json:"type"` - ShareID string `json:"shareId"` - Source nativecapture.CaptureTarget `json:"source"` - Audio bool `json:"audio"` - ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"` - AdapterIndex uint32 `json:"adapterIndex"` - EncoderIndex uint32 `json:"encoderIndex"` + Version int `json:"version"` + ID string `json:"id"` + Type string `json:"type"` + ShareID string `json:"shareId"` + Source nativecapture.CaptureTarget `json:"source"` + Audio bool `json:"audio"` + ExcludeAudio *nativecapture.CaptureTarget `json:"excludeAudio,omitempty"` + ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"` + AdapterIndex uint32 `json:"adapterIndex"` + EncoderIndex uint32 `json:"encoderIndex"` } type stopShareRequest struct { diff --git a/internal/app/nativehost/audio_mix_test.go b/internal/app/nativehost/audio_mix_test.go index 00302c5..bbfb7e3 100644 --- a/internal/app/nativehost/audio_mix_test.go +++ b/internal/app/nativehost/audio_mix_test.go @@ -88,6 +88,83 @@ func startMixedFixture(t *testing.T) (*Session, nativecapture.VideoOptions, <-ch return session, options, events } +func TestAudioExclusionFailureMutesOnlySourceAndNeverFallsBack(t *testing.T) { + session, options, events := startMixedFixture(t) + enabled := true + if err := session.SetMicrophone(&enabled, nil, nil); err != nil { + t.Fatal(err) + } + output := session.audioSource + session.mixer.mu.Lock() + microphone := session.mixer.microphone + session.mixer.mu.Unlock() + excluded := &nativecapture.CaptureTarget{Kind: "window", SourceID: "123", PID: 456, CreationTime: "789", Title: "Voice fixture"} + for _, fault := range []string{"PIIK_VIDEO_FAIL", "PIIK_EXCLUSION_FAIL"} { + if err := session.ReplaceSource(options, true, nil); err != nil { + t.Fatal(err) + } + video := session.stream + t.Setenv(fault, "1") + if err := session.ReplaceSource(options, true, excluded); err == nil { + t.Fatal("failed exclusion was accepted", fault) + } + if session.SourceAudio() == nil || *session.SourceAudio() || session.audioStream != nil || session.stream != video { + t.Fatal("failed exclusion retained unfiltered audio or retired video", fault) + } + t.Setenv(fault, "") + } + marker := filepath.Join(t.TempDir(), "excluded") + t.Setenv("PIIK_EXCLUSION_STARTED", marker) + if err := session.ReplaceSource(options, true, excluded); err != nil { + t.Fatal(err) + } + identity, err := os.ReadFile(marker) + if err != nil || string(identity) != "456:789" || !*session.SourceAudio() { + t.Fatal("replacement did not bind exclusion to the selected process lifetime", err) + } + sourceAudio := session.audioStream + profile := session.profile + profile.AudioBitrate = 128000 + if err := session.UpdateProfile(profile); err != nil { + t.Fatal(err) + } + session.SetPaused(true) + session.SetPaused(false) + if session.audioStream != sourceAudio { + t.Fatal("profile/pause replaced exclusion") + } + for len(events) > 0 { + <-events + } + if err := sourceAudio.Close(); err != nil { + t.Fatal(err) + } + timer := time.NewTimer(time.Second) + defer timer.Stop() + waiting := true + for waiting { + select { + case event := <-events: + if event.Type != "audio-state" { + continue + } + if event.SourceAudio || !event.Microphone || event.Failed { + t.Fatalf("source loss was misreported: %+v", event) + } + waiting = false + case <-timer.C: + t.Fatal("source loss was not reported") + } + } + session.mixer.mu.Lock() + sameMicrophone := session.mixer.microphone == microphone + sourceEnded := session.mixer.source == nil + session.mixer.mu.Unlock() + if !sourceEnded || !sameMicrophone || session.audioSource != output || !session.HasAudio() { + t.Fatal("exclusion loss disturbed microphone or output ownership") + } +} + func TestNativeMicrophonePreservesMediaOwnersAcrossInputChanges(t *testing.T) { session, options, events := startMixedFixture(t) check := func(err error) { @@ -130,7 +207,7 @@ func TestNativeMicrophonePreservesMediaOwnersAcrossInputChanges(t *testing.T) { } microphone = replaced options.Target.SourceID = "2" - check(session.ReplaceSource(options, true)) + check(session.ReplaceSource(options, true, nil)) session.mixer.mu.Lock() sameMicrophone := session.mixer.microphone == microphone session.mixer.mu.Unlock() diff --git a/internal/app/nativehost/audio_test.go b/internal/app/nativehost/audio_test.go index 0018fab..f2fc435 100644 --- a/internal/app/nativehost/audio_test.go +++ b/internal/app/nativehost/audio_test.go @@ -56,6 +56,17 @@ func runAudioRecoveryCapture() { } } audio := microphone || (len(os.Args) > 1 && os.Args[1] == "--capture-audio") + if !audio && os.Getenv("PIIK_VIDEO_FAIL") == "1" { + os.Exit(2) + } + if !microphone && audio && os.Args[2] == "exclude" { + if os.Getenv("PIIK_EXCLUSION_FAIL") == "1" { + os.Exit(2) + } + if marker := os.Getenv("PIIK_EXCLUSION_STARTED"); marker != "" { + _ = os.WriteFile(marker, []byte(os.Args[3]+":"+os.Args[4]), 0600) + } + } period := time.Second / 30 status := []byte(`{"state":"active","audio":true}`) if audio { @@ -179,7 +190,7 @@ func TestAudioEOFWaitsForExplicitSourceReplacement(t *testing.T) { waitPackets(&videoPackets, videoPackets.Load()+10, "video after audio EOF") before := audioPackets.Load() options.Target.SourceID = "2" - check(session.ReplaceSource(options, true)) + check(session.ReplaceSource(options, true, nil)) if session.currentStream() == previousVideo || session.currentAudioStream() == previousAudio || session.edge("audio-edge") != edge || session.audioSource != audioSource { t.Fatal("explicit replacement did not preserve the media owners") diff --git a/internal/app/nativehost/session.go b/internal/app/nativehost/session.go index 6d528d7..174b284 100644 --- a/internal/app/nativehost/session.go +++ b/internal/app/nativehost/session.go @@ -67,6 +67,7 @@ type Options struct { Video nativecapture.VideoOptions Profile QualityProfile AudioEnabled bool + ExcludeAudio *nativecapture.CaptureTarget MicrophoneMixing bool EdgeCapacity int BindAddress string @@ -138,6 +139,7 @@ func Start(parent context.Context, options Options) (*Session, error) { parent, options.CaptureProcess, options.Video.Target, + options.ExcludeAudio, ) if audioErr != nil { slog.DebugContext(parent, "piik-client", "event", "capture-audio-unavailable", "share", diagnostics.ID(options.ShareID), diagnostics.Error(audioErr)) @@ -401,6 +403,7 @@ func (session *Session) UpdateProfile(profile QualityProfile) error { func (session *Session) ReplaceSource( options nativecapture.VideoOptions, audioEnabled bool, + excludeAudio *nativecapture.CaptureTarget, ) error { session.updateMu.Lock() defer session.updateMu.Unlock() @@ -420,6 +423,21 @@ func (session *Session) ReplaceSource( if session.mixer == nil && audioEnabled != hasAudio { return errors.New("native source audio availability cannot change while sharing") } + if excludeAudio != nil { + if session.mixer == nil { + return errors.New("native audio exclusion requires mixed audio") + } + // A privacy change retires the old source before preparation. Failure must + // leave it silent rather than retaining unfiltered audio; video/mic stay live. + session.mu.Lock() + previousAudio := session.audioStream + session.audioStream = nil + session.mu.Unlock() + session.mixer.setSource(nil) + if previousAudio != nil { + _ = previousAudio.Close() + } + } options.Profile = profile.Video options.RestoreToken = "" replacement, state, err := session.prepareVideo(session.ctx, options, options.Target.Kind == "picker", false) @@ -432,6 +450,7 @@ func (session *Session) ReplaceSource( session.ctx, session.captureProcess, options.Target, + excludeAudio, ) if err != nil { _ = replacement.Close() @@ -452,11 +471,15 @@ func startAudioCapture( ctx context.Context, captureProcess string, target nativecapture.CaptureTarget, + excludeAudio *nativecapture.CaptureTarget, ) (*nativecapture.Stream, error) { + if excludeAudio != nil { + return nativecapture.StartAudio(ctx, captureProcess, *excludeAudio, true) + } if target.Kind == "display" || target.Kind == "picker" { return nativecapture.StartSystemAudio(ctx, captureProcess) } - return nativecapture.StartAudio(ctx, captureProcess, target) + return nativecapture.StartAudio(ctx, captureProcess, target, false) } func (session *Session) prepareVideo( @@ -560,7 +583,7 @@ func (session *Session) installCapture(next *nativecapture.Stream) error { // Preparation has consumed these events. Apply them to the new generation // before acknowledging it, just as the live reader handles unavailable layers. for _, layer := range unavailable { - if err := session.source.DisableLayer(layer); err != nil { + if err := session.disableCaptureLayer(next, layer); err != nil { return err } } @@ -575,6 +598,22 @@ func (session *Session) installCapture(next *nativecapture.Stream) error { return nil } +func (session *Session) disableCaptureLayer(stream *nativecapture.Stream, layer int) error { + session.mu.Lock() + if session.stream != stream || session.closed || session.ctx.Err() != nil { + session.mu.Unlock() + return nil + } + // Accept failure atomically with capture selection. Closing transports may + // wait for callbacks, so only the source-state change belongs under this lock. + retire, err := session.source.MarkLayerUnavailable(layer) + session.mu.Unlock() + if retire != nil { + retire() + } + return err +} + func (session *Session) CloseEdge(connectionID string) { session.mu.Lock() edge := session.edges[connectionID] @@ -831,7 +870,7 @@ func (session *Session) runVideo() error { if session.source == nil { return fail(errors.New("native output ended before its source state")) } - if err = session.source.DisableLayer(frame.Layer); err != nil { + if err = session.disableCaptureLayer(current, frame.Layer); err != nil { return fail(err) } case nativecapture.FramePCM: diff --git a/internal/app/nativehost/session_test.go b/internal/app/nativehost/session_test.go index a581b55..8f8003c 100644 --- a/internal/app/nativehost/session_test.go +++ b/internal/app/nativehost/session_test.go @@ -11,6 +11,7 @@ import ( "slices" "strconv" "strings" + "sync" "testing" "time" @@ -189,6 +190,122 @@ func TestCaptureCommitWaitsForReaderMetadataOrTermination(t *testing.T) { } } +func TestCaptureFailureBelongsToSelectedStream(t *testing.T) { + for _, replace := range []bool{false, true} { + t.Run(strconv.FormatBool(replace), func(t *testing.T) { + check := func(err error) { + t.Helper() + if err != nil { + t.Fatal(err) + } + } + t.Setenv("PIIK_NATIVEHOST_PIPE_FIXTURE", "backend") + t.Setenv("PIIK_CAPTURE_STARTING", string(captureStatePayload(t, + `{"state":"starting","codec":"vp8","adapterIndex":0,"adapterName":"Fixture","adapterIdentity":"0:0","encoderName":"VP8","encoderIdentity":"vp8"}`))) + t.Setenv("PIIK_CAPTURE_ACTIVE", string(captureStatePayload(t, `{"state":"active","codec":"vp8","width":1280,"height":720,"fps":30}`))) + t.Setenv("PIIK_CAPTURE_UNAVAILABLE", "0") + executable, err := os.Executable() + check(err) + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + options := nativecapture.VideoOptions{Target: nativecapture.CaptureTarget{Kind: "display", SourceID: "1", Title: "Fixture"}, Codec: "vp8", + Profile: nativecapture.VideoProfile{Width: 1280, Height: 720, Framerate: 30, Bitrate: 3_000_000, Preference: "balanced"}, OutputGroups: 4} + previous, err := nativecapture.StartVideo(ctx, executable, options) + check(err) + defer previous.Close() + t.Setenv("PIIK_CAPTURE_UNAVAILABLE", "") + replacement, err := nativecapture.StartVideo(ctx, executable, options) + check(err) + defer replacement.Close() + engine, err := mediaedge.NewEngine(mediaedge.EngineOptions{BindAddress: "127.0.0.1:0", IncludeLoopback: true}) + check(err) + defer engine.Close() + source, err := engine.NewSource("vp8", 4, 2, nil) + check(err) + defer source.Close() + check(configureCaptureOutputs(source, previous.Outputs())) + publication, err := engine.NewPublication(source, mediaedge.EdgeOptions{ConnectionID: "capture-failure-owner"}) + check(err) + active := make(chan struct{}, 1) + session := &Session{ctx: ctx, cancel: cancel, engine: engine, source: source, stream: previous, + videoOptions: options, ready: make(chan error, 1), events: func(_ context.Context, event Event) { + if event.State == "active" { + select { + case active <- struct{}{}: + default: + } + } + }} + // Delay the actual reader after its first identity check, without a + // production hook. Log I/O and ordinary scheduling can delay it here. + entered, release := make(chan struct{}), make(chan struct{}) + unblock := sync.OnceFunc(func() { close(release) }) + previousLog := slog.Default() + defer slog.SetDefault(previousLog) + slog.SetDefault(slog.New(slog.NewTextHandler(io.Discard, &slog.HandlerOptions{ + Level: slog.LevelDebug, ReplaceAttr: func(_ []string, attr slog.Attr) slog.Attr { + if attr.Key == "event" && attr.Value.String() == "capture-output-unavailable" { + close(entered) + <-release + } + return attr + }, + }))) + readerDone := make(chan error, 1) + go func() { readerDone <- session.runVideo() }() + defer func() { + unblock() + cancel() + _ = previous.Close() + _ = replacement.Close() + select { + case <-readerDone: + case <-time.After(5 * time.Second): + t.Error("capture reader did not retire") + } + }() + select { + case <-entered: + case err = <-readerDone: + readerDone <- err + t.Fatalf("reader stopped before output failure: %v", err) + case <-time.After(5 * time.Second): + t.Fatal("old output failure did not reach the reader") + } + committed := make(chan error, 1) + if replace { + go func() { + committed <- session.commitCapture(options, QualityProfile{Video: options.Profile}, replacement, CaptureState{}, nil, false) + }() + deadline := time.Now().Add(5 * time.Second) + for session.currentStream() != replacement { + if time.Now().After(deadline) { + t.Fatal("replacement was not selected") + } + time.Sleep(time.Millisecond) + } + } + unblock() + select { + case <-active: + case <-time.After(5 * time.Second): + t.Fatal("capture reader did not finish the failure or handoff") + } + if replace { + select { + case err = <-committed: + check(err) + case <-time.After(5 * time.Second): + t.Fatal("replacement installation was not acknowledged") + } + } + if err = publication.SetActiveCount(1); (err == nil) != replace { + t.Fatalf("publication availability after replacement=%v: %v", replace, err) + } + }) + } +} + func TestCaptureProfileRetainsSelectedBackendFromStarting(t *testing.T) { t.Setenv("PIIK_NATIVEHOST_PIPE_FIXTURE", "backend") t.Setenv("PIIK_CAPTURE_STARTING", string(captureStatePayload(t, diff --git a/internal/server/app/app.go b/internal/server/app/app.go index acf53d1..f58605b 100644 --- a/internal/server/app/app.go +++ b/internal/server/app/app.go @@ -154,6 +154,7 @@ func New(options Options) (*Server, error) { server.signalOptions = signal.Options{ Store: store, EndpointMediaCopyCapacity: configuration.EndpointMediaCopyCapacity, + SfuOnly: configuration.SFU != nil && configuration.SFU.Only, Ice: config.IceConfig(configuration), // The TS signaling server read natPredictionEnabled off the same // `ice` options object it forwarded; the Go IceConfig is already the diff --git a/internal/server/app/app_test.go b/internal/server/app/app_test.go index 8134f50..c0aee53 100644 --- a/internal/server/app/app_test.go +++ b/internal/server/app/app_test.go @@ -366,7 +366,17 @@ func TestCapabilitiesReportServicesDisabledByDefault(t *testing.T) { server := start(t, Options{Config: testConfig(t)}) server.do(http.MethodGet, "/api/capabilities"). - expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":false}`) + expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":false,"roomInteractions":true,"hostRoomSession":true}`) + +} + +func TestCapabilitiesReportSFUOnlyPolicy(t *testing.T) { + configuration := testConfig(t) + configuration.SFU = &config.SFUConfig{ListenHost: "127.0.0.1", Port: 0, Only: true} + configuration.NATPredictionEnabled = true + server := start(t, Options{Config: configuration}) + server.do(http.MethodGet, "/api/capabilities"). + expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":true,"sfuOnly":true,"natPrediction":false,"roomInteractions":true,"hostRoomSession":true}`) } func TestCapabilitiesReportOptionalNATPredictionWithoutExposingConfiguration(t *testing.T) { @@ -377,7 +387,7 @@ func TestCapabilitiesReportOptionalNATPredictionWithoutExposingConfiguration(t * server := start(t, Options{Config: configuration}) server.do(http.MethodGet, "/api/capabilities"). - expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":true}`). + expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":true,"roomInteractions":true,"hostRoomSession":true}`). expectHeader("Cache-Control", "no-store"). expectHeader("X-Content-Type-Options", "nosniff") @@ -420,7 +430,8 @@ func TestSFUListenerFollowsApplicationCloseAndEnd(t *testing.T) { t.Fatal("missing runtime owner") } server.do(http.MethodGet, "/api/capabilities"). - expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":true,"natPrediction":false}`) + expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":true,"natPrediction":false,"roomInteractions":true,"hostRoomSession":true}`) + var err error if ending { err = server.End(context.Background()) diff --git a/internal/server/app/routes.go b/internal/server/app/routes.go index 7a35806..278f518 100644 --- a/internal/server/app/routes.go +++ b/internal/server/app/routes.go @@ -80,7 +80,10 @@ func (s *Server) route( sendJSON(writer, http.StatusOK, protocol.RuntimeCapabilities{ ConnectionAttemptProgress4: true, Sfu: s.config.SFU != nil, - NatPrediction: s.config.NATPredictionEnabled, + SfuOnly: s.config.SFU != nil && s.config.SFU.Only, + NatPrediction: s.config.NATPredictionEnabled && !(s.config.SFU != nil && s.config.SFU.Only), + RoomInteractions: true, + HostRoomSession: true, }) return } diff --git a/internal/server/config/config.go b/internal/server/config/config.go index 481c0da..2a5bcd0 100644 --- a/internal/server/config/config.go +++ b/internal/server/config/config.go @@ -68,6 +68,7 @@ type SFUConfig struct { ListenHost string Port int PublicIP string + Only bool } // Config describes runtime configuration. Zero values mean open site @@ -273,8 +274,15 @@ func parseRoomDatabasePath(value string) (string, error) { } func parseSFU(env map[string]string) (*SFUConfig, error) { + only, err := parseBoolean(env["SFU_ONLY"], "SFU_ONLY") + if err != nil { + return nil, err + } rawPort := strings.TrimSpace(env["SFU_UDP_PORT"]) if rawPort == "" { + if only { + return nil, errors.New("SFU_ONLY requires SFU_UDP_PORT") + } return nil, nil } port, err := parseBoundedInteger(rawPort, 0, "SFU_UDP_PORT", 1, maxPort) @@ -298,6 +306,7 @@ func parseSFU(env map[string]string) (*SFUConfig, error) { ListenHost: listenHost, Port: int(port), PublicIP: publicIP, + Only: only, }, nil } diff --git a/internal/server/config/config_test.go b/internal/server/config/config_test.go index c6a5ed3..4d17164 100644 --- a/internal/server/config/config_test.go +++ b/internal/server/config/config_test.go @@ -187,6 +187,21 @@ func TestLoadSFUOptional(t *testing.T) { } } +func TestSFUOnlyRequiresAnEnabledListener(t *testing.T) { + configuration := mustLoad(t, map[string]string{"SFU_UDP_PORT": "7882", "SFU_ONLY": "true"}) + if !configuration.SFU.Only { + t.Fatal("SFU_ONLY was ignored") + } + for _, values := range []map[string]string{ + {"SFU_ONLY": "true"}, + {"SFU_ONLY": "yes", "SFU_UDP_PORT": "7882"}, + } { + if _, err := Load(values); err == nil { + t.Fatalf("accepted invalid SFU policy: %v", values) + } + } +} + func TestLoadAccepts(t *testing.T) { developmentPath, err := filepath.Abs(filepath.Join("state", "rooms.sqlite")) if err != nil { diff --git a/internal/server/protocol/client.go b/internal/server/protocol/client.go index 8bc3e54..ae318ab 100644 --- a/internal/server/protocol/client.go +++ b/internal/server/protocol/client.go @@ -26,6 +26,8 @@ type AuthenticateHostMessage struct { ViewerPresence bool `json:"viewerPresence,omitempty"` ConnectionAttemptProgress4 bool `json:"connectionAttemptProgress4,omitempty"` DisplayName *DisplayName `json:"displayName,omitempty"` + RoomOnly bool `json:"roomOnly,omitempty"` + RoomSession bool `json:"roomSession,omitempty"` } // AuthenticateViewerMessage is the viewer member of authenticateMessageSchema. @@ -172,6 +174,15 @@ type StopSharingMessage struct { ShareGeneration string `json:"shareGeneration,omitempty"` } +// StartSharingMessage begins a new media publication on an existing Host room session. +type StartSharingMessage struct { + Type string `json:"type"` + ShareGeneration string `json:"shareGeneration"` + SharingPaused bool `json:"sharingPaused,omitempty"` + QualitySettings *QualitySettings `json:"qualitySettings,omitempty"` + RoutePolicy RoutePolicy `json:"routePolicy"` +} + // AbandonRoomMessage is { type: "abandon-room" }. type AbandonRoomMessage struct { Type string `json:"type"` @@ -197,6 +208,7 @@ func (ResetSenderQualityMessage) isClientMessage() {} func (SetDisplayNameMessage) isClientMessage() {} func (SetSharingPausedMessage) isClientMessage() {} func (StopSharingMessage) isClientMessage() {} +func (StartSharingMessage) isClientMessage() {} func (AbandonRoomMessage) isClientMessage() {} // DecodeClientMessage parses JSON and validates the message schema. @@ -207,6 +219,10 @@ func DecodeClientMessage(data []byte) (ClientMessage, error) { return nil, err } switch messageType { + case "subscribe-room-interactions": + return decodeEmptyClientMessage(data, SubscribeRoomInteractionsMessage{Type: messageType}) + case "send-room-interaction": + return decodeSendRoomInteraction(data) case "authenticate": return decodeAuthenticate(data) case "signaling-challenge": @@ -247,6 +263,8 @@ func DecodeClientMessage(data []byte) (ClientMessage, error) { return decodeSetSharingPaused(data) case "stop-sharing": return decodeStopSharing(data) + case "start-sharing": + return decodeStartSharing(data) case "abandon-room": return decodeEmptyClientMessage(data, AbandonRoomMessage{Type: messageType}) } @@ -282,8 +300,9 @@ func decodeAuthenticate(data []byte) (ClientMessage, error) { if err := present.require("type", "protocol", "roomId", "role", "token", "clientId"); err != nil { return nil, err } - if err := present.optional("shareGeneration", "sharingPaused", "qualitySettings", + if err := present.optional("shareGeneration", "roomOnly", "roomSession", "sharingPaused", "qualitySettings", "routePolicy", "viewerPresence", "displayName", "connectionAttemptProgress4"); err != nil { + return nil, err } if message.Protocol != SignalingProtocol { @@ -301,6 +320,15 @@ func decodeAuthenticate(data []byte) (ClientMessage, error) { if present.has("shareGeneration") && !ValidOpaqueID(message.ShareGeneration) { return nil, errors.New("shareGeneration is not an opaque id") } + if present.has("roomOnly") && !message.RoomOnly { + return nil, errors.New("roomOnly must be true when present") + } + if present.has("roomSession") && !message.RoomSession { + return nil, errors.New("roomSession must be true when present") + } + if message.RoomOnly && !message.RoomSession { + return nil, errors.New("roomOnly requires roomSession") + } if present.has("viewerPresence") && !message.ViewerPresence { return nil, errors.New("viewerPresence must be true when present") } @@ -705,3 +733,21 @@ func decodeStopSharing(data []byte) (ClientMessage, error) { } return message, nil } + +func decodeStartSharing(data []byte) (ClientMessage, error) { + var message StartSharingMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err := present.require("type", "shareGeneration", "routePolicy"); err != nil { + return nil, err + } + if err := present.optional("sharingPaused", "qualitySettings"); err != nil { + return nil, err + } + if !ValidOpaqueID(message.ShareGeneration) { + return nil, errors.New("shareGeneration is not an opaque id") + } + return message, nil +} diff --git a/internal/server/protocol/contract_test.go b/internal/server/protocol/contract_test.go index 4931b9e..a3d74d3 100644 --- a/internal/server/protocol/contract_test.go +++ b/internal/server/protocol/contract_test.go @@ -16,6 +16,7 @@ func TestRuntimeCapabilities(t *testing.T) { {`{}`, RuntimeCapabilities{}}, {`{"connectionAttemptProgress4":true}`, RuntimeCapabilities{ConnectionAttemptProgress4: true}}, {`{"sfu":true,"extra":{"enabled":true}}`, RuntimeCapabilities{Sfu: true}}, + {`{"sfu":true,"sfuOnly":true}`, RuntimeCapabilities{Sfu: true, SfuOnly: true}}, {`{"natPrediction":true,"SFU":true}`, RuntimeCapabilities{NatPrediction: true}}, {`{"sfu":true,"natPrediction":true}`, RuntimeCapabilities{Sfu: true, NatPrediction: true}}, {`{"sfu":"ignored duplicate","sfu":true}`, RuntimeCapabilities{Sfu: true}}, @@ -26,7 +27,7 @@ func TestRuntimeCapabilities(t *testing.T) { } } for _, data := range []string{ - `null`, `[]`, `true`, `{"sfu":null}`, `{"sfu":"true"}`, + `null`, `[]`, `true`, `{"sfu":null}`, `{"sfu":"true"}`, `{"sfuOnly":null}`, `{"sfuOnly":"true"}`, `{"natPrediction":null}`, `{"natPrediction":1}`, `{"connectionAttemptProgress4":null}`, `{"connectionAttemptProgress4":4}`, } { @@ -698,6 +699,44 @@ func TestAuthenticateAppliesTheRoutePolicyDefault(t *testing.T) { } } +func TestAuthenticateAcceptsPersistentRoomSessionOptIn(t *testing.T) { + message, err := DecodeClientMessage([]byte( + `{"type":"authenticate","protocol":"piik-v23","roomId":"1234","role":"host",` + + `"token":"` + repeat("a", 43) + `","clientId":"client_12345678","roomSession":true,"roomOnly":true}`)) + if err != nil { + t.Fatal(err) + } + host, ok := message.(AuthenticateHostMessage) + if !ok || !host.RoomSession || !host.RoomOnly { + t.Fatalf("decoded %#v, want room-session opt-in and room-only state", message) + } + if _, err := DecodeClientMessage([]byte( + `{"type":"authenticate","protocol":"piik-v23","roomId":"1234","role":"host",` + + `"token":"` + repeat("a", 43) + `","clientId":"client_12345678","roomOnly":true}`)); err == nil { + t.Fatal("accepted roomOnly without roomSession opt-in") + } +} + +func TestStartSharingRequiresAnOpaqueGenerationAndRoutePolicy(t *testing.T) { + message, err := DecodeClientMessage([]byte( + `{"type":"start-sharing","shareGeneration":"share_12345678",` + + `"routePolicy":{"peerOnly":true,"topologyOptimization":false,"natPrediction":false}}`)) + if err != nil { + t.Fatal(err) + } + if _, ok := message.(StartSharingMessage); !ok { + t.Fatalf("decoded %#v, want StartSharingMessage", message) + } + for _, raw := range []string{ + `{"type":"start-sharing","shareGeneration":"short","routePolicy":{"peerOnly":true,"topologyOptimization":false,"natPrediction":false}}`, + `{"type":"start-sharing","shareGeneration":"share_12345678"}`, + } { + if _, err := DecodeClientMessage([]byte(raw)); err == nil { + t.Errorf("accepted invalid start-sharing message %s", raw) + } + } +} + // The sealed unions accept nothing outside their own message structs. func TestDecodeRejectsUnknownEnvelopes(t *testing.T) { for _, raw := range []string{`{}`, `[]`, `null`, `"authenticate"`, `{"type":42}`, diff --git a/internal/server/protocol/fixture_test.go b/internal/server/protocol/fixture_test.go index f02d054..7171630 100644 --- a/internal/server/protocol/fixture_test.go +++ b/internal/server/protocol/fixture_test.go @@ -55,6 +55,8 @@ func TestFixtureConstants(t *testing.T) { "viewerQualityEvidenceExpiryMs": float64(ViewerQualityEvidenceExpiryMs), "persistentNativeEdgeDegradedWindows": float64(PersistentNativeEdgeDegradedWindows), "maxDisplayNameCodePoints": float64(MaxDisplayNameCodePoints), + "maxChatCodePoints": float64(MaxChatCodePoints), + "interactionIntervalMs": float64(InteractionIntervalMs), "defaultViewerDisplayName": DefaultViewerDisplayName, "defaultHostDisplayNamePrefix": DefaultHostDisplayNamePrefix, "minViewerPasswordLength": float64(MinViewerPasswordLength), diff --git a/internal/server/protocol/interactions.go b/internal/server/protocol/interactions.go new file mode 100644 index 0000000..51f987f --- /dev/null +++ b/internal/server/protocol/interactions.go @@ -0,0 +1,194 @@ +package protocol + +import ( + "errors" + "strings" + "unicode/utf8" + + "golang.org/x/text/unicode/norm" +) + +const MaxChatCodePoints = 280 +const InteractionIntervalMs = 800 + +type InteractionPayload struct { + Kind string `json:"kind"` + Text string `json:"text,omitempty"` + Reaction string `json:"reaction,omitempty"` + TargetPeerID string `json:"targetPeerId,omitempty"` +} + +func (payload *InteractionPayload) UnmarshalJSON(data []byte) error { + type plain InteractionPayload + var value plain + fields, err := decodeObject(data, &value) + if err != nil { + return err + } + if err = fields.require("kind"); err != nil { + return err + } + switch value.Kind { + case "chat": + if err = fields.require("text"); err != nil { + return err + } + if fields.has("reaction") || fields.has("targetPeerId") || + hasUnpairedSurrogateEscape(fields["text"]) || !validChatText(value.Text) { + return errors.New("invalid chat payload") + } + case "reaction": + if err = fields.require("reaction"); err != nil { + return err + } + if err = fields.optional("targetPeerId"); err != nil { + return err + } + if fields.has("text") || (fields.has("targetPeerId") && !ValidOpaqueID(value.TargetPeerID)) { + return errors.New("invalid reaction payload") + } + switch value.Reaction { + case "tomato", "poop": + if !ValidOpaqueID(value.TargetPeerID) { + return errors.New("throwing a prop requires a participant") + } + case "wave", "heart", "clap", "laugh", "wow", "party", "fire", "eyes", "star", "sleep": + default: + return errors.New("unknown reaction") + } + default: + return errors.New("unknown interaction kind") + } + *payload = InteractionPayload(value) + return nil +} + +func validChatText(value string) bool { + return utf8.ValidString(value) && utf8.RuneCountInString(value) >= 1 && + utf8.RuneCountInString(value) <= MaxChatCodePoints && !forbiddenDisplayNameCharacters.MatchString(value) && + norm.NFC.IsNormalString(value) && strings.TrimFunc(value, IsJSWhitespace) == value +} + +type SubscribeRoomInteractionsMessage struct { + Type string `json:"type"` +} +type SendRoomInteractionMessage struct { + Type string `json:"type"` + RequestID string `json:"requestId"` + Payload InteractionPayload `json:"payload"` +} + +func (SubscribeRoomInteractionsMessage) isClientMessage() {} +func (SendRoomInteractionMessage) isClientMessage() {} + +func decodeSendRoomInteraction(data []byte) (ClientMessage, error) { + var message SendRoomInteractionMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err = present.require("type", "requestId", "payload"); err != nil { + return nil, err + } + if !ValidOpaqueID(message.RequestID) { + return nil, errors.New("invalid interaction requestId") + } + return message, nil +} + +type InteractionSender struct { + PeerID string `json:"peerId"` + Role Role `json:"role"` + DisplayName DisplayName `json:"displayName"` +} + +func (sender *InteractionSender) UnmarshalJSON(data []byte) error { + type plain InteractionSender + var value plain + present, err := decodeObject(data, &value) + if err != nil { + return err + } + if err = present.require("peerId", "role", "displayName"); err != nil { + return err + } + if !ValidOpaqueID(value.PeerID) { + return errors.New("invalid sender peerId") + } + *sender = InteractionSender(value) + return nil +} + +type RoomInteractionsReadyMessage struct { + Type string `json:"type"` + ServerTime Int `json:"serverTime"` +} +type RoomInteractionMessage struct { + Type string `json:"type"` + ID string `json:"id"` + RequestID string `json:"requestId"` + OccurredAt Int `json:"occurredAt"` + Sender InteractionSender `json:"sender"` + Payload InteractionPayload `json:"payload"` +} +type RoomInteractionRejectedMessage struct { + Type string `json:"type"` + RequestID string `json:"requestId"` + Reason string `json:"reason"` +} + +func (RoomInteractionsReadyMessage) isServerMessage() {} +func (RoomInteractionMessage) isServerMessage() {} +func (RoomInteractionRejectedMessage) isServerMessage() {} + +func decodeRoomInteractionsReady(data []byte) (ServerMessage, error) { + var message RoomInteractionsReadyMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err = present.require("type", "serverTime"); err != nil { + return nil, err + } + if !inRangeInt(message.ServerTime, 0, MaxSafeInteger) { + return nil, errors.New("invalid interaction server time") + } + return message, nil +} + +func decodeRoomInteraction(data []byte) (ServerMessage, error) { + var message RoomInteractionMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err = present.require("type", "id", "requestId", "occurredAt", "sender", "payload"); err != nil { + return nil, err + } + if !ValidOpaqueID(message.ID) || !ValidOpaqueID(message.RequestID) { + return nil, errors.New("invalid interaction identity") + } + if !inRangeInt(message.OccurredAt, 0, MaxSafeInteger) { + return nil, errors.New("invalid interaction time") + } + return message, nil +} +func decodeRoomInteractionRejected(data []byte) (ServerMessage, error) { + var message RoomInteractionRejectedMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err = present.require("type", "requestId", "reason"); err != nil { + return nil, err + } + if !ValidOpaqueID(message.RequestID) { + return nil, errors.New("invalid interaction requestId") + } + switch message.Reason { + case "rate-limited", "target-unavailable", "not-subscribed", "busy": + default: + return nil, errors.New("invalid rejection reason") + } + return message, nil +} diff --git a/internal/server/protocol/server.go b/internal/server/protocol/server.go index f25aa0a..3e1f9d1 100644 --- a/internal/server/protocol/server.go +++ b/internal/server/protocol/server.go @@ -57,6 +57,23 @@ type AuthenticatedViewerMessage struct { Role Role `json:"role"` } +// SharingStartedMessage acknowledges a new media publication on an existing Host session. +type SharingStartedMessage struct { + Type string `json:"type"` + ShareGeneration string `json:"shareGeneration"` + RoutePolicy RoutePolicy `json:"routePolicy"` + RouteRevision Int `json:"routeRevision"` + RouteAssignment ParticipantRouteAssignment `json:"routeAssignment"` + QualitySettings QualitySettings `json:"qualitySettings"` + Paused bool `json:"paused"` +} + +type SharingStartFailedMessage struct { + Type string `json:"type"` + ShareGeneration string `json:"shareGeneration"` + Code string `json:"code"` +} + // SignalingChallengeResponseMessage is the challenge echo. type SignalingChallengeResponseMessage struct { Type string `json:"type"` @@ -197,6 +214,8 @@ type ErrorMessage struct { func (AuthenticatedHostMessage) isServerMessage() {} func (AuthenticatedViewerMessage) isServerMessage() {} +func (SharingStartedMessage) isServerMessage() {} +func (SharingStartFailedMessage) isServerMessage() {} func (SignalingChallengeResponseMessage) isServerMessage() {} func (ServerSignalMessage) isServerMessage() {} func (ServerRestartRequestMessage) isServerMessage() {} @@ -230,8 +249,18 @@ func DecodeServerMessage(data []byte) (ServerMessage, error) { return nil, err } switch messageType { + case "room-interactions-ready": + return decodeRoomInteractionsReady(data) + case "room-interaction": + return decodeRoomInteraction(data) + case "room-interaction-rejected": + return decodeRoomInteractionRejected(data) case "authenticated": return decodeAuthenticated(data) + case "sharing-started": + return decodeSharingStarted(data) + case "sharing-start-failed": + return decodeSharingStartFailed(data) case "signaling-challenge-response": return decodeChallengeResponse(data) case "signal": @@ -386,6 +415,40 @@ func decodeAuthenticated(data []byte) (ServerMessage, error) { return nil, fmt.Errorf("unknown authenticated role %q", role) } +func decodeSharingStarted(data []byte) (ServerMessage, error) { + var message SharingStartedMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err := present.require("type", "shareGeneration", "routePolicy", "routeRevision", + "routeAssignment", "qualitySettings", "paused"); err != nil { + return nil, err + } + if !ValidOpaqueID(message.ShareGeneration) { + return nil, errors.New("shareGeneration is not an opaque id") + } + if !validRevision(message.RouteRevision) { + return nil, errors.New("routeRevision is out of range") + } + return message, nil +} + +func decodeSharingStartFailed(data []byte) (ServerMessage, error) { + var message SharingStartFailedMessage + present, err := decodeObject(data, &message) + if err != nil { + return nil, err + } + if err := present.require("type", "shareGeneration", "code"); err != nil { + return nil, err + } + if !ValidOpaqueID(message.ShareGeneration) || (message.Code != "FORBIDDEN" && message.Code != "SERVER_ERROR") { + return nil, errors.New("invalid sharing-start-failed response") + } + return message, nil +} + func decodeChallengeResponse(data []byte) (ServerMessage, error) { var message SignalingChallengeResponseMessage present, err := decodeObject(data, &message) diff --git a/internal/server/protocol/types.go b/internal/server/protocol/types.go index 4365aed..4bfd186 100644 --- a/internal/server/protocol/types.go +++ b/internal/server/protocol/types.go @@ -170,7 +170,10 @@ func (v *RoutePolicy) UnmarshalJSON(data []byte) error { type RuntimeCapabilities struct { ConnectionAttemptProgress4 bool `json:"connectionAttemptProgress4,omitempty"` Sfu bool `json:"sfu"` + SfuOnly bool `json:"sfuOnly,omitempty"` NatPrediction bool `json:"natPrediction"` + RoomInteractions bool `json:"roomInteractions,omitempty"` + HostRoomSession bool `json:"hostRoomSession,omitempty"` } // UnmarshalJSON implements json.Unmarshaler. @@ -183,12 +186,14 @@ func (v *RuntimeCapabilities) UnmarshalJSON(data []byte) error { if present == nil { return errors.New("runtime capabilities must be an object") } - if err := present.optional("sfu", "natPrediction", "connectionAttemptProgress4"); err != nil { + if err := present.optional("sfu", "sfuOnly", "roomInteractions", "hostRoomSession", "natPrediction", "connectionAttemptProgress4"); err != nil { + return err } for key, target := range map[string]*bool{ - "sfu": &v.Sfu, "natPrediction": &v.NatPrediction, + "sfu": &v.Sfu, "sfuOnly": &v.SfuOnly, "natPrediction": &v.NatPrediction, "connectionAttemptProgress4": &v.ConnectionAttemptProgress4, + "roomInteractions": &v.RoomInteractions, "hostRoomSession": &v.HostRoomSession, } { if value, ok := present[key]; ok { if err := json.Unmarshal(value, target); err != nil { diff --git a/internal/server/route/api.go b/internal/server/route/api.go index eaf052e..9c41d2d 100644 --- a/internal/server/route/api.go +++ b/internal/server/route/api.go @@ -455,6 +455,7 @@ type Options struct { EndpointMediaCopyCapacity int OperationTimeoutMs int64 SfuEnabled bool + SfuOnly bool QualityConvergenceEnabled bool NatPredictionEnabled bool } @@ -470,6 +471,7 @@ type Controller struct { endpointMediaCopyCapacity int operationTimeoutMs int64 sfuEnabled bool + sfuOnly bool qualityConvergenceEnabled bool natPredictionEnabled bool diff --git a/internal/server/route/candidates.go b/internal/server/route/candidates.go index 6b1cd35..931e3d8 100644 --- a/internal/server/route/candidates.go +++ b/internal/server/route/candidates.go @@ -70,7 +70,7 @@ func (c *Controller) buildCandidatePlans(childPeerID string, sfuOnly, preferUntr } var candidates []CandidateTuple switch { - case sfuOnly: + case sfuOnly || c.sfuOnly: if sfuCandidate != nil { candidates = []CandidateTuple{*sfuCandidate} } @@ -166,6 +166,9 @@ func (c *Controller) candidateValid(childPeerID string, plan CandidatePlan, att return false } if tuple.Kind == UpstreamPeer { + if c.sfuOnly { + return false + } parent, _ := c.participants.Get(tuple.ParentPeerID) if parent == nil || parent.sessionID == "" || parent.departureConfirmed || (att != nil && parent.sessionID != att.parentSessionID) || slices.Contains(c.descendantsOf(childPeerID), parent.peerID) || !c.sourceUsable(parent.peerID) || diff --git a/internal/server/route/controller.go b/internal/server/route/controller.go index bde5567..6463d73 100644 --- a/internal/server/route/controller.go +++ b/internal/server/route/controller.go @@ -271,6 +271,9 @@ func New(options Options) *Controller { if !isSafeInteger(options.OperationTimeoutMs) || options.OperationTimeoutMs <= 0 { panic(errors.New("Route operation timeout must be a positive integer")) } + if options.SfuOnly && !options.SfuEnabled { + panic(errors.New("SFU-only routing requires SFU availability")) + } c := &Controller{ hostPeerID: options.HostPeerID, debugRoomID: options.DebugRoomID, @@ -278,8 +281,9 @@ func New(options Options) *Controller { endpointMediaCopyCapacity: options.EndpointMediaCopyCapacity, operationTimeoutMs: options.OperationTimeoutMs, sfuEnabled: options.SfuEnabled, - qualityConvergenceEnabled: options.QualityConvergenceEnabled, - natPredictionEnabled: options.NatPredictionEnabled, + sfuOnly: options.SfuOnly, + qualityConvergenceEnabled: options.QualityConvergenceEnabled && !options.SfuOnly, + natPredictionEnabled: options.NatPredictionEnabled && !options.SfuOnly, routeTimings: make(map[string]*routeTimingRecord), qualityObservations: make(map[string]*routeQualityObservation), senderQualityObservations: make(map[string]*senderQualityObservation), @@ -291,6 +295,7 @@ func New(options Options) *Controller { "endpointCapacity", options.EndpointMediaCopyCapacity, "operationTimeoutMs", options.OperationTimeoutMs, "sfuEnabled", options.SfuEnabled, + "sfuOnly", options.SfuOnly, "qualityConvergenceEnabled", options.QualityConvergenceEnabled) return c } diff --git a/internal/server/route/sfu_only_test.go b/internal/server/route/sfu_only_test.go new file mode 100644 index 0000000..aafe397 --- /dev/null +++ b/internal/server/route/sfu_only_test.go @@ -0,0 +1,64 @@ +package route + +import "testing" + +func TestSFUOnlyUsesOnePublicationWithoutPeerConvergence(t *testing.T) { + routes := newController(1, Options{SfuEnabled: true, SfuOnly: true, + QualityConvergenceEnabled: true, NatPredictionEnabled: true}) + for index, viewer := range []string{A, B, C} { + now := int64(index * 10) + addViewer(routes, viewer, 1, &now) + op := must(t, routes.Reconcile(now).Operation) + eq(t, len(op.Candidates), 1) + publication := PublicationReuse + input := BeginInput{NowMs: now + 1, ConnectionID: viewer + "_sfu", + Reservation: sfuReuse(viewer)} + if index == 0 { + publication = PublicationCreate + input.PublicationGeneration = "publication" + input.PublicationConnectionID = "publication_connection" + input.Reservation = sfuCreate(viewer, "publication_resource") + } + eq(t, op.Candidates[0].Tuple, sfuTuple(publication)) + prepared := beginOperation(t, routes, input) + // Availability still commits only on the exact child's media-ready proof. + eq(t, hasEdge(routes, viewer), false) + eq(t, routes.CandidateReady(guardFor(viewer, prepared.Current.Revision, input.ConnectionID), + now+2, nil, CandidateProof{}).Accepted, true) + eq(t, edgeOf(t, routes, viewer).Kind, UpstreamSfu) + noOperation(t, routes.Reconcile(now+3).Operation) + } + eq(t, routes.physicalCopies(HOST), 1) + eq(t, routes.directContinuations.Len(), 0) + // A failed publication is replaced through the same bounded operation. + publication := routes.Publication() + eq(t, routes.InvalidateHostPublication(PublicationGuard{HostSessionID: "host_session", + RouteRevision: routes.Revision(), Generation: publication.Generation, + ConnectionID: publication.ConnectionID}, ms(40)), true) + op := must(t, routes.Reconcile(41).Operation) + eq(t, len(op.Candidates), 1) + eq(t, op.Candidates[0].Tuple, sfuTuple(PublicationReplace)) +} + +func TestSFUOnlyFailureDoesNotEscapeToPeers(t *testing.T) { + for _, admissionDenied := range []bool{false, true} { + routes := newController(2, Options{SfuEnabled: true, SfuOnly: true}) + addViewer(routes, A, 2, nil) + op := must(t, routes.Reconcile(0).Operation) + if admissionDenied { + routes.SkipCurrentCandidate(cursorGuard(op), 1, RejectionSfuAdmission) + } else { + prepared := beginOperation(t, routes, BeginInput{NowMs: 1, + ConnectionID: "sfu_connection", PublicationGeneration: "publication", + PublicationConnectionID: "publication_connection", Reservation: sfuCreate("edge", "publication")}) + routes.CandidateFailed(guardFor(A, prepared.Current.Revision, "sfu_connection"), 2) + } + noOperation(t, routes.Reconcile(3).Operation) + eq(t, hasEdge(routes, A), false) + // Resource release may retry SFU; it must not reopen a Peer opportunity. + routes.TouchExternalFacts() + retry := must(t, routes.Reconcile(4).Operation) + eq(t, len(retry.Candidates), 1) + eq(t, retry.Candidates[0].Tuple.Kind, UpstreamSfu) + } +} diff --git a/internal/server/signal/interactions.go b/internal/server/signal/interactions.go new file mode 100644 index 0000000..f3aac14 --- /dev/null +++ b/internal/server/signal/interactions.go @@ -0,0 +1,62 @@ +package signal + +import "github.com/TNTcraftHIM/Piik/internal/server/protocol" + +// Room authority remains in the authenticated session/store. Interaction state +// ends with that connection; no message history, routing state or retry worker +// lives on the server. Server.mu serializes membership checks and broadcast. +func (s *Server) handleRoomInteraction(sess *session, actor *authenticatedSession, message protocol.SendRoomInteractionMessage) { + reject := func(reason string) { + s.send(sess, protocol.RoomInteractionRejectedMessage{ + Type: "room-interaction-rejected", RequestID: message.RequestID, Reason: reason, + }) + } + if !sess.roomInteractions || actor.displayName == nil { + reject("not-subscribed") + return + } + now := s.now() + if now < sess.nextInteractionAtMs { + reject("rate-limited") + return + } + sess.nextInteractionAtMs = now + protocol.InteractionIntervalMs + if targetID := message.Payload.TargetPeerID; targetID != "" { + target, exists := s.connectedPeer(actor.roomID, targetID) + targetSession := s.sessionsByID[target.SessionID] + if !exists || targetSession == nil || targetSession.revoked || !targetSession.open() || !s.isCurrentSession(targetSession) { + reject("target-unavailable") + return + } + } + // Reserve the existing signaling queue for media/authority work when a + // reader stalls. Room data must not fill it and terminate a healthy share. + const maxInteractionQueueBytes = 16 * 1024 + if sess.queuedBytes >= maxInteractionQueueBytes { + reject("busy") + return + } + event := protocol.RoomInteractionMessage{ + Type: "room-interaction", ID: opaqueID(), RequestID: message.RequestID, + OccurredAt: protocol.Int(now), + Sender: protocol.InteractionSender{PeerID: actor.peerID, Role: actor.role, DisplayName: protocol.DisplayName(*actor.displayName)}, + Payload: message.Payload, + } + encoded, err := protocol.EncodeServerMessage(event) + if err != nil { + panic(err) + } + deliver := func(sessionID string) { + target := s.sessionsByID[sessionID] + if target != nil && target.roomInteractions && !target.revoked && target.open() && + s.isCurrentSession(target) && target.queuedBytes < maxInteractionQueueBytes { + s.sendEncoded(target, encoded) + } + } + if host, ok := s.store.GetConnectedHost(actor.roomID); ok { + deliver(host.SessionID) + } + for _, viewer := range s.store.GetConnectedViewers(actor.roomID) { + deliver(viewer.SessionID) + } +} diff --git a/internal/server/signal/interactions_test.go b/internal/server/signal/interactions_test.go new file mode 100644 index 0000000..48832f9 --- /dev/null +++ b/internal/server/signal/interactions_test.go @@ -0,0 +1,144 @@ +package signal + +import ( + "sync/atomic" + "testing" + "time" + + "github.com/TNTcraftHIM/Piik/internal/server/protocol" +) + +func subscribeInteractions(t *testing.T, client *testClient) { + t.Helper() + client.sendJSON(map[string]any{"type": "subscribe-room-interactions"}) + ready := client.next("room-interactions-ready") + if _, err := protocol.DecodeServerMessage(ready.raw); err != nil { + t.Fatal(err) + } +} + +func TestRoomInteractionsStayInsideAuthorizedOptedInMembership(t *testing.T) { + var now atomic.Int64 + now.Store(100_000) + h := startHarness(t, harnessOptions{maxViewersPerRoom: 4, now: now.Load}) + host, viewer, old := openClient(t, h), openClient(t, h), openClient(t, h) + authenticate(t, host, h.room, protocol.RoleHost, "host_interaction", -1, "", presenceOptions{displayName: "房主", viewerPresence: true}) + actor := authenticate(t, viewer, h.room, protocol.RoleViewer, "viewer_interaction", -1, "", presenceOptions{displayName: "小明", viewerPresence: true}) + authenticate(t, old, h.room, protocol.RoleViewer, "old_page_interaction", -1, "", presenceOptions{}) + otherRoom, err := h.server.CreateRoom(protocol.CodeEntryOpen, nil, "") + if err != nil { + t.Fatal(err) + } + outsider := openClient(t, h) + other := authenticate(t, outsider, otherRoom, protocol.RoleViewer, "outsider_interaction", -1, "", presenceOptions{displayName: "Elsewhere", viewerPresence: true}) + for _, client := range []*testClient{host, viewer, outsider} { + subscribeInteractions(t, client) + } + viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_1", "payload": map[string]any{"kind": "chat", "text": "你好 👋"}}) + hostEvent := host.next("room-interaction") + viewerEvent := viewer.next("room-interaction") + if string(hostEvent.raw) != string(viewerEvent.raw) { + t.Fatal("recipients received different events") + } + decoded, err := protocol.DecodeServerMessage(hostEvent.raw) + if err != nil { + t.Fatal(err) + } + event := decoded.(protocol.RoomInteractionMessage) + if event.OccurredAt != protocol.Int(now.Load()) { + t.Fatal("event time was not stamped at server acceptance") + } + if event.Sender.PeerID != actor.PeerID || event.Sender.DisplayName != "小明" { + t.Fatal("sender identity was not derived from authentication") + } + for _, client := range []*testClient{old, outsider} { + if _, ok := client.tryNext("room-interaction", 30*time.Millisecond); ok { + t.Fatal("interaction escaped its negotiated room boundary") + } + } + viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_2", "payload": map[string]any{"kind": "reaction", "reaction": "wave"}}) + expectMatch(t, viewer.next("room-interaction-rejected").raw, `{"reason":"rate-limited"}`) + now.Add(protocol.InteractionIntervalMs) + viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_3", "payload": map[string]any{"kind": "reaction", "reaction": "tomato", "targetPeerId": other.PeerID}}) + expectMatch(t, viewer.next("room-interaction-rejected").raw, `{"reason":"target-unavailable"}`) + // Room data and its rejections never terminate signaling or require media. + viewer.sendJSON(map[string]any{"type": "signaling-challenge", "sequence": 123}) + viewer.next("signaling-challenge-response") +} + +func TestRoomInteractionRequiresAuthenticationAndSubscription(t *testing.T) { + h := startHarness(t, harnessOptions{}) + guest := openClient(t, h) + guest.sendJSON(map[string]any{"type": "subscribe-room-interactions"}) + expectMatch(t, guest.next("error").raw, `{"code":"AUTH_REQUIRED"}`) + viewer := openClient(t, h) + authenticate(t, viewer, h.room, protocol.RoleViewer, "unsubscribed_viewer", -1, "", presenceOptions{displayName: "Friend", viewerPresence: true}) + viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_1", "payload": map[string]any{"kind": "chat", "text": "hello"}}) + expectMatch(t, viewer.next("room-interaction-rejected").raw, `{"reason":"not-subscribed"}`) +} + +func TestViewersCanInteractAfterHostDisconnects(t *testing.T) { + var now atomic.Int64 + now.Store(100_000) + h := startHarness(t, harnessOptions{now: now.Load}) + host, first, second := openClient(t, h), openClient(t, h), openClient(t, h) + hostActor := authenticate(t, host, h.room, protocol.RoleHost, "leaving_host", -1, "", + presenceOptions{displayName: "Host", roomSession: true}) + authenticate(t, first, h.room, protocol.RoleViewer, "remaining_first", -1, "", presenceOptions{displayName: "First"}) + secondActor := authenticate(t, second, h.room, protocol.RoleViewer, "remaining_second", -1, "", presenceOptions{displayName: "Second"}) + subscribeInteractions(t, first) + subscribeInteractions(t, second) + h.closeClient(host) + for _, payload := range []map[string]any{ + {"kind": "chat", "text": "still here"}, + {"kind": "reaction", "reaction": "wave"}, + {"kind": "reaction", "reaction": "tomato", "targetPeerId": secondActor.PeerID}, + } { + first.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "after_host_left", "payload": payload}) + if a, b := first.next("room-interaction"), second.next("room-interaction"); string(a.raw) != string(b.raw) { + t.Fatal("remaining viewers did not receive the same interaction") + } + now.Add(protocol.InteractionIntervalMs) + } + first.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "departed_target", + "payload": map[string]any{"kind": "reaction", "reaction": "tomato", "targetPeerId": hostActor.PeerID}}) + expectMatch(t, first.next("room-interaction-rejected").raw, `{"reason":"target-unavailable"}`) +} + +func TestRoomInteractionBackpressureDoesNotRetireMediaSignaling(t *testing.T) { + h := startHarness(t, harnessOptions{}) + host, viewer := openClient(t, h), openClient(t, h) + authenticate(t, host, h.room, protocol.RoleHost, "data_host_1234", -1, "", presenceOptions{displayName: "Host", viewerPresence: true}) + actor := authenticate(t, viewer, h.room, protocol.RoleViewer, "slow_viewer_1234", -1, "", presenceOptions{displayName: "Viewer", viewerPresence: true}) + subscribeInteractions(t, host) + subscribeInteractions(t, viewer) + h.locked(func() { + peer, ok := h.server.store.GetConnectedViewer(h.room.RoomID, actor.PeerID) + if !ok { + t.Fatal("missing viewer") + } + slow := h.server.sessionsByID[peer.SessionID] + // Inject a stalled writer under the same lock as a real broadcast, so + // the test does not depend on OS TCP buffer sizes or the reader's timing. + previous := slow.queuedBytes + slow.queuedBytes = 20 * 1024 + hostPeer, _ := h.server.store.GetConnectedHost(h.room.RoomID) + hostSession := h.server.sessionsByID[hostPeer.SessionID] + h.server.handleRoomInteraction(hostSession, hostSession.authenticated, protocol.SendRoomInteractionMessage{ + Type: "send-room-interaction", RequestID: "busy_request_1234", Payload: protocol.InteractionPayload{Kind: "chat", Text: "hello"}, + }) + if slow.terminated || slow.closeQueued { + t.Error("room data closed the signaling session") + } + if slow.queuedBytes != 20*1024 { + t.Error("room data entered a stalled signaling queue") + } + slow.queuedBytes = previous + }) + host.next("room-interaction") + if _, ok := viewer.tryNext("room-interaction", 30*time.Millisecond); ok { + t.Fatal("stalled recipient got room data") + } + viewer.sendJSON(map[string]any{"type": "signaling-challenge", "sequence": 456}) + viewer.next("signaling-challenge-response") +} diff --git a/internal/server/signal/router.go b/internal/server/signal/router.go index 60c6780..f6cda6b 100644 --- a/internal/server/signal/router.go +++ b/internal/server/signal/router.go @@ -75,6 +75,7 @@ type routerOptions struct { store *room.Store endpointMediaCopyCapacity int sfu *sfuFallback + sfuOnly bool hooks routerHooks now func() int64 afterFunc func(time.Duration, func()) func() bool @@ -160,6 +161,7 @@ type router struct { store *room.Store capacity int sfu *sfuFallback + sfuOnly bool hooks routerHooks now func() int64 afterFunc func(time.Duration, func()) func() bool @@ -187,6 +189,7 @@ func newRouter(options routerOptions) *router { store: options.store, capacity: options.endpointMediaCopyCapacity, sfu: options.sfu, + sfuOnly: options.sfuOnly, hooks: options.hooks, now: options.now, afterFunc: options.afterFunc, @@ -995,6 +998,7 @@ func (r *router) createController(roomID string, rm *roomRuntime, host authentic EndpointMediaCopyCapacity: r.capacity, OperationTimeoutMs: operationTimeoutMs, SfuEnabled: r.sfu != nil && !(routePolicy != nil && routePolicy.PeerOnly), + SfuOnly: r.sfuOnly, QualityConvergenceEnabled: routePolicy != nil && routePolicy.TopologyOptimization, NatPredictionEnabled: routePolicy != nil && routePolicy.NatPrediction, }) diff --git a/internal/server/signal/server.go b/internal/server/signal/server.go index bbf7561..4a351a0 100644 --- a/internal/server/signal/server.go +++ b/internal/server/signal/server.go @@ -72,6 +72,8 @@ type Options struct { EndpointMediaCopyCapacity int // SfuFallback is nil when embedded SFU media is disabled. SfuFallback *SfuFallback + // SfuOnly restricts the existing controller to server media candidates. + SfuOnly bool // Ice is the derived wire ICE configuration (config.IceConfig). Ice protocol.IceConfig // NATPredictionEnabled is the server capability that clamps a Host's @@ -215,6 +217,9 @@ func New(options Options) (*Server, error) { if options.SfuFallback != nil && (options.SfuFallback.Media == nil || options.SfuFallback.Admission == nil) { return nil, errors.New("SFU fallback requires media and resource admission") } + if options.SfuOnly && options.SfuFallback == nil { + return nil, errors.New("SFU-only routing requires SFU media and resource admission") + } s := &Server{ store: options.Store, endpointMediaCopyCapacity: options.EndpointMediaCopyCapacity, @@ -271,6 +276,7 @@ func New(options Options) (*Server, error) { store: s.store, endpointMediaCopyCapacity: s.endpointMediaCopyCapacity, sfu: fallback, + sfuOnly: options.SfuOnly, hooks: routerHooks{ sendToSession: s.sendToSession, shareGeneration: func(roomID string) string { return s.shares[roomID].generation }, @@ -713,6 +719,8 @@ type authRequest struct { viewerPresence bool connectionAttemptProgress4 bool displayName *string + roomOnly bool + roomSession bool } func authRequestOf(message protocol.ClientMessage) (authRequest, bool) { @@ -730,6 +738,8 @@ func authRequestOf(message protocol.ClientMessage) (authRequest, bool) { viewerPresence: m.ViewerPresence, connectionAttemptProgress4: m.ConnectionAttemptProgress4, displayName: displayNameString(m.DisplayName), + roomOnly: m.RoomOnly, + roomSession: m.RoomSession, }, true case protocol.AuthenticateViewerMessage: return authRequest{ @@ -758,6 +768,14 @@ func displayNameString(name *protocol.DisplayName) *string { // section. Password checks run in a goroutine so the reader can keep reading // while the KDF runs. func (s *Server) authenticate(sess *session, request authRequest) { + // Reject the conflicting preference before acquiring room or media authority, + // including reconnects from older pages. Never turn peer-only into SFU consent. + if request.role == protocol.RoleHost && !request.roomOnly && s.router.sfuOnly && request.routePolicy.PeerOnly { + s.sendError(sess, "FORBIDDEN", "This site requires server media; turn off Privacy mode or use another site") + sess.close(websocket.StatusCode(protocol.SignalCloseAuthenticationFailed), "Route policy conflict") + s.finishAuthenticating(sess) + return + } if request.role == protocol.RoleHost && !sess.siteAccessAuthenticated { s.sendError(sess, "AUTH_REQUIRED", "Site access is required") sess.close(websocket.StatusCode(protocol.SignalCloseAuthenticationFailed), "Authentication failed") @@ -889,28 +907,32 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part displayName: displayName, viewerPresence: request.viewerPresence, connectionAttemptProgress4: request.connectionAttemptProgress4, + roomSession: request.roomSession, } sess.authenticated = authenticated if participant.Role == protocol.RoleViewer { s.clearViewerEvidence(roomID, participant.PeerID) } s.clearViewerGrace(roomID, participant.PeerID) - routeParticipant := authenticatedRouteParticipant{ - roomID: roomID, - role: participant.Role, - peerID: participant.PeerID, - sessionID: sess.sessionID, + mediaActive := participant.Role == protocol.RoleHost && shareGeneration != "" + if participant.Role == protocol.RoleViewer { + mediaActive = s.activeHostShareGeneration(roomID) != "" } - if participant.Role == protocol.RoleHost { + routeParticipant := routeParticipant(sess, authenticated) + if participant.Role == protocol.RoleHost && mediaActive { policy := s.routePolicyOf(roomID) routeParticipant.routePolicy = &policy } - hybridState, assigned := s.connectRouteParticipant(routeParticipant) - if !assigned { - s.logger.Error("Media router did not assign an authenticated participant") - s.sendError(sess, "SERVER_ERROR", "Media assignment failed") - sess.close(websocket.StatusInternalError, "Media assignment failed") - return + hybridState := hybridAuthenticationState{routeAssignment: emptyAssignment()} + if participant.Role == protocol.RoleViewer || mediaActive { + var assigned bool + hybridState, assigned = s.connectRouteParticipant(routeParticipant) + if !assigned { + s.logger.Error("Media router did not assign an authenticated participant") + s.sendError(sess, "SERVER_ERROR", "Media assignment failed") + sess.close(websocket.StatusInternalError, "Media assignment failed") + return + } } // The viewer's media identity is derived from the committed edge, never // stored beside it; connectParticipant has just settled the graph. @@ -934,8 +956,11 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part qualitySettings = *share.qualitySettings } var roomShareGeneration *string - if share.generation != "" { - generation := share.generation + if participant.Role == protocol.RoleHost && mediaActive { + generation := shareGeneration + roomShareGeneration = &generation + } else if participant.Role == protocol.RoleViewer && mediaActive { + generation := s.activeHostShareGeneration(roomID) roomShareGeneration = &generation } if participant.Role == protocol.RoleHost { @@ -960,15 +985,15 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part ViewerPasswordEnabled: participant.ViewerPasswordEnabled, }) } else { - hostPaused := participant.HostOnline && - share.pausedGeneration == share.generation + hostOnline := participant.HostOnline && mediaActive + hostPaused := hostOnline && share.pausedGeneration == share.generation s.send(sess, protocol.AuthenticatedViewerMessage{ Type: "authenticated", Protocol: protocol.SignalingProtocol, PeerID: participant.PeerID, MaxViewers: protocol.Int(s.store.MaxViewersPerRoom()), EndpointMediaCopyCapacity: protocol.Int(s.endpointMediaCopyCapacity), - HostOnline: participant.HostOnline, + HostOnline: hostOnline, HostPaused: &hostPaused, ConnectionID: connectionID, IceConfig: s.ice, @@ -989,12 +1014,14 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part } } - if participant.Role == protocol.RoleHost { + if participant.Role == protocol.RoleHost && mediaActive { s.router.setPaused(roomID, s.shares[roomID].pausedGeneration == shareGeneration) } - s.router.completeAuthentication(routeParticipant, hybridState) + if participant.Role == protocol.RoleViewer || mediaActive { + s.router.completeAuthentication(routeParticipant, hybridState) + } - if participant.Role == protocol.RoleHost && shareGeneration != "" && + if participant.Role == protocol.RoleHost && mediaActive && s.shares[roomID].pausedGeneration == shareGeneration { s.sendToSession(sess.sessionID, protocol.PauseSharingSourceMessage{ Type: "pause-sharing-source", @@ -1004,7 +1031,7 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part s.sendViewerPresence(roomID) - if participant.Role == protocol.RoleHost { + if participant.Role == protocol.RoleHost && mediaActive { for _, viewer := range connectedViewers { s.sendToSession(viewer.SessionID, protocol.HostStatusMessage{ Type: "host-status", @@ -1021,6 +1048,15 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part // with. It returns the settled generation and must run with mu held, before the // `authenticated` message is built from the record it writes. func (s *Server) settleHostShare(roomID, sessionID string, request authRequest) string { + if request.roomOnly { + // A room-only reconnect must not resurrect a stale publication. The + // stop path normally already cleared it; this guard covers a socket + // loss between the client cleanup and the server's stop message. + if s.shares[roomID].generation != "" { + s.stopSharing(roomID) + } + return "" + } current := s.shares[roomID] currentGeneration := current.generation shareGeneration := request.shareGeneration @@ -1111,10 +1147,33 @@ func (s *Server) routePolicyOf(roomID string) protocol.RoutePolicy { return s.configuredRoutePolicy(protocol.DefaultRoutePolicy) } +// activeHostShareGeneration is the media truth for a room. The room share +// record intentionally keeps its generation as a stale reconnect fence after +// stopSharing, so membership or that fence alone cannot mean "publishing". +// Callers hold Server.mu. +func (s *Server) activeHostShareGeneration(roomID string) string { + host, ok := s.store.GetConnectedHost(roomID) + if !ok { + return "" + } + hostSession := s.sessionsByID[host.SessionID] + if hostSession == nil || hostSession.authenticated == nil || + hostSession.authenticated.role != protocol.RoleHost || + hostSession.authenticated.roomID != roomID { + return "" + } + generation := hostSession.authenticated.shareGeneration + if generation == "" || s.shares[roomID].generation != generation { + return "" + } + return generation +} + // configuredRoutePolicy restricts share preferences to the services this runtime owns. func (s *Server) configuredRoutePolicy(policy protocol.RoutePolicy) protocol.RoutePolicy { policy.PeerOnly = s.router.sfu == nil || policy.PeerOnly - policy.NatPrediction = s.natPredictionEnabled && policy.NatPrediction + policy.NatPrediction = s.natPredictionEnabled && !s.router.sfuOnly && policy.NatPrediction + policy.TopologyOptimization = !s.router.sfuOnly && policy.TopologyOptimization return policy } @@ -1131,6 +1190,12 @@ func routeParticipant(sess *session, authenticated *authenticatedSession) authen func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authenticatedSession, message protocol.ClientMessage) { switch m := message.(type) { + case protocol.SubscribeRoomInteractionsMessage: + // Explicit opt-in protects older v23 readers from unknown events. + sess.roomInteractions = true + s.send(sess, protocol.RoomInteractionsReadyMessage{Type: "room-interactions-ready", ServerTime: protocol.Int(s.now())}) + case protocol.SendRoomInteractionMessage: + s.handleRoomInteraction(sess, authenticated, m) case protocol.SignalingChallengeMessage: if !s.sessions.Has(sess) || sess.authenticated != authenticated { return @@ -1165,6 +1230,10 @@ func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authen s.sendError(sess, "FORBIDDEN", "Only the host may set quality settings") return } + if authenticated.shareGeneration == "" || + s.shares[authenticated.roomID].generation != authenticated.shareGeneration { + return + } share := s.shares[authenticated.roomID] share.qualitySettings = &m.QualitySettings s.shares[authenticated.roomID] = share @@ -1246,28 +1315,39 @@ func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authen s.shares[authenticated.roomID] = share s.router.setPaused(authenticated.roomID, m.Paused) s.broadcastHostStatus(authenticated.roomID, true, m.Paused) + case protocol.StartSharingMessage: + s.startSharing(sess, authenticated, m) case protocol.StopSharingMessage: if authenticated.role != protocol.RoleHost { s.sendError(sess, "FORBIDDEN", "只有当前房主可以停止分享") return } - if authenticated.shareGeneration == "" || - s.shares[authenticated.roomID].generation != authenticated.shareGeneration || + if authenticated.shareGeneration == "" { + return + } + if s.shares[authenticated.roomID].generation != authenticated.shareGeneration || (m.ShareGeneration != "" && m.ShareGeneration != authenticated.shareGeneration) { sess.close(websocket.StatusCode(protocol.SignalCloseSessionReplaced), "Sharing generation replaced") return } - // The participant leaves the store before stopSharing() - // so the sharing-stopped / host-status pair is what viewers see. - if s.sessions.Has(sess) && sess.authenticated == authenticated { - if _, err := s.store.DisconnectParticipant(authenticated.roomID, authenticated.peerID, sess.sessionID); err != nil { - // The TS threw out of the message handler; nothing caught it. - panic(fmt.Errorf("room store failed while stopping sharing: %w", err)) + if !authenticated.roomSession { + // Older clients treat stopping media as leaving the Host session. + // Keep that contract until the client explicitly opts into the + // persistent room-session lifecycle. + if s.sessions.Has(sess) && sess.authenticated == authenticated { + if _, err := s.store.DisconnectParticipant(authenticated.roomID, authenticated.peerID, sess.sessionID); err != nil { + panic(fmt.Errorf("room store failed while stopping sharing: %w", err)) + } } + s.stopSharing(authenticated.roomID) + sess.close(websocket.StatusNormalClosure, "Sharing stopped") + return } s.stopSharing(authenticated.roomID) + // Room membership and its interaction stream outlive the media + // publication. The next start-sharing message claims a fresh generation. + authenticated.shareGeneration = "" s.sendViewerPresence(authenticated.roomID) - sess.close(websocket.StatusNormalClosure, "Sharing stopped") case protocol.AbandonRoomMessage: if authenticated.role != protocol.RoleHost { s.sendError(sess, "FORBIDDEN", "Only the host may abandon the room") @@ -1280,6 +1360,76 @@ func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authen } } +// startSharing claims a new publication generation without replacing the +// authenticated Host participant. Room interactions therefore keep the same +// session while the media graph is rebuilt for the new source. +func (s *Server) startSharing(sess *session, authenticated *authenticatedSession, message protocol.StartSharingMessage) { + if authenticated.role != protocol.RoleHost { + s.sendError(sess, "FORBIDDEN", "Only the host may start sharing") + return + } + if !authenticated.roomSession { + s.sendError(sess, "FORBIDDEN", "Persistent room session is not enabled") + return + } + if !s.sessions.Has(sess) || sess.authenticated != authenticated || + authenticated.shareGeneration != "" || message.ShareGeneration == s.shares[authenticated.roomID].generation || + (s.router.sfuOnly && message.RoutePolicy.PeerOnly) { + s.send(sess, protocol.SharingStartFailedMessage{Type: "sharing-start-failed", + ShareGeneration: message.ShareGeneration, Code: "FORBIDDEN"}) + return + } + + // stop-sharing deliberately leaves the previous generation as a stale + // fence for reconnect validation. A fresh start supersedes that fence + // without emitting a second sharing-stopped event. + share := s.shares[authenticated.roomID] + share.generation = "" + share.pausedGeneration = "" + s.shares[authenticated.roomID] = share + request := authRequest{ + role: protocol.RoleHost, + roomID: authenticated.roomID, + shareGeneration: message.ShareGeneration, + sharingPaused: message.SharingPaused, + qualitySettings: message.QualitySettings, + routePolicy: message.RoutePolicy, + } + shareGeneration := s.settleHostShare(authenticated.roomID, sess.sessionID, request) + authenticated.shareGeneration = shareGeneration + routeParticipant := routeParticipant(sess, authenticated) + policy := s.routePolicyOf(authenticated.roomID) + routeParticipant.routePolicy = &policy + routeState, assigned := s.connectRouteParticipant(routeParticipant) + if !assigned { + authenticated.shareGeneration = "" + s.stopSharing(authenticated.roomID) + s.send(sess, protocol.SharingStartFailedMessage{Type: "sharing-start-failed", + ShareGeneration: message.ShareGeneration, Code: "SERVER_ERROR"}) + return + } + + share = s.shares[authenticated.roomID] + qualitySettings := protocol.DefaultQualitySettings + if share.qualitySettings != nil { + qualitySettings = *share.qualitySettings + } + s.send(sess, protocol.SharingStartedMessage{ + Type: "sharing-started", + ShareGeneration: shareGeneration, + RoutePolicy: s.routePolicyOf(authenticated.roomID), + RouteRevision: protocol.Int(routeState.routeRevision), + RouteAssignment: routeState.routeAssignment, + QualitySettings: qualitySettings, + Paused: share.pausedGeneration == shareGeneration, + }) + // Acknowledge publication authority before any media effects use it. + s.router.setPaused(authenticated.roomID, share.pausedGeneration == shareGeneration) + s.router.completeAuthentication(routeParticipant, routeState) + s.sendViewerPresence(authenticated.roomID) + s.broadcastHostStatus(authenticated.roomID, true, share.pausedGeneration == shareGeneration) +} + // --------------------------------------------------------------------------- // quality evidence // --------------------------------------------------------------------------- diff --git a/internal/server/signal/server_test.go b/internal/server/signal/server_test.go index ae9bd39..a1328e0 100644 --- a/internal/server/signal/server_test.go +++ b/internal/server/signal/server_test.go @@ -9,6 +9,7 @@ package signal // TS went through HTTP; the site-access cookie is a stub predicate. import ( + "bytes" "context" "encoding/json" "errors" @@ -80,6 +81,7 @@ func TestStopSharingDoesNotWriteRoomAuthority(t *testing.T) { h := startHarness(t, harnessOptions{store: store}) host := openClient(t, h) authenticated := authenticate(t, host, h.room, protocol.RoleHost, "storage-free-host", 1, "", presenceOptions{}) + host.ignore("route-update") h.locked(func() { if err := database.Close(); err != nil { t.Fatal(err) @@ -94,6 +96,151 @@ func TestStopSharingDoesNotWriteRoomAuthority(t *testing.T) { authenticate(t, resumed, h.room, protocol.RoleHost, "storage-free-host", 1, "", presenceOptions{}) } +func TestHostKeepsRoomSessionAndInteractionsAfterSharingStops(t *testing.T) { + h := startHarness(t, harnessOptions{maxViewersPerRoom: 2}) + host := openClient(t, h) + hostAuth := authenticate(t, host, h.room, protocol.RoleHost, "persistent-host", 1, + "first_share_generation_12345678", presenceOptions{displayName: "Host", viewerPresence: true, roomSession: true}) + viewer := openClient(t, h) + authenticate(t, viewer, h.room, protocol.RoleViewer, "persistent-viewer", 1, "", + presenceOptions{displayName: "Viewer", viewerPresence: true}) + host.ignore("route-update") + host.ignore("viewer-presence") + subscribeInteractions(t, host) + subscribeInteractions(t, viewer) + + host.sendJSON(map[string]any{"type": "stop-sharing", "shareGeneration": hostAuth.ShareGeneration}) + viewer.next("sharing-stopped") + viewer.next("host-status") + host.expectNone(80 * time.Millisecond) + h.locked(func() { + connected, ok := h.server.store.GetConnectedHost(h.room.RoomID) + if !ok || connected.SessionID == "" { + t.Fatal("stopping media must retain the Host room session") + } + }) + + host.sendJSON(map[string]any{ + "type": "send-room-interaction", + "requestId": "persistent_chat_request", + "payload": map[string]any{"kind": "chat", "text": "still here"}, + }) + host.next("room-interaction") + viewer.next("room-interaction") + + host.sendJSON(map[string]any{ + "type": "start-sharing", + "shareGeneration": "second_share_generation_12345678", + "routePolicy": map[string]any{"peerOnly": true, "topologyOptimization": true, "natPrediction": false}, + }) + started := host.next("sharing-started") + if !bytes.Contains(started.raw, []byte(`"shareGeneration":"second_share_generation_12345678"`)) { + t.Fatalf("unexpected sharing-started: %s", started.raw) + } + viewer.next("host-status") + h.closeClient(viewer) + h.closeClient(host) +} + +func TestIdleHostChatAndPublicationAcknowledgmentOrder(t *testing.T) { + h := startHarness(t, harnessOptions{}) + host := openClient(t, h) + authenticate(t, host, h.room, protocol.RoleHost, "room-only-host", 1, "", presenceOptions{roomSession: true, roomOnly: true, viewerPresence: true}) + host.ignore("viewer-presence") + viewer := openClient(t, h) + authenticate(t, viewer, h.room, protocol.RoleViewer, "room-only-viewer", 1, "", presenceOptions{}) + subscribeInteractions(t, host) + subscribeInteractions(t, viewer) + host.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "chat_before_capture", + "payload": map[string]any{"kind": "chat", "text": "before capture"}}) + host.next("room-interaction") + viewer.next("room-interaction") + host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "new_publication_generation", + "routePolicy": protocol.DefaultRoutePolicy}) + // No ignored route messages: acknowledgment must precede media effects. + host.sendJSON(map[string]any{"type": "signaling-challenge", "sequence": 1}) + host.next("signaling-challenge-response") + host.mu.Lock() + first := host.messages[0].typ + host.mu.Unlock() + if first != "sharing-started" { + t.Fatalf("first publication message = %s", first) + } + host.next("sharing-started") + host.next("route-update") +} + +func TestRejectedPublicationRetainsHostRoomSession(t *testing.T) { + h := startHarness(t, harnessOptions{}) + host := openClient(t, h) + authenticate(t, host, h.room, protocol.RoleHost, "rejected-start-host", 1, "", presenceOptions{roomSession: true, roomOnly: true, viewerPresence: true}) + host.ignore("viewer-presence") + subscribeInteractions(t, host) + // Inject the existing route-construction failure, after room admission. + h.locked(func() { h.server.router.capacity = 0 }) + host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "rejected_publication_generation", + "routePolicy": protocol.DefaultRoutePolicy}) + expectMatch(t, host.next("sharing-start-failed").raw, + `{"shareGeneration":"rejected_publication_generation","code":"SERVER_ERROR"}`) + h.locked(func() { + h.server.router.capacity = 2 + if h.server.activeHostShareGeneration(h.room.RoomID) != "" { + t.Fatal("failed publication remained active") + } + }) + host.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "chat_after_rejected_start", + "payload": map[string]any{"kind": "chat", "text": "still in room"}}) + host.next("room-interaction") + host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "next_publication_generation", + "routePolicy": protocol.DefaultRoutePolicy}) + host.next("sharing-started") + host.next("route-update") + host.ignore("route-update") + host.sendJSON(map[string]any{"type": "stop-sharing", "shareGeneration": "next_publication_generation"}) + host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "next_publication_generation", + "routePolicy": protocol.DefaultRoutePolicy}) + expectMatch(t, host.next("sharing-start-failed").raw, + `{"shareGeneration":"next_publication_generation","code":"FORBIDDEN"}`) +} + +func TestPersistentRoomSessionReconnectStaysMediaIdle(t *testing.T) { + h := startHarness(t, harnessOptions{maxViewersPerRoom: 2}) + host := openClient(t, h) + hostAuth := authenticate(t, host, h.room, protocol.RoleHost, "idle-reconnect-host", 1, + "idle_reconnect_generation_12345678", presenceOptions{roomSession: true}) + viewer := openClient(t, h) + authenticate(t, viewer, h.room, protocol.RoleViewer, "idle-reconnect-viewer", 1, "", presenceOptions{}) + host.ignore("route-update") + viewer.ignore("route-update") + host.sendJSON(map[string]any{"type": "stop-sharing", "shareGeneration": hostAuth.ShareGeneration}) + viewer.next("sharing-stopped") + viewer.next("host-status") + h.closeClient(viewer) + h.closeClient(host) + + resumed := openClient(t, h) + resumedAuth := authenticate(t, resumed, h.room, protocol.RoleHost, "idle-reconnect-host", 1, "", + presenceOptions{roomSession: true, roomOnly: true}) + if resumedAuth.ShareGeneration != nil { + t.Fatalf("room-only Host must not receive a media generation: %v", *resumedAuth.ShareGeneration) + } + h.locked(func() { + if runtime, ok := h.server.router.rooms.Get(h.room.RoomID); ok && runtime.controller != nil { + t.Fatal("room-only Host reconnect created a media controller") + } + }) + lateViewer := openClient(t, h) + lateAuth := authenticate(t, lateViewer, h.room, protocol.RoleViewer, "idle-reconnect-late-viewer", 1, "", presenceOptions{}) + if lateAuth.HostOnline || lateAuth.HostPaused == nil || *lateAuth.HostPaused { + t.Fatalf("idle room advertised active Host media: online=%v paused=%v", lateAuth.HostOnline, lateAuth.HostPaused) + } + if lateAuth.ShareGeneration != nil || lateAuth.RouteAssignment.Upstream.Kind != "none" { + t.Fatalf("idle room advertised a media route: generation=%v assignment=%+v", lateAuth.ShareGeneration, lateAuth.RouteAssignment) + } + h.closeClient(lateViewer) + h.closeClient(resumed) +} + // --------------------------------------------------------------------------- // harness // --------------------------------------------------------------------------- @@ -113,6 +260,7 @@ type harnessOptions struct { room *room.CreatedRoom afterFunc func(time.Duration, func()) func() bool sfu *SfuFallback + sfuOnly bool } type harness struct { @@ -189,6 +337,7 @@ func startHarness(t *testing.T, options harnessOptions) *harness { Store: store, EndpointMediaCopyCapacity: orValue(options.endpointMediaCopyCapacity, 2), SfuFallback: options.sfu, + SfuOnly: options.sfuOnly, Ice: config.IceConfig(config.Config{ STUNURLs: options.stunURLs, NATPredictionEnabled: options.natPredictionEnabled, @@ -803,6 +952,8 @@ type presenceOptions struct { sharingPaused *bool qualitySettings map[string]any routePolicy *protocol.RoutePolicy + roomSession bool + roomOnly bool } // authenticate ports the authenticate() helper. relayCapacity -1 is the TS @@ -833,6 +984,12 @@ func authenticate( if shareGeneration != "" { message["shareGeneration"] = shareGeneration } + if presence.roomSession { + message["roomSession"] = true + } + if presence.roomOnly { + message["roomOnly"] = true + } if presence.sharingPaused != nil { message["sharingPaused"] = *presence.sharingPaused } diff --git a/internal/server/signal/session.go b/internal/server/signal/session.go index 46dd9fa..e6c169c 100644 --- a/internal/server/signal/session.go +++ b/internal/server/signal/session.go @@ -23,6 +23,7 @@ type authenticatedSession struct { displayName *string viewerPresence bool connectionAttemptProgress4 bool + roomSession bool } // outbound is one queue item: a text frame or the close frame that ends the @@ -54,6 +55,8 @@ type session struct { challenged bool lastSignalingChallengeAtMs int64 authenticated *authenticatedSession + roomInteractions bool + nextInteractionAtMs int64 // authStop stops the authentication deadline; authGeneration is // bumped whenever the TS cleared the timer so a callback that lost the diff --git a/internal/server/signal/sfu_only_test.go b/internal/server/signal/sfu_only_test.go new file mode 100644 index 0000000..357e0ec --- /dev/null +++ b/internal/server/signal/sfu_only_test.go @@ -0,0 +1,34 @@ +package signal + +import ( + "testing" + + "github.com/TNTcraftHIM/Piik/internal/server/protocol" + "github.com/TNTcraftHIM/Piik/internal/server/sfu" + "github.com/TNTcraftHIM/Piik/internal/server/sfu/sfutest" +) + +func TestSFUOnlyRejectsPeerOnlyBeforeAcquiringShareAuthority(t *testing.T) { + h := startHarness(t, harnessOptions{sfuOnly: true, natPredictionEnabled: true, + sfu: &SfuFallback{Media: sfutest.New(), Admission: sfu.NewAdmission(sfu.AdmissionOptions{IngressCapacity: 2, EgressCapacity: 20})}}) + policy := protocol.RoutePolicy{PeerOnly: true, TopologyOptimization: true, NatPrediction: true} + rejected := openClient(t, h) + rejected.sendJSON(map[string]any{"type": "authenticate", "protocol": protocol.SignalingProtocol, + "roomId": h.room.RoomID, "role": "host", "clientId": "old-peer-only-host", + "token": h.room.HostToken, "routePolicy": policy}) + expectMatch(t, rejected.next("error").raw, `{"code":"FORBIDDEN"}`) + expectCloseCode(t, rejected, int(protocol.SignalCloseAuthenticationFailed)) + + // An idle room remains available. Refusing a conflicting share preserves it. + host := openClient(t, h) + authenticate(t, host, h.room, protocol.RoleHost, "idle-host", -1, "", + presenceOptions{roomSession: true, roomOnly: true, routePolicy: &policy}) + host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "sfu_only_share_generation", + "routePolicy": policy}) + expectMatch(t, host.next("sharing-start-failed").raw, `{"code":"FORBIDDEN"}`) + policy.PeerOnly = false + host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "sfu_only_share_generation", + "routePolicy": policy}) + expectMatch(t, host.next("sharing-started").raw, + `{"routePolicy":{"peerOnly":false,"topologyOptimization":false,"natPrediction":false}}`) +} diff --git a/native/capture/windows/README.md b/native/capture/windows/README.md index 2e781c3..83fea64 100644 --- a/native/capture/windows/README.md +++ b/native/capture/windows/README.md @@ -5,7 +5,8 @@ enumerates local displays and visible top-level windows, binds each window to its PID and process creation time, and has independent modes for: - capability discovery (`--probe`); -- process-tree or default-device loopback PCM (`--capture-audio`); or +- process-tree inclusion/exclusion or default-device loopback PCM + (`--capture-audio`); or - WGC/D3D11 screen/window video with adapter-bound Media Foundation H.264 or libvpx VP8 output (`--capture-video --codec auto|h264|vp8`). - bounded 320x180 BMP source previews (`--preview`), delivered once per target @@ -30,7 +31,12 @@ Video and audio run as separate bounded child processes. A source whose audio loopback cannot be initialized keeps video available and reports audio unavailable instead of failing the whole source. Process loopback is probed by activation rather than inferred from a Windows build number; display sources -use the standard render-device loopback available on Windows 10 and later. +normally use the standard render-device loopback available on Windows 10 and +later. The optional `--capture-audio exclude ` mode requires +build 20348+ and a successful exclusion probe. It captures all output devices +except the selected process tree, and retires when that process exits. The +[screen-audio contract](../../../docs/standards/media-quality.md#screen-audio) +owns selection and failure behavior. Auto measures delivered frame cadence through the target-profile encoding pipeline within a four-second selection budget; H.264 that sustains the target @@ -50,6 +56,19 @@ The checks include synthetic output-worker replacement and failure cases using WARP and a test codec, plus preview cleanup when WGC rejects a hidden test window. They do not capture a user screen or validate a physical encoder. +On a Windows machine with a physical D3D11 adapter, check SDR and HDR-to-SDR conversion: + +```powershell +powershell -NoProfile -File native/capture/windows/build.ps1 -OutputDirectory build/color-check -CheckColor +``` + +This sends synthetic RGB, full/limited-range BT.601/BT.709 NV12 and FP16 HDR bars +through the production converters. It checks range/matrix, highlight detail, +SDR/HDR replacement, white-level changes and retained-frame ownership without +capturing the desktop. Decoded H.264/VP8, Browser reception and real HDR displays +remain separate media acceptance; [fidelity research](../../../docs/research/media-fidelity.md) +records current evidence and limits. + The implementation follows Microsoft's MIT-licensed reference samples and official API contracts without copying their WIL framework. The retained MF fixture compiles the same encoder source with `PIIK_H264_FIXTURE`; there is diff --git a/native/capture/windows/build.ps1 b/native/capture/windows/build.ps1 index 6c33630..2a39a40 100644 --- a/native/capture/windows/build.ps1 +++ b/native/capture/windows/build.ps1 @@ -2,7 +2,8 @@ param( [Parameter(Mandatory = $true)] [string]$OutputDirectory, - [switch]$Check + [switch]$Check, + [switch]$CheckColor ) Set-StrictMode -Version Latest @@ -34,7 +35,7 @@ $includeFlags = @('', 'third_party\abseil-cpp', 'third_party\boringssl\src\inclu '/external:I "{0}"' -f (Join-Path $webrtc.Include $_).TrimEnd('\') } $compileFlags = '/nologo /c /std:c++20 /EHsc /GR /O2 /W4 /WX /MT /D_ITERATOR_DEBUG_LEVEL=0 /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 /DNTDDI_VERSION=0x0A00000A /external:W0 ' + ($includeFlags -join ' ') -$systemLibraries = 'ole32.lib mmdevapi.lib runtimeobject.lib user32.lib gdi32.lib dwmapi.lib shell32.lib mfplat.lib mf.lib mfuuid.lib d3d11.lib dxgi.lib dxguid.lib evr.lib oleaut32.lib windowsapp.lib winmm.lib ws2_32.lib strmiids.lib crypt32.lib dmoguids.lib iphlpapi.lib msdmo.lib secur32.lib wmcodecdspuuid.lib' +$systemLibraries = 'ole32.lib mmdevapi.lib runtimeobject.lib user32.lib gdi32.lib dwmapi.lib shell32.lib mfplat.lib mf.lib mfuuid.lib d3d11.lib d2d1.lib dxgi.lib dxguid.lib evr.lib oleaut32.lib windowsapp.lib winmm.lib ws2_32.lib strmiids.lib crypt32.lib dmoguids.lib iphlpapi.lib msdmo.lib secur32.lib wmcodecdspuuid.lib' $linkCommand = '"{0}" /nologo /libpath:"{1}"' -f $webrtc.Linker,$webrtc.RuntimeLibraries function Invoke-CaptureBuild([string]$command) { @@ -76,4 +77,16 @@ if ($Check) { if ($LASTEXITCODE -ne 0) { throw "Capture check failed: $name" } } } +if ($CheckColor) { + $source = Join-Path $helperDirectory 'frame_converter.probe.cpp' + $object = Join-Path $outputPath 'frame_converter.probe.obj' + $executable = Join-Path $outputPath 'frame_converter.probe.exe' + Invoke-CaptureBuild ('cl.exe {0} "{1}" /Fo:"{2}"' -f $compileFlags,$source,$object) + $dependencies = @('process_audio', 'capture_target', 'h264_encoder', 'adaptive_encoder') | ForEach-Object { + '"{0}"' -f (Join-Path $outputPath ($_ + '.obj')) + } + Invoke-CaptureBuild ('{0} /out:"{1}" "{2}" {3} "{4}" {5}' -f $linkCommand,$executable,$object,($dependencies -join ' '),$webrtc.Library,$systemLibraries) + & $executable + if ($LASTEXITCODE -ne 0) { throw 'Capture color check failed.' } +} Write-Output $executablePath diff --git a/native/capture/windows/capture_color.h b/native/capture/windows/capture_color.h new file mode 100644 index 0000000..b712528 --- /dev/null +++ b/native/capture/windows/capture_color.h @@ -0,0 +1,34 @@ +#pragma once + +#include "capture_error.h" + +#include +#include +#include + +namespace piik::capture::windows { + +// Untagged WebRTC SDR uses BT.601. Keep the converter's pixels and the H264 +// encoder's range/matrix declaration in agreement; VP8 has no equivalent VUI. +inline constexpr auto kSdrVideoColor = DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P601; +inline constexpr auto kDesktopColor = DXGI_COLOR_SPACE_RGB_FULL_G22_NONE_P709; + +inline void SetSdrVideoColor(IMFAttributes* type) { + Check(type->SetUINT32(MF_MT_YUV_MATRIX, MFVideoTransferMatrix_BT601), "video-type-matrix"); + Check(type->SetUINT32(MF_MT_VIDEO_NOMINAL_RANGE, MFNominalRange_16_235), "video-type-range"); + Check(type->SetUINT32(MF_MT_VIDEO_PRIMARIES, MFVideoPrimaries_SMPTE170M), "video-type-primaries"); + Check(type->SetUINT32(MF_MT_TRANSFER_FUNCTION, MFVideoTransFunc_709), "video-type-transfer"); +} + +// The decoder owns the incoming stream's declaration. Preserve it through the +// relay's local scale/re-encode boundary instead of guessing from resolution. +inline DXGI_COLOR_SPACE_TYPE DecodedVideoColor(IMFAttributes* type) { + const auto matrix = MFGetAttributeUINT32(type, MF_MT_YUV_MATRIX, MFVideoTransferMatrix_BT601); + const bool full = MFGetAttributeUINT32(type, MF_MT_VIDEO_NOMINAL_RANGE, + MFNominalRange_16_235) == MFNominalRange_0_255; + if (matrix == MFVideoTransferMatrix_BT709) + return full ? DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P709 : DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P709; + return full ? DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P601 : kSdrVideoColor; +} + +} // namespace piik::capture::windows diff --git a/native/capture/windows/capture_sdr.h b/native/capture/windows/capture_sdr.h new file mode 100644 index 0000000..7aa1d8d --- /dev/null +++ b/native/capture/windows/capture_sdr.h @@ -0,0 +1,209 @@ +#pragma once + +#include "capture_error.h" +#include "capture_target.h" + +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +namespace piik::capture::windows { + +struct CaptureColor final { + bool hdr = false; + float white_nits = 80.0f; + float peak_nits = 80.0f; + + DXGI_FORMAT Format() const { + return hdr ? DXGI_FORMAT_R16G16B16A16_FLOAT : DXGI_FORMAT_B8G8R8A8_UNORM; + } +}; + +// Windows invalidates the DXGI factory when display color characteristics +// change. Window movement additionally changes the source monitor; neither +// event belongs to encoder or media-route recovery. +class CaptureDisplayColor final { + public: + CaptureDisplayColor(TargetKind kind, UINT64 source_id) + : kind_(kind), source_id_(source_id) {} + + CaptureColor Resolve() { + const auto handle = static_cast(source_id_); + const HMONITOR monitor = kind_ == TargetKind::window + ? MonitorFromWindow(reinterpret_cast(handle), MONITOR_DEFAULTTONEAREST) + : reinterpret_cast(handle); + if (factory_ && factory_->IsCurrent() && monitor == monitor_) return color_; + + Microsoft::WRL::ComPtr factory; + if (FAILED(CreateDXGIFactory1(IID_PPV_ARGS(&factory)))) + return monitor == monitor_ ? color_ : CaptureColor{}; + CaptureColor next; + for (UINT adapter_index = 0;; ++adapter_index) { + Microsoft::WRL::ComPtr adapter; + if (FAILED(factory->EnumAdapters1(adapter_index, &adapter))) break; + for (UINT output_index = 0;; ++output_index) { + Microsoft::WRL::ComPtr output; + if (FAILED(adapter->EnumOutputs(output_index, &output))) break; + Microsoft::WRL::ComPtr extended; + DXGI_OUTPUT_DESC1 description{}; + if (FAILED(output.As(&extended)) || FAILED(extended->GetDesc1(&description)) || + description.Monitor != monitor) continue; + next.hdr = description.ColorSpace == DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020; + if (next.hdr) { + next.white_nits = ReadWhiteLevel(description.DeviceName); + // WGC has no per-frame MaxCLL. Use the source display's stable peak; + // absent usable metadata, retain the HDR10 luminance range. + const float peak = std::isfinite(description.MaxLuminance) && + description.MaxLuminance > 0.0f + ? description.MaxLuminance : 10'000.0f; + next.peak_nits = std::max(next.white_nits, peak); + } + factory_ = std::move(factory); + monitor_ = monitor; + color_ = next; + return color_; + } + } + return next; + } + + private: + static float ReadWhiteLevel(const wchar_t* device_name) { + UINT32 path_count = 0, mode_count = 0; + if (GetDisplayConfigBufferSizes(QDC_ONLY_ACTIVE_PATHS, &path_count, &mode_count) != + ERROR_SUCCESS) return 80.0f; + std::vector paths(path_count); + std::vector modes(mode_count); + if (QueryDisplayConfig(QDC_ONLY_ACTIVE_PATHS, &path_count, paths.data(), &mode_count, + modes.data(), nullptr) != ERROR_SUCCESS) return 80.0f; + for (UINT32 index = 0; index < path_count; ++index) { + const auto& path = paths[index]; + DISPLAYCONFIG_SOURCE_DEVICE_NAME name{}; + name.header = {DISPLAYCONFIG_DEVICE_INFO_GET_SOURCE_NAME, sizeof(name), + path.sourceInfo.adapterId, path.sourceInfo.id}; + if (DisplayConfigGetDeviceInfo(&name.header) != ERROR_SUCCESS || + wcscmp(name.viewGdiDeviceName, device_name) != 0) continue; + DISPLAYCONFIG_SDR_WHITE_LEVEL white{}; + white.header = {DISPLAYCONFIG_DEVICE_INFO_GET_SDR_WHITE_LEVEL, sizeof(white), + path.targetInfo.adapterId, path.targetInfo.id}; + if (DisplayConfigGetDeviceInfo(&white.header) == ERROR_SUCCESS && white.SDRWhiteLevel > 0) + return static_cast(white.SDRWhiteLevel) * 80.0f / 1000.0f; + } + return 80.0f; + } + + const TargetKind kind_; + const UINT64 source_id_; + Microsoft::WRL::ComPtr factory_; + HMONITOR monitor_ = nullptr; + CaptureColor color_; +}; + +// Produces an owned SDR texture before preview or output fanout. Ordinary SDR +// keeps its exact copy path. HDR uses Windows effects on the capture GPU once, +// rather than duplicating tone mapping in each encoder or reading pixels to CPU. +class CaptureSdrConverter final { + public: + explicit CaptureSdrConverter(ID3D11Device* device) : device_(device) { + device_->GetImmediateContext(&context_); + } + + Microsoft::WRL::ComPtr Convert(ID3D11Texture2D* source, + const CaptureColor& color) { + D3D11_TEXTURE2D_DESC description{}; + source->GetDesc(&description); + if (description.Format != color.Format()) + Fail("capture-color-format", "Captured frame and display color generation differ"); + description.Usage = D3D11_USAGE_DEFAULT; + description.CPUAccessFlags = description.MiscFlags = 0; + description.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET; + Microsoft::WRL::ComPtr owned; + Check(device_->CreateTexture2D(&description, nullptr, &owned), "capture-owned-input"); + context_->CopyResource(owned.Get(), source); + if (!color.hdr) return owned; + if (!drawing_) Initialize(); + + Microsoft::WRL::ComPtr surface; + Check(owned.As(&surface), "capture-hdr-surface"); + const auto input_properties = D2D1::BitmapProperties1(D2D1_BITMAP_OPTIONS_NONE, + D2D1::PixelFormat(description.Format, D2D1_ALPHA_MODE_IGNORE)); + Microsoft::WRL::ComPtr input; + Check(drawing_->CreateBitmapFromDxgiSurface(surface.Get(), &input_properties, &input), + "capture-hdr-bitmap"); + Check(tone_map_->SetValue(D2D1_HDRTONEMAP_PROP_INPUT_MAX_LUMINANCE, color.peak_nits), + "capture-hdr-peak"); + Check(tone_map_->SetValue(D2D1_HDRTONEMAP_PROP_OUTPUT_MAX_LUMINANCE, color.white_nits), + "capture-sdr-peak"); + Check(white_level_->SetValue(D2D1_WHITELEVELADJUSTMENT_PROP_OUTPUT_WHITE_LEVEL, color.white_nits), + "capture-sdr-white"); + + description.Format = DXGI_FORMAT_B8G8R8A8_UNORM; + Microsoft::WRL::ComPtr output; + Check(device_->CreateTexture2D(&description, nullptr, &output), "capture-sdr-texture"); + surface.Reset(); + Check(output.As(&surface), "capture-sdr-surface"); + const auto output_properties = D2D1::BitmapProperties1( + D2D1_BITMAP_OPTIONS_TARGET | D2D1_BITMAP_OPTIONS_CANNOT_DRAW, + D2D1::PixelFormat(description.Format, D2D1_ALPHA_MODE_IGNORE)); + Microsoft::WRL::ComPtr target; + Check(drawing_->CreateBitmapFromDxgiSurface(surface.Get(), &output_properties, &target), + "capture-sdr-bitmap"); + tone_map_->SetInput(0, input.Get()); + drawing_->SetTarget(target.Get()); + Microsoft::WRL::ComPtr image; + srgb_->GetOutput(&image); + drawing_->BeginDraw(); + drawing_->DrawImage(image.Get(), nullptr, nullptr, + D2D1_INTERPOLATION_MODE_NEAREST_NEIGHBOR, D2D1_COMPOSITE_MODE_SOURCE_COPY); + const HRESULT result = drawing_->EndDraw(); + drawing_->SetTarget(nullptr); + tone_map_->SetInput(0, nullptr); + Check(result, "capture-hdr-tone-map"); + return output; + } + + private: + void Initialize() { + Microsoft::WRL::ComPtr factory; + Check(D2D1CreateFactory(D2D1_FACTORY_TYPE_SINGLE_THREADED, IID_PPV_ARGS(&factory)), + "capture-d2d-factory"); + Microsoft::WRL::ComPtr dxgi; + Check(device_.As(&dxgi), "capture-d2d-dxgi"); + Microsoft::WRL::ComPtr device; + Check(factory->CreateDevice(dxgi.Get(), &device), "capture-d2d-device"); + Check(device->CreateDeviceContext(D2D1_DEVICE_CONTEXT_OPTIONS_NONE, &drawing_), + "capture-d2d-context"); + const D2D1_RENDERING_CONTROLS controls{D2D1_BUFFER_PRECISION_16BPC_FLOAT, {0, 0}}; + drawing_->SetRenderingControls(&controls); + Check(drawing_->CreateEffect(CLSID_D2D1HdrToneMap, &tone_map_), "capture-tone-map-effect"); + Check(tone_map_->SetValue(D2D1_HDRTONEMAP_PROP_DISPLAY_MODE, D2D1_HDRTONEMAP_DISPLAY_MODE_SDR), + "capture-tone-map-mode"); + Check(drawing_->CreateEffect(CLSID_D2D1WhiteLevelAdjustment, &white_level_), "capture-white-effect"); + Check(white_level_->SetValue(D2D1_WHITELEVELADJUSTMENT_PROP_INPUT_WHITE_LEVEL, 80.0f), + "capture-input-white"); + white_level_->SetInputEffect(0, tone_map_.Get()); + Check(drawing_->CreateEffect(CLSID_D2D1ColorManagement, &srgb_), "capture-srgb-effect"); + Microsoft::WRL::ComPtr linear, srgb; + Check(drawing_->CreateColorContext(D2D1_COLOR_SPACE_SCRGB, nullptr, 0, &linear), "capture-linear-color"); + Check(drawing_->CreateColorContext(D2D1_COLOR_SPACE_SRGB, nullptr, 0, &srgb), "capture-srgb-color"); + Check(srgb_->SetValue(D2D1_COLORMANAGEMENT_PROP_SOURCE_COLOR_CONTEXT, linear.Get()), "capture-input-color"); + Check(srgb_->SetValue(D2D1_COLORMANAGEMENT_PROP_DESTINATION_COLOR_CONTEXT, srgb.Get()), "capture-output-color"); + Check(srgb_->SetValue(D2D1_COLORMANAGEMENT_PROP_QUALITY, D2D1_COLORMANAGEMENT_QUALITY_BEST), + "capture-color-quality"); + srgb_->SetInputEffect(0, white_level_.Get()); + } + + Microsoft::WRL::ComPtr device_; + Microsoft::WRL::ComPtr context_; + Microsoft::WRL::ComPtr drawing_; + Microsoft::WRL::ComPtr tone_map_, white_level_, srgb_; +}; + +} // namespace piik::capture::windows diff --git a/native/capture/windows/capture_target.cpp b/native/capture/windows/capture_target.cpp index 8393792..4675cb3 100644 --- a/native/capture/windows/capture_target.cpp +++ b/native/capture/windows/capture_target.cpp @@ -19,6 +19,7 @@ #include "capture_target.h" #include "capture_border.h" #include "capture_geometry.h" +#include "capture_sdr.h" #include #include @@ -497,8 +498,11 @@ HRESULT CaptureWithWgc(TargetKind kind, UINT64 source_id, if (size.Width <= 0 || size.Height <= 0 || frame_ready == nullptr) { result = HRESULT_FROM_WIN32(ERROR_INVALID_DATA); } else { + windows::CaptureDisplayColor display_color(kind, source_id); + const auto color = display_color.Resolve(); + windows::CaptureSdrConverter sdr(device.Get()); pool = Direct3D11CaptureFramePool::CreateFreeThreaded( - capture_device, DirectXPixelFormat::B8G8R8A8UIntNormalized, 1, + capture_device, static_cast(color.Format()), 1, size); session = pool.CreateCaptureSession(item); frame_token = pool.FrameArrived( @@ -530,6 +534,7 @@ HRESULT CaptureWithWgc(TargetKind kind, UINT64 source_id, ::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>(); ComPtr texture; result = access->GetInterface(IID_PPV_ARGS(&texture)); + if (SUCCEEDED(result)) texture = sdr.Convert(texture.Get(), color); D3D11_TEXTURE2D_DESC description{}; if (SUCCEEDED(result)) texture->GetDesc(&description); if (SUCCEEDED(result) && @@ -1016,9 +1021,14 @@ HRESULT ValidateProcessTarget(DWORD pid, UINT64 expected_creation_time) { } HANDLE process = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid); if (process == nullptr) return HRESULT_FROM_WIN32(GetLastError()); + const HRESULT result = ValidateProcessTarget(process, expected_creation_time); + CloseHandle(process); + return result; +} + +HRESULT ValidateProcessTarget(HANDLE process, UINT64 expected_creation_time) { UINT64 actual_creation_time = 0; HRESULT result = ReadProcessCreationTime(process, &actual_creation_time); - CloseHandle(process); if (SUCCEEDED(result) && actual_creation_time != expected_creation_time) { result = HRESULT_FROM_WIN32(ERROR_INVALID_DATA); } diff --git a/native/capture/windows/capture_target.h b/native/capture/windows/capture_target.h index e1ccf2c..b18c072 100644 --- a/native/capture/windows/capture_target.h +++ b/native/capture/windows/capture_target.h @@ -13,6 +13,7 @@ int WriteMicrophoneList(); HRESULT WriteSourcePreview(TargetKind kind, UINT64 source_id, DWORD pid, UINT64 expected_creation_time); HRESULT ValidateProcessTarget(DWORD pid, UINT64 expected_creation_time); +HRESULT ValidateProcessTarget(HANDLE process, UINT64 expected_creation_time); HRESULT ValidateWindowTarget(UINT64 source_id, DWORD pid, UINT64 expected_creation_time); HRESULT ValidateDisplayTarget(UINT64 source_id); diff --git a/native/capture/windows/capture_target.test.cpp b/native/capture/windows/capture_target.test.cpp index 296a53b..4f02628 100644 --- a/native/capture/windows/capture_target.test.cpp +++ b/native/capture/windows/capture_target.test.cpp @@ -15,6 +15,8 @@ int main() { assert(GetProcessTimes(GetCurrentProcess(), &created, &exited, &kernel, &user)); const UINT64 creation_time = (static_cast(created.dwHighDateTime) << 32) | created.dwLowDateTime; + assert(SUCCEEDED(piik::capture::ValidateProcessTarget(GetCurrentProcess(), creation_time))); + assert(FAILED(piik::capture::ValidateProcessTarget(GetCurrentProcess(), creation_time + 1))); HANDLE previous_output = GetStdHandle(STD_OUTPUT_HANDLE); HANDLE sink = CreateFileW(L"NUL", GENERIC_WRITE, FILE_SHARE_WRITE, nullptr, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr); diff --git a/native/capture/windows/frame_converter.probe.cpp b/native/capture/windows/frame_converter.probe.cpp new file mode 100644 index 0000000..10da309 --- /dev/null +++ b/native/capture/windows/frame_converter.probe.cpp @@ -0,0 +1,218 @@ +// Physical D3D11 check; opt in with build.ps1 -CheckColor. No desktop capture. +#define wmain capture_main +#include "main.cpp" +#undef wmain +#include + +namespace { + +constexpr std::array kRgb{ + 0xff000000, 0xff202020, 0xff808080, 0xffebebeb, + 0xffffffff, 0xffff0000, 0xff00ff00, 0xff0000ff}; +constexpr std::array, 8> kYuv{{ + {16, 128, 128}, {44, 128, 128}, {126, 128, 128}, {218, 128, 128}, + {235, 128, 128}, {82, 90, 240}, {145, 54, 34}, {41, 240, 110}}}; +constexpr std::array, 8> kYuv709{{ + {16, 128, 128}, {44, 128, 128}, {126, 128, 128}, {218, 128, 128}, + {235, 128, 128}, {63, 102, 240}, {173, 42, 26}, {32, 240, 118}}}; +constexpr std::array, 8> kYuvFull{{ + {0, 128, 128}, {32, 128, 128}, {128, 128, 128}, {235, 128, 128}, + {255, 128, 128}, {76, 85, 255}, {150, 44, 21}, {29, 255, 107}}}; +constexpr std::array, 8> kYuv709Full{{ + {0, 128, 128}, {32, 128, 128}, {128, 128, 128}, {235, 128, 128}, + {255, 128, 128}, {54, 99, 255}, {182, 30, 12}, {18, 255, 116}}}; + +ComPtr ColorBars(const DeviceContext& device, UINT width, + UINT height, DXGI_COLOR_SPACE_TYPE color) { + const bool rgb = color == kDesktopColor; + const auto& yuv = color == DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P709 ? kYuv709 + : color == DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P709 ? kYuv709Full + : color == DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P601 ? kYuvFull : kYuv; + std::vector pixels(static_cast(width) * height * (rgb ? 4 : 2)); + for (UINT row = 0; row < height; ++row) { + for (UINT column = 0; column < width; ++column) { + const size_t bar = column * kRgb.size() / width; + const size_t index = static_cast(row) * width + column; + if (rgb) { + for (UINT channel = 0; channel < 4; ++channel) + pixels[index * 4 + channel] = static_cast(kRgb[bar] >> (channel * 8)); + } else { + pixels[index] = static_cast(yuv[bar][0]); + if (row < height / 2) + pixels[static_cast(width) * height + index] = + static_cast(yuv[bar][1 + column % 2]); + } + } + } + D3D11_TEXTURE2D_DESC description{}; + description.Width = width; + description.Height = height; + description.MipLevels = description.ArraySize = description.SampleDesc.Count = 1; + description.Format = rgb ? DXGI_FORMAT_B8G8R8A8_UNORM : DXGI_FORMAT_NV12; + description.BindFlags = D3D11_BIND_RENDER_TARGET | D3D11_BIND_SHADER_RESOURCE; + D3D11_SUBRESOURCE_DATA data{}; + data.pSysMem = pixels.data(); + data.SysMemPitch = width * (rgb ? 4 : 1); + ComPtr texture; + Check(device.device->CreateTexture2D(&description, &data, &texture), "color-source"); + return texture; +} + +void CheckColors(const DeviceContext& device, ID3D11Texture2D* texture) { + D3D11_TEXTURE2D_DESC description{}; + texture->GetDesc(&description); + description.Usage = D3D11_USAGE_STAGING; + description.BindFlags = 0; + description.CPUAccessFlags = D3D11_CPU_ACCESS_READ; + ComPtr staging; + Check(device.device->CreateTexture2D(&description, nullptr, &staging), "color-readback"); + device.context->CopyResource(staging.Get(), texture); + D3D11_MAPPED_SUBRESOURCE mapped{}; + Check(device.context->Map(staging.Get(), 0, D3D11_MAP_READ, 0, &mapped), "color-map"); + const auto* bytes = static_cast(mapped.pData); + bool correct = true; + for (size_t bar = 0; bar < kYuv.size(); ++bar) { + const auto x = description.Width * (2 * bar + 1) / (2 * kYuv.size()); + const size_t y = description.Height / 2; + const size_t uv = (description.Height + y / 2) * mapped.RowPitch + (x & ~size_t{1}); + const std::array actual{bytes[y * mapped.RowPitch + x], bytes[uv], bytes[uv + 1]}; + for (size_t channel = 0; channel < 3; ++channel) { + if (std::abs(actual[channel] - kYuv[bar][channel]) > 2) { + std::cerr << "bar=" << bar << " channel=" << channel + << " expected=" << kYuv[bar][channel] << " actual=" << actual[channel] << '\n'; + correct = false; + } + } + } + device.context->Unmap(staging.Get(), 0); + if (!correct) throw std::runtime_error("SDR range or matrix changed"); +} + +ComPtr LinearBars(const DeviceContext& device, UINT width, + UINT height, const std::array& nits) { + std::vector pixels(static_cast(width) * height * 4); + for (size_t pixel = 0; pixel < static_cast(width) * height; ++pixel) { + const auto value = DirectX::PackedVector::XMConvertFloatToHalf(nits[(pixel % width) * 8 / width] / 80.0f); + for (size_t channel = 0; channel < 3; ++channel) pixels[pixel * 4 + channel] = value; + pixels[pixel * 4 + 3] = DirectX::PackedVector::XMConvertFloatToHalf(1.0f); + } + D3D11_TEXTURE2D_DESC description{}; + description.Width = width; + description.Height = height; + description.MipLevels = description.ArraySize = description.SampleDesc.Count = 1; + description.Format = DXGI_FORMAT_R16G16B16A16_FLOAT; + description.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET; + D3D11_SUBRESOURCE_DATA data{}; + data.pSysMem = pixels.data(); + data.SysMemPitch = width * 8; + ComPtr texture; + Check(device.device->CreateTexture2D(&description, &data, &texture), "hdr-source"); + return texture; +} + +std::array ReadGrayBars(const DeviceContext& device, ID3D11Texture2D* texture) { + D3D11_TEXTURE2D_DESC description{}; + texture->GetDesc(&description); + description.Usage = D3D11_USAGE_STAGING; + description.BindFlags = 0; + description.CPUAccessFlags = D3D11_CPU_ACCESS_READ; + ComPtr staging; + Check(device.device->CreateTexture2D(&description, nullptr, &staging), "hdr-readback"); + device.context->CopyResource(staging.Get(), texture); + D3D11_MAPPED_SUBRESOURCE mapped{}; + Check(device.context->Map(staging.Get(), 0, D3D11_MAP_READ, 0, &mapped), "hdr-map"); + const auto* bytes = static_cast(mapped.pData); + std::array result{}; + bool neutral = true; + for (size_t bar = 0; bar < result.size(); ++bar) { + const size_t pixel = description.Height / 2 * mapped.RowPitch + + description.Width * (2 * bar + 1) / 16 * 4; + result[bar] = bytes[pixel]; + neutral = neutral && std::abs(result[bar] - bytes[pixel + 1]) <= 1 && + std::abs(result[bar] - bytes[pixel + 2]) <= 1; + } + device.context->Unmap(staging.Get(), 0); + if (!neutral) throw std::runtime_error("HDR gray acquired a color cast"); + return result; +} + +void CheckHdr(const DeviceContext& device) { + CaptureSdrConverter capture(device.device.Get()); + VideoProfile profile; + profile.width = 256; + profile.height = 128; + FrameConverter converter(device.device.Get(), profile); + for (const UINT width : {256u, 512u, 256u}) { + // SDR must retain the old pixels after an HDR frame, with the same owner. + auto sdr = ColorBars(device, width, width / 2, kDesktopColor); + auto owned = capture.Convert(sdr.Get(), {}); + CheckColors(device, converter.Convert(owned.Get(), width, width / 2, + {static_cast(width), static_cast(width / 2)}, kDesktopColor).Get()); + auto hdr = LinearBars(device, width, width / 2, {0, 5, 20, 80, 200, 400, 700, 1000}); + for (const float white : {80.0f, 240.0f, 80.0f}) { + auto mapped = capture.Convert(hdr.Get(), {true, white, 1000.0f}); + const auto levels = ReadGrayBars(device, mapped.Get()); + std::cout << "HDR white=" << white << " levels="; + for (const auto level : levels) std::cout << level << ','; + std::cout << '\n'; + if (levels.front() > 1 || levels.back() < 245) + throw std::runtime_error("HDR black/peak mapping is incorrect"); + for (size_t index = 1; index < levels.size(); ++index) + if (levels[index] <= levels[index - 1]) + throw std::runtime_error("HDR luminance detail was clipped before tone mapping"); + auto next = capture.Convert(sdr.Get(), {}); + if (ReadGrayBars(device, mapped.Get()) != levels) + throw std::runtime_error("Capture replacement modified a retained frame"); + CheckColors(device, converter.Convert(next.Get(), width, width / 2, + {static_cast(width), static_cast(width / 2)}, kDesktopColor).Get()); + } + } +} + +} // namespace + +int main() { + try { + Runtime runtime; + const auto adapters = EnumerateAdapters(); + if (adapters.empty()) throw std::runtime_error("A physical D3D11 adapter is required"); + for (const auto& adapter : adapters) { + auto device = CreateDevice(adapter); + for (const UINT output_width : {256u, 128u}) { + VideoProfile profile; + profile.width = output_width; + profile.height = output_width / 2; + FrameConverter converter(device.device.Get(), profile); + // Reuse the same converter across RGB/NV12 and size replacements. + for (const UINT input_width : {256u, 512u, 256u}) { + for (const auto color : {kDesktopColor, kSdrVideoColor, + DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P709, + DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P601, + DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P709, kDesktopColor}) { + const UINT input_height = input_width / 2; + auto source = ColorBars(device, input_width, input_height, color); + auto output = converter.Convert(source.Get(), input_width, input_height, + {static_cast(input_width), static_cast(input_height)}, + color); + try { + CheckColors(device, output.Get()); + } catch (...) { + std::cerr << "adapter=" << adapter.index << " color=" << color + << " input-width=" << input_width << " output-width=" << output_width << '\n'; + throw; + } + } + } + } + std::cout << "SDR conversion passed: " << NarrowAscii(adapter.description.Description) << '\n'; + CheckHdr(device); + std::cout << "HDR conversion passed: " << NarrowAscii(adapter.description.Description) << '\n'; + } + return 0; + } catch (const std::exception& error) { + std::cerr << error.what() << '\n'; + if (const auto* failure = dynamic_cast(&error)) + std::cerr << "stage=" << failure->stage() << " result=" << std::hex << failure->result() << '\n'; + return 1; + } +} diff --git a/native/capture/windows/h264_decoder.h b/native/capture/windows/h264_decoder.h index 0afad8c..f7e0bd5 100644 --- a/native/capture/windows/h264_decoder.h +++ b/native/capture/windows/h264_decoder.h @@ -1,6 +1,7 @@ #pragma once #include "capture_error.h" +#include "capture_color.h" #include "capture_input.h" @@ -56,6 +57,12 @@ class H264Decoder final { H264Decoder(const H264Decoder&) = delete; H264Decoder& operator=(const H264Decoder&) = delete; + DXGI_COLOR_SPACE_TYPE ColorSpace() const { + Microsoft::WRL::ComPtr type; + Check(decoder_->GetOutputCurrentType(0, &type), "current-output-type"); + return windows::DecodedVideoColor(type.Get()); + } + void Decode(const InputEnvelope& input, const Consumer& consume) { if (input.kind != 2 || input.data.empty() || input.data.size() > 4 * 1024 * 1024) throw std::runtime_error("Invalid H264 decoder input"); diff --git a/native/capture/windows/h264_encoder.cpp b/native/capture/windows/h264_encoder.cpp index 3d52b55..a9c01cb 100644 --- a/native/capture/windows/h264_encoder.cpp +++ b/native/capture/windows/h264_encoder.cpp @@ -1,6 +1,7 @@ #include "h264_encoder.h" #include "capture_output.h" +#include "capture_color.h" #include #include @@ -327,6 +328,7 @@ ComPtr CreateOutputType( "output-type-major"); Check(type->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_H264), "output-type-subtype"); + SetSdrVideoColor(type.Get()); Check(type->SetUINT32(MF_MT_AVG_BITRATE, profile.bit_rate), "output-type-bitrate"); Check(MFSetAttributeSize(type.Get(), MF_MT_FRAME_SIZE, profile.width, @@ -354,6 +356,7 @@ ComPtr CreateInputType( "input-type-major"); Check(type->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_NV12), "input-type-subtype"); + SetSdrVideoColor(type.Get()); Check(MFSetAttributeSize(type.Get(), MF_MT_FRAME_SIZE, profile.width, profile.height), "input-type-size"); diff --git a/native/capture/windows/main.cpp b/native/capture/windows/main.cpp index c957ab2..8f34976 100644 --- a/native/capture/windows/main.cpp +++ b/native/capture/windows/main.cpp @@ -3,7 +3,7 @@ #include #include -#include +#include #include #include #include @@ -32,6 +32,8 @@ #include "capture_target.h" #include "capture_border.h" #include "capture_geometry.h" +#include "capture_color.h" +#include "capture_sdr.h" #include "process_audio.h" #include "h264_encoder.h" #ifndef PIIK_H264_FIXTURE @@ -663,8 +665,9 @@ class FrameConverter final { public: explicit FrameConverter( ID3D11Device* device, - VideoProfile profile = kDefaultVideoProfile) - : device_(device), profile_(profile) { + VideoProfile profile = kDefaultVideoProfile, + DXGI_COLOR_SPACE_TYPE output_color = kSdrVideoColor) + : device_(device), profile_(profile), output_color_(output_color) { Check(device_->QueryInterface(IID_PPV_ARGS(&video_device_)), "video-processor-device"); ComPtr context; @@ -673,13 +676,30 @@ class FrameConverter final { } ComPtr Convert(ID3D11Texture2D* source, UINT32 width, - UINT32 height, SIZE presentation) { + UINT32 height, SIZE presentation, + DXGI_COLOR_SPACE_TYPE input_color) { if (source == nullptr || width == 0 || height == 0 || width > 16'384 || height > 16'384) { Fail("capture-size", "captured window dimensions are invalid"); } - if (!enumerator_ || width != input_width_ || height != input_height_) { - Configure(width, height); + ComPtr normalized; + if (output_color_ == kSdrVideoColor && input_color != kDesktopColor && + input_color != kSdrVideoColor) { + // YUV-to-YUV matrix/range changes are not reliable across tested drivers, + // even when reported as supported. Normalize through RGB on the GPU; + // ordinary RGB capture and canonical NV12 scaling stay single-pass. + if (!rgb_converter_) + rgb_converter_ = std::make_unique(device_.Get(), profile_, kDesktopColor); + normalized = rgb_converter_->Convert(source, width, height, presentation, input_color); + source = normalized.Get(); + width = profile_.width; + height = profile_.height; + presentation = {static_cast(width), static_cast(height)}; + input_color = kDesktopColor; + } + if (!enumerator_ || width != input_width_ || height != input_height_ || + input_color != input_color_) { + Configure(width, height, input_color); } D3D11_TEXTURE2D_DESC output_description = {}; @@ -687,7 +707,8 @@ class FrameConverter final { output_description.Height = profile_.height; output_description.MipLevels = 1; output_description.ArraySize = 1; - output_description.Format = DXGI_FORMAT_NV12; + output_description.Format = output_color_ == kDesktopColor + ? DXGI_FORMAT_B8G8R8A8_UNORM : DXGI_FORMAT_NV12; output_description.SampleDesc.Count = 1; output_description.Usage = D3D11_USAGE_DEFAULT; output_description.BindFlags = D3D11_BIND_RENDER_TARGET; @@ -745,7 +766,7 @@ class FrameConverter final { } private: - void Configure(UINT32 width, UINT32 height) { + void Configure(UINT32 width, UINT32 height, DXGI_COLOR_SPACE_TYPE input_color) { D3D11_VIDEO_PROCESSOR_CONTENT_DESC description = {}; description.InputFrameFormat = D3D11_VIDEO_FRAME_FORMAT_PROGRESSIVE; description.InputFrameRate = {profile_.frame_rate, 1}; @@ -762,20 +783,27 @@ class FrameConverter final { ComPtr processor; Check(video_device_->CreateVideoProcessor(enumerator.Get(), 0, &processor), "video-processor-create"); + // The incoming frame owns its range/matrix. Never infer these from size. + video_context_->VideoProcessorSetStreamColorSpace1(processor.Get(), 0, input_color); + video_context_->VideoProcessorSetOutputColorSpace1(processor.Get(), output_color_); enumerator_ = std::move(enumerator); processor_ = std::move(processor); input_width_ = width; input_height_ = height; + input_color_ = input_color; } ComPtr device_; VideoProfile profile_; + const DXGI_COLOR_SPACE_TYPE output_color_; + std::unique_ptr rgb_converter_; ComPtr video_device_; - ComPtr video_context_; + ComPtr video_context_; ComPtr enumerator_; ComPtr processor_; UINT32 input_width_ = 0; UINT32 input_height_ = 0; + DXGI_COLOR_SPACE_TYPE input_color_ = kDesktopColor; }; winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice @@ -933,6 +961,7 @@ struct ProductArguments final { VideoProfile profile; std::vector outputs; bool show_capture_border = false; + bool exclude_audio = false; }; DegradationPreference ParseDegradationPreference(const wchar_t* value) { @@ -1020,7 +1049,8 @@ ProductArguments ParseProductArguments(int count, wchar_t** values) { } if (count == 5 && std::wstring(values[1]) == L"--capture-audio") { arguments.mode = ProductArguments::Mode::audio; - arguments.target_kind = ParseTargetKind(values[2]); + arguments.exclude_audio = std::wstring(values[2]) == L"exclude"; + arguments.target_kind = arguments.exclude_audio ? piik::capture::TargetKind::window : ParseTargetKind(values[2]); } else if (count >= 16 && count <= 11 + 5 * piik::capture::kMaxOutputs && (count - 11) % 5 == 0 && std::wstring(values[1]) == L"--encoded-video" && std::wstring(values[2]) == L"--codec" && @@ -1126,7 +1156,7 @@ HRESULT RunAudioCapture(const ProductArguments& arguments) { stop.get(), should_stop, ready, pcm); } return piik::capture::CaptureProcessAudio( - arguments.pid, arguments.creation_time, stop.get(), + arguments.pid, arguments.creation_time, arguments.exclude_audio, stop.get(), should_stop, ready, pcm); } @@ -1153,6 +1183,7 @@ UINT32 WindowsBuild() { void WriteCapabilityProbe() { constexpr UINT32 kCreateForWindowMinimumBuild = 18'362; constexpr UINT32 kProcessLoopbackMinimumBuild = 19'041; + constexpr UINT32 kProcessExclusionMinimumBuild = 20'348; const UINT32 build = WindowsBuild(); bool window_capture = false; if (build >= kCreateForWindowMinimumBuild) { @@ -1167,6 +1198,8 @@ void WriteCapabilityProbe() { build >= kProcessLoopbackMinimumBuild && piik::capture::ProcessAudioAvailable(); const bool system_audio = piik::capture::SystemAudioAvailable(); + const bool process_audio_exclusion = build >= kProcessExclusionMinimumBuild && + piik::capture::ProcessAudioAvailable(true); std::vector adapters = EnumerateAdapters(); std::ostringstream output; @@ -1178,6 +1211,7 @@ void WriteCapabilityProbe() { << ",\"microphone\":true,\"softwareVP8\":true" << ",\"processAudio\":" << (process_audio ? "true" : "false") + << ",\"processAudioExclusion\":" << (process_audio_exclusion ? "true" : "false") << ",\"systemAudio\":" << (system_audio ? "true" : "false") << ",\"adapters\":["; for (size_t adapter_index = 0; adapter_index < adapters.size(); @@ -1361,6 +1395,7 @@ VideoEncoderSelection SelectVideoEncoder( struct CaptureInput final { ComPtr texture; + DXGI_COLOR_SPACE_TYPE color_space = kDesktopColor; UINT32 width = 0; UINT32 height = 0; SIZE presentation{}; @@ -1448,7 +1483,7 @@ class OutputWorker final { converted_height = height; } return converter->Convert(input->texture.Get(), input->width, - input->height, input->presentation); + input->height, input->presentation, input->color_space); }, profile_.width, profile_.height, input->timestamp, work.recovery, work.bitrate); } catch (...) { // Device loss is current shared evidence; an old codec failure only @@ -1703,12 +1738,13 @@ void RunEncodedVideo(ProductArguments arguments) { bool began = false; UINT64 last_timestamp = 0; auto submit = [&](ComPtr texture, UINT32 width, UINT32 height, - UINT64 timestamp, UINT64 duration) { + UINT64 timestamp, UINT64 duration, DXGI_COLOR_SPACE_TYPE color) { if ((began && timestamp <= last_timestamp) || width < arguments.profile.width || height < arguments.profile.height) { Fail("encoded-input-output", "decoded frame changed the source timeline or output bounds"); } auto input = std::make_shared(); input->texture = std::move(texture); + input->color_space = color; input->width = width; input->height = height; input->presentation = {static_cast(width), static_cast(height)}; @@ -1735,12 +1771,12 @@ void RunEncodedVideo(ProductArguments arguments) { Check(sample->GetSampleDuration(&duration), "decoded-duration"); if (timestamp < 0 || duration <= 0) Fail("decoded-timing", "decoder did not preserve source timing"); submit(OwnDecodedTexture(device, sample, width, height, stride), width, height, - static_cast(timestamp), static_cast(duration)); + static_cast(timestamp), static_cast(duration), h264->ColorSpace()); }); } else { const auto decoded = vp8->Decode(input.data); if (!decoded.nv12.empty()) submit(UploadDecodedNV12(device, decoded.nv12.data(), decoded.width, decoded.height), - decoded.width, decoded.height, input.timestamp, input.duration); + decoded.width, decoded.height, input.timestamp, input.duration, kSdrVideoColor); } } } @@ -1801,8 +1837,11 @@ void RunVideoCapture(ProductArguments arguments) { if (initial_size.Width <= 0 || initial_size.Height <= 0) { Fail("capture-size", "selected window has no capturable content"); } + CaptureDisplayColor display_color(arguments.target_kind, arguments.source_id); + CaptureSdrConverter sdr(device.device.Get()); + auto pool_format = display_color.Resolve().Format(); Direct3D11CaptureFramePool pool = Direct3D11CaptureFramePool::CreateFreeThreaded( - capture_device, DirectXPixelFormat::B8G8R8A8UIntNormalized, 2, + capture_device, static_cast(pool_format), 2, initial_size); GraphicsCaptureSession capture_session = pool.CreateCaptureSession(item); EnableFastCaptureUpdates(capture_session); @@ -1964,15 +2003,17 @@ void RunVideoCapture(ProductArguments arguments) { } auto content_size = latest.ContentSize(); if (content_size.Width <= 0 || content_size.Height <= 0) continue; + const auto color = display_color.Resolve(); if (content_size.Width != pool_size.Width || - content_size.Height != pool_size.Height) { + content_size.Height != pool_size.Height || color.Format() != pool_format) { latest.Close(); latest = nullptr; latest_input.reset(); pool.Recreate(capture_device, - DirectXPixelFormat::B8G8R8A8UIntNormalized, 2, + static_cast(color.Format()), 2, content_size); pool_size = content_size; + pool_format = color.Format(); continue; } ComPtr source = CaptureTexture(latest); @@ -1983,13 +2024,7 @@ void RunVideoCapture(ProductArguments arguments) { UINT32 content_height = std::min( source_description.Height, static_cast(content_size.Height)); auto input = std::make_shared(); - auto owned_description = source_description; - owned_description.Usage = D3D11_USAGE_DEFAULT; - owned_description.CPUAccessFlags = 0; - owned_description.MiscFlags = 0; - owned_description.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET; - Check(device.device->CreateTexture2D(&owned_description, nullptr, &input->texture), "capture-owned-input"); - device.context->CopyResource(input->texture.Get(), source.Get()); + input->texture = sdr.Convert(source.Get(), color); input->width = content_width; input->height = content_height; input->timestamp = timestamp; diff --git a/native/capture/windows/process_audio.cpp b/native/capture/windows/process_audio.cpp index ca84706..f439697 100644 --- a/native/capture/windows/process_audio.cpp +++ b/native/capture/windows/process_audio.cpp @@ -31,7 +31,8 @@ namespace { class ActivationHandler final : public RuntimeClass< RuntimeClassFlags, FtmBase, IActivateAudioInterfaceCompletionHandler> { public: - explicit ActivationHandler(HANDLE completed) : completed_(completed) {} + ActivationHandler() : completed_(CreateEventW(nullptr, FALSE, FALSE, nullptr)) {} + ~ActivationHandler() { if (completed_) CloseHandle(completed_); } STDMETHODIMP ActivateCompleted(IActivateAudioInterfaceAsyncOperation* operation) override { ComPtr activated; @@ -45,6 +46,7 @@ class ActivationHandler final : public RuntimeClass< HRESULT Result() const { return result_; } ComPtr Client() const { return client_; } + HANDLE Completed() const { return completed_; } private: HANDLE completed_ = nullptr; @@ -52,26 +54,29 @@ class ActivationHandler final : public RuntimeClass< ComPtr client_; }; -HRESULT ActivateProcessLoopback(DWORD pid, HANDLE completed, +HRESULT ActivateProcessLoopback(DWORD pid, bool exclude, ComPtr* client) { AUDIOCLIENT_ACTIVATION_PARAMS parameters{}; parameters.ActivationType = AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK; parameters.ProcessLoopbackParams.TargetProcessId = pid; parameters.ProcessLoopbackParams.ProcessLoopbackMode = - PROCESS_LOOPBACK_MODE_INCLUDE_TARGET_PROCESS_TREE; + exclude ? PROCESS_LOOPBACK_MODE_EXCLUDE_TARGET_PROCESS_TREE + : PROCESS_LOOPBACK_MODE_INCLUDE_TARGET_PROCESS_TREE; PROPVARIANT variant{}; variant.vt = VT_BLOB; variant.blob.cbSize = sizeof(parameters); variant.blob.pBlobData = reinterpret_cast(¶meters); - auto handler = Make(completed); + // The completion can outlive a timed-out caller; it owns its event handle. + auto handler = Make(); if (!handler) return E_OUTOFMEMORY; + if (!handler->Completed()) return HRESULT_FROM_WIN32(GetLastError()); ComPtr operation; HRESULT result = ActivateAudioInterfaceAsync(VIRTUAL_AUDIO_DEVICE_PROCESS_LOOPBACK, __uuidof(IAudioClient), &variant, handler.Get(), &operation); if (FAILED(result)) return result; - if (WaitForSingleObject(completed, 10'000) != WAIT_OBJECT_0) return HRESULT_FROM_WIN32(WAIT_TIMEOUT); + if (WaitForSingleObject(handler->Completed(), 10'000) != WAIT_OBJECT_0) return HRESULT_FROM_WIN32(WAIT_TIMEOUT); result = handler->Result(); if (SUCCEEDED(result)) *client = handler->Client(); return result; @@ -114,6 +119,14 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo return result; } HRESULT result = S_OK; + // Bind process audio to the selected lifetime, including while draining PCM. + // Exclusion must never become unfiltered system capture after its target dies. + const auto processAlive = [&]() -> HRESULT { + if (!process) return S_OK; + const DWORD wait = WaitForSingleObject(process, 0); + if (wait == WAIT_TIMEOUT) return S_OK; + return HRESULT_FROM_WIN32(wait == WAIT_OBJECT_0 ? ERROR_PROCESS_ABORTED : GetLastError()); + }; ComPtr capture; WAVEFORMATEX format{}; format.wFormatTag = WAVE_FORMAT_PCM; @@ -131,6 +144,7 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo if (SUCCEEDED(result)) result = client->GetService(IID_PPV_ARGS(&capture)); if (SUCCEEDED(result)) result = client->SetEventHandle(sampleReady); if (SUCCEEDED(result)) result = client->Start(); + if (SUCCEEDED(result)) result = processAlive(); if (SUCCEEDED(result)) result = ready_writer(); if (SUCCEEDED(result)) { std::array silence{}; @@ -171,6 +185,7 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo } UINT32 frames = 0; while (SUCCEEDED(result = capture->GetNextPacketSize(&frames)) && frames > 0) { + if (FAILED(result = processAlive())) break; BYTE* data = nullptr; DWORD flags = 0; UINT64 devicePosition = 0; @@ -198,6 +213,7 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo } capture->ReleaseBuffer(frames); while (pending.size() - consumed >= kAudioBytesPerChunk) { + if (FAILED(result = processAlive())) break; result = writer(nextTimestamp, pending.data() + consumed, kAudioBytesPerChunk); if (FAILED(result)) break; @@ -218,13 +234,10 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo } // namespace -bool ProcessAudioAvailable() { - HANDLE completed = CreateEventW(nullptr, FALSE, FALSE, nullptr); - if (completed == nullptr) return false; +bool ProcessAudioAvailable(bool exclude) { ComPtr client; const HRESULT result = - ActivateProcessLoopback(GetCurrentProcessId(), completed, &client); - CloseHandle(completed); + ActivateProcessLoopback(GetCurrentProcessId(), exclude, &client); return SUCCEEDED(result) && client != nullptr; } @@ -233,32 +246,24 @@ bool SystemAudioAvailable() { return SUCCEEDED(ActivateEndpoint(&client)) && client != nullptr; } -HRESULT CaptureProcessAudio(DWORD pid, UINT64 expectedCreationTime, +HRESULT CaptureProcessAudio(DWORD pid, UINT64 expectedCreationTime, bool exclude, HANDLE stop_event, const StopProbe& stop_probe, const ReadyWriter& ready_writer, const PCMWriter& writer) { HRESULT com_result = CoInitializeEx(nullptr, COINIT_MULTITHREADED); if (FAILED(com_result)) return com_result; - HRESULT result = ValidateProcessTarget(pid, expectedCreationTime); - HANDLE process = nullptr; - HANDLE completed = nullptr; + // Validate and watch the same handle so PID reuse cannot change the target. + HANDLE process = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION | SYNCHRONIZE, FALSE, pid); + HRESULT result = process ? ValidateProcessTarget(process, expectedCreationTime) + : HRESULT_FROM_WIN32(GetLastError()); ComPtr client; if (SUCCEEDED(result)) { - process = OpenProcess(SYNCHRONIZE, FALSE, pid); - if (process == nullptr) result = HRESULT_FROM_WIN32(GetLastError()); - } - if (SUCCEEDED(result)) { - completed = CreateEventW(nullptr, FALSE, FALSE, nullptr); - if (completed == nullptr) result = HRESULT_FROM_WIN32(GetLastError()); - } - if (SUCCEEDED(result)) { - result = ActivateProcessLoopback(pid, completed, &client); + result = ActivateProcessLoopback(pid, exclude, &client); } if (SUCCEEDED(result)) { result = CaptureAudioFrames(client, process, true, stop_event, stop_probe, ready_writer, writer); } - if (completed != nullptr) CloseHandle(completed); if (process != nullptr) CloseHandle(process); client.Reset(); CoUninitialize(); diff --git a/native/capture/windows/process_audio.h b/native/capture/windows/process_audio.h index 55e0215..1090149 100644 --- a/native/capture/windows/process_audio.h +++ b/native/capture/windows/process_audio.h @@ -21,11 +21,11 @@ using PCMWriter = using ReadyWriter = std::function; using StopProbe = std::function; -bool ProcessAudioAvailable(); +bool ProcessAudioAvailable(bool exclude = false); bool SystemAudioAvailable(); HRESULT CaptureMicrophone(const std::wstring& device_id, HANDLE stop_event, const StopProbe& stop_probe, const ReadyWriter& ready_writer, const PCMWriter& writer); -HRESULT CaptureProcessAudio(DWORD pid, UINT64 expected_creation_time, +HRESULT CaptureProcessAudio(DWORD pid, UINT64 expected_creation_time, bool exclude, HANDLE stop_event, const StopProbe& stop_probe, const ReadyWriter& ready_writer, const PCMWriter& writer); diff --git a/scripts/embedded-sfu-gate.ts b/scripts/embedded-sfu-gate.ts index 007ade2..b18e8bd 100644 --- a/scripts/embedded-sfu-gate.ts +++ b/scripts/embedded-sfu-gate.ts @@ -65,6 +65,7 @@ async function main(): Promise { const chromePath = process.env.CHROME_PATH?.trim(); if (!chromePath) throw new Error("CHROME_PATH is required"); const nativeArm = process.env.PIIK_EMBEDDED_SFU_NATIVE === "true"; + const sfuOnly = process.env.PIIK_EMBEDDED_SFU_ONLY === "true"; const displayArm = process.env.PIIK_EMBEDDED_SFU_DISPLAY === "true"; if (displayArm && nativeArm) throw new Error("Display and Native capture are separate arms"); const twoRooms = process.env.PIIK_EMBEDDED_SFU_TWO_ROOMS === "true"; @@ -93,6 +94,7 @@ async function main(): Promise { const sourceProfile = nativeArm ? await mkdtemp(join(tmpdir(), "piik-client-media-")) : null; const [sourcePort, sourceDebugPort] = nativeArm ? [await reservePort(), await reservePort()] : [0, 0]; const result = { passed: false, arm: `${nativeArm ? "native" : displayArm ? "display" : "browser"}-${codec}-simulcast${twoRooms ? "-two-rooms" : ""}`, stage: "start", + sfuOnly, stereoSeparationDb: null as number[] | null, startedAt: new Date().toISOString(), finishedAt: "", processes: [] as Array<{ role: string; pid: number | null }>, high: null as { frames: number; width: number; height: number } | null, low: null as { frames: number; width: number; height: number } | null, @@ -144,7 +146,8 @@ async function main(): Promise { PIIK_ENV: "development", PIIK_DEBUG: "route", LISTEN_HOST: "127.0.0.1", PORT: String(serverPort), PIIK_LOG_DIR: join(BUILD_ROOT, "logs"), PUBLIC_BASE_URL: origin, ALLOWED_ORIGINS: origin, - SFU_LISTEN_HOST: "127.0.0.1", SFU_UDP_PORT: String(mediaPort), SFU_PUBLIC_IP: "127.0.0.1" }, + SFU_LISTEN_HOST: "127.0.0.1", SFU_UDP_PORT: String(mediaPort), SFU_PUBLIC_IP: "127.0.0.1", + SFU_ONLY: String(sfuOnly) }, }); processStarted("server", server); server.stdout.resume(); @@ -257,6 +260,7 @@ async function main(): Promise { result.stage = "decoded-audio-energy"; await waitForSample(() => call(viewer!, "gate.audioEnergy()", 3_000), (energy) => { result.audioEnergy.high = energy; return energy > 0; }, 5_000); + if (!nativeArm) result.stereoSeparationDb = await call(viewer, "gate.audioSeparation()", 5_000); result.host = await call(host, "gate.snapshot()"); if (displayArm) { result.stage = "display-background"; @@ -357,7 +361,9 @@ async function main(): Promise { } result.finishedAt = new Date().toISOString(); result.passed = !result.error && result.host?.published === true && result.host.simulcast && - result.viewer?.decoded === true && result.viewer.committed && result.viewer.peerFailures > 0 && + result.viewer?.decoded === true && result.viewer.committed && + (sfuOnly ? result.viewer.peerFailures === 0 : result.viewer.peerFailures > 0) && + (nativeArm || result.stereoSeparationDb?.every(db => db > 20) === true) && result.viewer.audioKbps > 0 && result.audioEnergy.high > 0 && result.audioEnergy.low > result.audioEnergy.high && result.high !== null && result.low !== null && result.audioEnergy.recovered > 0 && result.publicationRetired && result.subscriptionRecovered && result.cleanup && result.nativeShareStopped && diff --git a/scripts/embedded-sfu-page.ts b/scripts/embedded-sfu-page.ts index 83246c3..207de8e 100644 --- a/scripts/embedded-sfu-page.ts +++ b/scripts/embedded-sfu-page.ts @@ -239,18 +239,58 @@ export async function startHost(nativeSource?: { title: string; port: number }, video: { width: 1280, height: 720, frameRate: 15 }, audio: false, }) : canvas.captureStream(15); audio = new AudioContext(); - const tone = audio.createOscillator(); const gain = audio.createGain(); gain.gain.value = 0.05; const destination = audio.createMediaStreamDestination(); - tone.connect(gain).connect(destination); - tone.start(); + const channels = audio.createChannelMerger(2); + [500, 1500].forEach((frequency, channel) => { + const tone = audio!.createOscillator(); + tone.frequency.value = frequency; + tone.connect(channels, 0, channel); + tone.start(); + }); + channels.connect(gain).connect(destination); await audio.resume(); source.addTrack(destination.stream.getAudioTracks()[0]!); await connect(room, true); return room; } +// Check decoded channels, not just /2 in SDP (Opus also uses that for mono). +export async function audioSeparation(stream = video?.srcObject): Promise { + if (!(stream instanceof MediaStream) || !stream.getAudioTracks().length) throw new Error("No received audio"); + const context = new AudioContext(); + try { + const input = context.createMediaStreamSource(stream); + const splitter = context.createChannelSplitter(2); + const sink = context.createGain(); + sink.gain.value = 0; + sink.connect(context.destination); + input.connect(splitter); + const analyzers = [0, 1].map(channel => { + const analyzer = context.createAnalyser(); + analyzer.fftSize = 8192; + analyzer.smoothingTimeConstant = 0; + splitter.connect(analyzer, channel); + analyzer.connect(sink); + return analyzer; + }); + await context.resume(); + await new Promise(resolve => setTimeout(resolve, 1000)); + return analyzers.map((analyzer, channel) => { + const bins = new Float32Array(analyzer.frequencyBinCount); + analyzer.getFloatFrequencyData(bins); + const level = (frequency: number) => { + const bin = Math.round(frequency * analyzer.fftSize / context.sampleRate); + return Math.max(-120, ...bins.slice(bin - 1, bin + 2)); + }; + return level(channel === 0 ? 500 : 1500) - level(channel === 0 ? 1500 : 500); + }); + } finally { + await context.close(); + } +} + export async function startViewer(room: CreateRoomResponse): Promise { await connect(room, false); } diff --git a/src/client/App.tsx b/src/client/App.tsx index 7ce6a31..254f64b 100644 --- a/src/client/App.tsx +++ b/src/client/App.tsx @@ -397,8 +397,11 @@ function SiteAccessGate({ } return ( diff --git a/src/client/components/living/CaptureSourcePicker.tsx b/src/client/components/living/CaptureSourcePicker.tsx index be4e1c6..bd6cba3 100644 --- a/src/client/components/living/CaptureSourcePicker.tsx +++ b/src/client/components/living/CaptureSourcePicker.tsx @@ -32,6 +32,7 @@ export type NativeSourceList = sources: NativeCaptureTarget[]; processAudio: boolean; systemAudio: boolean; + processAudioExclusion?: boolean; captureBorderControl?: boolean; }; @@ -51,6 +52,7 @@ export function CaptureSourcePicker({ loadCameras, initialAudio = true, initialShowCaptureBorder = false, + initialExcludeAudio, audioLocked = false, selectionDisabled = false, }: { @@ -60,7 +62,7 @@ export function CaptureSourcePicker({ initialCamera?: string; activeCameraVideo?: HTMLVideoElement | null; loadCameras?: typeof loadCameraPreviews; - onNative: (target: NativeCaptureTarget, audio: boolean, showCaptureBorder: boolean) => void; + onNative: (target: NativeCaptureTarget, audio: boolean, showCaptureBorder: boolean, excludeAudio?: NativeCaptureTarget) => void; onPreview: ( target: NativeCaptureTarget, signal?: AbortSignal, @@ -72,6 +74,7 @@ export function CaptureSourcePicker({ initialTab?: SourceTab; initialAudio?: boolean; initialShowCaptureBorder?: boolean; + initialExcludeAudio?: NativeCaptureTarget; audioLocked?: boolean; selectionDisabled?: boolean; }) { @@ -81,6 +84,7 @@ export function CaptureSourcePicker({ const [tab, setTab] = useState(null); const [shareAudio, setShareAudio] = useState(initialAudio); const [showCaptureBorder, setShowCaptureBorder] = useState(initialShowCaptureBorder); + const [excludeAudio, setExcludeAudio] = useState(initialExcludeAudio); const appDetected = nativeSources.kind === "ready" || nativeSources.kind === "failed" || nativeSources.kind === "unsupported" || nativeSources.kind === "incompatible"; const tabs = SOURCE_TABS.filter((value) => appDetected || value === "browser" || value === "camera"); @@ -122,6 +126,14 @@ export function CaptureSourcePicker({ ? nativeSources.processAudio : nativeSources.systemAudio); const anyNativeAudio = sources.some(supportsAudio); + // Source enumeration already owns process identity; do not create a second app scanner. + const audioApplications = nativeSources.kind === "ready" + ? [...new Map(nativeSources.sources.filter(target => target.kind === "window") + .map(target => [`${target.pid}:${target.creationTime}`, target])).values()] : []; + const exclusionKey = excludeAudio?.kind === "window" ? `${excludeAudio.pid}:${excludeAudio.creationTime}` : ""; + const canExcludeAudio = activeTab === "display" && nativeSources.kind === "ready" && nativeSources.processAudioExclusion; + const missingExclusion = !!excludeAudio && !audioApplications.some(target => `${target.pid}:${target.creationTime}` === exclusionKey); + const exclusionUnavailable = activeTab === "display" && shareAudio && !!excludeAudio && (!canExcludeAudio || missingExclusion); const refreshing = activeTab === "camera" ? cameraSources.busy : nativeSources.kind === "loading"; const refreshLabel = t(refreshing ? "host.sourcePicker.loading" : "host.sourcePicker.refresh"); const audioAction = t(!anyNativeAudio ? "host.noAudio" : audioLocked @@ -300,12 +312,12 @@ export function CaptureSourcePicker({ key={nativeCaptureTargetKey(target)} target={target} disabled={ - selectionDisabled || (audioLocked && shareAudio && !supportsAudio(target)) + selectionDisabled || exclusionUnavailable || (audioLocked && shareAudio && !supportsAudio(target)) } onPreview={onPreview} onSelect={() => onNative(target, shareAudio && supportsAudio(target), - supportsCaptureBorder && showCaptureBorder) + supportsCaptureBorder && showCaptureBorder, target.kind === "display" ? excludeAudio : undefined) } /> )) @@ -352,6 +364,32 @@ export function CaptureSourcePicker({ ) : null} + {canExcludeAudio && shareAudio && anyNativeAudio ? ( +
{ + if (event.currentTarget.open) event.currentTarget.scrollIntoView({ block: "nearest" }); + }}> + + + {!vis && {t("host.sourcePicker.excludeAudio")}{excludeAudio ? ` · ${excludeAudio.title}` : ""}} + + +
+ + + {t(excludeAudio ? "host.sourcePicker.excludeAudioHint" : "host.sourcePicker.excludeAudioChoose")} + +
+
+ ) : null} ) : null} diff --git a/src/client/components/living/Couch.tsx b/src/client/components/living/Couch.tsx index e6fe09d..1ef91ca 100644 --- a/src/client/components/living/Couch.tsx +++ b/src/client/components/living/Couch.tsx @@ -25,19 +25,27 @@ export interface CouchHostEntry { onSelect?: () => void; } +export interface CouchProps { + view: "host" | "viewer"; + host?: CouchHostEntry | null; + entries: CouchEntry[]; + selectedKey?: string | null; + onSelect?: (key: string) => void; + participantAction?: { + open: (key: string, trigger: HTMLButtonElement) => void; + target: string | null; + controls: string; + }; +} + export function Couch({ view, host, entries, selectedKey, onSelect, -}: { - view: "host" | "viewer"; - host?: CouchHostEntry | null; - entries: CouchEntry[]; - selectedKey?: string | null; - onSelect?: (key: string) => void; -}) { + participantAction, +}: CouchProps) { const { t } = useCopy(); const [couchRef, containerWidth] = useElementWidth(); const count = entries.length + (host ? 1 : 0); @@ -76,15 +84,16 @@ export function Couch({ aria-label={`${t("common.host")} · ${t("common.viewers")}`} > {host ? ( - - {host.onSelect ? ( + + {host.onSelect || participantAction ? ( ); return ( - + {pawn} ); diff --git a/src/client/components/living/FloatingPanel.tsx b/src/client/components/living/FloatingPanel.tsx new file mode 100644 index 0000000..29c215e --- /dev/null +++ b/src/client/components/living/FloatingPanel.tsx @@ -0,0 +1,222 @@ +import { useLayoutEffect, useRef, useState, type PointerEvent, type ReactNode, type RefObject } from "react"; +import { useCopy } from "../../ui/copy"; +import { Glyph, type GlyphName } from "../../ui/icons"; +import "./floating-panel.css"; + +const PANEL_SIZES = { default: { width: 420, height: 560 }, large: { width: 560, height: 680 } }; +const COMPACT_SIZES = { default: { width: 304, height: undefined }, large: { width: 380, height: 460 } }; +type PanelGesture = "move" | "nw" | "se"; +type PanelChange = { x: number; y: number } | { rect: DOMRect; corner: "nw" | "se"; dx: number; dy: number }; + +/** Native visibility owns the shell; content owns its session, draft and reading position. */ +export function FloatingPanel({ id, trigger, title, icon, compact = false, anchorOnOpen = false, className = "", children, onOpenChange }: { + id: string; trigger: string | RefObject; title: string; icon: GlyphName; compact?: boolean; className?: string; + anchorOnOpen?: boolean; + children: ReactNode; onOpenChange?: (open: boolean) => void; +}) { + const { t, vis } = useCopy(); + const [open, setOpen] = useState(false); + const panelRef = useRef(null); + const positionMenu = useRef(null); + const sizes = compact ? COMPACT_SIZES : PANEL_SIZES; + const position = useRef({ x: 1, y: 1 }); + const preferredSize = useRef(sizes.default); + const placePanel = useRef<((change?: PanelChange) => void) | null>(null); + const gesture = useRef<{ pointerId: number; x: number; y: number; rect: DOMRect; kind: PanelGesture } | null>(null); + const suppressDragClick = useRef(false); + const getTrigger = () => typeof trigger === "string" ? document.getElementById(trigger) : trigger.current; + const focusTrigger = () => { + const element = getTrigger(); + if (element?.isConnected) element.focus({ preventScroll: true }); + }; + + function startGesture(event: PointerEvent, kind: PanelGesture) { + if (!event.isPrimary || event.button !== 0 || !panelRef.current) return; + if (kind === "move" && (event.target as Element).closest("button, .lr-floating-positions")) return; + panelRef.current.getAnimations().forEach(animation => animation.finish()); + gesture.current = { pointerId: event.pointerId, x: event.clientX, y: event.clientY, + rect: panelRef.current.getBoundingClientRect(), kind }; + // Capture on the summary preserves its native click/keyboard activation. + const handle = (event.target as Element).closest("summary") ?? event.currentTarget; + handle.focus({ preventScroll: true }); + handle.setPointerCapture(event.pointerId); + } + useLayoutEffect(() => { + const panel = panelRef.current; + if (!open || !panel) return; + // The visual viewport keeps the composer above a phone's keyboard without + // resizing the living room or taking ownership of scrolling/focus. + const viewport = window.visualViewport; + const place = (change?: PanelChange) => { + const layout = document.documentElement; + const width = viewport?.width ?? layout.clientWidth, height = viewport?.height ?? layout.clientHeight; + const maxWidth = Math.max(0, width - 24), maxHeight = Math.max(0, height - 24); + const left = (viewport?.offsetLeft ?? 0) + 12, top = (viewport?.offsetTop ?? 0) + 12; + let point = change && "x" in change ? change : undefined; + if (change && "corner" in change) { + const { rect, corner, dx, dy } = change; + const nw = corner === "nw"; + // Hold the opposite corner steady; viewport changes only constrain the + // rendered size, while deliberate resizing writes the room UI preference. + const availableWidth = nw ? rect.right - left : left + maxWidth - rect.left; + const availableHeight = nw ? rect.bottom - top : top + maxHeight - rect.top; + const w = Math.min(availableWidth, Math.max(Math.min(compact ? 264 : 320, maxWidth), rect.width + (nw ? -dx : dx))); + const h = Math.min(availableHeight, Math.max(Math.min(compact ? 240 : 360, maxHeight), rect.height + (nw ? -dy : dy))); + preferredSize.current = { width: w, height: h }; + point = { x: nw ? rect.right - w : rect.left, y: nw ? rect.bottom - h : rect.top }; + } + panel.style.width = `${Math.min(preferredSize.current.width, maxWidth)}px`; + panel.style.maxHeight = `${maxHeight}px`; + const preferredHeight = preferredSize.current.height; + panel.style.height = preferredHeight === undefined ? "auto" : `${Math.min(preferredHeight, maxHeight)}px`; + const freeX = Math.max(0, maxWidth - panel.offsetWidth), freeY = Math.max(0, maxHeight - panel.offsetHeight); + // Keep the chosen relative position when the keyboard or viewport changes. + // Only deliberate movement writes it; a narrow viewport must not erase it. + if (point) { + if (freeX) position.current.x = Math.max(0, Math.min(1, (point.x - left) / freeX)); + if (freeY) position.current.y = Math.max(0, Math.min(1, (point.y - top) / freeY)); + } + panel.style.inset = `${top + freeY * position.current.y}px auto auto ${left + freeX * position.current.x}px`; + }; + const resize = () => place(); + const hideOnEscape = (event: KeyboardEvent) => { + // Inner native surfaces consume Escape first, even after focus moves out. + if (event.key !== "Escape" || event.defaultPrevented || event.isComposing || + document.querySelector('[popover="auto"]:popover-open, dialog:modal')) return; + // Native top-layer order follows opening, not DOM order. Consume one + // close request even when chat and the reaction palette are both open. + const panels = [...document.querySelectorAll(".lr-floating-panel:popover-open")]; + if (!panels.length) return; + const topmost = panels.reduce((top, next) => Number(next.dataset.openedAt) >= Number(top.dataset.openedAt) ? next : top); + if (topmost !== panel) return; + event.preventDefault(); + const restoreFocus = panel.contains(document.activeElement); + panel.hidePopover(); + if (restoreFocus) focusTrigger(); + }; + placePanel.current = place; + place(); + // Contextual palettes start beside their invoker, then use the same free + // movement owner. Retargeting an open palette must not undo a user's drag. + const anchor = anchorOnOpen && getTrigger()?.getBoundingClientRect(); + if (anchor) { + const top = (viewport?.offsetTop ?? 0) + 12; + const bottom = top + (viewport?.height ?? document.documentElement.clientHeight) - 24; + const below = anchor.bottom + 8; + const headroom = anchor.top - top >= panel.offsetHeight + 136 ? 128 : 0; + place({ x: anchor.x + anchor.width / 2 - panel.offsetWidth / 2, + y: below + panel.offsetHeight <= bottom ? below : anchor.top - panel.offsetHeight - 8 - headroom }); + } + viewport?.addEventListener("resize", resize); + viewport?.addEventListener("scroll", resize); + window.addEventListener("resize", resize); + // Content-sized palettes still fit after a target, language or status change. + const observer = new ResizeObserver(resize); + observer.observe(panel); + document.addEventListener("keydown", hideOnEscape); + return () => { + placePanel.current = null; + gesture.current = null; + viewport?.removeEventListener("resize", resize); + viewport?.removeEventListener("scroll", resize); + window.removeEventListener("resize", resize); + observer.disconnect(); + document.removeEventListener("keydown", hideOnEscape); + }; + }, [open, trigger, compact, anchorOnOpen]); + + return (); +} diff --git a/src/client/components/living/LauncherForm.tsx b/src/client/components/living/LauncherForm.tsx index 06af218..7ba7b5e 100644 --- a/src/client/components/living/LauncherForm.tsx +++ b/src/client/components/living/LauncherForm.tsx @@ -112,7 +112,7 @@ export function LauncherForm({ key={choice.mode} type="button" role="radio" - className={`lr-client-mode${mode === choice.mode ? " is-selected" : ""}`} + className={`lr-tile lr-client-mode${mode === choice.mode ? " is-selected" : ""}`} aria-checked={mode === choice.mode} tabIndex={mode === choice.mode ? 0 : -1} aria-label={t(choice.label)} diff --git a/src/client/components/living/MicrophoneDeviceSelect.tsx b/src/client/components/living/MicrophoneDeviceSelect.tsx index 28760d9..cd07f41 100644 --- a/src/client/components/living/MicrophoneDeviceSelect.tsx +++ b/src/client/components/living/MicrophoneDeviceSelect.tsx @@ -3,7 +3,7 @@ import type { CaptureDevice } from "../../media/capture-devices"; import { useCopy } from "../../ui/copy"; import { Glyph } from "../../ui/icons"; -// Shared presentation only: capture and replacement stay with the Host owner. +// Shared presentation only: the caller owns capture and device replacement. export function MicrophoneDeviceSelect({ value, load, onChange, disabled, revision, browser = true }: { value: string; load: () => Promise; onChange: (id: string) => void; disabled?: boolean; revision?: unknown; browser?: boolean; diff --git a/src/client/components/living/Pawn.tsx b/src/client/components/living/Pawn.tsx index 75c8f7f..edb5341 100644 --- a/src/client/components/living/Pawn.tsx +++ b/src/client/components/living/Pawn.tsx @@ -7,7 +7,7 @@ export function PersonShape({ eyes = true, gaze = 0, eyeClassName, host }: { eyes?: boolean | "closed"; gaze?: number; eyeClassName?: string; host?: boolean; }) { return <> - + {eyes ? {eyes === "closed" ? - + ; } diff --git a/src/client/components/living/PlaybackControls.tsx b/src/client/components/living/PlaybackControls.tsx index 38cadac..c368a4d 100644 --- a/src/client/components/living/PlaybackControls.tsx +++ b/src/client/components/living/PlaybackControls.tsx @@ -1,4 +1,4 @@ -import { useEffect, useRef, useState, type RefObject } from "react"; +import { useEffect, useRef, useState, type RefObject, type ReactNode } from "react"; import { ViewerAudio, type ViewerAudioSnapshot } from "../../media/viewer-audio"; import { useCopy } from "../../ui/copy"; import { debugError } from "../../lib/debug"; @@ -15,6 +15,7 @@ import "./playback-controls.css"; export function PlaybackControls({ videoRef, stream, audioTrackKey, canPlay, theaterMode, onPlay, onToggleTheater, onReconnect, reconnectAvailable, + extraActions, }: { videoRef: RefObject; stream: MediaStream | null; @@ -25,6 +26,7 @@ export function PlaybackControls({ onToggleTheater: () => void; onReconnect: () => void; reconnectAvailable: boolean; + extraActions?: ReactNode; }) { const { t, vis } = useCopy(); const audioRef = useRef(null); @@ -146,6 +148,7 @@ export function PlaybackControls({ + {extraActions} : null; +} + +function ConnectedToggle({ session, caption }: { session: RoomInteractionSession; caption: boolean }) { + const { overlayEnabled } = useSyncExternalStore(session.subscribe, session.getSnapshot); + return session.setOverlayEnabled(!overlayEnabled)} />; +} + +export function RoomChatOverlay({ session, visible }: { session: RoomInteractionSession | null; visible: boolean }) { + return session ? : null; +} + +function ConnectedOverlay({ session, visible }: { session: RoomInteractionSession; visible: boolean }) { + const { overlayMessages } = useSyncExternalStore(session.subscribe, session.getSnapshot); + useEffect(() => { + const sync = () => session.setOverlayVisible(visible && !document.hidden); + sync(); + document.addEventListener("visibilitychange", sync); + return () => { + document.removeEventListener("visibilitychange", sync); + session.setOverlayVisible(false); + }; + }, [session, visible]); + return visible ? : null; +} + +function ChatFlight({ message, now }: { + message: ReturnType["overlayMessages"][number]; now: () => number; +}) { + return { + // Mounting or returning from reduced motion uses the remaining lifetime. + for (const animation of event.currentTarget.getAnimations()) { + if (animation instanceof CSSAnimation && animation.animationName === "lr-chat-flight") { + animation.currentTime = Math.min(CHAT_OVERLAY_DURATION_MS, Math.max(0, CHAT_OVERLAY_DURATION_MS - (message.expiresAt - now()))); + } + } + }}> + {message.payload.kind === "chat" && message.payload.text} + ; +} diff --git a/src/client/components/living/RoomInteractions.tsx b/src/client/components/living/RoomInteractions.tsx new file mode 100644 index 0000000..efe3c68 --- /dev/null +++ b/src/client/components/living/RoomInteractions.tsx @@ -0,0 +1,228 @@ +import { useEffect, useId, useLayoutEffect, useRef, useState, useSyncExternalStore } from "react"; +import { normalizeChatText, MAX_CHAT_CODE_POINTS, REACTION_IDS, isThrow } from "../../../shared/room-interactions"; +import { RoomInteractionSession } from "../../lib/room-interactions"; +import { useCopy } from "../../ui/copy"; +import { Glyph } from "../../ui/icons"; +import { Btn, SwitchItem } from "./primitives"; +import { participantColor } from "./participant-color"; +import { FloatingPanel } from "./FloatingPanel"; +import { PawnSvg } from "./Pawn"; +import { Couch, type CouchProps } from "./Couch"; +import { RoomReaction, ReactionIcon } from "./RoomReaction"; +import { RoomChatToggle } from "./RoomChatOverlay"; +import "./room-interactions.css"; + +interface Participant { peerId: string; displayName: string } +type Props = CouchProps & { session: RoomInteractionSession | null }; + +export function RoomInteractions(props: Props) { + return props.session ? : ; +} + +type ConnectedProps = Props & { session: RoomInteractionSession }; +function ConnectedRoomInteractions(props: ConnectedProps) { + const state = useSyncExternalStore(props.session.subscribe, props.session.getSnapshot); + return ; +} + +function InteractionPanel({ session, state, ...couch }: ConnectedProps & { + state: ReturnType; +}) { + const { t, vis } = useCopy(); + const [chatOpen, setChatOpen] = useState(false); + const [effects, setEffects] = useState(true); + const [target, setTarget] = useState(null); + const [draft, setDraft] = useState(""); + const [lastSeen, setLastSeen] = useState(null); + const [visible, setVisible] = useState(false); + const [following, setFollowing] = useState(true); + const area = useRef(null); + const anchor = useRef(null); + const log = useRef(null); + const id = useId(); + const menuId = `${id}-person`; + const participants: Participant[] = [ + ...(couch.host ? [{ peerId: couch.host.key, displayName: couch.host.name }] : []), + ...couch.entries.map(entry => ({ peerId: entry.key, displayName: entry.name })), + ]; + const latest = state.messages.at(-1)?.id ?? null; + const unseenIndex = state.messages.findIndex(message => message.id === lastSeen); + const unread = latest === lastSeen ? 0 : state.messages.length - unseenIndex - 1; + const canSend = state.ready && !state.pending && !state.coolingDown; + const person = participants.find(person => person.peerId === target); + const rosterKey = participants.map(person => person.peerId).join(","); + const inspect = target === couch.host?.key && couch.host.onSelect ? () => { if (!couch.host?.selected) couch.host?.onSelect?.(); } + : couch.entries.some(entry => entry.key === target && entry.selectable !== false) && couch.onSelect + ? () => { if (target !== couch.selectedKey) couch.onSelect?.(target!); } : undefined; + + useEffect(() => { + const confirmed = state.confirmed; + if (confirmed?.payload.kind === "chat") { + const text = confirmed.payload.text; + setDraft(current => normalizeChatText(current) === text ? "" : current); + } + }, [state.confirmed]); + useLayoutEffect(() => { + const popup = document.getElementById(menuId); + if (!popup) return; + if (person && anchor.current?.isConnected) popup.showPopover({ source: anchor.current }); + else popup.hidePopover(); + }, [target, rosterKey, menuId]); + + function openReactions(key: string, trigger: HTMLButtonElement) { + const popup = document.getElementById(menuId); + if (target === key && popup?.matches(":popover-open")) { popup.hidePopover(); return; } + anchor.current = trigger; + setTarget(key); + } + useEffect(() => { + if (!chatOpen || !log.current) return; + let active = true; + let inView = false; + const sync = () => { if (active) setVisible(inView && document.visibilityState === "visible"); }; + const observer = new IntersectionObserver(entries => { + const entry = entries.at(-1); + inView = Boolean(entry?.isIntersecting && entry.intersectionRatio > 0); + sync(); + }); + observer.observe(log.current); + document.addEventListener("visibilitychange", sync); + return () => { + active = false; + observer.disconnect(); + document.removeEventListener("visibilitychange", sync); + setVisible(false); + }; + }, [chatOpen]); + useEffect(() => { + if (!chatOpen || !visible || !following || document.visibilityState !== "visible" || + !log.current?.getClientRects().length) return; + setLastSeen(latest); + log.current.scrollTop = log.current.scrollHeight; + }, [latest, chatOpen, visible, following]); + + return
+
+ {state.reactions.map(({ id, sender, payload }) => payload.kind === "reaction" &&

+ {t(payload.targetPeerId ? "interaction.receivedReaction" : "interaction.expressedReaction", { + sender: sender.displayName, + reaction: t(`interaction.reaction.${payload.reaction}`), + target: participants.find(person => person.peerId === payload.targetPeerId)?.displayName ?? t("interaction.participant"), + })} +

)} +
+ +
+
{ const node = event.currentTarget; setFollowing(node.scrollHeight - node.scrollTop - node.clientHeight < 40); }}> + {state.messages.length === 0 ?
+

{vis ? "···" : t("interaction.empty")}

+
: state.messages.map((message, index) => { + const previous = state.messages[index - 1]; + const continuation = previous?.sender.peerId === message.sender.peerId && previous.sender.displayName === message.sender.displayName; + return
+ {!continuation && } +
+ {message.sender.displayName} + {message.sender.role === "host" && · {t("common.host")}} +

{message.payload.kind === "chat" && message.payload.text}

+
+
; + })} +
+ {!following && state.messages.length > 0 && } +
+
{ + event.preventDefault(); + const text = normalizeChatText(draft); + if (text && canSend) { setFollowing(true); session.send({ kind: "chat", text }); } + }}> + setDraft(event.target.value)} maxLength={MAX_CHAT_CODE_POINTS * 2} + aria-label={t("interaction.placeholder")} placeholder={vis ? "···" : t("interaction.placeholder")} + autoComplete="off" + onKeyDown={event => { if (event.key === "Enter" && event.nativeEvent.isComposing) event.preventDefault(); }} /> + + +
+ +
+ {state.error ? <>{t(`interaction.error.${state.error}`)} + : !state.ready ? <>{t("interaction.reconnecting")} + : !vis && {t("interaction.ephemeral")}} +
+ {draft && } +
+
+
+ + {effects && state.reactions.map((reaction, index) => { + const payload = reaction.payload; + if (payload.kind !== "reaction") return null; + // Only one's own expression replaces one's previous expression. + // Receiving a gift never speaks for that person or hides their reaction. + const covered = !payload.targetPeerId && !isThrow(payload.reaction) && state.reactions.slice(index + 1).some(next => + next.payload.kind === "reaction" && !next.payload.targetPeerId && !isThrow(next.payload.reaction) && + next.sender.peerId === reaction.sender.peerId); + if (covered) return null; + return participants.some(person => person.peerId === reaction.sender.peerId) && + (!payload.targetPeerId || participants.some(person => person.peerId === payload.targetPeerId)) && + ; + })} +
+
+ + + {unread > 0 && {unread}} + +
+ { if (!open) setTarget(null); }}> +
+
+ + {person + ? + : } + {!vis && {t(target === state.peerId ? "interaction.express" : "interaction.toPerson")}} + {person?.displayName ?? t("interaction.participant")} + +
+ {[false, true].filter(throws => !throws || (person && target !== state.peerId)).map(throws =>
+ {REACTION_IDS.filter(reaction => isThrow(reaction) === throws).map(reaction => )} +
)} + {state.error || !state.ready ?
+ + {t(state.error ? `interaction.error.${state.error}` : "interaction.reconnecting")} +
: null} + {inspect && } + {target === state.peerId &&
+ + {vis && } +
} +
+
+
; +} diff --git a/src/client/components/living/RoomReaction.tsx b/src/client/components/living/RoomReaction.tsx new file mode 100644 index 0000000..51d1838 --- /dev/null +++ b/src/client/components/living/RoomReaction.tsx @@ -0,0 +1,119 @@ +import { useLayoutEffect, useRef, type CSSProperties, type RefObject } from "react"; +import { REACTION_DURATION_MS, REACTION_GLYPHS, isThrow, type ReactionId } from "../../../shared/room-interactions"; +import type { RoomInteraction } from "../../lib/room-interactions"; +import { identityHash } from "../../lib/identity-hash"; +import { Glyph } from "../../ui/icons"; + +/** Props use the same small silhouette in the picker and on the couch. */ +export function ReactionIcon({ reaction }: { reaction: ReactionId }) { + if (!isThrow(reaction)) return ; + return ; +} + +/** Fixed, event-seeded accents: no independent timer or mutable random stream. */ +function ReactionAccents({ reaction, seed }: { reaction: ReactionId; seed: number }) { + return + {reaction === "tomato" && } + {Array.from({ length: reaction === "sleep" ? 3 : 5 }, (_, index) => { + const angle = (-160 + index * 35 + (seed % 13)) * Math.PI / 180; + const radius = 27 + ((seed >>> (index * 4)) % 12); + return + {reaction === "heart" ? + : reaction === "sleep" ? z + : reaction === "poop" ? + : reaction === "tomato" ? + : reaction === "party" ? + : } + ; + })} + ; +} + +/** Event expiry owns lifetime. This component owns only placement and its own impact. */ +export function RoomReaction({ reaction, area, now }: { + reaction: RoomInteraction & { expiresAt: number }; area: RefObject; now: () => number; +}) { + const flight = useRef(null); + const payload = reaction.payload; + useLayoutEffect(() => { + const root = area.current; + const element = flight.current; + if (!root || !element || payload.kind !== "reaction") return; + const pawn = (peer: string) => root.querySelector(`[data-room-peer="${CSS.escape(peer)}"] .lr-person`); + const sender = pawn(reaction.sender.peerId); + const recipient = pawn(payload.targetPeerId ?? reaction.sender.peerId); + if (!sender || !recipient) return; + let frame = 0; + let impact: Animation | undefined; + const place = () => { + if (now() >= reaction.expiresAt || !sender.isConnected || !recipient.isConnected) { + element.style.visibility = "hidden"; + return; + } + frame = requestAnimationFrame(place); + if (document.hidden || !root.getClientRects().length) { + element.style.visibility = "hidden"; + impact?.cancel(); + return; + } + const rect = element.getBoundingClientRect(); + // Transforms do not trigger ResizeObserver. Follow the actual head for + // this event's remaining lifetime, including hover, lean and impact. + const from = sender.querySelector(".lr-person-head")?.getBoundingClientRect(); + const to = recipient.querySelector(".lr-person-head")?.getBoundingClientRect(); + if (!from || !to) { element.style.visibility = "hidden"; return; } + element.style.setProperty("--from-x", `${from.x - rect.x + from.width / 2}px`); + element.style.setProperty("--from-y", `${from.y - rect.y + from.height / 2}px`); + element.style.setProperty("--to-x", `${to.x - rect.x + to.width / 2}px`); + element.style.setProperty("--to-y", `${to.y - rect.y + to.height / 2}px`); + element.style.setProperty("--throw-direction", from.x > to.x ? "-1" : "1"); + element.style.visibility = "visible"; + }; + place(); + const body = recipient.querySelector(".lr-reaction-body"); + if (isThrow(payload.reaction) && payload.targetPeerId && body && + !window.matchMedia("(prefers-reduced-motion: reduce)").matches && !root.closest("[data-comic-reduced-motion]")) { + const elapsed = Math.max(0, REACTION_DURATION_MS - (reaction.expiresAt - now())); + const delay = REACTION_DURATION_MS * .32 - elapsed; + if (delay > -560) { + const direction = sender.getBoundingClientRect().x > recipient.getBoundingClientRect().x ? -1 : 1; + impact = body.animate([ + { transform: "none" }, + { transform: payload.reaction === "tomato" ? `rotate(${direction * 6}deg) scale(1.05,.95)` : `translateX(${direction * 4}px) rotate(${direction * 8}deg)` }, + { transform: `rotate(${-direction * 3}deg)` }, + { transform: "none" }, + ], { duration: 560, delay, easing: "ease-out" }); + } + } + return () => { cancelAnimationFrame(frame); impact?.cancel(); }; + }, [area, reaction, payload, now]); + if (payload.kind !== "reaction") return null; + const gift = Boolean(payload.targetPeerId && !isThrow(payload.reaction)); + return ; +} diff --git a/src/client/components/living/Tooltip.tsx b/src/client/components/living/Tooltip.tsx index 75a14fb..27ae5e0 100644 --- a/src/client/components/living/Tooltip.tsx +++ b/src/client/components/living/Tooltip.tsx @@ -294,15 +294,17 @@ export function Tooltip({ if (!interactionOpen) return; // Hover does not move keyboard focus into the trigger. const dismiss = (event: KeyboardEvent) => { - if (event.key !== "Escape") return; + if (event.key !== "Escape" || event.isComposing) return; // Close every hint channel, including a hover panel suppressed by a // focused neighbour. Only visible guidance consumes the outer action. const style = tipRef.current && getComputedStyle(tipRef.current); if (style?.visibility === "visible" && style.pointerEvents !== "none") event.preventDefault(); dismissPanel(); }; - document.addEventListener("keydown", dismiss); - return () => document.removeEventListener("keydown", dismiss); + // Guidance consumes Escape before its containing window, regardless of + // which surface mounted first. Outer handlers respect defaultPrevented. + document.addEventListener("keydown", dismiss, true); + return () => document.removeEventListener("keydown", dismiss, true); }, [interactionOpen]); useEffect(() => { diff --git a/src/client/components/living/floating-panel.css b/src/client/components/living/floating-panel.css new file mode 100644 index 0000000..b17fa80 --- /dev/null +++ b/src/client/components/living/floating-panel.css @@ -0,0 +1,33 @@ +.lr-floating-surface { color: var(--ink); background: var(--paper); border: 1px solid var(--control-edge); border-radius: 24px; box-shadow: 0 12px 36px rgb(16 26 44 / .2), var(--control-shadow); } +.lr-floating-panel { position: fixed; inset: auto 12px 12px auto; max-width: calc(100vw - 24px); max-height: calc(100dvh - 24px); margin: 0; padding: 16px; overflow: auto; overscroll-behavior: contain; transform-origin: right bottom; } +.lr-floating-panel:popover-open { display: flex; flex-direction: column; animation: lr-floating-enter 180ms var(--spring); } +.lr-floating-panel:focus { outline: none; } +.lr-floating-panel:focus-visible { outline: 2px solid var(--action); outline-offset: -2px; } +.lr-floating-header { display: flex; align-items: center; gap: 8px; min-height: 56px; flex-shrink: 0; margin: -16px -16px 0; padding: 6px 10px 6px 24px; border-bottom: 1px solid var(--wall-2); border-radius: 24px 24px 0 0; cursor: grab; user-select: none; touch-action: none; } +.lr-floating-header:active { cursor: grabbing; } +.lr-floating-header:focus-visible { outline: 2px solid var(--action); outline-offset: -3px; } +.lr-floating-title { display: flex; flex: 1; min-width: 0; align-items: center; gap: 7px; font-size: 14px; } +.lr-floating-title svg { flex: none; color: var(--muted); } +.lr-floating-title b { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } +.lr-floating-position { position: relative; flex: none; } +.lr-floating-position > summary { display: flex; align-items: center; gap: 5px; min-height: 40px; padding: 0 9px; border-radius: 12px; font-size: 12px; color: var(--muted); cursor: pointer; list-style: none; } +.lr-floating-position > summary::-webkit-details-marker { display: none; } +.lr-floating-position > summary:is(:hover, :focus-visible), .lr-floating-position[open] > summary { color: var(--ink); background: var(--control-hover); } +.lr-floating-positions { position: absolute; z-index: 2; top: calc(100% + 6px); right: 0; display: grid; grid-template-columns: 1fr 1fr; gap: 6px; width: 180px; padding: 8px; border: 1px solid var(--control-edge); border-radius: 18px; background: var(--paper); box-shadow: var(--control-shadow), 0 8px 24px #0002; } +.lr-floating-positions button { display: grid; justify-items: center; gap: 6px; min-height: 60px; padding: 8px 4px; border: 0; border-radius: 10px; color: var(--ink); background: transparent; font-size: 11px; } +.lr-floating-positions button:is(:hover, :focus-visible) { color: var(--action); background: var(--control-hover); } +.lr-floating-positions .lr-floating-size { border-top: 1px solid var(--wall-2); } +.lr-floating-resize { position: absolute; z-index: 3; display: grid; place-items: center; width: 28px; height: 28px; color: var(--muted); cursor: nwse-resize; touch-action: none; user-select: none; } +.lr-floating-resize.is-nw { top: 1px; left: 1px; transform: rotate(180deg); } +.lr-floating-resize.is-se { right: 1px; bottom: 1px; } +.lr-floating-resize:is(:hover, :focus-visible) { color: var(--action); } +.lr-floating-resize:focus-visible { outline: 2px solid var(--action); outline-offset: -4px; border-radius: 10px; } +@keyframes lr-floating-enter { from { opacity: 0; transform: translateY(8px) scale(.97); } to { opacity: 1; transform: none; } } +.lr-popover-close { display: grid; place-items: center; width: 36px; height: 36px; flex: none; background: transparent; color: var(--muted); border: 0; border-radius: 12px; cursor: pointer; } +.lr-popover-close:is(:hover, :focus-visible) { background: var(--control-hover); color: var(--ink); } +[data-comic-reduced-motion] .lr-floating-panel:popover-open { animation: none; } +@media (prefers-reduced-motion: reduce) { .lr-floating-panel:popover-open { animation: none; } } +@media (max-width: 420px) { + .lr-floating-panel { padding: 12px; } + .lr-floating-header { margin: -12px -12px 0; padding: 6px 10px 6px 24px; } +} diff --git a/src/client/components/living/hints/index.tsx b/src/client/components/living/hints/index.tsx index 85474a5..5cc31be 100644 --- a/src/client/components/living/hints/index.tsx +++ b/src/client/components/living/hints/index.tsx @@ -1,13 +1,13 @@ // Control hint scenes (操作提示漫画): small explanatory scenes used // as hover/focus tooltips in every mode, with captions added by Tooltip. // Same cast, palette, 320x96 canvas, and motion -// constitution as the state comics. Sets live in set1..set4.tsx; -// each must define exactly its SetNKind keys — the merge below then proves +// constitution as the state comics. Each scene group defines exactly its +// own HintKind subset — the merge below then proves // completeness at compile time. JSX-free shell: this module is also loaded // by the tsx preview harness, which uses the classic JSX runtime. import { createElement, memo, type CSSProperties } from "react"; -import type { HintKind, HintScene, MetricHintKind } from "../../../ui/visual-kinds"; +import type { HintKind, HintScene, MetricHintKind, InteractionHintKind } from "../../../ui/visual-kinds"; import { SET1_SCENES } from "./set1"; import { SET2_SCENES } from "./set2"; import { SET3_SCENES } from "./set3"; @@ -16,6 +16,7 @@ import { PLAYBACK_SCENES } from "./playback"; import { CONTROL_SCENES } from "./controls"; import { METRIC_SCENES } from "./metrics"; import { ADMISSION_SCENES } from "./admission"; +import { INTERACTION_SCENES } from "./interactions"; import { comicStyle, getComicPresentation, type ComicTone, type ComicMotion } from "../comic-presentation"; export type { HintKind, HintScene }; @@ -99,6 +100,7 @@ export const HINT_KINDS: readonly HintKind[] = [ "hint-pip-exit", "hint-pip-unavailable", ...Object.keys(METRIC_SCENES) as MetricHintKind[], + ...Object.keys(INTERACTION_SCENES) as InteractionHintKind[], ]; export const HINT_SCENES: Record = { @@ -110,6 +112,7 @@ export const HINT_SCENES: Record = { ...CONTROL_SCENES, ...METRIC_SCENES, ...ADMISSION_SCENES, + ...INTERACTION_SCENES, }; const HINT_KIND_SET: ReadonlySet = new Set(HINT_KINDS); diff --git a/src/client/components/living/hints/interactions.tsx b/src/client/components/living/hints/interactions.tsx new file mode 100644 index 0000000..c8f330b --- /dev/null +++ b/src/client/components/living/hints/interactions.tsx @@ -0,0 +1,59 @@ +import { FAINT, Frame, MiniTv, Pawn, SKY, YOU, rmBlock } from "../Comic"; +import { Glyph } from "../../../ui/icons"; +import type { ComicTheme, HintScene, InteractionHintKind } from "../../../ui/visual-kinds"; + +function InteractionMotion() { + return ; +} + +function Bubble({ x, y, color = YOU }: { x: number; y: number; color?: string }) { + return + + + ; +} + +function ChatHint({ theme, send = false }: { theme: ComicTheme; send?: boolean }) { + return <>{[0, 160].map((x, index) => + + + + {send || index ? : } + {Boolean(index) && } + )}; +} + +// The picture and chat stay present in both panels; only on-picture text changes. +function OverlayHint({ theme, show }: { theme: ComicTheme; show: boolean }) { + return <>{[0, 160].map((x, index) => + + + + + + + + + + )}; +} + +export const INTERACTION_SCENES: Record = { + "hint-chat-open": props => , + "hint-chat-send": props => , + "hint-chat-overlay-show": props => , + "hint-chat-overlay-hide": props => , +}; diff --git a/src/client/components/living/participant-motion.ts b/src/client/components/living/participant-motion.ts index c3e4679..57d3464 100644 --- a/src/client/components/living/participant-motion.ts +++ b/src/client/components/living/participant-motion.ts @@ -1,8 +1,8 @@ +import { identityHash } from "../../lib/identity-hash"; + /** UUID selects a quiet cadence; wall time keeps independently mounted views in phase. */ export function participantMotion(identity: string) { - let seed = 2166136261; - for (const char of identity) seed = Math.imul(seed ^ char.charCodeAt(0), 16777619); - seed >>>= 0; + const seed = identityHash(identity); return { duration: 42_000 + (seed % 17_003), offset: (seed >>> 8) % 59_003, diff --git a/src/client/components/living/primitives.tsx b/src/client/components/living/primitives.tsx index 40f5289..153bb5b 100644 --- a/src/client/components/living/primitives.tsx +++ b/src/client/components/living/primitives.tsx @@ -49,6 +49,7 @@ export function Btn({ onClick, type = "button", hint, + hintText, hintTone, hintMotion, draw, @@ -68,6 +69,8 @@ export function Btn({ type?: "button" | "submit"; /** Shared hint comic; text modes add the localized caption. */ hint?: ComicKind | HintKind; + /** Optional explanation beyond the control's accessible name. */ + hintText?: CopyKey; hintTone?: ComicTone; hintMotion?: ComicMotion; /** Draw-in spot id for state-beat toggle icons (see Glyph). */ @@ -114,7 +117,7 @@ export function Btn({ ); return hint ? ( - {button} + {button} ) : ( button ); diff --git a/src/client/components/living/room-chat-overlay.css b/src/client/components/living/room-chat-overlay.css new file mode 100644 index 0000000..b1a7e25 --- /dev/null +++ b/src/client/components/living/room-chat-overlay.css @@ -0,0 +1,25 @@ +.lr-tv-screen:has(> .lr-chat-overlay) { container-type: inline-size; } +.lr-chat-overlay { position: absolute; inset: 0; overflow: hidden; pointer-events: none; container-type: size; } +.lr-tv-screen:has(> .lr-playback) > .lr-chat-overlay { grid-area: 1 / 1 / 2 / 2; } +.lr-chat-flight { + position: absolute; left: 100%; top: calc(12px + var(--chat-lane) * 30px); + max-width: 80%; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; + padding: 2px 10px; border-radius: 999px; background: rgb(15 25 35 / .68); + color: #fff; font-size: clamp(14px, 2cqw, 18px); font-weight: 600; line-height: 24px; + text-shadow: 0 1px 2px rgb(0 0 0 / .4); + animation: lr-chat-flight linear both; +} +@keyframes lr-chat-flight { to { transform: translateX(calc(-100cqw - 100%)); } } +/* Fit complete 28px lines above playback controls. Visibility preserves the + running flight when the available picture space changes. */ +@container (height < 100px) { .lr-chat-flight[data-lane="2"] { visibility: hidden; } } +@container (height < 70px) { .lr-chat-flight[data-lane="1"] { visibility: hidden; } } +@container (height < 40px) { .lr-chat-flight { visibility: hidden; } } +.lr-chat-overlay-toggle { display: inline-flex; } +@container (max-width: 440px) { + .lr-playback:has(.lr-chat-overlay-toggle) .lr-btn { width: clamp(28px, 9cqw, 40px); } +} +[data-comic-reduced-motion] .lr-chat-flight { animation: none; left: 12px; } +@media (prefers-reduced-motion: reduce) { + .lr-chat-flight { animation: none; left: 12px; } +} diff --git a/src/client/components/living/room-interactions.css b/src/client/components/living/room-interactions.css new file mode 100644 index 0000000..4251464 --- /dev/null +++ b/src/client/components/living/room-interactions.css @@ -0,0 +1,195 @@ +.lr-room-interactions { position: relative; width: min(960px, 100%); margin: 0 auto; } +.lr-interaction-couch { position: relative; } +.lr-interaction-toggle { display: flex; justify-content: center; align-items: center; gap: 2px; width: fit-content; margin: 12px auto 4px; padding: 3px; background: var(--paper); border: 1px solid var(--control-edge); border-radius: 18px; } +.lr-interaction-chat-toggle { display: flex; position: relative; } +.lr-interaction-toggle .lr-btn { min-height: 44px; } +.lr-interaction-unread { align-self: flex-start; margin: -4px 0 0 -8px; z-index: 1; min-width: 22px; padding: 2px 6px; border-radius: 20px; color: var(--paper); background: var(--action); font: 700 11px var(--mono); text-align: center; } +.lr-chat-compose, .lr-chat-footer { flex-shrink: 0; } +.lr-interaction-effects { display: flex; align-items: center; gap: 8px; min-height: 40px; width: fit-content; margin-top: 8px; font-size: 12px; font-weight: 400; color: var(--muted); } +.lr-chat-history { position: relative; flex: 1; min-height: 96px; } +.lr-chat-log { height: 100%; overflow: auto; overscroll-behavior: contain; scrollbar-width: thin; scrollbar-gutter: stable; padding: 18px 2px; } +.lr-chat-latest { position: absolute; bottom: 8px; left: 50%; translate: -50% 0; display: flex; align-items: center; gap: 6px; min-height: 36px; max-width: calc(100% - 16px); padding: 6px 12px; border: 1px solid var(--control-edge); border-radius: 18px; background: var(--paper); color: var(--ink); box-shadow: var(--control-shadow); font-size: 12px; white-space: nowrap; } +.lr-chat-latest:is(:hover, :focus-visible) { background: var(--control-hover); } +.lr-chat-empty { display: grid; place-content: center; justify-items: center; gap: 10px; height: 100%; color: var(--muted); font-size: 13px; } +.lr-chat-empty p { margin: 0; } +.lr-chat-welcome { position: relative; display: flex; gap: 4px; padding-top: 20px; } +.lr-chat-welcome > svg:last-child { position: absolute; top: 0; right: 0; color: var(--action); } +.lr-chat-message { position: relative; display: flex; padding: 0 20px 0 34px; margin: 12px 0 0; } +.lr-chat-message:first-child { margin-top: 0; } +.lr-chat-message.is-self { justify-content: flex-end; padding: 0 34px 0 20px; } +.lr-chat-message.is-continuation { margin-top: 4px; } +.lr-chat-avatar { position: absolute; top: 1px; left: 0; } +.lr-chat-avatar .lr-person { width: 28px; height: 34px; } +.lr-chat-message.is-self .lr-chat-avatar { left: auto; right: 0; } +.lr-chat-bubble { min-width: 0; } +.lr-chat-author { display: block; margin: 0 4px 5px; font-size: 11px; color: var(--muted); overflow-wrap: anywhere; } +.lr-chat-message.is-self .lr-chat-author { text-align: right; } +.lr-chat-message p { padding: 9px 13px; margin: 0; border-radius: 4px 16px 16px; background: var(--wall); overflow-wrap: anywhere; white-space: pre-wrap; font-size: 14px; line-height: 1.6; } +.lr-chat-message.is-self p { background: color-mix(in srgb, var(--action) 11%, var(--paper)); border-radius: 16px 4px 16px 16px; } +.lr-chat-message.is-continuation p { border-radius: 16px; } +.lr-chat-compose { display: flex; align-items: center; gap: 4px; margin-top: 8px; padding: 4px; border: 1px solid var(--control-edge); border-radius: 20px; background: var(--wall); } +.lr-chat-compose:has(input:focus-visible) { outline: 3px solid var(--action); outline-offset: 2px; } +.lr-chat-compose > input { min-width: 0; flex: 1; width: 0; height: 44px; border: 0; outline: none; border-radius: 14px; padding: 0 10px; background: transparent; color: var(--ink); font-size: 16px; } +.lr-chat-compose .lr-btn { flex: none; width: 44px; height: 44px; min-width: 44px; min-height: 44px; padding: 0; border-radius: 15px; } +.lr-chat-footer { display: flex; align-items: center; gap: 10px; min-height: 40px; margin-top: 8px; padding-right: 12px; color: var(--muted); font-size: 11px; line-height: 1.5; } +.lr-chat-footer .lr-chat-overlay-toggle { flex: none; } +.lr-chat-footer .lr-btn { min-height: 36px; padding: 0 8px; } +.lr-chat-footer .lr-interaction-status { flex: 1; min-width: 0; min-height: 0; margin: 0; } +.lr-chat-count { flex: none; margin-left: auto; font-variant-numeric: tabular-nums; } +.lr-interaction-status { display: flex; align-items: center; gap: 6px; min-height: 24px; margin-top: 6px; font-size: 11px; color: var(--muted); } +.lr-person-menu { transform-origin: center bottom; } +.lr-person-menu-body { flex: 1; min-height: 0; padding-top: 12px; overflow: auto; overscroll-behavior: contain; scrollbar-width: thin; } +.lr-person-menu-target { margin-bottom: 10px; } +.lr-person-menu-name { display: flex; align-items: center; gap: 10px; min-width: 0; } +.lr-person-menu-name > span:last-child { min-width: 0; } +.lr-person-menu-name small { display: block; color: var(--muted); font-size: 11px; line-height: 1.6; } +.lr-person-menu-name b { display: block; font-size: 14px; font-weight: 650; overflow-wrap: anywhere; } +.lr-person-menu-portrait { display: grid; place-items: center; width: 34px; height: 40px; flex: none; } +.lr-person-menu-portrait .lr-person { width: 30px; height: 36px; } +.lr-reaction-grid { display: grid; grid-template-columns: repeat(5, minmax(0, 1fr)); gap: 4px; } +.lr-reaction-props { display: grid; grid-template-columns: 1fr 1fr; gap: 8px; margin-top: 10px; } +:is(.lr-reaction-grid, .lr-reaction-props) button { display: flex; align-items: center; justify-content: center; gap: 6px; min-height: 44px; padding: 4px; border: 1px solid transparent; background: transparent; color: inherit; border-radius: 14px; font-size: 26px; cursor: pointer; touch-action: manipulation; transition: background-color 140ms ease, border-color 140ms ease; } +.lr-reaction-props button { background: var(--wall); } +.lr-reaction-props button > span { font-size: 12px; } +:is(.lr-reaction-grid, .lr-reaction-props) button[aria-disabled="true"] { opacity: .4; cursor: default; } +:is(.lr-reaction-grid, .lr-reaction-props) button > :first-child { transition: transform 180ms var(--spring); } +:is(.lr-reaction-grid, .lr-reaction-props) button:is(:hover, :focus-visible):not([aria-disabled="true"]) { background: var(--control-hover); border-color: var(--action); } +:is(.lr-reaction-grid, .lr-reaction-props) button:is(:hover, :focus-visible):not([aria-disabled="true"]) > :first-child { transform: translateY(-2px); } +:is(.lr-reaction-grid, .lr-reaction-props) button:active:not([aria-disabled="true"]) > :first-child { transform: scale(.9); } +.lr-person-menu-details { display: flex; align-items: center; justify-content: center; gap: 8px; width: 100%; min-height: 40px; padding: 8px; margin-top: 8px; border: 0; border-radius: 12px; background: transparent; color: var(--muted); font-size: 12px; cursor: pointer; } +.lr-person-menu-details:is(:hover, :focus-visible) { background: var(--control-hover); color: var(--ink); } +.lr-person-menu .lr-interaction-effects { font-size: 11px; } +/* Theater keeps the same top-layer chat; the picture never changes size. */ +.lr-room.is-theater .lr-room-interactions { position: absolute; top: 12px; right: 12px; z-index: 5; width: auto; } +.lr-room.is-theater .lr-interaction-toggle { margin: 0 0 0 auto; } +.lr-room.is-theater .lr-interaction-couch { display: none; } +.lr-pawn[aria-expanded="true"] { background: color-mix(in srgb, var(--paper) 72%, transparent); border-radius: 18px; } +.lr-reaction-flight { position: absolute; inset: 0; pointer-events: none; z-index: 5; overflow: visible; } +/* The path positions the center; percentage translation centers any glyph size. */ +.lr-reaction-projectile { position: absolute; left: 0; top: 0; display: flex; width: max-content; translate: -50% -50%; font-size: 30px; line-height: 1; animation: lr-reaction-flight var(--reaction-duration) ease-out both; } +.lr-reaction-glyph { display: flex; animation: lr-reaction-wiggle var(--reaction-duration) ease-out both; } +.lr-reaction-glyph > svg { width: 34px; height: 34px; } +.is-tomato .lr-reaction-glyph { animation-name: lr-reaction-squash; } +.is-poop .lr-reaction-glyph { animation-name: lr-reaction-plop; } +:is(.is-clap, .is-laugh, .is-party) .lr-reaction-glyph { animation-name: lr-reaction-bounce; } +:is(.is-heart, .is-wow, .is-fire, .is-star) .lr-reaction-glyph { animation-name: lr-reaction-pulse; } +.is-eyes .lr-reaction-glyph { animation-name: lr-reaction-peek; } +.is-sleep .lr-reaction-glyph { animation-name: lr-reaction-doze; } +.lr-reaction-accents { position: absolute; left: 50%; top: 50%; translate: -50% -50%; overflow: visible; fill: #efb94e; } +.lr-reaction-accents.is-heart { fill: #e77585; } +.lr-reaction-accents.is-tomato { fill: #e96c55; } +.lr-reaction-accents.is-poop { fill: #b09b82; opacity: .65; } +.lr-reaction-accents.is-fire { fill: #ed9b58; } +.lr-reaction-accents:is(.is-eyes, .is-sleep) { fill: var(--muted); } +.lr-reaction-accents.is-sleep { font: 700 12px var(--mono); } +.lr-reaction-accents.is-party > :nth-child(2n) { fill: #65b7aa; } +.lr-reaction-accents.is-party > :nth-child(3n) { fill: #e77585; } +.lr-reaction-spark { animation: lr-reaction-spark-self var(--reaction-duration) ease-out both; } +.is-gift .lr-reaction-spark { animation-name: lr-reaction-spark-gift; } +.is-throw .lr-reaction-spark { animation-name: lr-reaction-spark-impact; } +.is-expression .lr-reaction-projectile { animation-name: lr-reaction-expression; } +.is-gift .lr-reaction-projectile { animation-name: lr-reaction-gift; } +.is-expression .is-sleep .lr-reaction-spark { animation-name: lr-reaction-drift; } +.lr-reaction-impact { fill: none; stroke: #e96c55; stroke-width: 1.5; transform-box: fill-box; transform-origin: center; animation: lr-reaction-impact var(--reaction-duration) ease-out both; } +/* Attribution follows the glyph without shifting its center off the head. */ +.lr-reaction-from { position: absolute; bottom: calc(100% + 4px); left: 50%; translate: -50% 0; display: flex; align-items: center; gap: 4px; width: max-content; padding: 4px 6px; border: 1px solid var(--control-edge); border-radius: 10px; background: var(--paper); color: var(--ink); box-shadow: var(--control-shadow); font-size: 11px; font-weight: 600; } +.lr-reaction-from > span { max-width: 80px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } +.lr-reaction-from > svg { flex: none; color: var(--muted); } +.lr-reaction-body { transform-box: fill-box; transform-origin: center bottom; } +@keyframes lr-reaction-flight { + 0% { opacity: 0; transform: translate(var(--from-x), var(--from-y)) scale(.3); } + 10% { opacity: 1; } + 22% { transform: translate(calc((var(--from-x) + var(--to-x)) / 2), calc(min(var(--from-y), var(--to-y)) - 48px)) rotate(calc(-15deg * var(--throw-direction))) scale(1); } + 32% { transform: translate(var(--to-x), var(--to-y)) rotate(calc(8deg * var(--throw-direction))) scale(1.12, .85); } + 44% { transform: translate(var(--to-x), calc(var(--to-y) - 31px)) scale(.85); } + 56%, 74% { opacity: 1; transform: translate(var(--to-x), calc(var(--to-y) - 23px)) scale(.75); } + 100% { opacity: 0; transform: translate(var(--to-x), calc(var(--to-y) - 29px)) scale(.65); } +} +@keyframes lr-reaction-expression { + 0% { opacity: 0; transform: translate(var(--to-x), calc(var(--to-y) - 13px)) scale(.75); } + 14%, 70% { opacity: 1; transform: translate(var(--to-x), calc(var(--to-y) - 33px)) scale(1); } + 100% { opacity: 0; transform: translate(var(--to-x), calc(var(--to-y) - 45px)); } +} +@keyframes lr-reaction-gift { + 0% { opacity: 0; transform: translate(var(--from-x), calc(var(--from-y) - 28px)) scale(.7); } + 12% { opacity: 1; transform: translate(var(--from-x), calc(var(--from-y) - 40px)) scale(.9); } + 35% { transform: translate(calc((var(--from-x) + var(--to-x)) / 2), calc(min(var(--from-y), var(--to-y)) - 100px)) scale(1); } + 52%, 80% { opacity: 1; transform: translate(var(--to-x), calc(var(--to-y) - 76px)) scale(1); } + 100% { opacity: 0; transform: translate(var(--to-x), calc(var(--to-y) - 84px)) scale(.94); } +} +@keyframes lr-reaction-wiggle { + 0%, 52%, 86%, 100% { transform: none; } + 60%, 76% { transform: rotate(-12deg); } + 68% { transform: rotate(12deg); } +} +@keyframes lr-reaction-bounce { + 0%, 52%, 68%, 84%, 100% { transform: none; } + 60%, 76% { transform: translateY(-5px) scale(1.06); } +} +@keyframes lr-reaction-pulse { + 0%, 52%, 72%, 100% { transform: none; } + 62% { transform: scale(1.15); } +} +@keyframes lr-reaction-peek { + 0%, 52%, 88%, 100% { transform: none; } + 60%, 64% { transform: translateX(-3px) rotate(-4deg); } + 74%, 78% { transform: translateX(3px) rotate(4deg); } +} +@keyframes lr-reaction-doze { + 0%, 52%, 100% { transform: none; } + 68% { transform: translateY(3px) rotate(-7deg); } + 84% { transform: translateY(1px) rotate(3deg); } +} +@keyframes lr-reaction-drift { + 0%, 14% { opacity: 0; transform: translateY(6px) scale(.5); } + 30% { opacity: .7; } + 90%, 100% { opacity: 0; transform: translate(var(--spark-x), var(--spark-y)) scale(1.1); } +} +@keyframes lr-reaction-impact { + 0%, 31% { opacity: 0; transform: scale(.4); } + 34% { opacity: .55; } + 48%, 100% { opacity: 0; transform: scale(1.6); } +} +@keyframes lr-reaction-squash { + 0%, 28%, 58%, 100% { transform: none; } + 34% { transform: scale(1.35, .65) rotate(-8deg); } + 44% { transform: scale(.9, 1.1) rotate(6deg); } +} +@keyframes lr-reaction-plop { + 0%, 28%, 64%, 100% { transform: none; } + 34% { transform: scale(1.2, .8); } + 43% { transform: rotate(-14deg); } + 53% { transform: rotate(9deg); } +} +@keyframes lr-reaction-spark-self { + 0%, 13% { opacity: 0; transform: scale(0); } + 22% { opacity: .85; } + 60%, 100% { opacity: 0; transform: translate(var(--spark-x), var(--spark-y)) rotate(var(--spark-turn)); } +} +@keyframes lr-reaction-spark-gift { + 0%, 51% { opacity: 0; transform: scale(0); } + 60% { opacity: .85; } + 95%, 100% { opacity: 0; transform: translate(var(--spark-x), var(--spark-y)) rotate(var(--spark-turn)); } +} +@keyframes lr-reaction-spark-impact { + 0%, 31% { opacity: 0; transform: scale(0); } + 38% { opacity: .85; } + 72%, 100% { opacity: 0; transform: translate(var(--spark-x), var(--spark-y)) rotate(var(--spark-turn)) scale(.6); } +} +[data-comic-reduced-motion] .lr-reaction-projectile { animation: none; transform: translate(var(--to-x), calc(var(--to-y) - 33px)); } +[data-comic-reduced-motion] .is-gift .lr-reaction-projectile { transform: translate(var(--to-x), calc(var(--to-y) - 76px)); } +[data-comic-reduced-motion] .lr-reaction-body { transform: none !important; } +[data-comic-reduced-motion] .lr-reaction-glyph { animation: none; } +[data-comic-reduced-motion] .lr-reaction-accents { display: none; } +[data-comic-reduced-motion] :is(.lr-reaction-grid, .lr-reaction-props) button > :first-child { transition: none; transform: none !important; } +@media (prefers-reduced-motion: reduce) { + .lr-reaction-projectile { animation: none; transform: translate(var(--to-x), calc(var(--to-y) - 33px)); } + .is-gift .lr-reaction-projectile { transform: translate(var(--to-x), calc(var(--to-y) - 76px)); } + .lr-reaction-body { transform: none !important; } + .lr-reaction-glyph { animation: none; } + .lr-reaction-accents { display: none; } + :is(.lr-reaction-grid, .lr-reaction-props) button > :first-child { transition: none; transform: none !important; } +} +@media (max-width: 420px) { + .lr-chat-compose { gap: 6px; } +} diff --git a/src/client/components/living/use-theater-mode.ts b/src/client/components/living/use-theater-mode.ts new file mode 100644 index 0000000..e7c6e2f --- /dev/null +++ b/src/client/components/living/use-theater-mode.ts @@ -0,0 +1,22 @@ +import { useEffect, useState } from "react"; + +/** The stage owns theater presentation; room membership and drafts stay mounted. */ +export function useTheaterMode() { + const [theaterMode, setTheaterMode] = useState(false); + useEffect(() => { + if (!theaterMode) return; + const exitOnEscape = (event: KeyboardEvent) => { + // A nonmodal popover can remain open after Tab moves focus outside it; + // native Escape must dismiss that surface before leaving theater mode. + if (event.key === "Escape" && !event.defaultPrevented && !event.isComposing && + !document.querySelector('[popover="auto"]:popover-open')) setTheaterMode(false); + }; + document.body.classList.add("lr-theater-open"); + window.addEventListener("keydown", exitOnEscape); + return () => { + document.body.classList.remove("lr-theater-open"); + window.removeEventListener("keydown", exitOnEscape); + }; + }, [theaterMode]); + return [theaterMode, setTheaterMode] as const; +} diff --git a/src/client/lib/debug.ts b/src/client/lib/debug.ts index 170b50f..18e27b3 100644 --- a/src/client/lib/debug.ts +++ b/src/client/lib/debug.ts @@ -30,7 +30,7 @@ const events: Array<{ record: BrowserDebugEvent; bytes: number }> = []; const startedAt = new Date().toISOString(); const started = performance.now(); let sequence = 0, retainedBytes = 0, evictedEvents = 0, truncatedEvents = 0; -const secretKey = /(?:token|password|passwd|secret|credential|authorization|cookie|private.?key|ice.?pwd|ufrag|usernamefragment|sdp|grant)|^(?:candidate|title|displayName)$/i; +const secretKey = /(?:token|password|passwd|secret|credential|authorization|cookie|private.?key|ice.?pwd|ufrag|usernamefragment|sdp|grant)|^(?:candidate|title|displayName|text)$/i; function safeText(value: string): string { return value diff --git a/src/client/lib/identity-hash.ts b/src/client/lib/identity-hash.ts new file mode 100644 index 0000000..65acd39 --- /dev/null +++ b/src/client/lib/identity-hash.ts @@ -0,0 +1,6 @@ +/** Stable decorative choices only; not a cryptographic or authority identifier. */ +export function identityHash(identity: string): number { + let seed = 2166136261; + for (const char of identity) seed = Math.imul(seed ^ char.charCodeAt(0), 16777619); + return seed >>> 0; +} diff --git a/src/client/lib/room-interactions.ts b/src/client/lib/room-interactions.ts new file mode 100644 index 0000000..fe29185 --- /dev/null +++ b/src/client/lib/room-interactions.ts @@ -0,0 +1,169 @@ +import { interactionPayloadSchema, type ClientMessage, type ServerMessage, type InteractionPayload } from "../../shared/protocol"; +import { INTERACTION_INTERVAL_MS, REACTION_DURATION_MS } from "../../shared/room-interactions"; +import { createOpaqueId } from "./opaque-id"; +import { identityHash } from "./identity-hash"; + +export type RoomInteraction = Extract; +type Rejection = Extract["reason"]; +export type InteractionError = Rejection | "offline" | "unconfirmed"; +export const CHAT_OVERLAY_DURATION_MS = 6000; +const CHAT_OVERLAY_LANES = 3; +interface InteractionState { + ready: boolean; + peerId: string | null; + messages: (RoomInteraction & { isSelf: boolean })[]; + reactions: (RoomInteraction & { expiresAt: number })[]; + overlayEnabled: boolean; + overlayMessages: (RoomInteraction & { expiresAt: number; lane: number })[]; + pending: { requestId: string; payload: InteractionPayload } | null; + confirmed: RoomInteraction | null; + coolingDown: boolean; + error: InteractionError | null; +} +const initialState = (): InteractionState => ({ ready: false, peerId: null, messages: [], reactions: [], + overlayEnabled: false, overlayMessages: [], + pending: null, confirmed: null, coolingDown: false, error: null }); + +// One owner per signaling client. Authentication admits room data; media +// generations and route changes deliberately do not reset this state. +export class RoomInteractionSession { + // Component state follows this room session, not reconnecting wire peer IDs. + readonly key = createOpaqueId(); + private state = initialState(); + private listeners = new Set<() => void>(); + private pendingTimer: ReturnType | undefined; + private cooldownTimer: ReturnType | undefined; + private effectsTimer: ReturnType | undefined; + private overlayVisible = false; + private closed = false; + private subscriptionStartedAt: number | null = null; + private serverOffset = 0; + + constructor(private readonly sendMessage: (message: ClientMessage) => boolean, readonly now = Date.now) {} + getSnapshot = () => this.state; + subscribe = (listener: () => void) => { + this.listeners.add(listener); + return () => { this.listeners.delete(listener); }; + }; + private update(patch: Partial) { + this.state = { ...this.state, ...patch }; + this.listeners.forEach(listener => listener()); + } + + authenticated(peerId: string) { + if (this.closed) return; + this.disconnected(); + this.update({ peerId, error: this.state.error === "unconfirmed" ? "unconfirmed" : null }); + this.subscriptionStartedAt = this.now(); + this.sendMessage({ type: "subscribe-room-interactions" }); + } + + receive(message: ServerMessage): boolean { + if (this.closed) return false; + if (message.type === "room-interactions-ready") { + if (this.subscriptionStartedAt === null) return true; + // One connection-scoped midpoint estimate; arrivals never move the clock. + this.serverOffset = message.serverTime - (this.subscriptionStartedAt + this.now()) / 2; + this.subscriptionStartedAt = null; + this.update({ ready: true }); + return true; + } + if (message.type === "room-interaction-rejected") { + if (message.requestId === this.state.pending?.requestId) { + clearTimeout(this.pendingTimer); + this.update({ pending: null, error: message.reason }); + } + return true; + } + if (message.type !== "room-interaction") return false; + if (!this.state.ready) return true; + const now = this.now(); + // Clock uncertainty may put a fresh event slightly ahead. Never extend its + // lifetime beyond one full effect, and never replay an expired effect. + const beganAt = Math.min(now, message.occurredAt - this.serverOffset); + const own = message.sender.peerId === this.state.peerId; + const confirmsPending = own && message.requestId === this.state.pending?.requestId; + const confirmed = confirmsPending ? message : this.state.confirmed; + const pending = confirmsPending ? null : this.state.pending; + if (!pending) clearTimeout(this.pendingTimer); + if (message.payload.kind === "chat") { + if (this.state.messages.some(item => item.id === message.id)) return true; + let overlayMessages = this.state.overlayMessages.filter(item => item.expiresAt > now); + const lane = identityHash(`${message.sender.peerId}:${message.id}`) % CHAT_OVERLAY_LANES; + const expiresAt = beganAt + CHAT_OVERLAY_DURATION_MS; + if (this.state.overlayEnabled && this.overlayVisible && expiresAt > now) { + // Newest on this fixed lane wins; missing messages cannot shift a later + // event to another lane, and bursts cannot accumulate a playback queue. + overlayMessages = [...overlayMessages.filter(item => item.lane !== lane), { ...message, lane, expiresAt }]; + } + this.update({ messages: [...this.state.messages.slice(-79), { ...message, isSelf: own }], + overlayMessages, pending, confirmed, error: confirmsPending ? null : this.state.error }); + } else { + if (this.state.reactions.some(item => item.id === message.id)) return true; + const expiresAt = beganAt + REACTION_DURATION_MS; + const reactions = this.state.reactions.filter(item => item.expiresAt > now); + this.update({ reactions: expiresAt > now ? [...reactions.slice(-7), { ...message, expiresAt }] : reactions, + pending, confirmed, error: confirmsPending ? null : this.state.error }); + } + this.expireEffects(); + return true; + } + + send(payload: InteractionPayload): boolean { + if (this.closed || !this.state.ready || this.state.pending || this.state.coolingDown || + !interactionPayloadSchema.safeParse(payload).success) return false; + const requestId = createOpaqueId(); + try { + if (!this.sendMessage({ type: "send-room-interaction", requestId, payload })) { + this.update({ error: "offline" }); + return false; + } + } catch { + this.update({ error: "offline" }); + return false; + } + this.update({ pending: { requestId, payload }, coolingDown: true, error: null }); + this.pendingTimer = setTimeout(() => this.update({ pending: null, error: "unconfirmed" }), 5000); + this.cooldownTimer = setTimeout(() => this.update({ coolingDown: false }), INTERACTION_INTERVAL_MS); + return true; + } + + setOverlayEnabled(enabled: boolean) { + if (this.closed || this.state.overlayEnabled === enabled) return; + this.update({ overlayEnabled: enabled, overlayMessages: [] }); + this.expireEffects(); + } + + setOverlayVisible(visible: boolean) { + if (this.closed || this.overlayVisible === visible) return; + this.overlayVisible = visible; + if (!visible) this.update({ overlayMessages: [] }); + this.expireEffects(); + } + + private expireEffects() { + clearTimeout(this.effectsTimer); + const next = Math.min(...[...this.state.reactions, ...this.state.overlayMessages].map(item => item.expiresAt)); + if (!Number.isFinite(next)) return; + this.effectsTimer = setTimeout(() => { + this.update({ reactions: this.state.reactions.filter(item => item.expiresAt > this.now()), + overlayMessages: this.state.overlayMessages.filter(item => item.expiresAt > this.now()) }); + this.expireEffects(); + }, Math.max(1, next - this.now())); + } + + disconnected() { + this.subscriptionStartedAt = null; + this.serverOffset = 0; + clearTimeout(this.pendingTimer); + clearTimeout(this.cooldownTimer); + clearTimeout(this.effectsTimer); + this.update({ ready: false, reactions: [], overlayMessages: [], coolingDown: false, pending: null, confirmed: null, + error: this.state.pending ? "unconfirmed" : this.state.error }); + } + close() { + this.closed = true; + this.disconnected(); + this.update(initialState()); + } +} diff --git a/src/client/lib/signaling.ts b/src/client/lib/signaling.ts index 158e66d..d34acdb 100644 --- a/src/client/lib/signaling.ts +++ b/src/client/lib/signaling.ts @@ -5,11 +5,13 @@ import { type ClientMessage, type DisplayName, type QualitySettings, + type RoutePolicy, type ServerMessage, } from "../../shared/protocol"; import type { SignalConnectionState } from "../types"; import { qualitySettingsEqual } from "../media/quality"; import { debugEvent } from "./debug"; +import { RoomInteractionSession } from "./room-interactions"; type WithoutProtocolEnvelope = T extends { type: string; @@ -74,9 +76,10 @@ function signalUrl(): string { } export class SignalingClient { + interactions: RoomInteractionSession | null = null; private socket: WebSocket | null = null; private stopped = false; - private authenticated = false; + private authenticatedPeerId: string | null = null; private reconnectAttempt = 0; private reconnectTimer: number | null = null; private authenticationTimer: number | null = null; @@ -90,12 +93,27 @@ export class SignalingClient { private previousChallenge: PendingSignalingChallenge | null = null; private visibilityListenerAttached = false; private hostQualityIntent: HostQualityIntent | null = null; + private pendingShareGeneration: string | null = null; constructor( private readonly identity: SignalingIdentity, private readonly events: SignalingEvents, private readonly now: () => number = Date.now, - ) {} + capabilities: { roomInteractions?: boolean } = {}, + ) { + if (capabilities.roomInteractions) this.enableRoomInteractions(); + } + + // Capability discovery may finish after media authentication. Enabling room + // data must not replace the signaling connection or its healthy media. + enableRoomInteractions(): RoomInteractionSession | null { + if (this.stopped) return null; + if (!this.interactions) { + this.interactions = new RoomInteractionSession(message => this.send(message), this.now); + if (this.authenticatedPeerId) this.interactions.authenticated(this.authenticatedPeerId); + } + return this.interactions; + } start(): void { if (this.socket || this.stopped) { @@ -108,12 +126,14 @@ export class SignalingClient { } stop(): void { + this.interactions?.close(); this.stopped = true; - this.authenticated = false; + this.authenticatedPeerId = null; this.clearTimers(); this.detachVisibilityListener(); this.terminalMessage = null; this.hostQualityIntent = null; + this.pendingShareGeneration = null; const socket = this.socket; this.socket = null; if (socket && socket.readyState < WebSocket.CLOSING) { @@ -125,12 +145,14 @@ export class SignalingClient { send(message: ClientMessage): boolean { if ( - !this.authenticated || + !this.authenticatedPeerId || !this.socket || this.socket.readyState !== WebSocket.OPEN ) { return false; } + if (message.type === "send-room-interaction" && + this.socket.bufferedAmount > 16 * 1024) return false; this.socket.send(JSON.stringify(message)); debugEvent("signal", "sent", { generation: this.socketGeneration, message }); return true; @@ -166,6 +188,50 @@ export class SignalingClient { } } + ownsHostRoom(roomId: string, hostToken: string): boolean { + return this.identity.role === "host" && this.identity.roomId === roomId && this.identity.token === hostToken; + } + + wantsHostPublication(shareGeneration: string): boolean { + return this.identity.role === "host" && this.identity.roomOnly !== true && + this.identity.shareGeneration === shareGeneration; + } + + /** Keep the authenticated room session and claim a fresh media generation. */ + startSharing( + shareGeneration: string, + qualitySettings: QualitySettings, + routePolicy: RoutePolicy, + sharingPaused = false, + ): boolean { + if (this.identity.role !== "host" || this.stopped || !shareGeneration) { + return false; + } + this.identity.roomOnly = undefined; + this.identity.shareGeneration = shareGeneration; + this.identity.sharingPaused = sharingPaused; + this.identity.qualitySettings = { ...qualitySettings }; + this.identity.routePolicy = { ...routePolicy }; + this.pendingShareGeneration = shareGeneration; + const sent = this.flushHostShareIntent(); + return sent || this.authenticatedPeerId === null; + } + + /** Stop only the media publication; the room and interaction session stay alive. */ + stopSharing(shareGeneration: string): boolean { + if (this.identity.role !== "host" || this.stopped || !shareGeneration || + this.identity.shareGeneration !== shareGeneration) { + return false; + } + const sent = this.send({ type: "stop-sharing", shareGeneration }); + this.identity.shareGeneration = undefined; + this.identity.sharingPaused = false; + this.identity.roomOnly = true; + this.pendingShareGeneration = null; + this.hostQualityIntent = null; + return sent; + } + confirmSharingPaused(): void { if (this.identity.role === "host") { this.identity.sharingPaused = true; @@ -237,7 +303,7 @@ export class SignalingClient { }; socket.send(JSON.stringify(authenticate)); this.authenticationTimer = window.setTimeout(() => { - if (!this.authenticated && this.socket === socket) { + if (!this.authenticatedPeerId && this.socket === socket) { socket.close(4000, "authentication timeout"); } }, 8_000); @@ -275,7 +341,7 @@ export class SignalingClient { if ( message.type === "error" && message.code === "INVALID_MESSAGE" && - !this.authenticated + !this.authenticatedPeerId ) { this.terminateForProtocolMismatch(); return; @@ -291,7 +357,7 @@ export class SignalingClient { } if (message.type === "authenticated") { - this.authenticated = true; + this.authenticatedPeerId = message.peerId; this.reconnectAttempt = 0; this.clearAuthenticationTimer(); if (this.terminalMessage) { @@ -305,7 +371,21 @@ export class SignalingClient { this.events.onStatus("connected"); debugEvent("signal", "authenticated", { role: this.identity.role, generation }); this.refreshSignalingWatchdog(); + this.interactions?.authenticated(message.peerId); + this.reconcileHostShareState(message); } + if ((message.type === "sharing-started" || message.type === "sharing-start-failed") && + this.pendingShareGeneration === message.shareGeneration) { + this.pendingShareGeneration = null; + } + if (message.type === "sharing-start-failed" && this.identity.role === "host" && + this.identity.shareGeneration === message.shareGeneration) { + this.identity.shareGeneration = undefined; + this.identity.sharingPaused = false; + this.identity.roomOnly = true; + this.hostQualityIntent = null; + } + if (this.interactions?.receive(message)) return; if ( message.type === "host-status" && this.identity.role === "host" && @@ -328,7 +408,8 @@ export class SignalingClient { return; } this.socket = null; - this.authenticated = false; + this.authenticatedPeerId = null; + this.interactions?.disconnected(); debugEvent("signal", "closed", { role: this.identity.role, code: event.code, reconnecting: !this.stopped && shouldReconnectSignaling(event.code) }); this.clearSignalingWatchdog(); @@ -392,6 +473,84 @@ export class SignalingClient { return true; } + private flushHostShareIntent(): boolean { + const shareGeneration = this.pendingShareGeneration; + if ( + !shareGeneration || + this.identity.role !== "host" || + this.identity.shareGeneration !== shareGeneration + ) { + return false; + } + try { + if (!this.send({ + type: "start-sharing", + shareGeneration, + sharingPaused: this.identity.sharingPaused, + qualitySettings: this.identity.qualitySettings, + routePolicy: this.identity.routePolicy, + })) { + return false; + } + return true; + } catch { + return false; + } + } + + private reconcileHostShareState( + message: Extract, + ): void { + if (this.identity.role !== "host" || this.identity.roomSession !== true) { + return; + } + const desiredGeneration = this.identity.shareGeneration; + if (this.identity.roomOnly === true) { + this.pendingShareGeneration = null; + if (message.shareGeneration !== null) { + // stopSharing may have been requested while authentication was still + // in flight. The authenticated reply is the first point where the + // server can accept the idempotent stop for that session. + this.send({ + type: "stop-sharing", + shareGeneration: message.shareGeneration, + }); + } + return; + } + if (!desiredGeneration) { + this.pendingShareGeneration = null; + return; + } + if (message.shareGeneration === desiredGeneration) { + // Authentication already claimed this generation; sending a second + // start-sharing would be rejected as a duplicate publication. + this.pendingShareGeneration = null; + return; + } + if (message.shareGeneration === null) { + this.flushHostShareIntent(); + return; + } + if (this.pendingShareGeneration === desiredGeneration) { + // A stop/start can cross the authentication response: the socket was + // authenticated with the old generation, while the UI already owns a + // fresh one. WebSocket ordering makes this pair one serialized server + // transition; keep the new intent and retire the stale publication. + if (this.send({ + type: "stop-sharing", + shareGeneration: message.shareGeneration, + })) { + this.flushHostShareIntent(); + } + return; + } + // Without a pending local intent, the server's active generation is + // authoritative. Do not retry an intent that no longer describes this + // authenticated session. + this.pendingShareGeneration = null; + } + private reconcileHostQualityIntent( message: Extract, ): void { @@ -521,7 +680,8 @@ export class SignalingClient { return; } this.socket = null; - this.authenticated = false; + this.authenticatedPeerId = null; + this.interactions?.disconnected(); this.clearAuthenticationTimer(); this.clearSignalingWatchdog(); this.events.onStatus("reconnecting"); @@ -573,7 +733,7 @@ export class SignalingClient { private isWatchdogEligible(): boolean { return ( - this.authenticated && + this.authenticatedPeerId !== null && this.isDocumentVisible() && this.socket?.readyState === WebSocket.OPEN ); diff --git a/src/client/locales/en.ts b/src/client/locales/en.ts index 211b668..e400d2e 100644 --- a/src/client/locales/en.ts +++ b/src/client/locales/en.ts @@ -118,6 +118,60 @@ export const en: Record = { "host.device.listFailed": "Could not load devices. Try refreshing.", "host.device.permission.microphone": "Turn on the microphone and allow access to see the available devices here.", "host.microphone.volume": "Microphone volume", + "interaction.open": "Chat", + "interaction.title": "Chat", + "interaction.close": "Hide chat window", + "floating.move": "Move window", + "floating.moveHint": "Drag the title bar, use arrow keys to adjust, or choose a corner under Position.", + "floating.position": "Position", + "floating.resize": "Resize window", + "floating.resizeHint": "Drag a corner or use arrow keys; preset sizes are under Position.", + "floating.size.default": "Default size", + "floating.size.large": "Large size", + "floating.position.topLeft": "Top left", + "floating.position.topRight": "Top right", + "floating.position.bottomLeft": "Bottom left", + "floating.position.bottomRight": "Bottom right", + "interaction.overlay.label": "On screen", + "interaction.overlay.show": "Show chat on screen", + "interaction.overlay.hide": "Hide chat on screen", + "interaction.overlay.showHint": "Show new messages over the picture, just for you.", + "interaction.overlay.hideHint": "Hide on-screen messages. They remain in the chat window.", + "interaction.effects": "Show reaction effects", + "interaction.ephemeral": "Messages stay on this page and clear when you refresh.", + "interaction.empty": "Make yourself at home. Say hello.", + "interaction.prop.tomato": "Toss tomato", + "interaction.prop.poop": "Toss poop", + "interaction.placeholder": "Say something…", + "interaction.send": "Send", + "interaction.reactions": "Reactions", + "interaction.express": "React", + "interaction.toPerson": "To", + "interaction.withPerson": "Interact with {name}", + "interaction.participant": "a room participant", + "interaction.receivedReaction": "{sender} to {target}: {reaction}.", + "interaction.expressedReaction": "{sender}: {reaction}.", + "interaction.unread": "{count} new messages", + "interaction.latest": "Back to latest", + "interaction.reconnecting": "Connect to send messages and reactions", + "interaction.error.offline": "Could not send. Try again shortly.", + "interaction.error.unconfirmed": "No confirmation from the server yet.", + "interaction.error.rate-limited": "A little too fast. Try again in a moment.", + "interaction.error.target-unavailable": "They just left. Try another person.", + "interaction.error.not-subscribed": "Reactions and chat are unavailable. Rejoin the room.", + "interaction.error.busy": "The connection is busy. Try again shortly.", + "interaction.reaction.wave": "Wave", + "interaction.reaction.heart": "Love it", + "interaction.reaction.clap": "Applause", + "interaction.reaction.laugh": "Laugh", + "interaction.reaction.wow": "Mind blown", + "interaction.reaction.party": "Celebrate", + "interaction.reaction.fire": "On fire", + "interaction.reaction.eyes": "Watching", + "interaction.reaction.star": "Gold star", + "interaction.reaction.sleep": "Sleepy", + "interaction.reaction.tomato": "Throw a tomato", + "interaction.reaction.poop": "Throw some poop", "host.microphone.mute": "Mute microphone", "host.microphone.pending": "Opening microphone", "host.microphone.browserOnly": "This capture method does not support a microphone yet. Choose Browser or Camera.", @@ -133,6 +187,11 @@ export const en: Record = { "host.sourcePicker.tab.display": "Screens", "host.sourcePicker.windowAudio": "Application sound", "host.sourcePicker.systemAudio": "System sound", + "host.sourcePicker.excludeAudio": "Exclude application audio", + "host.sourcePicker.excludeNone": "None", + "host.sourcePicker.excludeMissing": "Not listed; select again", + "host.sourcePicker.excludeAudioChoose": "Choose an application window to exclude. If it is missing, open its window and refresh.", + "host.sourcePicker.excludeAudioHint": "Share audio from all output devices except the selected process and its children. If it exits, source audio stops until you select again.", "host.sourcePicker.showCaptureBorder": "Show capture border", "host.sourcePicker.showCaptureBorderHint": "Turn off to hide the border when Windows allows it. Other captures may keep it visible.", "host.sourcePicker.empty": "No sources available", @@ -149,8 +208,8 @@ export const en: Record = { "host.sourcePicker.system": "System screen or window picker", "host.sourcePicker.systemAction": "Choose a screen or window with the system picker", "host.sourcePicker.audioHint": "Application sound for windows; system sound for screens", - "host.sourcePicker.refresh": "Refresh screens and windows", - "host.sourcePicker.loading": "Finding local screens and windows", + "host.sourcePicker.refresh": "Refresh sharing sources", + "host.sourcePicker.loading": "Finding sharing sources", "host.join": "Join a room", "host.starting": "Connecting", "host.cancelStart": "Cancel", @@ -241,6 +300,7 @@ export const en: Record = { "host.quality.1080p60": "1080p\u00a0· 60\u00a0fps", "host.advanced": "Sharing settings", "host.settings.button": "Settings", + "host.settings.picture": "Picture", "host.settings.sound": "Sound", "host.settings.technical": "Connection & encoding", @@ -259,6 +319,8 @@ export const en: Record = { "host.advanced.audio.music": "Music", "host.advanced.audio.veryHigh": "Studio", "host.advanced.route": "Route policy", + "host.advanced.route.sfuOnly": "Server media", + "host.advanced.route.sfuOnlyHint": "This site sends all shared video and audio through its media server", "host.advanced.route.topo": "Auto topology tuning", "host.advanced.route.topoHint": "Gradually pick healthier paths while live", "host.advanced.route.peerOnly": "Privacy mode", diff --git a/src/client/locales/zh.ts b/src/client/locales/zh.ts index dd0c4f0..3346dfe 100644 --- a/src/client/locales/zh.ts +++ b/src/client/locales/zh.ts @@ -118,6 +118,60 @@ export const zh = { "host.device.listFailed": "暂时无法获取设备,请刷新重试。", "host.device.permission.microphone": "开启麦克风并允许访问后,可以在这里选择设备。", "host.microphone.volume": "麦克风音量", + "interaction.open": "聊天", + "interaction.title": "聊天", + "interaction.close": "收起聊天窗口", + "floating.move": "移动窗口", + "floating.moveHint": "拖动标题栏移动窗口;方向键可微调,也可以从「位置」选择角落。", + "floating.position": "位置", + "floating.resize": "调整窗口大小", + "floating.resizeHint": "拖动角落或按方向键调整大小,也可以从「位置」选择预设尺寸。", + "floating.size.default": "默认大小", + "floating.size.large": "大尺寸", + "floating.position.topLeft": "左上角", + "floating.position.topRight": "右上角", + "floating.position.bottomLeft": "左下角", + "floating.position.bottomRight": "右下角", + "interaction.overlay.label": "弹幕", + "interaction.overlay.show": "显示弹幕", + "interaction.overlay.hide": "隐藏弹幕", + "interaction.overlay.showHint": "在画面上显示新消息,仅对自己生效。", + "interaction.overlay.hideHint": "隐藏画面上的消息,聊天窗口仍可查看。", + "interaction.effects": "显示互动动效", + "interaction.ephemeral": "仅保留本页收到的消息,刷新后清空。", + "interaction.empty": "坐这儿,聊两句。", + "interaction.prop.tomato": "丢番茄", + "interaction.prop.poop": "丢便便", + "interaction.placeholder": "说点什么…", + "interaction.send": "发送", + "interaction.reactions": "表情与小道具", + "interaction.express": "发表情", + "interaction.toPerson": "送给", + "interaction.withPerson": "与 {name} 互动", + "interaction.participant": "房间成员", + "interaction.receivedReaction": "{sender} 对 {target} 使用了「{reaction}」。", + "interaction.expressedReaction": "{sender}:{reaction}。", + "interaction.unread": "{count} 条新消息", + "interaction.latest": "回到最新消息", + "interaction.reconnecting": "连接后即可发送", + "interaction.error.offline": "暂时未能发送,请稍后重试。", + "interaction.error.unconfirmed": "暂未收到发送确认。", + "interaction.error.rate-limited": "发得有点快,稍等一下再试。", + "interaction.error.target-unavailable": "对方刚刚离开了,换个人试试。", + "interaction.error.not-subscribed": "互动暂不可用,请重新进入房间。", + "interaction.error.busy": "连接正忙,稍后再试。", + "interaction.reaction.wave": "打个招呼", + "interaction.reaction.heart": "比个心", + "interaction.reaction.clap": "鼓掌", + "interaction.reaction.laugh": "笑出声", + "interaction.reaction.wow": "震惊", + "interaction.reaction.party": "撒花", + "interaction.reaction.fire": "太燃了", + "interaction.reaction.eyes": "围观", + "interaction.reaction.star": "亮个星", + "interaction.reaction.sleep": "困了", + "interaction.reaction.tomato": "丢番茄", + "interaction.reaction.poop": "丢便便", "host.microphone.mute": "关闭麦克风", "host.microphone.pending": "正在开启麦克风", "host.microphone.browserOnly": "此采集方式暂不支持麦克风,可改用浏览器或摄像头。", @@ -133,6 +187,11 @@ export const zh = { "host.sourcePicker.tab.display": "屏幕", "host.sourcePicker.windowAudio": "应用声音", "host.sourcePicker.systemAudio": "系统声音", + "host.sourcePicker.excludeAudio": "排除应用声音", + "host.sourcePicker.excludeNone": "不排除", + "host.sourcePicker.excludeMissing": "不在列表中,请重新选择", + "host.sourcePicker.excludeAudioChoose": "选择要排除的应用窗口;未列出的应用请先打开窗口,再刷新。", + "host.sourcePicker.excludeAudioHint": "分享所有输出设备的声音,但不包含所选进程及其子进程。应用退出后来源声音会停止,需重新选择。", "host.sourcePicker.showCaptureBorder": "显示采集边框", "host.sourcePicker.showCaptureBorderHint": "关闭后尝试隐藏黄框;Windows 或其他采集程序可能仍要求显示。", "host.sourcePicker.empty": "暂无可用来源", @@ -149,8 +208,8 @@ export const zh = { "host.sourcePicker.system": "系统屏幕与窗口选择器", "host.sourcePicker.systemAction": "使用系统选择器选择屏幕或窗口", "host.sourcePicker.audioHint": "窗口分享应用声音,整屏分享系统声音", - "host.sourcePicker.refresh": "刷新屏幕和窗口", - "host.sourcePicker.loading": "正在查找本机屏幕和窗口", + "host.sourcePicker.refresh": "刷新分享来源", + "host.sourcePicker.loading": "正在查找分享来源", "host.join": "加入房间", "host.starting": "正在连接", "host.cancelStart": "取消", @@ -241,6 +300,7 @@ export const zh = { "host.quality.1080p60": "1080p\u00a0· 60\u00a0帧", "host.advanced": "分享设置", "host.settings.button": "设置", + "host.settings.picture": "画面", "host.settings.sound": "声音", "host.settings.technical": "连接与编码", @@ -259,6 +319,8 @@ export const zh = { "host.advanced.audio.music": "音乐", "host.advanced.audio.veryHigh": "保真", "host.advanced.route": "路由策略", + "host.advanced.route.sfuOnly": "服务器转发", + "host.advanced.route.sfuOnlyHint": "此站点的共享画面与声音统一由媒体服务器转发", "host.advanced.route.topo": "自动优化拓扑", "host.advanced.route.topoHint": "观看中逐步选择更健康的连接", "host.advanced.route.peerOnly": "隐私模式", diff --git a/src/client/media/browser-encoding-output.ts b/src/client/media/browser-encoding-output.ts index de03a0d..40a7490 100644 --- a/src/client/media/browser-encoding-output.ts +++ b/src/client/media/browser-encoding-output.ts @@ -26,7 +26,7 @@ export interface BrowserEncodingOutputSnapshot { } type Queue = { producerId: string; frames: RTCEncodedVideoFrame[]; needKey: boolean; requestKey: () => Promise }; -type Selection = { queue: Queue; onSelected: () => void }; +type Selection = { queue: Queue; onSelected: () => void; accept: (metadata: RTCEncodedVideoFrameMetadata) => boolean }; // Bound pending encoded data; a lost dependency chain resumes on a fresh key. const MAX_QUEUED_FRAMES = 4; @@ -46,7 +46,7 @@ export class BrowserEncodingOutput { private readonly writer: WritableStreamDefaultWriter; private current: Queue | undefined; private pending: Selection | undefined; - private carrier: RTCEncodedVideoFrame | undefined; + private carrier: { frame: RTCEncodedVideoFrame; recovery: boolean } | undefined; private writing: Promise | undefined; private ownKey: (() => Promise) | undefined; private raw: false | "key" | true = false; @@ -80,15 +80,25 @@ export class BrowserEncodingOutput { }); } - select(producerId: string, requestKey: () => Promise, onSelected: () => void): void { + select(producerId: string, requestKey: () => Promise, onSelected: () => void, + accept: Selection["accept"]): void { if (this.abort.signal.aborted) return; this.epoch++; this.carrier = undefined; - this.pending = { queue: { producerId, frames: [], needKey: true, requestKey }, onSelected }; + this.pending = { queue: { producerId, frames: [], needKey: true, requestKey }, onSelected, accept }; debugEvent("encoding-pool", "selection-requested", { ...this.identity, producerId }); if (!this.paused) this.request(this.pending.queue); } + cancelSelection(producerId: string): boolean { + if (this.pending?.queue.producerId !== producerId) return true; + // Once an accepted key starts writing, that queue owns the wire. Keep its + // pool reference until completion, including across pause/resume. + if (this.current === this.pending.queue) return false; + this.pending = undefined; + return true; + } + push(producerId: string, frame: RTCEncodedVideoFrame): void { if (this.abort.signal.aborted || this.paused || frame.data.byteLength === 0) return; try { @@ -152,30 +162,37 @@ export class BrowserEncodingOutput { const epoch = this.epoch; await this.writing; if (this.abort.signal.aborted || this.paused || this.epoch !== epoch) return; - if (this.raw && !this.pending?.queue.frames[0]) { - if (this.raw === "key" && frame.type !== "key") return; - await this.write(frame, null); - return; - } } - // A native carrier key can be a remote PLI. Recover the selected real - // stream, without requiring its frame type to match this clock frame. - if (frame.type === "key" && this.current && this.pending?.queue.frames[0]?.type !== "key" && - this.current.frames[0]?.type !== "key") this.recover(this.current); - this.carrier = frame; + // Coalescing keeps the newest clock, but must not overwrite a recovery + // request received while the previous output write is still pending. + this.carrier = { frame, recovery: frame.type === "key" || this.carrier?.recovery === true }; const writing = this.drain(); if (this.raw) await writing; } private drain(): Promise | undefined { if (this.abort.signal.aborted || this.paused || this.writing || !this.carrier) return; - const selection = this.pending?.queue.frames[0]?.type === "key" ? this.pending : undefined; - if (this.raw && !selection) return; - const queue = selection?.queue ?? this.current, frame = queue?.frames.shift(); - if (!queue || !frame) return; - const own = this.carrier; - this.carrier = undefined; try { + let selection = this.pending?.queue.frames[0]?.type === "key" ? this.pending : undefined; + // Preparation is not commitment: validate the actual key while the old + // stream is still intact, before consuming either queue or writing bytes. + if (selection && selection.queue !== this.current && !selection.accept(selection.queue.frames[0]!.getMetadata())) { + debugEvent("encoding-pool", "selection-rejected", { ...this.identity, producerId: selection.queue.producerId }); + this.pending = undefined; + selection = undefined; + } + const { frame: own, recovery } = this.carrier; + // A carrier key can be a remote PLI. Only an accepted candidate key can + // replace the current stream's recovery; a rejected one proves nothing. + if (recovery && !selection && this.current && this.current.frames[0]?.type !== "key") this.recover(this.current); + if (this.raw && !selection) { + this.carrier = undefined; + if (this.raw === "key" && own.type !== "key") return; + return this.write(own, null); + } + const queue = selection?.queue ?? this.current, frame = queue?.frames.shift(); + if (!queue || !frame) return; + this.carrier = undefined; const copy = copyFrame(frame, own.timestamp); this.current = queue; this.raw = false; diff --git a/src/client/media/browser-encoding-pool.ts b/src/client/media/browser-encoding-pool.ts index 7fa1c06..97a92c1 100644 --- a/src/client/media/browser-encoding-pool.ts +++ b/src/client/media/browser-encoding-pool.ts @@ -98,6 +98,54 @@ export class BrowserEncodingPool { return [...this.members].filter((member) => member.current === group || member.pending?.group === group); } + private demand(group: Group, except?: Member): number | undefined { + const budgets = this.references(group).flatMap((member) => member !== except && + this.compatible(member, group) && member.budget !== undefined ? [member.budget] : []); + return budgets.length ? Math.max(...budgets) : undefined; + } + + private hasHigherDemand(member: Member, group: Group): boolean { + return (this.demand(group) ?? 0) > member.budget!; + } + + private canJoin(member: Member, group: Group): boolean { + // A pending newcomer cannot prove compatibility with its own demand or + // raise the incumbent owner's allocation merely to reuse its encoder. + const demand = this.demand(group, member); + return demand === undefined || demand === member.budget; + } + + private canReuse(member: Member, group: Group, current: Group): boolean { + const output = group.output, previous = current.output; + // A quiet scene does not make a higher rate owner compatible. Check both + // allocations before pending membership can raise the candidate's budget. + return this.compatible(member, group) && this.canJoin(member, group) && + group.budget <= member.budget! && group.budget >= current.budget && + fits(output, member.budget) && output?.reason === "none" && !!previous && + noRegression(output, previous.width, previous.height, previous.fps, member.profile); + } + + private acceptsFrame(member: Member, group: Group, metadata: RTCEncodedVideoFrameMetadata): boolean { + if (member.disabled || member.paused || !this.compatible(member, group) || group.updating || + member.budget === undefined || group.budget > member.budget || !this.canJoin(member, group)) return false; + const current = member.current; + const optional = current && this.compatible(member, current) && + !this.hasHigherDemand(member, current); + // Initial output and necessary downgrades have no healthy picture to beat. + if (current ? !optional : member.carrier) return true; + const actual = member.encoded.snapshot(); + if (actual.lastProducerId !== (current?.producer.id ?? null)) return false; + if (!metadata.width || !metadata.height || actual.width === null || actual.height === null || + metadata.width < actual.width || metadata.height < actual.height) { + // A real key can disprove the last statistics sample. Do not let another + // reconciliation reuse that sample before the normal poll replaces it. + group.output = undefined; + return false; + } + return current ? this.canReuse(member, group, current) : !!group.output && + noRegression(group.output, actual.width, actual.height, member.raw?.framesPerSecond ?? null, member.profile); + } + private async poll(): Promise { if (this.polling || this.disposed) return; this.polling = true; @@ -155,6 +203,12 @@ export class BrowserEncodingPool { } private reconcile(): void { + // Remove obsolete requests before they can displace a current member in + // this pass. Applied rates may still belong to the previous stats sample. + for (const member of this.members) { + if (member.pending && member.budget !== undefined && !this.canJoin(member, member.pending.group) && + member.encoded.cancelSelection(member.pending.group.producer.id)) member.pending = undefined; + } const changing = new Set(); for (const group of this.groups) { const references = this.references(group); @@ -186,12 +240,12 @@ export class BrowserEncodingPool { // A shared producer can outgrow a waiting child's native allocation. // Replan that membership; never wait forever for its budget to catch up. if (!pending.selecting && pending.group.budget > member.budget && - !fits(pending.group.output, member.budget) && this.references(pending.group).some((other) => other !== member)) { + this.references(pending.group).some((other) => other !== member)) { member.pending = undefined; continue; } if (!pending.selecting && member.current && this.compatible(member, member.current) && - fits(member.current.output, member.budget) && pending.group.budget < member.current.budget) { + !this.hasHigherDemand(member, member.current) && !this.canReuse(member, pending.group, member.current)) { member.pending = undefined; continue; } @@ -223,6 +277,12 @@ export class BrowserEncodingPool { if (!member.disposed && !applied) this.failSource(member.source); }).catch(() => this.failSource(member.source)).finally(() => { member.replacement = undefined; }); this.prune(); + }, (metadata) => { + if (member.disposed || member.pending !== pending) return false; + if (this.acceptsFrame(member, pending.group, metadata)) return true; + member.pending = undefined; + this.prune(); + return false; }); } } @@ -237,10 +297,10 @@ export class BrowserEncodingPool { let earlier = true; for (const group of this.groups) { if (group === current) { earlier = false; continue; } - const improves = group.output && current.output && (group.output.width > current.output.width || - group.output.height > current.output.height || Math.round(group.output.fps) > Math.round(current.output.fps)); - if ((earlier || improves) && this.compatible(member, group) && fits(group.output, member.budget) && group.output?.reason === "none" && - noRegression(group.output!, current.output.width, current.output.height, current.output.fps, member.profile)) { + // A newer output must improve within the same Host ceilings used below. + const improves = group.output && current.output && !noRegression(current.output, + group.output.width, group.output.height, group.output.fps, member.profile); + if ((earlier || improves) && this.canReuse(member, group, current)) { member.pending = { group }; break; } } @@ -248,22 +308,18 @@ export class BrowserEncodingPool { } // A short output-rate spike is not new demand. Keep the rate owner and // let its native encoder adapt unless another child needs a higher rate. - if (current && this.compatible(member, current) && !this.references(current).some((other) => - this.compatible(other, current) && other.budget !== undefined && other.budget > member.budget!)) continue; - const candidates = [...this.groups].filter((group) => this.compatible(member, group)); - let target = candidates.find((group) => group !== current && fits(group.output, member.budget) && - (!current || group.budget === member.budget)); - target ??= candidates.find((group) => group !== current && group.budget === member.budget); + if (current && this.compatible(member, current) && !this.hasHigherDemand(member, current)) continue; + const candidates = [...this.groups].filter((group) => group !== current && this.compatible(member, group) && + this.demand(group) === member.budget); + let target = candidates.find((group) => fits(group.output, member.budget)) ?? candidates[0]; if (!target) { target = this.startGroup(member, member.budget); } member.pending = { group: target }; } for (const group of this.groups) { - const references = this.references(group); - const budgets = references.flatMap((member) => this.compatible(member, group) && member.budget !== undefined ? [member.budget] : []); - if (group.ready && budgets.length) { - const budget = Math.max(...budgets); + const budget = this.demand(group); + if (group.ready && budget !== undefined) { if (budget !== group.budget && !group.updating) { group.budget = budget; group.updating = group.producer.update(group.profile, budget).catch(() => { diff --git a/src/client/media/browser-encoding-producer.ts b/src/client/media/browser-encoding-producer.ts index 5af13c9..799bc7a 100644 --- a/src/client/media/browser-encoding-producer.ts +++ b/src/client/media/browser-encoding-producer.ts @@ -1,3 +1,4 @@ +import { parse, write } from "sdp-transform"; import { createOpaqueId } from "../lib/opaque-id"; import { debugError, debugEvent } from "../lib/debug"; import { debugRtcFailure, debugRtcStats, debugTrack, observeDebugConnection } from "../lib/debug-webrtc"; @@ -119,7 +120,7 @@ export class BrowserEncodingProducer { await receive.setRemoteDescription(send.localDescription!); await flushReceive(); this.checkAlive(); - await receive.setLocalDescription(await receive.createAnswer()); + await receive.setLocalDescription(this.seedLocalAnswer(await receive.createAnswer())); this.checkAlive(); await send.setRemoteDescription(receive.localDescription!); await flushSend(); @@ -217,6 +218,25 @@ export class BrowserEncodingProducer { return this.videoSender; } + private seedLocalAnswer(answer: RTCSessionDescriptionInit): RTCSessionDescriptionInit { + const kbps = Math.floor(Math.min(this.budget, this.desiredProfile.maxBitrate) / 1_000); + if (!Number.isFinite(kbps) || kbps <= 0 || !answer.sdp) return answer; + const session = parse(answer.sdp); + const media = session.media.find((section) => section.type === "video" && section.port !== 0); + const codec = media?.rtp.find((entry) => `video/${entry.codec}`.toLowerCase() === this.codec.mimeType.toLowerCase()); + if (!media || !codec) return answer; + const format = media.fmtp.find((entry) => entry.payload === codec.payload); + const parameters = (format?.config ?? "").split(";").map((part) => part.trim()) + .filter((part) => part && !/^x-google-start-bitrate\s*=/i.test(part)); + parameters.push(`x-google-start-bitrate=${kbps}`); + if (format) format.config = parameters.join(";"); + else media.fmtp.push({ payload: codec.payload, config: parameters.join(";") }); + // This same-Browser encoder already has a downstream native allocation. + // WebRTC reads send rates from the remote answer; an offer hint is dropped. + // Seed once, without a minimum or an external SDP/codec-identity change. + return { ...answer, sdp: write(session) }; + } + private async applyCurrent(): Promise { const sender = this.requireSender(), track = this.input!; const profile = this.desiredProfile; diff --git a/src/client/native/client.ts b/src/client/native/client.ts index f9c50e6..4a12986 100644 --- a/src/client/native/client.ts +++ b/src/client/native/client.ts @@ -74,6 +74,7 @@ export interface NativeShareInput { shareId: string; source: NativeCaptureTarget; audio: boolean; + excludeAudio?: NativeCaptureTarget; showCaptureBorder?: boolean; adapterIndex: number; encoderIndex: number; @@ -281,11 +282,13 @@ export class NativeClient { input: NativeShareInput, ): Promise<{ audio: boolean; codec: NativeVideoCodec; sourceAudio?: boolean }> { // Older Apps reject unknown command fields. - const { showCaptureBorder = false, ...shareInput } = input; + const { showCaptureBorder = false, excludeAudio, ...shareInput } = input; + this.validateAudioExclusion(excludeAudio); const response = await this.request( "start-share", { ...shareInput, + ...(excludeAudio ? { excludeAudio } : {}), ...(this.health.nativeMedia.microphone ? { microphoneMixing: true } : {}), ...(this.health.nativeMedia.captureBorderControl ? { showCaptureBorder } : {}), }, @@ -310,19 +313,28 @@ export class NativeClient { } } + private validateAudioExclusion(target: NativeCaptureTarget | undefined): void { + if (target && !this.health.nativeMedia.processAudioExclusion) { + throw new Error("Piik App does not support audio exclusion"); + } + } + async replaceShareSource( shareId: string, source: NativeCaptureTarget, audio: boolean, path: NativeCapturePath, showCaptureBorder = false, + excludeAudio?: NativeCaptureTarget, ): Promise { + this.validateAudioExclusion(excludeAudio); const response = await this.request( "replace-share-source", { shareId, source, audio, + ...(excludeAudio ? { excludeAudio } : {}), adapterIndex: path.adapterIndex, encoderIndex: path.encoderIndex, ...(this.health.nativeMedia.captureBorderControl ? { showCaptureBorder } : {}), diff --git a/src/client/native/wire.ts b/src/client/native/wire.ts index 4326d5d..8f4542a 100644 --- a/src/client/native/wire.ts +++ b/src/client/native/wire.ts @@ -36,6 +36,7 @@ export const nativeHealthSchema = nativeDiscoveryIdentitySchema.extend({ video: z.boolean().default(false), processAudio: z.boolean().default(false), systemAudio: z.boolean().default(false), + processAudioExclusion: z.boolean().default(false), microphone: z.boolean().default(false), // API availability, not Windows approval or observed border visibility. captureBorderControl: z.boolean().default(false), diff --git a/src/client/pages/AppLauncherPage.tsx b/src/client/pages/AppLauncherPage.tsx index 142498b..d227b00 100644 --- a/src/client/pages/AppLauncherPage.tsx +++ b/src/client/pages/AppLauncherPage.tsx @@ -131,7 +131,7 @@ export function AppLauncherPage() { const updateText = update ? `${t(updateKey)} · ${update.version}` : ""; const updateLink = update ? ( ')}`; -const SOURCES = { kind: "ready", processAudio: true, systemAudio: true, captureBorderControl: true, sources: [ +const SOURCES = { kind: "ready", processAudio: true, systemAudio: true, processAudioExclusion: true, captureBorderControl: true, sources: [ { kind: "window", sourceId: "101", pid: 1, creationTime: "1", title: "Sketchbook" }, { kind: "window", sourceId: "102", pid: 2, creationTime: "2", title: "A short film" }, + { kind: "window", sourceId: "104", pid: 3, creationTime: "3", title: "Voice call" }, { kind: "display", sourceId: "103", title: "Display 1" }, ] } satisfies NativeSourceList; const previewSource = async () => POSTER; @@ -36,6 +45,7 @@ const previewMicrophones = async () => [{ id: "headset", label: "USB Headset" }, export function ControlsPreview() { const { t, lang, vis } = useCopy(); const en = lang === "en"; + const displayPreview = new URLSearchParams(window.location.search).get("source") === "display"; const [sound, setSound] = useState(true); const [microphone, setMicrophone] = useState(false); const [microphoneVolume, setMicrophoneVolume] = useState(1); @@ -54,7 +64,7 @@ export function ControlsPreview() { const [passwordVisible, setPasswordVisible] = useState(false); const [name, setName] = useState("Piik friend"); const [invalid, setInvalid] = useState(false); - const [sourceOpen, setSourceOpen] = useState(false); + const [sourceOpen, setSourceOpen] = useState(displayPreview); const [sourceState, setSourceState] = useState("ready"); const [captureBorderAvailable, setCaptureBorderAvailable] = useState(true); const [showCaptureBorder, setShowCaptureBorder] = useState(false); @@ -92,6 +102,7 @@ export function ControlsPreview() { hint={paused ? "hint-resume" : "hint-pause"} onClick={() => setPaused(!paused)} /> setSharingSettings(value => !value)} /> @@ -108,6 +119,8 @@ export function ControlsPreview() { undefined} label={t("host.advanced.route.peerOnly")} note={t("host.advanced.route.peerOnlyRequired")} hint="hint-route-p2p-required" /> + undefined} label={t("host.advanced.route.sfuOnly")} + note={t("host.advanced.route.sfuOnlyHint")} hint="hint-route-sfu" />
{sourceOpen ? { setSourceOpen(false); notify(); }} onCamera={() => { setSourceOpen(false); notify(); }} loadCameras={previewCameras} @@ -217,5 +234,70 @@ export function ControlsPreview() { onSubmit={event => event.preventDefault()} />
+
+
09

{en ? "A little company" : "聊两句,丢个番茄。"}

+

{en ? "Click a person for reactions. Drag or resize either window." : "点小人发表情;聊天与表情窗都能拖动、缩放。"}

+ +
; } + +function InteractionControlsPreview() { + const [session, setSession] = useState(null); + const [viewer, setViewer] = useState(false); + const [crowded, setCrowded] = useState(false); + const [reaction, setReaction] = useState("heart"); + const [theater, setTheater] = useTheaterMode(); + const video = useRef(null); + const { t, lang } = useCopy(); + useEffect(() => { + const sender = viewer + ? { peerId: "preview-friend", role: "viewer" as const, displayName: "Piik friend" } + : { peerId: "preview-host", role: "host" as const, displayName: "Piik" }; + const model = new RoomInteractionSession(message => { + queueMicrotask(() => { + if (message.type === "subscribe-room-interactions") model.receive({ type: "room-interactions-ready", serverTime: Date.now() }); + if (message.type === "send-room-interaction") model.receive({ type: "room-interaction", id: createOpaqueId(), + requestId: message.requestId, occurredAt: Date.now(), sender, payload: message.payload }); + }); + return true; + }); + model.authenticated(sender.peerId); + setSession(model); + return () => model.close(); + }, [viewer]); + const receiveSample = (payload: InteractionPayload) => session?.receive({ + type: "room-interaction", id: createOpaqueId(), requestId: createOpaqueId(), occurredAt: Date.now(), + sender: viewer ? { peerId: "preview-host", role: "host", displayName: "Piik" } + : { peerId: "preview-friend", role: "viewer", displayName: "Piik friend" }, payload, + }); + return <>
+ + + + + +
+ + ({ + key: index ? `preview-friend-${index}` : "preview-friend", name: index ? `Friend ${index + 1}` : "Piik friend", + you: viewer && index === 0, + status: deriveParticipantStatus({ upstream: { kind: "peer", peerId: "preview-host" }, mediaReady: true }, true), + })) + } />
; +} diff --git a/src/client/pages/HostPage.tsx b/src/client/pages/HostPage.tsx index 9c6add6..a705395 100644 --- a/src/client/pages/HostPage.tsx +++ b/src/client/pages/HostPage.tsx @@ -25,9 +25,12 @@ import { import { browserCaptureDevices } from "../media/capture-devices"; import { AppHeader, LedStrip } from "../components/living/Header"; import { WelcomeLine } from "../components/living/WelcomeLine"; -import { Couch, type CouchEntry } from "../components/living/Couch"; +import type { CouchEntry } from "../components/living/Couch"; import { SharingSettings } from "../components/living/SharingSettings"; import { HostMicrophone, HostMicrophoneSettings } from "../components/living/HostMicrophone"; +import { RoomInteractions } from "../components/living/RoomInteractions"; +import { RoomChatOverlay } from "../components/living/RoomChatOverlay"; +import type { RoomInteractionSession } from "../lib/room-interactions"; import { HostAudio } from "../media/host-audio"; import { CaptureSourcePicker, @@ -330,7 +333,10 @@ function hostTerminationKey(reason: SignalingTerminationReason): CopyKey { } interface HostPageProps { + roomInteractionsAvailable?: boolean; + hostRoomSessionAvailable?: boolean; sfuAvailable?: boolean; + sfuOnly?: boolean; natPredictionAvailable?: boolean; connectionAttemptProgress4?: boolean; launchedByClient?: boolean; @@ -344,12 +350,16 @@ type ShareSourceSelection = client: NativeClient; target: NativeCaptureTarget; audio: boolean; + excludeAudio?: NativeCaptureTarget; showCaptureBorder: boolean; path: NativeCapturePath; }; export function HostPage({ + roomInteractionsAvailable = false, + hostRoomSessionAvailable = false, sfuAvailable = false, + sfuOnly = false, natPredictionAvailable = false, connectionAttemptProgress4 = false, launchedByClient = false, @@ -367,7 +377,8 @@ export function HostPage({ () => ({ ...DEFAULT_ROUTE_POLICY, peerOnly: !sfuAvailable, - natPrediction: natPredictionAvailable, + topologyOptimization: !sfuOnly, + natPrediction: natPredictionAvailable && !sfuOnly, }), ); const [videoCodecMode, setVideoCodecMode] = @@ -381,6 +392,7 @@ export function HostPage({ const [stream, setStream] = useState(null); const hostAudioRef = useRef(null); const [microphoneEnabled, setMicrophoneEnabled] = useState(false); + const [interactionSession, setInteractionSession] = useState(null); const [microphoneVolume, setMicrophoneVolume] = useState(1); const [microphonePending, setMicrophonePending] = useState(false); const [microphoneDevices, setMicrophoneDevices] = useState({ browser: "", native: "" }); @@ -444,6 +456,13 @@ export function HostPage({ useEffect(() => { roomRef.current = room; }, [room]); + useEffect(() => { + let active = true; + if (hostRoomSessionAvailable && room) void roomInitializationRef.current.then(() => { + if (active && isCurrentRoomAuthority(room)) connectRoomSignal(room).start(); + }); + return () => { active = false; }; + }, [hostRoomSessionAvailable, room?.roomId, room?.hostToken]); const [creationProfile, setCreationProfile] = useState(readCreationProfile); const creationProfileRef = useRef(creationProfile); @@ -580,12 +599,14 @@ export function HostPage({ const nativeClientRef = useRef(null); const nativeClientConnectRef = useRef | null>(null); const nativeShareGenerationRef = useRef(null); + const serverShareGenerationRef = useRef(null); const nativeMediaBridgeRef = useRef(null); const nativeMediaIngressRef = useRef(null); const nativeEventCleanupRef = useRef<(() => void) | null>(null); const nativeClientCloseCleanupRef = useRef<(() => void) | null>(null); const nativeModeRef = useRef(false); const nativeSourceAudioRef = useRef(undefined); + const nativeAudioSelectionRef = useRef<{ enabled: boolean; exclude?: NativeCaptureTarget } | null>(null); const nativeSourceRequestRef = useRef(null); const nativePreviewTailRef = useRef>(Promise.resolve()); const nativeSourcePathRef = useRef(null); @@ -688,6 +709,7 @@ export function HostPage({ browserVideoPoolRef.current = null; activeHostChildPeerIdsRef.current = []; endpointMediaCopyCapacityRef.current = MAX_ENDPOINT_MEDIA_CHILDREN; + serverShareGenerationRef.current = null; hostPeerIdRef.current = null; if (copiedResetTimerRef.current !== null) { window.clearTimeout(copiedResetTimerRef.current); @@ -836,7 +858,7 @@ export function HostPage({ return warning; } - function disposeResources(notifyServer: boolean): void { + function disposeResources(notifyServer: boolean, keepRoomSession = false): void { sourceSwitchRef.current = null; qualityChangeRef.current = null; pendingQualityChangeRef.current = null; @@ -851,14 +873,20 @@ export function HostPage({ if (notifyServer) { const shareGeneration = shareGenerationRef.current; if (shareGeneration) { - signal.send({ type: "stop-sharing", shareGeneration }); + if (keepRoomSession) signal.stopSharing(shareGeneration); + else signal.send({ type: "stop-sharing", shareGeneration }); } } - // Never carry a terminal message into a later authentication: the next - // sharing generation may already be reusing this room. - signal.stop(); + if (!keepRoomSession) { + // Never carry a terminal message into a later authentication: the next + // sharing generation may already be reusing this room. + signal.stop(); + } + } + if (!keepRoomSession) { + signalRef.current = null; + setInteractionSession(null); } - signalRef.current = null; shareGenerationRef.current = null; disposeNativeShare(); peersRef.current.forEach((peer) => peer.dispose()); @@ -868,8 +896,11 @@ export function HostPage({ browserVideoPoolRef.current?.dispose(); browserVideoPoolRef.current = null; activeHostChildPeerIdsRef.current = []; - endpointMediaCopyCapacityRef.current = MAX_ENDPOINT_MEDIA_CHILDREN; - hostPeerIdRef.current = null; + if (!keepRoomSession) { + endpointMediaCopyCapacityRef.current = MAX_ENDPOINT_MEDIA_CHILDREN; + hostPeerIdRef.current = null; + } + serverShareGenerationRef.current = null; void hostSfuRouteRef.current?.disconnect(); hostSfuRouteRef.current = null; hostAudioRef.current?.dispose(); @@ -880,16 +911,16 @@ export function HostPage({ retiringStreamRef.current?.getTracks().forEach((track) => track.stop()); streamRef.current = null; retiringStreamRef.current = null; - iceConfigRef.current = null; + if (!keepRoomSession) iceConfigRef.current = null; setStream(null); setDetails(null); setPeerSnapshots(new Map()); - setParticipantPresence([]); + if (!keepRoomSession) setParticipantPresence([]); viewerQualityEvidenceStore.clear(); cancelViewerQualityEvidenceRender(); setViewerQualityEvidence(viewerQualityEvidenceStore.getSnapshot()); activeRouteRevisionRef.current = 0; - setSignalStatus("offline"); + if (!keepRoomSession) setSignalStatus("offline"); setSwitchingSource(false); setChangingQuality(false); sharingPausedRef.current = false; @@ -943,6 +974,7 @@ export function HostPage({ notifyServer = true, comic: ComicKind = "share-ended", tone: ComicTone = "off", + keepRoomSession = hostRoomSessionAvailable, ): void { const generation = activeGenerationRef.current; if (generation === null || generationRef.current !== generation) { @@ -955,7 +987,7 @@ export function HostPage({ if (notifyServer && currentRoom) { writePreferredRoom(currentRoom.roomId); } - disposeResources(notifyServer); + disposeResources(notifyServer, keepRoomSession); setNoticeValue({ ...(typeof message === "string" ? { kind: "text", text: message } @@ -987,7 +1019,9 @@ export function HostPage({ ); const replacement = hostRoomFromCreated(response); if (wasSharing) { - endSharing({ key: "host.roomReplaced" }, false); + endSharing({ key: "host.roomReplaced" }, false, "share-ended", "off", false); + } else if (signalRef.current) { + disposeResources(false); } replaceViewerInvite(activeRoom.roomId, null); if (!(await writeHostRoom( @@ -1054,7 +1088,7 @@ export function HostPage({ if (forgetRoom(activeRoom)) { // endSharing already stated the recovery path; the raw room error // would only overwrite it with the same 404 in server wording. - endSharing({ key: "host.roomInvalid" }, false, "room-not-found", "bad"); + endSharing({ key: "host.roomInvalid" }, false, "room-not-found", "bad", false); return; } } @@ -1156,7 +1190,7 @@ export function HostPage({ shareGeneration: string, selection: Extract, ): Promise { - const { client, target, audio, showCaptureBorder, path } = selection; + const { client, target, audio, excludeAudio, showCaptureBorder, path } = selection; let bridge: NativeMediaBridge | null = null; let shareStarted = false; let nativeEventCleanup: (() => void) | null = null; @@ -1181,6 +1215,7 @@ export function HostPage({ ); nativeMediaBridgeRef.current = bridge; nativeSourceAudioRef.current = undefined; + nativeAudioSelectionRef.current = { enabled: audio, exclude: excludeAudio }; nativeEventCleanup = client.onEvent((event) => { if (event.shareId !== shareGeneration || !isCurrentShare(generation, shareGeneration) || nativeClientRef.current !== client) return; if (event.type === "audio-state") { @@ -1203,6 +1238,7 @@ export function HostPage({ shareId: shareGeneration, source: target, audio, + excludeAudio: audio ? excludeAudio : undefined, showCaptureBorder, adapterIndex: path.adapterIndex, encoderIndex: path.encoderIndex, @@ -1392,6 +1428,7 @@ export function HostPage({ sources, processAudio: client.health.nativeMedia.processAudio, systemAudio: client.health.nativeMedia.systemAudio, + processAudioExclusion: client.health.nativeMedia.processAudioExclusion, captureBorderControl: client.health.nativeMedia.captureBorderControl, }); } catch (error) { @@ -1484,6 +1521,7 @@ export function HostPage({ target: NativeCaptureTarget, audio: boolean, showCaptureBorder: boolean, + excludeAudio?: NativeCaptureTarget, ): void { if (nativeSources?.kind !== "ready") return; const client = nativeClientRef.current; @@ -1491,10 +1529,10 @@ export function HostPage({ if (!client || !path) return; setShowCaptureBorder(showCaptureBorder); if (phase === "live" && nativeModeRef.current) { - void switchNativeSource(client, target, audio, path, showCaptureBorder); + void switchNativeSource(client, target, audio, path, showCaptureBorder, excludeAudio); return; } - void startSharing({ kind: "native", client, target, audio, showCaptureBorder, path }); + void startSharing({ kind: "native", client, target, audio, excludeAudio, showCaptureBorder, path }); } function disposeNativeShare(expectedShare = nativeShareGenerationRef.current): void { @@ -1511,6 +1549,7 @@ export function HostPage({ nativeShareGenerationRef.current = null; nativeModeRef.current = false; nativeSourceAudioRef.current = undefined; + nativeAudioSelectionRef.current = null; setNativeActive(false); if (!client || !shareGeneration) { releaseUnusedNativeClient(); @@ -2214,7 +2253,6 @@ export function HostPage({ function handleSignalMessage( message: ServerMessage, generation: number, - activeRoom: HostRoomState, reauthenticated: boolean, pendingQualitySettings: QualitySettings | null, ): void { @@ -2223,29 +2261,8 @@ export function HostPage({ } if (message.type === "authenticated" && message.role === "host") { discardPreparedHostChild(); - hostPeerIdRef.current = message.peerId; - endpointMediaCopyCapacityRef.current = message.endpointMediaCopyCapacity; - const authenticatedProfile = { - codeEntryPolicy: message.codeEntryPolicy, - roomPassword: message.viewerPasswordEnabled - ? creationProfileRef.current.roomPassword - : null, - }; - saveCreationProfile(authenticatedProfile); - creationProfileRef.current = authenticatedProfile; - setCreationProfile(authenticatedProfile); - setViewerPasswordEnabled(message.viewerPasswordEnabled); - setViewerPasswordDraft(authenticatedProfile.roomPassword ?? ""); - setViewerPasswordVisible(false); routePolicyRef.current = { ...message.routePolicy }; setRoutePolicy({ ...message.routePolicy }); - setRoom((current) => - mergeAuthenticatedHostRoom( - current, - activeRoom.roomId, - message.codeEntryPolicy, - ), - ); const currentQualitySettings = pendingQualitySettings ?? message.qualitySettings; activeRouteRevisionRef.current = message.routeRevision; @@ -2289,9 +2306,24 @@ export function HostPage({ }); return; } - if (message.type === "viewer-presence") { - retainViewerQualityEvidenceForPresence(message.viewers); - setParticipantPresence(message.viewers); + if (message.type === "sharing-started") { + if (message.shareGeneration !== shareGenerationRef.current) return; + serverShareGenerationRef.current = message.shareGeneration; + routePolicyRef.current = { ...message.routePolicy }; + setRoutePolicy({ ...message.routePolicy }); + commitQuality(message.qualitySettings); + activeRouteRevisionRef.current = message.routeRevision; + sharingPausedRef.current = message.paused; + setSharingPaused(message.paused); + setPhase("live"); + const route = ensureHostSfuRoute(generation); + void route + .resyncAuthoritative({ + revision: message.routeRevision, + phase: "active", + assignment: message.routeAssignment, + }) + .then(() => syncHostSfuQualityWarning(route, generation)); return; } if (message.type === "pause-sharing-source") { @@ -2395,29 +2427,237 @@ export function HostPage({ }); return; } - if (message.type === "room-closed") { - if (!forgetRoom(activeRoom)) { + } + + function connectRoomSignal(activeRoom: HostRoomState, shareGeneration?: string, allowReplacement = true): SignalingClient { + const previous = signalRef.current; + if (hostRoomSessionAvailable && previous?.ownsHostRoom(activeRoom.roomId, activeRoom.hostToken)) return previous; + previous?.stop(); + let authenticated = false; + const hostClientId = getStableClientId("host", activeRoom.roomId); + hostClientIdRef.current = hostClientId; + const hostFallback = defaultHostDisplayName(visRef.current); + const initialDisplayName = readDisplayName(hostFallback); + displayNameRef.current = initialDisplayName; + setDisplayName(initialDisplayName); + setDisplayNameDraft(initialDisplayName); + setDisplayNameError(null); + const signal = new SignalingClient( + { + roomId: activeRoom.roomId, + role: "host", + token: activeRoom.hostToken, + clientId: hostClientId, + ...(hostRoomSessionAvailable ? { roomSession: true as const, roomOnly: true as const } : { shareGeneration }), + sharingPaused: false, + qualitySettings: qualitySettingsRef.current, + routePolicy: routePolicyRef.current, + viewerPresence: true, + ...(connectionAttemptProgress4 ? { connectionAttemptProgress4: true } : {}), + displayName: initialDisplayName, + }, + { + onStatus: (status) => { + if (signalRef.current === signal && isCurrentRoomAuthority(activeRoom)) { + setSignalStatus(status); + } + }, + onTerminated: (reason) => { + if (signalRef.current !== signal || !isCurrentRoomAuthority(activeRoom)) return; + if (activeGenerationRef.current !== null) { + endSharing({ key: hostTerminationKey(reason) }, false, "signal-failed", "bad", false); + } else { + disposeResources(false); + setNoticeKey(hostTerminationKey(reason), "signal-failed", "bad"); + setPhase("error"); + } + if (reason === "SESSION_REPLACED") forgetRoom(activeRoom, true); + }, + onAccessRequired: () => { + if (signalRef.current !== signal || !isCurrentRoomAuthority(activeRoom)) return; + if (activeGenerationRef.current !== null) { + endSharing({ key: "gate.expired" }, false, "access-denied", "bad", false); + } else { + disposeResources(false); + setNoticeKey("gate.expired", "access-denied", "bad"); + setPhase("error"); + } + onAuthorizationRequired?.(); + }, + onMessage: (message) => { + if (signalRef.current !== signal || !isCurrentRoomAuthority(activeRoom)) return; + const currentGeneration = activeGenerationRef.current; + const currentShareGeneration = shareGenerationRef.current; + const localShareActive = + currentGeneration !== null && currentShareGeneration !== null; + if ( + !authenticated && + message.type === "error" && + message.code === "INVALID_TOKEN" && + allowReplacement + ) { + if (!localShareActive || !signal.wantsHostPublication(currentShareGeneration!)) { + signalRef.current = null; + signal.stop(); + setInteractionSession(null); + setSignalStatus("offline"); + forgetRoom(activeRoom); + // Capture may still be awaiting permission. Its ready path owns + // creation/publication; room recovery must not start it early. + if (!localShareActive) { + setNoticeKey("host.roomInvalid", "room-not-found", "bad"); + setPhase("error"); + } + return; + } + signalRef.current = null; + signal.stop(); + setSignalStatus("offline"); + roomInitializationRef.current = createReplacementRoom(activeRoom, currentGeneration!, currentShareGeneration!); + return; + } + if (message.type === "error" && message.code === "INVALID_TOKEN") { + forgetRoom(activeRoom); + if (localShareActive) { + endSharing({ key: "host.roomInvalid" }, false, "room-not-found", "bad", false); + } else { + disposeResources(false); + forgetRoom(activeRoom); + setNoticeKey("host.roomInvalid", "room-not-found", "bad"); + setPhase("error"); + } + return; + } + if (message.type === "room-closed") { + if (!forgetRoom(activeRoom)) return; + if (localShareActive) { + endSharing(say("host.roomClosed"), false, "room-closed", "off", false); + } else { + disposeResources(false); + setNoticeKey("host.roomClosed", "room-closed", "off"); + setPhase("ended"); + } + return; + } + if (message.type === "error" && message.code === "HOST_ALREADY_CONNECTED") { + if (localShareActive) { + endSharing(hostServerErrorNotice(message.code), false, "signal-failed", "bad", false); + } else { + disposeResources(false); + setNotice(hostServerErrorNotice(message.code), "signal-failed", "bad"); + setPhase("error"); + } + return; + } + const reauthenticated = + authenticated && + message.type === "authenticated" && + message.role === "host"; + if (message.type === "authenticated" && message.role === "host") { + authenticated = true; + iceConfigRef.current = message.iceConfig; + hostPeerIdRef.current = message.peerId; + endpointMediaCopyCapacityRef.current = message.endpointMediaCopyCapacity; + serverShareGenerationRef.current = message.shareGeneration; + const profile = { + codeEntryPolicy: message.codeEntryPolicy, + roomPassword: message.viewerPasswordEnabled ? creationProfileRef.current.roomPassword : null, + }; + saveCreationProfile(profile); + creationProfileRef.current = profile; + setCreationProfile(profile); + setViewerPasswordEnabled(message.viewerPasswordEnabled); + setViewerPasswordDraft(profile.roomPassword ?? ""); + setViewerPasswordVisible(false); + setRoom(current => mergeAuthenticatedHostRoom(current, activeRoom.roomId, message.codeEntryPolicy)); + peersRef.current.forEach((peer) => + peer.updateIceConfig(message.iceConfig), + ); + void writeHostRoom( + activeRoom, + () => signalRef.current === signal && isCurrentRoomAuthority(activeRoom), + ); + writePreferredRoom(activeRoom.roomId); + } + if ( + message.type === "sharing-started" && + currentShareGeneration !== null && + message.shareGeneration === currentShareGeneration + ) { + serverShareGenerationRef.current = message.shareGeneration; + } + if (message.type === "sharing-start-failed") { + if (localShareActive && message.shareGeneration === currentShareGeneration) { + endSharing(hostServerErrorNotice(message.code), false, "signal-failed", "bad"); + } + return; + } + if (message.type === "viewer-presence") { + retainViewerQualityEvidenceForPresence(message.viewers); + setParticipantPresence(message.viewers); + return; + } + if (message.type === "error") { + if (message.code !== "AUTH_REQUIRED") setNotice(hostServerErrorNotice(message.code), "signal-failed", "bad"); + return; + } + const mediaActive = + currentGeneration !== null && + currentShareGeneration !== null && + serverShareGenerationRef.current === currentShareGeneration; + if (message.type === "authenticated" && message.role === "host" && mediaActive) { + setPhase("live"); + } + if (!mediaActive) return; + handleSignalMessage( + message, + currentGeneration!, + reauthenticated, + reauthenticated + ? signal.pendingHostQualitySettings(currentShareGeneration!) + : null, + ); + }, + }, + undefined, + { roomInteractions: roomInteractionsAvailable }, + ); + signalRef.current = signal; + setInteractionSession(signal.interactions); + return signal; + } + + async function createReplacementRoom(activeRoom: HostRoomState, generation: number, shareGeneration: string): Promise { + try { + const response = await createRoom( + creationProfileRef.current.codeEntryPolicy, + creationProfileRef.current.roomPassword, + readPreferredRoomId(), + ); + const replacement = hostRoomFromCreated(response); + if (!isCurrentShare(generation, shareGeneration) || !isCurrentRoomAuthority(activeRoom)) { + closeAbandonedRoom(replacement); return; } - endSharing(say("host.roomClosed"), false, "room-closed", "off"); - return; - } - if (message.type === "error") { - if (message.code === "INVALID_TOKEN") { - if (!forgetRoom(activeRoom)) { - return; - } - endSharing({ key: "host.roomInvalid" }, false, "room-not-found", "bad"); + forgetRoom(activeRoom); + if (!(await writeHostRoom(replacement, () => isCurrentShare(generation, shareGeneration)))) { + closeAbandonedRoom(replacement); + throw new Error("Host room is already open in another tab"); + } + roomRef.current = replacement; + setRoom(replacement); + const signal = connectRoomSignal(replacement, shareGeneration, false); + if (hostRoomSessionAvailable) signal.startSharing(shareGeneration, qualitySettingsRef.current, routePolicyRef.current); + signal.start(); + } catch (error) { + if (!isCurrentShare(generation, shareGeneration)) return; + endSharing({ key: "host.roomInvalid" }, false, "room-not-found", "bad", false); + if (error instanceof ApiError && error.status === 401 && onAuthorizationRequired) { + onAuthorizationRequired(); return; } - if (message.code === "AUTH_REQUIRED") { - return; - } - if (message.code === "HOST_ALREADY_CONNECTED") { - endSharing(hostServerErrorNotice(message.code), false, "signal-failed", "bad"); - return; - } - setNotice(hostServerErrorNotice(message.code), "signal-failed", "bad"); + setNoticeError(error, "room", "television"); + setPhase("error"); } } @@ -2538,158 +2778,9 @@ export function HostPage({ setRoom(createdRoom); claimedRoom = true; } - let replacementAttempted = false; - const connectSignal = (activeRoom: HostRoomState): SignalingClient => { - let authenticated = false; - const hostClientId = getStableClientId("host", activeRoom.roomId); - hostClientIdRef.current = hostClientId; - const hostFallback = defaultHostDisplayName(visRef.current); - const initialDisplayName = readDisplayName(hostFallback); - displayNameRef.current = initialDisplayName; - setDisplayName(initialDisplayName); - setDisplayNameDraft(initialDisplayName); - setDisplayNameError(null); - const signal = new SignalingClient( - { - roomId: activeRoom.roomId, - role: "host", - token: activeRoom.hostToken, - clientId: hostClientId, - shareGeneration, - sharingPaused: false, - qualitySettings: qualitySettingsRef.current, - routePolicy: routePolicyRef.current, - viewerPresence: true, - ...(connectionAttemptProgress4 ? { connectionAttemptProgress4: true } : {}), - displayName: initialDisplayName, - }, - { - onStatus: (status) => { - if ( - isCurrentShare(generation, shareGeneration) && - signalRef.current === signal - ) { - setSignalStatus(status); - } - }, - onTerminated: (reason) => { - if ( - isCurrentShare(generation, shareGeneration) && - signalRef.current === signal - ) { - endSharing({ key: hostTerminationKey(reason) }, false, "signal-failed", "bad"); - if (reason === "SESSION_REPLACED") forgetRoom(activeRoom, true); - } - }, - onAccessRequired: () => { - if ( - isCurrentShare(generation, shareGeneration) && - signalRef.current === signal - ) { - endSharing({ key: "gate.expired" }, false, "access-denied", "bad"); - onAuthorizationRequired?.(); - } - }, - onMessage: (message) => { - if ( - !isCurrentShare(generation, shareGeneration) || - signalRef.current !== signal - ) { - return; - } - if ( - !authenticated && - message.type === "error" && - message.code === "INVALID_TOKEN" && - !replacementAttempted - ) { - replacementAttempted = true; - signalRef.current = null; - signal.stop(); - setSignalStatus("offline"); - forgetRoom(activeRoom); - void createReplacementRoom(); - return; - } - const reauthenticated = - authenticated && - message.type === "authenticated" && - message.role === "host"; - if (message.type === "authenticated" && message.role === "host") { - authenticated = true; - iceConfigRef.current = message.iceConfig; - peersRef.current.forEach((peer) => - peer.updateIceConfig(message.iceConfig), - ); - void writeHostRoom( - activeRoom, - () => isCurrentShare(generation, shareGeneration) && - signalRef.current === signal, - ); - writePreferredRoom(activeRoom.roomId); - setPhase("live"); - } - handleSignalMessage( - message, - generation, - activeRoom, - reauthenticated, - reauthenticated - ? signal.pendingHostQualitySettings(shareGeneration) - : null, - ); - }, - }, - ); - return signal; - }; - const createReplacementRoom = async (): Promise => { - try { - const response = await createRoom( - creationProfileRef.current.codeEntryPolicy, - creationProfileRef.current.roomPassword, - readPreferredRoomId(), - ); - const replacement = hostRoomFromCreated(response); - if (!isCurrentShare(generation, shareGeneration)) { - captured?.getTracks().forEach((track) => track.stop()); - disposeNativeShare(shareGeneration); - closeAbandonedRoom(replacement); - return; - } - if (!(await writeHostRoom( - replacement, - () => isCurrentShare(generation, shareGeneration), - ))) { - closeAbandonedRoom(replacement); - throw new Error("Host room is already open in another tab"); - } - roomRef.current = replacement; - setRoom(replacement); - const replacementSignal = connectSignal(replacement); - signalRef.current = replacementSignal; - replacementSignal.start(); - } catch (error) { - if (!isCurrentShare(generation, shareGeneration)) { - if (nativeStarted) disposeNativeShare(shareGeneration); - return; - } - activeGenerationRef.current = null; - disposeResources(false); - if ( - error instanceof ApiError && - error.status === 401 && - onAuthorizationRequired - ) { - onAuthorizationRequired(); - return; - } - setNoticeError(error, "room", "television"); - setPhase("error"); - } - }; - const signal = connectSignal(createdRoom); - signalRef.current = signal; + const signal = connectRoomSignal(createdRoom, shareGeneration); + if (hostRoomSessionAvailable) signal.startSharing(shareGeneration, qualitySettingsRef.current, routePolicyRef.current); + signal.start(); } catch (error) { if (!isCurrentShare(generation, shareGeneration)) { @@ -2724,6 +2815,7 @@ export function HostPage({ audio: boolean, path: NativeCapturePath, showCaptureBorder: boolean, + excludeAudio?: NativeCaptureTarget, ): Promise { const generation = activeGenerationRef.current; const shareGeneration = nativeShareGenerationRef.current; @@ -2750,12 +2842,16 @@ export function HostPage({ sourceSwitchRef.current !== token || nativeClientRef.current !== client ) return; + // Preserve an exclusion request even on failure: the App mutes source + // audio before replacing it, and the next picker must not default to all. + if (audio && excludeAudio) nativeAudioSelectionRef.current = { enabled: audio, exclude: excludeAudio }; await client.replaceShareSource( shareGeneration, target, audio, path, showCaptureBorder, + audio ? excludeAudio : undefined, ); if ( !isCurrentGeneration(generation) || @@ -2765,6 +2861,7 @@ export function HostPage({ return; } invalidateSenderQualityEvidence(); + nativeAudioSelectionRef.current = { enabled: audio, exclude: excludeAudio }; if (routePolicyRef.current.topologyOptimization) { signalRef.current?.send({ type: "reset-sender-quality" }); } @@ -3444,6 +3541,8 @@ export function HostPage({ {stream ? (