diff --git a/.env.example b/.env.example
index 3b71f8a..20ba2eb 100644
--- a/.env.example
+++ b/.env.example
@@ -35,6 +35,8 @@ SFU_UDP_PORT=
SFU_LISTEN_HOST=0.0.0.0
# Optional advertised IPv4 override when the listener's address is behind NAT.
SFU_PUBLIC_IP=
+# Require server media for every room instead of P2P-first. Requires SFU_UDP_PORT.
+SFU_ONLY=false
# Ordinary peer ICE discovery. Production requires at least one STUN URL.
# These are advertised addresses, not local listener addresses. They may use
diff --git a/AGENTS.md b/AGENTS.md
index 171e7d1..131e228 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -30,7 +30,7 @@
[routing/transport](./docs/standards/routing-transport.md),
[media quality](./docs/standards/media-quality.md), and
[presentation/lifecycle](./docs/standards/presentation-lifecycle.md).
-- Media stays automatic and P2P-first; central services provide room authority,
+- Media stays automatic and defaults to P2P-first; central services provide room authority,
signaling, STUN, observability, and bounded embedded SFU/UDP fallback using
mature LiveKit media components under [ADR-0013](./docs/adr/0013-embedded-node-local-media.md). Route
changes must preserve the single-graph, single-operation model in
diff --git a/README.md b/README.md
index 58e85d9..a48d317 100644
--- a/README.md
+++ b/README.md
@@ -34,6 +34,7 @@ watch in their browsers.
- **Share from Piik App or an existing Piik site.** Choose a screen, window, browser tab or camera; available sources and audio depend on the platform.
- **Direct connections first.** Media travels between participants where possible (P2P). A self-hosted server can provide automatic media forwarding (SFU) as a fallback.
- **Rooms you control.** Invitations, room codes and optional room passwords, for one host and up to 20 viewers.
+- **Chat while watching.** Send messages and reactions to friends, with optional on-screen chat.
- **Flexible viewing.** Light and dark themes, playback controls, picture-in-picture and a connection topology view.
- **One program to host a site.** Web UI, room management and optional media forwarding are packaged together.
diff --git a/README.zh-CN.md b/README.zh-CN.md
index 511d839..11131a2 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -33,6 +33,7 @@ Piik 是一个免费开源的屏幕共享工具,适合游戏围观、一起看
- **用 Piik App 或已有 Piik 站点分享。** 支持屏幕、窗口、浏览器标签页和摄像头;可选来源与音频能力取决于平台。
- **优先直接连接。** 画面优先在设备之间传输(P2P),自建站点可启用媒体转发(SFU)自动兜底。
- **房间由你管理。** 支持邀请链接、房间号和可选的房间密码,一位房主、最多 20 位观众。
+- **边看边聊。** 发送文字、和朋友互送表情,也可开启弹幕。
- **灵活的观看方式。** 支持深浅主题、播放控制、画中画和连接拓扑查看。
- **一个程序就能建站。** 网页、房间管理和可选的媒体转发打包在同一服务端程序中。
diff --git a/deploy/container/.env.example b/deploy/container/.env.example
index c83e7a5..89c06ac 100644
--- a/deploy/container/.env.example
+++ b/deploy/container/.env.example
@@ -8,6 +8,8 @@ SITE_ACCESS_PASSWORD=
# Optional automatic media fallback: set BOTH values and allow UDP 7882.
# SFU_UDP_PORT=7882
# SFU_PUBLIC_IP=203.0.113.10
+# Optional: require SFU for all rooms instead of P2P-first (requires SFU_UDP_PORT).
+# SFU_ONLY=true
# Optional NAT survey: also allow UDP 3479 and 3480.
# NAT_PREDICTION_ENABLED=true
diff --git a/docs/README.md b/docs/README.md
index 9b51ca1..4666b81 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -99,7 +99,8 @@ and [website preview and publishing](./operations/website.md).
| Why a design was chosen | [Architecture decisions](./adr/) |
| Measurements and platform limits | [Research](./research/) and [verification status](./verification-status.md) |
| Camera and Host microphone | [Capture scope and limits](./research/camera-and-microphone.md) |
-| Room voice proposal and media-fidelity boundaries | [Design assessment](./research/room-voice.md) |
+| Unreleased text, reactions and floating room tools | [Room-interaction candidate](./research/room-interactions.md) |
+| Stereo verification and HDR-to-SDR assessment | [Media fidelity](./research/media-fidelity.md) |
## Shared conventions
diff --git a/docs/adr/0005-automatic-hybrid-media-routing.md b/docs/adr/0005-automatic-hybrid-media-routing.md
index 4b3b970..2fc2de6 100644
--- a/docs/adr/0005-automatic-hybrid-media-routing.md
+++ b/docs/adr/0005-automatic-hybrid-media-routing.md
@@ -28,6 +28,14 @@ Hosted and Local always use this controller. The former rollout switch and
separate Host-star authority are removed; SFU availability is still selected by
configuration and the room's privacy policy, not by another routing mode.
+The optional Hosted `SFU_ONLY` policy restricts this controller's candidate
+eligibility to SFU. It reuses the publication/subscription ledger, deadlines,
+first-frame commit and recovery. It does not create another graph or scheduler.
+It disables Peer relay and background Peer work; admission failure does not
+relax the restriction. The service rejects an explicit peer-only request instead
+of treating it as permission to forward media. Default P2P-first and App Local
+behavior are unchanged.
+
### One Capacity Rule
Every non-server endpoint uses one server-authoritative steady outbound media-
diff --git a/docs/adr/0008-window-scoped-audio-capture.md b/docs/adr/0008-window-scoped-audio-capture.md
index 9283c79..820837d 100644
--- a/docs/adr/0008-window-scoped-audio-capture.md
+++ b/docs/adr/0008-window-scoped-audio-capture.md
@@ -2,7 +2,7 @@
- Status: accepted for Browser capture and the Windows App audio paths
- Date: 2026-08-21
-- Last updated: 2026-08-27
+- Last updated: 2026-10-01
## Context
@@ -26,8 +26,8 @@ platform probe and physical gate report it.
Track presence does not prove its source or isolation.
3. Returned screen-audio tracks use `contentHint = "music"`. Current bounded
sender ceilings and narrow Opus stereo answer normalization are owned by
- [media quality](../standards/media-quality.md). There is no Web Audio mixer,
- resampler, second representation, or independent audio clock.
+ [media quality](../standards/media-quality.md). Audio scope adds no second
+ representation or independent clock; Host commentary uses the existing mixer.
4. Browser source switching replaces the current screen-audio track in the same
persistent media stream. Local Viewer play, mute, and volume stay inside the
native media element.
@@ -38,6 +38,13 @@ platform probe and physical gate report it.
probe and physical gate pass. WGC/MF video and the App media boundary are owned by
[ADR-0010](./0010-cross-platform-client-runtime.md); measurements are in
[Native App media](../research/native-client-media.md).
+7. Windows App screen sharing can optionally exclude one selected process tree
+ through WASAPI's exclusion mode, on build 20348+ with a successful capability
+ probe. This mode spans all render endpoints; it is not restricted to the
+ default playback device. Reuse source enumeration and the existing PCM/mixed
+ output. Generic Browser capture and arbitrary unrelated process sets are not
+ covered. [Media quality](../standards/media-quality.md#screen-audio) owns
+ selection lifetime and failure behavior.
## Consequences
diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md
index 776a68a..983962e 100644
--- a/docs/guide/getting-started.md
+++ b/docs/guide/getting-started.md
@@ -7,6 +7,7 @@ watching, using Piik online, using Piik App and sharing through an existing site
[Join a room](#join-a-friends-room) · [Use online](#use-piik-online) ·
[Piik App](#share-with-piik-app) · [Existing site](#share-from-an-existing-site) ·
+[Chat and reactions](#chat-and-reactions) ·
[Troubleshooting](#when-something-gets-in-the-way)
Viewers need a browser and an invitation. To start sharing, download Piik App
@@ -154,6 +155,36 @@ then select a picture card to start sharing. Device names and previews depend on
browser permissions; while a camera is shared, other cameras are listed by name.
Use headphones to keep speaker sound from feeding back into the microphone.
+### Exclude a voice app from screen audio
+
+On supported Windows Apps, open the **Screens** tab and enable **System sound**.
+Under **Exclude application audio**, select a window belonging to the voice app,
+then choose the screen to share. Open the app's window and refresh if it is absent.
+This excludes one process and its children. Other apps remain audible, including
+those playing through other output devices.
+It does not mute your Piik microphone or prevent speaker sound reaching a microphone.
+
+If the excluded app closes or restarts, source audio stops. Use **Switch source**
+and select it again. Turning source audio off retains the exclusion choice;
+select **None** to restore unfiltered system audio. This control requires an updated page and a Windows App
+whose capability probe supports process exclusion; it is not a Browser-capture filter.
+
+## Chat and reactions
+
+Once the host starts the first share, participants can select **Chat** below the
+sofa to send messages. Select a person on the sofa to open their reaction menu:
+your own reactions appear above you; reactions sent to a friend fly towards them.
+The chat window can be moved and resized without rearranging the page.
+
+**On screen**, beside the message box or in the viewer's playback bar, shows new
+messages over the picture. It is off by default and only changes your own view.
+System video fullscreen and picture in picture cannot show these overlays.
+
+Stopping a share keeps the room's chat available. App Local and Public invite
+rooms require the App to stay open. Each page retains only its latest received
+messages: refreshing clears them, and late joiners receive no earlier history.
+Chat travels through the room's server and is **not end-to-end encrypted**.
+
## When something gets in the way
For sound, capture, startup or playback issues, see [Troubleshooting](./troubleshooting.md).
diff --git a/docs/guide/getting-started.zh-CN.md b/docs/guide/getting-started.zh-CN.md
index ef3ef23..147e12f 100644
--- a/docs/guide/getting-started.zh-CN.md
+++ b/docs/guide/getting-started.zh-CN.md
@@ -6,7 +6,8 @@ Piik 支持一位房主分享屏幕,最多 20 位受邀观众通过浏览器
本指南介绍加入房间、使用在线版、使用 Piik App,以及通过已有站点分享的方法。
[加入房间](#加入朋友的房间) · [在线版](#使用在线版) ·
-[Piik App](#用-piik-app-分享) · [已有站点](#使用已有站点分享) · [问题排查](#遇到问题了)
+[Piik App](#用-piik-app-分享) · [已有站点](#使用已有站点分享) ·
+[聊天与表情](#聊天与表情) · [问题排查](#遇到问题了)
观众只需要浏览器和邀请链接。
分享画面推荐下载 Piik App,选择 **公网邀请**。
@@ -133,6 +134,31 @@ Linux 原生采集还需要[系统组件](../../native/capture/linux/README.md)
设备名称和可用预览取决于浏览器权限;正在分享摄像头时,其他摄像头只显示名称。
建议佩戴耳机,避免扬声器声音被麦克风再次收进去。
+### 分享屏幕时排除语音软件的声音
+
+在支持此功能的 Windows App 中,打开 **屏幕** 页签并开启 **系统声音**,
+在 **排除应用声音** 中选择语音软件的窗口,再选择要分享的屏幕。
+未列出的应用请先打开窗口,再刷新。此功能排除所选进程及其子进程,
+其他应用的声音仍会被分享,包括输出到其他播放设备的声音。
+它不影响 Piik 麦克风,也不能消除扬声器被麦克风收音产生的回声。
+
+所选应用关闭或重启后,来源声音会停止;请通过 **切换来源** 重新选择。
+暂时关闭来源声音会保留排除选择;需要恢复完整的系统声音时,选择 **不排除**。
+此控件需要新版页面和通过进程排除能力检测的 Windows App,不适用于浏览器自带的分享选择器。
+
+## 聊天与表情
+
+房主首次开始分享、创建房间后,参与者可以点击沙发下方的 **聊天** 发送消息。
+点击沙发上的人物可打开表情菜单:给自己发的表情出现在头顶,送给朋友的表情会飞向对方。
+聊天窗口可以移动和缩放,不会挤动页面。
+
+输入框旁和观众播放控制栏中的 **弹幕** 可以把新消息显示在画面上,默认关闭,只影响自己的观看。
+系统视频全屏和画中画无法显示弹幕。
+
+停止分享后仍可继续聊天;App 的本地房间和公网邀请需要保持 App 运行。
+每个页面只保留本页最近收到的消息,刷新后清空,后来加入的人看不到此前的记录。
+聊天消息经过房间所在的服务器,**不提供端到端加密**。
+
## 遇到问题了
声音、采集、启动或播放遇到问题时,先看[问题排查](./troubleshooting.zh-CN.md)。
diff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.md
index b5ae60e..fed063f 100644
--- a/docs/guide/troubleshooting.md
+++ b/docs/guide/troubleshooting.md
@@ -147,8 +147,9 @@ Try these in order, stopping when the picture arrives:
6. **Invite a friend on another network.** A Viewer who can receive the picture
and has spare forwarding capacity may give Piik another path to you. Piik
chooses this automatically; extra people help only when those connections work.
-7. **Use a site with media fallback.** Its operator must enable SFU forwarding,
- and the Host must turn off **Privacy mode** before sharing. The public
+7. **Use a site with media fallback.** Its operator must enable SFU forwarding.
+ If the site offers **Privacy mode**, the Host must turn it off before sharing.
+ Sites configured for **Server media** already require SFU. The public
`demo.piik.tv` site and the App's **Public invite** mode do not provide SFU.
[Self-hosting](../operations/self-hosting.md#optional-media-fallback) is an
advanced option. Both P2P and SFU media currently use UDP; a network that blocks
diff --git a/docs/guide/troubleshooting.zh-CN.md b/docs/guide/troubleshooting.zh-CN.md
index 92d197b..6bae1e0 100644
--- a/docs/guide/troubleshooting.zh-CN.md
+++ b/docs/guide/troubleshooting.zh-CN.md
@@ -123,7 +123,8 @@ NAT(网络地址转换)让多台设备共用一个上网地址。不同路
修改路由器工作模式前,先备份配置。
6. **请另一网络的朋友加入。** 如果对方能正常看到画面,且有余力转发,Piik 可能经由这位观众
把画面送到你这里。线路由 Piik 自动选择,能否帮上忙取决于这些连接是否可用。
-7. **使用有媒体兜底的站点。** 管理员需要先启用 SFU 转发,房主在分享前关闭 **隐私模式**。
+7. **使用有媒体兜底的站点。** 管理员需要先启用 SFU 转发。如果页面提供 **隐私模式**,
+ 房主应在分享前关闭;配置为 **服务器转发** 的站点已经强制使用 SFU。
公开站点 `demo.piik.tv` 和 App 的 **公网邀请** 模式均没有 SFU 兜底。
有服务器的用户也可以按 [部署指南](../operations/self-hosting.zh-CN.md#可选启用媒体兜底) 配置,属于进阶操作。
目前 P2P 和 SFU 媒体都使用 UDP;网络完全禁止 UDP 时,仍需换网络或由管理员放行。
diff --git a/docs/operations/self-hosting.md b/docs/operations/self-hosting.md
index fb73ea0..a8c1108 100644
--- a/docs/operations/self-hosting.md
+++ b/docs/operations/self-hosting.md
@@ -112,6 +112,9 @@ permission; `self` allows this site to request access.
Allow **TCP 80/443** for HTTPS and **UDP 3478** for STUN in the server firewall
and cloud security group. Keep TCP 8787 private. The STUN hostname must resolve
directly to the server; a CDN HTTP proxy does not forward its UDP traffic.
+See the [complete port table](../standards/configuration.md#public-and-private-ports)
+for optional services. Embedded SFU media shares the single UDP port set by
+`SFU_UDP_PORT`.
Open `https://share.example.com/healthz`; it should return `{"status":"ok"}`.
Then open the site, share a screen and join from another device. This initial
@@ -122,9 +125,15 @@ configuration uses P2P media, so participants need a usable UDP path.
Add `SFU_UDP_PORT=7882` to `.env`, allow UDP 7882, and restart Piik to enable
automatic SFU fallback. When the server is behind NAT, also set `SFU_PUBLIC_IP`
to its reachable public IPv4 address. The same binary provides the fallback.
+Allow server forwarding capacity and outbound bandwidth for Viewers actually
+using the SFU.
The Host must turn off **Privacy mode** before sharing to allow this route.
Both direct and SFU media need a usable UDP connection.
+To send every room through your server, also set `SFU_ONLY=true` and restart.
+The sharing settings will show **Server media**. Plan server bandwidth for every
+Viewer; this mode does not fall back to P2P if the server route fails.
+
## Keep it running and update
For automatic startup, use the [systemd or container guide](./service-management.md).
diff --git a/docs/operations/self-hosting.zh-CN.md b/docs/operations/self-hosting.zh-CN.md
index 9ad9cdd..13501a2 100644
--- a/docs/operations/self-hosting.zh-CN.md
+++ b/docs/operations/self-hosting.zh-CN.md
@@ -101,6 +101,8 @@ nginx 可参考[配置示例](../../deploy/nginx/piik.conf.example)。如果沿
在服务器防火墙和云平台安全组中放行 **TCP 80/443**(HTTPS)和 **UDP 3478**(STUN)。
TCP 8787 仅供本机反向代理访问。STUN 域名需要直接解析到服务器,不能只经过 CDN 的 HTTP 代理。
+可选服务的端口见[完整端口表](../standards/configuration.md#public-and-private-ports)。
+内置 SFU 的媒体流复用 `SFU_UDP_PORT` 指定的单个 UDP 端口。
打开 `https://share.example.com/healthz`,应返回 `{"status":"ok"}`。
随后打开站点,分享一个画面,并用另一台设备加入验证。
@@ -111,9 +113,14 @@ TCP 8787 仅供本机反向代理访问。STUN 域名需要直接解析到服务
在 `.env` 中添加 `SFU_UDP_PORT=7882`,放行 UDP 7882,再重启 Piik,即可启用自动 SFU 兜底。
如果服务器处于 NAT 后方,还需将 `SFU_PUBLIC_IP` 设置为外部可达的公网 IPv4 地址。
这项功能由同一个服务端程序提供。
+服务器需要为实际经由 SFU 观看的观众承担转发负载和出口带宽。
房主需要在开始分享前关闭 **隐私模式**,才会允许使用这条线路。
直连和 SFU 媒体都需要可用的 UDP 通路。
+如需所有房间都通过服务器转发,再设置 `SFU_ONLY=true` 并重启。
+分享设置中会显示 **服务器转发**。请为每位观众预留服务器带宽;
+服务器线路失败时,此模式不会改走 P2P。
+
## 长期运行与更新
需要开机启动时,参阅 [systemd 与容器配置](./service-management.zh-CN.md)。
diff --git a/docs/project-memory.md b/docs/project-memory.md
index 8b4f721..e257a60 100644
--- a/docs/project-memory.md
+++ b/docs/project-memory.md
@@ -1,6 +1,6 @@
# Project Memory
-Last updated: 2026-09-29
+Last updated: 2026-10-01
Piik is private, low-latency screen sharing for one Host and up to 20
authenticated friends. The current product surface is Web Host, Web Viewer, and
@@ -43,7 +43,8 @@ blank keeps the self-contained site open.
## Product Map
- [Rooms and access](./standards/rooms-access.md) owns room codes,
- invitations, code entry, credentials, and lightweight/SQLite persistence.
+ invitations, code entry, credentials, room interactions and lightweight/SQLite
+ persistence.
- [Routing and transport](./standards/routing-transport.md) owns P2P-first
distribution, endpoint capacity, SFU fallback, recovery, and privacy limits.
- [Capture and media quality](./standards/media-quality.md) owns Browser capture,
@@ -70,8 +71,9 @@ implementation and routine UI detail.
available, and anonymous provenance distinguishes ordinary from predicted
selected paths. The configuration default remains stock ICE. The only
application fallback is one bounded embedded SFU/UDP
- publication; Piik configures no TURN or TCP media route. A pre-share
- peer-only policy can exclude that SFU suffix for one share generation.
+ publication; Piik configures no TURN or TCP media route. Hosted operators may
+ require SFU-only media through the same controller. On P2P-first sites, a
+ pre-share peer-only policy can exclude that SFU suffix for one share generation.
- Every endpoint shares one steady outbound-copy cap, default `2` and limited to
`1..3`. One room controller owns one committed graph, one reconcile loop, and
one serial child operation. First decoded frame commits availability work;
diff --git a/docs/research/browser-local-encoding-pool.md b/docs/research/browser-local-encoding-pool.md
index 74a2b9d..873c448 100644
--- a/docs/research/browser-local-encoding-pool.md
+++ b/docs/research/browser-local-encoding-pool.md
@@ -93,6 +93,14 @@ maximum queued service; reverse feedback is unshaped. They are not a model of
public NAT, geographic RTT or every congestion/loss pattern. Reduced quality
and temporary stalls under real constraints remain possible.
+A 2026-09-30 actual-tab fault check separated A at 400 kbps from healthy B,
+then failed only A's producer. Both H264 and VP8 retained the outgoing
+connections, returned to ordinary encoding and retired all pooled groups.
+B retained 1904x1048 at about 30 fps; its largest observed frame-callback gap
+around fallback was 271 ms for H264 and 80 ms for VP8. Source-wide fallback
+therefore has a measurable handoff cost, even without a resolution drop or
+connection failure. This does not establish low-end hardware capacity.
+
### Balanced Startup And Recovery
Serial Chrome 152 checks on 2026-09-25 reproduced two composition defects in
@@ -316,12 +324,19 @@ complexity: changing to simple bars restored 1080p, retained after restoring
the textured scene. WebRTC's [quality scaler](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/modules/video_coding/utility/quality_scaler.cc)
uses QP thresholds and sampling history as well as bandwidth-related adaptation.
Its default two-second checks and longer sampling after down-adaptation can
-outlast a short disturbance. Native Chrome 154 logging also confirmed a stable
-QP deadband with the NVIDIA H264 MFT: after one 1048p-to-786p reduction, smoothed
-QP stayed between the recorded 24/37 thresholds for the remaining 90-second
-observation. Capture remained full size, but no low-QP upscale was requested.
-This is not merely old samples taking time to expire; restored bandwidth alone
-does not require this scaler to restore resolution.
+outlast a short disturbance. A Chrome 154 ordinary-sender tab control on
+2026-09-30 remained at 524p / about 30 fps after 120 recovery seconds following two
+one-second rate/loss pulses. Actual capture stayed 1904x1048 / 30 fps; the native
+log's slow QP filter settled around 27, above its upscale threshold of 24.
+Changing only the scene to simple moving bars restored full size in about
+12 seconds. Restoring the textured scene then retained full size for the
+remaining 30 seconds. The sender made only its two startup parameter writes;
+no pool, reconnect, preference reset or frame reset caused that recovery.
+This confirms a content-dependent stable state, not merely stale samples or
+a blocked recovery operation. Restored bandwidth alone does not require this
+scaler to restore resolution. The scaler and default-threshold source matched
+inspected upstream main on that date; this does not establish every device's
+behavior or the field report's cause.
A matched three-arm comparison used fixed 5 Mbps, alternating 5.0/4.9 Mbps
every 500 ms, and repeated identical 5 Mbps writes. Native reconfiguration
@@ -335,6 +350,149 @@ perceptual blur measurement. They do not justify delaying real demand, bitrate
compensation or periodic resets; equal product updates are already deduplicated.
The field's repeated brief blur still lacks matching Host histories.
+**Hardware encoder startup can block sibling encoding.** On 2026-10-01,
+Windows Chrome 154.0.8037.58 tests added three independent Canvas H264 producers,
+coinciding with 174/221/225 ms gaps in the existing producer; captured frames
+continued within 47 ms. The new producers never joined the pool or replaced its
+output, and the existing
+producer's parameters did not change. Their NVIDIA initialization took
+161/163/162 ms. Substituting software VP8, confirmed by RTCStats as libvpx, gave
+60/48/109 ms existing-producer gaps in the same startup windows.
+A subsequent trace recorded 148 ms of `CreateAndInitializeVEA` on the shared
+Renderer Media thread, matching GPU-side initialization. Frames timestamped
+during that interval entered encoding together on the same thread after it
+returned. This establishes blocking in the local reproduction, consistent with
+[Chromium's synchronous initialization IPC](https://github.com/chromium/chromium/blob/154.0.8037.58/media/mojo/clients/mojo_video_encode_accelerator.cc#L117).
+The trace buffer lost events elsewhere; this conclusion uses complete task and
+frame events, not missing events. It does not identify the driver's internal
+cost, explain earlier downscaling or attribute unmatched field reports. No
+codec-default change, spare prewarmed encoder or pool reset follows from it.
+The Windows [VEA provider](https://github.com/chromium/chromium/blob/154.0.8037.58/media/mojo/services/mojo_video_encode_accelerator_provider.cc)
+already assigns each service a dedicated COM STA runner; the trace likewise
+places the old service's encoding and new initialization on different threads.
+Asynchronous Renderer initialization is therefore a plausible upstream repair,
+not merely moving the wait onto another shared queue. It still needs a patched
+Browser comparison, including initialization failure and callback ordering;
+shared GPU/driver cost remains unproved.
+
+**Optional reuse must preserve the existing rate owner.** A two-Viewer,
+textured-tab H264 comparison exposed a Piik handoff defect. During recovery
+from a one-second rate/loss pulse, a healthy Viewer moved from a 5 Mbps group
+to a lower-budget group because its sampled output was 29 rather than 27 fps
+at the same 1048p. Both reported `qualityLimitationReason:none`, but the same
+NVIDIA encoder implementation's interval-average QP was about 51 on the
+candidate versus 37 on the current output; the candidate then fell to 786p. The
+[stats definition](https://www.w3.org/TR/webrtc-stats/#dom-rtcqualitylimitationreason-none)
+describes resolution/FPS limitation, not equal quantization quality.
+
+Pending membership raised the candidate's budget before the existing
+lower-budget rejection ran. The candidate repair checks this constraint before
+membership and reuses one non-regression predicate before submitting a pending
+selection. FPS comparisons also respect the Host ceiling; a sampled 32 fps
+under a 30 fps ceiling is not an improvement over 30 fps. Initial output and
+necessary downgrades retain their path, without a QP policy, timer or encoder reset.
+
+Selection also needs proof at the actual write boundary: while waiting for a key,
+a candidate can downscale after passing the sampled comparison. The output now
+asks the pool to revalidate current authority, budget and actual key dimensions
+before replacing its queue or writing bytes. Rejection clears both pending owners,
+invalidates a contradicted sample and retains prior output and recovery requests.
+H264/VP8 display-capture checks preserve live/paused settings, source replacement
+and ordinary fallback; played audio remains present. Brief video stalls remain.
+
+The bounded H264 and VP8 two-Viewer controls for this admission repair retained
+the healthy sibling at 1048p and about 29 fps through both recovery intervals.
+The constrained Viewer still adapted and recovered. This establishes one
+avoidable sibling degradation path, not the field room's cause or elimination
+of all brief blur. The affected admission rules predate the display-isolation
+repair. Raw observations stay in ignored fixtures; comparisons read native stats
+without calling the producer's startup-transition helper.
+
+Current demand also matters at group admission. Two-Viewer traces showed a
+newcomer raising an incumbent group's allocation because matching used the
+previous applied budget. The incumbent then needed another encoder, while the
+newcomer's former encoder retired. Matching current native demand prevents that
+exchange; a pending newcomer cannot establish compatibility with its own vote.
+Cancellation must stop at accepted output writing, including a pause that delays
+its completion. These are ownership repairs, not suppression of real demand
+changes or removal of the measured hardware initialization cost.
+
+**Quiet content is not rate compatibility.** A separate fixed-allocation
+comparison kept the two outgoing H264 ceilings at 3 and 5 Mbps, with no network
+shaping, while alternating simple and textured moving content. Actual native
+allocations stayed at those ceilings during the later scene cycles. Reuse based
+on a quiet 500 ms byte-rate sample repeatedly moved the weaker child into the
+5 Mbps producer; returning motion split it out again. Five producers were created
+over the run. Requiring the candidate's rate budget to fit the child's allocation
+at initial admission, optional reuse and pending revalidation retained two
+producers with no scene-driven handoffs. The healthy H264 sibling retained full
+1048p; the weaker output continued native adaptation. VP8 also retained two
+producers through the same scene changes. Three controlled regressions fail
+when these checks are removed, including a paused pending child whose candidate
+budget rises. Unequal demands may retain an additional producer even during a
+quiet scene; bounded membership and native recovery remain unchanged. These
+checks remove unnecessary encoder churn, not all content-dependent adaptation.
+
+**A known allocation should initialize the local encoder.** A replacement local
+PeerConnection otherwise starts its own estimate at WebRTC's default 300 kbps,
+even when the outgoing connection already allocates 4 Mbps. Chrome 154's native
+logs confirm this second cold start. On the same isolated 1904x1048 textured tab,
+seeding only the local answer with that 4 Mbps allocation increased VP8 output
+in the first second from 6 to 27 frames; its first sampled target was 3.33 Mbps
+rather than 2.70 Mbps. The 400 kbps control retained its native 333 kbps initial
+encoder target and two first-second frames with either version. These bounded
+results do not promise faster first frames or remove weak-network adaptation.
+
+The hint uses the existing Host-bounded allocation, not a multiplier or floor.
+WebRTC reads the send configuration from the
+[remote description](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/pc/channel.cc)
+and accepts a positive `x-google-start-bitrate` in its
+[codec bitrate configuration](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/media/engine/webrtc_media_engine.cc).
+An offer-only control lost the hint in Chrome's answer and did not improve
+startup. Piik therefore changes only its owned local answer; external SDP,
+codec identity and subsequent sender-budget updates stay unchanged. The
+application writes this hint once per producer, not on each budget change.
+The full two-Viewer VP8 follow-up retained the healthy child at 1048p through
+two one-second 400 kbps / 4% loss pulses at 40 ms configured RTT. The constrained
+child still down-adapted and froze during recovery, but regained 1048p within
+each 40-second recovery window. This is not a freeze-free claim. A separate
+audio/loss check delivered all 97 scheduled audio pulses to both played outputs;
+H264 quiet-start, live/paused settings, source replacement and ordinary-fallback
+checks passed. All owned test connections, processes and profiles retired.
+
+**Residual VP8 stalls do not require an output wait.** The ordinary two-Viewer
+control with the same source and rate/loss pulses also froze and downscaled,
+ending the recovery windows at 786p and 524p. It entered the second pulse at a
+lower resolution than pooling, so their recovery FPS is not a fair cost comparison.
+Replaying the pooled child's budget sequence on a warmed local producer without
+a shaper, carrier, queue or handoff produced nine frames in the same eight-second
+low-budget interval; adding the two original keyframe requests yielded eight,
+matching the original interval. Startup protection had already ended. Native
+logs show frame dropping and the scaler's spaced downscales. This establishes
+stalls under that budget sequence, not the cause of the sequence: keyframe bursts
+can still affect downstream estimation, and parameter reconfiguration remains
+in the replay. These results do not support removing dependency recovery or
+startup protection, growing queues or repeatedly resetting encoders. Matched
+field evidence and broader transport/allocation effects remain open.
+The rate/loss run with audio retained all 65 pulses after startup at both
+played outputs, despite video stalls; this does not establish gap-free video.
+A separate six-second complete RTP-media drop on one child, preserving RTCP
+and signaling, resumed its played video 157 ms after release for VP8 and 128 ms
+for H264. Both healthy siblings retained full dimensions; H264 still recorded
+brief sibling stalls. All post-release audio pulses arrived. This bounded local
+check does not establish recovery for an interrupted relay or reporter's network.
+
+A matched single-Viewer control separates pool handoffs from ordinary adaptation.
+Pooled sending used one producer with no handoffs; ordinary sending used no pool
+and made only two parameter writes. After the second one-second rate/loss pulse,
+both reached 262p before ending their 40-second recovery window at 390p / 15 fps.
+Actual source frames stayed 1904x1048 at about 30 fps. Native logs showed quality
+scaler down-adaptation, Chromium's below-360p software fallback and subsequent
+recovery to NVIDIA H264 at 390p; this was not a hardware failure. Pool reuse and
+frequent application-level budget writes are therefore not necessary for this
+bounded low-output state. Their effect on the intervening trajectory and the
+unmatched field reports remains distinct.
+
The tab case exposed a more specific capture-feedback problem. A Chromium trace
kept the source geometry at 1904x1049 while capture supplied a 268x148 content
rectangle. Feedback rose to 66,600 pixels, but capture did not grow.
diff --git a/docs/research/media-fidelity.md b/docs/research/media-fidelity.md
new file mode 100644
index 0000000..eaa7ec2
--- /dev/null
+++ b/docs/research/media-fidelity.md
@@ -0,0 +1,162 @@
+# Media Fidelity
+
+Reviewed 2026-10-01. [Media quality](../standards/media-quality.md) owns behavior;
+[TODO](../todo.md) owns remaining work. These synthetic checks do not establish
+all-device acceptance or full HDR support.
+
+## Stereo
+
+Browser display capture requests two channels without speech processing. The
+Host mixer uses a stereo Web Audio destination; Native capture and mixing use
+48 kHz, two-channel PCM16. Native Opus uses the audio application mode. Browser,
+Native and SFU receivers request stereo. The existing 64/128/192 kbps choices
+are ceilings, not a lossless-audio promise.
+
+`opus/48000/2` alone does not prove stereo content: Opus permits mono packets
+under that declaration. [RFC 7587](https://www.rfc-editor.org/rfc/rfc7587.html#section-7)
+distinguishes receive preferences from packet content. Test distinct signals
+after decoding instead of inferring fidelity from SDP or a two-channel mixer.
+
+The embedded SFU browser gate sends 500 Hz left and 1500 Hz right, and measures
+decoded channel separation. On local Chrome 154, both the default fallback and
+SFU-only arms retained more than 77 dB separation in both channels at the 64 kbps
+ceiling, including normal video reception. Source changes and subscription
+recovery retain the existing audio gate. Run `npm run gate:embedded-sfu` with `PIIK_EMBEDDED_SFU_GATE=true`,
+`CHROME_PATH`, and optional `PIIK_EMBEDDED_SFU_ONLY=true`; it cleans up its own
+processes and ports. This is synthetic local evidence, not an acoustic or
+public-network measurement.
+
+A local Chrome 154 check through the production Host/Viewer peers retained more
+than 57 dB separation after direct reception and after one decoding/re-encoding
+relay, with a third tone mixed as Host microphone input. The receiver used its
+normal video playback sink; a receiver-only Web Audio probe without playback
+returned silence and was not a fidelity result. The Native Opus regression test
+independently requires more than 20 dB decoded separation at all three bitrates.
+These checks support keeping the existing stereo path. They do not cover every
+physical source device, browser or acoustic setup, and do not establish surround
+sound support.
+
+## SDR Conversion
+
+[#445](https://github.com/TNTcraftHIM/Piik/issues/445) reports washed-out Native
+capture on an SDR Windows 10 system. Its proposed explanation is not established:
+Microsoft documents full-range RGB as the
+[video processor's default input range](https://learn.microsoft.com/en-us/windows/win32/api/d3d11/ns-d3d11-d3d11_video_processor_color_space).
+The confirmed gap was that `FrameConverter` left its input/output range and
+matrix implicit. The same converter handles BGRA capture and decoded NV12
+relay input; correcting only screen capture would leave the relay inconsistent.
+
+A local D3D11 check through the previous converter produced full-range NV12 from
+BGRA color bars (black Y=0, white Y=255). The production `AdaptiveEncoder` VP8
+output then reproduced crushed shadows and clipped highlights in Chrome 154:
+
+| Source gray RGB | Previous conversion, WebRTC playback | Explicit limited-range BT.601 |
+| --- | --- | --- |
+| 0 | 0 | 0 |
+| 32 | 19 | 33 |
+| 128 | 130 | 128 |
+| 235 | 255 | 235 |
+| 255 | 255 | 255 |
+
+The fixture encoded 256x128 color bars with the production converter/encoder,
+injected those keyframes into a local VP8 PeerConnection without the color-space
+RTP extension, and sampled the playing video into a canvas. The experimental
+conversion also restored red/green/blue bars to within one channel value of the
+source. This establishes a bounded SDR defect, not the cause of every color
+report.
+
+WebCodecs and WebRTC are not interchangeable acceptance paths: explicit BT.709
+looked correct through WebCodecs but shifted primary colors through this WebRTC
+receiver. Chromium's
+[remote-frame conversion](https://chromium.googlesource.com/chromium/src/+/master/third_party/blink/renderer/modules/peerconnection/media_stream_remote_video_source.cc)
+can use BT.601 when color metadata is unspecified. White Y=235 is normal for
+limited-range YUV; it must become display RGB=255, not be treated as a defect by
+itself.
+
+The candidate now declares RGB capture as full range and reads decoded H.264
+color metadata from Media Foundation. Unspecified SDR and VP8 use the existing
+WebRTC BT.601 convention. The converter produces limited-range BT.601 NV12;
+the H.264 media types declare matching range, matrix, primaries and transfer.
+Setting only matrix/range did not produce matching H.264 VUI on the tested
+NVIDIA encoder. No wire field or user setting is added.
+
+Use the Windows 10
+[explicit video color-space API](https://learn.microsoft.com/en-us/windows/win32/api/d3d11_1/nf-d3d11_1-id3d11videocontext1-videoprocessorsetstreamcolorspace1).
+The legacy nominal-range API squeezed decoded NV12 a second time in local tests.
+Even with the explicit API and a positive format-conversion capability result,
+NV12 BT.709-to-BT.601 conversion changed levels on NVIDIA when scaling and left
+the matrix unchanged on AMD. Noncanonical NV12 therefore passes through RGB on
+the GPU before the shared output conversion. Ordinary RGB capture and already
+canonical NV12 keep one pass; no CPU readback, vendor branch or custom shader
+is added to the product path.
+
+The opt-in `-CheckColor` probe in the [Windows capture guide](../../native/capture/windows/README.md)
+checks full/limited-range BT.601/BT.709 input, scaling and input replacement
+through the production converter. It passed on local RTX 4070 SUPER and AMD
+Radeon adapters. RTX H.264/VP8 encoding followed by Chrome 154 WebRTC playback
+also passed direct reception, decode/scale/re-encode and explicitly tagged
+BT.709 H.264 input to both output codecs; sampled RGB errors were at most 5/255.
+AMD hardware encoding was not accepted locally: MFT activation returned
+`0x8007000e` before color configuration. These local checks do not settle the
+reporter's Windows 10 environment, all hardware encoders, HDR content or missing
+upstream color declarations.
+
+## HDR To SDR
+
+[#420](https://github.com/TNTcraftHIM/Piik/issues/420) reports overexposure with
+both App and Browser capture on one Windows HDR machine. The report does not
+locate the failing stage.
+
+The previous native WGC frame pool and thumbnails requested BGRA8, discarding
+HDR range before the existing 8-bit H.264/VP8 encoder. Microsoft's
+[capture guidance](https://learn.microsoft.com/en-us/windows/apps/develop/media-authoring-processing/screen-capture)
+warns that HDR needs floating-point capture throughout the input pipeline to
+avoid clipping. Reducing brightness after encoding cannot restore clipped
+highlights. This is a confirmed implementation gap, not proof that every
+reported Browser failure has the same cause.
+
+The candidate uses one `CaptureSdrConverter` before native output fanout and in
+thumbnails. HDR displays use an scRGB FP16 pool, then Windows Direct2D's
+[HDR tone-map effect](https://learn.microsoft.com/en-us/windows/win32/direct2d/hdr-tone-map-effect)
+compresses luminance, the
+[white-level effect](https://learn.microsoft.com/en-us/windows/win32/direct2d/white-level-adjustment-effect)
+accounts for the source display's SDR white, and color management converts to
+sRGB BGRA8. Ordinary SDR retains its texture-copy path. No custom shader,
+CPU readback, new packaged dependency or per-output tone mapper is added.
+
+Source-monitor movement and DXGI factory invalidation refresh color metadata,
+following [GetDesc1's contract](https://learn.microsoft.com/en-us/windows/win32/api/dxgi1_6/nf-dxgi1_6-idxgioutput6-getdesc1).
+A format change recreates the WGC pool through its existing resize boundary;
+it does not restart routes or encoders. WGC supplies no per-frame MaxCLL, so
+conversion uses the display peak (at least the configured SDR white), or the
+HDR10 10,000-nit range when usable peak metadata is absent. This stable platform
+policy avoids a second image-content adaptation loop; real-display appearance
+remains an acceptance requirement.
+
+The retained `-CheckColor` probe passed on RTX 4070 SUPER and AMD Radeon:
+0–1000-nit neutral bars preserved distinguishable highlights, source white
+80/240/80 nits and SDR/HDR size changes preserved ownership and ordinary SDR
+colors. Production H.264/VP8 output decoded over Chrome 154 WebRTC reproduced the
+converted SDR gray bars within 1/255. These are synthetic input checks, not
+observations of an HDR game or the reporter's machine.
+
+Warmed GPU timestamp measurements for conversion, including the owned input
+copy, had the following local medians. Dimensions are captured input sizes,
+not newly supported sharing presets.
+
+| Adapter | 1920x1080 | 3840x2160 |
+| --- | --- | --- |
+| RTX 4070 SUPER | 0.045 ms | 0.240 ms |
+| AMD Radeon | 2.39 ms | 11.17 ms |
+
+HDR conversion has a real GPU cost on weaker devices. These bounded measurements
+exclude encoding and do not establish gaming/endurance performance. Current
+monitors were SDR: physical HDR highlights, SDR content on HDR displays,
+thumbnails, mixed displays and mode changes under load remain unaccepted.
+Browser capture owns its own conversion; inspect the acquired frame before
+assigning a downstream bug.
+
+Full HDR requires a separate end-to-end decision for bit depth, codec support,
+color metadata, decoding and mixed Viewer capabilities. It is not a quality
+dropdown addition. Preserve ordinary SDR routing and existing compatibility
+with HDR-to-SDR; no full-HDR or surround implementation is claimed.
diff --git a/docs/research/native-client-media.md b/docs/research/native-client-media.md
index bb6a810..f5f9112 100644
--- a/docs/research/native-client-media.md
+++ b/docs/research/native-client-media.md
@@ -1,11 +1,47 @@
# Native App Media Evidence
-- Reviewed: 2026-09-29
+- Reviewed: 2026-10-01
- Scope: platform capture, shared encoded sources, Pion transport and Browser
decode; current behavior belongs to [media quality](../standards/media-quality.md)
- Status: Windows physical native Host and Viewer gates passed; macOS and Linux adapters
compile and package but still require physical media gates
+## Windows Audio Exclusion
+
+A Windows 11 build 26200 check used two independent WinMM processes emitting
+500 Hz and 1500 Hz tones, through the production capture helper and Go PCM
+reader. Default-device loopback captured both at approximately 1799 amplitude;
+process inclusion retained the first with under 0.3 amplitude of the other,
+and exclusion retained the second with under 0.04 of the excluded tone. Target
+exit retired capture in about 42 ms; attempting the stale identity failed.
+No raw audio was saved. The extended capability probe completed in 300 ms.
+
+A parent/child follow-up used a silent selected parent with a 1900 Hz child,
+plus an independent 2900 Hz process. Process-scoped reference amplitudes were
+about 1799; exclusion retained the independent tone at 1800 and reduced the
+selected child's tone to 9.3 (under 0.6%). Parent exit retired capture in 64 ms;
+its stale identity was rejected. The reference uses each signal's own process
+capture: default-device loopback also includes unrelated playback and endpoint
+processing, so its amplitude is not a process-capture gain reference.
+
+The session fixture separately verifies failed exclusion replacement mutes
+source audio while preserving video, microphone and encoded audio output; pause
+and audio-profile changes retain the selected input. These checks establish a
+local Windows implementation and one direct child, not coverage of every voice
+application's process tree or simultaneous playback across several output devices. Those remain
+device acceptance boundaries. [ADR-0008](../adr/0008-window-scoped-audio-capture.md)
+owns the platform choice.
+
+The [Windows activation contract](https://learn.microsoft.com/en-us/windows/win32/api/audioclientactivationparams/ns-audioclientactivationparams-audioclient_process_loopback_params)
+accepts one PID and its process tree, not a list of unrelated processes.
+The [win-capture-audio implementation](https://github.com/bozbez/win-capture-audio/releases/tag/v2.2.0-beta)
+supports broader exclusion by tracking audio sessions, capturing non-matching
+processes and mixing them. Applying that design here would require session
+discovery, process-tree deduplication and bounded multi-input mixing; Piik's
+current source-plus-microphone mixer does not own that inventory. Multiple
+exclusions therefore require a capture design change, not just a multi-select
+control. Combining exclusion streams does not itself exclude their combined set.
+
## Initial Physical Evidence (2026-09-05)
These checks established the original native media paths. The Browser-mediated
@@ -271,6 +307,33 @@ that result nor the allocation timings establish the reported system-wide lag's
cause. Retain the existing implementation; reopen optimization from a matched
bottleneck.
+## Native H.264 Motion Quality
+
+Issue [#432](https://github.com/TNTcraftHIM/Piik/issues/432) reports more visible
+blocking with native H.264 than VP8 on v1.6.7. Review of its attached Browser
+report confirms 720p30 in both comparison windows, roughly 11 Mbps received,
+and no reported packet loss; the 1080p60 H.264 window receives roughly 10 Mbps.
+Those samples establish delivered format and transport observations, not equal
+source content or objective picture quality. No encoded stream, reference
+frames or QP measurements accompany that report.
+
+The reporter retracted the initial CBR bitrate-ceiling diagnosis after finding
+that the experimental fixture changed bitrate during the measured interval.
+That switching belongs to `PIIK_H264_FIXTURE` in `windows/main.cpp`, not live
+capture. The corrected experiment reports CBR reaching its requested bitrate.
+Production `h264_encoder.cpp` explicitly requests and validates Baseline; the
+Browser's negotiated `profile-level-id` alone is not an inspection of emitted
+SPS. The later experimental CABAC result cannot establish the production
+Baseline bitstream's behavior. Microsoft's [encoder reference](https://learn.microsoft.com/en-us/windows/win32/medfound/h-264-video-encoder)
+distinguishes profile, rate control and optional quality/speed settings.
+
+The original 720p comparison remains unresolved. An equal-content comparison
+must preserve actual bitrate, frame rate, latency and production codec settings,
+inspect the resulting bitstream and retain reference/decoded pictures. Raising
+the bitrate ceiling, lowering the default frame rate or changing profile is not
+justified by these logs alone; those changes have bandwidth, motion and receiver
+compatibility consequences.
+
## Implementation Boundary
- `nativecapture` owns the child process, source identity, and bounded frame protocol.
diff --git a/docs/research/room-interactions.md b/docs/research/room-interactions.md
new file mode 100644
index 0000000..a114cd9
--- /dev/null
+++ b/docs/research/room-interactions.md
@@ -0,0 +1,260 @@
+# Room Interaction Design
+
+The owner accepted the text/reaction UI and model with bounded local acceptance.
+[Status](../status.md#accepted-interaction-phase) owns release readiness and
+[TODO](../todo.md#now) owns remaining work. Reviewed 2026-10-02.
+This document records the implementation design and its tradeoffs;
+[rooms and access](../standards/rooms-access.md#room-interactions) owns authority
+and retention, while [presentation and lifecycle](../standards/presentation-lifecycle.md#product-surface)
+owns room/publication lifetime.
+
+## Model And Owners
+
+| Concern | Owner | Lifetime |
+| --- | --- | --- |
+| Identity, room membership and access | Existing room store and authenticated signaling session | Current admitted participant |
+| Screen/camera plus Host commentary | Existing capture and media owners | Current share and source |
+| Text and reactions | `signal/interactions.go`, shared protocol and `RoomInteractionSession` | Current authenticated room session; bounded page-local presentation |
+| Chat panel, optional chat overlay and reaction placement | Shared `RoomInteractions` and `RoomChatOverlay`; `RoomInteractionSession` owns bounded events | Current room UI |
+
+The existing room authority admits interactions through the same authenticated
+WebSocket; there is no second admission or identity system.
+
+Messages do not need a media route, decoded frame, or SFU. They do not mutate the
+media graph, add a port, create an extra connection, or restart capture. Pause,
+source replacement and media recovery keep the current conversation. In this
+candidate, a capable Host explicitly opts into a persistent room session.
+Restoring an existing room connects that session without requesting capture or
+starting media. The capture-ready owner requests publication; merely entering a
+capture picker or awaiting permission cannot publish. Stopping the publication
+retires only its media generation, while the authenticated Host and its
+interaction session remain available. A later `start-sharing` claims a
+fresh generation on that same signaling connection. Room close, access loss,
+session replacement and page teardown still retire the whole session. A Host
+without the capability keeps the existing stop-and-leave behavior.
+
+Room restoration and authority recovery share one awaited owner; connection reuse
+checks the actual room/token binding. Publication acknowledgment precedes media
+effects. A rejected start names its attempted generation, retires only that local
+attempt and keeps room interactions available; it cannot stop another generation.
+Opening the Host page alone still does not create an unused room.
+
+The server treats the current Host session's non-empty generation, matched to the
+room's reconnect fence, as the only publishing truth. Room membership and Host
+presence are separate, so a late Viewer in an idle room receives no media route
+even when the Host remains available for room interactions.
+Connected Viewers can continue interacting when the Host disconnects, while the
+room server and their admitted sessions remain available. A departed participant
+cannot receive a targeted reaction. Closing an App that also runs the room server
+ends that service; merely leaving the Host page does not.
+
+## Delivery, Compatibility And Bounds
+
+- `/api/capabilities` advertises optional `roomInteractions` and
+ `hostRoomSession`; absence means the corresponding behavior is unavailable.
+ The page explicitly subscribes after authentication. Only subscribed current
+ sessions receive interaction events. The Host room-session behavior also
+ requires the `roomSession: true` authentication opt-in; older clients retain
+ the original stop-and-leave contract. Existing v23 media messages remain
+ valid, and publication start/result messages are sent only to an opted-in
+ session.
+- The viewer discovers this optional capability independently of first-picture
+ startup. Late capability discovery enables data on the existing connection.
+- A strict payload union admits plain chat text or a registered reaction. Chat is
+ at most 280 Unicode code points, normalized to NFC, without control/bidi
+ characters. It renders as text, with no markup, links, attachments or commands.
+- The server accepts at most one interaction per 800 ms per connection. It
+ validates identity, subscription and target before fan-out. A slow recipient's
+ data is skipped at 16 KiB queued output, reserving the existing bounded queue
+ for signaling. The browser also declines sends under output backpressure.
+- The sender waits for its server echo, not a guessed success or a read receipt.
+ A rejection or a 5-second confirmation timeout preserves the draft. There is
+ no offline outbox or automatic retry; manually retrying an unconfirmed send
+ may duplicate a message that arrived before the confirmation was lost. Only
+ the matching pending request may confirm a send or clear its draft/error;
+ late echoes may enter history without confirming a newer attempt.
+- The page retains the latest 80 received chat messages and at most eight
+ simultaneous reaction effects. Effects expire after 2.4 seconds. Reconnection
+ clears pending effects and re-subscribes, without replaying messages or actions.
+ Same-room re-admission retains history, self attribution and the local draft,
+ even when its wire peer ID changes. Refresh, leaving, authority loss or room
+ session replacement clears the page conversation; late joiners receive only
+ new messages. Nothing persists in Browser storage.
+- There is no server history, database table or message body in diagnostic
+ exports. Transport is the site's HTTPS/WSS connection; **chat is not end-to-end
+ encrypted**, and the server operator can access live message contents.
+
+## Interaction Design
+
+Chat uses the shared `FloatingPanel` shell for its nonmodal window. The shell
+owns native visibility, placement, movement, resizing and focus return; the
+conversation retains its session, draft, reading position and unread state.
+It opens initially at the lower right, without
+resizing the picture or moving the couch. Drag its title bar, use arrow keys on
+the focused bar, or explicitly choose a corner under Position. Its summary also
+allows dragging; a drag never activates the menu. Resize from either diagonal
+corner, use arrow keys on a resize handle, or choose a preset under Position.
+Resizing holds the opposite corner steady. Size and position belong to this room
+UI only; viewport and keyboard changes bound them without erasing preferences.
+The native manual
+popover keeps the conversation visible while operating the picture or reactions.
+Close, the entry toggle, or Escape hides it; this preserves the room conversation. Inner
+menus consume Escape first, then the most recently opened utility window, then
+theater mode. Input-method
+composition is not a dismissal request. Opening focuses the panel without opening
+the keyboard; explicit close returns focus to the entry. Nested disclosures
+cannot change panel visibility.
+The visual viewport bounds it above an onscreen keyboard.
+Hidden messages are not marked read. A bounded scrolling log keeps arrivals from
+growing the panel. Consecutive messages from the same sender share visual identity;
+each message retains its accessible author. Hiding and reopening preserves the
+reader's position; a latest-message action resumes following. Sending waits for
+confirmation without locking draft editing or introducing a message queue.
+The conversation entry sits below the couch, separate from
+the Host's sharing controls; theater retains its compact conversation entry.
+Selecting a person opens the compact reaction palette in the shared floating
+shell, initially beside the selected person (below when space permits). Each
+opening anchors there before allowing free movement. Its portrait and name identify the
+target even after moving the window. Selecting a different participant keeps its
+placement and any manually chosen size; selecting the same person toggles it.
+The default palette height follows its content, bounded by the visible viewport.
+One's own palette expresses a reaction above oneself and holds the local effects switch; other
+people's palettes offer targeted expressions and two labelled throwing props.
+Connection details remain secondary when that participant has inspectable data.
+Close or Escape dismisses the palette; it also closes if its participant leaves.
+Sending keeps it open for repeated interaction, with the existing short cooldown.
+Confirmation never changes menu visibility; failure stays visible in the card.
+Opening or closing a reaction card preserves the chat draft and unread state.
+Self expressions follow the sender's head, including idle and interaction motion.
+Only that sender's newest self expression replaces their earlier one. A targeted
+emoji is a delivery: it travels from sender to recipient, retains a compact
+source label above the centered glyph, and settles separately above the
+recipient's own expression. Late mounts and reduced motion retain that attribution without
+replaying elapsed motion. A gift never replaces the recipient's self expression;
+all delivered events remain in the accessible log.
+Tomato/poop throws require choosing a current participant, land with a small
+character response and settle above the head, keeping the face visible. A tomato
+squashes with a few droplets; poop wobbles with a small puff. Other expressions
+use short gestures and sparse accents. All decorative descendants share the
+event's remaining phase and expiry; event ID and sender identity seed their
+variation. No particles, stains or animation timers outlive the event.
+A departed participant is never silently replaced as target.
+Everyone sees the same server-issued event; placement follows their own couch
+layout. Effects have a couch-only coordinate owner, independent of panel layout,
+and never become a video overlay or a media status. Impact transforms leave the
+UUID-owned idle animation, names and status indicators alone. Hiding effects or
+retiring the event cancels its own animation; showing a still-live event uses its
+remaining lifetime. Users can hide effects; reduced motion uses a static stamp.
+Theater mode hides the couch and retains a chat entry, using the same floating
+panel and draft without changing the picture's size. HTML player fullscreen
+keeps the existing danmaku control; it does not include the conversation panel.
+Unread messages advance only while the log is onscreen, the document is visible
+and the reader is following its latest messages. A hidden or scrolled-back log
+does not consume arrivals. These are local unread indicators, not read receipts.
+
+The optional chat overlay (danmaku) projects newly received chat over the picture;
+it adds no message kind, transport or delivery promise. It defaults off, with the
+same local switch beside the chat composer and Viewer playback bar, including HTML
+fullscreen. At most three single-line comments appear for up to six seconds each.
+Font size follows the picture width, with a fixed readable contrast and duration
+rather than separate typography, opacity and speed preferences.
+Each event's sender and ID choose its lane, and a newer event replaces that lane's
+previous flight. All received text remains in the full chat log, without a playback queue. Text is clipped
+to the picture space above playback controls; smaller spaces show fewer complete
+lanes without restarting their timelines. It does not intercept playback gestures. Showing the overlay
+does not mark the chat log as read. Hidden documents, unavailable pictures,
+source changes and disconnection clear flights; showing or reconnecting never
+replays earlier comments. Reduced motion uses temporary stationary text. Native
+video-only fullscreen and picture in picture cannot include HTML overlays.
+
+Server confirmation means acceptance, not an acknowledgment from every recipient.
+Delivered chat follows server order; congestion can leave gaps without reordering
+the received subset. The same event retains its lane and time basis even when
+preceding events are missed. Visibility, local switches and missing messages may
+change which effects are shown; geometry and font size follow the local viewport.
+
+Reaction animations are decorative. The same bounded events also provide
+localized sender/action/target descriptions in a polite accessible log, including
+when visual effects are hidden. Reduced motion removes decorative movement from
+both the flight and the picker controls.
+
+Chinese, English and visual modes share the component and behavior. Visual mode
+retains user-entered text and participant names, with icon controls and localized
+accessible names. Host identity reuses the existing crown; neither chat colour nor
+reaction motion claims media connectivity, capture permission or delivery status.
+
+## Presentation Synchronization
+
+The server stamps acceptance time once and broadcasts the same event in its
+existing serialized order. A subscription response provides server time; the
+room session estimates its offset from the local round-trip midpoint and keeps
+that estimate until reconnection. Effect expiry and both renderers share that
+session's clock. Late effects start at their remaining phase; expired chat still
+enters the log and confirms a matching send, without replaying an effect.
+Returning from reduced motion also respects the original expiry.
+
+This is approximate alignment, limited by subscription round-trip asymmetry,
+queueing and clock drift; it does not promise frame-exact synchronization. There
+is no continuous clock exchange, sorting by timestamp, or animation-frame stream.
+Changing a device's clock mid-connection can disturb effect timing until the
+next subscription; message order and delivery do not depend on that clock.
+Participant motion and event lanes reuse the same identity hash, with an event's
+ID and actor identity determining its lane independently of prior arrivals.
+A shared mutable random-number stream would drift whenever a client misses or
+hides an event. Responsive layout remains local.
+This follows the same-input principle described in
+[Deterministic Lockstep](https://gafferongames.com/post/deterministic_lockstep/),
+without introducing lockstep waiting or frame-by-frame simulation.
+
+## History Direction (Not Yet Implemented)
+
+History ownership remains undecided. A Host cache can also miss live messages and
+is unavailable while the Host is absent; treating its copy as server-confirmed
+history would change message authority. Compare that cost with a bounded room-server
+memory buffer before adding replay. Either design needs explicit access and
+retention boundaries. Recovered text would enter the log, never replay old
+danmaku or reaction effects. Current behavior remains the no-history contract above.
+
+## Audio Boundary
+
+The Host microphone remains commentary mixed into the shared source.
+Bidirectional room voice is not part of the product plan.
+Text, reactions and danmaku introduce no microphone capture, voice
+admission or additional media peers.
+
+## Primary References And Rationale
+
+- [LiveKit media and data](https://docs.livekit.io/frontends/build/media-data/)
+ separates publications from finite data. Piik already has an authenticated
+ signaling channel; adding the LiveKit room service solely for chat is unnecessary.
+- [LiveKit text streams](https://docs.livekit.io/transport/data/text-streams/)
+ distinguishes current-participant delivery from persistent history. The
+ candidate likewise promises transient room messages, with explicit local bounds.
+- [WebSocket buffering](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/bufferedAmount)
+ and [WebSocket API limitations](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API)
+ motivate bounded queues and explicit send failure rather than an accumulating outbox.
+- [WebRTC data channels](https://www.w3.org/TR/webrtc/#rtcdatachannel)
+ depend on peer connections. Using them here would couple chat readiness and
+ fan-out to a screen-sharing route; the existing room WebSocket avoids that.
+- [React state identity](https://react.dev/learn/preserving-and-resetting-state)
+ supports resetting the panel with its room session rather than a reconnecting
+ transport identity. [ARIA log semantics](https://www.w3.org/TR/wai-aria-1.2/#log)
+ provide sequential, non-interrupting announcements alongside visual effects.
+- [Meet reactions](https://support.google.com/meet/answer/13151720?hl=en)
+ attach expressions to participants and let viewers reduce visual distraction.
+ Piik likewise keeps reactions outside the shared picture, with explicit local
+ display preference and a separate accessible event description.
+- [Microsoft participant cards](https://support.microsoft.com/en-us/outlook/profile-cards-in-microsoft-365)
+ place person-specific actions on the selected person. Piik uses that direct
+ entry on its existing couch rather than requiring a second participant list.
+- [Native popovers](https://developer.mozilla.org/en-US/docs/Web/API/Popover_API/Using)
+ provide top-layer placement, light dismissal and focus return without adding
+ a second overlay framework. Piik owns the bounded card's anchor and contents.
+- [Animation time](https://developer.mozilla.org/en-US/docs/Web/API/Animation/currentTime)
+ lets a newly mounted or resumed effect seek to its remaining phase. The
+ [clock comparison](https://developer.mozilla.org/en-US/docs/Web/API/Performance/now#ticking_during_sleep)
+ explains why replacing wall time with a monotonic clock alone would not solve
+ cross-platform sleep and resume.
+- [niconico playback controls](https://help.nicochannel.jp/hc/ja/articles/4414625162777-%E5%8B%95%E7%94%BB%E3%81%AE%E8%A6%96%E8%81%B4%E6%96%B9%E6%B3%95)
+ keep comment display optional alongside playback. Piik likewise separates
+ each viewer's overlay preference from message delivery and the conversation.
diff --git a/docs/research/room-voice.md b/docs/research/room-voice.md
deleted file mode 100644
index bfb5c4b..0000000
--- a/docs/research/room-voice.md
+++ /dev/null
@@ -1,287 +0,0 @@
-# Room Voice: Design Assessment
-
-Reviewed 2026-09-24 against public `v1.6.3` (`f423df0a`). This is a proposal,
-not an accepted product contract or implementation authorization. The separate
-room-interaction experiment supplies candidate lifecycle behavior; it has not
-been integrated. [TODO](../todo.md#now) owns the outstanding work and decisions.
-
-## Recommendation
-
-Keep existing room authority and the Host-rooted screen route. Model room voice
-as participant-owned microphone publications and independent listening. Chat,
-screen playback and voice share admission, not a media lifetime. Do not turn the
-screen tree into a conference graph or introduce another room/session store.
-
-There is no equally small solution that simultaneously promises all 21 members
-can talk, preserves every P2P-only deployment, and supplies conference-grade
-connectivity. Choose the first voice scope before implementing its transport.
-For the next technical experiment, assess a small direct group and audio
-recapture isolation first. Four total voice participants is a useful test size,
-**not an accepted product limit or a demonstrated performance guarantee**.
-If full-room voice is required initially, assess a mature conference SFU before
-building a mesh intended to grow into one.
-
-## Current Owners And Actual Gaps
-
-| Current owner | Verified behavior | Consequence for voice |
-| --- | --- | --- |
-| Room store and authenticated signaling | Room admission, participant identity, grant revocation and session replacement | Extend this authority; no voice login or second membership store |
-| `HostAudio` in `src/client/media/host-audio.ts` | Source and microphone mixed into one share-scoped track | This is commentary; it cannot supply independent conversation |
-| `internal/app/nativehost/audio_mix.go` | Native source/mic PCM share one mixer and output clock | Keep source audio here; avoid duplicating the Host mic across outputs |
-| `src/client/media/viewer-audio.ts` | Audio follows the picture's play/pause/mute state | Room speech needs its own playback owner |
-| Screen controller and media edges | One Host source, one upstream per Viewer, bounded screen-copy capacity | Voice must not borrow screen generations, first-video-frame proof or copy slots |
-| Embedded SFU admission/media | One committed room publication, video-required metadata, share-scoped resource identity | Multiple audio-only publishers need a new capability, not another flag |
-
-The SFU constraints are explicit in `validPublicationMedia` in
-`internal/server/sfu/media.go`, `ResourceFence` and `CommitPublication` in
-`internal/server/sfu/admission.go`, and the Host/Viewer dispatch in
-`internal/server/signal/router_sfu_signal.go`.
-[ADR-0013](../adr/0013-embedded-node-local-media.md) embeds Pion/LiveKit components;
-it does not embed LiveKit's complete room service.
-
-The interaction candidate separates a retained Host room session from screen
-publication. That is a useful foundation, not permission to allocate rooms on
-page visit. Preserve the accepted entry: first share creates the room and an
-existing room can resume. Retaining opted-in interaction after stopping media is
-the candidate lifecycle, not published behavior. A separate voice-only
-room-creation flow would be another product decision.
-
-## Transport Choice
-
-| Approach | Benefit | Cost and boundary |
-| --- | --- | --- |
-| Small direct voice mesh | Keeps media P2P and does not require a reachable central media server | Each sender sends to every listener; needs a total voice-member bound, separate resource accounting and real NAT/device tests |
-| Mature multi-publisher SFU | Each speaker uploads once; suitable direction for full-room voice | New publication/subscription integration and central-media availability/privacy decision |
-| Host forwarding or mixing hub | Fewer peer relationships | Reintroduces Host dependency, forwarding/mixing and echo ownership; unsuitable for Host-independent conversation |
-| Custom voice relay tree or automatic mesh/SFU switching | Potentially combines deployment options | Adds another routing/recovery problem; outside the first scope |
-
-For a full bidirectional mesh, `N` participants need `N*(N-1)/2` pair connections
-and up to `N-1` microphone copies per sender: 21 people means 210 pairs and 20
-copies; four means six pairs and three copies. These are topology counts, not
-bandwidth or CPU measurements. Limiting speakers alone does not bound sender
-upload: one speaker and 20 listeners still need 20 copies. The screen's existing
-`C=1..3` limit must retain its meaning; any independent voice budget needs explicit
-acceptance and combined-load measurement.
-
-Two SFU implementations deserve a bounded comparison: a complete mature room
-engine, or a narrow participant-audio adapter around existing media components.
-The former adds deployment and potentially competing session ownership; the
-latter leaves Piik responsible for admission, subscriptions, recovery and cleanup.
-Do not describe the adapter as mature conference behavior merely because its
-forwarder uses mature libraries. LiveKit's
-[room/participant/track model](https://docs.livekit.io/intro/basics/rooms-participants-tracks/)
-and [SFU architecture](https://docs.livekit.io/reference/internals/livekit-sfu/)
-show the layers involved.
-
-Select one topology for the first capability. Jitsi's
-[two-party P2P design](https://jitsi.org/blog/p2p4121/) retains its bridge connection
-and changes transport when group size changes: useful evidence that a hybrid
-mode has real lifecycle cost. It is not a reason to copy that complexity here.
-
-Deployment is part of this decision. Demo and App Local/public-link modes are
-P2P-only today. Public-link tunneling carries HTTP/WebSocket control traffic,
-not a public SFU media endpoint. Existing STUN is not TURN, and existing embedded
-UDP media does not provide media TCP/TLS. A full conference engine's
-[connectivity options](https://docs.livekit.io/transport/self-hosting/ports-firewall/)
-are not inherited by importing its library. This proposal enables no new service
-or port and does not change Demo configuration.
-
-## Proposed Responsibility Boundaries
-
-```text
-Existing room authority and authenticated signaling
- +-- screen publication -> existing P2P/SFU screen route -> picture/source sound
- +-- interaction events -> bounded chat/reactions
- +-- voice admission -> participant microphone -> selected voice transport
- -> independent voice playback
-```
-
-These are responsibility boundaries, not a request for a generic media framework
-or one new service per line. Keep the first implementation cohesive:
-
-| Responsibility | Owns | Does not own |
-| --- | --- | --- |
-| Room authority | Who belongs, who may publish/listen, bounds and retirement | OS microphone permission or local volume |
-| Participant voice session | Explicit join intent, current admitted publication/subscriptions and their transport lifetime | Screen routes, share generation or a second reconnect policy |
-| Microphone input | One acquisition, device choice, local mute, replacement and disposal | Remote permission to turn capture on |
-| Voice playback | Received tracks, local volume/person mute, output capability and autoplay recovery | The video's pause/mute state or remote speaking permission |
-
-Do not attach room speech to a screen peer merely to reduce connection count:
-screen parent replacement and share retirement have different lifetimes. The
-selected voice transport owns its connections and counts them explicitly.
-
-Prefer a Browser-owned room microphone for both roles, including a Host using
-App screen capture. This reuses platform speech processing and avoids a second
-native voice service. Reuse existing device-selection semantics, not native
-device identifiers. Browser microphone acquisition requires a secure context;
-plain HTTP LAN pages and unsupported browsers must have truthful capability
-presentation. App detection alone does not establish microphone availability.
-See the [capture specification](https://www.w3.org/TR/mediacapture-streams/).
-Browser voice also does not inherit the native Pion path's port mapping, NAT
-assistance or native media fallback just because an App is attached. Measure
-voice reachability independently; a working screen path does not prove it.
-
-Room voice and source sound need different audio treatment. Voice uses the
-platform's speech/AEC facilities; music/game sound keeps its fidelity settings.
-Do not force both through the existing source-audio Opus profile or write a new
-voice codec, AEC, PCM subtraction routine or virtual-audio driver.
-
-## Consent, Lifecycle And Presentation
-
-Proposed first permission model: admitted members may explicitly join voice and
-then unmute themselves; existing room removal remains the authority boundary.
-Start with a closed microphone. Joining may unlock playback, but only explicit
-unmute requests input permission. If Host-only speaking approval is required,
-specify it before adding a moderation UI. No remote action may start a microphone;
-LiveKit also disables [remote unmute by default](https://docs.livekit.io/intro/basics/rooms-participants-tracks/participants/).
-
-| Event | Proposed behavior |
-| --- | --- |
-| Share start/stop/pause, source or quality change | Voice stays independent |
-| Local mic mute | Silence that publication without leaving voice; retain explicit device intent |
-| Local person mute or voice volume | Change local listening only; picture volume remains separate |
-| Leave voice | Release mic and voice media, remain in room/chat |
-| Host leaves page while room service remains | Retire Host voice; other admitted participants may continue |
-| App exits in Local/public-link mode | Room service disappears; independent voice cannot outlive its authority |
-| Site-mode App media fails | Browser voice may continue while the Site room authority remains |
-| Revocation, replacement or room close | Retire affected resources and reject stale completions/signaling |
-| Mic permission/device failure | Preserve picture and other voices; show the actual input failure |
-| Hidden page or phone interruption | Visibility alone is not leave/mute; report actual suspension/device loss |
-
-Transient signaling loss is not automatically a media failure or a grant
-revocation. Current `presence.go` and `route.Controller.DisconnectSession`
-distinguish disconnected sessions from confirmed departure; Viewers have bounded
-membership grace. Voice must consume that authority owner's transitions, with no
-parallel grace timer or indefinite stale-session publication. Before implementation,
-define and test the Host case, reservation retention, reconnect re-admission and
-revocation during an outage. Do not extrapolate Viewer grace to a Host or reuse
-an old session ID as current authority. This is a design gate, not a claim that
-screen recovery already implements voice recovery.
-
-One input owner prevents duplicate captures. Retire late permission results after
-leave; prepare a replacement device before releasing a healthy old input; keep a
-failed explicit selection visible rather than silently choosing another mic.
-Repeated mute/join/leave must converge without a second saved effective-state flag.
-Ordinary publication mute need not renegotiate every time; mature systems
-[distinguish mute, unpublish and unsubscribe](https://docs.livekit.io/intro/basics/rooms-participants-tracks/tracks/).
-
-Use the existing participants/sofa and room actions, not a conference grid.
-First controls: join/leave voice, mic toggle and input choice/check, independent
-voice volume, and local person mute in the participant menu. Keep source-sound
-settings with the picture. Default to system output; show output selection only
-where supported. Do not add deafen, global push-to-talk or a permission-role matrix
-until their need and restoration behavior are established.
-
-Display local intent, authorized membership, transport readiness and actual
-playback separately. A speaking pulse follows observed audio activity, not a
-connected flag or UUID animation seed. Catch blocked playback, including tracks
-arriving after joining, and offer an explicit enable-audio action; compare
-[LiveKit's autoplay handling](https://docs.livekit.io/reference/components/react/component/startaudio/).
-Mobile lock-screen calling remains unproven; platform suspension is not fixed by
-extra timers. See [browser lifecycle guidance](https://developer.chrome.com/docs/web-platform/page-lifecycle-api).
-
-## Audio Isolation Is A Prerequisite
-
-Whole-system loopback records received room speech played on the captured output
-endpoint. That speech can then be sent back through screen audio. Headphones
-remove an acoustic path, not this digital recapture; microphone AEC does not clean
-the separate source-audio track. This matters even if only the Host shares a screen.
-
-Windows exposes narrower process-tree capture and process exclusion. Current
-Piik includes the selected process tree for window sound or captures the output
-endpoint for system sound; it does not implement process exclusion. Excluding
-the browser that plays Piik voice could also exclude intended browser content.
-Test actual process ownership and output routing rather than assuming a window
-selection isolates sound. See Microsoft's
-[loopback description](https://learn.microsoft.com/en-us/windows/win32/coreaudio/loopback-recording)
-and [process-capture sample](https://learn.microsoft.com/en-us/samples/microsoft/windows-classic-samples/applicationloopbackaudio-sample/).
-
-Browser `restrictOwnAudio` is relevant; `suppressLocalAudioPlayback` has a different
-purpose. Chrome documents the former as excluding audio originating from the
-capturing document when system audio is captured. Check support, applied settings
-and decoded output on supported platforms; a hint is not cross-browser proof.
-See the [Screen Capture specification](https://www.w3.org/TR/screen-capture/)
-and [Chrome's implementation announcement](https://developer.chrome.com/blog/chrome-141-beta).
-
-Verify source sound and incoming voices using distinguishable signals. Cover
-native window/system capture, browser tab/window/system capture, source changes,
-same-browser content and output-device changes. Where isolation cannot be
-established, a supported combination or explicit feature limitation must be
-chosen before release. Never silently discard source sound or promise that a
-headphone hint solves it.
-
-## Compatibility, Trust And Bounds
-
-The Host microphone must have one distribution path. Sending it in both existing
-commentary and independent voice causes duplicates and defeats individual mute.
-Moving it exclusively to voice can make older/non-participating Viewers lose
-commentary. Choose an explicit room capability/participation boundary or a
-coordinated major-version transition; do not ship a silent behavior change.
-The [versioning owner](../standards/versioning.md) governs that choice. This
-assessment does not select a release number or authorize removal of native mic.
-Once that boundary is selected, define the handoff before coding: retire the old
-microphone distribution before enabling the new one, and test failure/cancellation
-without silently losing commentary for Viewers who cannot receive room voice.
-
-Browser-first capture can avoid new native voice commands, but does not make
-room signaling automatically compatible. Trace strict readers in both directions;
-advertise support before new messages. Keep stable peer identity, authenticated
-session, grant and media connection identity distinct. Reuse an existing identity
-only when its lifetime matches; do not add a room-wide epoch for convenience.
-
-Bind signaling to current authenticated participants and exact publication/peer
-resources; bound memberships, publications, subscriptions, candidates and message
-rates before allocation. Leave/revoke must release actual resources before their
-slots become reusable. No audio through chat events or durable voice recordings.
-
-P2P revocation has an enforcement limit: the server can refuse signaling and
-conforming endpoints can stop capture/drop tracks, but it cannot intercept media
-between two modified peers. An SFU can enforce its own forwarding removal. Do not
-promise identical server-enforced muting across these models, or hide the central
-server's media access under the existing P2P privacy label.
-
-## Implementation Sequence And Acceptance
-
-1. Reconcile the isolated interaction candidate with current main and accept its
- room/picture separation on its own. Preserve the existing staged work; do not
- merge an old branch wholesale. Voice need not block chat/UI acceptance.
-2. Select total voice membership, deployment coverage, speaking policy and the
- commentary/compatibility boundary. Record accepted changes in the existing
- product owners and an ADR only when architectural decisions are accepted.
-3. Run an isolated transport/audio-isolation experiment before shipping UI.
- Measure screen plus voice CPU, upload, number of connections, audible latency,
- NAT failure, echo and cleanup. Use one topology; stop if the required scope
- implies a custom conference engine or unsupported audio isolation.
-4. Implement one vertical slice through the owners above: explicit join, muted
- input, unmute, remote audio, local mute and leave. Then expand participants and
- exact authority teardown. Reuse existing controls, status/comic meanings and
- diagnostic export; avoid separate feature-local rules.
-5. Accept faults as well as the happy path: permission resolves after leave,
- device switch fails, source changes, autoplay is blocked, signaling drops
- during healthy audio, reconnect races revoke/replacement, Host leaves, service
- exits, mobile suspends, and repeated join/leave leaks no resources. Test quiet
- and muted participants without video-frame admission requirements. Include
- supported old/new endpoints and Chinese/English/visual presentation.
-
-Transport experiments must report measured limits rather than treating a
-successful localhost call as public-network or mobile acceptance. This planning
-round ran no voice, acoustic-echo or device-performance experiment.
-
-## Independent Fidelity Work
-
-Stereo acceptance can proceed separately: distinguishable left/right signals
-through Browser, Native, relay and SFU, with microphone mixing and source changes.
-The released native answer fix negotiates receive-side stereo; SDP alone does not
-prove decoded channel separation. Opus negotiation follows
-[RFC 7587 section 7](https://www.rfc-editor.org/rfc/rfc7587.html#section-7).
-Do not add surround channels as a prerequisite for voice.
-
-For [HDR issue #420](https://github.com/TNTcraftHIM/Piik/issues/420), investigate
-correct SDR output first: preserve HDR range during native capture, then use
-platform tone/gamut mapping before the existing SDR encoder. Windows documents
-[floating-point capture](https://learn.microsoft.com/en-us/windows/apps/develop/media-authoring-processing/screen-capture)
-and a [tone-map effect](https://learn.microsoft.com/en-us/windows/win32/direct2d/hdr-tone-map-effect).
-Validate mixed SDR/HDR content, highlights, SDR reference output and display-mode
-changes. Already-clipped browser input cannot be restored downstream. Full HDR
-transport/metadata, surround sound, recording and transcription remain outside
-this proposal.
diff --git a/docs/standards/configuration.md b/docs/standards/configuration.md
index fac3d95..4288653 100644
--- a/docs/standards/configuration.md
+++ b/docs/standards/configuration.md
@@ -56,6 +56,7 @@ Automatic SFU fallback runs inside the Hosted process when `SFU_UDP_PORT` is set
| `SFU_UDP_PORT` | Optional UDP media port `1..65535`; unset or blank disables SFU and fixes Privacy mode on for every room. Set `7882` for the standard public listener and allow the Host to choose Privacy mode. |
| `SFU_LISTEN_HOST` | IPv4 bind address, default `0.0.0.0`; independent of HTTP `LISTEN_HOST`. Read only when SFU is enabled. |
| `SFU_PUBLIC_IP` | Optional explicit IPv4 advertised-address override for a host behind NAT. Read only when SFU is enabled. |
+| `SFU_ONLY` | `false` by default. `true` requires `SFU_UDP_PORT` and sends every room through SFU, without Peer candidates or background P2P convergence. Applies after Server restart. |
SFU control uses the application's authenticated signaling connection. No
separate control origin or infrastructure credentials are configured. Local and
@@ -67,6 +68,13 @@ configured STUN and peer relays remain; no media-server fallback is possible.
The server enforces this even if a Host requests hybrid mode. There is no
separate `SFU_ENABLED` flag to conflict with the listener configuration.
+With `SFU_ONLY=true`, the page shows a locked **Server media** policy in place of
+P2P settings. An older page requesting Privacy mode is refused before sharing;
+refresh it or turn Privacy mode off. Capacity exhaustion and connection failure
+use the existing bounded SFU admission/recovery path, never a P2P escape.
+This policy needs sufficient server bandwidth and a working UDP route; it is not
+a TURN/TCP fallback. App Local/public-link and the public Demo remain P2P-only.
+
The SQLite parent directory must exist and be writable. The systemd template
sets `/var/lib/piik/rooms.sqlite` under its managed state directory; the
container image sets `/home/nonroot/rooms.sqlite` under its writable data
diff --git a/docs/standards/engineering.md b/docs/standards/engineering.md
index 109232d..dcbf258 100644
--- a/docs/standards/engineering.md
+++ b/docs/standards/engineering.md
@@ -148,7 +148,10 @@ above rather than searching the tree.
Closing a Browser connection may leave its API promises pending. Its owner
cancels the corresponding waits so a serialized successor never depends on
completion from a retired connection; late results still need identity checks.
- Later cancellation must not erase a failure already observed before cleanup.
+ Failure effects need the same final identity check as successful results,
+ serialized with replacement; an earlier input check alone is insufficient.
+ Later cancellation must not erase a failure already accepted by its current
+ owner. Perform blocking retirement after releasing that owner's state lock.
A caller deadline does not cancel a dependency that ignores it. Keep late
work bounded and serialize its cleanup; verify the dependency's actual lifetime.
Construction is part of acquisition: if it throws, release resources created
diff --git a/docs/standards/media-quality.md b/docs/standards/media-quality.md
index 99db584..cace60c 100644
--- a/docs/standards/media-quality.md
+++ b/docs/standards/media-quality.md
@@ -39,6 +39,17 @@ production behavior and remaining acceptance.
frame matches its desktop source dimensions. Other frames retain their own
aspect ratio. This changes only the encoded presentation, never the game or
display settings; vendor-private scaling is not inferred.
+- Windows SDR conversion owns one output color contract: limited-range BT.601
+ NV12 with matching H.264 metadata. RGB capture and decoded video retain their
+ input range/matrix through conversion and relay scaling; dimensions do not
+ determine color space. Unspecified SDR follows the WebRTC convention.
+- Windows native HDR sources retain scRGB FP16 until platform tone mapping,
+ source-display white-level adjustment and sRGB conversion produce SDR. Live
+ capture and source thumbnails share this owner; conversion precedes output
+ fanout and does not add a media route or per-encoder tone curve. Ordinary SDR
+ keeps its copy path. Source-display color changes refresh capture input,
+ not room authority or transport. [Fidelity evidence](../research/media-fidelity.md)
+ owns platform checks; this is HDR-to-SDR, not end-to-end HDR delivery.
- Where Windows supports border control, the native source picker offers
**Show capture border**, off by default. The choice stays in the Host page and
follows native source and quality changes. Source previews request borderless
@@ -177,12 +188,32 @@ closing a Browser PeerConnection alone does not guarantee that its promises
settle. Cancellation starts before joining work that depends on it, while a
replacement still waits for the retired owner's cleanup.
+Browser and Native encoder-group admission uses current native demand, not the
+previously applied rate. A new or pending join must not displace an existing
+rate owner by raising or retaining its budget against that owner's demand.
+Revalidate pending admission before planning other memberships.
+
Eligible Browser P2P parents use the source-owned pool in
[ADR-0014](../adr/0014-browser-node-local-encoding-pool.md). Compatible direct
children share one independent local WebRTC producer; incompatible demands
-remain separate. Each outgoing connection keeps native transport, allocation
-and recovery, with a tiny carrier supplying its RTP clock. Producers use
-the existing native video target under Host ceilings. Actual forwarded-frame
+remain separate. Joining another producer must fit the child's native allocation;
+a quiet scene's low byte rate does not prove that producer's rate budget fits.
+Pending separation remains necessary while another compatible member needs a
+higher native budget; quiet output alone cannot cancel it. When that demand
+leaves or the child's budget recovers, the current producer can adapt in place.
+Optional reuse also preserves the healthy current output's rate budget and
+observed quality within Host ceilings through recovery-frame commitment, using
+the actual candidate frame and current output rather than preparation alone.
+Rejected membership retains current delivery and its recovery requests.
+A recovery frame already being written retains its owner until completion,
+including across pause/resume; cancellation belongs to the output that knows
+whether writing has started.
+Each outgoing connection keeps native transport, allocation and recovery, with
+a tiny carrier supplying its RTP clock. Producers use the existing native video
+target under Host ceilings. A new producer seeds its
+local WebRTC start rate from that allocation, without a minimum or a change to
+the outgoing connection's estimate. Subsequent budgets use the existing sender
+parameters; initialization does not add a recovery controller. Actual forwarded-frame
and producer observations remain distinct from the tiny carrier's statistics.
Producer startup protection begins at actual outgoing publication, excluding
local warmup and paused frames. The synthetic carrier uses screen-content
@@ -294,6 +325,17 @@ A Native audio-process failure does not end healthy video. A later explicit
source replacement resumes audio through the existing track and encoder owner;
it does not trigger an automatic capture retry or change room audio topology.
+Optional Windows screen-audio exclusion binds one selected process tree to its
+PID and creation time under [ADR-0008](../adr/0008-window-scoped-audio-capture.md).
+The selection lasts for the share, survives source-audio off/on and refresh,
+and is never saved across App runs or rebound by executable name. Changing to a
+window source or explicitly selecting no exclusion clears it. A requested
+exclusion retires previous source audio before replacement preparation; failure
+or target exit leaves source audio silent while healthy video and the Host
+microphone continue. Only an explicit source selection resumes it. A missing
+capability must reject the request, never discard the exclusion. Old pages keep
+their existing audio behavior.
+
## Observable Truth
Requested resolution, frame rate, bitrate, content intent, codec, and audio
diff --git a/docs/standards/naming.md b/docs/standards/naming.md
index 8e31015..0a7f6d3 100644
--- a/docs/standards/naming.md
+++ b/docs/standards/naming.md
@@ -40,6 +40,7 @@ Set the register by the reader's task:
| Surface | Register |
| --- | --- |
| Website | Light, everyday language in introductions and illustration captions; keep the current restraint |
+| App and Web controls | Direct feature and action names, including panel titles and settings; “聊天” / “Chat” rather than an invitation or joke |
| README | A friendly introduction, followed by direct explanations of features, limits and first steps |
| Documentation index | Clear topic names and short descriptions that help readers choose a guide |
| Deployment, configuration and maintenance guides | Formal, precise instructions: prerequisites, actions, expected results and recovery steps |
@@ -102,6 +103,8 @@ in the welcome or illustration caption. Do not soften a failure into a joke,
invent connection progress or promise guaranteed connectivity or absolute privacy.
Screen sharing includes games, creative work and showing a useful discovery.
+Playfulness belongs in optional welcome/waiting lines, empty-state captions and
+illustration reactions; it does not rename the control that performs the action.
Operational pages omit introductory sentences that repeat their controls.
Helper text should explain a meaningful choice or prevent an error.
diff --git a/docs/standards/presentation-lifecycle.md b/docs/standards/presentation-lifecycle.md
index 4fe86bc..5191363 100644
--- a/docs/standards/presentation-lifecycle.md
+++ b/docs/standards/presentation-lifecycle.md
@@ -24,6 +24,15 @@ room identity, current state, roster, and one manual media-reconnect action.
Controls and text must remain usable without overlap or horizontal overflow on
desktop and mobile viewports.
+A capable, opted-in authenticated room session
+outlives its current publication. Stopping capture retires media, not chat;
+restoring an existing room joins it without starting capture. Entering the page
+alone does not create a room. Room close, lost authority and page teardown still
+retire the session. Connected Viewers may keep interacting while the Host page
+is absent, provided the room service and their admitted sessions remain available.
+[Rooms and access](./rooms-access.md#room-interactions) owns interaction authority
+and retention; [status](../status.md#accepted-interaction-phase) owns release readiness.
+
Piik App opens this same application in the system Browser. Its small
startup surface selects Local, temporary public invitation, or a configured
Site before entering the Host workspace. Viewer links opened at that activated
diff --git a/docs/standards/rooms-access.md b/docs/standards/rooms-access.md
index c908780..26fb6b6 100644
--- a/docs/standards/rooms-access.md
+++ b/docs/standards/rooms-access.md
@@ -1,6 +1,6 @@
# Rooms And Access
-This file owns the current room, admission, invitation, and persistence
+This file owns the current room, admission, invitation, interaction, and persistence
contract. [ADR-0002](../adr/0002-memory-resident-protected-rooms.md) explains
the storage decision; implementation detail belongs in code and tests.
@@ -91,6 +91,26 @@ recreate a revoked subscription; no separate media token survives revocation.
Changing code-entry policy or password affects later
code-only attempts and does not silently revoke invitations.
+## Room Interactions
+
+Text and reactions reuse the authenticated room signaling connection. The server
+derives sender identity from that session and accepts targeted reactions only
+for participants still connected to the same room. Interaction subscriptions
+inherit existing membership, replacement and revocation checks; they grant no
+additional authority and do not modify media routes.
+
+Delivery is transient and bounded. The server keeps no message history; each
+page keeps only a bounded set of received messages and effects, without writing
+them to storage or diagnostics. Re-admission to the same room may retain that
+page's conversation, but refresh, leaving or authority loss clears it. There is
+no history replay or offline outbox. HTTPS/WSS protects transport; chat is not
+end-to-end encrypted, and the site operator can access live message contents.
+
+[Presentation and lifecycle](./presentation-lifecycle.md#product-surface) owns
+the distinction between a room session and its current media publication.
+[Interaction research](../research/room-interactions.md) records delivery bounds,
+compatibility and design rationale.
+
## Persistence Modes
- **Stable mode (Hosted default):** one exact-schema SQLite owner persists room
diff --git a/docs/standards/routing-transport.md b/docs/standards/routing-transport.md
index c5fb992..19bfe17 100644
--- a/docs/standards/routing-transport.md
+++ b/docs/standards/routing-transport.md
@@ -7,7 +7,7 @@ owns ports and [self-hosting](../operations/self-hosting.md) owns service setup.
## Topology
-- Media is automatic and P2P-first. Users do not select parents, route types, or
+- Media is automatic and defaults to P2P-first. Users do not select parents, route types, or
network transports.
- The Host is the only source. Every Viewer has at most one active upstream, and
the committed media graph is acyclic and source-reachable.
@@ -21,9 +21,17 @@ same bounded P2P graph without its SFU fallback and fixes Privacy mode on for al
rooms, including App Local and public-link rooms. The server enforces that policy
for waiting and active participants; there is no separate Host-star mode.
+A Hosted operator may require SFU-only media through `SFU_ONLY=true`. The same
+controller then admits only SFU candidates: no Peer parents, Viewer relays or
+background P2P convergence. The Host still publishes once, and every Viewer
+uses the existing SFU subscription ledger and first-frame commit. Capacity or
+connection failure never permits a Peer route. An explicitly peer-only share
+is rejected before acquiring media authority; it is never silently forwarded.
+The page advertises this site policy and hides inapplicable P2P preferences.
+
## Per-Share Route Policy
-The Host chooses route policy before sharing and it remains fixed for that
+On a P2P-first site, the Host chooses route policy before sharing and it remains fixed for that
share generation:
- when SFU is available, default hybrid mode keeps P2P first and permits the
diff --git a/docs/standards/visual-language.md b/docs/standards/visual-language.md
index 39d5f74..857a117 100644
--- a/docs/standards/visual-language.md
+++ b/docs/standards/visual-language.md
@@ -56,6 +56,15 @@ do not stretch short values to fill a phone row. Check actual control bounds
around breakpoints, in both orientations and all presentation modes; page-level
overflow clipping is not evidence that the content fits.
+Persistent room utilities use the shared floating panel without reflowing the
+living room. The shell owns visibility, focus return, movement, resizing and
+viewport/keyboard bounds; contents retain their domain state. Support title-bar
+dragging and keyboard placement. An interaction palette starts beside its chosen
+participant, stays open after sending and keeps its position when retargeted.
+One-shot settings and guidance retain their existing disclosures. Escape closes
+inner guidance first, then the most recently opened floating panel, then theater
+mode; an IME composition must not dismiss these surfaces.
+
Keep the application language control's segmented Chinese, English and visual
shortcuts. When additional catalogs exist, insert one language menu
after the visual shortcut; all extra languages share that slot. Show the
@@ -172,6 +181,8 @@ Participant characters should leave room for that distinctive presence.
| Browser application | Browser chrome around content; not a second design of television |
| A captured window or display | Window title bar or display stand, with the shared media metaphor kept distinct |
| A camera source or Host commentary | `camera` identifies the alternative picture source; `microphone` identifies the enabled Host voice input and `microphoneOff` its muted state. Neither replaces the speaker metaphor for source/playback sound. |
+| Room text and reactions | `chat` opens the conversation, `danmaku` controls its optional picture overlay, `smile` identifies reactions and `send` submits the chosen content. These do not represent media transport or voice. |
+| Floating utility placement | `grip` marks a draggable title bar; `move` opens placement controls. Closing a panel hides its presentation, not its room session. |
| Server forwarding media | A server on the media path; opening a Site alone does not imply SFU |
These are **identity/object colours**, not status. A green pawn remains green
diff --git a/docs/status.md b/docs/status.md
index 63f8ebf..1a086f7 100644
--- a/docs/status.md
+++ b/docs/status.md
@@ -1,6 +1,6 @@
# Current Status
-Last updated: 2026-09-29
+Last updated: 2026-10-02
This is the compact execution/deployment index. Product modules own behavior,
[verification status](./verification-status.md) owns unresolved physical limits,
@@ -35,6 +35,24 @@ Website delivery includes a bilingual reader documentation center generated
from the repository's guides alongside the homepage, with search and detailed
troubleshooting separate from the homepage FAQ.
+## Accepted Interaction Phase
+
+The accepted [room-interaction design](./research/room-interactions.md) includes
+the verified media repairs. It adds authenticated text and reactions
+over existing signaling, with optional local chat overlay and a shared floating
+panel. Its opted-in room session can outlive a publication; capture, quality and
+route owners remain unchanged. Optional Windows screen-audio exclusion uses the
+existing source/mixer boundary. Hosted Server
+also accepts an optional SFU-only policy through the same route controller.
+Windows native capture and thumbnails now share automatic HDR-to-SDR conversion;
+synthetic GPU and Browser reception checks pass, with physical HDR display
+acceptance still pending in [verification status](./verification-status.md).
+Room voice is outside the product plan; Host microphone commentary remains.
+4K remains deferred. The owner accepted the UI/model and bounded local acceptance
+and authorized v1.7.0 publication. Physical device limits remain open.
+Release artifacts and operator deployment records own delivery completion;
+[TODO](./todo.md#now) owns remaining work.
+
## Deployment
Source, package identity, GitHub repository and local worktree paths use Piik.
diff --git a/docs/todo.md b/docs/todo.md
index c5dbee7..18c47af 100644
--- a/docs/todo.md
+++ b/docs/todo.md
@@ -1,70 +1,58 @@
# Current TODO Ledger
-Last reviewed: 2026-09-30
+Last reviewed: 2026-10-02
Only **Now** is executable. Product modules own behavior; Git/PRs own completed
history. A parked idea is not implementation authority.
## Now
-- [ ] **Published-product bug and experience acceptance.** Prioritize share
- startup, audio/source changes, sustained viewing, recovery and retirement,
- admission, and narrow/touch UI. Verify confirmed fixes through the existing
- owners before resuming feature polish. Keep field reports without matched
- evidence separate from locally reproduced defects.
-- [ ] **Sustained Browser degradation.** The H264 field cause remains open.
+- [ ] **Authorized release delivery.** The owner accepted the room-interaction
+ phase, complete bilingual release notes and bounded acceptance, and authorized
+ v1.7.0 publication. Follow
+ [the integration workflow](../CONTRIBUTING.md#full-integration-and-release-workflow)
+ for one squash merge, packaging, publication and scoped postflight. Release
+ artifacts and operator records own completion; do not add a release-record
+ commit. No implementation work remains open for this phase.
+
+## Held By Owner
+
+- [ ] **Browser-internal encoder work.** Paused by the owner on 2026-10-01.
[Comparisons and Chromium traces](./research/browser-local-encoding-pool.md#sustained-h264-recovery)
- distinguish encoder recovery hysteresis from a tab-capture feedback/size-step
- lock reproduced in bare WebRTC. Verify the upstream soft-target/hard-maximum
- repair candidate against a patched Browser, which remains unbuilt. The Piik
- sender boundary now isolates display frames while preserving native idle refresh,
- capture intent and operation retirement. Real hidden-tab SFU checks pass;
- broader device/background coverage remains. Native logs confirm content-driven
- QP hysteresis and a cost from bitrate reconfiguration, but not repeated brief
- blur. Match field Host/Viewer evidence
- before assigning the field cause or declaring repeated brief blur fixed.
- Retain existing clone/startup protections and candidate proof; no periodic
- reset or quality floor. A recent Piik regression is not established.
+ own the evidence for synchronous H264 initialization, native adaptation and
+ the unbuilt upstream repair candidate. Keep the verified Piik budget and
+ handoff repairs; do not add periodic resets, a quality floor or a Browser fork.
+ The field room's cause and all brief blur are not established as resolved.
## Deferred Feature Work
-Feature design and interaction polish are paused while the bug/experience phase
-is active. Retain these decisions for resumption, not as parallel work.
+These proposals remain deferred beyond the accepted interaction phase.
- [ ] **Passive App attachment: design hold.** Site mode authorizes one selected
origin and supplies native media without starting a local room server. A
passive replacement needs an accepted site-consent/discovery flow; it must not
admit arbitrary sites or add another runtime owner. Keep Site mode until that
decision; removal of the Demo prefill does not authorize changing site trust.
-- [ ] **Refine the isolated room-interaction prototype.** Review chat, optional
- danmaku and participant-targeted reactions for usability, placement and intended
- cross-view synchronization. Preserve room authorization and media-route owners.
- Accept its combined layout: sharing settings with the picture, floating chat
- without page reflow and room actions before diagnostics. Check narrow-screen
- keyboard avoidance, its theater entry and participant menu on real devices;
- keep chat available independently of sharing.
- Retain current room creation: the first share creates a room, an existing room
- can resume, and stopping media keeps the opted-in interaction session. Do not
- add visit-triggered creation or a separate pre-share room-creation entry.
- This experiment does not authorize integration or publication.
-- [ ] **Room voice: design before implementation.** The
- [voice assessment](./research/room-voice.md) maps current owners, mature options
- and a staged verification plan. Resolve total voice membership/topology,
- P2P-only deployment coverage, speaking authority and the Host-commentary
- compatibility boundary before implementation. Validate source-audio isolation
- from received speech before exposing simultaneous sharing and voice. Keep
- room authority, microphone and screen lifetimes distinct; no new room store,
- screen-tree reversal or automatic mesh/SFU controller is approved.
-- [ ] **Verify media fidelity independently of voice.**
- Verify actual left/right separation through Browser, Native and SFU paths;
- receive-side stereo negotiation alone does not establish audible fidelity.
- Assess HDR capture to correct SDR output for ordinary viewers before full HDR,
- with issue [#420](https://github.com/TNTcraftHIM/Piik/issues/420) as the reported
- case. Full HDR and surround sound remain design decisions. These findings do
- not establish physical device acceptance or authorize a new release.
## Awaiting Device Or Reporter Evidence
+- [ ] **HDR, audio-exclusion and mobile interaction device coverage.** Complete
+ the remaining physical checks in
+ [verification status](./verification-status.md#candidate-evidence-boundary):
+ real HDR/mixed displays under load, real voice-app process trees across playback
+ devices, and phone keyboard/theater/participant-menu interaction. Local GPU,
+ decoded-video, synthetic audio and responsive-browser evidence do not cover
+ those environments. Issues [#420](https://github.com/TNTcraftHIM/Piik/issues/420)
+ (Browser HDR) and [#445](https://github.com/TNTcraftHIM/Piik/issues/445)
+ (Windows 10 colors) still need matched capture evidence; do not assign them
+ the locally reproduced Native conversion defects.
+- [ ] **Native H.264 motion quality (#432).** The original 720p30 H.264/VP8
+ visual comparison remains unresolved. Browser evidence confirms similar
+ delivered bitrate/format and zero reported loss, but contains no matched
+ pictures or bitstream. The reporter withdrew the fixture-contaminated CBR
+ ceiling claim. [Evidence review](./research/native-client-media.md#native-h264-motion-quality)
+ owns that distinction. Reproduce with identical content and unchanged
+ production settings before proposing profile, default-FPS or bitrate changes.
- [ ] **Self-hosted room creation HTTP 403.** Retest affected deployments using
the [address checks](./guide/troubleshooting.md#room-creation-returns-403).
Obtain the configured public address/origin and response details; distinguish
@@ -78,6 +66,9 @@ is active. Retain these decisions for resumption, not as parallel work.
evidence before calling that an ICE/NAT failure. HTTP 1033 belongs to the
public-link item below, before media routing. Local fixes and STUN checks do
not establish these reporters' causes.
+ [#443](https://github.com/TNTcraftHIM/Piik/issues/443)'s Host report reaches Native
+ prepare/answer/candidate processing without a connected remote edge; obtain
+ the same attempt's Viewer and App transport reports to locate the failure.
- [ ] **Camera and Host microphone device coverage.** The owner accepted the
sharing layout and authorized release with these physical limits recorded.
Verify real audio levels/echo, multiple-device replacement and native mixing on
@@ -92,6 +83,9 @@ is active. Retain these decisions for resumption, not as parallel work.
[runtime evidence](./research/cross-platform-client-runtime.md#public-invitation-startup).
Demo access failures and public-link HTTP 1033 also remain unconfirmed; check
connector, DNS/provider and remote access separately from WebRTC availability.
+ [#434](https://github.com/TNTcraftHIM/Piik/issues/434)'s attached log records DNS
+ refusal during Cloudflare edge discovery, before readiness; retrying media
+ cannot repair that resolver failure. Do not assign it to every startup report.
- [ ] **Interruption during established viewing.** A Viewer reportedly returns
to P2P connecting after watching for a while. Include
[#429](https://github.com/TNTcraftHIM/Piik/issues/429)'s reported SFU-to-P2P
@@ -126,6 +120,10 @@ is active. Retain these decisions for resumption, not as parallel work.
also needs paired evidence; distinguish activation, codec configuration and
actual output before changing the encoder contract. Local NVIDIA success
does not settle AMD activation.
+ In [#437](https://github.com/TNTcraftHIM/Piik/issues/437), both manually selected
+ codecs reportedly work; its Browser report records Auto startup returning
+ `operation-failed` after 6.85 seconds, before the request timeout. Obtain the
+ same attempt's App diagnostics to locate selection/activation failure.
- [ ] **Share ends after entering a game.** Screen sharing reportedly works
until entering a game freezes the picture, followed seconds later by share
termination. Version, capture path, codec and matched diagnostics are unknown.
@@ -249,3 +247,7 @@ not establish better connection success or speed; this note adds no retry policy
11. **Release-operation policy.** Protected release environments, immutable
draft assets and changing mirror-failure policy remain unaccepted proposals;
evaluate their benefit before adding release machinery.
+12. **Linux ARM64 Server distribution (#439).** Evaluate archives and container
+ publishing together with deployment names, update links and runtime checks.
+ A cross-compile alone does not establish a supported package. This is a
+ community proposal, not part of the current accepted candidate.
diff --git a/docs/verification-status.md b/docs/verification-status.md
index 470100a..70aaa27 100644
--- a/docs/verification-status.md
+++ b/docs/verification-status.md
@@ -1,6 +1,6 @@
# Verification Status
-Last updated: 2026-09-17
+Last updated: 2026-10-01
This file owns physical evidence limits that change how the product may be
described. [Status](./status.md) owns the execution/deployment index.
@@ -47,6 +47,26 @@ or a measured improvement worth its implementation and maintenance cost.
## Candidate Evidence Boundary
+The Windows HDR-to-SDR candidate has synthetic FP16 conversion and decoded
+H.264/VP8 Browser reception evidence. Local displays were in SDR mode; real HDR
+content, SDR white on HDR displays, mixed-display movement, display-mode changes
+and sustained GPU contention still require physical acceptance. The
+[fidelity assessment](./research/media-fidelity.md#hdr-to-sdr) owns measurements.
+These checks do not establish Browser-capture HDR behavior or end-to-end HDR.
+
+Windows source-audio exclusion has bounded parent/child tone-isolation and
+target-exit evidence. It does not establish coverage of arbitrary process trees
+or real voice applications across playback devices. The
+[audio assessment](./research/native-client-media.md#windows-audio-exclusion)
+owns that distinction.
+
+Room interactions have multi-page Browser evidence for stop/resume, idle-room
+join, Host departure/return, reconnect, invitation revocation and event timing.
+Responsive Browser checks cover narrow portrait and landscape layouts, but not
+real phone keyboards, Safari behavior or every touch/device combination. The
+owner accepted this bounded UI/model walkthrough for phase closure; these
+physical limits remain open and are not release authorization.
+
The 2026-09-11 functional test acceptance used two genuine Windows App packages
and matching Server source/Web builds under the same private contract. Both App/Site version
directions delivered decoded 1080p H264 from an isolated native test window.
diff --git a/internal/app/app.go b/internal/app/app.go
index b645a17..f65036a 100644
--- a/internal/app/app.go
+++ b/internal/app/app.go
@@ -621,14 +621,15 @@ func discoverNativeMedia(ctx context.Context, configuredPath string) nativeRunti
captureProcess: path,
capture: capabilities,
capabilities: loopback.NativeMediaCapabilities{
- ReceiverReuse: true,
- Video: summary.Video,
- CaptureBorderControl: summary.CaptureBorderControl,
- Microphone: summary.Microphone,
- ProcessAudio: summary.ProcessAudio,
- SystemAudio: summary.SystemAudio,
- HardwareH264: summary.HardwareH264,
- SoftwareVP8: summary.SoftwareVP8,
+ ReceiverReuse: true,
+ Video: summary.Video,
+ CaptureBorderControl: summary.CaptureBorderControl,
+ Microphone: summary.Microphone,
+ ProcessAudio: summary.ProcessAudio,
+ ProcessAudioExclusion: summary.ProcessAudioExclusion,
+ SystemAudio: summary.SystemAudio,
+ HardwareH264: summary.HardwareH264,
+ SoftwareVP8: summary.SoftwareVP8,
},
}
}
diff --git a/internal/app/app_test.go b/internal/app/app_test.go
index 40ff38d..2b17851 100644
--- a/internal/app/app_test.go
+++ b/internal/app/app_test.go
@@ -121,6 +121,12 @@ func TestPublicLinkStartupDistinguishesCancellationFromFailure(t *testing.T) {
}
cancel()
}
+ waitForResult := 5 * time.Second
+ if mode == "failed" {
+ // Startup includes three retry pauses and four real child launches;
+ // this is not the cancellation/ready-tunnel retirement deadline.
+ waitForResult = 15 * time.Second
+ }
select {
case err = <-done:
joined = true
@@ -139,7 +145,7 @@ func TestPublicLinkStartupDistinguishesCancellationFromFailure(t *testing.T) {
if ready != (mode == "ready" || mode == "ready-failed") {
t.Fatalf("startup readiness = %v for %s", ready, mode)
}
- case <-time.After(5 * time.Second):
+ case <-time.After(waitForResult):
t.Fatal("App did not retire its local authority and tunnel")
}
content, err := os.ReadFile(filepath.Join(directory, "logs", "client.log"))
diff --git a/internal/app/loopback/server.go b/internal/app/loopback/server.go
index 798b931..a2f4e66 100644
--- a/internal/app/loopback/server.go
+++ b/internal/app/loopback/server.go
@@ -69,14 +69,15 @@ type Health struct {
}
type NativeMediaCapabilities struct {
- ReceiverReuse bool `json:"receiverReuse,omitempty"`
- Video bool `json:"video"`
- CaptureBorderControl bool `json:"captureBorderControl,omitempty"`
- Microphone bool `json:"microphone,omitempty"`
- ProcessAudio bool `json:"processAudio"`
- SystemAudio bool `json:"systemAudio"`
- HardwareH264 bool `json:"hardwareH264"`
- SoftwareVP8 bool `json:"softwareVP8"`
+ ReceiverReuse bool `json:"receiverReuse,omitempty"`
+ Video bool `json:"video"`
+ CaptureBorderControl bool `json:"captureBorderControl,omitempty"`
+ Microphone bool `json:"microphone,omitempty"`
+ ProcessAudio bool `json:"processAudio"`
+ ProcessAudioExclusion bool `json:"processAudioExclusion,omitempty"`
+ SystemAudio bool `json:"systemAudio"`
+ HardwareH264 bool `json:"hardwareH264"`
+ SoftwareVP8 bool `json:"softwareVP8"`
}
type Server struct {
diff --git a/internal/app/mediaedge/edge_test.go b/internal/app/mediaedge/edge_test.go
index 6b5a495..700c8de 100644
--- a/internal/app/mediaedge/edge_test.go
+++ b/internal/app/mediaedge/edge_test.go
@@ -65,9 +65,11 @@ func TestOutputPlanRetiresOnlyFailedLayerConsumers(t *testing.T) {
lowerTracker, higherTracker := trackers.AddTracker(0), trackers.AddTracker(1)
lowerTracker.Observe(0, 1200, 1188, true, 90000, nil)
higherTracker.Observe(0, 1200, 1188, true, 90000, nil)
- if err = source.DisableLayer(0); err != nil {
+ retire, err := source.MarkLayerUnavailable(0)
+ if err != nil {
t.Fatal(err)
}
+ retire()
if low.State() != webrtc.PeerConnectionStateClosed || high.State() != webrtc.PeerConnectionStateConnected {
t.Fatalf("failed/sibling state = %s/%s", low.State(), high.State())
}
@@ -174,6 +176,59 @@ func TestRelayFailureCannotRetireReconfiguredProfile(t *testing.T) {
}
}
+func TestRetiredRelayFailureCannotDisableLaterDemand(t *testing.T) {
+ engine, err := NewEngine(EngineOptions{BindAddress: "127.0.0.1:0", IncludeLoopback: true})
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = engine.Close() })
+ source, err := engine.NewSource("vp8", 2, 2, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = source.Close() })
+ source.relay = &relayDerivation{source: source}
+ if err = source.SetFormat(1, 640, 360); err != nil {
+ t.Fatal(err)
+ }
+ if err = source.SetRelayProfile(nativecapture.VideoProfile{
+ Width: 1280, Height: 720, Framerate: 30, Bitrate: 3_000_000, Preference: "balanced",
+ }); err != nil {
+ t.Fatal(err)
+ }
+ high, _, _ := connectedReceiver(t, engine, source, "retiring-relay-high")
+ low, _, _ := connectedReceiver(t, engine, source, "retiring-relay-low")
+ if err = low.SetTargetLayer(0); err != nil {
+ t.Fatal(err)
+ }
+ ctx, cancel := context.WithCancelCause(t.Context())
+ defer cancel(nil)
+ done := make(chan struct{})
+ t.Cleanup(func() { close(done) })
+ run := &relayRun{owner: source.relay, ctx: ctx, cancel: cancel, done: done,
+ plan: relayPlan{profile: source.relayProfile, format: source.formats[1].Load()}}
+ source.relay.run = run
+ if err = low.Close(); err != nil {
+ t.Fatal(err)
+ }
+ if !errors.Is(context.Cause(ctx), context.Canceled) {
+ t.Fatal("last lower consumer did not retire the derivation normally")
+ }
+ // A failure already read from the old process can reach the source lock
+ // after cancellation, before the process exits and the next run starts.
+ retire, err := source.markLayerUnavailable(0, run)
+ if err != nil || retire != nil || source.outputBitrates[0] == 0 {
+ t.Fatal("normal retirement disabled a reusable output slot")
+ }
+ rejoined, _, _ := connectedReceiver(t, engine, source, "retiring-relay-rejoined")
+ if err = rejoined.SetTargetLayer(0); err != nil {
+ t.Fatalf("later lower demand was rejected: %v", err)
+ }
+ if rejoined.State() != webrtc.PeerConnectionStateConnected || high.State() != webrtc.PeerConnectionStateConnected {
+ t.Fatal("late failure retired a current consumer or healthy sibling")
+ }
+}
+
func TestShutdownClosesSiblingTransportsBeforeWaitingForWrites(t *testing.T) {
for _, owner := range []string{"source", "engine"} {
t.Run(owner, func(t *testing.T) {
diff --git a/internal/app/mediaedge/groups.go b/internal/app/mediaedge/groups.go
index 15f3c81..b0d8dbe 100644
--- a/internal/app/mediaedge/groups.go
+++ b/internal/app/mediaedge/groups.go
@@ -95,6 +95,23 @@ func (source *Source) planGroups(demands []outputDemand) (OutputPlan, error) {
delete(source.memberships, consumer)
}
}
+ compatible := func(group *outputGroup, demand outputDemand) bool {
+ for _, other := range demands {
+ if other.active && other.lower && other.budget != demand.budget &&
+ (other.consumer.CurrentSource() == group.media.Source || source.memberships[other.consumer] == group) {
+ return false
+ }
+ }
+ return true
+ }
+ // Revalidate pending joins before they can prevent an incumbent from
+ // adjusting in place. Membership order must not give stale demand priority.
+ for _, demand := range demands {
+ group := source.memberships[demand.consumer]
+ if demand.active && demand.lower && group != nil && demand.consumer.CurrentSource() != group.media.Source && !compatible(group, demand) {
+ delete(source.memberships, demand.consumer)
+ }
+ }
// The previous frame's active bit alone may outlive its last handoff.
inUse := func(group *outputGroup) bool {
if group == nil || !group.active {
@@ -119,6 +136,9 @@ func (source *Source) planGroups(demands []outputDemand) (OutputPlan, error) {
for slot := 0; slot < len(source.outputBitrates); slot++ {
group := source.groups[slot]
if group != nil && source.outputBitrates[slot] > 0 && group.budget == demand.budget {
+ if demand.consumer.CurrentSource() != group.media.Source && !compatible(group, demand) {
+ continue
+ }
if selected == nil || inUse(group) {
selected = group
}
@@ -152,15 +172,7 @@ func (source *Source) planGroups(demands []outputDemand) (OutputPlan, error) {
if source.outputBitrates[group.slot] == 0 {
continue
}
- compatible := true
- for _, other := range demands {
- if other.active && other.lower && other.budget != demand.budget &&
- (other.consumer.CurrentSource() == group.media.Source || source.memberships[other.consumer] == group) {
- compatible = false
- break
- }
- }
- if compatible {
+ if compatible(group, demand) {
selected = group
break
}
diff --git a/internal/app/mediaedge/groups_test.go b/internal/app/mediaedge/groups_test.go
index d0ebf6b..7b23156 100644
--- a/internal/app/mediaedge/groups_test.go
+++ b/internal/app/mediaedge/groups_test.go
@@ -1,6 +1,7 @@
package mediaedge
import (
+ "fmt"
"testing"
"time"
@@ -253,6 +254,64 @@ func newGroupLifecycleFixture(t *testing.T) (*Source, *groupTestConsumer, *outpu
return source, b, group, demands
}
+func TestOutputGroupsPreserveCurrentDemandDuringAdmission(t *testing.T) {
+ for _, pending := range []bool{false, true} {
+ for _, reverse := range []bool{false, true} {
+ t.Run(fmt.Sprintf("pending=%v/reverse=%v", pending, reverse), func(t *testing.T) {
+ source, b, gb, demands := newGroupLifecycleFixture(t)
+ a := demands[0].consumer
+ ga := source.memberships[a]
+ source.installGroup(ga, false)
+ source.installGroup(gb, false)
+ groupCount := len(source.groups)
+ demands[1].budget = 100_000
+ if pending {
+ if _, err := source.planGroups(demands); err != nil {
+ t.Fatal(err)
+ }
+ if source.memberships[b] != ga || b.CurrentSource() != gb.media.Source {
+ t.Fatal("fixture did not prepare a compatible rejoin")
+ }
+ }
+ source.groupRecovery.Store(0)
+ demands[0].budget = 60_000
+ if reverse {
+ demands[0], demands[1] = demands[1], demands[0]
+ }
+ plan, err := source.planGroups(demands)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if source.memberships[a] != ga || source.memberships[b] != gb || len(source.groups) != groupCount ||
+ plan.Bitrates[ga.slot] != 60_000 || plan.Bitrates[gb.slot] != 100_000 || source.groupRecovery.Load() != 0 {
+ t.Fatalf("admission displaced an incumbent: a=%v b=%v groups=%d/%d rates=%v recovery=%d",
+ source.memberships[a] == ga, source.memberships[b] == gb, len(source.groups), groupCount,
+ plan.Bitrates, source.groupRecovery.Load())
+ }
+ })
+ }
+ }
+}
+
+func TestOutputGroupsRetainOriginalRecoveryAfterPendingJoin(t *testing.T) {
+ source, b, gb, demands := newGroupLifecycleFixture(t)
+ source.installGroup(source.memberships[demands[0].consumer], false)
+ source.installGroup(gb, false)
+ demands[1].budget = demands[0].budget
+ if _, err := source.planGroups(demands); err != nil {
+ t.Fatal(err)
+ }
+ source.groupRecovery.Store(0)
+ demands[0].budget = 60_000
+ demands[1].lower, demands[1].original = false, true
+ if _, err := source.planGroups(demands); err != nil {
+ t.Fatal(err)
+ }
+ if source.memberships[b] != nil || source.groupRecovery.Load()&2 == 0 || b.CurrentSource() != gb.media.Source {
+ t.Fatal("return to original lost recovery or replaced the current source early")
+ }
+}
+
func TestOutputGroupRejectsRecoveryBeforeChangingAttachment(t *testing.T) {
for _, path := range []string{"derived", "original"} {
for _, stamp := range []string{"future", "duplicate", "before-generation-start"} {
diff --git a/internal/app/mediaedge/publication_test.go b/internal/app/mediaedge/publication_test.go
index f36fd15..4b94b37 100644
--- a/internal/app/mediaedge/publication_test.go
+++ b/internal/app/mediaedge/publication_test.go
@@ -73,9 +73,11 @@ func TestPublicationOwnsOneReservationAndSourceLifetime(t *testing.T) {
if err != nil {
t.Fatal(err)
}
- if err = source.DisableLayer(0); err != nil {
+ retire, err := source.MarkLayerUnavailable(0)
+ if err != nil {
t.Fatal(err)
}
+ retire()
if _, err = engine.NewPublication(source, EdgeOptions{ConnectionID: "failed-output"}); err == nil {
t.Fatal("failed output was admitted into publication metadata")
}
@@ -117,9 +119,11 @@ func TestPublicationAdmissionIgnoresUnrelatedFailedOutputs(t *testing.T) {
t.Fatal(err)
}
for _, slot := range []int{2, 3, 0, 1} {
- if err = source.DisableLayer(slot); err != nil {
+ retire, err := source.MarkLayerUnavailable(slot)
+ if err != nil {
t.Fatal(err)
}
+ retire()
candidate, createErr := engine.NewPublication(source, EdgeOptions{ConnectionID: "replacement"})
if slot >= 2 {
if createErr != nil {
diff --git a/internal/app/mediaedge/source.go b/internal/app/mediaedge/source.go
index de2ef3d..77a781f 100644
--- a/internal/app/mediaedge/source.go
+++ b/internal/app/mediaedge/source.go
@@ -1,6 +1,7 @@
package mediaedge
import (
+ "context"
"errors"
"io"
"sync"
@@ -211,12 +212,10 @@ func (source *Source) ConfigureOutputs(ceilings []uint32) error {
return nil
}
-func (source *Source) DisableLayer(layer int) error {
- retire, err := source.markLayerUnavailable(layer, nil)
- if retire != nil {
- retire()
- }
- return err
+// MarkLayerUnavailable commits output failure without closing transports. The
+// caller must run the returned retirement after releasing its capture-owner lock.
+func (source *Source) MarkLayerUnavailable(layer int) (func(), error) {
+ return source.markLayerUnavailable(layer, nil)
}
func (source *Source) markLayerUnavailable(layer int, run *relayRun) (func(), error) {
@@ -227,8 +226,10 @@ func (source *Source) markLayerUnavailable(layer int, run *relayRun) (func(), er
source.writeMu.Lock()
defer source.writeMu.Unlock()
source.mu.Lock()
- if run != nil && (run.owner.run != run || source.relayProfile != run.plan.profile ||
- source.formats[len(source.formats)-1].Load() != run.plan.format) {
+ // Normal retirement cannot disable a later run's slots. A real cancellation
+ // cause still owns failure isolation until that run has finished retiring.
+ if run != nil && (run.owner.run != run || errors.Is(context.Cause(run.ctx), context.Canceled) ||
+ source.relayProfile != run.plan.profile || source.formats[len(source.formats)-1].Load() != run.plan.format) {
source.mu.Unlock()
return nil, nil
}
diff --git a/internal/app/nativeaudio/encoder_test.go b/internal/app/nativeaudio/encoder_test.go
index 82ab344..0744054 100644
--- a/internal/app/nativeaudio/encoder_test.go
+++ b/internal/app/nativeaudio/encoder_test.go
@@ -85,6 +85,58 @@ func TestEncodePreservesAudiblePCM(t *testing.T) {
}
}
+func TestEncodePreservesStereoSeparation(t *testing.T) {
+ for _, bitrate := range []int{64_000, 128_000, 192_000} {
+ encoder, err := NewEncoder(bitrate)
+ if err != nil {
+ t.Fatal(err)
+ }
+ decoder, err := gopus.NewDecoder(gopus.DefaultDecoderConfig(SampleRate, Channels))
+ if err != nil {
+ t.Fatal(err)
+ }
+ pcm := make([]byte, FrameBytes)
+ decoded := make([]int16, FrameSamples*Channels)
+ frequencies := [2]float64{500, 1500}
+ var tones [2][2]complex128
+ for block := 0; block < 30; block++ {
+ for frame := 0; frame < FrameSamples; frame++ {
+ for channel, frequency := range frequencies {
+ phase := 2 * math.Pi * frequency * float64(block*FrameSamples+frame) / SampleRate
+ binary.LittleEndian.PutUint16(pcm[(frame*Channels+channel)*2:], uint16(int16(8_000*math.Sin(phase))))
+ }
+ }
+ packet, err := encoder.Encode(pcm)
+ if err != nil {
+ t.Fatal(err)
+ }
+ count, err := decoder.DecodeInt16(packet, decoded)
+ if err != nil || count != FrameSamples {
+ t.Fatalf("decode: %d samples, %v", count, err)
+ }
+ if block < 5 {
+ continue // Exclude codec startup; compare frequency energy, not phase delay.
+ }
+ for frame := 0; frame < count; frame++ {
+ for tone, frequency := range frequencies {
+ phase := 2 * math.Pi * frequency * float64(block*FrameSamples+frame) / SampleRate
+ for channel := range Channels {
+ tones[channel][tone] += complex(float64(decoded[frame*Channels+channel]), 0) * complex(math.Cos(phase), math.Sin(phase))
+ }
+ }
+ }
+ }
+ for channel := range Channels {
+ wanted, other := tones[channel][channel], tones[channel][1-channel]
+ level := math.Hypot(real(wanted), imag(wanted))
+ leak := math.Hypot(real(other), imag(other))
+ if level < 1 || level < 10*leak {
+ t.Fatalf("%d bps channel %d lost separation: signal %f, leakage %f", bitrate, channel, level, leak)
+ }
+ }
+ }
+}
+
func TestNewEncoderRejectsOutOfRangeBitrate(t *testing.T) {
for _, bitrate := range []int{0, 5_999, 510_001} {
if _, err := NewEncoder(bitrate); err == nil {
diff --git a/internal/app/nativecapture/probe.go b/internal/app/nativecapture/probe.go
index cc30fcb..8f4a0a1 100644
--- a/internal/app/nativecapture/probe.go
+++ b/internal/app/nativecapture/probe.go
@@ -42,36 +42,39 @@ type Adapter struct {
}
type Capabilities struct {
- Protocol int `json:"protocol"`
- Platform string `json:"platform"`
- PlatformBuild string `json:"platformBuild"`
- VideoCapture bool `json:"videoCapture"`
- CaptureBorderControl bool `json:"captureBorderControl,omitempty"`
- Microphone bool `json:"microphone,omitempty"`
- ProcessAudio bool `json:"processAudio"`
- SystemAudio bool `json:"systemAudio"`
- SoftwareVP8 bool `json:"softwareVP8"`
- Adapters []Adapter `json:"adapters"`
+ Protocol int `json:"protocol"`
+ Platform string `json:"platform"`
+ PlatformBuild string `json:"platformBuild"`
+ VideoCapture bool `json:"videoCapture"`
+ CaptureBorderControl bool `json:"captureBorderControl,omitempty"`
+ Microphone bool `json:"microphone,omitempty"`
+ ProcessAudio bool `json:"processAudio"`
+ ProcessAudioExclusion bool `json:"processAudioExclusion,omitempty"`
+ SystemAudio bool `json:"systemAudio"`
+ SoftwareVP8 bool `json:"softwareVP8"`
+ Adapters []Adapter `json:"adapters"`
}
type Summary struct {
- Video bool
- CaptureBorderControl bool
- Microphone bool
- ProcessAudio bool
- SystemAudio bool
- HardwareH264 bool
- SoftwareVP8 bool
+ Video bool
+ CaptureBorderControl bool
+ Microphone bool
+ ProcessAudio bool
+ ProcessAudioExclusion bool
+ SystemAudio bool
+ HardwareH264 bool
+ SoftwareVP8 bool
}
func (capabilities Capabilities) Summary() Summary {
summary := Summary{
- Video: capabilities.VideoCapture,
- CaptureBorderControl: capabilities.VideoCapture && capabilities.CaptureBorderControl,
- Microphone: capabilities.Microphone,
- ProcessAudio: capabilities.ProcessAudio,
- SystemAudio: capabilities.SystemAudio,
- SoftwareVP8: capabilities.SoftwareVP8,
+ Video: capabilities.VideoCapture,
+ CaptureBorderControl: capabilities.VideoCapture && capabilities.CaptureBorderControl,
+ Microphone: capabilities.Microphone,
+ ProcessAudio: capabilities.ProcessAudio,
+ ProcessAudioExclusion: capabilities.ProcessAudioExclusion && capabilities.Microphone,
+ SystemAudio: capabilities.SystemAudio,
+ SoftwareVP8: capabilities.SoftwareVP8,
}
for _, adapter := range capabilities.Adapters {
if len(adapter.HardwareH264) > 0 {
diff --git a/internal/app/nativecapture/probe_test.go b/internal/app/nativecapture/probe_test.go
index 6f2d1ef..893b955 100644
--- a/internal/app/nativecapture/probe_test.go
+++ b/internal/app/nativecapture/probe_test.go
@@ -33,6 +33,22 @@ func TestDecodeProbeRejectsUnknownTrailingAndDuplicateData(t *testing.T) {
}
}
+func TestAudioExclusionCapabilityRequiresExplicitSupportAndMixing(t *testing.T) {
+ capabilities, err := decodeProbe([]byte(validProbe))
+ if err != nil || capabilities.Summary().ProcessAudioExclusion {
+ t.Fatal("old capture probe advertised audio exclusion", err)
+ }
+ capabilities, err = decodeProbe([]byte(strings.Replace(validProbe, `"processAudio":true`,
+ `"processAudio":true,"processAudioExclusion":true,"microphone":true`, 1)))
+ if err != nil || !capabilities.Summary().ProcessAudioExclusion {
+ t.Fatal("explicit audio exclusion capability was lost", err)
+ }
+ capabilities.Microphone = false
+ if capabilities.Summary().ProcessAudioExclusion {
+ t.Fatal("audio exclusion requires the replaceable mixed source")
+ }
+}
+
func TestSummaryDoesNotConflateVideoAudioAndEncoder(t *testing.T) {
capabilities, err := decodeProbe([]byte(
`{"protocol":7,"platform":"windows","platformBuild":"19045","videoCapture":true,"processAudio":false,"systemAudio":true,"adapters":[{"index":0,"name":"GPU","identity":"0x0:0x1","hardwareH264":[]}]}`,
diff --git a/internal/app/nativecapture/stream.go b/internal/app/nativecapture/stream.go
index b8caf53..2b31d66 100644
--- a/internal/app/nativecapture/stream.go
+++ b/internal/app/nativecapture/stream.go
@@ -281,13 +281,17 @@ func appendOutputArguments(arguments []string, outputs []OutputProfile) ([]strin
return arguments, nil
}
-func StartAudio(parent context.Context, executable string, target CaptureTarget) (*Stream, error) {
- if !target.Valid() {
+func StartAudio(parent context.Context, executable string, target CaptureTarget, exclude bool) (*Stream, error) {
+ if !target.Valid() || (exclude && target.Kind != "window") {
return nil, errors.New("native audio target is invalid")
}
+ kind := target.Kind
+ if exclude {
+ kind = "exclude"
+ }
return startAudioStream(parent, executable, []string{
"--capture-audio",
- target.Kind,
+ kind,
strconv.FormatUint(uint64(target.PID), 10),
zeroWhenEmpty(target.CreationTime),
})
diff --git a/internal/app/nativecontrol/session.go b/internal/app/nativecontrol/session.go
index 19d80f4..40a59a7 100644
--- a/internal/app/nativecontrol/session.go
+++ b/internal/app/nativecontrol/session.go
@@ -182,6 +182,12 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any,
!validQualitySettings(request.Profile) {
return nil, protocolViolation("native start-share request is invalid")
}
+ if err := session.validateAudioExclusion(request.Source, request.Audio, request.ExcludeAudio); err != nil {
+ return nil, err
+ }
+ if request.ExcludeAudio != nil && !request.MicrophoneMixing {
+ return nil, errors.New("native audio exclusion requires mixed audio")
+ }
slog.Debug("piik-client", "event", "native-profile-requested", "requestId", envelope.ID,
"share", diagnostics.ID(request.ShareID), "profile", nativeQualityProfile(request.Profile), "codec", request.Codec, "audio", request.Audio,
"sourceKind", request.Source.Kind, "adapterIndex", request.AdapterIndex, "encoderIndex", request.EncoderIndex)
@@ -245,6 +251,9 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any,
!validIdentities(request.ShareID) || !request.Source.Valid() {
return nil, protocolViolation("native replace-share-source request is invalid")
}
+ if err := session.validateAudioExclusion(request.Source, request.Audio, request.ExcludeAudio); err != nil {
+ return nil, err
+ }
host := session.current(request.ShareID)
if host == nil {
return nil, errors.New("native share does not exist")
@@ -261,7 +270,7 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any,
ShowCaptureBorder: request.ShowCaptureBorder && session.capabilities.CaptureBorderControl,
AdapterIndex: request.AdapterIndex,
EncoderIndex: request.EncoderIndex,
- }, audio)
+ }, audio, request.ExcludeAudio)
if err != nil {
return nil, err
}
@@ -534,6 +543,19 @@ func (session *Session) finishHostOperation(done chan struct{}, result any) {
session.mu.Unlock()
}
+func (session *Session) validateAudioExclusion(source nativecapture.CaptureTarget, audio bool, excluded *nativecapture.CaptureTarget) error {
+ if excluded == nil {
+ return nil
+ }
+ if !audio || source.Kind != "display" || excluded.Kind != "window" || !excluded.Valid() {
+ return protocolViolation("native audio exclusion target is invalid")
+ }
+ if !session.capabilities.Summary().ProcessAudioExclusion || !session.capabilities.SystemAudio {
+ return errors.New("native audio exclusion is unavailable")
+ }
+ return nil
+}
+
func (session *Session) shareUpdateResult(request requestEnvelope, shareID string, profile nativehost.QualityProfile, err error) any {
if err != nil {
return operationFailure(request, err)
@@ -571,6 +593,7 @@ func (session *Session) startShare(
Profile: profile.Video,
},
MicrophoneMixing: request.MicrophoneMixing && session.capabilities.Microphone,
+ ExcludeAudio: request.ExcludeAudio,
Profile: profile,
EdgeCapacity: request.EdgeCapacity,
AudioEnabled: request.Audio && session.capabilities.Summary().AudioFor(
diff --git a/internal/app/nativecontrol/session_test.go b/internal/app/nativecontrol/session_test.go
index acd5d15..cad4b95 100644
--- a/internal/app/nativecontrol/session_test.go
+++ b/internal/app/nativecontrol/session_test.go
@@ -641,3 +641,32 @@ func TestReplaceShareSourceKeepsOneStrictTargetShape(t *testing.T) {
t.Fatalf("malformed target was treated as an operational failure: %v", err)
}
}
+
+func TestAudioExclusionValidatesScopeWithoutDroppingRequestedPolicy(t *testing.T) {
+ session := &Session{capabilities: nativecapture.Capabilities{Microphone: true, SystemAudio: true, ProcessAudioExclusion: true}}
+ display := nativecapture.CaptureTarget{Kind: "display", SourceID: "1", Title: "Screen"}
+ window := nativecapture.CaptureTarget{Kind: "window", SourceID: "2", PID: 123, CreationTime: "456", Title: "Voice fixture"}
+ if err := session.validateAudioExclusion(display, true, &window); err != nil {
+ t.Fatal(err)
+ }
+ for _, invalid := range []struct {
+ source nativecapture.CaptureTarget
+ audio bool
+ excluded nativecapture.CaptureTarget
+ }{
+ {display, false, window}, {window, true, window}, {display, true, display},
+ {display, true, nativecapture.CaptureTarget{Kind: "window", SourceID: "2", PID: 123, Title: "Missing lifetime"}},
+ } {
+ if _, ok := session.validateAudioExclusion(invalid.source, invalid.audio, &invalid.excluded).(protocolViolation); !ok {
+ t.Fatal("malformed audio exclusion was not rejected")
+ }
+ }
+ session.capabilities.ProcessAudioExclusion = false
+ err := session.validateAudioExclusion(display, true, &window)
+ if _, malformed := err.(protocolViolation); err == nil || malformed {
+ t.Fatal("unsupported audio exclusion must fail the operation, not silently widen capture or close control")
+ }
+ if err := session.validateAudioExclusion(display, true, nil); err != nil {
+ t.Fatal("legacy capture became dependent on exclusion support")
+ }
+}
diff --git a/internal/app/nativecontrol/wire.go b/internal/app/nativecontrol/wire.go
index a68404d..b69ab1c 100644
--- a/internal/app/nativecontrol/wire.go
+++ b/internal/app/nativecontrol/wire.go
@@ -42,19 +42,20 @@ type qualitySettings struct {
}
type startShareRequest struct {
- Version int `json:"version"`
- ID string `json:"id"`
- Type string `json:"type"`
- ShareID string `json:"shareId"`
- Source nativecapture.CaptureTarget `json:"source"`
- Audio bool `json:"audio"`
- MicrophoneMixing bool `json:"microphoneMixing,omitempty"`
- ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"`
- AdapterIndex uint32 `json:"adapterIndex"`
- EncoderIndex uint32 `json:"encoderIndex"`
- EdgeCapacity int `json:"edgeCapacity"`
- Profile qualitySettings `json:"profile"`
- Codec string `json:"codec"`
+ Version int `json:"version"`
+ ID string `json:"id"`
+ Type string `json:"type"`
+ ShareID string `json:"shareId"`
+ Source nativecapture.CaptureTarget `json:"source"`
+ Audio bool `json:"audio"`
+ MicrophoneMixing bool `json:"microphoneMixing,omitempty"`
+ ExcludeAudio *nativecapture.CaptureTarget `json:"excludeAudio,omitempty"`
+ ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"`
+ AdapterIndex uint32 `json:"adapterIndex"`
+ EncoderIndex uint32 `json:"encoderIndex"`
+ EdgeCapacity int `json:"edgeCapacity"`
+ Profile qualitySettings `json:"profile"`
+ Codec string `json:"codec"`
}
type microphoneRequest struct {
@@ -81,15 +82,16 @@ type updateShareRequest struct {
}
type replaceShareSourceRequest struct {
- Version int `json:"version"`
- ID string `json:"id"`
- Type string `json:"type"`
- ShareID string `json:"shareId"`
- Source nativecapture.CaptureTarget `json:"source"`
- Audio bool `json:"audio"`
- ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"`
- AdapterIndex uint32 `json:"adapterIndex"`
- EncoderIndex uint32 `json:"encoderIndex"`
+ Version int `json:"version"`
+ ID string `json:"id"`
+ Type string `json:"type"`
+ ShareID string `json:"shareId"`
+ Source nativecapture.CaptureTarget `json:"source"`
+ Audio bool `json:"audio"`
+ ExcludeAudio *nativecapture.CaptureTarget `json:"excludeAudio,omitempty"`
+ ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"`
+ AdapterIndex uint32 `json:"adapterIndex"`
+ EncoderIndex uint32 `json:"encoderIndex"`
}
type stopShareRequest struct {
diff --git a/internal/app/nativehost/audio_mix_test.go b/internal/app/nativehost/audio_mix_test.go
index 00302c5..bbfb7e3 100644
--- a/internal/app/nativehost/audio_mix_test.go
+++ b/internal/app/nativehost/audio_mix_test.go
@@ -88,6 +88,83 @@ func startMixedFixture(t *testing.T) (*Session, nativecapture.VideoOptions, <-ch
return session, options, events
}
+func TestAudioExclusionFailureMutesOnlySourceAndNeverFallsBack(t *testing.T) {
+ session, options, events := startMixedFixture(t)
+ enabled := true
+ if err := session.SetMicrophone(&enabled, nil, nil); err != nil {
+ t.Fatal(err)
+ }
+ output := session.audioSource
+ session.mixer.mu.Lock()
+ microphone := session.mixer.microphone
+ session.mixer.mu.Unlock()
+ excluded := &nativecapture.CaptureTarget{Kind: "window", SourceID: "123", PID: 456, CreationTime: "789", Title: "Voice fixture"}
+ for _, fault := range []string{"PIIK_VIDEO_FAIL", "PIIK_EXCLUSION_FAIL"} {
+ if err := session.ReplaceSource(options, true, nil); err != nil {
+ t.Fatal(err)
+ }
+ video := session.stream
+ t.Setenv(fault, "1")
+ if err := session.ReplaceSource(options, true, excluded); err == nil {
+ t.Fatal("failed exclusion was accepted", fault)
+ }
+ if session.SourceAudio() == nil || *session.SourceAudio() || session.audioStream != nil || session.stream != video {
+ t.Fatal("failed exclusion retained unfiltered audio or retired video", fault)
+ }
+ t.Setenv(fault, "")
+ }
+ marker := filepath.Join(t.TempDir(), "excluded")
+ t.Setenv("PIIK_EXCLUSION_STARTED", marker)
+ if err := session.ReplaceSource(options, true, excluded); err != nil {
+ t.Fatal(err)
+ }
+ identity, err := os.ReadFile(marker)
+ if err != nil || string(identity) != "456:789" || !*session.SourceAudio() {
+ t.Fatal("replacement did not bind exclusion to the selected process lifetime", err)
+ }
+ sourceAudio := session.audioStream
+ profile := session.profile
+ profile.AudioBitrate = 128000
+ if err := session.UpdateProfile(profile); err != nil {
+ t.Fatal(err)
+ }
+ session.SetPaused(true)
+ session.SetPaused(false)
+ if session.audioStream != sourceAudio {
+ t.Fatal("profile/pause replaced exclusion")
+ }
+ for len(events) > 0 {
+ <-events
+ }
+ if err := sourceAudio.Close(); err != nil {
+ t.Fatal(err)
+ }
+ timer := time.NewTimer(time.Second)
+ defer timer.Stop()
+ waiting := true
+ for waiting {
+ select {
+ case event := <-events:
+ if event.Type != "audio-state" {
+ continue
+ }
+ if event.SourceAudio || !event.Microphone || event.Failed {
+ t.Fatalf("source loss was misreported: %+v", event)
+ }
+ waiting = false
+ case <-timer.C:
+ t.Fatal("source loss was not reported")
+ }
+ }
+ session.mixer.mu.Lock()
+ sameMicrophone := session.mixer.microphone == microphone
+ sourceEnded := session.mixer.source == nil
+ session.mixer.mu.Unlock()
+ if !sourceEnded || !sameMicrophone || session.audioSource != output || !session.HasAudio() {
+ t.Fatal("exclusion loss disturbed microphone or output ownership")
+ }
+}
+
func TestNativeMicrophonePreservesMediaOwnersAcrossInputChanges(t *testing.T) {
session, options, events := startMixedFixture(t)
check := func(err error) {
@@ -130,7 +207,7 @@ func TestNativeMicrophonePreservesMediaOwnersAcrossInputChanges(t *testing.T) {
}
microphone = replaced
options.Target.SourceID = "2"
- check(session.ReplaceSource(options, true))
+ check(session.ReplaceSource(options, true, nil))
session.mixer.mu.Lock()
sameMicrophone := session.mixer.microphone == microphone
session.mixer.mu.Unlock()
diff --git a/internal/app/nativehost/audio_test.go b/internal/app/nativehost/audio_test.go
index 0018fab..f2fc435 100644
--- a/internal/app/nativehost/audio_test.go
+++ b/internal/app/nativehost/audio_test.go
@@ -56,6 +56,17 @@ func runAudioRecoveryCapture() {
}
}
audio := microphone || (len(os.Args) > 1 && os.Args[1] == "--capture-audio")
+ if !audio && os.Getenv("PIIK_VIDEO_FAIL") == "1" {
+ os.Exit(2)
+ }
+ if !microphone && audio && os.Args[2] == "exclude" {
+ if os.Getenv("PIIK_EXCLUSION_FAIL") == "1" {
+ os.Exit(2)
+ }
+ if marker := os.Getenv("PIIK_EXCLUSION_STARTED"); marker != "" {
+ _ = os.WriteFile(marker, []byte(os.Args[3]+":"+os.Args[4]), 0600)
+ }
+ }
period := time.Second / 30
status := []byte(`{"state":"active","audio":true}`)
if audio {
@@ -179,7 +190,7 @@ func TestAudioEOFWaitsForExplicitSourceReplacement(t *testing.T) {
waitPackets(&videoPackets, videoPackets.Load()+10, "video after audio EOF")
before := audioPackets.Load()
options.Target.SourceID = "2"
- check(session.ReplaceSource(options, true))
+ check(session.ReplaceSource(options, true, nil))
if session.currentStream() == previousVideo || session.currentAudioStream() == previousAudio ||
session.edge("audio-edge") != edge || session.audioSource != audioSource {
t.Fatal("explicit replacement did not preserve the media owners")
diff --git a/internal/app/nativehost/session.go b/internal/app/nativehost/session.go
index 6d528d7..174b284 100644
--- a/internal/app/nativehost/session.go
+++ b/internal/app/nativehost/session.go
@@ -67,6 +67,7 @@ type Options struct {
Video nativecapture.VideoOptions
Profile QualityProfile
AudioEnabled bool
+ ExcludeAudio *nativecapture.CaptureTarget
MicrophoneMixing bool
EdgeCapacity int
BindAddress string
@@ -138,6 +139,7 @@ func Start(parent context.Context, options Options) (*Session, error) {
parent,
options.CaptureProcess,
options.Video.Target,
+ options.ExcludeAudio,
)
if audioErr != nil {
slog.DebugContext(parent, "piik-client", "event", "capture-audio-unavailable", "share", diagnostics.ID(options.ShareID), diagnostics.Error(audioErr))
@@ -401,6 +403,7 @@ func (session *Session) UpdateProfile(profile QualityProfile) error {
func (session *Session) ReplaceSource(
options nativecapture.VideoOptions,
audioEnabled bool,
+ excludeAudio *nativecapture.CaptureTarget,
) error {
session.updateMu.Lock()
defer session.updateMu.Unlock()
@@ -420,6 +423,21 @@ func (session *Session) ReplaceSource(
if session.mixer == nil && audioEnabled != hasAudio {
return errors.New("native source audio availability cannot change while sharing")
}
+ if excludeAudio != nil {
+ if session.mixer == nil {
+ return errors.New("native audio exclusion requires mixed audio")
+ }
+ // A privacy change retires the old source before preparation. Failure must
+ // leave it silent rather than retaining unfiltered audio; video/mic stay live.
+ session.mu.Lock()
+ previousAudio := session.audioStream
+ session.audioStream = nil
+ session.mu.Unlock()
+ session.mixer.setSource(nil)
+ if previousAudio != nil {
+ _ = previousAudio.Close()
+ }
+ }
options.Profile = profile.Video
options.RestoreToken = ""
replacement, state, err := session.prepareVideo(session.ctx, options, options.Target.Kind == "picker", false)
@@ -432,6 +450,7 @@ func (session *Session) ReplaceSource(
session.ctx,
session.captureProcess,
options.Target,
+ excludeAudio,
)
if err != nil {
_ = replacement.Close()
@@ -452,11 +471,15 @@ func startAudioCapture(
ctx context.Context,
captureProcess string,
target nativecapture.CaptureTarget,
+ excludeAudio *nativecapture.CaptureTarget,
) (*nativecapture.Stream, error) {
+ if excludeAudio != nil {
+ return nativecapture.StartAudio(ctx, captureProcess, *excludeAudio, true)
+ }
if target.Kind == "display" || target.Kind == "picker" {
return nativecapture.StartSystemAudio(ctx, captureProcess)
}
- return nativecapture.StartAudio(ctx, captureProcess, target)
+ return nativecapture.StartAudio(ctx, captureProcess, target, false)
}
func (session *Session) prepareVideo(
@@ -560,7 +583,7 @@ func (session *Session) installCapture(next *nativecapture.Stream) error {
// Preparation has consumed these events. Apply them to the new generation
// before acknowledging it, just as the live reader handles unavailable layers.
for _, layer := range unavailable {
- if err := session.source.DisableLayer(layer); err != nil {
+ if err := session.disableCaptureLayer(next, layer); err != nil {
return err
}
}
@@ -575,6 +598,22 @@ func (session *Session) installCapture(next *nativecapture.Stream) error {
return nil
}
+func (session *Session) disableCaptureLayer(stream *nativecapture.Stream, layer int) error {
+ session.mu.Lock()
+ if session.stream != stream || session.closed || session.ctx.Err() != nil {
+ session.mu.Unlock()
+ return nil
+ }
+ // Accept failure atomically with capture selection. Closing transports may
+ // wait for callbacks, so only the source-state change belongs under this lock.
+ retire, err := session.source.MarkLayerUnavailable(layer)
+ session.mu.Unlock()
+ if retire != nil {
+ retire()
+ }
+ return err
+}
+
func (session *Session) CloseEdge(connectionID string) {
session.mu.Lock()
edge := session.edges[connectionID]
@@ -831,7 +870,7 @@ func (session *Session) runVideo() error {
if session.source == nil {
return fail(errors.New("native output ended before its source state"))
}
- if err = session.source.DisableLayer(frame.Layer); err != nil {
+ if err = session.disableCaptureLayer(current, frame.Layer); err != nil {
return fail(err)
}
case nativecapture.FramePCM:
diff --git a/internal/app/nativehost/session_test.go b/internal/app/nativehost/session_test.go
index a581b55..8f8003c 100644
--- a/internal/app/nativehost/session_test.go
+++ b/internal/app/nativehost/session_test.go
@@ -11,6 +11,7 @@ import (
"slices"
"strconv"
"strings"
+ "sync"
"testing"
"time"
@@ -189,6 +190,122 @@ func TestCaptureCommitWaitsForReaderMetadataOrTermination(t *testing.T) {
}
}
+func TestCaptureFailureBelongsToSelectedStream(t *testing.T) {
+ for _, replace := range []bool{false, true} {
+ t.Run(strconv.FormatBool(replace), func(t *testing.T) {
+ check := func(err error) {
+ t.Helper()
+ if err != nil {
+ t.Fatal(err)
+ }
+ }
+ t.Setenv("PIIK_NATIVEHOST_PIPE_FIXTURE", "backend")
+ t.Setenv("PIIK_CAPTURE_STARTING", string(captureStatePayload(t,
+ `{"state":"starting","codec":"vp8","adapterIndex":0,"adapterName":"Fixture","adapterIdentity":"0:0","encoderName":"VP8","encoderIdentity":"vp8"}`)))
+ t.Setenv("PIIK_CAPTURE_ACTIVE", string(captureStatePayload(t, `{"state":"active","codec":"vp8","width":1280,"height":720,"fps":30}`)))
+ t.Setenv("PIIK_CAPTURE_UNAVAILABLE", "0")
+ executable, err := os.Executable()
+ check(err)
+ ctx, cancel := context.WithCancel(t.Context())
+ defer cancel()
+ options := nativecapture.VideoOptions{Target: nativecapture.CaptureTarget{Kind: "display", SourceID: "1", Title: "Fixture"}, Codec: "vp8",
+ Profile: nativecapture.VideoProfile{Width: 1280, Height: 720, Framerate: 30, Bitrate: 3_000_000, Preference: "balanced"}, OutputGroups: 4}
+ previous, err := nativecapture.StartVideo(ctx, executable, options)
+ check(err)
+ defer previous.Close()
+ t.Setenv("PIIK_CAPTURE_UNAVAILABLE", "")
+ replacement, err := nativecapture.StartVideo(ctx, executable, options)
+ check(err)
+ defer replacement.Close()
+ engine, err := mediaedge.NewEngine(mediaedge.EngineOptions{BindAddress: "127.0.0.1:0", IncludeLoopback: true})
+ check(err)
+ defer engine.Close()
+ source, err := engine.NewSource("vp8", 4, 2, nil)
+ check(err)
+ defer source.Close()
+ check(configureCaptureOutputs(source, previous.Outputs()))
+ publication, err := engine.NewPublication(source, mediaedge.EdgeOptions{ConnectionID: "capture-failure-owner"})
+ check(err)
+ active := make(chan struct{}, 1)
+ session := &Session{ctx: ctx, cancel: cancel, engine: engine, source: source, stream: previous,
+ videoOptions: options, ready: make(chan error, 1), events: func(_ context.Context, event Event) {
+ if event.State == "active" {
+ select {
+ case active <- struct{}{}:
+ default:
+ }
+ }
+ }}
+ // Delay the actual reader after its first identity check, without a
+ // production hook. Log I/O and ordinary scheduling can delay it here.
+ entered, release := make(chan struct{}), make(chan struct{})
+ unblock := sync.OnceFunc(func() { close(release) })
+ previousLog := slog.Default()
+ defer slog.SetDefault(previousLog)
+ slog.SetDefault(slog.New(slog.NewTextHandler(io.Discard, &slog.HandlerOptions{
+ Level: slog.LevelDebug, ReplaceAttr: func(_ []string, attr slog.Attr) slog.Attr {
+ if attr.Key == "event" && attr.Value.String() == "capture-output-unavailable" {
+ close(entered)
+ <-release
+ }
+ return attr
+ },
+ })))
+ readerDone := make(chan error, 1)
+ go func() { readerDone <- session.runVideo() }()
+ defer func() {
+ unblock()
+ cancel()
+ _ = previous.Close()
+ _ = replacement.Close()
+ select {
+ case <-readerDone:
+ case <-time.After(5 * time.Second):
+ t.Error("capture reader did not retire")
+ }
+ }()
+ select {
+ case <-entered:
+ case err = <-readerDone:
+ readerDone <- err
+ t.Fatalf("reader stopped before output failure: %v", err)
+ case <-time.After(5 * time.Second):
+ t.Fatal("old output failure did not reach the reader")
+ }
+ committed := make(chan error, 1)
+ if replace {
+ go func() {
+ committed <- session.commitCapture(options, QualityProfile{Video: options.Profile}, replacement, CaptureState{}, nil, false)
+ }()
+ deadline := time.Now().Add(5 * time.Second)
+ for session.currentStream() != replacement {
+ if time.Now().After(deadline) {
+ t.Fatal("replacement was not selected")
+ }
+ time.Sleep(time.Millisecond)
+ }
+ }
+ unblock()
+ select {
+ case <-active:
+ case <-time.After(5 * time.Second):
+ t.Fatal("capture reader did not finish the failure or handoff")
+ }
+ if replace {
+ select {
+ case err = <-committed:
+ check(err)
+ case <-time.After(5 * time.Second):
+ t.Fatal("replacement installation was not acknowledged")
+ }
+ }
+ if err = publication.SetActiveCount(1); (err == nil) != replace {
+ t.Fatalf("publication availability after replacement=%v: %v", replace, err)
+ }
+ })
+ }
+}
+
func TestCaptureProfileRetainsSelectedBackendFromStarting(t *testing.T) {
t.Setenv("PIIK_NATIVEHOST_PIPE_FIXTURE", "backend")
t.Setenv("PIIK_CAPTURE_STARTING", string(captureStatePayload(t,
diff --git a/internal/server/app/app.go b/internal/server/app/app.go
index acf53d1..f58605b 100644
--- a/internal/server/app/app.go
+++ b/internal/server/app/app.go
@@ -154,6 +154,7 @@ func New(options Options) (*Server, error) {
server.signalOptions = signal.Options{
Store: store,
EndpointMediaCopyCapacity: configuration.EndpointMediaCopyCapacity,
+ SfuOnly: configuration.SFU != nil && configuration.SFU.Only,
Ice: config.IceConfig(configuration),
// The TS signaling server read natPredictionEnabled off the same
// `ice` options object it forwarded; the Go IceConfig is already the
diff --git a/internal/server/app/app_test.go b/internal/server/app/app_test.go
index 8134f50..c0aee53 100644
--- a/internal/server/app/app_test.go
+++ b/internal/server/app/app_test.go
@@ -366,7 +366,17 @@ func TestCapabilitiesReportServicesDisabledByDefault(t *testing.T) {
server := start(t, Options{Config: testConfig(t)})
server.do(http.MethodGet, "/api/capabilities").
- expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":false}`)
+ expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":false,"roomInteractions":true,"hostRoomSession":true}`)
+
+}
+
+func TestCapabilitiesReportSFUOnlyPolicy(t *testing.T) {
+ configuration := testConfig(t)
+ configuration.SFU = &config.SFUConfig{ListenHost: "127.0.0.1", Port: 0, Only: true}
+ configuration.NATPredictionEnabled = true
+ server := start(t, Options{Config: configuration})
+ server.do(http.MethodGet, "/api/capabilities").
+ expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":true,"sfuOnly":true,"natPrediction":false,"roomInteractions":true,"hostRoomSession":true}`)
}
func TestCapabilitiesReportOptionalNATPredictionWithoutExposingConfiguration(t *testing.T) {
@@ -377,7 +387,7 @@ func TestCapabilitiesReportOptionalNATPredictionWithoutExposingConfiguration(t *
server := start(t, Options{Config: configuration})
server.do(http.MethodGet, "/api/capabilities").
- expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":true}`).
+ expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":true,"roomInteractions":true,"hostRoomSession":true}`).
expectHeader("Cache-Control", "no-store").
expectHeader("X-Content-Type-Options", "nosniff")
@@ -420,7 +430,8 @@ func TestSFUListenerFollowsApplicationCloseAndEnd(t *testing.T) {
t.Fatal("missing runtime owner")
}
server.do(http.MethodGet, "/api/capabilities").
- expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":true,"natPrediction":false}`)
+ expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":true,"natPrediction":false,"roomInteractions":true,"hostRoomSession":true}`)
+
var err error
if ending {
err = server.End(context.Background())
diff --git a/internal/server/app/routes.go b/internal/server/app/routes.go
index 7a35806..278f518 100644
--- a/internal/server/app/routes.go
+++ b/internal/server/app/routes.go
@@ -80,7 +80,10 @@ func (s *Server) route(
sendJSON(writer, http.StatusOK, protocol.RuntimeCapabilities{
ConnectionAttemptProgress4: true,
Sfu: s.config.SFU != nil,
- NatPrediction: s.config.NATPredictionEnabled,
+ SfuOnly: s.config.SFU != nil && s.config.SFU.Only,
+ NatPrediction: s.config.NATPredictionEnabled && !(s.config.SFU != nil && s.config.SFU.Only),
+ RoomInteractions: true,
+ HostRoomSession: true,
})
return
}
diff --git a/internal/server/config/config.go b/internal/server/config/config.go
index 481c0da..2a5bcd0 100644
--- a/internal/server/config/config.go
+++ b/internal/server/config/config.go
@@ -68,6 +68,7 @@ type SFUConfig struct {
ListenHost string
Port int
PublicIP string
+ Only bool
}
// Config describes runtime configuration. Zero values mean open site
@@ -273,8 +274,15 @@ func parseRoomDatabasePath(value string) (string, error) {
}
func parseSFU(env map[string]string) (*SFUConfig, error) {
+ only, err := parseBoolean(env["SFU_ONLY"], "SFU_ONLY")
+ if err != nil {
+ return nil, err
+ }
rawPort := strings.TrimSpace(env["SFU_UDP_PORT"])
if rawPort == "" {
+ if only {
+ return nil, errors.New("SFU_ONLY requires SFU_UDP_PORT")
+ }
return nil, nil
}
port, err := parseBoundedInteger(rawPort, 0, "SFU_UDP_PORT", 1, maxPort)
@@ -298,6 +306,7 @@ func parseSFU(env map[string]string) (*SFUConfig, error) {
ListenHost: listenHost,
Port: int(port),
PublicIP: publicIP,
+ Only: only,
}, nil
}
diff --git a/internal/server/config/config_test.go b/internal/server/config/config_test.go
index c6a5ed3..4d17164 100644
--- a/internal/server/config/config_test.go
+++ b/internal/server/config/config_test.go
@@ -187,6 +187,21 @@ func TestLoadSFUOptional(t *testing.T) {
}
}
+func TestSFUOnlyRequiresAnEnabledListener(t *testing.T) {
+ configuration := mustLoad(t, map[string]string{"SFU_UDP_PORT": "7882", "SFU_ONLY": "true"})
+ if !configuration.SFU.Only {
+ t.Fatal("SFU_ONLY was ignored")
+ }
+ for _, values := range []map[string]string{
+ {"SFU_ONLY": "true"},
+ {"SFU_ONLY": "yes", "SFU_UDP_PORT": "7882"},
+ } {
+ if _, err := Load(values); err == nil {
+ t.Fatalf("accepted invalid SFU policy: %v", values)
+ }
+ }
+}
+
func TestLoadAccepts(t *testing.T) {
developmentPath, err := filepath.Abs(filepath.Join("state", "rooms.sqlite"))
if err != nil {
diff --git a/internal/server/protocol/client.go b/internal/server/protocol/client.go
index 8bc3e54..ae318ab 100644
--- a/internal/server/protocol/client.go
+++ b/internal/server/protocol/client.go
@@ -26,6 +26,8 @@ type AuthenticateHostMessage struct {
ViewerPresence bool `json:"viewerPresence,omitempty"`
ConnectionAttemptProgress4 bool `json:"connectionAttemptProgress4,omitempty"`
DisplayName *DisplayName `json:"displayName,omitempty"`
+ RoomOnly bool `json:"roomOnly,omitempty"`
+ RoomSession bool `json:"roomSession,omitempty"`
}
// AuthenticateViewerMessage is the viewer member of authenticateMessageSchema.
@@ -172,6 +174,15 @@ type StopSharingMessage struct {
ShareGeneration string `json:"shareGeneration,omitempty"`
}
+// StartSharingMessage begins a new media publication on an existing Host room session.
+type StartSharingMessage struct {
+ Type string `json:"type"`
+ ShareGeneration string `json:"shareGeneration"`
+ SharingPaused bool `json:"sharingPaused,omitempty"`
+ QualitySettings *QualitySettings `json:"qualitySettings,omitempty"`
+ RoutePolicy RoutePolicy `json:"routePolicy"`
+}
+
// AbandonRoomMessage is { type: "abandon-room" }.
type AbandonRoomMessage struct {
Type string `json:"type"`
@@ -197,6 +208,7 @@ func (ResetSenderQualityMessage) isClientMessage() {}
func (SetDisplayNameMessage) isClientMessage() {}
func (SetSharingPausedMessage) isClientMessage() {}
func (StopSharingMessage) isClientMessage() {}
+func (StartSharingMessage) isClientMessage() {}
func (AbandonRoomMessage) isClientMessage() {}
// DecodeClientMessage parses JSON and validates the message schema.
@@ -207,6 +219,10 @@ func DecodeClientMessage(data []byte) (ClientMessage, error) {
return nil, err
}
switch messageType {
+ case "subscribe-room-interactions":
+ return decodeEmptyClientMessage(data, SubscribeRoomInteractionsMessage{Type: messageType})
+ case "send-room-interaction":
+ return decodeSendRoomInteraction(data)
case "authenticate":
return decodeAuthenticate(data)
case "signaling-challenge":
@@ -247,6 +263,8 @@ func DecodeClientMessage(data []byte) (ClientMessage, error) {
return decodeSetSharingPaused(data)
case "stop-sharing":
return decodeStopSharing(data)
+ case "start-sharing":
+ return decodeStartSharing(data)
case "abandon-room":
return decodeEmptyClientMessage(data, AbandonRoomMessage{Type: messageType})
}
@@ -282,8 +300,9 @@ func decodeAuthenticate(data []byte) (ClientMessage, error) {
if err := present.require("type", "protocol", "roomId", "role", "token", "clientId"); err != nil {
return nil, err
}
- if err := present.optional("shareGeneration", "sharingPaused", "qualitySettings",
+ if err := present.optional("shareGeneration", "roomOnly", "roomSession", "sharingPaused", "qualitySettings",
"routePolicy", "viewerPresence", "displayName", "connectionAttemptProgress4"); err != nil {
+
return nil, err
}
if message.Protocol != SignalingProtocol {
@@ -301,6 +320,15 @@ func decodeAuthenticate(data []byte) (ClientMessage, error) {
if present.has("shareGeneration") && !ValidOpaqueID(message.ShareGeneration) {
return nil, errors.New("shareGeneration is not an opaque id")
}
+ if present.has("roomOnly") && !message.RoomOnly {
+ return nil, errors.New("roomOnly must be true when present")
+ }
+ if present.has("roomSession") && !message.RoomSession {
+ return nil, errors.New("roomSession must be true when present")
+ }
+ if message.RoomOnly && !message.RoomSession {
+ return nil, errors.New("roomOnly requires roomSession")
+ }
if present.has("viewerPresence") && !message.ViewerPresence {
return nil, errors.New("viewerPresence must be true when present")
}
@@ -705,3 +733,21 @@ func decodeStopSharing(data []byte) (ClientMessage, error) {
}
return message, nil
}
+
+func decodeStartSharing(data []byte) (ClientMessage, error) {
+ var message StartSharingMessage
+ present, err := decodeObject(data, &message)
+ if err != nil {
+ return nil, err
+ }
+ if err := present.require("type", "shareGeneration", "routePolicy"); err != nil {
+ return nil, err
+ }
+ if err := present.optional("sharingPaused", "qualitySettings"); err != nil {
+ return nil, err
+ }
+ if !ValidOpaqueID(message.ShareGeneration) {
+ return nil, errors.New("shareGeneration is not an opaque id")
+ }
+ return message, nil
+}
diff --git a/internal/server/protocol/contract_test.go b/internal/server/protocol/contract_test.go
index 4931b9e..a3d74d3 100644
--- a/internal/server/protocol/contract_test.go
+++ b/internal/server/protocol/contract_test.go
@@ -16,6 +16,7 @@ func TestRuntimeCapabilities(t *testing.T) {
{`{}`, RuntimeCapabilities{}},
{`{"connectionAttemptProgress4":true}`, RuntimeCapabilities{ConnectionAttemptProgress4: true}},
{`{"sfu":true,"extra":{"enabled":true}}`, RuntimeCapabilities{Sfu: true}},
+ {`{"sfu":true,"sfuOnly":true}`, RuntimeCapabilities{Sfu: true, SfuOnly: true}},
{`{"natPrediction":true,"SFU":true}`, RuntimeCapabilities{NatPrediction: true}},
{`{"sfu":true,"natPrediction":true}`, RuntimeCapabilities{Sfu: true, NatPrediction: true}},
{`{"sfu":"ignored duplicate","sfu":true}`, RuntimeCapabilities{Sfu: true}},
@@ -26,7 +27,7 @@ func TestRuntimeCapabilities(t *testing.T) {
}
}
for _, data := range []string{
- `null`, `[]`, `true`, `{"sfu":null}`, `{"sfu":"true"}`,
+ `null`, `[]`, `true`, `{"sfu":null}`, `{"sfu":"true"}`, `{"sfuOnly":null}`, `{"sfuOnly":"true"}`,
`{"natPrediction":null}`, `{"natPrediction":1}`,
`{"connectionAttemptProgress4":null}`, `{"connectionAttemptProgress4":4}`,
} {
@@ -698,6 +699,44 @@ func TestAuthenticateAppliesTheRoutePolicyDefault(t *testing.T) {
}
}
+func TestAuthenticateAcceptsPersistentRoomSessionOptIn(t *testing.T) {
+ message, err := DecodeClientMessage([]byte(
+ `{"type":"authenticate","protocol":"piik-v23","roomId":"1234","role":"host",` +
+ `"token":"` + repeat("a", 43) + `","clientId":"client_12345678","roomSession":true,"roomOnly":true}`))
+ if err != nil {
+ t.Fatal(err)
+ }
+ host, ok := message.(AuthenticateHostMessage)
+ if !ok || !host.RoomSession || !host.RoomOnly {
+ t.Fatalf("decoded %#v, want room-session opt-in and room-only state", message)
+ }
+ if _, err := DecodeClientMessage([]byte(
+ `{"type":"authenticate","protocol":"piik-v23","roomId":"1234","role":"host",` +
+ `"token":"` + repeat("a", 43) + `","clientId":"client_12345678","roomOnly":true}`)); err == nil {
+ t.Fatal("accepted roomOnly without roomSession opt-in")
+ }
+}
+
+func TestStartSharingRequiresAnOpaqueGenerationAndRoutePolicy(t *testing.T) {
+ message, err := DecodeClientMessage([]byte(
+ `{"type":"start-sharing","shareGeneration":"share_12345678",` +
+ `"routePolicy":{"peerOnly":true,"topologyOptimization":false,"natPrediction":false}}`))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, ok := message.(StartSharingMessage); !ok {
+ t.Fatalf("decoded %#v, want StartSharingMessage", message)
+ }
+ for _, raw := range []string{
+ `{"type":"start-sharing","shareGeneration":"short","routePolicy":{"peerOnly":true,"topologyOptimization":false,"natPrediction":false}}`,
+ `{"type":"start-sharing","shareGeneration":"share_12345678"}`,
+ } {
+ if _, err := DecodeClientMessage([]byte(raw)); err == nil {
+ t.Errorf("accepted invalid start-sharing message %s", raw)
+ }
+ }
+}
+
// The sealed unions accept nothing outside their own message structs.
func TestDecodeRejectsUnknownEnvelopes(t *testing.T) {
for _, raw := range []string{`{}`, `[]`, `null`, `"authenticate"`, `{"type":42}`,
diff --git a/internal/server/protocol/fixture_test.go b/internal/server/protocol/fixture_test.go
index f02d054..7171630 100644
--- a/internal/server/protocol/fixture_test.go
+++ b/internal/server/protocol/fixture_test.go
@@ -55,6 +55,8 @@ func TestFixtureConstants(t *testing.T) {
"viewerQualityEvidenceExpiryMs": float64(ViewerQualityEvidenceExpiryMs),
"persistentNativeEdgeDegradedWindows": float64(PersistentNativeEdgeDegradedWindows),
"maxDisplayNameCodePoints": float64(MaxDisplayNameCodePoints),
+ "maxChatCodePoints": float64(MaxChatCodePoints),
+ "interactionIntervalMs": float64(InteractionIntervalMs),
"defaultViewerDisplayName": DefaultViewerDisplayName,
"defaultHostDisplayNamePrefix": DefaultHostDisplayNamePrefix,
"minViewerPasswordLength": float64(MinViewerPasswordLength),
diff --git a/internal/server/protocol/interactions.go b/internal/server/protocol/interactions.go
new file mode 100644
index 0000000..51f987f
--- /dev/null
+++ b/internal/server/protocol/interactions.go
@@ -0,0 +1,194 @@
+package protocol
+
+import (
+ "errors"
+ "strings"
+ "unicode/utf8"
+
+ "golang.org/x/text/unicode/norm"
+)
+
+const MaxChatCodePoints = 280
+const InteractionIntervalMs = 800
+
+type InteractionPayload struct {
+ Kind string `json:"kind"`
+ Text string `json:"text,omitempty"`
+ Reaction string `json:"reaction,omitempty"`
+ TargetPeerID string `json:"targetPeerId,omitempty"`
+}
+
+func (payload *InteractionPayload) UnmarshalJSON(data []byte) error {
+ type plain InteractionPayload
+ var value plain
+ fields, err := decodeObject(data, &value)
+ if err != nil {
+ return err
+ }
+ if err = fields.require("kind"); err != nil {
+ return err
+ }
+ switch value.Kind {
+ case "chat":
+ if err = fields.require("text"); err != nil {
+ return err
+ }
+ if fields.has("reaction") || fields.has("targetPeerId") ||
+ hasUnpairedSurrogateEscape(fields["text"]) || !validChatText(value.Text) {
+ return errors.New("invalid chat payload")
+ }
+ case "reaction":
+ if err = fields.require("reaction"); err != nil {
+ return err
+ }
+ if err = fields.optional("targetPeerId"); err != nil {
+ return err
+ }
+ if fields.has("text") || (fields.has("targetPeerId") && !ValidOpaqueID(value.TargetPeerID)) {
+ return errors.New("invalid reaction payload")
+ }
+ switch value.Reaction {
+ case "tomato", "poop":
+ if !ValidOpaqueID(value.TargetPeerID) {
+ return errors.New("throwing a prop requires a participant")
+ }
+ case "wave", "heart", "clap", "laugh", "wow", "party", "fire", "eyes", "star", "sleep":
+ default:
+ return errors.New("unknown reaction")
+ }
+ default:
+ return errors.New("unknown interaction kind")
+ }
+ *payload = InteractionPayload(value)
+ return nil
+}
+
+func validChatText(value string) bool {
+ return utf8.ValidString(value) && utf8.RuneCountInString(value) >= 1 &&
+ utf8.RuneCountInString(value) <= MaxChatCodePoints && !forbiddenDisplayNameCharacters.MatchString(value) &&
+ norm.NFC.IsNormalString(value) && strings.TrimFunc(value, IsJSWhitespace) == value
+}
+
+type SubscribeRoomInteractionsMessage struct {
+ Type string `json:"type"`
+}
+type SendRoomInteractionMessage struct {
+ Type string `json:"type"`
+ RequestID string `json:"requestId"`
+ Payload InteractionPayload `json:"payload"`
+}
+
+func (SubscribeRoomInteractionsMessage) isClientMessage() {}
+func (SendRoomInteractionMessage) isClientMessage() {}
+
+func decodeSendRoomInteraction(data []byte) (ClientMessage, error) {
+ var message SendRoomInteractionMessage
+ present, err := decodeObject(data, &message)
+ if err != nil {
+ return nil, err
+ }
+ if err = present.require("type", "requestId", "payload"); err != nil {
+ return nil, err
+ }
+ if !ValidOpaqueID(message.RequestID) {
+ return nil, errors.New("invalid interaction requestId")
+ }
+ return message, nil
+}
+
+type InteractionSender struct {
+ PeerID string `json:"peerId"`
+ Role Role `json:"role"`
+ DisplayName DisplayName `json:"displayName"`
+}
+
+func (sender *InteractionSender) UnmarshalJSON(data []byte) error {
+ type plain InteractionSender
+ var value plain
+ present, err := decodeObject(data, &value)
+ if err != nil {
+ return err
+ }
+ if err = present.require("peerId", "role", "displayName"); err != nil {
+ return err
+ }
+ if !ValidOpaqueID(value.PeerID) {
+ return errors.New("invalid sender peerId")
+ }
+ *sender = InteractionSender(value)
+ return nil
+}
+
+type RoomInteractionsReadyMessage struct {
+ Type string `json:"type"`
+ ServerTime Int `json:"serverTime"`
+}
+type RoomInteractionMessage struct {
+ Type string `json:"type"`
+ ID string `json:"id"`
+ RequestID string `json:"requestId"`
+ OccurredAt Int `json:"occurredAt"`
+ Sender InteractionSender `json:"sender"`
+ Payload InteractionPayload `json:"payload"`
+}
+type RoomInteractionRejectedMessage struct {
+ Type string `json:"type"`
+ RequestID string `json:"requestId"`
+ Reason string `json:"reason"`
+}
+
+func (RoomInteractionsReadyMessage) isServerMessage() {}
+func (RoomInteractionMessage) isServerMessage() {}
+func (RoomInteractionRejectedMessage) isServerMessage() {}
+
+func decodeRoomInteractionsReady(data []byte) (ServerMessage, error) {
+ var message RoomInteractionsReadyMessage
+ present, err := decodeObject(data, &message)
+ if err != nil {
+ return nil, err
+ }
+ if err = present.require("type", "serverTime"); err != nil {
+ return nil, err
+ }
+ if !inRangeInt(message.ServerTime, 0, MaxSafeInteger) {
+ return nil, errors.New("invalid interaction server time")
+ }
+ return message, nil
+}
+
+func decodeRoomInteraction(data []byte) (ServerMessage, error) {
+ var message RoomInteractionMessage
+ present, err := decodeObject(data, &message)
+ if err != nil {
+ return nil, err
+ }
+ if err = present.require("type", "id", "requestId", "occurredAt", "sender", "payload"); err != nil {
+ return nil, err
+ }
+ if !ValidOpaqueID(message.ID) || !ValidOpaqueID(message.RequestID) {
+ return nil, errors.New("invalid interaction identity")
+ }
+ if !inRangeInt(message.OccurredAt, 0, MaxSafeInteger) {
+ return nil, errors.New("invalid interaction time")
+ }
+ return message, nil
+}
+func decodeRoomInteractionRejected(data []byte) (ServerMessage, error) {
+ var message RoomInteractionRejectedMessage
+ present, err := decodeObject(data, &message)
+ if err != nil {
+ return nil, err
+ }
+ if err = present.require("type", "requestId", "reason"); err != nil {
+ return nil, err
+ }
+ if !ValidOpaqueID(message.RequestID) {
+ return nil, errors.New("invalid interaction requestId")
+ }
+ switch message.Reason {
+ case "rate-limited", "target-unavailable", "not-subscribed", "busy":
+ default:
+ return nil, errors.New("invalid rejection reason")
+ }
+ return message, nil
+}
diff --git a/internal/server/protocol/server.go b/internal/server/protocol/server.go
index f25aa0a..3e1f9d1 100644
--- a/internal/server/protocol/server.go
+++ b/internal/server/protocol/server.go
@@ -57,6 +57,23 @@ type AuthenticatedViewerMessage struct {
Role Role `json:"role"`
}
+// SharingStartedMessage acknowledges a new media publication on an existing Host session.
+type SharingStartedMessage struct {
+ Type string `json:"type"`
+ ShareGeneration string `json:"shareGeneration"`
+ RoutePolicy RoutePolicy `json:"routePolicy"`
+ RouteRevision Int `json:"routeRevision"`
+ RouteAssignment ParticipantRouteAssignment `json:"routeAssignment"`
+ QualitySettings QualitySettings `json:"qualitySettings"`
+ Paused bool `json:"paused"`
+}
+
+type SharingStartFailedMessage struct {
+ Type string `json:"type"`
+ ShareGeneration string `json:"shareGeneration"`
+ Code string `json:"code"`
+}
+
// SignalingChallengeResponseMessage is the challenge echo.
type SignalingChallengeResponseMessage struct {
Type string `json:"type"`
@@ -197,6 +214,8 @@ type ErrorMessage struct {
func (AuthenticatedHostMessage) isServerMessage() {}
func (AuthenticatedViewerMessage) isServerMessage() {}
+func (SharingStartedMessage) isServerMessage() {}
+func (SharingStartFailedMessage) isServerMessage() {}
func (SignalingChallengeResponseMessage) isServerMessage() {}
func (ServerSignalMessage) isServerMessage() {}
func (ServerRestartRequestMessage) isServerMessage() {}
@@ -230,8 +249,18 @@ func DecodeServerMessage(data []byte) (ServerMessage, error) {
return nil, err
}
switch messageType {
+ case "room-interactions-ready":
+ return decodeRoomInteractionsReady(data)
+ case "room-interaction":
+ return decodeRoomInteraction(data)
+ case "room-interaction-rejected":
+ return decodeRoomInteractionRejected(data)
case "authenticated":
return decodeAuthenticated(data)
+ case "sharing-started":
+ return decodeSharingStarted(data)
+ case "sharing-start-failed":
+ return decodeSharingStartFailed(data)
case "signaling-challenge-response":
return decodeChallengeResponse(data)
case "signal":
@@ -386,6 +415,40 @@ func decodeAuthenticated(data []byte) (ServerMessage, error) {
return nil, fmt.Errorf("unknown authenticated role %q", role)
}
+func decodeSharingStarted(data []byte) (ServerMessage, error) {
+ var message SharingStartedMessage
+ present, err := decodeObject(data, &message)
+ if err != nil {
+ return nil, err
+ }
+ if err := present.require("type", "shareGeneration", "routePolicy", "routeRevision",
+ "routeAssignment", "qualitySettings", "paused"); err != nil {
+ return nil, err
+ }
+ if !ValidOpaqueID(message.ShareGeneration) {
+ return nil, errors.New("shareGeneration is not an opaque id")
+ }
+ if !validRevision(message.RouteRevision) {
+ return nil, errors.New("routeRevision is out of range")
+ }
+ return message, nil
+}
+
+func decodeSharingStartFailed(data []byte) (ServerMessage, error) {
+ var message SharingStartFailedMessage
+ present, err := decodeObject(data, &message)
+ if err != nil {
+ return nil, err
+ }
+ if err := present.require("type", "shareGeneration", "code"); err != nil {
+ return nil, err
+ }
+ if !ValidOpaqueID(message.ShareGeneration) || (message.Code != "FORBIDDEN" && message.Code != "SERVER_ERROR") {
+ return nil, errors.New("invalid sharing-start-failed response")
+ }
+ return message, nil
+}
+
func decodeChallengeResponse(data []byte) (ServerMessage, error) {
var message SignalingChallengeResponseMessage
present, err := decodeObject(data, &message)
diff --git a/internal/server/protocol/types.go b/internal/server/protocol/types.go
index 4365aed..4bfd186 100644
--- a/internal/server/protocol/types.go
+++ b/internal/server/protocol/types.go
@@ -170,7 +170,10 @@ func (v *RoutePolicy) UnmarshalJSON(data []byte) error {
type RuntimeCapabilities struct {
ConnectionAttemptProgress4 bool `json:"connectionAttemptProgress4,omitempty"`
Sfu bool `json:"sfu"`
+ SfuOnly bool `json:"sfuOnly,omitempty"`
NatPrediction bool `json:"natPrediction"`
+ RoomInteractions bool `json:"roomInteractions,omitempty"`
+ HostRoomSession bool `json:"hostRoomSession,omitempty"`
}
// UnmarshalJSON implements json.Unmarshaler.
@@ -183,12 +186,14 @@ func (v *RuntimeCapabilities) UnmarshalJSON(data []byte) error {
if present == nil {
return errors.New("runtime capabilities must be an object")
}
- if err := present.optional("sfu", "natPrediction", "connectionAttemptProgress4"); err != nil {
+ if err := present.optional("sfu", "sfuOnly", "roomInteractions", "hostRoomSession", "natPrediction", "connectionAttemptProgress4"); err != nil {
+
return err
}
for key, target := range map[string]*bool{
- "sfu": &v.Sfu, "natPrediction": &v.NatPrediction,
+ "sfu": &v.Sfu, "sfuOnly": &v.SfuOnly, "natPrediction": &v.NatPrediction,
"connectionAttemptProgress4": &v.ConnectionAttemptProgress4,
+ "roomInteractions": &v.RoomInteractions, "hostRoomSession": &v.HostRoomSession,
} {
if value, ok := present[key]; ok {
if err := json.Unmarshal(value, target); err != nil {
diff --git a/internal/server/route/api.go b/internal/server/route/api.go
index eaf052e..9c41d2d 100644
--- a/internal/server/route/api.go
+++ b/internal/server/route/api.go
@@ -455,6 +455,7 @@ type Options struct {
EndpointMediaCopyCapacity int
OperationTimeoutMs int64
SfuEnabled bool
+ SfuOnly bool
QualityConvergenceEnabled bool
NatPredictionEnabled bool
}
@@ -470,6 +471,7 @@ type Controller struct {
endpointMediaCopyCapacity int
operationTimeoutMs int64
sfuEnabled bool
+ sfuOnly bool
qualityConvergenceEnabled bool
natPredictionEnabled bool
diff --git a/internal/server/route/candidates.go b/internal/server/route/candidates.go
index 6b1cd35..931e3d8 100644
--- a/internal/server/route/candidates.go
+++ b/internal/server/route/candidates.go
@@ -70,7 +70,7 @@ func (c *Controller) buildCandidatePlans(childPeerID string, sfuOnly, preferUntr
}
var candidates []CandidateTuple
switch {
- case sfuOnly:
+ case sfuOnly || c.sfuOnly:
if sfuCandidate != nil {
candidates = []CandidateTuple{*sfuCandidate}
}
@@ -166,6 +166,9 @@ func (c *Controller) candidateValid(childPeerID string, plan CandidatePlan, att
return false
}
if tuple.Kind == UpstreamPeer {
+ if c.sfuOnly {
+ return false
+ }
parent, _ := c.participants.Get(tuple.ParentPeerID)
if parent == nil || parent.sessionID == "" || parent.departureConfirmed || (att != nil && parent.sessionID != att.parentSessionID) ||
slices.Contains(c.descendantsOf(childPeerID), parent.peerID) || !c.sourceUsable(parent.peerID) ||
diff --git a/internal/server/route/controller.go b/internal/server/route/controller.go
index bde5567..6463d73 100644
--- a/internal/server/route/controller.go
+++ b/internal/server/route/controller.go
@@ -271,6 +271,9 @@ func New(options Options) *Controller {
if !isSafeInteger(options.OperationTimeoutMs) || options.OperationTimeoutMs <= 0 {
panic(errors.New("Route operation timeout must be a positive integer"))
}
+ if options.SfuOnly && !options.SfuEnabled {
+ panic(errors.New("SFU-only routing requires SFU availability"))
+ }
c := &Controller{
hostPeerID: options.HostPeerID,
debugRoomID: options.DebugRoomID,
@@ -278,8 +281,9 @@ func New(options Options) *Controller {
endpointMediaCopyCapacity: options.EndpointMediaCopyCapacity,
operationTimeoutMs: options.OperationTimeoutMs,
sfuEnabled: options.SfuEnabled,
- qualityConvergenceEnabled: options.QualityConvergenceEnabled,
- natPredictionEnabled: options.NatPredictionEnabled,
+ sfuOnly: options.SfuOnly,
+ qualityConvergenceEnabled: options.QualityConvergenceEnabled && !options.SfuOnly,
+ natPredictionEnabled: options.NatPredictionEnabled && !options.SfuOnly,
routeTimings: make(map[string]*routeTimingRecord),
qualityObservations: make(map[string]*routeQualityObservation),
senderQualityObservations: make(map[string]*senderQualityObservation),
@@ -291,6 +295,7 @@ func New(options Options) *Controller {
"endpointCapacity", options.EndpointMediaCopyCapacity,
"operationTimeoutMs", options.OperationTimeoutMs,
"sfuEnabled", options.SfuEnabled,
+ "sfuOnly", options.SfuOnly,
"qualityConvergenceEnabled", options.QualityConvergenceEnabled)
return c
}
diff --git a/internal/server/route/sfu_only_test.go b/internal/server/route/sfu_only_test.go
new file mode 100644
index 0000000..aafe397
--- /dev/null
+++ b/internal/server/route/sfu_only_test.go
@@ -0,0 +1,64 @@
+package route
+
+import "testing"
+
+func TestSFUOnlyUsesOnePublicationWithoutPeerConvergence(t *testing.T) {
+ routes := newController(1, Options{SfuEnabled: true, SfuOnly: true,
+ QualityConvergenceEnabled: true, NatPredictionEnabled: true})
+ for index, viewer := range []string{A, B, C} {
+ now := int64(index * 10)
+ addViewer(routes, viewer, 1, &now)
+ op := must(t, routes.Reconcile(now).Operation)
+ eq(t, len(op.Candidates), 1)
+ publication := PublicationReuse
+ input := BeginInput{NowMs: now + 1, ConnectionID: viewer + "_sfu",
+ Reservation: sfuReuse(viewer)}
+ if index == 0 {
+ publication = PublicationCreate
+ input.PublicationGeneration = "publication"
+ input.PublicationConnectionID = "publication_connection"
+ input.Reservation = sfuCreate(viewer, "publication_resource")
+ }
+ eq(t, op.Candidates[0].Tuple, sfuTuple(publication))
+ prepared := beginOperation(t, routes, input)
+ // Availability still commits only on the exact child's media-ready proof.
+ eq(t, hasEdge(routes, viewer), false)
+ eq(t, routes.CandidateReady(guardFor(viewer, prepared.Current.Revision, input.ConnectionID),
+ now+2, nil, CandidateProof{}).Accepted, true)
+ eq(t, edgeOf(t, routes, viewer).Kind, UpstreamSfu)
+ noOperation(t, routes.Reconcile(now+3).Operation)
+ }
+ eq(t, routes.physicalCopies(HOST), 1)
+ eq(t, routes.directContinuations.Len(), 0)
+ // A failed publication is replaced through the same bounded operation.
+ publication := routes.Publication()
+ eq(t, routes.InvalidateHostPublication(PublicationGuard{HostSessionID: "host_session",
+ RouteRevision: routes.Revision(), Generation: publication.Generation,
+ ConnectionID: publication.ConnectionID}, ms(40)), true)
+ op := must(t, routes.Reconcile(41).Operation)
+ eq(t, len(op.Candidates), 1)
+ eq(t, op.Candidates[0].Tuple, sfuTuple(PublicationReplace))
+}
+
+func TestSFUOnlyFailureDoesNotEscapeToPeers(t *testing.T) {
+ for _, admissionDenied := range []bool{false, true} {
+ routes := newController(2, Options{SfuEnabled: true, SfuOnly: true})
+ addViewer(routes, A, 2, nil)
+ op := must(t, routes.Reconcile(0).Operation)
+ if admissionDenied {
+ routes.SkipCurrentCandidate(cursorGuard(op), 1, RejectionSfuAdmission)
+ } else {
+ prepared := beginOperation(t, routes, BeginInput{NowMs: 1,
+ ConnectionID: "sfu_connection", PublicationGeneration: "publication",
+ PublicationConnectionID: "publication_connection", Reservation: sfuCreate("edge", "publication")})
+ routes.CandidateFailed(guardFor(A, prepared.Current.Revision, "sfu_connection"), 2)
+ }
+ noOperation(t, routes.Reconcile(3).Operation)
+ eq(t, hasEdge(routes, A), false)
+ // Resource release may retry SFU; it must not reopen a Peer opportunity.
+ routes.TouchExternalFacts()
+ retry := must(t, routes.Reconcile(4).Operation)
+ eq(t, len(retry.Candidates), 1)
+ eq(t, retry.Candidates[0].Tuple.Kind, UpstreamSfu)
+ }
+}
diff --git a/internal/server/signal/interactions.go b/internal/server/signal/interactions.go
new file mode 100644
index 0000000..f3aac14
--- /dev/null
+++ b/internal/server/signal/interactions.go
@@ -0,0 +1,62 @@
+package signal
+
+import "github.com/TNTcraftHIM/Piik/internal/server/protocol"
+
+// Room authority remains in the authenticated session/store. Interaction state
+// ends with that connection; no message history, routing state or retry worker
+// lives on the server. Server.mu serializes membership checks and broadcast.
+func (s *Server) handleRoomInteraction(sess *session, actor *authenticatedSession, message protocol.SendRoomInteractionMessage) {
+ reject := func(reason string) {
+ s.send(sess, protocol.RoomInteractionRejectedMessage{
+ Type: "room-interaction-rejected", RequestID: message.RequestID, Reason: reason,
+ })
+ }
+ if !sess.roomInteractions || actor.displayName == nil {
+ reject("not-subscribed")
+ return
+ }
+ now := s.now()
+ if now < sess.nextInteractionAtMs {
+ reject("rate-limited")
+ return
+ }
+ sess.nextInteractionAtMs = now + protocol.InteractionIntervalMs
+ if targetID := message.Payload.TargetPeerID; targetID != "" {
+ target, exists := s.connectedPeer(actor.roomID, targetID)
+ targetSession := s.sessionsByID[target.SessionID]
+ if !exists || targetSession == nil || targetSession.revoked || !targetSession.open() || !s.isCurrentSession(targetSession) {
+ reject("target-unavailable")
+ return
+ }
+ }
+ // Reserve the existing signaling queue for media/authority work when a
+ // reader stalls. Room data must not fill it and terminate a healthy share.
+ const maxInteractionQueueBytes = 16 * 1024
+ if sess.queuedBytes >= maxInteractionQueueBytes {
+ reject("busy")
+ return
+ }
+ event := protocol.RoomInteractionMessage{
+ Type: "room-interaction", ID: opaqueID(), RequestID: message.RequestID,
+ OccurredAt: protocol.Int(now),
+ Sender: protocol.InteractionSender{PeerID: actor.peerID, Role: actor.role, DisplayName: protocol.DisplayName(*actor.displayName)},
+ Payload: message.Payload,
+ }
+ encoded, err := protocol.EncodeServerMessage(event)
+ if err != nil {
+ panic(err)
+ }
+ deliver := func(sessionID string) {
+ target := s.sessionsByID[sessionID]
+ if target != nil && target.roomInteractions && !target.revoked && target.open() &&
+ s.isCurrentSession(target) && target.queuedBytes < maxInteractionQueueBytes {
+ s.sendEncoded(target, encoded)
+ }
+ }
+ if host, ok := s.store.GetConnectedHost(actor.roomID); ok {
+ deliver(host.SessionID)
+ }
+ for _, viewer := range s.store.GetConnectedViewers(actor.roomID) {
+ deliver(viewer.SessionID)
+ }
+}
diff --git a/internal/server/signal/interactions_test.go b/internal/server/signal/interactions_test.go
new file mode 100644
index 0000000..48832f9
--- /dev/null
+++ b/internal/server/signal/interactions_test.go
@@ -0,0 +1,144 @@
+package signal
+
+import (
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "github.com/TNTcraftHIM/Piik/internal/server/protocol"
+)
+
+func subscribeInteractions(t *testing.T, client *testClient) {
+ t.Helper()
+ client.sendJSON(map[string]any{"type": "subscribe-room-interactions"})
+ ready := client.next("room-interactions-ready")
+ if _, err := protocol.DecodeServerMessage(ready.raw); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func TestRoomInteractionsStayInsideAuthorizedOptedInMembership(t *testing.T) {
+ var now atomic.Int64
+ now.Store(100_000)
+ h := startHarness(t, harnessOptions{maxViewersPerRoom: 4, now: now.Load})
+ host, viewer, old := openClient(t, h), openClient(t, h), openClient(t, h)
+ authenticate(t, host, h.room, protocol.RoleHost, "host_interaction", -1, "", presenceOptions{displayName: "房主", viewerPresence: true})
+ actor := authenticate(t, viewer, h.room, protocol.RoleViewer, "viewer_interaction", -1, "", presenceOptions{displayName: "小明", viewerPresence: true})
+ authenticate(t, old, h.room, protocol.RoleViewer, "old_page_interaction", -1, "", presenceOptions{})
+ otherRoom, err := h.server.CreateRoom(protocol.CodeEntryOpen, nil, "")
+ if err != nil {
+ t.Fatal(err)
+ }
+ outsider := openClient(t, h)
+ other := authenticate(t, outsider, otherRoom, protocol.RoleViewer, "outsider_interaction", -1, "", presenceOptions{displayName: "Elsewhere", viewerPresence: true})
+ for _, client := range []*testClient{host, viewer, outsider} {
+ subscribeInteractions(t, client)
+ }
+ viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_1", "payload": map[string]any{"kind": "chat", "text": "你好 👋"}})
+ hostEvent := host.next("room-interaction")
+ viewerEvent := viewer.next("room-interaction")
+ if string(hostEvent.raw) != string(viewerEvent.raw) {
+ t.Fatal("recipients received different events")
+ }
+ decoded, err := protocol.DecodeServerMessage(hostEvent.raw)
+ if err != nil {
+ t.Fatal(err)
+ }
+ event := decoded.(protocol.RoomInteractionMessage)
+ if event.OccurredAt != protocol.Int(now.Load()) {
+ t.Fatal("event time was not stamped at server acceptance")
+ }
+ if event.Sender.PeerID != actor.PeerID || event.Sender.DisplayName != "小明" {
+ t.Fatal("sender identity was not derived from authentication")
+ }
+ for _, client := range []*testClient{old, outsider} {
+ if _, ok := client.tryNext("room-interaction", 30*time.Millisecond); ok {
+ t.Fatal("interaction escaped its negotiated room boundary")
+ }
+ }
+ viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_2", "payload": map[string]any{"kind": "reaction", "reaction": "wave"}})
+ expectMatch(t, viewer.next("room-interaction-rejected").raw, `{"reason":"rate-limited"}`)
+ now.Add(protocol.InteractionIntervalMs)
+ viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_3", "payload": map[string]any{"kind": "reaction", "reaction": "tomato", "targetPeerId": other.PeerID}})
+ expectMatch(t, viewer.next("room-interaction-rejected").raw, `{"reason":"target-unavailable"}`)
+ // Room data and its rejections never terminate signaling or require media.
+ viewer.sendJSON(map[string]any{"type": "signaling-challenge", "sequence": 123})
+ viewer.next("signaling-challenge-response")
+}
+
+func TestRoomInteractionRequiresAuthenticationAndSubscription(t *testing.T) {
+ h := startHarness(t, harnessOptions{})
+ guest := openClient(t, h)
+ guest.sendJSON(map[string]any{"type": "subscribe-room-interactions"})
+ expectMatch(t, guest.next("error").raw, `{"code":"AUTH_REQUIRED"}`)
+ viewer := openClient(t, h)
+ authenticate(t, viewer, h.room, protocol.RoleViewer, "unsubscribed_viewer", -1, "", presenceOptions{displayName: "Friend", viewerPresence: true})
+ viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_1", "payload": map[string]any{"kind": "chat", "text": "hello"}})
+ expectMatch(t, viewer.next("room-interaction-rejected").raw, `{"reason":"not-subscribed"}`)
+}
+
+func TestViewersCanInteractAfterHostDisconnects(t *testing.T) {
+ var now atomic.Int64
+ now.Store(100_000)
+ h := startHarness(t, harnessOptions{now: now.Load})
+ host, first, second := openClient(t, h), openClient(t, h), openClient(t, h)
+ hostActor := authenticate(t, host, h.room, protocol.RoleHost, "leaving_host", -1, "",
+ presenceOptions{displayName: "Host", roomSession: true})
+ authenticate(t, first, h.room, protocol.RoleViewer, "remaining_first", -1, "", presenceOptions{displayName: "First"})
+ secondActor := authenticate(t, second, h.room, protocol.RoleViewer, "remaining_second", -1, "", presenceOptions{displayName: "Second"})
+ subscribeInteractions(t, first)
+ subscribeInteractions(t, second)
+ h.closeClient(host)
+ for _, payload := range []map[string]any{
+ {"kind": "chat", "text": "still here"},
+ {"kind": "reaction", "reaction": "wave"},
+ {"kind": "reaction", "reaction": "tomato", "targetPeerId": secondActor.PeerID},
+ } {
+ first.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "after_host_left", "payload": payload})
+ if a, b := first.next("room-interaction"), second.next("room-interaction"); string(a.raw) != string(b.raw) {
+ t.Fatal("remaining viewers did not receive the same interaction")
+ }
+ now.Add(protocol.InteractionIntervalMs)
+ }
+ first.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "departed_target",
+ "payload": map[string]any{"kind": "reaction", "reaction": "tomato", "targetPeerId": hostActor.PeerID}})
+ expectMatch(t, first.next("room-interaction-rejected").raw, `{"reason":"target-unavailable"}`)
+}
+
+func TestRoomInteractionBackpressureDoesNotRetireMediaSignaling(t *testing.T) {
+ h := startHarness(t, harnessOptions{})
+ host, viewer := openClient(t, h), openClient(t, h)
+ authenticate(t, host, h.room, protocol.RoleHost, "data_host_1234", -1, "", presenceOptions{displayName: "Host", viewerPresence: true})
+ actor := authenticate(t, viewer, h.room, protocol.RoleViewer, "slow_viewer_1234", -1, "", presenceOptions{displayName: "Viewer", viewerPresence: true})
+ subscribeInteractions(t, host)
+ subscribeInteractions(t, viewer)
+ h.locked(func() {
+ peer, ok := h.server.store.GetConnectedViewer(h.room.RoomID, actor.PeerID)
+ if !ok {
+ t.Fatal("missing viewer")
+ }
+ slow := h.server.sessionsByID[peer.SessionID]
+ // Inject a stalled writer under the same lock as a real broadcast, so
+ // the test does not depend on OS TCP buffer sizes or the reader's timing.
+ previous := slow.queuedBytes
+ slow.queuedBytes = 20 * 1024
+ hostPeer, _ := h.server.store.GetConnectedHost(h.room.RoomID)
+ hostSession := h.server.sessionsByID[hostPeer.SessionID]
+ h.server.handleRoomInteraction(hostSession, hostSession.authenticated, protocol.SendRoomInteractionMessage{
+ Type: "send-room-interaction", RequestID: "busy_request_1234", Payload: protocol.InteractionPayload{Kind: "chat", Text: "hello"},
+ })
+ if slow.terminated || slow.closeQueued {
+ t.Error("room data closed the signaling session")
+ }
+ if slow.queuedBytes != 20*1024 {
+ t.Error("room data entered a stalled signaling queue")
+ }
+ slow.queuedBytes = previous
+ })
+ host.next("room-interaction")
+ if _, ok := viewer.tryNext("room-interaction", 30*time.Millisecond); ok {
+ t.Fatal("stalled recipient got room data")
+ }
+ viewer.sendJSON(map[string]any{"type": "signaling-challenge", "sequence": 456})
+ viewer.next("signaling-challenge-response")
+}
diff --git a/internal/server/signal/router.go b/internal/server/signal/router.go
index 60c6780..f6cda6b 100644
--- a/internal/server/signal/router.go
+++ b/internal/server/signal/router.go
@@ -75,6 +75,7 @@ type routerOptions struct {
store *room.Store
endpointMediaCopyCapacity int
sfu *sfuFallback
+ sfuOnly bool
hooks routerHooks
now func() int64
afterFunc func(time.Duration, func()) func() bool
@@ -160,6 +161,7 @@ type router struct {
store *room.Store
capacity int
sfu *sfuFallback
+ sfuOnly bool
hooks routerHooks
now func() int64
afterFunc func(time.Duration, func()) func() bool
@@ -187,6 +189,7 @@ func newRouter(options routerOptions) *router {
store: options.store,
capacity: options.endpointMediaCopyCapacity,
sfu: options.sfu,
+ sfuOnly: options.sfuOnly,
hooks: options.hooks,
now: options.now,
afterFunc: options.afterFunc,
@@ -995,6 +998,7 @@ func (r *router) createController(roomID string, rm *roomRuntime, host authentic
EndpointMediaCopyCapacity: r.capacity,
OperationTimeoutMs: operationTimeoutMs,
SfuEnabled: r.sfu != nil && !(routePolicy != nil && routePolicy.PeerOnly),
+ SfuOnly: r.sfuOnly,
QualityConvergenceEnabled: routePolicy != nil && routePolicy.TopologyOptimization,
NatPredictionEnabled: routePolicy != nil && routePolicy.NatPrediction,
})
diff --git a/internal/server/signal/server.go b/internal/server/signal/server.go
index bbf7561..4a351a0 100644
--- a/internal/server/signal/server.go
+++ b/internal/server/signal/server.go
@@ -72,6 +72,8 @@ type Options struct {
EndpointMediaCopyCapacity int
// SfuFallback is nil when embedded SFU media is disabled.
SfuFallback *SfuFallback
+ // SfuOnly restricts the existing controller to server media candidates.
+ SfuOnly bool
// Ice is the derived wire ICE configuration (config.IceConfig).
Ice protocol.IceConfig
// NATPredictionEnabled is the server capability that clamps a Host's
@@ -215,6 +217,9 @@ func New(options Options) (*Server, error) {
if options.SfuFallback != nil && (options.SfuFallback.Media == nil || options.SfuFallback.Admission == nil) {
return nil, errors.New("SFU fallback requires media and resource admission")
}
+ if options.SfuOnly && options.SfuFallback == nil {
+ return nil, errors.New("SFU-only routing requires SFU media and resource admission")
+ }
s := &Server{
store: options.Store,
endpointMediaCopyCapacity: options.EndpointMediaCopyCapacity,
@@ -271,6 +276,7 @@ func New(options Options) (*Server, error) {
store: s.store,
endpointMediaCopyCapacity: s.endpointMediaCopyCapacity,
sfu: fallback,
+ sfuOnly: options.SfuOnly,
hooks: routerHooks{
sendToSession: s.sendToSession,
shareGeneration: func(roomID string) string { return s.shares[roomID].generation },
@@ -713,6 +719,8 @@ type authRequest struct {
viewerPresence bool
connectionAttemptProgress4 bool
displayName *string
+ roomOnly bool
+ roomSession bool
}
func authRequestOf(message protocol.ClientMessage) (authRequest, bool) {
@@ -730,6 +738,8 @@ func authRequestOf(message protocol.ClientMessage) (authRequest, bool) {
viewerPresence: m.ViewerPresence,
connectionAttemptProgress4: m.ConnectionAttemptProgress4,
displayName: displayNameString(m.DisplayName),
+ roomOnly: m.RoomOnly,
+ roomSession: m.RoomSession,
}, true
case protocol.AuthenticateViewerMessage:
return authRequest{
@@ -758,6 +768,14 @@ func displayNameString(name *protocol.DisplayName) *string {
// section. Password checks run in a goroutine so the reader can keep reading
// while the KDF runs.
func (s *Server) authenticate(sess *session, request authRequest) {
+ // Reject the conflicting preference before acquiring room or media authority,
+ // including reconnects from older pages. Never turn peer-only into SFU consent.
+ if request.role == protocol.RoleHost && !request.roomOnly && s.router.sfuOnly && request.routePolicy.PeerOnly {
+ s.sendError(sess, "FORBIDDEN", "This site requires server media; turn off Privacy mode or use another site")
+ sess.close(websocket.StatusCode(protocol.SignalCloseAuthenticationFailed), "Route policy conflict")
+ s.finishAuthenticating(sess)
+ return
+ }
if request.role == protocol.RoleHost && !sess.siteAccessAuthenticated {
s.sendError(sess, "AUTH_REQUIRED", "Site access is required")
sess.close(websocket.StatusCode(protocol.SignalCloseAuthenticationFailed), "Authentication failed")
@@ -889,28 +907,32 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part
displayName: displayName,
viewerPresence: request.viewerPresence,
connectionAttemptProgress4: request.connectionAttemptProgress4,
+ roomSession: request.roomSession,
}
sess.authenticated = authenticated
if participant.Role == protocol.RoleViewer {
s.clearViewerEvidence(roomID, participant.PeerID)
}
s.clearViewerGrace(roomID, participant.PeerID)
- routeParticipant := authenticatedRouteParticipant{
- roomID: roomID,
- role: participant.Role,
- peerID: participant.PeerID,
- sessionID: sess.sessionID,
+ mediaActive := participant.Role == protocol.RoleHost && shareGeneration != ""
+ if participant.Role == protocol.RoleViewer {
+ mediaActive = s.activeHostShareGeneration(roomID) != ""
}
- if participant.Role == protocol.RoleHost {
+ routeParticipant := routeParticipant(sess, authenticated)
+ if participant.Role == protocol.RoleHost && mediaActive {
policy := s.routePolicyOf(roomID)
routeParticipant.routePolicy = &policy
}
- hybridState, assigned := s.connectRouteParticipant(routeParticipant)
- if !assigned {
- s.logger.Error("Media router did not assign an authenticated participant")
- s.sendError(sess, "SERVER_ERROR", "Media assignment failed")
- sess.close(websocket.StatusInternalError, "Media assignment failed")
- return
+ hybridState := hybridAuthenticationState{routeAssignment: emptyAssignment()}
+ if participant.Role == protocol.RoleViewer || mediaActive {
+ var assigned bool
+ hybridState, assigned = s.connectRouteParticipant(routeParticipant)
+ if !assigned {
+ s.logger.Error("Media router did not assign an authenticated participant")
+ s.sendError(sess, "SERVER_ERROR", "Media assignment failed")
+ sess.close(websocket.StatusInternalError, "Media assignment failed")
+ return
+ }
}
// The viewer's media identity is derived from the committed edge, never
// stored beside it; connectParticipant has just settled the graph.
@@ -934,8 +956,11 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part
qualitySettings = *share.qualitySettings
}
var roomShareGeneration *string
- if share.generation != "" {
- generation := share.generation
+ if participant.Role == protocol.RoleHost && mediaActive {
+ generation := shareGeneration
+ roomShareGeneration = &generation
+ } else if participant.Role == protocol.RoleViewer && mediaActive {
+ generation := s.activeHostShareGeneration(roomID)
roomShareGeneration = &generation
}
if participant.Role == protocol.RoleHost {
@@ -960,15 +985,15 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part
ViewerPasswordEnabled: participant.ViewerPasswordEnabled,
})
} else {
- hostPaused := participant.HostOnline &&
- share.pausedGeneration == share.generation
+ hostOnline := participant.HostOnline && mediaActive
+ hostPaused := hostOnline && share.pausedGeneration == share.generation
s.send(sess, protocol.AuthenticatedViewerMessage{
Type: "authenticated",
Protocol: protocol.SignalingProtocol,
PeerID: participant.PeerID,
MaxViewers: protocol.Int(s.store.MaxViewersPerRoom()),
EndpointMediaCopyCapacity: protocol.Int(s.endpointMediaCopyCapacity),
- HostOnline: participant.HostOnline,
+ HostOnline: hostOnline,
HostPaused: &hostPaused,
ConnectionID: connectionID,
IceConfig: s.ice,
@@ -989,12 +1014,14 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part
}
}
- if participant.Role == protocol.RoleHost {
+ if participant.Role == protocol.RoleHost && mediaActive {
s.router.setPaused(roomID, s.shares[roomID].pausedGeneration == shareGeneration)
}
- s.router.completeAuthentication(routeParticipant, hybridState)
+ if participant.Role == protocol.RoleViewer || mediaActive {
+ s.router.completeAuthentication(routeParticipant, hybridState)
+ }
- if participant.Role == protocol.RoleHost && shareGeneration != "" &&
+ if participant.Role == protocol.RoleHost && mediaActive &&
s.shares[roomID].pausedGeneration == shareGeneration {
s.sendToSession(sess.sessionID, protocol.PauseSharingSourceMessage{
Type: "pause-sharing-source",
@@ -1004,7 +1031,7 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part
s.sendViewerPresence(roomID)
- if participant.Role == protocol.RoleHost {
+ if participant.Role == protocol.RoleHost && mediaActive {
for _, viewer := range connectedViewers {
s.sendToSession(viewer.SessionID, protocol.HostStatusMessage{
Type: "host-status",
@@ -1021,6 +1048,15 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part
// with. It returns the settled generation and must run with mu held, before the
// `authenticated` message is built from the record it writes.
func (s *Server) settleHostShare(roomID, sessionID string, request authRequest) string {
+ if request.roomOnly {
+ // A room-only reconnect must not resurrect a stale publication. The
+ // stop path normally already cleared it; this guard covers a socket
+ // loss between the client cleanup and the server's stop message.
+ if s.shares[roomID].generation != "" {
+ s.stopSharing(roomID)
+ }
+ return ""
+ }
current := s.shares[roomID]
currentGeneration := current.generation
shareGeneration := request.shareGeneration
@@ -1111,10 +1147,33 @@ func (s *Server) routePolicyOf(roomID string) protocol.RoutePolicy {
return s.configuredRoutePolicy(protocol.DefaultRoutePolicy)
}
+// activeHostShareGeneration is the media truth for a room. The room share
+// record intentionally keeps its generation as a stale reconnect fence after
+// stopSharing, so membership or that fence alone cannot mean "publishing".
+// Callers hold Server.mu.
+func (s *Server) activeHostShareGeneration(roomID string) string {
+ host, ok := s.store.GetConnectedHost(roomID)
+ if !ok {
+ return ""
+ }
+ hostSession := s.sessionsByID[host.SessionID]
+ if hostSession == nil || hostSession.authenticated == nil ||
+ hostSession.authenticated.role != protocol.RoleHost ||
+ hostSession.authenticated.roomID != roomID {
+ return ""
+ }
+ generation := hostSession.authenticated.shareGeneration
+ if generation == "" || s.shares[roomID].generation != generation {
+ return ""
+ }
+ return generation
+}
+
// configuredRoutePolicy restricts share preferences to the services this runtime owns.
func (s *Server) configuredRoutePolicy(policy protocol.RoutePolicy) protocol.RoutePolicy {
policy.PeerOnly = s.router.sfu == nil || policy.PeerOnly
- policy.NatPrediction = s.natPredictionEnabled && policy.NatPrediction
+ policy.NatPrediction = s.natPredictionEnabled && !s.router.sfuOnly && policy.NatPrediction
+ policy.TopologyOptimization = !s.router.sfuOnly && policy.TopologyOptimization
return policy
}
@@ -1131,6 +1190,12 @@ func routeParticipant(sess *session, authenticated *authenticatedSession) authen
func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authenticatedSession, message protocol.ClientMessage) {
switch m := message.(type) {
+ case protocol.SubscribeRoomInteractionsMessage:
+ // Explicit opt-in protects older v23 readers from unknown events.
+ sess.roomInteractions = true
+ s.send(sess, protocol.RoomInteractionsReadyMessage{Type: "room-interactions-ready", ServerTime: protocol.Int(s.now())})
+ case protocol.SendRoomInteractionMessage:
+ s.handleRoomInteraction(sess, authenticated, m)
case protocol.SignalingChallengeMessage:
if !s.sessions.Has(sess) || sess.authenticated != authenticated {
return
@@ -1165,6 +1230,10 @@ func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authen
s.sendError(sess, "FORBIDDEN", "Only the host may set quality settings")
return
}
+ if authenticated.shareGeneration == "" ||
+ s.shares[authenticated.roomID].generation != authenticated.shareGeneration {
+ return
+ }
share := s.shares[authenticated.roomID]
share.qualitySettings = &m.QualitySettings
s.shares[authenticated.roomID] = share
@@ -1246,28 +1315,39 @@ func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authen
s.shares[authenticated.roomID] = share
s.router.setPaused(authenticated.roomID, m.Paused)
s.broadcastHostStatus(authenticated.roomID, true, m.Paused)
+ case protocol.StartSharingMessage:
+ s.startSharing(sess, authenticated, m)
case protocol.StopSharingMessage:
if authenticated.role != protocol.RoleHost {
s.sendError(sess, "FORBIDDEN", "只有当前房主可以停止分享")
return
}
- if authenticated.shareGeneration == "" ||
- s.shares[authenticated.roomID].generation != authenticated.shareGeneration ||
+ if authenticated.shareGeneration == "" {
+ return
+ }
+ if s.shares[authenticated.roomID].generation != authenticated.shareGeneration ||
(m.ShareGeneration != "" && m.ShareGeneration != authenticated.shareGeneration) {
sess.close(websocket.StatusCode(protocol.SignalCloseSessionReplaced), "Sharing generation replaced")
return
}
- // The participant leaves the store before stopSharing()
- // so the sharing-stopped / host-status pair is what viewers see.
- if s.sessions.Has(sess) && sess.authenticated == authenticated {
- if _, err := s.store.DisconnectParticipant(authenticated.roomID, authenticated.peerID, sess.sessionID); err != nil {
- // The TS threw out of the message handler; nothing caught it.
- panic(fmt.Errorf("room store failed while stopping sharing: %w", err))
+ if !authenticated.roomSession {
+ // Older clients treat stopping media as leaving the Host session.
+ // Keep that contract until the client explicitly opts into the
+ // persistent room-session lifecycle.
+ if s.sessions.Has(sess) && sess.authenticated == authenticated {
+ if _, err := s.store.DisconnectParticipant(authenticated.roomID, authenticated.peerID, sess.sessionID); err != nil {
+ panic(fmt.Errorf("room store failed while stopping sharing: %w", err))
+ }
}
+ s.stopSharing(authenticated.roomID)
+ sess.close(websocket.StatusNormalClosure, "Sharing stopped")
+ return
}
s.stopSharing(authenticated.roomID)
+ // Room membership and its interaction stream outlive the media
+ // publication. The next start-sharing message claims a fresh generation.
+ authenticated.shareGeneration = ""
s.sendViewerPresence(authenticated.roomID)
- sess.close(websocket.StatusNormalClosure, "Sharing stopped")
case protocol.AbandonRoomMessage:
if authenticated.role != protocol.RoleHost {
s.sendError(sess, "FORBIDDEN", "Only the host may abandon the room")
@@ -1280,6 +1360,76 @@ func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authen
}
}
+// startSharing claims a new publication generation without replacing the
+// authenticated Host participant. Room interactions therefore keep the same
+// session while the media graph is rebuilt for the new source.
+func (s *Server) startSharing(sess *session, authenticated *authenticatedSession, message protocol.StartSharingMessage) {
+ if authenticated.role != protocol.RoleHost {
+ s.sendError(sess, "FORBIDDEN", "Only the host may start sharing")
+ return
+ }
+ if !authenticated.roomSession {
+ s.sendError(sess, "FORBIDDEN", "Persistent room session is not enabled")
+ return
+ }
+ if !s.sessions.Has(sess) || sess.authenticated != authenticated ||
+ authenticated.shareGeneration != "" || message.ShareGeneration == s.shares[authenticated.roomID].generation ||
+ (s.router.sfuOnly && message.RoutePolicy.PeerOnly) {
+ s.send(sess, protocol.SharingStartFailedMessage{Type: "sharing-start-failed",
+ ShareGeneration: message.ShareGeneration, Code: "FORBIDDEN"})
+ return
+ }
+
+ // stop-sharing deliberately leaves the previous generation as a stale
+ // fence for reconnect validation. A fresh start supersedes that fence
+ // without emitting a second sharing-stopped event.
+ share := s.shares[authenticated.roomID]
+ share.generation = ""
+ share.pausedGeneration = ""
+ s.shares[authenticated.roomID] = share
+ request := authRequest{
+ role: protocol.RoleHost,
+ roomID: authenticated.roomID,
+ shareGeneration: message.ShareGeneration,
+ sharingPaused: message.SharingPaused,
+ qualitySettings: message.QualitySettings,
+ routePolicy: message.RoutePolicy,
+ }
+ shareGeneration := s.settleHostShare(authenticated.roomID, sess.sessionID, request)
+ authenticated.shareGeneration = shareGeneration
+ routeParticipant := routeParticipant(sess, authenticated)
+ policy := s.routePolicyOf(authenticated.roomID)
+ routeParticipant.routePolicy = &policy
+ routeState, assigned := s.connectRouteParticipant(routeParticipant)
+ if !assigned {
+ authenticated.shareGeneration = ""
+ s.stopSharing(authenticated.roomID)
+ s.send(sess, protocol.SharingStartFailedMessage{Type: "sharing-start-failed",
+ ShareGeneration: message.ShareGeneration, Code: "SERVER_ERROR"})
+ return
+ }
+
+ share = s.shares[authenticated.roomID]
+ qualitySettings := protocol.DefaultQualitySettings
+ if share.qualitySettings != nil {
+ qualitySettings = *share.qualitySettings
+ }
+ s.send(sess, protocol.SharingStartedMessage{
+ Type: "sharing-started",
+ ShareGeneration: shareGeneration,
+ RoutePolicy: s.routePolicyOf(authenticated.roomID),
+ RouteRevision: protocol.Int(routeState.routeRevision),
+ RouteAssignment: routeState.routeAssignment,
+ QualitySettings: qualitySettings,
+ Paused: share.pausedGeneration == shareGeneration,
+ })
+ // Acknowledge publication authority before any media effects use it.
+ s.router.setPaused(authenticated.roomID, share.pausedGeneration == shareGeneration)
+ s.router.completeAuthentication(routeParticipant, routeState)
+ s.sendViewerPresence(authenticated.roomID)
+ s.broadcastHostStatus(authenticated.roomID, true, share.pausedGeneration == shareGeneration)
+}
+
// ---------------------------------------------------------------------------
// quality evidence
// ---------------------------------------------------------------------------
diff --git a/internal/server/signal/server_test.go b/internal/server/signal/server_test.go
index ae9bd39..a1328e0 100644
--- a/internal/server/signal/server_test.go
+++ b/internal/server/signal/server_test.go
@@ -9,6 +9,7 @@ package signal
// TS went through HTTP; the site-access cookie is a stub predicate.
import (
+ "bytes"
"context"
"encoding/json"
"errors"
@@ -80,6 +81,7 @@ func TestStopSharingDoesNotWriteRoomAuthority(t *testing.T) {
h := startHarness(t, harnessOptions{store: store})
host := openClient(t, h)
authenticated := authenticate(t, host, h.room, protocol.RoleHost, "storage-free-host", 1, "", presenceOptions{})
+ host.ignore("route-update")
h.locked(func() {
if err := database.Close(); err != nil {
t.Fatal(err)
@@ -94,6 +96,151 @@ func TestStopSharingDoesNotWriteRoomAuthority(t *testing.T) {
authenticate(t, resumed, h.room, protocol.RoleHost, "storage-free-host", 1, "", presenceOptions{})
}
+func TestHostKeepsRoomSessionAndInteractionsAfterSharingStops(t *testing.T) {
+ h := startHarness(t, harnessOptions{maxViewersPerRoom: 2})
+ host := openClient(t, h)
+ hostAuth := authenticate(t, host, h.room, protocol.RoleHost, "persistent-host", 1,
+ "first_share_generation_12345678", presenceOptions{displayName: "Host", viewerPresence: true, roomSession: true})
+ viewer := openClient(t, h)
+ authenticate(t, viewer, h.room, protocol.RoleViewer, "persistent-viewer", 1, "",
+ presenceOptions{displayName: "Viewer", viewerPresence: true})
+ host.ignore("route-update")
+ host.ignore("viewer-presence")
+ subscribeInteractions(t, host)
+ subscribeInteractions(t, viewer)
+
+ host.sendJSON(map[string]any{"type": "stop-sharing", "shareGeneration": hostAuth.ShareGeneration})
+ viewer.next("sharing-stopped")
+ viewer.next("host-status")
+ host.expectNone(80 * time.Millisecond)
+ h.locked(func() {
+ connected, ok := h.server.store.GetConnectedHost(h.room.RoomID)
+ if !ok || connected.SessionID == "" {
+ t.Fatal("stopping media must retain the Host room session")
+ }
+ })
+
+ host.sendJSON(map[string]any{
+ "type": "send-room-interaction",
+ "requestId": "persistent_chat_request",
+ "payload": map[string]any{"kind": "chat", "text": "still here"},
+ })
+ host.next("room-interaction")
+ viewer.next("room-interaction")
+
+ host.sendJSON(map[string]any{
+ "type": "start-sharing",
+ "shareGeneration": "second_share_generation_12345678",
+ "routePolicy": map[string]any{"peerOnly": true, "topologyOptimization": true, "natPrediction": false},
+ })
+ started := host.next("sharing-started")
+ if !bytes.Contains(started.raw, []byte(`"shareGeneration":"second_share_generation_12345678"`)) {
+ t.Fatalf("unexpected sharing-started: %s", started.raw)
+ }
+ viewer.next("host-status")
+ h.closeClient(viewer)
+ h.closeClient(host)
+}
+
+func TestIdleHostChatAndPublicationAcknowledgmentOrder(t *testing.T) {
+ h := startHarness(t, harnessOptions{})
+ host := openClient(t, h)
+ authenticate(t, host, h.room, protocol.RoleHost, "room-only-host", 1, "", presenceOptions{roomSession: true, roomOnly: true, viewerPresence: true})
+ host.ignore("viewer-presence")
+ viewer := openClient(t, h)
+ authenticate(t, viewer, h.room, protocol.RoleViewer, "room-only-viewer", 1, "", presenceOptions{})
+ subscribeInteractions(t, host)
+ subscribeInteractions(t, viewer)
+ host.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "chat_before_capture",
+ "payload": map[string]any{"kind": "chat", "text": "before capture"}})
+ host.next("room-interaction")
+ viewer.next("room-interaction")
+ host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "new_publication_generation",
+ "routePolicy": protocol.DefaultRoutePolicy})
+ // No ignored route messages: acknowledgment must precede media effects.
+ host.sendJSON(map[string]any{"type": "signaling-challenge", "sequence": 1})
+ host.next("signaling-challenge-response")
+ host.mu.Lock()
+ first := host.messages[0].typ
+ host.mu.Unlock()
+ if first != "sharing-started" {
+ t.Fatalf("first publication message = %s", first)
+ }
+ host.next("sharing-started")
+ host.next("route-update")
+}
+
+func TestRejectedPublicationRetainsHostRoomSession(t *testing.T) {
+ h := startHarness(t, harnessOptions{})
+ host := openClient(t, h)
+ authenticate(t, host, h.room, protocol.RoleHost, "rejected-start-host", 1, "", presenceOptions{roomSession: true, roomOnly: true, viewerPresence: true})
+ host.ignore("viewer-presence")
+ subscribeInteractions(t, host)
+ // Inject the existing route-construction failure, after room admission.
+ h.locked(func() { h.server.router.capacity = 0 })
+ host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "rejected_publication_generation",
+ "routePolicy": protocol.DefaultRoutePolicy})
+ expectMatch(t, host.next("sharing-start-failed").raw,
+ `{"shareGeneration":"rejected_publication_generation","code":"SERVER_ERROR"}`)
+ h.locked(func() {
+ h.server.router.capacity = 2
+ if h.server.activeHostShareGeneration(h.room.RoomID) != "" {
+ t.Fatal("failed publication remained active")
+ }
+ })
+ host.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "chat_after_rejected_start",
+ "payload": map[string]any{"kind": "chat", "text": "still in room"}})
+ host.next("room-interaction")
+ host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "next_publication_generation",
+ "routePolicy": protocol.DefaultRoutePolicy})
+ host.next("sharing-started")
+ host.next("route-update")
+ host.ignore("route-update")
+ host.sendJSON(map[string]any{"type": "stop-sharing", "shareGeneration": "next_publication_generation"})
+ host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "next_publication_generation",
+ "routePolicy": protocol.DefaultRoutePolicy})
+ expectMatch(t, host.next("sharing-start-failed").raw,
+ `{"shareGeneration":"next_publication_generation","code":"FORBIDDEN"}`)
+}
+
+func TestPersistentRoomSessionReconnectStaysMediaIdle(t *testing.T) {
+ h := startHarness(t, harnessOptions{maxViewersPerRoom: 2})
+ host := openClient(t, h)
+ hostAuth := authenticate(t, host, h.room, protocol.RoleHost, "idle-reconnect-host", 1,
+ "idle_reconnect_generation_12345678", presenceOptions{roomSession: true})
+ viewer := openClient(t, h)
+ authenticate(t, viewer, h.room, protocol.RoleViewer, "idle-reconnect-viewer", 1, "", presenceOptions{})
+ host.ignore("route-update")
+ viewer.ignore("route-update")
+ host.sendJSON(map[string]any{"type": "stop-sharing", "shareGeneration": hostAuth.ShareGeneration})
+ viewer.next("sharing-stopped")
+ viewer.next("host-status")
+ h.closeClient(viewer)
+ h.closeClient(host)
+
+ resumed := openClient(t, h)
+ resumedAuth := authenticate(t, resumed, h.room, protocol.RoleHost, "idle-reconnect-host", 1, "",
+ presenceOptions{roomSession: true, roomOnly: true})
+ if resumedAuth.ShareGeneration != nil {
+ t.Fatalf("room-only Host must not receive a media generation: %v", *resumedAuth.ShareGeneration)
+ }
+ h.locked(func() {
+ if runtime, ok := h.server.router.rooms.Get(h.room.RoomID); ok && runtime.controller != nil {
+ t.Fatal("room-only Host reconnect created a media controller")
+ }
+ })
+ lateViewer := openClient(t, h)
+ lateAuth := authenticate(t, lateViewer, h.room, protocol.RoleViewer, "idle-reconnect-late-viewer", 1, "", presenceOptions{})
+ if lateAuth.HostOnline || lateAuth.HostPaused == nil || *lateAuth.HostPaused {
+ t.Fatalf("idle room advertised active Host media: online=%v paused=%v", lateAuth.HostOnline, lateAuth.HostPaused)
+ }
+ if lateAuth.ShareGeneration != nil || lateAuth.RouteAssignment.Upstream.Kind != "none" {
+ t.Fatalf("idle room advertised a media route: generation=%v assignment=%+v", lateAuth.ShareGeneration, lateAuth.RouteAssignment)
+ }
+ h.closeClient(lateViewer)
+ h.closeClient(resumed)
+}
+
// ---------------------------------------------------------------------------
// harness
// ---------------------------------------------------------------------------
@@ -113,6 +260,7 @@ type harnessOptions struct {
room *room.CreatedRoom
afterFunc func(time.Duration, func()) func() bool
sfu *SfuFallback
+ sfuOnly bool
}
type harness struct {
@@ -189,6 +337,7 @@ func startHarness(t *testing.T, options harnessOptions) *harness {
Store: store,
EndpointMediaCopyCapacity: orValue(options.endpointMediaCopyCapacity, 2),
SfuFallback: options.sfu,
+ SfuOnly: options.sfuOnly,
Ice: config.IceConfig(config.Config{
STUNURLs: options.stunURLs,
NATPredictionEnabled: options.natPredictionEnabled,
@@ -803,6 +952,8 @@ type presenceOptions struct {
sharingPaused *bool
qualitySettings map[string]any
routePolicy *protocol.RoutePolicy
+ roomSession bool
+ roomOnly bool
}
// authenticate ports the authenticate() helper. relayCapacity -1 is the TS
@@ -833,6 +984,12 @@ func authenticate(
if shareGeneration != "" {
message["shareGeneration"] = shareGeneration
}
+ if presence.roomSession {
+ message["roomSession"] = true
+ }
+ if presence.roomOnly {
+ message["roomOnly"] = true
+ }
if presence.sharingPaused != nil {
message["sharingPaused"] = *presence.sharingPaused
}
diff --git a/internal/server/signal/session.go b/internal/server/signal/session.go
index 46dd9fa..e6c169c 100644
--- a/internal/server/signal/session.go
+++ b/internal/server/signal/session.go
@@ -23,6 +23,7 @@ type authenticatedSession struct {
displayName *string
viewerPresence bool
connectionAttemptProgress4 bool
+ roomSession bool
}
// outbound is one queue item: a text frame or the close frame that ends the
@@ -54,6 +55,8 @@ type session struct {
challenged bool
lastSignalingChallengeAtMs int64
authenticated *authenticatedSession
+ roomInteractions bool
+ nextInteractionAtMs int64
// authStop stops the authentication deadline; authGeneration is
// bumped whenever the TS cleared the timer so a callback that lost the
diff --git a/internal/server/signal/sfu_only_test.go b/internal/server/signal/sfu_only_test.go
new file mode 100644
index 0000000..357e0ec
--- /dev/null
+++ b/internal/server/signal/sfu_only_test.go
@@ -0,0 +1,34 @@
+package signal
+
+import (
+ "testing"
+
+ "github.com/TNTcraftHIM/Piik/internal/server/protocol"
+ "github.com/TNTcraftHIM/Piik/internal/server/sfu"
+ "github.com/TNTcraftHIM/Piik/internal/server/sfu/sfutest"
+)
+
+func TestSFUOnlyRejectsPeerOnlyBeforeAcquiringShareAuthority(t *testing.T) {
+ h := startHarness(t, harnessOptions{sfuOnly: true, natPredictionEnabled: true,
+ sfu: &SfuFallback{Media: sfutest.New(), Admission: sfu.NewAdmission(sfu.AdmissionOptions{IngressCapacity: 2, EgressCapacity: 20})}})
+ policy := protocol.RoutePolicy{PeerOnly: true, TopologyOptimization: true, NatPrediction: true}
+ rejected := openClient(t, h)
+ rejected.sendJSON(map[string]any{"type": "authenticate", "protocol": protocol.SignalingProtocol,
+ "roomId": h.room.RoomID, "role": "host", "clientId": "old-peer-only-host",
+ "token": h.room.HostToken, "routePolicy": policy})
+ expectMatch(t, rejected.next("error").raw, `{"code":"FORBIDDEN"}`)
+ expectCloseCode(t, rejected, int(protocol.SignalCloseAuthenticationFailed))
+
+ // An idle room remains available. Refusing a conflicting share preserves it.
+ host := openClient(t, h)
+ authenticate(t, host, h.room, protocol.RoleHost, "idle-host", -1, "",
+ presenceOptions{roomSession: true, roomOnly: true, routePolicy: &policy})
+ host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "sfu_only_share_generation",
+ "routePolicy": policy})
+ expectMatch(t, host.next("sharing-start-failed").raw, `{"code":"FORBIDDEN"}`)
+ policy.PeerOnly = false
+ host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "sfu_only_share_generation",
+ "routePolicy": policy})
+ expectMatch(t, host.next("sharing-started").raw,
+ `{"routePolicy":{"peerOnly":false,"topologyOptimization":false,"natPrediction":false}}`)
+}
diff --git a/native/capture/windows/README.md b/native/capture/windows/README.md
index 2e781c3..83fea64 100644
--- a/native/capture/windows/README.md
+++ b/native/capture/windows/README.md
@@ -5,7 +5,8 @@ enumerates local displays and visible top-level windows, binds each window to
its PID and process creation time, and has independent modes for:
- capability discovery (`--probe`);
-- process-tree or default-device loopback PCM (`--capture-audio`); or
+- process-tree inclusion/exclusion or default-device loopback PCM
+ (`--capture-audio`); or
- WGC/D3D11 screen/window video with adapter-bound Media Foundation H.264 or
libvpx VP8 output (`--capture-video --codec auto|h264|vp8`).
- bounded 320x180 BMP source previews (`--preview`), delivered once per target
@@ -30,7 +31,12 @@ Video and audio run as separate bounded child processes. A source whose audio
loopback cannot be initialized keeps video available and reports audio
unavailable instead of failing the whole source. Process loopback is probed by
activation rather than inferred from a Windows build number; display sources
-use the standard render-device loopback available on Windows 10 and later.
+normally use the standard render-device loopback available on Windows 10 and
+later. The optional `--capture-audio exclude ` mode requires
+build 20348+ and a successful exclusion probe. It captures all output devices
+except the selected process tree, and retires when that process exits. The
+[screen-audio contract](../../../docs/standards/media-quality.md#screen-audio)
+owns selection and failure behavior.
Auto measures delivered frame cadence through the target-profile encoding
pipeline within a four-second selection budget; H.264 that sustains the target
@@ -50,6 +56,19 @@ The checks include synthetic output-worker replacement and failure cases using
WARP and a test codec, plus preview cleanup when WGC rejects a hidden test window.
They do not capture a user screen or validate a physical encoder.
+On a Windows machine with a physical D3D11 adapter, check SDR and HDR-to-SDR conversion:
+
+```powershell
+powershell -NoProfile -File native/capture/windows/build.ps1 -OutputDirectory build/color-check -CheckColor
+```
+
+This sends synthetic RGB, full/limited-range BT.601/BT.709 NV12 and FP16 HDR bars
+through the production converters. It checks range/matrix, highlight detail,
+SDR/HDR replacement, white-level changes and retained-frame ownership without
+capturing the desktop. Decoded H.264/VP8, Browser reception and real HDR displays
+remain separate media acceptance; [fidelity research](../../../docs/research/media-fidelity.md)
+records current evidence and limits.
+
The implementation follows Microsoft's MIT-licensed reference samples and
official API contracts without copying their WIL framework. The retained MF
fixture compiles the same encoder source with `PIIK_H264_FIXTURE`; there is
diff --git a/native/capture/windows/build.ps1 b/native/capture/windows/build.ps1
index 6c33630..2a39a40 100644
--- a/native/capture/windows/build.ps1
+++ b/native/capture/windows/build.ps1
@@ -2,7 +2,8 @@
param(
[Parameter(Mandatory = $true)]
[string]$OutputDirectory,
- [switch]$Check
+ [switch]$Check,
+ [switch]$CheckColor
)
Set-StrictMode -Version Latest
@@ -34,7 +35,7 @@ $includeFlags = @('', 'third_party\abseil-cpp', 'third_party\boringssl\src\inclu
'/external:I "{0}"' -f (Join-Path $webrtc.Include $_).TrimEnd('\')
}
$compileFlags = '/nologo /c /std:c++20 /EHsc /GR /O2 /W4 /WX /MT /D_ITERATOR_DEBUG_LEVEL=0 /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 /DNTDDI_VERSION=0x0A00000A /external:W0 ' + ($includeFlags -join ' ')
-$systemLibraries = 'ole32.lib mmdevapi.lib runtimeobject.lib user32.lib gdi32.lib dwmapi.lib shell32.lib mfplat.lib mf.lib mfuuid.lib d3d11.lib dxgi.lib dxguid.lib evr.lib oleaut32.lib windowsapp.lib winmm.lib ws2_32.lib strmiids.lib crypt32.lib dmoguids.lib iphlpapi.lib msdmo.lib secur32.lib wmcodecdspuuid.lib'
+$systemLibraries = 'ole32.lib mmdevapi.lib runtimeobject.lib user32.lib gdi32.lib dwmapi.lib shell32.lib mfplat.lib mf.lib mfuuid.lib d3d11.lib d2d1.lib dxgi.lib dxguid.lib evr.lib oleaut32.lib windowsapp.lib winmm.lib ws2_32.lib strmiids.lib crypt32.lib dmoguids.lib iphlpapi.lib msdmo.lib secur32.lib wmcodecdspuuid.lib'
$linkCommand = '"{0}" /nologo /libpath:"{1}"' -f $webrtc.Linker,$webrtc.RuntimeLibraries
function Invoke-CaptureBuild([string]$command) {
@@ -76,4 +77,16 @@ if ($Check) {
if ($LASTEXITCODE -ne 0) { throw "Capture check failed: $name" }
}
}
+if ($CheckColor) {
+ $source = Join-Path $helperDirectory 'frame_converter.probe.cpp'
+ $object = Join-Path $outputPath 'frame_converter.probe.obj'
+ $executable = Join-Path $outputPath 'frame_converter.probe.exe'
+ Invoke-CaptureBuild ('cl.exe {0} "{1}" /Fo:"{2}"' -f $compileFlags,$source,$object)
+ $dependencies = @('process_audio', 'capture_target', 'h264_encoder', 'adaptive_encoder') | ForEach-Object {
+ '"{0}"' -f (Join-Path $outputPath ($_ + '.obj'))
+ }
+ Invoke-CaptureBuild ('{0} /out:"{1}" "{2}" {3} "{4}" {5}' -f $linkCommand,$executable,$object,($dependencies -join ' '),$webrtc.Library,$systemLibraries)
+ & $executable
+ if ($LASTEXITCODE -ne 0) { throw 'Capture color check failed.' }
+}
Write-Output $executablePath
diff --git a/native/capture/windows/capture_color.h b/native/capture/windows/capture_color.h
new file mode 100644
index 0000000..b712528
--- /dev/null
+++ b/native/capture/windows/capture_color.h
@@ -0,0 +1,34 @@
+#pragma once
+
+#include "capture_error.h"
+
+#include
+#include
+#include
+
+namespace piik::capture::windows {
+
+// Untagged WebRTC SDR uses BT.601. Keep the converter's pixels and the H264
+// encoder's range/matrix declaration in agreement; VP8 has no equivalent VUI.
+inline constexpr auto kSdrVideoColor = DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P601;
+inline constexpr auto kDesktopColor = DXGI_COLOR_SPACE_RGB_FULL_G22_NONE_P709;
+
+inline void SetSdrVideoColor(IMFAttributes* type) {
+ Check(type->SetUINT32(MF_MT_YUV_MATRIX, MFVideoTransferMatrix_BT601), "video-type-matrix");
+ Check(type->SetUINT32(MF_MT_VIDEO_NOMINAL_RANGE, MFNominalRange_16_235), "video-type-range");
+ Check(type->SetUINT32(MF_MT_VIDEO_PRIMARIES, MFVideoPrimaries_SMPTE170M), "video-type-primaries");
+ Check(type->SetUINT32(MF_MT_TRANSFER_FUNCTION, MFVideoTransFunc_709), "video-type-transfer");
+}
+
+// The decoder owns the incoming stream's declaration. Preserve it through the
+// relay's local scale/re-encode boundary instead of guessing from resolution.
+inline DXGI_COLOR_SPACE_TYPE DecodedVideoColor(IMFAttributes* type) {
+ const auto matrix = MFGetAttributeUINT32(type, MF_MT_YUV_MATRIX, MFVideoTransferMatrix_BT601);
+ const bool full = MFGetAttributeUINT32(type, MF_MT_VIDEO_NOMINAL_RANGE,
+ MFNominalRange_16_235) == MFNominalRange_0_255;
+ if (matrix == MFVideoTransferMatrix_BT709)
+ return full ? DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P709 : DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P709;
+ return full ? DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P601 : kSdrVideoColor;
+}
+
+} // namespace piik::capture::windows
diff --git a/native/capture/windows/capture_sdr.h b/native/capture/windows/capture_sdr.h
new file mode 100644
index 0000000..7aa1d8d
--- /dev/null
+++ b/native/capture/windows/capture_sdr.h
@@ -0,0 +1,209 @@
+#pragma once
+
+#include "capture_error.h"
+#include "capture_target.h"
+
+#include
+#include
+#include
+#include
+#include
+
+#include
+#include
+#include
+#include
+
+namespace piik::capture::windows {
+
+struct CaptureColor final {
+ bool hdr = false;
+ float white_nits = 80.0f;
+ float peak_nits = 80.0f;
+
+ DXGI_FORMAT Format() const {
+ return hdr ? DXGI_FORMAT_R16G16B16A16_FLOAT : DXGI_FORMAT_B8G8R8A8_UNORM;
+ }
+};
+
+// Windows invalidates the DXGI factory when display color characteristics
+// change. Window movement additionally changes the source monitor; neither
+// event belongs to encoder or media-route recovery.
+class CaptureDisplayColor final {
+ public:
+ CaptureDisplayColor(TargetKind kind, UINT64 source_id)
+ : kind_(kind), source_id_(source_id) {}
+
+ CaptureColor Resolve() {
+ const auto handle = static_cast(source_id_);
+ const HMONITOR monitor = kind_ == TargetKind::window
+ ? MonitorFromWindow(reinterpret_cast(handle), MONITOR_DEFAULTTONEAREST)
+ : reinterpret_cast(handle);
+ if (factory_ && factory_->IsCurrent() && monitor == monitor_) return color_;
+
+ Microsoft::WRL::ComPtr factory;
+ if (FAILED(CreateDXGIFactory1(IID_PPV_ARGS(&factory))))
+ return monitor == monitor_ ? color_ : CaptureColor{};
+ CaptureColor next;
+ for (UINT adapter_index = 0;; ++adapter_index) {
+ Microsoft::WRL::ComPtr adapter;
+ if (FAILED(factory->EnumAdapters1(adapter_index, &adapter))) break;
+ for (UINT output_index = 0;; ++output_index) {
+ Microsoft::WRL::ComPtr output;
+ if (FAILED(adapter->EnumOutputs(output_index, &output))) break;
+ Microsoft::WRL::ComPtr extended;
+ DXGI_OUTPUT_DESC1 description{};
+ if (FAILED(output.As(&extended)) || FAILED(extended->GetDesc1(&description)) ||
+ description.Monitor != monitor) continue;
+ next.hdr = description.ColorSpace == DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020;
+ if (next.hdr) {
+ next.white_nits = ReadWhiteLevel(description.DeviceName);
+ // WGC has no per-frame MaxCLL. Use the source display's stable peak;
+ // absent usable metadata, retain the HDR10 luminance range.
+ const float peak = std::isfinite(description.MaxLuminance) &&
+ description.MaxLuminance > 0.0f
+ ? description.MaxLuminance : 10'000.0f;
+ next.peak_nits = std::max(next.white_nits, peak);
+ }
+ factory_ = std::move(factory);
+ monitor_ = monitor;
+ color_ = next;
+ return color_;
+ }
+ }
+ return next;
+ }
+
+ private:
+ static float ReadWhiteLevel(const wchar_t* device_name) {
+ UINT32 path_count = 0, mode_count = 0;
+ if (GetDisplayConfigBufferSizes(QDC_ONLY_ACTIVE_PATHS, &path_count, &mode_count) !=
+ ERROR_SUCCESS) return 80.0f;
+ std::vector paths(path_count);
+ std::vector modes(mode_count);
+ if (QueryDisplayConfig(QDC_ONLY_ACTIVE_PATHS, &path_count, paths.data(), &mode_count,
+ modes.data(), nullptr) != ERROR_SUCCESS) return 80.0f;
+ for (UINT32 index = 0; index < path_count; ++index) {
+ const auto& path = paths[index];
+ DISPLAYCONFIG_SOURCE_DEVICE_NAME name{};
+ name.header = {DISPLAYCONFIG_DEVICE_INFO_GET_SOURCE_NAME, sizeof(name),
+ path.sourceInfo.adapterId, path.sourceInfo.id};
+ if (DisplayConfigGetDeviceInfo(&name.header) != ERROR_SUCCESS ||
+ wcscmp(name.viewGdiDeviceName, device_name) != 0) continue;
+ DISPLAYCONFIG_SDR_WHITE_LEVEL white{};
+ white.header = {DISPLAYCONFIG_DEVICE_INFO_GET_SDR_WHITE_LEVEL, sizeof(white),
+ path.targetInfo.adapterId, path.targetInfo.id};
+ if (DisplayConfigGetDeviceInfo(&white.header) == ERROR_SUCCESS && white.SDRWhiteLevel > 0)
+ return static_cast(white.SDRWhiteLevel) * 80.0f / 1000.0f;
+ }
+ return 80.0f;
+ }
+
+ const TargetKind kind_;
+ const UINT64 source_id_;
+ Microsoft::WRL::ComPtr factory_;
+ HMONITOR monitor_ = nullptr;
+ CaptureColor color_;
+};
+
+// Produces an owned SDR texture before preview or output fanout. Ordinary SDR
+// keeps its exact copy path. HDR uses Windows effects on the capture GPU once,
+// rather than duplicating tone mapping in each encoder or reading pixels to CPU.
+class CaptureSdrConverter final {
+ public:
+ explicit CaptureSdrConverter(ID3D11Device* device) : device_(device) {
+ device_->GetImmediateContext(&context_);
+ }
+
+ Microsoft::WRL::ComPtr Convert(ID3D11Texture2D* source,
+ const CaptureColor& color) {
+ D3D11_TEXTURE2D_DESC description{};
+ source->GetDesc(&description);
+ if (description.Format != color.Format())
+ Fail("capture-color-format", "Captured frame and display color generation differ");
+ description.Usage = D3D11_USAGE_DEFAULT;
+ description.CPUAccessFlags = description.MiscFlags = 0;
+ description.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET;
+ Microsoft::WRL::ComPtr owned;
+ Check(device_->CreateTexture2D(&description, nullptr, &owned), "capture-owned-input");
+ context_->CopyResource(owned.Get(), source);
+ if (!color.hdr) return owned;
+ if (!drawing_) Initialize();
+
+ Microsoft::WRL::ComPtr surface;
+ Check(owned.As(&surface), "capture-hdr-surface");
+ const auto input_properties = D2D1::BitmapProperties1(D2D1_BITMAP_OPTIONS_NONE,
+ D2D1::PixelFormat(description.Format, D2D1_ALPHA_MODE_IGNORE));
+ Microsoft::WRL::ComPtr input;
+ Check(drawing_->CreateBitmapFromDxgiSurface(surface.Get(), &input_properties, &input),
+ "capture-hdr-bitmap");
+ Check(tone_map_->SetValue(D2D1_HDRTONEMAP_PROP_INPUT_MAX_LUMINANCE, color.peak_nits),
+ "capture-hdr-peak");
+ Check(tone_map_->SetValue(D2D1_HDRTONEMAP_PROP_OUTPUT_MAX_LUMINANCE, color.white_nits),
+ "capture-sdr-peak");
+ Check(white_level_->SetValue(D2D1_WHITELEVELADJUSTMENT_PROP_OUTPUT_WHITE_LEVEL, color.white_nits),
+ "capture-sdr-white");
+
+ description.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
+ Microsoft::WRL::ComPtr output;
+ Check(device_->CreateTexture2D(&description, nullptr, &output), "capture-sdr-texture");
+ surface.Reset();
+ Check(output.As(&surface), "capture-sdr-surface");
+ const auto output_properties = D2D1::BitmapProperties1(
+ D2D1_BITMAP_OPTIONS_TARGET | D2D1_BITMAP_OPTIONS_CANNOT_DRAW,
+ D2D1::PixelFormat(description.Format, D2D1_ALPHA_MODE_IGNORE));
+ Microsoft::WRL::ComPtr target;
+ Check(drawing_->CreateBitmapFromDxgiSurface(surface.Get(), &output_properties, &target),
+ "capture-sdr-bitmap");
+ tone_map_->SetInput(0, input.Get());
+ drawing_->SetTarget(target.Get());
+ Microsoft::WRL::ComPtr image;
+ srgb_->GetOutput(&image);
+ drawing_->BeginDraw();
+ drawing_->DrawImage(image.Get(), nullptr, nullptr,
+ D2D1_INTERPOLATION_MODE_NEAREST_NEIGHBOR, D2D1_COMPOSITE_MODE_SOURCE_COPY);
+ const HRESULT result = drawing_->EndDraw();
+ drawing_->SetTarget(nullptr);
+ tone_map_->SetInput(0, nullptr);
+ Check(result, "capture-hdr-tone-map");
+ return output;
+ }
+
+ private:
+ void Initialize() {
+ Microsoft::WRL::ComPtr factory;
+ Check(D2D1CreateFactory(D2D1_FACTORY_TYPE_SINGLE_THREADED, IID_PPV_ARGS(&factory)),
+ "capture-d2d-factory");
+ Microsoft::WRL::ComPtr dxgi;
+ Check(device_.As(&dxgi), "capture-d2d-dxgi");
+ Microsoft::WRL::ComPtr device;
+ Check(factory->CreateDevice(dxgi.Get(), &device), "capture-d2d-device");
+ Check(device->CreateDeviceContext(D2D1_DEVICE_CONTEXT_OPTIONS_NONE, &drawing_),
+ "capture-d2d-context");
+ const D2D1_RENDERING_CONTROLS controls{D2D1_BUFFER_PRECISION_16BPC_FLOAT, {0, 0}};
+ drawing_->SetRenderingControls(&controls);
+ Check(drawing_->CreateEffect(CLSID_D2D1HdrToneMap, &tone_map_), "capture-tone-map-effect");
+ Check(tone_map_->SetValue(D2D1_HDRTONEMAP_PROP_DISPLAY_MODE, D2D1_HDRTONEMAP_DISPLAY_MODE_SDR),
+ "capture-tone-map-mode");
+ Check(drawing_->CreateEffect(CLSID_D2D1WhiteLevelAdjustment, &white_level_), "capture-white-effect");
+ Check(white_level_->SetValue(D2D1_WHITELEVELADJUSTMENT_PROP_INPUT_WHITE_LEVEL, 80.0f),
+ "capture-input-white");
+ white_level_->SetInputEffect(0, tone_map_.Get());
+ Check(drawing_->CreateEffect(CLSID_D2D1ColorManagement, &srgb_), "capture-srgb-effect");
+ Microsoft::WRL::ComPtr linear, srgb;
+ Check(drawing_->CreateColorContext(D2D1_COLOR_SPACE_SCRGB, nullptr, 0, &linear), "capture-linear-color");
+ Check(drawing_->CreateColorContext(D2D1_COLOR_SPACE_SRGB, nullptr, 0, &srgb), "capture-srgb-color");
+ Check(srgb_->SetValue(D2D1_COLORMANAGEMENT_PROP_SOURCE_COLOR_CONTEXT, linear.Get()), "capture-input-color");
+ Check(srgb_->SetValue(D2D1_COLORMANAGEMENT_PROP_DESTINATION_COLOR_CONTEXT, srgb.Get()), "capture-output-color");
+ Check(srgb_->SetValue(D2D1_COLORMANAGEMENT_PROP_QUALITY, D2D1_COLORMANAGEMENT_QUALITY_BEST),
+ "capture-color-quality");
+ srgb_->SetInputEffect(0, white_level_.Get());
+ }
+
+ Microsoft::WRL::ComPtr device_;
+ Microsoft::WRL::ComPtr context_;
+ Microsoft::WRL::ComPtr drawing_;
+ Microsoft::WRL::ComPtr tone_map_, white_level_, srgb_;
+};
+
+} // namespace piik::capture::windows
diff --git a/native/capture/windows/capture_target.cpp b/native/capture/windows/capture_target.cpp
index 8393792..4675cb3 100644
--- a/native/capture/windows/capture_target.cpp
+++ b/native/capture/windows/capture_target.cpp
@@ -19,6 +19,7 @@
#include "capture_target.h"
#include "capture_border.h"
#include "capture_geometry.h"
+#include "capture_sdr.h"
#include
#include
@@ -497,8 +498,11 @@ HRESULT CaptureWithWgc(TargetKind kind, UINT64 source_id,
if (size.Width <= 0 || size.Height <= 0 || frame_ready == nullptr) {
result = HRESULT_FROM_WIN32(ERROR_INVALID_DATA);
} else {
+ windows::CaptureDisplayColor display_color(kind, source_id);
+ const auto color = display_color.Resolve();
+ windows::CaptureSdrConverter sdr(device.Get());
pool = Direct3D11CaptureFramePool::CreateFreeThreaded(
- capture_device, DirectXPixelFormat::B8G8R8A8UIntNormalized, 1,
+ capture_device, static_cast(color.Format()), 1,
size);
session = pool.CreateCaptureSession(item);
frame_token = pool.FrameArrived(
@@ -530,6 +534,7 @@ HRESULT CaptureWithWgc(TargetKind kind, UINT64 source_id,
::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>();
ComPtr texture;
result = access->GetInterface(IID_PPV_ARGS(&texture));
+ if (SUCCEEDED(result)) texture = sdr.Convert(texture.Get(), color);
D3D11_TEXTURE2D_DESC description{};
if (SUCCEEDED(result)) texture->GetDesc(&description);
if (SUCCEEDED(result) &&
@@ -1016,9 +1021,14 @@ HRESULT ValidateProcessTarget(DWORD pid, UINT64 expected_creation_time) {
}
HANDLE process = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid);
if (process == nullptr) return HRESULT_FROM_WIN32(GetLastError());
+ const HRESULT result = ValidateProcessTarget(process, expected_creation_time);
+ CloseHandle(process);
+ return result;
+}
+
+HRESULT ValidateProcessTarget(HANDLE process, UINT64 expected_creation_time) {
UINT64 actual_creation_time = 0;
HRESULT result = ReadProcessCreationTime(process, &actual_creation_time);
- CloseHandle(process);
if (SUCCEEDED(result) && actual_creation_time != expected_creation_time) {
result = HRESULT_FROM_WIN32(ERROR_INVALID_DATA);
}
diff --git a/native/capture/windows/capture_target.h b/native/capture/windows/capture_target.h
index e1ccf2c..b18c072 100644
--- a/native/capture/windows/capture_target.h
+++ b/native/capture/windows/capture_target.h
@@ -13,6 +13,7 @@ int WriteMicrophoneList();
HRESULT WriteSourcePreview(TargetKind kind, UINT64 source_id, DWORD pid,
UINT64 expected_creation_time);
HRESULT ValidateProcessTarget(DWORD pid, UINT64 expected_creation_time);
+HRESULT ValidateProcessTarget(HANDLE process, UINT64 expected_creation_time);
HRESULT ValidateWindowTarget(UINT64 source_id, DWORD pid,
UINT64 expected_creation_time);
HRESULT ValidateDisplayTarget(UINT64 source_id);
diff --git a/native/capture/windows/capture_target.test.cpp b/native/capture/windows/capture_target.test.cpp
index 296a53b..4f02628 100644
--- a/native/capture/windows/capture_target.test.cpp
+++ b/native/capture/windows/capture_target.test.cpp
@@ -15,6 +15,8 @@ int main() {
assert(GetProcessTimes(GetCurrentProcess(), &created, &exited, &kernel, &user));
const UINT64 creation_time = (static_cast(created.dwHighDateTime) << 32) |
created.dwLowDateTime;
+ assert(SUCCEEDED(piik::capture::ValidateProcessTarget(GetCurrentProcess(), creation_time)));
+ assert(FAILED(piik::capture::ValidateProcessTarget(GetCurrentProcess(), creation_time + 1)));
HANDLE previous_output = GetStdHandle(STD_OUTPUT_HANDLE);
HANDLE sink = CreateFileW(L"NUL", GENERIC_WRITE, FILE_SHARE_WRITE, nullptr,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr);
diff --git a/native/capture/windows/frame_converter.probe.cpp b/native/capture/windows/frame_converter.probe.cpp
new file mode 100644
index 0000000..10da309
--- /dev/null
+++ b/native/capture/windows/frame_converter.probe.cpp
@@ -0,0 +1,218 @@
+// Physical D3D11 check; opt in with build.ps1 -CheckColor. No desktop capture.
+#define wmain capture_main
+#include "main.cpp"
+#undef wmain
+#include
+
+namespace {
+
+constexpr std::array kRgb{
+ 0xff000000, 0xff202020, 0xff808080, 0xffebebeb,
+ 0xffffffff, 0xffff0000, 0xff00ff00, 0xff0000ff};
+constexpr std::array, 8> kYuv{{
+ {16, 128, 128}, {44, 128, 128}, {126, 128, 128}, {218, 128, 128},
+ {235, 128, 128}, {82, 90, 240}, {145, 54, 34}, {41, 240, 110}}};
+constexpr std::array, 8> kYuv709{{
+ {16, 128, 128}, {44, 128, 128}, {126, 128, 128}, {218, 128, 128},
+ {235, 128, 128}, {63, 102, 240}, {173, 42, 26}, {32, 240, 118}}};
+constexpr std::array, 8> kYuvFull{{
+ {0, 128, 128}, {32, 128, 128}, {128, 128, 128}, {235, 128, 128},
+ {255, 128, 128}, {76, 85, 255}, {150, 44, 21}, {29, 255, 107}}};
+constexpr std::array, 8> kYuv709Full{{
+ {0, 128, 128}, {32, 128, 128}, {128, 128, 128}, {235, 128, 128},
+ {255, 128, 128}, {54, 99, 255}, {182, 30, 12}, {18, 255, 116}}};
+
+ComPtr ColorBars(const DeviceContext& device, UINT width,
+ UINT height, DXGI_COLOR_SPACE_TYPE color) {
+ const bool rgb = color == kDesktopColor;
+ const auto& yuv = color == DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P709 ? kYuv709
+ : color == DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P709 ? kYuv709Full
+ : color == DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P601 ? kYuvFull : kYuv;
+ std::vector pixels(static_cast(width) * height * (rgb ? 4 : 2));
+ for (UINT row = 0; row < height; ++row) {
+ for (UINT column = 0; column < width; ++column) {
+ const size_t bar = column * kRgb.size() / width;
+ const size_t index = static_cast(row) * width + column;
+ if (rgb) {
+ for (UINT channel = 0; channel < 4; ++channel)
+ pixels[index * 4 + channel] = static_cast(kRgb[bar] >> (channel * 8));
+ } else {
+ pixels[index] = static_cast(yuv[bar][0]);
+ if (row < height / 2)
+ pixels[static_cast(width) * height + index] =
+ static_cast(yuv[bar][1 + column % 2]);
+ }
+ }
+ }
+ D3D11_TEXTURE2D_DESC description{};
+ description.Width = width;
+ description.Height = height;
+ description.MipLevels = description.ArraySize = description.SampleDesc.Count = 1;
+ description.Format = rgb ? DXGI_FORMAT_B8G8R8A8_UNORM : DXGI_FORMAT_NV12;
+ description.BindFlags = D3D11_BIND_RENDER_TARGET | D3D11_BIND_SHADER_RESOURCE;
+ D3D11_SUBRESOURCE_DATA data{};
+ data.pSysMem = pixels.data();
+ data.SysMemPitch = width * (rgb ? 4 : 1);
+ ComPtr texture;
+ Check(device.device->CreateTexture2D(&description, &data, &texture), "color-source");
+ return texture;
+}
+
+void CheckColors(const DeviceContext& device, ID3D11Texture2D* texture) {
+ D3D11_TEXTURE2D_DESC description{};
+ texture->GetDesc(&description);
+ description.Usage = D3D11_USAGE_STAGING;
+ description.BindFlags = 0;
+ description.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
+ ComPtr staging;
+ Check(device.device->CreateTexture2D(&description, nullptr, &staging), "color-readback");
+ device.context->CopyResource(staging.Get(), texture);
+ D3D11_MAPPED_SUBRESOURCE mapped{};
+ Check(device.context->Map(staging.Get(), 0, D3D11_MAP_READ, 0, &mapped), "color-map");
+ const auto* bytes = static_cast(mapped.pData);
+ bool correct = true;
+ for (size_t bar = 0; bar < kYuv.size(); ++bar) {
+ const auto x = description.Width * (2 * bar + 1) / (2 * kYuv.size());
+ const size_t y = description.Height / 2;
+ const size_t uv = (description.Height + y / 2) * mapped.RowPitch + (x & ~size_t{1});
+ const std::array actual{bytes[y * mapped.RowPitch + x], bytes[uv], bytes[uv + 1]};
+ for (size_t channel = 0; channel < 3; ++channel) {
+ if (std::abs(actual[channel] - kYuv[bar][channel]) > 2) {
+ std::cerr << "bar=" << bar << " channel=" << channel
+ << " expected=" << kYuv[bar][channel] << " actual=" << actual[channel] << '\n';
+ correct = false;
+ }
+ }
+ }
+ device.context->Unmap(staging.Get(), 0);
+ if (!correct) throw std::runtime_error("SDR range or matrix changed");
+}
+
+ComPtr LinearBars(const DeviceContext& device, UINT width,
+ UINT height, const std::array& nits) {
+ std::vector pixels(static_cast(width) * height * 4);
+ for (size_t pixel = 0; pixel < static_cast(width) * height; ++pixel) {
+ const auto value = DirectX::PackedVector::XMConvertFloatToHalf(nits[(pixel % width) * 8 / width] / 80.0f);
+ for (size_t channel = 0; channel < 3; ++channel) pixels[pixel * 4 + channel] = value;
+ pixels[pixel * 4 + 3] = DirectX::PackedVector::XMConvertFloatToHalf(1.0f);
+ }
+ D3D11_TEXTURE2D_DESC description{};
+ description.Width = width;
+ description.Height = height;
+ description.MipLevels = description.ArraySize = description.SampleDesc.Count = 1;
+ description.Format = DXGI_FORMAT_R16G16B16A16_FLOAT;
+ description.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET;
+ D3D11_SUBRESOURCE_DATA data{};
+ data.pSysMem = pixels.data();
+ data.SysMemPitch = width * 8;
+ ComPtr texture;
+ Check(device.device->CreateTexture2D(&description, &data, &texture), "hdr-source");
+ return texture;
+}
+
+std::array ReadGrayBars(const DeviceContext& device, ID3D11Texture2D* texture) {
+ D3D11_TEXTURE2D_DESC description{};
+ texture->GetDesc(&description);
+ description.Usage = D3D11_USAGE_STAGING;
+ description.BindFlags = 0;
+ description.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
+ ComPtr staging;
+ Check(device.device->CreateTexture2D(&description, nullptr, &staging), "hdr-readback");
+ device.context->CopyResource(staging.Get(), texture);
+ D3D11_MAPPED_SUBRESOURCE mapped{};
+ Check(device.context->Map(staging.Get(), 0, D3D11_MAP_READ, 0, &mapped), "hdr-map");
+ const auto* bytes = static_cast(mapped.pData);
+ std::array result{};
+ bool neutral = true;
+ for (size_t bar = 0; bar < result.size(); ++bar) {
+ const size_t pixel = description.Height / 2 * mapped.RowPitch +
+ description.Width * (2 * bar + 1) / 16 * 4;
+ result[bar] = bytes[pixel];
+ neutral = neutral && std::abs(result[bar] - bytes[pixel + 1]) <= 1 &&
+ std::abs(result[bar] - bytes[pixel + 2]) <= 1;
+ }
+ device.context->Unmap(staging.Get(), 0);
+ if (!neutral) throw std::runtime_error("HDR gray acquired a color cast");
+ return result;
+}
+
+void CheckHdr(const DeviceContext& device) {
+ CaptureSdrConverter capture(device.device.Get());
+ VideoProfile profile;
+ profile.width = 256;
+ profile.height = 128;
+ FrameConverter converter(device.device.Get(), profile);
+ for (const UINT width : {256u, 512u, 256u}) {
+ // SDR must retain the old pixels after an HDR frame, with the same owner.
+ auto sdr = ColorBars(device, width, width / 2, kDesktopColor);
+ auto owned = capture.Convert(sdr.Get(), {});
+ CheckColors(device, converter.Convert(owned.Get(), width, width / 2,
+ {static_cast(width), static_cast(width / 2)}, kDesktopColor).Get());
+ auto hdr = LinearBars(device, width, width / 2, {0, 5, 20, 80, 200, 400, 700, 1000});
+ for (const float white : {80.0f, 240.0f, 80.0f}) {
+ auto mapped = capture.Convert(hdr.Get(), {true, white, 1000.0f});
+ const auto levels = ReadGrayBars(device, mapped.Get());
+ std::cout << "HDR white=" << white << " levels=";
+ for (const auto level : levels) std::cout << level << ',';
+ std::cout << '\n';
+ if (levels.front() > 1 || levels.back() < 245)
+ throw std::runtime_error("HDR black/peak mapping is incorrect");
+ for (size_t index = 1; index < levels.size(); ++index)
+ if (levels[index] <= levels[index - 1])
+ throw std::runtime_error("HDR luminance detail was clipped before tone mapping");
+ auto next = capture.Convert(sdr.Get(), {});
+ if (ReadGrayBars(device, mapped.Get()) != levels)
+ throw std::runtime_error("Capture replacement modified a retained frame");
+ CheckColors(device, converter.Convert(next.Get(), width, width / 2,
+ {static_cast(width), static_cast(width / 2)}, kDesktopColor).Get());
+ }
+ }
+}
+
+} // namespace
+
+int main() {
+ try {
+ Runtime runtime;
+ const auto adapters = EnumerateAdapters();
+ if (adapters.empty()) throw std::runtime_error("A physical D3D11 adapter is required");
+ for (const auto& adapter : adapters) {
+ auto device = CreateDevice(adapter);
+ for (const UINT output_width : {256u, 128u}) {
+ VideoProfile profile;
+ profile.width = output_width;
+ profile.height = output_width / 2;
+ FrameConverter converter(device.device.Get(), profile);
+ // Reuse the same converter across RGB/NV12 and size replacements.
+ for (const UINT input_width : {256u, 512u, 256u}) {
+ for (const auto color : {kDesktopColor, kSdrVideoColor,
+ DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P709,
+ DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P601,
+ DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P709, kDesktopColor}) {
+ const UINT input_height = input_width / 2;
+ auto source = ColorBars(device, input_width, input_height, color);
+ auto output = converter.Convert(source.Get(), input_width, input_height,
+ {static_cast(input_width), static_cast(input_height)},
+ color);
+ try {
+ CheckColors(device, output.Get());
+ } catch (...) {
+ std::cerr << "adapter=" << adapter.index << " color=" << color
+ << " input-width=" << input_width << " output-width=" << output_width << '\n';
+ throw;
+ }
+ }
+ }
+ }
+ std::cout << "SDR conversion passed: " << NarrowAscii(adapter.description.Description) << '\n';
+ CheckHdr(device);
+ std::cout << "HDR conversion passed: " << NarrowAscii(adapter.description.Description) << '\n';
+ }
+ return 0;
+ } catch (const std::exception& error) {
+ std::cerr << error.what() << '\n';
+ if (const auto* failure = dynamic_cast(&error))
+ std::cerr << "stage=" << failure->stage() << " result=" << std::hex << failure->result() << '\n';
+ return 1;
+ }
+}
diff --git a/native/capture/windows/h264_decoder.h b/native/capture/windows/h264_decoder.h
index 0afad8c..f7e0bd5 100644
--- a/native/capture/windows/h264_decoder.h
+++ b/native/capture/windows/h264_decoder.h
@@ -1,6 +1,7 @@
#pragma once
#include "capture_error.h"
+#include "capture_color.h"
#include "capture_input.h"
@@ -56,6 +57,12 @@ class H264Decoder final {
H264Decoder(const H264Decoder&) = delete;
H264Decoder& operator=(const H264Decoder&) = delete;
+ DXGI_COLOR_SPACE_TYPE ColorSpace() const {
+ Microsoft::WRL::ComPtr type;
+ Check(decoder_->GetOutputCurrentType(0, &type), "current-output-type");
+ return windows::DecodedVideoColor(type.Get());
+ }
+
void Decode(const InputEnvelope& input, const Consumer& consume) {
if (input.kind != 2 || input.data.empty() || input.data.size() > 4 * 1024 * 1024)
throw std::runtime_error("Invalid H264 decoder input");
diff --git a/native/capture/windows/h264_encoder.cpp b/native/capture/windows/h264_encoder.cpp
index 3d52b55..a9c01cb 100644
--- a/native/capture/windows/h264_encoder.cpp
+++ b/native/capture/windows/h264_encoder.cpp
@@ -1,6 +1,7 @@
#include "h264_encoder.h"
#include "capture_output.h"
+#include "capture_color.h"
#include
#include
@@ -327,6 +328,7 @@ ComPtr CreateOutputType(
"output-type-major");
Check(type->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_H264),
"output-type-subtype");
+ SetSdrVideoColor(type.Get());
Check(type->SetUINT32(MF_MT_AVG_BITRATE, profile.bit_rate),
"output-type-bitrate");
Check(MFSetAttributeSize(type.Get(), MF_MT_FRAME_SIZE, profile.width,
@@ -354,6 +356,7 @@ ComPtr CreateInputType(
"input-type-major");
Check(type->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_NV12),
"input-type-subtype");
+ SetSdrVideoColor(type.Get());
Check(MFSetAttributeSize(type.Get(), MF_MT_FRAME_SIZE, profile.width,
profile.height),
"input-type-size");
diff --git a/native/capture/windows/main.cpp b/native/capture/windows/main.cpp
index c957ab2..8f34976 100644
--- a/native/capture/windows/main.cpp
+++ b/native/capture/windows/main.cpp
@@ -3,7 +3,7 @@
#include
#include
-#include
+#include
#include
#include
#include
@@ -32,6 +32,8 @@
#include "capture_target.h"
#include "capture_border.h"
#include "capture_geometry.h"
+#include "capture_color.h"
+#include "capture_sdr.h"
#include "process_audio.h"
#include "h264_encoder.h"
#ifndef PIIK_H264_FIXTURE
@@ -663,8 +665,9 @@ class FrameConverter final {
public:
explicit FrameConverter(
ID3D11Device* device,
- VideoProfile profile = kDefaultVideoProfile)
- : device_(device), profile_(profile) {
+ VideoProfile profile = kDefaultVideoProfile,
+ DXGI_COLOR_SPACE_TYPE output_color = kSdrVideoColor)
+ : device_(device), profile_(profile), output_color_(output_color) {
Check(device_->QueryInterface(IID_PPV_ARGS(&video_device_)),
"video-processor-device");
ComPtr context;
@@ -673,13 +676,30 @@ class FrameConverter final {
}
ComPtr Convert(ID3D11Texture2D* source, UINT32 width,
- UINT32 height, SIZE presentation) {
+ UINT32 height, SIZE presentation,
+ DXGI_COLOR_SPACE_TYPE input_color) {
if (source == nullptr || width == 0 || height == 0 ||
width > 16'384 || height > 16'384) {
Fail("capture-size", "captured window dimensions are invalid");
}
- if (!enumerator_ || width != input_width_ || height != input_height_) {
- Configure(width, height);
+ ComPtr normalized;
+ if (output_color_ == kSdrVideoColor && input_color != kDesktopColor &&
+ input_color != kSdrVideoColor) {
+ // YUV-to-YUV matrix/range changes are not reliable across tested drivers,
+ // even when reported as supported. Normalize through RGB on the GPU;
+ // ordinary RGB capture and canonical NV12 scaling stay single-pass.
+ if (!rgb_converter_)
+ rgb_converter_ = std::make_unique(device_.Get(), profile_, kDesktopColor);
+ normalized = rgb_converter_->Convert(source, width, height, presentation, input_color);
+ source = normalized.Get();
+ width = profile_.width;
+ height = profile_.height;
+ presentation = {static_cast(width), static_cast(height)};
+ input_color = kDesktopColor;
+ }
+ if (!enumerator_ || width != input_width_ || height != input_height_ ||
+ input_color != input_color_) {
+ Configure(width, height, input_color);
}
D3D11_TEXTURE2D_DESC output_description = {};
@@ -687,7 +707,8 @@ class FrameConverter final {
output_description.Height = profile_.height;
output_description.MipLevels = 1;
output_description.ArraySize = 1;
- output_description.Format = DXGI_FORMAT_NV12;
+ output_description.Format = output_color_ == kDesktopColor
+ ? DXGI_FORMAT_B8G8R8A8_UNORM : DXGI_FORMAT_NV12;
output_description.SampleDesc.Count = 1;
output_description.Usage = D3D11_USAGE_DEFAULT;
output_description.BindFlags = D3D11_BIND_RENDER_TARGET;
@@ -745,7 +766,7 @@ class FrameConverter final {
}
private:
- void Configure(UINT32 width, UINT32 height) {
+ void Configure(UINT32 width, UINT32 height, DXGI_COLOR_SPACE_TYPE input_color) {
D3D11_VIDEO_PROCESSOR_CONTENT_DESC description = {};
description.InputFrameFormat = D3D11_VIDEO_FRAME_FORMAT_PROGRESSIVE;
description.InputFrameRate = {profile_.frame_rate, 1};
@@ -762,20 +783,27 @@ class FrameConverter final {
ComPtr processor;
Check(video_device_->CreateVideoProcessor(enumerator.Get(), 0, &processor),
"video-processor-create");
+ // The incoming frame owns its range/matrix. Never infer these from size.
+ video_context_->VideoProcessorSetStreamColorSpace1(processor.Get(), 0, input_color);
+ video_context_->VideoProcessorSetOutputColorSpace1(processor.Get(), output_color_);
enumerator_ = std::move(enumerator);
processor_ = std::move(processor);
input_width_ = width;
input_height_ = height;
+ input_color_ = input_color;
}
ComPtr device_;
VideoProfile profile_;
+ const DXGI_COLOR_SPACE_TYPE output_color_;
+ std::unique_ptr rgb_converter_;
ComPtr video_device_;
- ComPtr video_context_;
+ ComPtr video_context_;
ComPtr enumerator_;
ComPtr processor_;
UINT32 input_width_ = 0;
UINT32 input_height_ = 0;
+ DXGI_COLOR_SPACE_TYPE input_color_ = kDesktopColor;
};
winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice
@@ -933,6 +961,7 @@ struct ProductArguments final {
VideoProfile profile;
std::vector outputs;
bool show_capture_border = false;
+ bool exclude_audio = false;
};
DegradationPreference ParseDegradationPreference(const wchar_t* value) {
@@ -1020,7 +1049,8 @@ ProductArguments ParseProductArguments(int count, wchar_t** values) {
}
if (count == 5 && std::wstring(values[1]) == L"--capture-audio") {
arguments.mode = ProductArguments::Mode::audio;
- arguments.target_kind = ParseTargetKind(values[2]);
+ arguments.exclude_audio = std::wstring(values[2]) == L"exclude";
+ arguments.target_kind = arguments.exclude_audio ? piik::capture::TargetKind::window : ParseTargetKind(values[2]);
} else if (count >= 16 && count <= 11 + 5 * piik::capture::kMaxOutputs && (count - 11) % 5 == 0 &&
std::wstring(values[1]) == L"--encoded-video" &&
std::wstring(values[2]) == L"--codec" &&
@@ -1126,7 +1156,7 @@ HRESULT RunAudioCapture(const ProductArguments& arguments) {
stop.get(), should_stop, ready, pcm);
}
return piik::capture::CaptureProcessAudio(
- arguments.pid, arguments.creation_time, stop.get(),
+ arguments.pid, arguments.creation_time, arguments.exclude_audio, stop.get(),
should_stop,
ready, pcm);
}
@@ -1153,6 +1183,7 @@ UINT32 WindowsBuild() {
void WriteCapabilityProbe() {
constexpr UINT32 kCreateForWindowMinimumBuild = 18'362;
constexpr UINT32 kProcessLoopbackMinimumBuild = 19'041;
+ constexpr UINT32 kProcessExclusionMinimumBuild = 20'348;
const UINT32 build = WindowsBuild();
bool window_capture = false;
if (build >= kCreateForWindowMinimumBuild) {
@@ -1167,6 +1198,8 @@ void WriteCapabilityProbe() {
build >= kProcessLoopbackMinimumBuild &&
piik::capture::ProcessAudioAvailable();
const bool system_audio = piik::capture::SystemAudioAvailable();
+ const bool process_audio_exclusion = build >= kProcessExclusionMinimumBuild &&
+ piik::capture::ProcessAudioAvailable(true);
std::vector adapters = EnumerateAdapters();
std::ostringstream output;
@@ -1178,6 +1211,7 @@ void WriteCapabilityProbe() {
<< ",\"microphone\":true,\"softwareVP8\":true"
<< ",\"processAudio\":"
<< (process_audio ? "true" : "false")
+ << ",\"processAudioExclusion\":" << (process_audio_exclusion ? "true" : "false")
<< ",\"systemAudio\":" << (system_audio ? "true" : "false")
<< ",\"adapters\":[";
for (size_t adapter_index = 0; adapter_index < adapters.size();
@@ -1361,6 +1395,7 @@ VideoEncoderSelection SelectVideoEncoder(
struct CaptureInput final {
ComPtr texture;
+ DXGI_COLOR_SPACE_TYPE color_space = kDesktopColor;
UINT32 width = 0;
UINT32 height = 0;
SIZE presentation{};
@@ -1448,7 +1483,7 @@ class OutputWorker final {
converted_height = height;
}
return converter->Convert(input->texture.Get(), input->width,
- input->height, input->presentation);
+ input->height, input->presentation, input->color_space);
}, profile_.width, profile_.height, input->timestamp, work.recovery, work.bitrate);
} catch (...) {
// Device loss is current shared evidence; an old codec failure only
@@ -1703,12 +1738,13 @@ void RunEncodedVideo(ProductArguments arguments) {
bool began = false;
UINT64 last_timestamp = 0;
auto submit = [&](ComPtr texture, UINT32 width, UINT32 height,
- UINT64 timestamp, UINT64 duration) {
+ UINT64 timestamp, UINT64 duration, DXGI_COLOR_SPACE_TYPE color) {
if ((began && timestamp <= last_timestamp) || width < arguments.profile.width || height < arguments.profile.height) {
Fail("encoded-input-output", "decoded frame changed the source timeline or output bounds");
}
auto input = std::make_shared();
input->texture = std::move(texture);
+ input->color_space = color;
input->width = width;
input->height = height;
input->presentation = {static_cast(width), static_cast(height)};
@@ -1735,12 +1771,12 @@ void RunEncodedVideo(ProductArguments arguments) {
Check(sample->GetSampleDuration(&duration), "decoded-duration");
if (timestamp < 0 || duration <= 0) Fail("decoded-timing", "decoder did not preserve source timing");
submit(OwnDecodedTexture(device, sample, width, height, stride), width, height,
- static_cast(timestamp), static_cast(duration));
+ static_cast(timestamp), static_cast(duration), h264->ColorSpace());
});
} else {
const auto decoded = vp8->Decode(input.data);
if (!decoded.nv12.empty()) submit(UploadDecodedNV12(device, decoded.nv12.data(), decoded.width, decoded.height),
- decoded.width, decoded.height, input.timestamp, input.duration);
+ decoded.width, decoded.height, input.timestamp, input.duration, kSdrVideoColor);
}
}
}
@@ -1801,8 +1837,11 @@ void RunVideoCapture(ProductArguments arguments) {
if (initial_size.Width <= 0 || initial_size.Height <= 0) {
Fail("capture-size", "selected window has no capturable content");
}
+ CaptureDisplayColor display_color(arguments.target_kind, arguments.source_id);
+ CaptureSdrConverter sdr(device.device.Get());
+ auto pool_format = display_color.Resolve().Format();
Direct3D11CaptureFramePool pool = Direct3D11CaptureFramePool::CreateFreeThreaded(
- capture_device, DirectXPixelFormat::B8G8R8A8UIntNormalized, 2,
+ capture_device, static_cast(pool_format), 2,
initial_size);
GraphicsCaptureSession capture_session = pool.CreateCaptureSession(item);
EnableFastCaptureUpdates(capture_session);
@@ -1964,15 +2003,17 @@ void RunVideoCapture(ProductArguments arguments) {
}
auto content_size = latest.ContentSize();
if (content_size.Width <= 0 || content_size.Height <= 0) continue;
+ const auto color = display_color.Resolve();
if (content_size.Width != pool_size.Width ||
- content_size.Height != pool_size.Height) {
+ content_size.Height != pool_size.Height || color.Format() != pool_format) {
latest.Close();
latest = nullptr;
latest_input.reset();
pool.Recreate(capture_device,
- DirectXPixelFormat::B8G8R8A8UIntNormalized, 2,
+ static_cast(color.Format()), 2,
content_size);
pool_size = content_size;
+ pool_format = color.Format();
continue;
}
ComPtr source = CaptureTexture(latest);
@@ -1983,13 +2024,7 @@ void RunVideoCapture(ProductArguments arguments) {
UINT32 content_height = std::min(
source_description.Height, static_cast(content_size.Height));
auto input = std::make_shared();
- auto owned_description = source_description;
- owned_description.Usage = D3D11_USAGE_DEFAULT;
- owned_description.CPUAccessFlags = 0;
- owned_description.MiscFlags = 0;
- owned_description.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET;
- Check(device.device->CreateTexture2D(&owned_description, nullptr, &input->texture), "capture-owned-input");
- device.context->CopyResource(input->texture.Get(), source.Get());
+ input->texture = sdr.Convert(source.Get(), color);
input->width = content_width;
input->height = content_height;
input->timestamp = timestamp;
diff --git a/native/capture/windows/process_audio.cpp b/native/capture/windows/process_audio.cpp
index ca84706..f439697 100644
--- a/native/capture/windows/process_audio.cpp
+++ b/native/capture/windows/process_audio.cpp
@@ -31,7 +31,8 @@ namespace {
class ActivationHandler final : public RuntimeClass<
RuntimeClassFlags, FtmBase, IActivateAudioInterfaceCompletionHandler> {
public:
- explicit ActivationHandler(HANDLE completed) : completed_(completed) {}
+ ActivationHandler() : completed_(CreateEventW(nullptr, FALSE, FALSE, nullptr)) {}
+ ~ActivationHandler() { if (completed_) CloseHandle(completed_); }
STDMETHODIMP ActivateCompleted(IActivateAudioInterfaceAsyncOperation* operation) override {
ComPtr activated;
@@ -45,6 +46,7 @@ class ActivationHandler final : public RuntimeClass<
HRESULT Result() const { return result_; }
ComPtr Client() const { return client_; }
+ HANDLE Completed() const { return completed_; }
private:
HANDLE completed_ = nullptr;
@@ -52,26 +54,29 @@ class ActivationHandler final : public RuntimeClass<
ComPtr client_;
};
-HRESULT ActivateProcessLoopback(DWORD pid, HANDLE completed,
+HRESULT ActivateProcessLoopback(DWORD pid, bool exclude,
ComPtr* client) {
AUDIOCLIENT_ACTIVATION_PARAMS parameters{};
parameters.ActivationType = AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK;
parameters.ProcessLoopbackParams.TargetProcessId = pid;
parameters.ProcessLoopbackParams.ProcessLoopbackMode =
- PROCESS_LOOPBACK_MODE_INCLUDE_TARGET_PROCESS_TREE;
+ exclude ? PROCESS_LOOPBACK_MODE_EXCLUDE_TARGET_PROCESS_TREE
+ : PROCESS_LOOPBACK_MODE_INCLUDE_TARGET_PROCESS_TREE;
PROPVARIANT variant{};
variant.vt = VT_BLOB;
variant.blob.cbSize = sizeof(parameters);
variant.blob.pBlobData = reinterpret_cast(¶meters);
- auto handler = Make(completed);
+ // The completion can outlive a timed-out caller; it owns its event handle.
+ auto handler = Make();
if (!handler) return E_OUTOFMEMORY;
+ if (!handler->Completed()) return HRESULT_FROM_WIN32(GetLastError());
ComPtr operation;
HRESULT result = ActivateAudioInterfaceAsync(VIRTUAL_AUDIO_DEVICE_PROCESS_LOOPBACK,
__uuidof(IAudioClient), &variant,
handler.Get(), &operation);
if (FAILED(result)) return result;
- if (WaitForSingleObject(completed, 10'000) != WAIT_OBJECT_0) return HRESULT_FROM_WIN32(WAIT_TIMEOUT);
+ if (WaitForSingleObject(handler->Completed(), 10'000) != WAIT_OBJECT_0) return HRESULT_FROM_WIN32(WAIT_TIMEOUT);
result = handler->Result();
if (SUCCEEDED(result)) *client = handler->Client();
return result;
@@ -114,6 +119,14 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo
return result;
}
HRESULT result = S_OK;
+ // Bind process audio to the selected lifetime, including while draining PCM.
+ // Exclusion must never become unfiltered system capture after its target dies.
+ const auto processAlive = [&]() -> HRESULT {
+ if (!process) return S_OK;
+ const DWORD wait = WaitForSingleObject(process, 0);
+ if (wait == WAIT_TIMEOUT) return S_OK;
+ return HRESULT_FROM_WIN32(wait == WAIT_OBJECT_0 ? ERROR_PROCESS_ABORTED : GetLastError());
+ };
ComPtr capture;
WAVEFORMATEX format{};
format.wFormatTag = WAVE_FORMAT_PCM;
@@ -131,6 +144,7 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo
if (SUCCEEDED(result)) result = client->GetService(IID_PPV_ARGS(&capture));
if (SUCCEEDED(result)) result = client->SetEventHandle(sampleReady);
if (SUCCEEDED(result)) result = client->Start();
+ if (SUCCEEDED(result)) result = processAlive();
if (SUCCEEDED(result)) result = ready_writer();
if (SUCCEEDED(result)) {
std::array silence{};
@@ -171,6 +185,7 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo
}
UINT32 frames = 0;
while (SUCCEEDED(result = capture->GetNextPacketSize(&frames)) && frames > 0) {
+ if (FAILED(result = processAlive())) break;
BYTE* data = nullptr;
DWORD flags = 0;
UINT64 devicePosition = 0;
@@ -198,6 +213,7 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo
}
capture->ReleaseBuffer(frames);
while (pending.size() - consumed >= kAudioBytesPerChunk) {
+ if (FAILED(result = processAlive())) break;
result = writer(nextTimestamp, pending.data() + consumed,
kAudioBytesPerChunk);
if (FAILED(result)) break;
@@ -218,13 +234,10 @@ HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loo
} // namespace
-bool ProcessAudioAvailable() {
- HANDLE completed = CreateEventW(nullptr, FALSE, FALSE, nullptr);
- if (completed == nullptr) return false;
+bool ProcessAudioAvailable(bool exclude) {
ComPtr client;
const HRESULT result =
- ActivateProcessLoopback(GetCurrentProcessId(), completed, &client);
- CloseHandle(completed);
+ ActivateProcessLoopback(GetCurrentProcessId(), exclude, &client);
return SUCCEEDED(result) && client != nullptr;
}
@@ -233,32 +246,24 @@ bool SystemAudioAvailable() {
return SUCCEEDED(ActivateEndpoint(&client)) && client != nullptr;
}
-HRESULT CaptureProcessAudio(DWORD pid, UINT64 expectedCreationTime,
+HRESULT CaptureProcessAudio(DWORD pid, UINT64 expectedCreationTime, bool exclude,
HANDLE stop_event, const StopProbe& stop_probe,
const ReadyWriter& ready_writer,
const PCMWriter& writer) {
HRESULT com_result = CoInitializeEx(nullptr, COINIT_MULTITHREADED);
if (FAILED(com_result)) return com_result;
- HRESULT result = ValidateProcessTarget(pid, expectedCreationTime);
- HANDLE process = nullptr;
- HANDLE completed = nullptr;
+ // Validate and watch the same handle so PID reuse cannot change the target.
+ HANDLE process = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION | SYNCHRONIZE, FALSE, pid);
+ HRESULT result = process ? ValidateProcessTarget(process, expectedCreationTime)
+ : HRESULT_FROM_WIN32(GetLastError());
ComPtr client;
if (SUCCEEDED(result)) {
- process = OpenProcess(SYNCHRONIZE, FALSE, pid);
- if (process == nullptr) result = HRESULT_FROM_WIN32(GetLastError());
- }
- if (SUCCEEDED(result)) {
- completed = CreateEventW(nullptr, FALSE, FALSE, nullptr);
- if (completed == nullptr) result = HRESULT_FROM_WIN32(GetLastError());
- }
- if (SUCCEEDED(result)) {
- result = ActivateProcessLoopback(pid, completed, &client);
+ result = ActivateProcessLoopback(pid, exclude, &client);
}
if (SUCCEEDED(result)) {
result = CaptureAudioFrames(client, process, true, stop_event, stop_probe,
ready_writer, writer);
}
- if (completed != nullptr) CloseHandle(completed);
if (process != nullptr) CloseHandle(process);
client.Reset();
CoUninitialize();
diff --git a/native/capture/windows/process_audio.h b/native/capture/windows/process_audio.h
index 55e0215..1090149 100644
--- a/native/capture/windows/process_audio.h
+++ b/native/capture/windows/process_audio.h
@@ -21,11 +21,11 @@ using PCMWriter =
using ReadyWriter = std::function;
using StopProbe = std::function;
-bool ProcessAudioAvailable();
+bool ProcessAudioAvailable(bool exclude = false);
bool SystemAudioAvailable();
HRESULT CaptureMicrophone(const std::wstring& device_id, HANDLE stop_event, const StopProbe& stop_probe,
const ReadyWriter& ready_writer, const PCMWriter& writer);
-HRESULT CaptureProcessAudio(DWORD pid, UINT64 expected_creation_time,
+HRESULT CaptureProcessAudio(DWORD pid, UINT64 expected_creation_time, bool exclude,
HANDLE stop_event, const StopProbe& stop_probe,
const ReadyWriter& ready_writer,
const PCMWriter& writer);
diff --git a/scripts/embedded-sfu-gate.ts b/scripts/embedded-sfu-gate.ts
index 007ade2..b18e8bd 100644
--- a/scripts/embedded-sfu-gate.ts
+++ b/scripts/embedded-sfu-gate.ts
@@ -65,6 +65,7 @@ async function main(): Promise {
const chromePath = process.env.CHROME_PATH?.trim();
if (!chromePath) throw new Error("CHROME_PATH is required");
const nativeArm = process.env.PIIK_EMBEDDED_SFU_NATIVE === "true";
+ const sfuOnly = process.env.PIIK_EMBEDDED_SFU_ONLY === "true";
const displayArm = process.env.PIIK_EMBEDDED_SFU_DISPLAY === "true";
if (displayArm && nativeArm) throw new Error("Display and Native capture are separate arms");
const twoRooms = process.env.PIIK_EMBEDDED_SFU_TWO_ROOMS === "true";
@@ -93,6 +94,7 @@ async function main(): Promise {
const sourceProfile = nativeArm ? await mkdtemp(join(tmpdir(), "piik-client-media-")) : null;
const [sourcePort, sourceDebugPort] = nativeArm ? [await reservePort(), await reservePort()] : [0, 0];
const result = { passed: false, arm: `${nativeArm ? "native" : displayArm ? "display" : "browser"}-${codec}-simulcast${twoRooms ? "-two-rooms" : ""}`, stage: "start",
+ sfuOnly, stereoSeparationDb: null as number[] | null,
startedAt: new Date().toISOString(), finishedAt: "", processes: [] as Array<{ role: string; pid: number | null }>,
high: null as { frames: number; width: number; height: number } | null,
low: null as { frames: number; width: number; height: number } | null,
@@ -144,7 +146,8 @@ async function main(): Promise {
PIIK_ENV: "development", PIIK_DEBUG: "route", LISTEN_HOST: "127.0.0.1", PORT: String(serverPort),
PIIK_LOG_DIR: join(BUILD_ROOT, "logs"),
PUBLIC_BASE_URL: origin, ALLOWED_ORIGINS: origin,
- SFU_LISTEN_HOST: "127.0.0.1", SFU_UDP_PORT: String(mediaPort), SFU_PUBLIC_IP: "127.0.0.1" },
+ SFU_LISTEN_HOST: "127.0.0.1", SFU_UDP_PORT: String(mediaPort), SFU_PUBLIC_IP: "127.0.0.1",
+ SFU_ONLY: String(sfuOnly) },
});
processStarted("server", server);
server.stdout.resume();
@@ -257,6 +260,7 @@ async function main(): Promise {
result.stage = "decoded-audio-energy";
await waitForSample(() => call(viewer!, "gate.audioEnergy()", 3_000),
(energy) => { result.audioEnergy.high = energy; return energy > 0; }, 5_000);
+ if (!nativeArm) result.stereoSeparationDb = await call(viewer, "gate.audioSeparation()", 5_000);
result.host = await call(host, "gate.snapshot()");
if (displayArm) {
result.stage = "display-background";
@@ -357,7 +361,9 @@ async function main(): Promise {
}
result.finishedAt = new Date().toISOString();
result.passed = !result.error && result.host?.published === true && result.host.simulcast &&
- result.viewer?.decoded === true && result.viewer.committed && result.viewer.peerFailures > 0 &&
+ result.viewer?.decoded === true && result.viewer.committed &&
+ (sfuOnly ? result.viewer.peerFailures === 0 : result.viewer.peerFailures > 0) &&
+ (nativeArm || result.stereoSeparationDb?.every(db => db > 20) === true) &&
result.viewer.audioKbps > 0 && result.audioEnergy.high > 0 && result.audioEnergy.low > result.audioEnergy.high &&
result.high !== null && result.low !== null && result.audioEnergy.recovered > 0 &&
result.publicationRetired && result.subscriptionRecovered && result.cleanup && result.nativeShareStopped &&
diff --git a/scripts/embedded-sfu-page.ts b/scripts/embedded-sfu-page.ts
index 83246c3..207de8e 100644
--- a/scripts/embedded-sfu-page.ts
+++ b/scripts/embedded-sfu-page.ts
@@ -239,18 +239,58 @@ export async function startHost(nativeSource?: { title: string; port: number },
video: { width: 1280, height: 720, frameRate: 15 }, audio: false,
}) : canvas.captureStream(15);
audio = new AudioContext();
- const tone = audio.createOscillator();
const gain = audio.createGain();
gain.gain.value = 0.05;
const destination = audio.createMediaStreamDestination();
- tone.connect(gain).connect(destination);
- tone.start();
+ const channels = audio.createChannelMerger(2);
+ [500, 1500].forEach((frequency, channel) => {
+ const tone = audio!.createOscillator();
+ tone.frequency.value = frequency;
+ tone.connect(channels, 0, channel);
+ tone.start();
+ });
+ channels.connect(gain).connect(destination);
await audio.resume();
source.addTrack(destination.stream.getAudioTracks()[0]!);
await connect(room, true);
return room;
}
+// Check decoded channels, not just /2 in SDP (Opus also uses that for mono).
+export async function audioSeparation(stream = video?.srcObject): Promise {
+ if (!(stream instanceof MediaStream) || !stream.getAudioTracks().length) throw new Error("No received audio");
+ const context = new AudioContext();
+ try {
+ const input = context.createMediaStreamSource(stream);
+ const splitter = context.createChannelSplitter(2);
+ const sink = context.createGain();
+ sink.gain.value = 0;
+ sink.connect(context.destination);
+ input.connect(splitter);
+ const analyzers = [0, 1].map(channel => {
+ const analyzer = context.createAnalyser();
+ analyzer.fftSize = 8192;
+ analyzer.smoothingTimeConstant = 0;
+ splitter.connect(analyzer, channel);
+ analyzer.connect(sink);
+ return analyzer;
+ });
+ await context.resume();
+ await new Promise(resolve => setTimeout(resolve, 1000));
+ return analyzers.map((analyzer, channel) => {
+ const bins = new Float32Array(analyzer.frequencyBinCount);
+ analyzer.getFloatFrequencyData(bins);
+ const level = (frequency: number) => {
+ const bin = Math.round(frequency * analyzer.fftSize / context.sampleRate);
+ return Math.max(-120, ...bins.slice(bin - 1, bin + 2));
+ };
+ return level(channel === 0 ? 500 : 1500) - level(channel === 0 ? 1500 : 500);
+ });
+ } finally {
+ await context.close();
+ }
+}
+
export async function startViewer(room: CreateRoomResponse): Promise {
await connect(room, false);
}
diff --git a/src/client/App.tsx b/src/client/App.tsx
index 7ce6a31..254f64b 100644
--- a/src/client/App.tsx
+++ b/src/client/App.tsx
@@ -397,8 +397,11 @@ function SiteAccessGate({
}
return (
diff --git a/src/client/components/living/CaptureSourcePicker.tsx b/src/client/components/living/CaptureSourcePicker.tsx
index be4e1c6..bd6cba3 100644
--- a/src/client/components/living/CaptureSourcePicker.tsx
+++ b/src/client/components/living/CaptureSourcePicker.tsx
@@ -32,6 +32,7 @@ export type NativeSourceList =
sources: NativeCaptureTarget[];
processAudio: boolean;
systemAudio: boolean;
+ processAudioExclusion?: boolean;
captureBorderControl?: boolean;
};
@@ -51,6 +52,7 @@ export function CaptureSourcePicker({
loadCameras,
initialAudio = true,
initialShowCaptureBorder = false,
+ initialExcludeAudio,
audioLocked = false,
selectionDisabled = false,
}: {
@@ -60,7 +62,7 @@ export function CaptureSourcePicker({
initialCamera?: string;
activeCameraVideo?: HTMLVideoElement | null;
loadCameras?: typeof loadCameraPreviews;
- onNative: (target: NativeCaptureTarget, audio: boolean, showCaptureBorder: boolean) => void;
+ onNative: (target: NativeCaptureTarget, audio: boolean, showCaptureBorder: boolean, excludeAudio?: NativeCaptureTarget) => void;
onPreview: (
target: NativeCaptureTarget,
signal?: AbortSignal,
@@ -72,6 +74,7 @@ export function CaptureSourcePicker({
initialTab?: SourceTab;
initialAudio?: boolean;
initialShowCaptureBorder?: boolean;
+ initialExcludeAudio?: NativeCaptureTarget;
audioLocked?: boolean;
selectionDisabled?: boolean;
}) {
@@ -81,6 +84,7 @@ export function CaptureSourcePicker({
const [tab, setTab] = useState(null);
const [shareAudio, setShareAudio] = useState(initialAudio);
const [showCaptureBorder, setShowCaptureBorder] = useState(initialShowCaptureBorder);
+ const [excludeAudio, setExcludeAudio] = useState(initialExcludeAudio);
const appDetected = nativeSources.kind === "ready" || nativeSources.kind === "failed" ||
nativeSources.kind === "unsupported" || nativeSources.kind === "incompatible";
const tabs = SOURCE_TABS.filter((value) => appDetected || value === "browser" || value === "camera");
@@ -122,6 +126,14 @@ export function CaptureSourcePicker({
? nativeSources.processAudio
: nativeSources.systemAudio);
const anyNativeAudio = sources.some(supportsAudio);
+ // Source enumeration already owns process identity; do not create a second app scanner.
+ const audioApplications = nativeSources.kind === "ready"
+ ? [...new Map(nativeSources.sources.filter(target => target.kind === "window")
+ .map(target => [`${target.pid}:${target.creationTime}`, target])).values()] : [];
+ const exclusionKey = excludeAudio?.kind === "window" ? `${excludeAudio.pid}:${excludeAudio.creationTime}` : "";
+ const canExcludeAudio = activeTab === "display" && nativeSources.kind === "ready" && nativeSources.processAudioExclusion;
+ const missingExclusion = !!excludeAudio && !audioApplications.some(target => `${target.pid}:${target.creationTime}` === exclusionKey);
+ const exclusionUnavailable = activeTab === "display" && shareAudio && !!excludeAudio && (!canExcludeAudio || missingExclusion);
const refreshing = activeTab === "camera" ? cameraSources.busy : nativeSources.kind === "loading";
const refreshLabel = t(refreshing ? "host.sourcePicker.loading" : "host.sourcePicker.refresh");
const audioAction = t(!anyNativeAudio ? "host.noAudio" : audioLocked
@@ -300,12 +312,12 @@ export function CaptureSourcePicker({
key={nativeCaptureTargetKey(target)}
target={target}
disabled={
- selectionDisabled || (audioLocked && shareAudio && !supportsAudio(target))
+ selectionDisabled || exclusionUnavailable || (audioLocked && shareAudio && !supportsAudio(target))
}
onPreview={onPreview}
onSelect={() =>
onNative(target, shareAudio && supportsAudio(target),
- supportsCaptureBorder && showCaptureBorder)
+ supportsCaptureBorder && showCaptureBorder, target.kind === "display" ? excludeAudio : undefined)
}
/>
))
@@ -352,6 +364,32 @@ export function CaptureSourcePicker({
) : null}
+ {canExcludeAudio && shareAudio && anyNativeAudio ? (
+ {
+ if (event.currentTarget.open) event.currentTarget.scrollIntoView({ block: "nearest" });
+ }}>
+
+
+ {!vis && {t("host.sourcePicker.excludeAudio")}{excludeAudio ? ` · ${excludeAudio.title}` : ""}}
+
+
+
;
+}
diff --git a/src/client/components/living/RoomReaction.tsx b/src/client/components/living/RoomReaction.tsx
new file mode 100644
index 0000000..51d1838
--- /dev/null
+++ b/src/client/components/living/RoomReaction.tsx
@@ -0,0 +1,119 @@
+import { useLayoutEffect, useRef, type CSSProperties, type RefObject } from "react";
+import { REACTION_DURATION_MS, REACTION_GLYPHS, isThrow, type ReactionId } from "../../../shared/room-interactions";
+import type { RoomInteraction } from "../../lib/room-interactions";
+import { identityHash } from "../../lib/identity-hash";
+import { Glyph } from "../../ui/icons";
+
+/** Props use the same small silhouette in the picker and on the couch. */
+export function ReactionIcon({ reaction }: { reaction: ReactionId }) {
+ if (!isThrow(reaction)) return {REACTION_GLYPHS[reaction]};
+ return ;
+}
+
+/** Fixed, event-seeded accents: no independent timer or mutable random stream. */
+function ReactionAccents({ reaction, seed }: { reaction: ReactionId; seed: number }) {
+ return ;
+}
+
+/** Event expiry owns lifetime. This component owns only placement and its own impact. */
+export function RoomReaction({ reaction, area, now }: {
+ reaction: RoomInteraction & { expiresAt: number }; area: RefObject; now: () => number;
+}) {
+ const flight = useRef(null);
+ const payload = reaction.payload;
+ useLayoutEffect(() => {
+ const root = area.current;
+ const element = flight.current;
+ if (!root || !element || payload.kind !== "reaction") return;
+ const pawn = (peer: string) => root.querySelector(`[data-room-peer="${CSS.escape(peer)}"] .lr-person`);
+ const sender = pawn(reaction.sender.peerId);
+ const recipient = pawn(payload.targetPeerId ?? reaction.sender.peerId);
+ if (!sender || !recipient) return;
+ let frame = 0;
+ let impact: Animation | undefined;
+ const place = () => {
+ if (now() >= reaction.expiresAt || !sender.isConnected || !recipient.isConnected) {
+ element.style.visibility = "hidden";
+ return;
+ }
+ frame = requestAnimationFrame(place);
+ if (document.hidden || !root.getClientRects().length) {
+ element.style.visibility = "hidden";
+ impact?.cancel();
+ return;
+ }
+ const rect = element.getBoundingClientRect();
+ // Transforms do not trigger ResizeObserver. Follow the actual head for
+ // this event's remaining lifetime, including hover, lean and impact.
+ const from = sender.querySelector(".lr-person-head")?.getBoundingClientRect();
+ const to = recipient.querySelector(".lr-person-head")?.getBoundingClientRect();
+ if (!from || !to) { element.style.visibility = "hidden"; return; }
+ element.style.setProperty("--from-x", `${from.x - rect.x + from.width / 2}px`);
+ element.style.setProperty("--from-y", `${from.y - rect.y + from.height / 2}px`);
+ element.style.setProperty("--to-x", `${to.x - rect.x + to.width / 2}px`);
+ element.style.setProperty("--to-y", `${to.y - rect.y + to.height / 2}px`);
+ element.style.setProperty("--throw-direction", from.x > to.x ? "-1" : "1");
+ element.style.visibility = "visible";
+ };
+ place();
+ const body = recipient.querySelector(".lr-reaction-body");
+ if (isThrow(payload.reaction) && payload.targetPeerId && body &&
+ !window.matchMedia("(prefers-reduced-motion: reduce)").matches && !root.closest("[data-comic-reduced-motion]")) {
+ const elapsed = Math.max(0, REACTION_DURATION_MS - (reaction.expiresAt - now()));
+ const delay = REACTION_DURATION_MS * .32 - elapsed;
+ if (delay > -560) {
+ const direction = sender.getBoundingClientRect().x > recipient.getBoundingClientRect().x ? -1 : 1;
+ impact = body.animate([
+ { transform: "none" },
+ { transform: payload.reaction === "tomato" ? `rotate(${direction * 6}deg) scale(1.05,.95)` : `translateX(${direction * 4}px) rotate(${direction * 8}deg)` },
+ { transform: `rotate(${-direction * 3}deg)` },
+ { transform: "none" },
+ ], { duration: 560, delay, easing: "ease-out" });
+ }
+ }
+ return () => { cancelAnimationFrame(frame); impact?.cancel(); };
+ }, [area, reaction, payload, now]);
+ if (payload.kind !== "reaction") return null;
+ const gift = Boolean(payload.targetPeerId && !isThrow(payload.reaction));
+ return
+ {
+ // CSS restarts after display:none or reduced motion. Event expiry still
+ // owns time, including a late mount when effects are shown again.
+ for (const animation of (event.target as Element).getAnimations()) {
+ if (animation instanceof CSSAnimation && animation.animationName.startsWith("lr-reaction-")) {
+ animation.currentTime = Math.min(REACTION_DURATION_MS, Math.max(0, REACTION_DURATION_MS - (reaction.expiresAt - now())));
+ }
+ }
+ }}>{gift && {reaction.sender.displayName}}
+
+
+