okooooooo/Piik
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add room interactions and media sharing improvements

Browse Source 
## Problem and outcome / 问题与结果

This phase adds lightweight room interaction without making media depend on chat,
and adds Windows source-audio exclusion and optional SFU-only hosting at their
existing owners. It also corrects confirmed encoder/capture ownership and native
color-conversion defects. Host microphone commentary remains on the shared media
path; room voice and 4K are outside this phase.

## Release notes

### 中文

这是一次功能更新：Piik 加入了文字聊天与表情互动，并完善了 Windows 声音采集、画面颜色和自部署选项。

#### 新功能

- **边看边聊。** 在可移动、可缩放的聊天窗口中发送消息，或开启弹幕。停止分享后，房间中的朋友仍可继续聊天。
- **表情互动。** 点击沙发上的人物，给自己发表情，或把番茄等表情送给朋友；各端按同一事件显示动画。
- **排除应用声音。** Windows App 分享整个屏幕时，可排除一个应用及其子进程的声音，适合同时使用语音软件。所选应用退出后，来源声音停止，避免恢复为未过滤的系统声音。
- **可选的纯 SFU 模式。** 自部署管理员可让共享画面和声音统一通过媒体服务器转发。默认仍为优先 P2P；App 公网邀请保持纯 P2P。
- **Windows HDR 转 SDR。** 原生采集与缩略图自动将 HDR 来源转换为 SDR，沿用现有观看链路。

#### 修复与改进

- 修复原生颜色转换的范围与矩阵处理，避免已复现的暗部丢失、高光裁切和颜色偏差。
- 修复共享编码器切换时使用过期需求或过期帧信息的问题，保护正在观看的画面；修复换源后旧采集事件可能影响新来源的问题。
- 统一新控件、浮窗与来源选择文案，补充中英文使用说明。

#### 使用与升级

聊天与表情需要新版站点页面和 Server；声音排除需要新版页面与支持此能力的 Windows App。聊天只保留本页最近收到的消息，刷新后清空，不补发历史，且不提供端到端加密。

HDR 转换输出为 SDR，不代表端到端 HDR 或多声道支持。既有房主麦克风与立体声能力保留。

### English

This feature update brings text chat and reactions to Piik, alongside improvements to Windows audio capture, picture colors and self-hosting options.

#### New features

- **Chat while watching.** Send messages in a movable, resizable window, or enable on-screen chat. Friends can keep chatting after the host stops sharing.
- **Reactions.** Select a person on the sofa to react above yourself or send a tomato and other reactions to a friend. Participants animate the same shared event.
- **Exclude application audio.** When sharing a whole screen with Windows App, exclude one application and its child processes, useful when using a separate voice app. Source audio stops if the selected app exits, preventing a return to unfiltered system sound.
- **Optional SFU-only hosting.** Site operators can require shared video and audio to pass through the media server. P2P-first remains the default; App Public invite remains P2P-only.
- **Windows HDR-to-SDR conversion.** Native capture and source thumbnails automatically convert HDR sources to SDR for the existing viewing path.

#### Fixes and improvements

- Correct native color-conversion ranges and matrices, addressing reproduced crushed shadows, clipped highlights and color shifts.
- Correct stale demand and frame evidence during shared-encoder handoffs to protect the current picture. Prevent events from a replaced capture from affecting its successor.
- Align new controls, floating panels and source-picker copy, with updated Chinese and English guides.

#### Using this update

Chat and reactions require updated site pages and Server. Audio exclusion requires updated pages and a Windows App that supports the capability. Chat retains only recent messages received by the current page, clears on refresh and does not replay history. It is not end-to-end encrypted.

HDR conversion produces SDR; it does not add end-to-end HDR or surround sound. Existing host microphone commentary and stereo support remain available.

## Verification / 验证

- Web type checks, 78 test files / 1275 tests and production build: PASS.
- Go core tests, formatting/vet and target builds: PASS; Darwin cgo remains platform-limited on this Windows runner.
- Native build/check/color probes and synthetic HDR-to-SDR H.264/VP8 browser reception: PASS.
- Access/privacy gate, Native H.264 SFU-only media gate and local Auto/H.264 Host gate: PASS.
- Three real production pages: chat/reactions, stop/resume, idle-room join, host departure/return, offline draft recovery and invitation revocation: PASS.
- Synthetic parent/child process-audio exclusion and target lifetime: PASS; no raw audio retained.
- Windows candidate archive extraction and packaged Local startup/shutdown: PASS.
- Documentation and website build: PASS.
- [Candidate CI](https://github.com/TNTcraftHIM/Piik/actions/runs/36873766646) at
  `f26cd208d03bc0dcbe353f8cac217f993216b0a3`: Go core and race checks, Web tests,
  Server/container checks and Windows/macOS/Linux App packaging all PASS.
  Publication was skipped as intended.

The owner accepted the UI/model and bounded local walkthrough for phase closure.
Physical HDR/mixed displays, mobile keyboard,
real voice applications across output devices and macOS/Linux capture remain
unverified. Existing field reports without matched evidence are not claimed
resolved. The owner authorized v1.7.0 publication on 2026-10-02. The final
documentation-only commit records that authorization; product code matches
the verified candidate above.

## Impact / 影响

Authorized version: **v1.7.0**, matching the local release planner. Current wire
contracts remain Browser/server 23, Native control 9 and capture 7. Optional
capabilities gate new functionality; existing room/media authority remains owned
by the existing services. No new ports, message persistence, room voice or
automatic room creation on page visit. Updated README, guides, standards,
research evidence and TODO stay with the candidate phase.
This commit is contained in:
AAA authored and GitHub committed 2026-10-02 01:36:45 +08:00
1 parent 95cc11f2c1
commit 1b9f5bd2a2
141 files changed
+7004 -986
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.env.example
+2
View File
@@ -35,6 +35,8 @@ SFU_UDP_PORT=
SFU_LISTEN_HOST=0.0.0.0
# Optional advertised IPv4 override when the listener's address is behind NAT.
SFU_PUBLIC_IP=
# Require server media for every room instead of P2P-first. Requires SFU_UDP_PORT.
SFU_ONLY=false
# Ordinary peer ICE discovery. Production requires at least one STUN URL.
# These are advertised addresses, not local listener addresses. They may use
AGENTS.md
+1 -1
View File
@@ -30,7 +30,7 @@
[routing/transport](./docs/standards/routing-transport.md),
[media quality](./docs/standards/media-quality.md), and
[presentation/lifecycle](./docs/standards/presentation-lifecycle.md).
- Media stays automatic and P2P-first; central services provide room authority,
- Media stays automatic and defaults to P2P-first; central services provide room authority,
signaling, STUN, observability, and bounded embedded SFU/UDP fallback using
mature LiveKit media components under [ADR-0013](./docs/adr/0013-embedded-node-local-media.md). Route
changes must preserve the single-graph, single-operation model in
README.md
+1
View File
@@ -34,6 +34,7 @@ watch in their browsers.
- **Share from Piik App or an existing Piik site.** Choose a screen, window, browser tab or camera; available sources and audio depend on the platform.
- **Direct connections first.** Media travels between participants where possible (P2P). A self-hosted server can provide automatic media forwarding (SFU) as a fallback.
- **Rooms you control.** Invitations, room codes and optional room passwords, for one host and up to 20 viewers.
- **Chat while watching.** Send messages and reactions to friends, with optional on-screen chat.
- **Flexible viewing.** Light and dark themes, playback controls, picture-in-picture and a connection topology view.
- **One program to host a site.** Web UI, room management and optional media forwarding are packaged together.
README.zh-CN.md
+1
View File
@@ -33,6 +33,7 @@ Piik 是一个免费开源的屏幕共享工具，适合游戏围观、一起看
- **用 Piik App 或已有 Piik 站点分享。** 支持屏幕、窗口、浏览器标签页和摄像头；可选来源与音频能力取决于平台。
- **优先直接连接。** 画面优先在设备之间传输（P2P），自建站点可启用媒体转发（SFU）自动兜底。
- **房间由你管理。** 支持邀请链接、房间号和可选的房间密码，一位房主、最多 20 位观众。
- **边看边聊。** 发送文字、和朋友互送表情，也可开启弹幕。
- **灵活的观看方式。** 支持深浅主题、播放控制、画中画和连接拓扑查看。
- **一个程序就能建站。** 网页、房间管理和可选的媒体转发打包在同一服务端程序中。
deploy/container/.env.example
+2
View File
@@ -8,6 +8,8 @@ SITE_ACCESS_PASSWORD=
# Optional automatic media fallback: set BOTH values and allow UDP 7882.
# SFU_UDP_PORT=7882
# SFU_PUBLIC_IP=203.0.113.10
# Optional: require SFU for all rooms instead of P2P-first (requires SFU_UDP_PORT).
# SFU_ONLY=true
# Optional NAT survey: also allow UDP 3479 and 3480.
# NAT_PREDICTION_ENABLED=true
docs/README.md
+2 -1
View File
@@ -99,7 +99,8 @@ and [website preview and publishing](./operations/website.md).
| Why a design was chosen | [Architecture decisions](./adr/) |
| Measurements and platform limits | [Research](./research/) and [verification status](./verification-status.md) |
| Camera and Host microphone | [Capture scope and limits](./research/camera-and-microphone.md) |
| Room voice proposal and media-fidelity boundaries | [Design assessment](./research/room-voice.md) |
| Unreleased text, reactions and floating room tools | [Room-interaction candidate](./research/room-interactions.md) |
| Stereo verification and HDR-to-SDR assessment | [Media fidelity](./research/media-fidelity.md) |
## Shared conventions
docs/adr/0005-automatic-hybrid-media-routing.md
+8
View File
@@ -28,6 +28,14 @@ Hosted and Local always use this controller. The former rollout switch and
separate Host-star authority are removed; SFU availability is still selected by
configuration and the room's privacy policy, not by another routing mode.
The optional Hosted `SFU_ONLY` policy restricts this controller's candidate
eligibility to SFU. It reuses the publication/subscription ledger, deadlines,
first-frame commit and recovery. It does not create another graph or scheduler.
It disables Peer relay and background Peer work; admission failure does not
relax the restriction. The service rejects an explicit peer-only request instead
of treating it as permission to forward media. Default P2P-first and App Local
behavior are unchanged.
### One Capacity Rule
Every non-server endpoint uses one server-authoritative steady outbound media-
docs/adr/0008-window-scoped-audio-capture.md
+10 -3
View File
@@ -2,7 +2,7 @@
- Status: accepted for Browser capture and the Windows App audio paths
- Date: 2026-08-21
- Last updated: 2026-08-27
- Last updated: 2026-10-01
## Context
@@ -26,8 +26,8 @@ platform probe and physical gate report it.
Track presence does not prove its source or isolation.
3. Returned screen-audio tracks use `contentHint = "music"`. Current bounded
sender ceilings and narrow Opus stereo answer normalization are owned by
[media quality](../standards/media-quality.md). There is no Web Audio mixer,
resampler, second representation, or independent audio clock.
[media quality](../standards/media-quality.md). Audio scope adds no second
representation or independent clock; Host commentary uses the existing mixer.
4. Browser source switching replaces the current screen-audio track in the same
persistent media stream. Local Viewer play, mute, and volume stay inside the
native media element.
@@ -38,6 +38,13 @@ platform probe and physical gate report it.
probe and physical gate pass. WGC/MF video and the App media boundary are owned by
[ADR-0010](./0010-cross-platform-client-runtime.md); measurements are in
[Native App media](../research/native-client-media.md).
7. Windows App screen sharing can optionally exclude one selected process tree
through WASAPI's exclusion mode, on build 20348+ with a successful capability
probe. This mode spans all render endpoints; it is not restricted to the
default playback device. Reuse source enumeration and the existing PCM/mixed
output. Generic Browser capture and arbitrary unrelated process sets are not
covered. [Media quality](../standards/media-quality.md#screen-audio) owns
selection lifetime and failure behavior.
## Consequences
docs/guide/getting-started.md
+31
View File
@@ -7,6 +7,7 @@ watching, using Piik online, using Piik App and sharing through an existing site
[Join a room](#join-a-friends-room) · [Use online](#use-piik-online) ·
[Piik App](#share-with-piik-app) · [Existing site](#share-from-an-existing-site) ·
[Chat and reactions](#chat-and-reactions) ·
[Troubleshooting](#when-something-gets-in-the-way)
Viewers need a browser and an invitation. To start sharing, download Piik App
@@ -154,6 +155,36 @@ then select a picture card to start sharing. Device names and previews depend on
browser permissions; while a camera is shared, other cameras are listed by name.
Use headphones to keep speaker sound from feeding back into the microphone.
### Exclude a voice app from screen audio
On supported Windows Apps, open the **Screens** tab and enable **System sound**.
Under **Exclude application audio**, select a window belonging to the voice app,
then choose the screen to share. Open the app's window and refresh if it is absent.
This excludes one process and its children. Other apps remain audible, including
those playing through other output devices.
It does not mute your Piik microphone or prevent speaker sound reaching a microphone.
If the excluded app closes or restarts, source audio stops. Use **Switch source**
and select it again. Turning source audio off retains the exclusion choice;
select **None** to restore unfiltered system audio. This control requires an updated page and a Windows App
whose capability probe supports process exclusion; it is not a Browser-capture filter.
## Chat and reactions
Once the host starts the first share, participants can select **Chat** below the
sofa to send messages. Select a person on the sofa to open their reaction menu:
your own reactions appear above you; reactions sent to a friend fly towards them.
The chat window can be moved and resized without rearranging the page.
**On screen**, beside the message box or in the viewer's playback bar, shows new
messages over the picture. It is off by default and only changes your own view.
System video fullscreen and picture in picture cannot show these overlays.
Stopping a share keeps the room's chat available. App Local and Public invite
rooms require the App to stay open. Each page retains only its latest received
messages: refreshing clears them, and late joiners receive no earlier history.
Chat travels through the room's server and is **not end-to-end encrypted**.
## When something gets in the way
For sound, capture, startup or playback issues, see [Troubleshooting](./troubleshooting.md).
docs/guide/getting-started.zh-CN.md
+27 -1
View File
@@ -6,7 +6,8 @@ Piik 支持一位房主分享屏幕，最多 20 位受邀观众通过浏览器
本指南介绍加入房间、使用在线版、使用 Piik App，以及通过已有站点分享的方法。
[加入房间](#加入朋友的房间) · [在线版](#使用在线版) ·
[Piik App](#用-piik-app-分享) · [已有站点](#使用已有站点分享) · [问题排查](#遇到问题了)
[Piik App](#用-piik-app-分享) · [已有站点](#使用已有站点分享) ·
[聊天与表情](#聊天与表情) · [问题排查](#遇到问题了)
观众只需要浏览器和邀请链接。
分享画面推荐下载 Piik App，选择 **公网邀请**。
@@ -133,6 +134,31 @@ Linux 原生采集还需要[系统组件](../../native/capture/linux/README.md)
设备名称和可用预览取决于浏览器权限；正在分享摄像头时，其他摄像头只显示名称。
建议佩戴耳机，避免扬声器声音被麦克风再次收进去。
### 分享屏幕时排除语音软件的声音
在支持此功能的 Windows App 中，打开 **屏幕** 页签并开启 **系统声音**，
在 **排除应用声音** 中选择语音软件的窗口，再选择要分享的屏幕。
未列出的应用请先打开窗口，再刷新。此功能排除所选进程及其子进程，
其他应用的声音仍会被分享，包括输出到其他播放设备的声音。
它不影响 Piik 麦克风，也不能消除扬声器被麦克风收音产生的回声。
所选应用关闭或重启后，来源声音会停止；请通过 **切换来源** 重新选择。
暂时关闭来源声音会保留排除选择；需要恢复完整的系统声音时，选择 **不排除**。
此控件需要新版页面和通过进程排除能力检测的 Windows App，不适用于浏览器自带的分享选择器。
## 聊天与表情
房主首次开始分享、创建房间后，参与者可以点击沙发下方的 **聊天** 发送消息。
点击沙发上的人物可打开表情菜单：给自己发的表情出现在头顶，送给朋友的表情会飞向对方。
聊天窗口可以移动和缩放，不会挤动页面。
输入框旁和观众播放控制栏中的 **弹幕** 可以把新消息显示在画面上，默认关闭，只影响自己的观看。
系统视频全屏和画中画无法显示弹幕。
停止分享后仍可继续聊天；App 的本地房间和公网邀请需要保持 App 运行。
每个页面只保留本页最近收到的消息，刷新后清空，后来加入的人看不到此前的记录。
聊天消息经过房间所在的服务器，**不提供端到端加密**。
## 遇到问题了
声音、采集、启动或播放遇到问题时，先看[问题排查](./troubleshooting.zh-CN.md)。
docs/guide/troubleshooting.md
+3 -2
View File
@@ -147,8 +147,9 @@ Try these in order, stopping when the picture arrives:
6. **Invite a friend on another network.** A Viewer who can receive the picture
and has spare forwarding capacity may give Piik another path to you. Piik
chooses this automatically; extra people help only when those connections work.
7. **Use a site with media fallback.** Its operator must enable SFU forwarding,
and the Host must turn off **Privacy mode** before sharing. The public
7. **Use a site with media fallback.** Its operator must enable SFU forwarding.
If the site offers **Privacy mode**, the Host must turn it off before sharing.
Sites configured for **Server media** already require SFU. The public
`demo.piik.tv` site and the App's **Public invite** mode do not provide SFU.
[Self-hosting](../operations/self-hosting.md#optional-media-fallback) is an
advanced option. Both P2P and SFU media currently use UDP; a network that blocks
docs/guide/troubleshooting.zh-CN.md
+2 -1
View File
@@ -123,7 +123,8 @@ NAT（网络地址转换）让多台设备共用一个上网地址。不同路
修改路由器工作模式前，先备份配置。
6. **请另一网络的朋友加入。** 如果对方能正常看到画面，且有余力转发，Piik 可能经由这位观众
把画面送到你这里。线路由 Piik 自动选择，能否帮上忙取决于这些连接是否可用。
7. **使用有媒体兜底的站点。** 管理员需要先启用 SFU 转发，房主在分享前关闭 **隐私模式**。
7. **使用有媒体兜底的站点。** 管理员需要先启用 SFU 转发。如果页面提供 **隐私模式**，
房主应在分享前关闭；配置为 **服务器转发** 的站点已经强制使用 SFU。
公开站点 `demo.piik.tv` 和 App 的 **公网邀请** 模式均没有 SFU 兜底。
有服务器的用户也可以按 [部署指南](../operations/self-hosting.zh-CN.md#可选启用媒体兜底) 配置，属于进阶操作。
目前 P2P 和 SFU 媒体都使用 UDP；网络完全禁止 UDP 时，仍需换网络或由管理员放行。
docs/operations/self-hosting.md
+9
View File
@@ -112,6 +112,9 @@ permission; `self` allows this site to request access.
Allow **TCP 80/443** for HTTPS and **UDP 3478** for STUN in the server firewall
and cloud security group. Keep TCP 8787 private. The STUN hostname must resolve
directly to the server; a CDN HTTP proxy does not forward its UDP traffic.
See the [complete port table](../standards/configuration.md#public-and-private-ports)
for optional services. Embedded SFU media shares the single UDP port set by
`SFU_UDP_PORT`.
Open `https://share.example.com/healthz`; it should return `{"status":"ok"}`.
Then open the site, share a screen and join from another device. This initial
@@ -122,9 +125,15 @@ configuration uses P2P media, so participants need a usable UDP path.
Add `SFU_UDP_PORT=7882` to `.env`, allow UDP 7882, and restart Piik to enable
automatic SFU fallback. When the server is behind NAT, also set `SFU_PUBLIC_IP`
to its reachable public IPv4 address. The same binary provides the fallback.
Allow server forwarding capacity and outbound bandwidth for Viewers actually
using the SFU.
The Host must turn off **Privacy mode** before sharing to allow this route.
Both direct and SFU media need a usable UDP connection.
To send every room through your server, also set `SFU_ONLY=true` and restart.
The sharing settings will show **Server media**. Plan server bandwidth for every
Viewer; this mode does not fall back to P2P if the server route fails.
## Keep it running and update
For automatic startup, use the [systemd or container guide](./service-management.md).
docs/operations/self-hosting.zh-CN.md
+7
View File
@@ -101,6 +101,8 @@ nginx 可参考[配置示例](../../deploy/nginx/piik.conf.example)。如果沿
在服务器防火墙和云平台安全组中放行 **TCP 80/443**（HTTPS）和 **UDP 3478**（STUN）。
TCP 8787 仅供本机反向代理访问。STUN 域名需要直接解析到服务器，不能只经过 CDN 的 HTTP 代理。
可选服务的端口见[完整端口表](../standards/configuration.md#public-and-private-ports)。
内置 SFU 的媒体流复用 `SFU_UDP_PORT` 指定的单个 UDP 端口。
打开 `https://share.example.com/healthz`，应返回 `{"status":"ok"}`。
随后打开站点，分享一个画面，并用另一台设备加入验证。
@@ -111,9 +113,14 @@ TCP 8787 仅供本机反向代理访问。STUN 域名需要直接解析到服务
在 `.env` 中添加 `SFU_UDP_PORT=7882`，放行 UDP 7882，再重启 Piik，即可启用自动 SFU 兜底。
如果服务器处于 NAT 后方，还需将 `SFU_PUBLIC_IP` 设置为外部可达的公网 IPv4 地址。
这项功能由同一个服务端程序提供。
服务器需要为实际经由 SFU 观看的观众承担转发负载和出口带宽。
房主需要在开始分享前关闭 **隐私模式**，才会允许使用这条线路。
直连和 SFU 媒体都需要可用的 UDP 通路。
如需所有房间都通过服务器转发，再设置 `SFU_ONLY=true` 并重启。
分享设置中会显示 **服务器转发**。请为每位观众预留服务器带宽；
服务器线路失败时，此模式不会改走 P2P。
## 长期运行与更新
需要开机启动时，参阅 [systemd 与容器配置](./service-management.zh-CN.md)。
docs/project-memory.md
+6 -4
View File
@@ -1,6 +1,6 @@
# Project Memory
Last updated: 2026-09-29
Last updated: 2026-10-01
Piik is private, low-latency screen sharing for one Host and up to 20
authenticated friends. The current product surface is Web Host, Web Viewer, and
@@ -43,7 +43,8 @@ blank keeps the self-contained site open.
## Product Map
- [Rooms and access](./standards/rooms-access.md) owns room codes,
invitations, code entry, credentials, and lightweight/SQLite persistence.
invitations, code entry, credentials, room interactions and lightweight/SQLite
persistence.
- [Routing and transport](./standards/routing-transport.md) owns P2P-first
distribution, endpoint capacity, SFU fallback, recovery, and privacy limits.
- [Capture and media quality](./standards/media-quality.md) owns Browser capture,
@@ -70,8 +71,9 @@ implementation and routine UI detail.
available, and anonymous provenance distinguishes ordinary from predicted
selected paths. The configuration default remains stock ICE. The only
application fallback is one bounded embedded SFU/UDP
publication; Piik configures no TURN or TCP media route. A pre-share
peer-only policy can exclude that SFU suffix for one share generation.
publication; Piik configures no TURN or TCP media route. Hosted operators may
require SFU-only media through the same controller. On P2P-first sites, a
pre-share peer-only policy can exclude that SFU suffix for one share generation.
- Every endpoint shares one steady outbound-copy cap, default `2` and limited to
`1..3`. One room controller owns one committed graph, one reconcile loop, and
one serial child operation. First decoded frame commits availability work;
docs/research/browser-local-encoding-pool.md
+164 -6
View File
@@ -93,6 +93,14 @@ maximum queued service; reverse feedback is unshaped. They are not a model of
public NAT, geographic RTT or every congestion/loss pattern. Reduced quality
and temporary stalls under real constraints remain possible.
A 2026-09-30 actual-tab fault check separated A at 400 kbps from healthy B,
then failed only A's producer. Both H264 and VP8 retained the outgoing
connections, returned to ordinary encoding and retired all pooled groups.
B retained 1904x1048 at about 30 fps; its largest observed frame-callback gap
around fallback was 271 ms for H264 and 80 ms for VP8. Source-wide fallback
therefore has a measurable handoff cost, even without a resolution drop or
connection failure. This does not establish low-end hardware capacity.
### Balanced Startup And Recovery
Serial Chrome 152 checks on 2026-09-25 reproduced two composition defects in
@@ -316,12 +324,19 @@ complexity: changing to simple bars restored 1080p, retained after restoring
the textured scene. WebRTC's [quality scaler](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/modules/video_coding/utility/quality_scaler.cc)
uses QP thresholds and sampling history as well as bandwidth-related adaptation.
Its default two-second checks and longer sampling after down-adaptation can
outlast a short disturbance. Native Chrome 154 logging also confirmed a stable
QP deadband with the NVIDIA H264 MFT: after one 1048p-to-786p reduction, smoothed
QP stayed between the recorded 24/37 thresholds for the remaining 90-second
observation. Capture remained full size, but no low-QP upscale was requested.
This is not merely old samples taking time to expire; restored bandwidth alone
does not require this scaler to restore resolution.
outlast a short disturbance. A Chrome 154 ordinary-sender tab control on
2026-09-30 remained at 524p / about 30 fps after 120 recovery seconds following two
one-second rate/loss pulses. Actual capture stayed 1904x1048 / 30 fps; the native
log's slow QP filter settled around 27, above its upscale threshold of 24.
Changing only the scene to simple moving bars restored full size in about
12 seconds. Restoring the textured scene then retained full size for the
remaining 30 seconds. The sender made only its two startup parameter writes;
no pool, reconnect, preference reset or frame reset caused that recovery.
This confirms a content-dependent stable state, not merely stale samples or
a blocked recovery operation. Restored bandwidth alone does not require this
scaler to restore resolution. The scaler and default-threshold source matched
inspected upstream main on that date; this does not establish every device's
behavior or the field report's cause.
A matched three-arm comparison used fixed 5 Mbps, alternating 5.0/4.9 Mbps
every 500 ms, and repeated identical 5 Mbps writes. Native reconfiguration
@@ -335,6 +350,149 @@ perceptual blur measurement. They do not justify delaying real demand, bitrate
compensation or periodic resets; equal product updates are already deduplicated.
The field's repeated brief blur still lacks matching Host histories.
**Hardware encoder startup can block sibling encoding.** On 2026-10-01,
Windows Chrome 154.0.8037.58 tests added three independent Canvas H264 producers,
coinciding with 174/221/225 ms gaps in the existing producer; captured frames
continued within 47 ms. The new producers never joined the pool or replaced its
output, and the existing
producer's parameters did not change. Their NVIDIA initialization took
161/163/162 ms. Substituting software VP8, confirmed by RTCStats as libvpx, gave
60/48/109 ms existing-producer gaps in the same startup windows.
A subsequent trace recorded 148 ms of `CreateAndInitializeVEA` on the shared
Renderer Media thread, matching GPU-side initialization. Frames timestamped
during that interval entered encoding together on the same thread after it
returned. This establishes blocking in the local reproduction, consistent with
[Chromium's synchronous initialization IPC](https://github.com/chromium/chromium/blob/154.0.8037.58/media/mojo/clients/mojo_video_encode_accelerator.cc#L117).
The trace buffer lost events elsewhere; this conclusion uses complete task and
frame events, not missing events. It does not identify the driver's internal
cost, explain earlier downscaling or attribute unmatched field reports. No
codec-default change, spare prewarmed encoder or pool reset follows from it.
The Windows [VEA provider](https://github.com/chromium/chromium/blob/154.0.8037.58/media/mojo/services/mojo_video_encode_accelerator_provider.cc)
already assigns each service a dedicated COM STA runner; the trace likewise
places the old service's encoding and new initialization on different threads.
Asynchronous Renderer initialization is therefore a plausible upstream repair,
not merely moving the wait onto another shared queue. It still needs a patched
Browser comparison, including initialization failure and callback ordering;
shared GPU/driver cost remains unproved.
**Optional reuse must preserve the existing rate owner.** A two-Viewer,
textured-tab H264 comparison exposed a Piik handoff defect. During recovery
from a one-second rate/loss pulse, a healthy Viewer moved from a 5 Mbps group
to a lower-budget group because its sampled output was 29 rather than 27 fps
at the same 1048p. Both reported `qualityLimitationReason:none`, but the same
NVIDIA encoder implementation's interval-average QP was about 51 on the
candidate versus 37 on the current output; the candidate then fell to 786p. The
[stats definition](https://www.w3.org/TR/webrtc-stats/#dom-rtcqualitylimitationreason-none)
describes resolution/FPS limitation, not equal quantization quality.
Pending membership raised the candidate's budget before the existing
lower-budget rejection ran. The candidate repair checks this constraint before
membership and reuses one non-regression predicate before submitting a pending
selection. FPS comparisons also respect the Host ceiling; a sampled 32 fps
under a 30 fps ceiling is not an improvement over 30 fps. Initial output and
necessary downgrades retain their path, without a QP policy, timer or encoder reset.
Selection also needs proof at the actual write boundary: while waiting for a key,
a candidate can downscale after passing the sampled comparison. The output now
asks the pool to revalidate current authority, budget and actual key dimensions
before replacing its queue or writing bytes. Rejection clears both pending owners,
invalidates a contradicted sample and retains prior output and recovery requests.
H264/VP8 display-capture checks preserve live/paused settings, source replacement
and ordinary fallback; played audio remains present. Brief video stalls remain.
The bounded H264 and VP8 two-Viewer controls for this admission repair retained
the healthy sibling at 1048p and about 29 fps through both recovery intervals.
The constrained Viewer still adapted and recovered. This establishes one
avoidable sibling degradation path, not the field room's cause or elimination
of all brief blur. The affected admission rules predate the display-isolation
repair. Raw observations stay in ignored fixtures; comparisons read native stats
without calling the producer's startup-transition helper.
Current demand also matters at group admission. Two-Viewer traces showed a
newcomer raising an incumbent group's allocation because matching used the
previous applied budget. The incumbent then needed another encoder, while the
newcomer's former encoder retired. Matching current native demand prevents that
exchange; a pending newcomer cannot establish compatibility with its own vote.
Cancellation must stop at accepted output writing, including a pause that delays
its completion. These are ownership repairs, not suppression of real demand
changes or removal of the measured hardware initialization cost.
**Quiet content is not rate compatibility.** A separate fixed-allocation
comparison kept the two outgoing H264 ceilings at 3 and 5 Mbps, with no network
shaping, while alternating simple and textured moving content. Actual native
allocations stayed at those ceilings during the later scene cycles. Reuse based
on a quiet 500 ms byte-rate sample repeatedly moved the weaker child into the
5 Mbps producer; returning motion split it out again. Five producers were created
over the run. Requiring the candidate's rate budget to fit the child's allocation
at initial admission, optional reuse and pending revalidation retained two
producers with no scene-driven handoffs. The healthy H264 sibling retained full
1048p; the weaker output continued native adaptation. VP8 also retained two
producers through the same scene changes. Three controlled regressions fail
when these checks are removed, including a paused pending child whose candidate
budget rises. Unequal demands may retain an additional producer even during a
quiet scene; bounded membership and native recovery remain unchanged. These
checks remove unnecessary encoder churn, not all content-dependent adaptation.
**A known allocation should initialize the local encoder.** A replacement local
PeerConnection otherwise starts its own estimate at WebRTC's default 300 kbps,
even when the outgoing connection already allocates 4 Mbps. Chrome 154's native
logs confirm this second cold start. On the same isolated 1904x1048 textured tab,
seeding only the local answer with that 4 Mbps allocation increased VP8 output
in the first second from 6 to 27 frames; its first sampled target was 3.33 Mbps
rather than 2.70 Mbps. The 400 kbps control retained its native 333 kbps initial
encoder target and two first-second frames with either version. These bounded
results do not promise faster first frames or remove weak-network adaptation.
The hint uses the existing Host-bounded allocation, not a multiplier or floor.
WebRTC reads the send configuration from the
[remote description](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/pc/channel.cc)
and accepts a positive `x-google-start-bitrate` in its
[codec bitrate configuration](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/media/engine/webrtc_media_engine.cc).
An offer-only control lost the hint in Chrome's answer and did not improve
startup. Piik therefore changes only its owned local answer; external SDP,
codec identity and subsequent sender-budget updates stay unchanged. The
application writes this hint once per producer, not on each budget change.
The full two-Viewer VP8 follow-up retained the healthy child at 1048p through
two one-second 400 kbps / 4% loss pulses at 40 ms configured RTT. The constrained
child still down-adapted and froze during recovery, but regained 1048p within
each 40-second recovery window. This is not a freeze-free claim. A separate
audio/loss check delivered all 97 scheduled audio pulses to both played outputs;
H264 quiet-start, live/paused settings, source replacement and ordinary-fallback
checks passed. All owned test connections, processes and profiles retired.
**Residual VP8 stalls do not require an output wait.** The ordinary two-Viewer
control with the same source and rate/loss pulses also froze and downscaled,
ending the recovery windows at 786p and 524p. It entered the second pulse at a
lower resolution than pooling, so their recovery FPS is not a fair cost comparison.
Replaying the pooled child's budget sequence on a warmed local producer without
a shaper, carrier, queue or handoff produced nine frames in the same eight-second
low-budget interval; adding the two original keyframe requests yielded eight,
matching the original interval. Startup protection had already ended. Native
logs show frame dropping and the scaler's spaced downscales. This establishes
stalls under that budget sequence, not the cause of the sequence: keyframe bursts
can still affect downstream estimation, and parameter reconfiguration remains
in the replay. These results do not support removing dependency recovery or
startup protection, growing queues or repeatedly resetting encoders. Matched
field evidence and broader transport/allocation effects remain open.
The rate/loss run with audio retained all 65 pulses after startup at both
played outputs, despite video stalls; this does not establish gap-free video.
A separate six-second complete RTP-media drop on one child, preserving RTCP
and signaling, resumed its played video 157 ms after release for VP8 and 128 ms
for H264. Both healthy siblings retained full dimensions; H264 still recorded
brief sibling stalls. All post-release audio pulses arrived. This bounded local
check does not establish recovery for an interrupted relay or reporter's network.
A matched single-Viewer control separates pool handoffs from ordinary adaptation.
Pooled sending used one producer with no handoffs; ordinary sending used no pool
and made only two parameter writes. After the second one-second rate/loss pulse,
both reached 262p before ending their 40-second recovery window at 390p / 15 fps.
Actual source frames stayed 1904x1048 at about 30 fps. Native logs showed quality
scaler down-adaptation, Chromium's below-360p software fallback and subsequent
recovery to NVIDIA H264 at 390p; this was not a hardware failure. Pool reuse and
frequent application-level budget writes are therefore not necessary for this
bounded low-output state. Their effect on the intervening trajectory and the
unmatched field reports remains distinct.
The tab case exposed a more specific capture-feedback problem. A Chromium trace
kept the source geometry at 1904x1049 while capture supplied a 268x148 content
rectangle. Feedback rose to 66,600 pixels, but capture did not grow.
docs/research/media-fidelity.md
+162
View File
@@ -0,0 +1,162 @@
# Media Fidelity
Reviewed 2026-10-01. [Media quality](../standards/media-quality.md) owns behavior;
[TODO](../todo.md) owns remaining work. These synthetic checks do not establish
all-device acceptance or full HDR support.
## Stereo
Browser display capture requests two channels without speech processing. The
Host mixer uses a stereo Web Audio destination; Native capture and mixing use
48 kHz, two-channel PCM16. Native Opus uses the audio application mode. Browser,
Native and SFU receivers request stereo. The existing 64/128/192 kbps choices
are ceilings, not a lossless-audio promise.
`opus/48000/2` alone does not prove stereo content: Opus permits mono packets
under that declaration. [RFC 7587](https://www.rfc-editor.org/rfc/rfc7587.html#section-7)
distinguishes receive preferences from packet content. Test distinct signals
after decoding instead of inferring fidelity from SDP or a two-channel mixer.
The embedded SFU browser gate sends 500 Hz left and 1500 Hz right, and measures
decoded channel separation. On local Chrome 154, both the default fallback and
SFU-only arms retained more than 77 dB separation in both channels at the 64 kbps
ceiling, including normal video reception. Source changes and subscription
recovery retain the existing audio gate. Run `npm run gate:embedded-sfu` with `PIIK_EMBEDDED_SFU_GATE=true`,
`CHROME_PATH`, and optional `PIIK_EMBEDDED_SFU_ONLY=true`; it cleans up its own
processes and ports. This is synthetic local evidence, not an acoustic or
public-network measurement.
A local Chrome 154 check through the production Host/Viewer peers retained more
than 57 dB separation after direct reception and after one decoding/re-encoding
relay, with a third tone mixed as Host microphone input. The receiver used its
normal video playback sink; a receiver-only Web Audio probe without playback
returned silence and was not a fidelity result. The Native Opus regression test
independently requires more than 20 dB decoded separation at all three bitrates.
These checks support keeping the existing stereo path. They do not cover every
physical source device, browser or acoustic setup, and do not establish surround
sound support.
## SDR Conversion
[#445](https://github.com/TNTcraftHIM/Piik/issues/445) reports washed-out Native
capture on an SDR Windows 10 system. Its proposed explanation is not established:
Microsoft documents full-range RGB as the
[video processor's default input range](https://learn.microsoft.com/en-us/windows/win32/api/d3d11/ns-d3d11-d3d11_video_processor_color_space).
The confirmed gap was that `FrameConverter` left its input/output range and
matrix implicit. The same converter handles BGRA capture and decoded NV12
relay input; correcting only screen capture would leave the relay inconsistent.
A local D3D11 check through the previous converter produced full-range NV12 from
BGRA color bars (black Y=0, white Y=255). The production `AdaptiveEncoder` VP8
output then reproduced crushed shadows and clipped highlights in Chrome 154:
| Source gray RGB | Previous conversion, WebRTC playback | Explicit limited-range BT.601 |
| --- | --- | --- |
| 0 | 0 | 0 |
| 32 | 19 | 33 |
| 128 | 130 | 128 |
| 235 | 255 | 235 |
| 255 | 255 | 255 |
The fixture encoded 256x128 color bars with the production converter/encoder,
injected those keyframes into a local VP8 PeerConnection without the color-space
RTP extension, and sampled the playing video into a canvas. The experimental
conversion also restored red/green/blue bars to within one channel value of the
source. This establishes a bounded SDR defect, not the cause of every color
report.
WebCodecs and WebRTC are not interchangeable acceptance paths: explicit BT.709
looked correct through WebCodecs but shifted primary colors through this WebRTC
receiver. Chromium's
[remote-frame conversion](https://chromium.googlesource.com/chromium/src/+/master/third_party/blink/renderer/modules/peerconnection/media_stream_remote_video_source.cc)
can use BT.601 when color metadata is unspecified. White Y=235 is normal for
limited-range YUV; it must become display RGB=255, not be treated as a defect by
itself.
The candidate now declares RGB capture as full range and reads decoded H.264
color metadata from Media Foundation. Unspecified SDR and VP8 use the existing
WebRTC BT.601 convention. The converter produces limited-range BT.601 NV12;
the H.264 media types declare matching range, matrix, primaries and transfer.
Setting only matrix/range did not produce matching H.264 VUI on the tested
NVIDIA encoder. No wire field or user setting is added.
Use the Windows 10
[explicit video color-space API](https://learn.microsoft.com/en-us/windows/win32/api/d3d11_1/nf-d3d11_1-id3d11videocontext1-videoprocessorsetstreamcolorspace1).
The legacy nominal-range API squeezed decoded NV12 a second time in local tests.
Even with the explicit API and a positive format-conversion capability result,
NV12 BT.709-to-BT.601 conversion changed levels on NVIDIA when scaling and left
the matrix unchanged on AMD. Noncanonical NV12 therefore passes through RGB on
the GPU before the shared output conversion. Ordinary RGB capture and already
canonical NV12 keep one pass; no CPU readback, vendor branch or custom shader
is added to the product path.
The opt-in `-CheckColor` probe in the [Windows capture guide](../../native/capture/windows/README.md)
checks full/limited-range BT.601/BT.709 input, scaling and input replacement
through the production converter. It passed on local RTX 4070 SUPER and AMD
Radeon adapters. RTX H.264/VP8 encoding followed by Chrome 154 WebRTC playback
also passed direct reception, decode/scale/re-encode and explicitly tagged
BT.709 H.264 input to both output codecs; sampled RGB errors were at most 5/255.
AMD hardware encoding was not accepted locally: MFT activation returned
`0x8007000e` before color configuration. These local checks do not settle the
reporter's Windows 10 environment, all hardware encoders, HDR content or missing
upstream color declarations.
## HDR To SDR
[#420](https://github.com/TNTcraftHIM/Piik/issues/420) reports overexposure with
both App and Browser capture on one Windows HDR machine. The report does not
locate the failing stage.
The previous native WGC frame pool and thumbnails requested BGRA8, discarding
HDR range before the existing 8-bit H.264/VP8 encoder. Microsoft's
[capture guidance](https://learn.microsoft.com/en-us/windows/apps/develop/media-authoring-processing/screen-capture)
warns that HDR needs floating-point capture throughout the input pipeline to
avoid clipping. Reducing brightness after encoding cannot restore clipped
highlights. This is a confirmed implementation gap, not proof that every
reported Browser failure has the same cause.
The candidate uses one `CaptureSdrConverter` before native output fanout and in
thumbnails. HDR displays use an scRGB FP16 pool, then Windows Direct2D's
[HDR tone-map effect](https://learn.microsoft.com/en-us/windows/win32/direct2d/hdr-tone-map-effect)
compresses luminance, the
[white-level effect](https://learn.microsoft.com/en-us/windows/win32/direct2d/white-level-adjustment-effect)
accounts for the source display's SDR white, and color management converts to
sRGB BGRA8. Ordinary SDR retains its texture-copy path. No custom shader,
CPU readback, new packaged dependency or per-output tone mapper is added.
Source-monitor movement and DXGI factory invalidation refresh color metadata,
following [GetDesc1's contract](https://learn.microsoft.com/en-us/windows/win32/api/dxgi1_6/nf-dxgi1_6-idxgioutput6-getdesc1).
A format change recreates the WGC pool through its existing resize boundary;
it does not restart routes or encoders. WGC supplies no per-frame MaxCLL, so
conversion uses the display peak (at least the configured SDR white), or the
HDR10 10,000-nit range when usable peak metadata is absent. This stable platform
policy avoids a second image-content adaptation loop; real-display appearance
remains an acceptance requirement.
The retained `-CheckColor` probe passed on RTX 4070 SUPER and AMD Radeon:
0–1000-nit neutral bars preserved distinguishable highlights, source white
80/240/80 nits and SDR/HDR size changes preserved ownership and ordinary SDR
colors. Production H.264/VP8 output decoded over Chrome 154 WebRTC reproduced the
converted SDR gray bars within 1/255. These are synthetic input checks, not
observations of an HDR game or the reporter's machine.
Warmed GPU timestamp measurements for conversion, including the owned input
copy, had the following local medians. Dimensions are captured input sizes,
not newly supported sharing presets.
| Adapter | 1920x1080 | 3840x2160 |
| --- | --- | --- |
| RTX 4070 SUPER | 0.045 ms | 0.240 ms |
| AMD Radeon | 2.39 ms | 11.17 ms |
HDR conversion has a real GPU cost on weaker devices. These bounded measurements
exclude encoding and do not establish gaming/endurance performance. Current
monitors were SDR: physical HDR highlights, SDR content on HDR displays,
thumbnails, mixed displays and mode changes under load remain unaccepted.
Browser capture owns its own conversion; inspect the acquired frame before
assigning a downstream bug.
Full HDR requires a separate end-to-end decision for bit depth, codec support,
color metadata, decoding and mixed Viewer capabilities. It is not a quality
dropdown addition. Preserve ordinary SDR routing and existing compatibility
with HDR-to-SDR; no full-HDR or surround implementation is claimed.
docs/research/native-client-media.md
+64 -1
View File
@@ -1,11 +1,47 @@
# Native App Media Evidence
- Reviewed: 2026-09-29
- Reviewed: 2026-10-01
- Scope: platform capture, shared encoded sources, Pion transport and Browser
decode; current behavior belongs to [media quality](../standards/media-quality.md)
- Status: Windows physical native Host and Viewer gates passed; macOS and Linux adapters
compile and package but still require physical media gates
## Windows Audio Exclusion
A Windows 11 build 26200 check used two independent WinMM processes emitting
500 Hz and 1500 Hz tones, through the production capture helper and Go PCM
reader. Default-device loopback captured both at approximately 1799 amplitude;
process inclusion retained the first with under 0.3 amplitude of the other,
and exclusion retained the second with under 0.04 of the excluded tone. Target
exit retired capture in about 42 ms; attempting the stale identity failed.
No raw audio was saved. The extended capability probe completed in 300 ms.
A parent/child follow-up used a silent selected parent with a 1900 Hz child,
plus an independent 2900 Hz process. Process-scoped reference amplitudes were
about 1799; exclusion retained the independent tone at 1800 and reduced the
selected child's tone to 9.3 (under 0.6%). Parent exit retired capture in 64 ms;
its stale identity was rejected. The reference uses each signal's own process
capture: default-device loopback also includes unrelated playback and endpoint
processing, so its amplitude is not a process-capture gain reference.
The session fixture separately verifies failed exclusion replacement mutes
source audio while preserving video, microphone and encoded audio output; pause
and audio-profile changes retain the selected input. These checks establish a
local Windows implementation and one direct child, not coverage of every voice
application's process tree or simultaneous playback across several output devices. Those remain
device acceptance boundaries. [ADR-0008](../adr/0008-window-scoped-audio-capture.md)
owns the platform choice.
The [Windows activation contract](https://learn.microsoft.com/en-us/windows/win32/api/audioclientactivationparams/ns-audioclientactivationparams-audioclient_process_loopback_params)
accepts one PID and its process tree, not a list of unrelated processes.
The [win-capture-audio implementation](https://github.com/bozbez/win-capture-audio/releases/tag/v2.2.0-beta)
supports broader exclusion by tracking audio sessions, capturing non-matching
processes and mixing them. Applying that design here would require session
discovery, process-tree deduplication and bounded multi-input mixing; Piik's
current source-plus-microphone mixer does not own that inventory. Multiple
exclusions therefore require a capture design change, not just a multi-select
control. Combining exclusion streams does not itself exclude their combined set.
## Initial Physical Evidence (2026-09-05)
These checks established the original native media paths. The Browser-mediated
@@ -271,6 +307,33 @@ that result nor the allocation timings establish the reported system-wide lag's
cause. Retain the existing implementation; reopen optimization from a matched
bottleneck.
## Native H.264 Motion Quality
Issue [#432](https://github.com/TNTcraftHIM/Piik/issues/432) reports more visible
blocking with native H.264 than VP8 on v1.6.7. Review of its attached Browser
report confirms 720p30 in both comparison windows, roughly 11 Mbps received,
and no reported packet loss; the 1080p60 H.264 window receives roughly 10 Mbps.
Those samples establish delivered format and transport observations, not equal
source content or objective picture quality. No encoded stream, reference
frames or QP measurements accompany that report.
The reporter retracted the initial CBR bitrate-ceiling diagnosis after finding
that the experimental fixture changed bitrate during the measured interval.
That switching belongs to `PIIK_H264_FIXTURE` in `windows/main.cpp`, not live
capture. The corrected experiment reports CBR reaching its requested bitrate.
Production `h264_encoder.cpp` explicitly requests and validates Baseline; the
Browser's negotiated `profile-level-id` alone is not an inspection of emitted
SPS. The later experimental CABAC result cannot establish the production
Baseline bitstream's behavior. Microsoft's [encoder reference](https://learn.microsoft.com/en-us/windows/win32/medfound/h-264-video-encoder)
distinguishes profile, rate control and optional quality/speed settings.
The original 720p comparison remains unresolved. An equal-content comparison
must preserve actual bitrate, frame rate, latency and production codec settings,
inspect the resulting bitstream and retain reference/decoded pictures. Raising
the bitrate ceiling, lowering the default frame rate or changing profile is not
justified by these logs alone; those changes have bandwidth, motion and receiver
compatibility consequences.
## Implementation Boundary
- `nativecapture` owns the child process, source identity, and bounded frame protocol.
docs/research/room-interactions.md
+260
View File
@@ -0,0 +1,260 @@
# Room Interaction Design
The owner accepted the text/reaction UI and model with bounded local acceptance.
[Status](../status.md#accepted-interaction-phase) owns release readiness and
[TODO](../todo.md#now) owns remaining work. Reviewed 2026-10-02.
This document records the implementation design and its tradeoffs;
[rooms and access](../standards/rooms-access.md#room-interactions) owns authority
and retention, while [presentation and lifecycle](../standards/presentation-lifecycle.md#product-surface)
owns room/publication lifetime.
## Model And Owners
| Concern | Owner | Lifetime |
| --- | --- | --- |
| Identity, room membership and access | Existing room store and authenticated signaling session | Current admitted participant |
| Screen/camera plus Host commentary | Existing capture and media owners | Current share and source |
| Text and reactions | `signal/interactions.go`, shared protocol and `RoomInteractionSession` | Current authenticated room session; bounded page-local presentation |
| Chat panel, optional chat overlay and reaction placement | Shared `RoomInteractions` and `RoomChatOverlay`; `RoomInteractionSession` owns bounded events | Current room UI |
The existing room authority admits interactions through the same authenticated
WebSocket; there is no second admission or identity system.
Messages do not need a media route, decoded frame, or SFU. They do not mutate the
media graph, add a port, create an extra connection, or restart capture. Pause,
source replacement and media recovery keep the current conversation. In this
candidate, a capable Host explicitly opts into a persistent room session.
Restoring an existing room connects that session without requesting capture or
starting media. The capture-ready owner requests publication; merely entering a
capture picker or awaiting permission cannot publish. Stopping the publication
retires only its media generation, while the authenticated Host and its
interaction session remain available. A later `start-sharing` claims a
fresh generation on that same signaling connection. Room close, access loss,
session replacement and page teardown still retire the whole session. A Host
without the capability keeps the existing stop-and-leave behavior.
Room restoration and authority recovery share one awaited owner; connection reuse
checks the actual room/token binding. Publication acknowledgment precedes media
effects. A rejected start names its attempted generation, retires only that local
attempt and keeps room interactions available; it cannot stop another generation.
Opening the Host page alone still does not create an unused room.
The server treats the current Host session's non-empty generation, matched to the
room's reconnect fence, as the only publishing truth. Room membership and Host
presence are separate, so a late Viewer in an idle room receives no media route
even when the Host remains available for room interactions.
Connected Viewers can continue interacting when the Host disconnects, while the
room server and their admitted sessions remain available. A departed participant
cannot receive a targeted reaction. Closing an App that also runs the room server
ends that service; merely leaving the Host page does not.
## Delivery, Compatibility And Bounds
- `/api/capabilities` advertises optional `roomInteractions` and
`hostRoomSession`; absence means the corresponding behavior is unavailable.
The page explicitly subscribes after authentication. Only subscribed current
sessions receive interaction events. The Host room-session behavior also
requires the `roomSession: true` authentication opt-in; older clients retain
the original stop-and-leave contract. Existing v23 media messages remain
valid, and publication start/result messages are sent only to an opted-in
session.
- The viewer discovers this optional capability independently of first-picture
startup. Late capability discovery enables data on the existing connection.
- A strict payload union admits plain chat text or a registered reaction. Chat is
at most 280 Unicode code points, normalized to NFC, without control/bidi
characters. It renders as text, with no markup, links, attachments or commands.
- The server accepts at most one interaction per 800 ms per connection. It
validates identity, subscription and target before fan-out. A slow recipient's
data is skipped at 16 KiB queued output, reserving the existing bounded queue
for signaling. The browser also declines sends under output backpressure.
- The sender waits for its server echo, not a guessed success or a read receipt.
A rejection or a 5-second confirmation timeout preserves the draft. There is
no offline outbox or automatic retry; manually retrying an unconfirmed send
may duplicate a message that arrived before the confirmation was lost. Only
the matching pending request may confirm a send or clear its draft/error;
late echoes may enter history without confirming a newer attempt.
- The page retains the latest 80 received chat messages and at most eight
simultaneous reaction effects. Effects expire after 2.4 seconds. Reconnection
clears pending effects and re-subscribes, without replaying messages or actions.
Same-room re-admission retains history, self attribution and the local draft,
even when its wire peer ID changes. Refresh, leaving, authority loss or room
session replacement clears the page conversation; late joiners receive only
new messages. Nothing persists in Browser storage.
- There is no server history, database table or message body in diagnostic
exports. Transport is the site's HTTPS/WSS connection; **chat is not end-to-end
encrypted**, and the server operator can access live message contents.
## Interaction Design
Chat uses the shared `FloatingPanel` shell for its nonmodal window. The shell
owns native visibility, placement, movement, resizing and focus return; the
conversation retains its session, draft, reading position and unread state.
It opens initially at the lower right, without
resizing the picture or moving the couch. Drag its title bar, use arrow keys on
the focused bar, or explicitly choose a corner under Position. Its summary also
allows dragging; a drag never activates the menu. Resize from either diagonal
corner, use arrow keys on a resize handle, or choose a preset under Position.
Resizing holds the opposite corner steady. Size and position belong to this room
UI only; viewport and keyboard changes bound them without erasing preferences.
The native manual
popover keeps the conversation visible while operating the picture or reactions.
Close, the entry toggle, or Escape hides it; this preserves the room conversation. Inner
menus consume Escape first, then the most recently opened utility window, then
theater mode. Input-method
composition is not a dismissal request. Opening focuses the panel without opening
the keyboard; explicit close returns focus to the entry. Nested disclosures
cannot change panel visibility.
The visual viewport bounds it above an onscreen keyboard.
Hidden messages are not marked read. A bounded scrolling log keeps arrivals from
growing the panel. Consecutive messages from the same sender share visual identity;
each message retains its accessible author. Hiding and reopening preserves the
reader's position; a latest-message action resumes following. Sending waits for
confirmation without locking draft editing or introducing a message queue.
The conversation entry sits below the couch, separate from
the Host's sharing controls; theater retains its compact conversation entry.
Selecting a person opens the compact reaction palette in the shared floating
shell, initially beside the selected person (below when space permits). Each
opening anchors there before allowing free movement. Its portrait and name identify the
target even after moving the window. Selecting a different participant keeps its
placement and any manually chosen size; selecting the same person toggles it.
The default palette height follows its content, bounded by the visible viewport.
One's own palette expresses a reaction above oneself and holds the local effects switch; other
people's palettes offer targeted expressions and two labelled throwing props.
Connection details remain secondary when that participant has inspectable data.
Close or Escape dismisses the palette; it also closes if its participant leaves.
Sending keeps it open for repeated interaction, with the existing short cooldown.
Confirmation never changes menu visibility; failure stays visible in the card.
Opening or closing a reaction card preserves the chat draft and unread state.
Self expressions follow the sender's head, including idle and interaction motion.
Only that sender's newest self expression replaces their earlier one. A targeted
emoji is a delivery: it travels from sender to recipient, retains a compact
source label above the centered glyph, and settles separately above the
recipient's own expression. Late mounts and reduced motion retain that attribution without
replaying elapsed motion. A gift never replaces the recipient's self expression;
all delivered events remain in the accessible log.
Tomato/poop throws require choosing a current participant, land with a small
character response and settle above the head, keeping the face visible. A tomato
squashes with a few droplets; poop wobbles with a small puff. Other expressions
use short gestures and sparse accents. All decorative descendants share the
event's remaining phase and expiry; event ID and sender identity seed their
variation. No particles, stains or animation timers outlive the event.
A departed participant is never silently replaced as target.
Everyone sees the same server-issued event; placement follows their own couch
layout. Effects have a couch-only coordinate owner, independent of panel layout,
and never become a video overlay or a media status. Impact transforms leave the
UUID-owned idle animation, names and status indicators alone. Hiding effects or
retiring the event cancels its own animation; showing a still-live event uses its
remaining lifetime. Users can hide effects; reduced motion uses a static stamp.
Theater mode hides the couch and retains a chat entry, using the same floating
panel and draft without changing the picture's size. HTML player fullscreen
keeps the existing danmaku control; it does not include the conversation panel.
Unread messages advance only while the log is onscreen, the document is visible
and the reader is following its latest messages. A hidden or scrolled-back log
does not consume arrivals. These are local unread indicators, not read receipts.
The optional chat overlay (danmaku) projects newly received chat over the picture;
it adds no message kind, transport or delivery promise. It defaults off, with the
same local switch beside the chat composer and Viewer playback bar, including HTML
fullscreen. At most three single-line comments appear for up to six seconds each.
Font size follows the picture width, with a fixed readable contrast and duration
rather than separate typography, opacity and speed preferences.
Each event's sender and ID choose its lane, and a newer event replaces that lane's
previous flight. All received text remains in the full chat log, without a playback queue. Text is clipped
to the picture space above playback controls; smaller spaces show fewer complete
lanes without restarting their timelines. It does not intercept playback gestures. Showing the overlay
does not mark the chat log as read. Hidden documents, unavailable pictures,
source changes and disconnection clear flights; showing or reconnecting never
replays earlier comments. Reduced motion uses temporary stationary text. Native
video-only fullscreen and picture in picture cannot include HTML overlays.
Server confirmation means acceptance, not an acknowledgment from every recipient.
Delivered chat follows server order; congestion can leave gaps without reordering
the received subset. The same event retains its lane and time basis even when
preceding events are missed. Visibility, local switches and missing messages may
change which effects are shown; geometry and font size follow the local viewport.
Reaction animations are decorative. The same bounded events also provide
localized sender/action/target descriptions in a polite accessible log, including
when visual effects are hidden. Reduced motion removes decorative movement from
both the flight and the picker controls.
Chinese, English and visual modes share the component and behavior. Visual mode
retains user-entered text and participant names, with icon controls and localized
accessible names. Host identity reuses the existing crown; neither chat colour nor
reaction motion claims media connectivity, capture permission or delivery status.
## Presentation Synchronization
The server stamps acceptance time once and broadcasts the same event in its
existing serialized order. A subscription response provides server time; the
room session estimates its offset from the local round-trip midpoint and keeps
that estimate until reconnection. Effect expiry and both renderers share that
session's clock. Late effects start at their remaining phase; expired chat still
enters the log and confirms a matching send, without replaying an effect.
Returning from reduced motion also respects the original expiry.
This is approximate alignment, limited by subscription round-trip asymmetry,
queueing and clock drift; it does not promise frame-exact synchronization. There
is no continuous clock exchange, sorting by timestamp, or animation-frame stream.
Changing a device's clock mid-connection can disturb effect timing until the
next subscription; message order and delivery do not depend on that clock.
Participant motion and event lanes reuse the same identity hash, with an event's
ID and actor identity determining its lane independently of prior arrivals.
A shared mutable random-number stream would drift whenever a client misses or
hides an event. Responsive layout remains local.
This follows the same-input principle described in
[Deterministic Lockstep](https://gafferongames.com/post/deterministic_lockstep/),
without introducing lockstep waiting or frame-by-frame simulation.
## History Direction (Not Yet Implemented)
History ownership remains undecided. A Host cache can also miss live messages and
is unavailable while the Host is absent; treating its copy as server-confirmed
history would change message authority. Compare that cost with a bounded room-server
memory buffer before adding replay. Either design needs explicit access and
retention boundaries. Recovered text would enter the log, never replay old
danmaku or reaction effects. Current behavior remains the no-history contract above.
## Audio Boundary
The Host microphone remains commentary mixed into the shared source.
Bidirectional room voice is not part of the product plan.
Text, reactions and danmaku introduce no microphone capture, voice
admission or additional media peers.
## Primary References And Rationale
- [LiveKit media and data](https://docs.livekit.io/frontends/build/media-data/)
separates publications from finite data. Piik already has an authenticated
signaling channel; adding the LiveKit room service solely for chat is unnecessary.
- [LiveKit text streams](https://docs.livekit.io/transport/data/text-streams/)
distinguishes current-participant delivery from persistent history. The
candidate likewise promises transient room messages, with explicit local bounds.
- [WebSocket buffering](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/bufferedAmount)
and [WebSocket API limitations](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API)
motivate bounded queues and explicit send failure rather than an accumulating outbox.
- [WebRTC data channels](https://www.w3.org/TR/webrtc/#rtcdatachannel)
depend on peer connections. Using them here would couple chat readiness and
fan-out to a screen-sharing route; the existing room WebSocket avoids that.
- [React state identity](https://react.dev/learn/preserving-and-resetting-state)
supports resetting the panel with its room session rather than a reconnecting
transport identity. [ARIA log semantics](https://www.w3.org/TR/wai-aria-1.2/#log)
provide sequential, non-interrupting announcements alongside visual effects.
- [Meet reactions](https://support.google.com/meet/answer/13151720?hl=en)
attach expressions to participants and let viewers reduce visual distraction.
Piik likewise keeps reactions outside the shared picture, with explicit local
display preference and a separate accessible event description.
- [Microsoft participant cards](https://support.microsoft.com/en-us/outlook/profile-cards-in-microsoft-365)
place person-specific actions on the selected person. Piik uses that direct
entry on its existing couch rather than requiring a second participant list.
- [Native popovers](https://developer.mozilla.org/en-US/docs/Web/API/Popover_API/Using)
provide top-layer placement, light dismissal and focus return without adding
a second overlay framework. Piik owns the bounded card's anchor and contents.
- [Animation time](https://developer.mozilla.org/en-US/docs/Web/API/Animation/currentTime)
lets a newly mounted or resumed effect seek to its remaining phase. The
[clock comparison](https://developer.mozilla.org/en-US/docs/Web/API/Performance/now#ticking_during_sleep)
explains why replacing wall time with a monotonic clock alone would not solve
cross-platform sleep and resume.
- [niconico playback controls](https://help.nicochannel.jp/hc/ja/articles/4414625162777-%E5%8B%95%E7%94%BB%E3%81%AE%E8%A6%96%E8%81%B4%E6%96%B9%E6%B3%95)
keep comment display optional alongside playback. Piik likewise separates
each viewer's overlay preference from message delivery and the conversation.
docs/research/room-voice.md
-287
View File
@@ -1,287 +0,0 @@
# Room Voice: Design Assessment
Reviewed 2026-09-24 against public `v1.6.3` (`f423df0a`). This is a proposal,
not an accepted product contract or implementation authorization. The separate
room-interaction experiment supplies candidate lifecycle behavior; it has not
been integrated. [TODO](../todo.md#now) owns the outstanding work and decisions.
## Recommendation
Keep existing room authority and the Host-rooted screen route. Model room voice
as participant-owned microphone publications and independent listening. Chat,
screen playback and voice share admission, not a media lifetime. Do not turn the
screen tree into a conference graph or introduce another room/session store.
There is no equally small solution that simultaneously promises all 21 members
can talk, preserves every P2P-only deployment, and supplies conference-grade
connectivity. Choose the first voice scope before implementing its transport.
For the next technical experiment, assess a small direct group and audio
recapture isolation first. Four total voice participants is a useful test size,
**not an accepted product limit or a demonstrated performance guarantee**.
If full-room voice is required initially, assess a mature conference SFU before
building a mesh intended to grow into one.
## Current Owners And Actual Gaps
| Current owner | Verified behavior | Consequence for voice |
| --- | --- | --- |
| Room store and authenticated signaling | Room admission, participant identity, grant revocation and session replacement | Extend this authority; no voice login or second membership store |
| `HostAudio` in `src/client/media/host-audio.ts` | Source and microphone mixed into one share-scoped track | This is commentary; it cannot supply independent conversation |
| `internal/app/nativehost/audio_mix.go` | Native source/mic PCM share one mixer and output clock | Keep source audio here; avoid duplicating the Host mic across outputs |
| `src/client/media/viewer-audio.ts` | Audio follows the picture's play/pause/mute state | Room speech needs its own playback owner |
| Screen controller and media edges | One Host source, one upstream per Viewer, bounded screen-copy capacity | Voice must not borrow screen generations, first-video-frame proof or copy slots |
| Embedded SFU admission/media | One committed room publication, video-required metadata, share-scoped resource identity | Multiple audio-only publishers need a new capability, not another flag |
The SFU constraints are explicit in `validPublicationMedia` in
`internal/server/sfu/media.go`, `ResourceFence` and `CommitPublication` in
`internal/server/sfu/admission.go`, and the Host/Viewer dispatch in
`internal/server/signal/router_sfu_signal.go`.
[ADR-0013](../adr/0013-embedded-node-local-media.md) embeds Pion/LiveKit components;
it does not embed LiveKit's complete room service.
The interaction candidate separates a retained Host room session from screen
publication. That is a useful foundation, not permission to allocate rooms on
page visit. Preserve the accepted entry: first share creates the room and an
existing room can resume. Retaining opted-in interaction after stopping media is
the candidate lifecycle, not published behavior. A separate voice-only
room-creation flow would be another product decision.
## Transport Choice
| Approach | Benefit | Cost and boundary |
| --- | --- | --- |
| Small direct voice mesh | Keeps media P2P and does not require a reachable central media server | Each sender sends to every listener; needs a total voice-member bound, separate resource accounting and real NAT/device tests |
| Mature multi-publisher SFU | Each speaker uploads once; suitable direction for full-room voice | New publication/subscription integration and central-media availability/privacy decision |
| Host forwarding or mixing hub | Fewer peer relationships | Reintroduces Host dependency, forwarding/mixing and echo ownership; unsuitable for Host-independent conversation |
| Custom voice relay tree or automatic mesh/SFU switching | Potentially combines deployment options | Adds another routing/recovery problem; outside the first scope |
For a full bidirectional mesh, `N` participants need `N*(N-1)/2` pair connections
and up to `N-1` microphone copies per sender: 21 people means 210 pairs and 20
copies; four means six pairs and three copies. These are topology counts, not
bandwidth or CPU measurements. Limiting speakers alone does not bound sender
upload: one speaker and 20 listeners still need 20 copies. The screen's existing
`C=1..3` limit must retain its meaning; any independent voice budget needs explicit
acceptance and combined-load measurement.
Two SFU implementations deserve a bounded comparison: a complete mature room
engine, or a narrow participant-audio adapter around existing media components.
The former adds deployment and potentially competing session ownership; the
latter leaves Piik responsible for admission, subscriptions, recovery and cleanup.
Do not describe the adapter as mature conference behavior merely because its
forwarder uses mature libraries. LiveKit's
[room/participant/track model](https://docs.livekit.io/intro/basics/rooms-participants-tracks/)
and [SFU architecture](https://docs.livekit.io/reference/internals/livekit-sfu/)
show the layers involved.
Select one topology for the first capability. Jitsi's
[two-party P2P design](https://jitsi.org/blog/p2p4121/) retains its bridge connection
and changes transport when group size changes: useful evidence that a hybrid
mode has real lifecycle cost. It is not a reason to copy that complexity here.
Deployment is part of this decision. Demo and App Local/public-link modes are
P2P-only today. Public-link tunneling carries HTTP/WebSocket control traffic,
not a public SFU media endpoint. Existing STUN is not TURN, and existing embedded
UDP media does not provide media TCP/TLS. A full conference engine's
[connectivity options](https://docs.livekit.io/transport/self-hosting/ports-firewall/)
are not inherited by importing its library. This proposal enables no new service
or port and does not change Demo configuration.
## Proposed Responsibility Boundaries
```text
Existing room authority and authenticated signaling
+-- screen publication -> existing P2P/SFU screen route -> picture/source sound
+-- interaction events -> bounded chat/reactions
+-- voice admission -> participant microphone -> selected voice transport
-> independent voice playback
```
These are responsibility boundaries, not a request for a generic media framework
or one new service per line. Keep the first implementation cohesive:
| Responsibility | Owns | Does not own |
| --- | --- | --- |
| Room authority | Who belongs, who may publish/listen, bounds and retirement | OS microphone permission or local volume |
| Participant voice session | Explicit join intent, current admitted publication/subscriptions and their transport lifetime | Screen routes, share generation or a second reconnect policy |
| Microphone input | One acquisition, device choice, local mute, replacement and disposal | Remote permission to turn capture on |
| Voice playback | Received tracks, local volume/person mute, output capability and autoplay recovery | The video's pause/mute state or remote speaking permission |
Do not attach room speech to a screen peer merely to reduce connection count:
screen parent replacement and share retirement have different lifetimes. The
selected voice transport owns its connections and counts them explicitly.
Prefer a Browser-owned room microphone for both roles, including a Host using
App screen capture. This reuses platform speech processing and avoids a second
native voice service. Reuse existing device-selection semantics, not native
device identifiers. Browser microphone acquisition requires a secure context;
plain HTTP LAN pages and unsupported browsers must have truthful capability
presentation. App detection alone does not establish microphone availability.
See the [capture specification](https://www.w3.org/TR/mediacapture-streams/).
Browser voice also does not inherit the native Pion path's port mapping, NAT
assistance or native media fallback just because an App is attached. Measure
voice reachability independently; a working screen path does not prove it.
Room voice and source sound need different audio treatment. Voice uses the
platform's speech/AEC facilities; music/game sound keeps its fidelity settings.
Do not force both through the existing source-audio Opus profile or write a new
voice codec, AEC, PCM subtraction routine or virtual-audio driver.
## Consent, Lifecycle And Presentation
Proposed first permission model: admitted members may explicitly join voice and
then unmute themselves; existing room removal remains the authority boundary.
Start with a closed microphone. Joining may unlock playback, but only explicit
unmute requests input permission. If Host-only speaking approval is required,
specify it before adding a moderation UI. No remote action may start a microphone;
LiveKit also disables [remote unmute by default](https://docs.livekit.io/intro/basics/rooms-participants-tracks/participants/).
| Event | Proposed behavior |
| --- | --- |
| Share start/stop/pause, source or quality change | Voice stays independent |
| Local mic mute | Silence that publication without leaving voice; retain explicit device intent |
| Local person mute or voice volume | Change local listening only; picture volume remains separate |
| Leave voice | Release mic and voice media, remain in room/chat |
| Host leaves page while room service remains | Retire Host voice; other admitted participants may continue |
| App exits in Local/public-link mode | Room service disappears; independent voice cannot outlive its authority |
| Site-mode App media fails | Browser voice may continue while the Site room authority remains |
| Revocation, replacement or room close | Retire affected resources and reject stale completions/signaling |
| Mic permission/device failure | Preserve picture and other voices; show the actual input failure |
| Hidden page or phone interruption | Visibility alone is not leave/mute; report actual suspension/device loss |
Transient signaling loss is not automatically a media failure or a grant
revocation. Current `presence.go` and `route.Controller.DisconnectSession`
distinguish disconnected sessions from confirmed departure; Viewers have bounded
membership grace. Voice must consume that authority owner's transitions, with no
parallel grace timer or indefinite stale-session publication. Before implementation,
define and test the Host case, reservation retention, reconnect re-admission and
revocation during an outage. Do not extrapolate Viewer grace to a Host or reuse
an old session ID as current authority. This is a design gate, not a claim that
screen recovery already implements voice recovery.
One input owner prevents duplicate captures. Retire late permission results after
leave; prepare a replacement device before releasing a healthy old input; keep a
failed explicit selection visible rather than silently choosing another mic.
Repeated mute/join/leave must converge without a second saved effective-state flag.
Ordinary publication mute need not renegotiate every time; mature systems
[distinguish mute, unpublish and unsubscribe](https://docs.livekit.io/intro/basics/rooms-participants-tracks/tracks/).
Use the existing participants/sofa and room actions, not a conference grid.
First controls: join/leave voice, mic toggle and input choice/check, independent
voice volume, and local person mute in the participant menu. Keep source-sound
settings with the picture. Default to system output; show output selection only
where supported. Do not add deafen, global push-to-talk or a permission-role matrix
until their need and restoration behavior are established.
Display local intent, authorized membership, transport readiness and actual
playback separately. A speaking pulse follows observed audio activity, not a
connected flag or UUID animation seed. Catch blocked playback, including tracks
arriving after joining, and offer an explicit enable-audio action; compare
[LiveKit's autoplay handling](https://docs.livekit.io/reference/components/react/component/startaudio/).
Mobile lock-screen calling remains unproven; platform suspension is not fixed by
extra timers. See [browser lifecycle guidance](https://developer.chrome.com/docs/web-platform/page-lifecycle-api).
## Audio Isolation Is A Prerequisite
Whole-system loopback records received room speech played on the captured output
endpoint. That speech can then be sent back through screen audio. Headphones
remove an acoustic path, not this digital recapture; microphone AEC does not clean
the separate source-audio track. This matters even if only the Host shares a screen.
Windows exposes narrower process-tree capture and process exclusion. Current
Piik includes the selected process tree for window sound or captures the output
endpoint for system sound; it does not implement process exclusion. Excluding
the browser that plays Piik voice could also exclude intended browser content.
Test actual process ownership and output routing rather than assuming a window
selection isolates sound. See Microsoft's
[loopback description](https://learn.microsoft.com/en-us/windows/win32/coreaudio/loopback-recording)
and [process-capture sample](https://learn.microsoft.com/en-us/samples/microsoft/windows-classic-samples/applicationloopbackaudio-sample/).
Browser `restrictOwnAudio` is relevant; `suppressLocalAudioPlayback` has a different
purpose. Chrome documents the former as excluding audio originating from the
capturing document when system audio is captured. Check support, applied settings
and decoded output on supported platforms; a hint is not cross-browser proof.
See the [Screen Capture specification](https://www.w3.org/TR/screen-capture/)
and [Chrome's implementation announcement](https://developer.chrome.com/blog/chrome-141-beta).
Verify source sound and incoming voices using distinguishable signals. Cover
native window/system capture, browser tab/window/system capture, source changes,
same-browser content and output-device changes. Where isolation cannot be
established, a supported combination or explicit feature limitation must be
chosen before release. Never silently discard source sound or promise that a
headphone hint solves it.
## Compatibility, Trust And Bounds
The Host microphone must have one distribution path. Sending it in both existing
commentary and independent voice causes duplicates and defeats individual mute.
Moving it exclusively to voice can make older/non-participating Viewers lose
commentary. Choose an explicit room capability/participation boundary or a
coordinated major-version transition; do not ship a silent behavior change.
The [versioning owner](../standards/versioning.md) governs that choice. This
assessment does not select a release number or authorize removal of native mic.
Once that boundary is selected, define the handoff before coding: retire the old
microphone distribution before enabling the new one, and test failure/cancellation
without silently losing commentary for Viewers who cannot receive room voice.
Browser-first capture can avoid new native voice commands, but does not make
room signaling automatically compatible. Trace strict readers in both directions;
advertise support before new messages. Keep stable peer identity, authenticated
session, grant and media connection identity distinct. Reuse an existing identity
only when its lifetime matches; do not add a room-wide epoch for convenience.
Bind signaling to current authenticated participants and exact publication/peer
resources; bound memberships, publications, subscriptions, candidates and message
rates before allocation. Leave/revoke must release actual resources before their
slots become reusable. No audio through chat events or durable voice recordings.
P2P revocation has an enforcement limit: the server can refuse signaling and
conforming endpoints can stop capture/drop tracks, but it cannot intercept media
between two modified peers. An SFU can enforce its own forwarding removal. Do not
promise identical server-enforced muting across these models, or hide the central
server's media access under the existing P2P privacy label.
## Implementation Sequence And Acceptance
1. Reconcile the isolated interaction candidate with current main and accept its
room/picture separation on its own. Preserve the existing staged work; do not
merge an old branch wholesale. Voice need not block chat/UI acceptance.
2. Select total voice membership, deployment coverage, speaking policy and the
commentary/compatibility boundary. Record accepted changes in the existing
product owners and an ADR only when architectural decisions are accepted.
3. Run an isolated transport/audio-isolation experiment before shipping UI.
Measure screen plus voice CPU, upload, number of connections, audible latency,
NAT failure, echo and cleanup. Use one topology; stop if the required scope
implies a custom conference engine or unsupported audio isolation.
4. Implement one vertical slice through the owners above: explicit join, muted
input, unmute, remote audio, local mute and leave. Then expand participants and
exact authority teardown. Reuse existing controls, status/comic meanings and
diagnostic export; avoid separate feature-local rules.
5. Accept faults as well as the happy path: permission resolves after leave,
device switch fails, source changes, autoplay is blocked, signaling drops
during healthy audio, reconnect races revoke/replacement, Host leaves, service
exits, mobile suspends, and repeated join/leave leaks no resources. Test quiet
and muted participants without video-frame admission requirements. Include
supported old/new endpoints and Chinese/English/visual presentation.
Transport experiments must report measured limits rather than treating a
successful localhost call as public-network or mobile acceptance. This planning
round ran no voice, acoustic-echo or device-performance experiment.
## Independent Fidelity Work
Stereo acceptance can proceed separately: distinguishable left/right signals
through Browser, Native, relay and SFU, with microphone mixing and source changes.
The released native answer fix negotiates receive-side stereo; SDP alone does not
prove decoded channel separation. Opus negotiation follows
[RFC 7587 section 7](https://www.rfc-editor.org/rfc/rfc7587.html#section-7).
Do not add surround channels as a prerequisite for voice.
For [HDR issue #420](https://github.com/TNTcraftHIM/Piik/issues/420), investigate
correct SDR output first: preserve HDR range during native capture, then use
platform tone/gamut mapping before the existing SDR encoder. Windows documents
[floating-point capture](https://learn.microsoft.com/en-us/windows/apps/develop/media-authoring-processing/screen-capture)
and a [tone-map effect](https://learn.microsoft.com/en-us/windows/win32/direct2d/hdr-tone-map-effect).
Validate mixed SDR/HDR content, highlights, SDR reference output and display-mode
changes. Already-clipped browser input cannot be restored downstream. Full HDR
transport/metadata, surround sound, recording and transcription remain outside
this proposal.
docs/standards/configuration.md
+8
View File
@@ -56,6 +56,7 @@ Automatic SFU fallback runs inside the Hosted process when `SFU_UDP_PORT` is set
| `SFU_UDP_PORT` | Optional UDP media port `1..65535`; unset or blank disables SFU and fixes Privacy mode on for every room. Set `7882` for the standard public listener and allow the Host to choose Privacy mode. |
| `SFU_LISTEN_HOST` | IPv4 bind address, default `0.0.0.0`; independent of HTTP `LISTEN_HOST`. Read only when SFU is enabled. |
| `SFU_PUBLIC_IP` | Optional explicit IPv4 advertised-address override for a host behind NAT. Read only when SFU is enabled. |
| `SFU_ONLY` | `false` by default. `true` requires `SFU_UDP_PORT` and sends every room through SFU, without Peer candidates or background P2P convergence. Applies after Server restart. |
SFU control uses the application's authenticated signaling connection. No
separate control origin or infrastructure credentials are configured. Local and
@@ -67,6 +68,13 @@ configured STUN and peer relays remain; no media-server fallback is possible.
The server enforces this even if a Host requests hybrid mode. There is no
separate `SFU_ENABLED` flag to conflict with the listener configuration.
With `SFU_ONLY=true`, the page shows a locked **Server media** policy in place of
P2P settings. An older page requesting Privacy mode is refused before sharing;
refresh it or turn Privacy mode off. Capacity exhaustion and connection failure
use the existing bounded SFU admission/recovery path, never a P2P escape.
This policy needs sufficient server bandwidth and a working UDP route; it is not
a TURN/TCP fallback. App Local/public-link and the public Demo remain P2P-only.
The SQLite parent directory must exist and be writable. The systemd template
sets `/var/lib/piik/rooms.sqlite` under its managed state directory; the
container image sets `/home/nonroot/rooms.sqlite` under its writable data
docs/standards/engineering.md
+4 -1
View File
@@ -148,7 +148,10 @@ above rather than searching the tree.
Closing a Browser connection may leave its API promises pending. Its owner
cancels the corresponding waits so a serialized successor never depends on
completion from a retired connection; late results still need identity checks.
Later cancellation must not erase a failure already observed before cleanup.
Failure effects need the same final identity check as successful results,
serialized with replacement; an earlier input check alone is insufficient.
Later cancellation must not erase a failure already accepted by its current
owner. Perform blocking retirement after releasing that owner's state lock.
A caller deadline does not cancel a dependency that ignores it. Keep late
work bounded and serialize its cleanup; verify the dependency's actual lifetime.
Construction is part of acquisition: if it throws, release resources created
docs/standards/media-quality.md
+45 -3
View File
@@ -39,6 +39,17 @@ production behavior and remaining acceptance.
frame matches its desktop source dimensions. Other frames retain their own
aspect ratio. This changes only the encoded presentation, never the game or
display settings; vendor-private scaling is not inferred.
- Windows SDR conversion owns one output color contract: limited-range BT.601
NV12 with matching H.264 metadata. RGB capture and decoded video retain their
input range/matrix through conversion and relay scaling; dimensions do not
determine color space. Unspecified SDR follows the WebRTC convention.
- Windows native HDR sources retain scRGB FP16 until platform tone mapping,
source-display white-level adjustment and sRGB conversion produce SDR. Live
capture and source thumbnails share this owner; conversion precedes output
fanout and does not add a media route or per-encoder tone curve. Ordinary SDR
keeps its copy path. Source-display color changes refresh capture input,
not room authority or transport. [Fidelity evidence](../research/media-fidelity.md)
owns platform checks; this is HDR-to-SDR, not end-to-end HDR delivery.
- Where Windows supports border control, the native source picker offers
**Show capture border**, off by default. The choice stays in the Host page and
follows native source and quality changes. Source previews request borderless
@@ -177,12 +188,32 @@ closing a Browser PeerConnection alone does not guarantee that its promises
settle. Cancellation starts before joining work that depends on it, while a
replacement still waits for the retired owner's cleanup.
Browser and Native encoder-group admission uses current native demand, not the
previously applied rate. A new or pending join must not displace an existing
rate owner by raising or retaining its budget against that owner's demand.
Revalidate pending admission before planning other memberships.
Eligible Browser P2P parents use the source-owned pool in
[ADR-0014](../adr/0014-browser-node-local-encoding-pool.md). Compatible direct
children share one independent local WebRTC producer; incompatible demands
remain separate. Each outgoing connection keeps native transport, allocation
and recovery, with a tiny carrier supplying its RTP clock. Producers use
the existing native video target under Host ceilings. Actual forwarded-frame
remain separate. Joining another producer must fit the child's native allocation;
a quiet scene's low byte rate does not prove that producer's rate budget fits.
Pending separation remains necessary while another compatible member needs a
higher native budget; quiet output alone cannot cancel it. When that demand
leaves or the child's budget recovers, the current producer can adapt in place.
Optional reuse also preserves the healthy current output's rate budget and
observed quality within Host ceilings through recovery-frame commitment, using
the actual candidate frame and current output rather than preparation alone.
Rejected membership retains current delivery and its recovery requests.
A recovery frame already being written retains its owner until completion,
including across pause/resume; cancellation belongs to the output that knows
whether writing has started.
Each outgoing connection keeps native transport, allocation and recovery, with
a tiny carrier supplying its RTP clock. Producers use the existing native video
target under Host ceilings. A new producer seeds its
local WebRTC start rate from that allocation, without a minimum or a change to
the outgoing connection's estimate. Subsequent budgets use the existing sender
parameters; initialization does not add a recovery controller. Actual forwarded-frame
and producer observations remain distinct from the tiny carrier's statistics.
Producer startup protection begins at actual outgoing publication, excluding
local warmup and paused frames. The synthetic carrier uses screen-content
@@ -294,6 +325,17 @@ A Native audio-process failure does not end healthy video. A later explicit
source replacement resumes audio through the existing track and encoder owner;
it does not trigger an automatic capture retry or change room audio topology.
Optional Windows screen-audio exclusion binds one selected process tree to its
PID and creation time under [ADR-0008](../adr/0008-window-scoped-audio-capture.md).
The selection lasts for the share, survives source-audio off/on and refresh,
and is never saved across App runs or rebound by executable name. Changing to a
window source or explicitly selecting no exclusion clears it. A requested
exclusion retires previous source audio before replacement preparation; failure
or target exit leaves source audio silent while healthy video and the Host
microphone continue. Only an explicit source selection resumes it. A missing
capability must reject the request, never discard the exclusion. Old pages keep
their existing audio behavior.
## Observable Truth
Requested resolution, frame rate, bitrate, content intent, codec, and audio
docs/standards/naming.md
+3
View File
@@ -40,6 +40,7 @@ Set the register by the reader's task:
| Surface | Register |
| --- | --- |
| Website | Light, everyday language in introductions and illustration captions; keep the current restraint |
| App and Web controls | Direct feature and action names, including panel titles and settings; “聊天” / “Chat” rather than an invitation or joke |
| README | A friendly introduction, followed by direct explanations of features, limits and first steps |
| Documentation index | Clear topic names and short descriptions that help readers choose a guide |
| Deployment, configuration and maintenance guides | Formal, precise instructions: prerequisites, actions, expected results and recovery steps |
@@ -102,6 +103,8 @@ in the welcome or illustration caption. Do not soften a failure into a joke,
invent connection progress or promise guaranteed connectivity or absolute privacy.
Screen sharing includes games, creative work and showing a useful discovery.
Playfulness belongs in optional welcome/waiting lines, empty-state captions and
illustration reactions; it does not rename the control that performs the action.
Operational pages omit introductory sentences that repeat their controls.
Helper text should explain a meaningful choice or prevent an error.
docs/standards/presentation-lifecycle.md
+9
View File
@@ -24,6 +24,15 @@ room identity, current state, roster, and one manual media-reconnect action.
Controls and text must remain usable without overlap or horizontal overflow on
desktop and mobile viewports.
A capable, opted-in authenticated room session
outlives its current publication. Stopping capture retires media, not chat;
restoring an existing room joins it without starting capture. Entering the page
alone does not create a room. Room close, lost authority and page teardown still
retire the session. Connected Viewers may keep interacting while the Host page
is absent, provided the room service and their admitted sessions remain available.
[Rooms and access](./rooms-access.md#room-interactions) owns interaction authority
and retention; [status](../status.md#accepted-interaction-phase) owns release readiness.
Piik App opens this same application in the system Browser. Its small
startup surface selects Local, temporary public invitation, or a configured
Site before entering the Host workspace. Viewer links opened at that activated
docs/standards/rooms-access.md
+21 -1
View File
@@ -1,6 +1,6 @@
# Rooms And Access
This file owns the current room, admission, invitation, and persistence
This file owns the current room, admission, invitation, interaction, and persistence
contract. [ADR-0002](../adr/0002-memory-resident-protected-rooms.md) explains
the storage decision; implementation detail belongs in code and tests.
@@ -91,6 +91,26 @@ recreate a revoked subscription; no separate media token survives revocation.
Changing code-entry policy or password affects later
code-only attempts and does not silently revoke invitations.
## Room Interactions
Text and reactions reuse the authenticated room signaling connection. The server
derives sender identity from that session and accepts targeted reactions only
for participants still connected to the same room. Interaction subscriptions
inherit existing membership, replacement and revocation checks; they grant no
additional authority and do not modify media routes.
Delivery is transient and bounded. The server keeps no message history; each
page keeps only a bounded set of received messages and effects, without writing
them to storage or diagnostics. Re-admission to the same room may retain that
page's conversation, but refresh, leaving or authority loss clears it. There is
no history replay or offline outbox. HTTPS/WSS protects transport; chat is not
end-to-end encrypted, and the site operator can access live message contents.
[Presentation and lifecycle](./presentation-lifecycle.md#product-surface) owns
the distinction between a room session and its current media publication.
[Interaction research](../research/room-interactions.md) records delivery bounds,
compatibility and design rationale.
## Persistence Modes
- **Stable mode (Hosted default):** one exact-schema SQLite owner persists room
docs/standards/routing-transport.md
+10 -2
View File
@@ -7,7 +7,7 @@ owns ports and [self-hosting](../operations/self-hosting.md) owns service setup.
## Topology
- Media is automatic and P2P-first. Users do not select parents, route types, or
- Media is automatic and defaults to P2P-first. Users do not select parents, route types, or
network transports.
- The Host is the only source. Every Viewer has at most one active upstream, and
the committed media graph is acyclic and source-reachable.
@@ -21,9 +21,17 @@ same bounded P2P graph without its SFU fallback and fixes Privacy mode on for al
rooms, including App Local and public-link rooms. The server enforces that policy
for waiting and active participants; there is no separate Host-star mode.
A Hosted operator may require SFU-only media through `SFU_ONLY=true`. The same
controller then admits only SFU candidates: no Peer parents, Viewer relays or
background P2P convergence. The Host still publishes once, and every Viewer
uses the existing SFU subscription ledger and first-frame commit. Capacity or
connection failure never permits a Peer route. An explicitly peer-only share
is rejected before acquiring media authority; it is never silently forwarded.
The page advertises this site policy and hides inapplicable P2P preferences.
## Per-Share Route Policy
The Host chooses route policy before sharing and it remains fixed for that
On a P2P-first site, the Host chooses route policy before sharing and it remains fixed for that
share generation:
- when SFU is available, default hybrid mode keeps P2P first and permits the
docs/standards/visual-language.md
+11
View File
@@ -56,6 +56,15 @@ do not stretch short values to fill a phone row. Check actual control bounds
around breakpoints, in both orientations and all presentation modes; page-level
overflow clipping is not evidence that the content fits.
Persistent room utilities use the shared floating panel without reflowing the
living room. The shell owns visibility, focus return, movement, resizing and
viewport/keyboard bounds; contents retain their domain state. Support title-bar
dragging and keyboard placement. An interaction palette starts beside its chosen
participant, stays open after sending and keeps its position when retargeted.
One-shot settings and guidance retain their existing disclosures. Escape closes
inner guidance first, then the most recently opened floating panel, then theater
mode; an IME composition must not dismiss these surfaces.
Keep the application language control's segmented Chinese, English and visual
shortcuts. When additional catalogs exist, insert one language menu
after the visual shortcut; all extra languages share that slot. Show the
@@ -172,6 +181,8 @@ Participant characters should leave room for that distinctive presence.
| Browser application | Browser chrome around content; not a second design of television |
| A captured window or display | Window title bar or display stand, with the shared media metaphor kept distinct |
| A camera source or Host commentary | `camera` identifies the alternative picture source; `microphone` identifies the enabled Host voice input and `microphoneOff` its muted state. Neither replaces the speaker metaphor for source/playback sound. |
| Room text and reactions | `chat` opens the conversation, `danmaku` controls its optional picture overlay, `smile` identifies reactions and `send` submits the chosen content. These do not represent media transport or voice. |
| Floating utility placement | `grip` marks a draggable title bar; `move` opens placement controls. Closing a panel hides its presentation, not its room session. |
| Server forwarding media | A server on the media path; opening a Site alone does not imply SFU |
These are **identity/object colours**, not status. A green pawn remains green
docs/status.md
+19 -1
View File
@@ -1,6 +1,6 @@
# Current Status
Last updated: 2026-09-29
Last updated: 2026-10-02
This is the compact execution/deployment index. Product modules own behavior,
[verification status](./verification-status.md) owns unresolved physical limits,
@@ -35,6 +35,24 @@ Website delivery includes a bilingual reader documentation center generated
from the repository's guides alongside the homepage, with search and detailed
troubleshooting separate from the homepage FAQ.
## Accepted Interaction Phase
The accepted [room-interaction design](./research/room-interactions.md) includes
the verified media repairs. It adds authenticated text and reactions
over existing signaling, with optional local chat overlay and a shared floating
panel. Its opted-in room session can outlive a publication; capture, quality and
route owners remain unchanged. Optional Windows screen-audio exclusion uses the
existing source/mixer boundary. Hosted Server
also accepts an optional SFU-only policy through the same route controller.
Windows native capture and thumbnails now share automatic HDR-to-SDR conversion;
synthetic GPU and Browser reception checks pass, with physical HDR display
acceptance still pending in [verification status](./verification-status.md).
Room voice is outside the product plan; Host microphone commentary remains.
4K remains deferred. The owner accepted the UI/model and bounded local acceptance
and authorized v1.7.0 publication. Physical device limits remain open.
Release artifacts and operator deployment records own delivery completion;
[TODO](./todo.md#now) owns remaining work.
## Deployment
Source, package identity, GitHub repository and local worktree paths use Piik.
docs/todo.md
+48 -46
View File
@@ -1,70 +1,58 @@
# Current TODO Ledger
Last reviewed: 2026-09-30
Last reviewed: 2026-10-02
Only **Now** is executable. Product modules own behavior; Git/PRs own completed
history. A parked idea is not implementation authority.
## Now
- [ ] **Published-product bug and experience acceptance.** Prioritize share
startup, audio/source changes, sustained viewing, recovery and retirement,
admission, and narrow/touch UI. Verify confirmed fixes through the existing
owners before resuming feature polish. Keep field reports without matched
evidence separate from locally reproduced defects.
- [ ] **Sustained Browser degradation.** The H264 field cause remains open.
- [ ] **Authorized release delivery.** The owner accepted the room-interaction
phase, complete bilingual release notes and bounded acceptance, and authorized
v1.7.0 publication. Follow
[the integration workflow](../CONTRIBUTING.md#full-integration-and-release-workflow)
for one squash merge, packaging, publication and scoped postflight. Release
artifacts and operator records own completion; do not add a release-record
commit. No implementation work remains open for this phase.
## Held By Owner
- [ ] **Browser-internal encoder work.** Paused by the owner on 2026-10-01.
[Comparisons and Chromium traces](./research/browser-local-encoding-pool.md#sustained-h264-recovery)
distinguish encoder recovery hysteresis from a tab-capture feedback/size-step
lock reproduced in bare WebRTC. Verify the upstream soft-target/hard-maximum
repair candidate against a patched Browser, which remains unbuilt. The Piik
sender boundary now isolates display frames while preserving native idle refresh,
capture intent and operation retirement. Real hidden-tab SFU checks pass;
broader device/background coverage remains. Native logs confirm content-driven
QP hysteresis and a cost from bitrate reconfiguration, but not repeated brief
blur. Match field Host/Viewer evidence
before assigning the field cause or declaring repeated brief blur fixed.
Retain existing clone/startup protections and candidate proof; no periodic
reset or quality floor. A recent Piik regression is not established.
own the evidence for synchronous H264 initialization, native adaptation and
the unbuilt upstream repair candidate. Keep the verified Piik budget and
handoff repairs; do not add periodic resets, a quality floor or a Browser fork.
The field room's cause and all brief blur are not established as resolved.
## Deferred Feature Work
Feature design and interaction polish are paused while the bug/experience phase
is active. Retain these decisions for resumption, not as parallel work.
These proposals remain deferred beyond the accepted interaction phase.
- [ ] **Passive App attachment: design hold.** Site mode authorizes one selected
origin and supplies native media without starting a local room server. A
passive replacement needs an accepted site-consent/discovery flow; it must not
admit arbitrary sites or add another runtime owner. Keep Site mode until that
decision; removal of the Demo prefill does not authorize changing site trust.
- [ ] **Refine the isolated room-interaction prototype.** Review chat, optional
danmaku and participant-targeted reactions for usability, placement and intended
cross-view synchronization. Preserve room authorization and media-route owners.
Accept its combined layout: sharing settings with the picture, floating chat
without page reflow and room actions before diagnostics. Check narrow-screen
keyboard avoidance, its theater entry and participant menu on real devices;
keep chat available independently of sharing.
Retain current room creation: the first share creates a room, an existing room
can resume, and stopping media keeps the opted-in interaction session. Do not
add visit-triggered creation or a separate pre-share room-creation entry.
This experiment does not authorize integration or publication.
- [ ] **Room voice: design before implementation.** The
[voice assessment](./research/room-voice.md) maps current owners, mature options
and a staged verification plan. Resolve total voice membership/topology,
P2P-only deployment coverage, speaking authority and the Host-commentary
compatibility boundary before implementation. Validate source-audio isolation
from received speech before exposing simultaneous sharing and voice. Keep
room authority, microphone and screen lifetimes distinct; no new room store,
screen-tree reversal or automatic mesh/SFU controller is approved.
- [ ] **Verify media fidelity independently of voice.**
Verify actual left/right separation through Browser, Native and SFU paths;
receive-side stereo negotiation alone does not establish audible fidelity.
Assess HDR capture to correct SDR output for ordinary viewers before full HDR,
with issue [#420](https://github.com/TNTcraftHIM/Piik/issues/420) as the reported
case. Full HDR and surround sound remain design decisions. These findings do
not establish physical device acceptance or authorize a new release.
## Awaiting Device Or Reporter Evidence
- [ ] **HDR, audio-exclusion and mobile interaction device coverage.** Complete
the remaining physical checks in
[verification status](./verification-status.md#candidate-evidence-boundary):
real HDR/mixed displays under load, real voice-app process trees across playback
devices, and phone keyboard/theater/participant-menu interaction. Local GPU,
decoded-video, synthetic audio and responsive-browser evidence do not cover
those environments. Issues [#420](https://github.com/TNTcraftHIM/Piik/issues/420)
(Browser HDR) and [#445](https://github.com/TNTcraftHIM/Piik/issues/445)
(Windows 10 colors) still need matched capture evidence; do not assign them
the locally reproduced Native conversion defects.
- [ ] **Native H.264 motion quality (#432).** The original 720p30 H.264/VP8
visual comparison remains unresolved. Browser evidence confirms similar
delivered bitrate/format and zero reported loss, but contains no matched
pictures or bitstream. The reporter withdrew the fixture-contaminated CBR
ceiling claim. [Evidence review](./research/native-client-media.md#native-h264-motion-quality)
owns that distinction. Reproduce with identical content and unchanged
production settings before proposing profile, default-FPS or bitrate changes.
- [ ] **Self-hosted room creation HTTP 403.** Retest affected deployments using
the [address checks](./guide/troubleshooting.md#room-creation-returns-403).
Obtain the configured public address/origin and response details; distinguish
@@ -78,6 +66,9 @@ is active. Retain these decisions for resumption, not as parallel work.
evidence before calling that an ICE/NAT failure. HTTP 1033 belongs to the
public-link item below, before media routing. Local fixes and STUN checks do
not establish these reporters' causes.
[#443](https://github.com/TNTcraftHIM/Piik/issues/443)'s Host report reaches Native
prepare/answer/candidate processing without a connected remote edge; obtain
the same attempt's Viewer and App transport reports to locate the failure.
- [ ] **Camera and Host microphone device coverage.** The owner accepted the
sharing layout and authorized release with these physical limits recorded.
Verify real audio levels/echo, multiple-device replacement and native mixing on
@@ -92,6 +83,9 @@ is active. Retain these decisions for resumption, not as parallel work.
[runtime evidence](./research/cross-platform-client-runtime.md#public-invitation-startup).
Demo access failures and public-link HTTP 1033 also remain unconfirmed; check
connector, DNS/provider and remote access separately from WebRTC availability.
[#434](https://github.com/TNTcraftHIM/Piik/issues/434)'s attached log records DNS
refusal during Cloudflare edge discovery, before readiness; retrying media
cannot repair that resolver failure. Do not assign it to every startup report.
- [ ] **Interruption during established viewing.** A Viewer reportedly returns
to P2P connecting after watching for a while. Include
[#429](https://github.com/TNTcraftHIM/Piik/issues/429)'s reported SFU-to-P2P
@@ -126,6 +120,10 @@ is active. Retain these decisions for resumption, not as parallel work.
also needs paired evidence; distinguish activation, codec configuration and
actual output before changing the encoder contract. Local NVIDIA success
does not settle AMD activation.
In [#437](https://github.com/TNTcraftHIM/Piik/issues/437), both manually selected
codecs reportedly work; its Browser report records Auto startup returning
`operation-failed` after 6.85 seconds, before the request timeout. Obtain the
same attempt's App diagnostics to locate selection/activation failure.
- [ ] **Share ends after entering a game.** Screen sharing reportedly works
until entering a game freezes the picture, followed seconds later by share
termination. Version, capture path, codec and matched diagnostics are unknown.
@@ -249,3 +247,7 @@ not establish better connection success or speed; this note adds no retry policy
11. **Release-operation policy.** Protected release environments, immutable
draft assets and changing mirror-failure policy remain unaccepted proposals;
evaluate their benefit before adding release machinery.
12. **Linux ARM64 Server distribution (#439).** Evaluate archives and container
publishing together with deployment names, update links and runtime checks.
A cross-compile alone does not establish a supported package. This is a
community proposal, not part of the current accepted candidate.
docs/verification-status.md
+21 -1
View File
@@ -1,6 +1,6 @@
# Verification Status
Last updated: 2026-09-17
Last updated: 2026-10-01
This file owns physical evidence limits that change how the product may be
described. [Status](./status.md) owns the execution/deployment index.
@@ -47,6 +47,26 @@ or a measured improvement worth its implementation and maintenance cost.
## Candidate Evidence Boundary
The Windows HDR-to-SDR candidate has synthetic FP16 conversion and decoded
H.264/VP8 Browser reception evidence. Local displays were in SDR mode; real HDR
content, SDR white on HDR displays, mixed-display movement, display-mode changes
and sustained GPU contention still require physical acceptance. The
[fidelity assessment](./research/media-fidelity.md#hdr-to-sdr) owns measurements.
These checks do not establish Browser-capture HDR behavior or end-to-end HDR.
Windows source-audio exclusion has bounded parent/child tone-isolation and
target-exit evidence. It does not establish coverage of arbitrary process trees
or real voice applications across playback devices. The
[audio assessment](./research/native-client-media.md#windows-audio-exclusion)
owns that distinction.
Room interactions have multi-page Browser evidence for stop/resume, idle-room
join, Host departure/return, reconnect, invitation revocation and event timing.
Responsive Browser checks cover narrow portrait and landscape layouts, but not
real phone keyboards, Safari behavior or every touch/device combination. The
owner accepted this bounded UI/model walkthrough for phase closure; these
physical limits remain open and are not release authorization.
The 2026-09-11 functional test acceptance used two genuine Windows App packages
and matching Server source/Web builds under the same private contract. Both App/Site version
directions delivered decoded 1080p H264 from an isolated native test window.
internal/app/app.go
+9 -8
View File
@@ -621,14 +621,15 @@ func discoverNativeMedia(ctx context.Context, configuredPath string) nativeRunti
captureProcess: path,
capture: capabilities,
capabilities: loopback.NativeMediaCapabilities{
ReceiverReuse: true,
Video: summary.Video,
CaptureBorderControl: summary.CaptureBorderControl,
Microphone: summary.Microphone,
ProcessAudio: summary.ProcessAudio,
SystemAudio: summary.SystemAudio,
HardwareH264: summary.HardwareH264,
SoftwareVP8: summary.SoftwareVP8,
ReceiverReuse: true,
Video: summary.Video,
CaptureBorderControl: summary.CaptureBorderControl,
Microphone: summary.Microphone,
ProcessAudio: summary.ProcessAudio,
ProcessAudioExclusion: summary.ProcessAudioExclusion,
SystemAudio: summary.SystemAudio,
HardwareH264: summary.HardwareH264,
SoftwareVP8: summary.SoftwareVP8,
},
}
}
internal/app/app_test.go
+7 -1
View File
@@ -121,6 +121,12 @@ func TestPublicLinkStartupDistinguishesCancellationFromFailure(t *testing.T) {
}
cancel()
}
waitForResult := 5 * time.Second
if mode == "failed" {
// Startup includes three retry pauses and four real child launches;
// this is not the cancellation/ready-tunnel retirement deadline.
waitForResult = 15 * time.Second
}
select {
case err = <-done:
joined = true
@@ -139,7 +145,7 @@ func TestPublicLinkStartupDistinguishesCancellationFromFailure(t *testing.T) {
if ready != (mode == "ready" || mode == "ready-failed") {
t.Fatalf("startup readiness = %v for %s", ready, mode)
}
case <-time.After(5 * time.Second):
case <-time.After(waitForResult):
t.Fatal("App did not retire its local authority and tunnel")
}
content, err := os.ReadFile(filepath.Join(directory, "logs", "client.log"))
internal/app/loopback/server.go
+9 -8
View File
@@ -69,14 +69,15 @@ type Health struct {
}
type NativeMediaCapabilities struct {
ReceiverReuse bool `json:"receiverReuse,omitempty"`
Video bool `json:"video"`
CaptureBorderControl bool `json:"captureBorderControl,omitempty"`
Microphone bool `json:"microphone,omitempty"`
ProcessAudio bool `json:"processAudio"`
SystemAudio bool `json:"systemAudio"`
HardwareH264 bool `json:"hardwareH264"`
SoftwareVP8 bool `json:"softwareVP8"`
ReceiverReuse bool `json:"receiverReuse,omitempty"`
Video bool `json:"video"`
CaptureBorderControl bool `json:"captureBorderControl,omitempty"`
Microphone bool `json:"microphone,omitempty"`
ProcessAudio bool `json:"processAudio"`
ProcessAudioExclusion bool `json:"processAudioExclusion,omitempty"`
SystemAudio bool `json:"systemAudio"`
HardwareH264 bool `json:"hardwareH264"`
SoftwareVP8 bool `json:"softwareVP8"`
}
type Server struct {
internal/app/mediaedge/edge_test.go
+56 -1
View File
@@ -65,9 +65,11 @@ func TestOutputPlanRetiresOnlyFailedLayerConsumers(t *testing.T) {
lowerTracker, higherTracker := trackers.AddTracker(0), trackers.AddTracker(1)
lowerTracker.Observe(0, 1200, 1188, true, 90000, nil)
higherTracker.Observe(0, 1200, 1188, true, 90000, nil)
if err = source.DisableLayer(0); err != nil {
retire, err := source.MarkLayerUnavailable(0)
if err != nil {
t.Fatal(err)
}
retire()
if low.State() != webrtc.PeerConnectionStateClosed || high.State() != webrtc.PeerConnectionStateConnected {
t.Fatalf("failed/sibling state = %s/%s", low.State(), high.State())
}
@@ -174,6 +176,59 @@ func TestRelayFailureCannotRetireReconfiguredProfile(t *testing.T) {
}
}
func TestRetiredRelayFailureCannotDisableLaterDemand(t *testing.T) {
engine, err := NewEngine(EngineOptions{BindAddress: "127.0.0.1:0", IncludeLoopback: true})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = engine.Close() })
source, err := engine.NewSource("vp8", 2, 2, nil)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = source.Close() })
source.relay = &relayDerivation{source: source}
if err = source.SetFormat(1, 640, 360); err != nil {
t.Fatal(err)
}
if err = source.SetRelayProfile(nativecapture.VideoProfile{
Width: 1280, Height: 720, Framerate: 30, Bitrate: 3_000_000, Preference: "balanced",
}); err != nil {
t.Fatal(err)
}
high, _, _ := connectedReceiver(t, engine, source, "retiring-relay-high")
low, _, _ := connectedReceiver(t, engine, source, "retiring-relay-low")
if err = low.SetTargetLayer(0); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancelCause(t.Context())
defer cancel(nil)
done := make(chan struct{})
t.Cleanup(func() { close(done) })
run := &relayRun{owner: source.relay, ctx: ctx, cancel: cancel, done: done,
plan: relayPlan{profile: source.relayProfile, format: source.formats[1].Load()}}
source.relay.run = run
if err = low.Close(); err != nil {
t.Fatal(err)
}
if !errors.Is(context.Cause(ctx), context.Canceled) {
t.Fatal("last lower consumer did not retire the derivation normally")
}
// A failure already read from the old process can reach the source lock
// after cancellation, before the process exits and the next run starts.
retire, err := source.markLayerUnavailable(0, run)
if err != nil || retire != nil || source.outputBitrates[0] == 0 {
t.Fatal("normal retirement disabled a reusable output slot")
}
rejoined, _, _ := connectedReceiver(t, engine, source, "retiring-relay-rejoined")
if err = rejoined.SetTargetLayer(0); err != nil {
t.Fatalf("later lower demand was rejected: %v", err)
}
if rejoined.State() != webrtc.PeerConnectionStateConnected || high.State() != webrtc.PeerConnectionStateConnected {
t.Fatal("late failure retired a current consumer or healthy sibling")
}
}
func TestShutdownClosesSiblingTransportsBeforeWaitingForWrites(t *testing.T) {
for _, owner := range []string{"source", "engine"} {
t.Run(owner, func(t *testing.T) {
internal/app/mediaedge/groups.go
+21 -9
View File
@@ -95,6 +95,23 @@ func (source *Source) planGroups(demands []outputDemand) (OutputPlan, error) {
delete(source.memberships, consumer)
}
}
compatible := func(group *outputGroup, demand outputDemand) bool {
for _, other := range demands {
if other.active && other.lower && other.budget != demand.budget &&
(other.consumer.CurrentSource() == group.media.Source || source.memberships[other.consumer] == group) {
return false
}
}
return true
}
// Revalidate pending joins before they can prevent an incumbent from
// adjusting in place. Membership order must not give stale demand priority.
for _, demand := range demands {
group := source.memberships[demand.consumer]
if demand.active && demand.lower && group != nil && demand.consumer.CurrentSource() != group.media.Source && !compatible(group, demand) {
delete(source.memberships, demand.consumer)
}
}
// The previous frame's active bit alone may outlive its last handoff.
inUse := func(group *outputGroup) bool {
if group == nil || !group.active {
@@ -119,6 +136,9 @@ func (source *Source) planGroups(demands []outputDemand) (OutputPlan, error) {
for slot := 0; slot < len(source.outputBitrates); slot++ {
group := source.groups[slot]
if group != nil && source.outputBitrates[slot] > 0 && group.budget == demand.budget {
if demand.consumer.CurrentSource() != group.media.Source && !compatible(group, demand) {
continue
}
if selected == nil || inUse(group) {
selected = group
}
@@ -152,15 +172,7 @@ func (source *Source) planGroups(demands []outputDemand) (OutputPlan, error) {
if source.outputBitrates[group.slot] == 0 {
continue
}
compatible := true
for _, other := range demands {
if other.active && other.lower && other.budget != demand.budget &&
(other.consumer.CurrentSource() == group.media.Source || source.memberships[other.consumer] == group) {
compatible = false
break
}
}
if compatible {
if compatible(group, demand) {
selected = group
break
}
internal/app/mediaedge/groups_test.go
+59
View File
@@ -1,6 +1,7 @@
package mediaedge
import (
"fmt"
"testing"
"time"
@@ -253,6 +254,64 @@ func newGroupLifecycleFixture(t *testing.T) (*Source, *groupTestConsumer, *outpu
return source, b, group, demands
}
func TestOutputGroupsPreserveCurrentDemandDuringAdmission(t *testing.T) {
for _, pending := range []bool{false, true} {
for _, reverse := range []bool{false, true} {
t.Run(fmt.Sprintf("pending=%v/reverse=%v", pending, reverse), func(t *testing.T) {
source, b, gb, demands := newGroupLifecycleFixture(t)
a := demands[0].consumer
ga := source.memberships[a]
source.installGroup(ga, false)
source.installGroup(gb, false)
groupCount := len(source.groups)
demands[1].budget = 100_000
if pending {
if _, err := source.planGroups(demands); err != nil {
t.Fatal(err)
}
if source.memberships[b] != ga || b.CurrentSource() != gb.media.Source {
t.Fatal("fixture did not prepare a compatible rejoin")
}
}
source.groupRecovery.Store(0)
demands[0].budget = 60_000
if reverse {
demands[0], demands[1] = demands[1], demands[0]
}
plan, err := source.planGroups(demands)
if err != nil {
t.Fatal(err)
}
if source.memberships[a] != ga || source.memberships[b] != gb || len(source.groups) != groupCount ||
plan.Bitrates[ga.slot] != 60_000 || plan.Bitrates[gb.slot] != 100_000 || source.groupRecovery.Load() != 0 {
t.Fatalf("admission displaced an incumbent: a=%v b=%v groups=%d/%d rates=%v recovery=%d",
source.memberships[a] == ga, source.memberships[b] == gb, len(source.groups), groupCount,
plan.Bitrates, source.groupRecovery.Load())
}
})
}
}
}
func TestOutputGroupsRetainOriginalRecoveryAfterPendingJoin(t *testing.T) {
source, b, gb, demands := newGroupLifecycleFixture(t)
source.installGroup(source.memberships[demands[0].consumer], false)
source.installGroup(gb, false)
demands[1].budget = demands[0].budget
if _, err := source.planGroups(demands); err != nil {
t.Fatal(err)
}
source.groupRecovery.Store(0)
demands[0].budget = 60_000
demands[1].lower, demands[1].original = false, true
if _, err := source.planGroups(demands); err != nil {
t.Fatal(err)
}
if source.memberships[b] != nil || source.groupRecovery.Load()&2 == 0 || b.CurrentSource() != gb.media.Source {
t.Fatal("return to original lost recovery or replaced the current source early")
}
}
func TestOutputGroupRejectsRecoveryBeforeChangingAttachment(t *testing.T) {
for _, path := range []string{"derived", "original"} {
for _, stamp := range []string{"future", "duplicate", "before-generation-start"} {
internal/app/mediaedge/publication_test.go
+6 -2
View File
@@ -73,9 +73,11 @@ func TestPublicationOwnsOneReservationAndSourceLifetime(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if err = source.DisableLayer(0); err != nil {
retire, err := source.MarkLayerUnavailable(0)
if err != nil {
t.Fatal(err)
}
retire()
if _, err = engine.NewPublication(source, EdgeOptions{ConnectionID: "failed-output"}); err == nil {
t.Fatal("failed output was admitted into publication metadata")
}
@@ -117,9 +119,11 @@ func TestPublicationAdmissionIgnoresUnrelatedFailedOutputs(t *testing.T) {
t.Fatal(err)
}
for _, slot := range []int{2, 3, 0, 1} {
if err = source.DisableLayer(slot); err != nil {
retire, err := source.MarkLayerUnavailable(slot)
if err != nil {
t.Fatal(err)
}
retire()
candidate, createErr := engine.NewPublication(source, EdgeOptions{ConnectionID: "replacement"})
if slot >= 2 {
if createErr != nil {
internal/app/mediaedge/source.go
+9 -8
View File
@@ -1,6 +1,7 @@
package mediaedge
import (
"context"
"errors"
"io"
"sync"
@@ -211,12 +212,10 @@ func (source *Source) ConfigureOutputs(ceilings []uint32) error {
return nil
}
func (source *Source) DisableLayer(layer int) error {
retire, err := source.markLayerUnavailable(layer, nil)
if retire != nil {
retire()
}
return err
// MarkLayerUnavailable commits output failure without closing transports. The
// caller must run the returned retirement after releasing its capture-owner lock.
func (source *Source) MarkLayerUnavailable(layer int) (func(), error) {
return source.markLayerUnavailable(layer, nil)
}
func (source *Source) markLayerUnavailable(layer int, run *relayRun) (func(), error) {
@@ -227,8 +226,10 @@ func (source *Source) markLayerUnavailable(layer int, run *relayRun) (func(), er
source.writeMu.Lock()
defer source.writeMu.Unlock()
source.mu.Lock()
if run != nil && (run.owner.run != run || source.relayProfile != run.plan.profile ||
source.formats[len(source.formats)-1].Load() != run.plan.format) {
// Normal retirement cannot disable a later run's slots. A real cancellation
// cause still owns failure isolation until that run has finished retiring.
if run != nil && (run.owner.run != run || errors.Is(context.Cause(run.ctx), context.Canceled) ||
source.relayProfile != run.plan.profile || source.formats[len(source.formats)-1].Load() != run.plan.format) {
source.mu.Unlock()
return nil, nil
}
internal/app/nativeaudio/encoder_test.go
+52
View File
@@ -85,6 +85,58 @@ func TestEncodePreservesAudiblePCM(t *testing.T) {
}
}
func TestEncodePreservesStereoSeparation(t *testing.T) {
for _, bitrate := range []int{64_000, 128_000, 192_000} {
encoder, err := NewEncoder(bitrate)
if err != nil {
t.Fatal(err)
}
decoder, err := gopus.NewDecoder(gopus.DefaultDecoderConfig(SampleRate, Channels))
if err != nil {
t.Fatal(err)
}
pcm := make([]byte, FrameBytes)
decoded := make([]int16, FrameSamples*Channels)
frequencies := [2]float64{500, 1500}
var tones [2][2]complex128
for block := 0; block < 30; block++ {
for frame := 0; frame < FrameSamples; frame++ {
for channel, frequency := range frequencies {
phase := 2 * math.Pi * frequency * float64(block*FrameSamples+frame) / SampleRate
binary.LittleEndian.PutUint16(pcm[(frame*Channels+channel)*2:], uint16(int16(8_000*math.Sin(phase))))
}
}
packet, err := encoder.Encode(pcm)
if err != nil {
t.Fatal(err)
}
count, err := decoder.DecodeInt16(packet, decoded)
if err != nil || count != FrameSamples {
t.Fatalf("decode: %d samples, %v", count, err)
}
if block < 5 {
continue // Exclude codec startup; compare frequency energy, not phase delay.
}
for frame := 0; frame < count; frame++ {
for tone, frequency := range frequencies {
phase := 2 * math.Pi * frequency * float64(block*FrameSamples+frame) / SampleRate
for channel := range Channels {
tones[channel][tone] += complex(float64(decoded[frame*Channels+channel]), 0) * complex(math.Cos(phase), math.Sin(phase))
}
}
}
}
for channel := range Channels {
wanted, other := tones[channel][channel], tones[channel][1-channel]
level := math.Hypot(real(wanted), imag(wanted))
leak := math.Hypot(real(other), imag(other))
if level < 1 || level < 10*leak {
t.Fatalf("%d bps channel %d lost separation: signal %f, leakage %f", bitrate, channel, level, leak)
}
}
}
}
func TestNewEncoderRejectsOutOfRangeBitrate(t *testing.T) {
for _, bitrate := range []int{0, 5_999, 510_001} {
if _, err := NewEncoder(bitrate); err == nil {
internal/app/nativecapture/probe.go
+26 -23
View File
@@ -42,36 +42,39 @@ type Adapter struct {
}
type Capabilities struct {
Protocol int `json:"protocol"`
Platform string `json:"platform"`
PlatformBuild string `json:"platformBuild"`
VideoCapture bool `json:"videoCapture"`
CaptureBorderControl bool `json:"captureBorderControl,omitempty"`
Microphone bool `json:"microphone,omitempty"`
ProcessAudio bool `json:"processAudio"`
SystemAudio bool `json:"systemAudio"`
SoftwareVP8 bool `json:"softwareVP8"`
Adapters []Adapter `json:"adapters"`
Protocol int `json:"protocol"`
Platform string `json:"platform"`
PlatformBuild string `json:"platformBuild"`
VideoCapture bool `json:"videoCapture"`
CaptureBorderControl bool `json:"captureBorderControl,omitempty"`
Microphone bool `json:"microphone,omitempty"`
ProcessAudio bool `json:"processAudio"`
ProcessAudioExclusion bool `json:"processAudioExclusion,omitempty"`
SystemAudio bool `json:"systemAudio"`
SoftwareVP8 bool `json:"softwareVP8"`
Adapters []Adapter `json:"adapters"`
}
type Summary struct {
Video bool
CaptureBorderControl bool
Microphone bool
ProcessAudio bool
SystemAudio bool
HardwareH264 bool
SoftwareVP8 bool
Video bool
CaptureBorderControl bool
Microphone bool
ProcessAudio bool
ProcessAudioExclusion bool
SystemAudio bool
HardwareH264 bool
SoftwareVP8 bool
}
func (capabilities Capabilities) Summary() Summary {
summary := Summary{
Video: capabilities.VideoCapture,
CaptureBorderControl: capabilities.VideoCapture && capabilities.CaptureBorderControl,
Microphone: capabilities.Microphone,
ProcessAudio: capabilities.ProcessAudio,
SystemAudio: capabilities.SystemAudio,
SoftwareVP8: capabilities.SoftwareVP8,
Video: capabilities.VideoCapture,
CaptureBorderControl: capabilities.VideoCapture && capabilities.CaptureBorderControl,
Microphone: capabilities.Microphone,
ProcessAudio: capabilities.ProcessAudio,
ProcessAudioExclusion: capabilities.ProcessAudioExclusion && capabilities.Microphone,
SystemAudio: capabilities.SystemAudio,
SoftwareVP8: capabilities.SoftwareVP8,
}
for _, adapter := range capabilities.Adapters {
if len(adapter.HardwareH264) > 0 {
internal/app/nativecapture/probe_test.go
+16
View File
@@ -33,6 +33,22 @@ func TestDecodeProbeRejectsUnknownTrailingAndDuplicateData(t *testing.T) {
}
}
func TestAudioExclusionCapabilityRequiresExplicitSupportAndMixing(t *testing.T) {
capabilities, err := decodeProbe([]byte(validProbe))
if err != nil || capabilities.Summary().ProcessAudioExclusion {
t.Fatal("old capture probe advertised audio exclusion", err)
}
capabilities, err = decodeProbe([]byte(strings.Replace(validProbe, `"processAudio":true`,
`"processAudio":true,"processAudioExclusion":true,"microphone":true`, 1)))
if err != nil || !capabilities.Summary().ProcessAudioExclusion {
t.Fatal("explicit audio exclusion capability was lost", err)
}
capabilities.Microphone = false
if capabilities.Summary().ProcessAudioExclusion {
t.Fatal("audio exclusion requires the replaceable mixed source")
}
}
func TestSummaryDoesNotConflateVideoAudioAndEncoder(t *testing.T) {
capabilities, err := decodeProbe([]byte(
`{"protocol":7,"platform":"windows","platformBuild":"19045","videoCapture":true,"processAudio":false,"systemAudio":true,"adapters":[{"index":0,"name":"GPU","identity":"0x0:0x1","hardwareH264":[]}]}`,
internal/app/nativecapture/stream.go
+7 -3
View File
@@ -281,13 +281,17 @@ func appendOutputArguments(arguments []string, outputs []OutputProfile) ([]strin
return arguments, nil
}
func StartAudio(parent context.Context, executable string, target CaptureTarget) (*Stream, error) {
if !target.Valid() {
func StartAudio(parent context.Context, executable string, target CaptureTarget, exclude bool) (*Stream, error) {
if !target.Valid() || (exclude && target.Kind != "window") {
return nil, errors.New("native audio target is invalid")
}
kind := target.Kind
if exclude {
kind = "exclude"
}
return startAudioStream(parent, executable, []string{
"--capture-audio",
target.Kind,
kind,
strconv.FormatUint(uint64(target.PID), 10),
zeroWhenEmpty(target.CreationTime),
})
internal/app/nativecontrol/session.go
+24 -1
View File
@@ -182,6 +182,12 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any,
!validQualitySettings(request.Profile) {
return nil, protocolViolation("native start-share request is invalid")
}
if err := session.validateAudioExclusion(request.Source, request.Audio, request.ExcludeAudio); err != nil {
return nil, err
}
if request.ExcludeAudio != nil && !request.MicrophoneMixing {
return nil, errors.New("native audio exclusion requires mixed audio")
}
slog.Debug("piik-client", "event", "native-profile-requested", "requestId", envelope.ID,
"share", diagnostics.ID(request.ShareID), "profile", nativeQualityProfile(request.Profile), "codec", request.Codec, "audio", request.Audio,
"sourceKind", request.Source.Kind, "adapterIndex", request.AdapterIndex, "encoderIndex", request.EncoderIndex)
@@ -245,6 +251,9 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any,
!validIdentities(request.ShareID) || !request.Source.Valid() {
return nil, protocolViolation("native replace-share-source request is invalid")
}
if err := session.validateAudioExclusion(request.Source, request.Audio, request.ExcludeAudio); err != nil {
return nil, err
}
host := session.current(request.ShareID)
if host == nil {
return nil, errors.New("native share does not exist")
@@ -261,7 +270,7 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any,
ShowCaptureBorder: request.ShowCaptureBorder && session.capabilities.CaptureBorderControl,
AdapterIndex: request.AdapterIndex,
EncoderIndex: request.EncoderIndex,
}, audio)
}, audio, request.ExcludeAudio)
if err != nil {
return nil, err
}
@@ -534,6 +543,19 @@ func (session *Session) finishHostOperation(done chan struct{}, result any) {
session.mu.Unlock()
}
func (session *Session) validateAudioExclusion(source nativecapture.CaptureTarget, audio bool, excluded *nativecapture.CaptureTarget) error {
if excluded == nil {
return nil
}
if !audio || source.Kind != "display" || excluded.Kind != "window" || !excluded.Valid() {
return protocolViolation("native audio exclusion target is invalid")
}
if !session.capabilities.Summary().ProcessAudioExclusion || !session.capabilities.SystemAudio {
return errors.New("native audio exclusion is unavailable")
}
return nil
}
func (session *Session) shareUpdateResult(request requestEnvelope, shareID string, profile nativehost.QualityProfile, err error) any {
if err != nil {
return operationFailure(request, err)
@@ -571,6 +593,7 @@ func (session *Session) startShare(
Profile: profile.Video,
},
MicrophoneMixing: request.MicrophoneMixing && session.capabilities.Microphone,
ExcludeAudio: request.ExcludeAudio,
Profile: profile,
EdgeCapacity: request.EdgeCapacity,
AudioEnabled: request.Audio && session.capabilities.Summary().AudioFor(
internal/app/nativecontrol/session_test.go
+29
View File
@@ -641,3 +641,32 @@ func TestReplaceShareSourceKeepsOneStrictTargetShape(t *testing.T) {
t.Fatalf("malformed target was treated as an operational failure: %v", err)
}
}
func TestAudioExclusionValidatesScopeWithoutDroppingRequestedPolicy(t *testing.T) {
session := &Session{capabilities: nativecapture.Capabilities{Microphone: true, SystemAudio: true, ProcessAudioExclusion: true}}
display := nativecapture.CaptureTarget{Kind: "display", SourceID: "1", Title: "Screen"}
window := nativecapture.CaptureTarget{Kind: "window", SourceID: "2", PID: 123, CreationTime: "456", Title: "Voice fixture"}
if err := session.validateAudioExclusion(display, true, &window); err != nil {
t.Fatal(err)
}
for _, invalid := range []struct {
source nativecapture.CaptureTarget
audio bool
excluded nativecapture.CaptureTarget
}{
{display, false, window}, {window, true, window}, {display, true, display},
{display, true, nativecapture.CaptureTarget{Kind: "window", SourceID: "2", PID: 123, Title: "Missing lifetime"}},
} {
if _, ok := session.validateAudioExclusion(invalid.source, invalid.audio, &invalid.excluded).(protocolViolation); !ok {
t.Fatal("malformed audio exclusion was not rejected")
}
}
session.capabilities.ProcessAudioExclusion = false
err := session.validateAudioExclusion(display, true, &window)
if _, malformed := err.(protocolViolation); err == nil || malformed {
t.Fatal("unsupported audio exclusion must fail the operation, not silently widen capture or close control")
}
if err := session.validateAudioExclusion(display, true, nil); err != nil {
t.Fatal("legacy capture became dependent on exclusion support")
}
}
internal/app/nativecontrol/wire.go
+24 -22
View File
@@ -42,19 +42,20 @@ type qualitySettings struct {
}
type startShareRequest struct {
Version int `json:"version"`
ID string `json:"id"`
Type string `json:"type"`
ShareID string `json:"shareId"`
Source nativecapture.CaptureTarget `json:"source"`
Audio bool `json:"audio"`
MicrophoneMixing bool `json:"microphoneMixing,omitempty"`
ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"`
AdapterIndex uint32 `json:"adapterIndex"`
EncoderIndex uint32 `json:"encoderIndex"`
EdgeCapacity int `json:"edgeCapacity"`
Profile qualitySettings `json:"profile"`
Codec string `json:"codec"`
Version int `json:"version"`
ID string `json:"id"`
Type string `json:"type"`
ShareID string `json:"shareId"`
Source nativecapture.CaptureTarget `json:"source"`
Audio bool `json:"audio"`
MicrophoneMixing bool `json:"microphoneMixing,omitempty"`
ExcludeAudio *nativecapture.CaptureTarget `json:"excludeAudio,omitempty"`
ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"`
AdapterIndex uint32 `json:"adapterIndex"`
EncoderIndex uint32 `json:"encoderIndex"`
EdgeCapacity int `json:"edgeCapacity"`
Profile qualitySettings `json:"profile"`
Codec string `json:"codec"`
}
type microphoneRequest struct {
@@ -81,15 +82,16 @@ type updateShareRequest struct {
}
type replaceShareSourceRequest struct {
Version int `json:"version"`
ID string `json:"id"`
Type string `json:"type"`
ShareID string `json:"shareId"`
Source nativecapture.CaptureTarget `json:"source"`
Audio bool `json:"audio"`
ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"`
AdapterIndex uint32 `json:"adapterIndex"`
EncoderIndex uint32 `json:"encoderIndex"`
Version int `json:"version"`
ID string `json:"id"`
Type string `json:"type"`
ShareID string `json:"shareId"`
Source nativecapture.CaptureTarget `json:"source"`
Audio bool `json:"audio"`
ExcludeAudio *nativecapture.CaptureTarget `json:"excludeAudio,omitempty"`
ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"`
AdapterIndex uint32 `json:"adapterIndex"`
EncoderIndex uint32 `json:"encoderIndex"`
}
type stopShareRequest struct {
internal/app/nativehost/audio_mix_test.go
+78 -1
View File
@@ -88,6 +88,83 @@ func startMixedFixture(t *testing.T) (*Session, nativecapture.VideoOptions, <-ch
return session, options, events
}
func TestAudioExclusionFailureMutesOnlySourceAndNeverFallsBack(t *testing.T) {
session, options, events := startMixedFixture(t)
enabled := true
if err := session.SetMicrophone(&enabled, nil, nil); err != nil {
t.Fatal(err)
}
output := session.audioSource
session.mixer.mu.Lock()
microphone := session.mixer.microphone
session.mixer.mu.Unlock()
excluded := &nativecapture.CaptureTarget{Kind: "window", SourceID: "123", PID: 456, CreationTime: "789", Title: "Voice fixture"}
for _, fault := range []string{"PIIK_VIDEO_FAIL", "PIIK_EXCLUSION_FAIL"} {
if err := session.ReplaceSource(options, true, nil); err != nil {
t.Fatal(err)
}
video := session.stream
t.Setenv(fault, "1")
if err := session.ReplaceSource(options, true, excluded); err == nil {
t.Fatal("failed exclusion was accepted", fault)
}
if session.SourceAudio() == nil || *session.SourceAudio() || session.audioStream != nil || session.stream != video {
t.Fatal("failed exclusion retained unfiltered audio or retired video", fault)
}
t.Setenv(fault, "")
}
marker := filepath.Join(t.TempDir(), "excluded")
t.Setenv("PIIK_EXCLUSION_STARTED", marker)
if err := session.ReplaceSource(options, true, excluded); err != nil {
t.Fatal(err)
}
identity, err := os.ReadFile(marker)
if err != nil || string(identity) != "456:789" || !*session.SourceAudio() {
t.Fatal("replacement did not bind exclusion to the selected process lifetime", err)
}
sourceAudio := session.audioStream
profile := session.profile
profile.AudioBitrate = 128000
if err := session.UpdateProfile(profile); err != nil {
t.Fatal(err)
}
session.SetPaused(true)
session.SetPaused(false)
if session.audioStream != sourceAudio {
t.Fatal("profile/pause replaced exclusion")
}
for len(events) > 0 {
<-events
}
if err := sourceAudio.Close(); err != nil {
t.Fatal(err)
}
timer := time.NewTimer(time.Second)
defer timer.Stop()
waiting := true
for waiting {
select {
case event := <-events:
if event.Type != "audio-state" {
continue
}
if event.SourceAudio || !event.Microphone || event.Failed {
t.Fatalf("source loss was misreported: %+v", event)
}
waiting = false
case <-timer.C:
t.Fatal("source loss was not reported")
}
}
session.mixer.mu.Lock()
sameMicrophone := session.mixer.microphone == microphone
sourceEnded := session.mixer.source == nil
session.mixer.mu.Unlock()
if !sourceEnded || !sameMicrophone || session.audioSource != output || !session.HasAudio() {
t.Fatal("exclusion loss disturbed microphone or output ownership")
}
}
func TestNativeMicrophonePreservesMediaOwnersAcrossInputChanges(t *testing.T) {
session, options, events := startMixedFixture(t)
check := func(err error) {
@@ -130,7 +207,7 @@ func TestNativeMicrophonePreservesMediaOwnersAcrossInputChanges(t *testing.T) {
}
microphone = replaced
options.Target.SourceID = "2"
check(session.ReplaceSource(options, true))
check(session.ReplaceSource(options, true, nil))
session.mixer.mu.Lock()
sameMicrophone := session.mixer.microphone == microphone
session.mixer.mu.Unlock()
internal/app/nativehost/audio_test.go
+12 -1
View File
@@ -56,6 +56,17 @@ func runAudioRecoveryCapture() {
}
}
audio := microphone || (len(os.Args) > 1 && os.Args[1] == "--capture-audio")
if !audio && os.Getenv("PIIK_VIDEO_FAIL") == "1" {
os.Exit(2)
}
if !microphone && audio && os.Args[2] == "exclude" {
if os.Getenv("PIIK_EXCLUSION_FAIL") == "1" {
os.Exit(2)
}
if marker := os.Getenv("PIIK_EXCLUSION_STARTED"); marker != "" {
_ = os.WriteFile(marker, []byte(os.Args[3]+":"+os.Args[4]), 0600)
}
}
period := time.Second / 30
status := []byte(`{"state":"active","audio":true}`)
if audio {
@@ -179,7 +190,7 @@ func TestAudioEOFWaitsForExplicitSourceReplacement(t *testing.T) {
waitPackets(&videoPackets, videoPackets.Load()+10, "video after audio EOF")
before := audioPackets.Load()
options.Target.SourceID = "2"
check(session.ReplaceSource(options, true))
check(session.ReplaceSource(options, true, nil))
if session.currentStream() == previousVideo || session.currentAudioStream() == previousAudio ||
session.edge("audio-edge") != edge || session.audioSource != audioSource {
t.Fatal("explicit replacement did not preserve the media owners")
internal/app/nativehost/session.go
+42 -3
View File
@@ -67,6 +67,7 @@ type Options struct {
Video nativecapture.VideoOptions
Profile QualityProfile
AudioEnabled bool
ExcludeAudio *nativecapture.CaptureTarget
MicrophoneMixing bool
EdgeCapacity int
BindAddress string
@@ -138,6 +139,7 @@ func Start(parent context.Context, options Options) (*Session, error) {
parent,
options.CaptureProcess,
options.Video.Target,
options.ExcludeAudio,
)
if audioErr != nil {
slog.DebugContext(parent, "piik-client", "event", "capture-audio-unavailable", "share", diagnostics.ID(options.ShareID), diagnostics.Error(audioErr))
@@ -401,6 +403,7 @@ func (session *Session) UpdateProfile(profile QualityProfile) error {
func (session *Session) ReplaceSource(
options nativecapture.VideoOptions,
audioEnabled bool,
excludeAudio *nativecapture.CaptureTarget,
) error {
session.updateMu.Lock()
defer session.updateMu.Unlock()
@@ -420,6 +423,21 @@ func (session *Session) ReplaceSource(
if session.mixer == nil && audioEnabled != hasAudio {
return errors.New("native source audio availability cannot change while sharing")
}
if excludeAudio != nil {
if session.mixer == nil {
return errors.New("native audio exclusion requires mixed audio")
}
// A privacy change retires the old source before preparation. Failure must
// leave it silent rather than retaining unfiltered audio; video/mic stay live.
session.mu.Lock()
previousAudio := session.audioStream
session.audioStream = nil
session.mu.Unlock()
session.mixer.setSource(nil)
if previousAudio != nil {
_ = previousAudio.Close()
}
}
options.Profile = profile.Video
options.RestoreToken = ""
replacement, state, err := session.prepareVideo(session.ctx, options, options.Target.Kind == "picker", false)
@@ -432,6 +450,7 @@ func (session *Session) ReplaceSource(
session.ctx,
session.captureProcess,
options.Target,
excludeAudio,
)
if err != nil {
_ = replacement.Close()
@@ -452,11 +471,15 @@ func startAudioCapture(
ctx context.Context,
captureProcess string,
target nativecapture.CaptureTarget,
excludeAudio *nativecapture.CaptureTarget,
) (*nativecapture.Stream, error) {
if excludeAudio != nil {
return nativecapture.StartAudio(ctx, captureProcess, *excludeAudio, true)
}
if target.Kind == "display" || target.Kind == "picker" {
return nativecapture.StartSystemAudio(ctx, captureProcess)
}
return nativecapture.StartAudio(ctx, captureProcess, target)
return nativecapture.StartAudio(ctx, captureProcess, target, false)
}
func (session *Session) prepareVideo(
@@ -560,7 +583,7 @@ func (session *Session) installCapture(next *nativecapture.Stream) error {
// Preparation has consumed these events. Apply them to the new generation
// before acknowledging it, just as the live reader handles unavailable layers.
for _, layer := range unavailable {
if err := session.source.DisableLayer(layer); err != nil {
if err := session.disableCaptureLayer(next, layer); err != nil {
return err
}
}
@@ -575,6 +598,22 @@ func (session *Session) installCapture(next *nativecapture.Stream) error {
return nil
}
func (session *Session) disableCaptureLayer(stream *nativecapture.Stream, layer int) error {
session.mu.Lock()
if session.stream != stream || session.closed || session.ctx.Err() != nil {
session.mu.Unlock()
return nil
}
// Accept failure atomically with capture selection. Closing transports may
// wait for callbacks, so only the source-state change belongs under this lock.
retire, err := session.source.MarkLayerUnavailable(layer)
session.mu.Unlock()
if retire != nil {
retire()
}
return err
}
func (session *Session) CloseEdge(connectionID string) {
session.mu.Lock()
edge := session.edges[connectionID]
@@ -831,7 +870,7 @@ func (session *Session) runVideo() error {
if session.source == nil {
return fail(errors.New("native output ended before its source state"))
}
if err = session.source.DisableLayer(frame.Layer); err != nil {
if err = session.disableCaptureLayer(current, frame.Layer); err != nil {
return fail(err)
}
case nativecapture.FramePCM:
internal/app/nativehost/session_test.go
+117
View File
@@ -11,6 +11,7 @@ import (
"slices"
"strconv"
"strings"
"sync"
"testing"
"time"
@@ -189,6 +190,122 @@ func TestCaptureCommitWaitsForReaderMetadataOrTermination(t *testing.T) {
}
}
func TestCaptureFailureBelongsToSelectedStream(t *testing.T) {
for _, replace := range []bool{false, true} {
t.Run(strconv.FormatBool(replace), func(t *testing.T) {
check := func(err error) {
t.Helper()
if err != nil {
t.Fatal(err)
}
}
t.Setenv("PIIK_NATIVEHOST_PIPE_FIXTURE", "backend")
t.Setenv("PIIK_CAPTURE_STARTING", string(captureStatePayload(t,
`{"state":"starting","codec":"vp8","adapterIndex":0,"adapterName":"Fixture","adapterIdentity":"0:0","encoderName":"VP8","encoderIdentity":"vp8"}`)))
t.Setenv("PIIK_CAPTURE_ACTIVE", string(captureStatePayload(t, `{"state":"active","codec":"vp8","width":1280,"height":720,"fps":30}`)))
t.Setenv("PIIK_CAPTURE_UNAVAILABLE", "0")
executable, err := os.Executable()
check(err)
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
options := nativecapture.VideoOptions{Target: nativecapture.CaptureTarget{Kind: "display", SourceID: "1", Title: "Fixture"}, Codec: "vp8",
Profile: nativecapture.VideoProfile{Width: 1280, Height: 720, Framerate: 30, Bitrate: 3_000_000, Preference: "balanced"}, OutputGroups: 4}
previous, err := nativecapture.StartVideo(ctx, executable, options)
check(err)
defer previous.Close()
t.Setenv("PIIK_CAPTURE_UNAVAILABLE", "")
replacement, err := nativecapture.StartVideo(ctx, executable, options)
check(err)
defer replacement.Close()
engine, err := mediaedge.NewEngine(mediaedge.EngineOptions{BindAddress: "127.0.0.1:0", IncludeLoopback: true})
check(err)
defer engine.Close()
source, err := engine.NewSource("vp8", 4, 2, nil)
check(err)
defer source.Close()
check(configureCaptureOutputs(source, previous.Outputs()))
publication, err := engine.NewPublication(source, mediaedge.EdgeOptions{ConnectionID: "capture-failure-owner"})
check(err)
active := make(chan struct{}, 1)
session := &Session{ctx: ctx, cancel: cancel, engine: engine, source: source, stream: previous,
videoOptions: options, ready: make(chan error, 1), events: func(_ context.Context, event Event) {
if event.State == "active" {
select {
case active <- struct{}{}:
default:
}
}
}}
// Delay the actual reader after its first identity check, without a
// production hook. Log I/O and ordinary scheduling can delay it here.
entered, release := make(chan struct{}), make(chan struct{})
unblock := sync.OnceFunc(func() { close(release) })
previousLog := slog.Default()
defer slog.SetDefault(previousLog)
slog.SetDefault(slog.New(slog.NewTextHandler(io.Discard, &slog.HandlerOptions{
Level: slog.LevelDebug, ReplaceAttr: func(_ []string, attr slog.Attr) slog.Attr {
if attr.Key == "event" && attr.Value.String() == "capture-output-unavailable" {
close(entered)
<-release
}
return attr
},
})))
readerDone := make(chan error, 1)
go func() { readerDone <- session.runVideo() }()
defer func() {
unblock()
cancel()
_ = previous.Close()
_ = replacement.Close()
select {
case <-readerDone:
case <-time.After(5 * time.Second):
t.Error("capture reader did not retire")
}
}()
select {
case <-entered:
case err = <-readerDone:
readerDone <- err
t.Fatalf("reader stopped before output failure: %v", err)
case <-time.After(5 * time.Second):
t.Fatal("old output failure did not reach the reader")
}
committed := make(chan error, 1)
if replace {
go func() {
committed <- session.commitCapture(options, QualityProfile{Video: options.Profile}, replacement, CaptureState{}, nil, false)
}()
deadline := time.Now().Add(5 * time.Second)
for session.currentStream() != replacement {
if time.Now().After(deadline) {
t.Fatal("replacement was not selected")
}
time.Sleep(time.Millisecond)
}
}
unblock()
select {
case <-active:
case <-time.After(5 * time.Second):
t.Fatal("capture reader did not finish the failure or handoff")
}
if replace {
select {
case err = <-committed:
check(err)
case <-time.After(5 * time.Second):
t.Fatal("replacement installation was not acknowledged")
}
}
if err = publication.SetActiveCount(1); (err == nil) != replace {
t.Fatalf("publication availability after replacement=%v: %v", replace, err)
}
})
}
}
func TestCaptureProfileRetainsSelectedBackendFromStarting(t *testing.T) {
t.Setenv("PIIK_NATIVEHOST_PIPE_FIXTURE", "backend")
t.Setenv("PIIK_CAPTURE_STARTING", string(captureStatePayload(t,
internal/server/app/app.go
+1
View File
@@ -154,6 +154,7 @@ func New(options Options) (*Server, error) {
server.signalOptions = signal.Options{
Store: store,
EndpointMediaCopyCapacity: configuration.EndpointMediaCopyCapacity,
SfuOnly: configuration.SFU != nil && configuration.SFU.Only,
Ice: config.IceConfig(configuration),
// The TS signaling server read natPredictionEnabled off the same
// `ice` options object it forwarded; the Go IceConfig is already the
internal/server/app/app_test.go
+14 -3
View File
@@ -366,7 +366,17 @@ func TestCapabilitiesReportServicesDisabledByDefault(t *testing.T) {
server := start(t, Options{Config: testConfig(t)})
server.do(http.MethodGet, "/api/capabilities").
expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":false}`)
expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":false,"roomInteractions":true,"hostRoomSession":true}`)
}
func TestCapabilitiesReportSFUOnlyPolicy(t *testing.T) {
configuration := testConfig(t)
configuration.SFU = &config.SFUConfig{ListenHost: "127.0.0.1", Port: 0, Only: true}
configuration.NATPredictionEnabled = true
server := start(t, Options{Config: configuration})
server.do(http.MethodGet, "/api/capabilities").
expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":true,"sfuOnly":true,"natPrediction":false,"roomInteractions":true,"hostRoomSession":true}`)
}
func TestCapabilitiesReportOptionalNATPredictionWithoutExposingConfiguration(t *testing.T) {
@@ -377,7 +387,7 @@ func TestCapabilitiesReportOptionalNATPredictionWithoutExposingConfiguration(t *
server := start(t, Options{Config: configuration})
server.do(http.MethodGet, "/api/capabilities").
expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":true}`).
expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":false,"natPrediction":true,"roomInteractions":true,"hostRoomSession":true}`).
expectHeader("Cache-Control", "no-store").
expectHeader("X-Content-Type-Options", "nosniff")
@@ -420,7 +430,8 @@ func TestSFUListenerFollowsApplicationCloseAndEnd(t *testing.T) {
t.Fatal("missing runtime owner")
}
server.do(http.MethodGet, "/api/capabilities").
expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":true,"natPrediction":false}`)
expect(http.StatusOK, `{"connectionAttemptProgress4":true,"sfu":true,"natPrediction":false,"roomInteractions":true,"hostRoomSession":true}`)
var err error
if ending {
err = server.End(context.Background())
internal/server/app/routes.go
+4 -1
View File
@@ -80,7 +80,10 @@ func (s *Server) route(
sendJSON(writer, http.StatusOK, protocol.RuntimeCapabilities{
ConnectionAttemptProgress4: true,
Sfu: s.config.SFU != nil,
NatPrediction: s.config.NATPredictionEnabled,
SfuOnly: s.config.SFU != nil && s.config.SFU.Only,
NatPrediction: s.config.NATPredictionEnabled && !(s.config.SFU != nil && s.config.SFU.Only),
RoomInteractions: true,
HostRoomSession: true,
})
return
}
internal/server/config/config.go
+9
View File
@@ -68,6 +68,7 @@ type SFUConfig struct {
ListenHost string
Port int
PublicIP string
Only bool
}
// Config describes runtime configuration. Zero values mean open site
@@ -273,8 +274,15 @@ func parseRoomDatabasePath(value string) (string, error) {
}
func parseSFU(env map[string]string) (*SFUConfig, error) {
only, err := parseBoolean(env["SFU_ONLY"], "SFU_ONLY")
if err != nil {
return nil, err
}
rawPort := strings.TrimSpace(env["SFU_UDP_PORT"])
if rawPort == "" {
if only {
return nil, errors.New("SFU_ONLY requires SFU_UDP_PORT")
}
return nil, nil
}
port, err := parseBoundedInteger(rawPort, 0, "SFU_UDP_PORT", 1, maxPort)
@@ -298,6 +306,7 @@ func parseSFU(env map[string]string) (*SFUConfig, error) {
ListenHost: listenHost,
Port: int(port),
PublicIP: publicIP,
Only: only,
}, nil
}
internal/server/config/config_test.go
+15
View File
@@ -187,6 +187,21 @@ func TestLoadSFUOptional(t *testing.T) {
}
}
func TestSFUOnlyRequiresAnEnabledListener(t *testing.T) {
configuration := mustLoad(t, map[string]string{"SFU_UDP_PORT": "7882", "SFU_ONLY": "true"})
if !configuration.SFU.Only {
t.Fatal("SFU_ONLY was ignored")
}
for _, values := range []map[string]string{
{"SFU_ONLY": "true"},
{"SFU_ONLY": "yes", "SFU_UDP_PORT": "7882"},
} {
if _, err := Load(values); err == nil {
t.Fatalf("accepted invalid SFU policy: %v", values)
}
}
}
func TestLoadAccepts(t *testing.T) {
developmentPath, err := filepath.Abs(filepath.Join("state", "rooms.sqlite"))
if err != nil {
internal/server/protocol/client.go
+47 -1
View File
@@ -26,6 +26,8 @@ type AuthenticateHostMessage struct {
ViewerPresence bool `json:"viewerPresence,omitempty"`
ConnectionAttemptProgress4 bool `json:"connectionAttemptProgress4,omitempty"`
DisplayName *DisplayName `json:"displayName,omitempty"`
RoomOnly bool `json:"roomOnly,omitempty"`
RoomSession bool `json:"roomSession,omitempty"`
}
// AuthenticateViewerMessage is the viewer member of authenticateMessageSchema.
@@ -172,6 +174,15 @@ type StopSharingMessage struct {
ShareGeneration string `json:"shareGeneration,omitempty"`
}
// StartSharingMessage begins a new media publication on an existing Host room session.
type StartSharingMessage struct {
Type string `json:"type"`
ShareGeneration string `json:"shareGeneration"`
SharingPaused bool `json:"sharingPaused,omitempty"`
QualitySettings *QualitySettings `json:"qualitySettings,omitempty"`
RoutePolicy RoutePolicy `json:"routePolicy"`
}
// AbandonRoomMessage is { type: "abandon-room" }.
type AbandonRoomMessage struct {
Type string `json:"type"`
@@ -197,6 +208,7 @@ func (ResetSenderQualityMessage) isClientMessage() {}
func (SetDisplayNameMessage) isClientMessage() {}
func (SetSharingPausedMessage) isClientMessage() {}
func (StopSharingMessage) isClientMessage() {}
func (StartSharingMessage) isClientMessage() {}
func (AbandonRoomMessage) isClientMessage() {}
// DecodeClientMessage parses JSON and validates the message schema.
@@ -207,6 +219,10 @@ func DecodeClientMessage(data []byte) (ClientMessage, error) {
return nil, err
}
switch messageType {
case "subscribe-room-interactions":
return decodeEmptyClientMessage(data, SubscribeRoomInteractionsMessage{Type: messageType})
case "send-room-interaction":
return decodeSendRoomInteraction(data)
case "authenticate":
return decodeAuthenticate(data)
case "signaling-challenge":
@@ -247,6 +263,8 @@ func DecodeClientMessage(data []byte) (ClientMessage, error) {
return decodeSetSharingPaused(data)
case "stop-sharing":
return decodeStopSharing(data)
case "start-sharing":
return decodeStartSharing(data)
case "abandon-room":
return decodeEmptyClientMessage(data, AbandonRoomMessage{Type: messageType})
}
@@ -282,8 +300,9 @@ func decodeAuthenticate(data []byte) (ClientMessage, error) {
if err := present.require("type", "protocol", "roomId", "role", "token", "clientId"); err != nil {
return nil, err
}
if err := present.optional("shareGeneration", "sharingPaused", "qualitySettings",
if err := present.optional("shareGeneration", "roomOnly", "roomSession", "sharingPaused", "qualitySettings",
"routePolicy", "viewerPresence", "displayName", "connectionAttemptProgress4"); err != nil {
return nil, err
}
if message.Protocol != SignalingProtocol {
@@ -301,6 +320,15 @@ func decodeAuthenticate(data []byte) (ClientMessage, error) {
if present.has("shareGeneration") && !ValidOpaqueID(message.ShareGeneration) {
return nil, errors.New("shareGeneration is not an opaque id")
}
if present.has("roomOnly") && !message.RoomOnly {
return nil, errors.New("roomOnly must be true when present")
}
if present.has("roomSession") && !message.RoomSession {
return nil, errors.New("roomSession must be true when present")
}
if message.RoomOnly && !message.RoomSession {
return nil, errors.New("roomOnly requires roomSession")
}
if present.has("viewerPresence") && !message.ViewerPresence {
return nil, errors.New("viewerPresence must be true when present")
}
@@ -705,3 +733,21 @@ func decodeStopSharing(data []byte) (ClientMessage, error) {
}
return message, nil
}
func decodeStartSharing(data []byte) (ClientMessage, error) {
var message StartSharingMessage
present, err := decodeObject(data, &message)
if err != nil {
return nil, err
}
if err := present.require("type", "shareGeneration", "routePolicy"); err != nil {
return nil, err
}
if err := present.optional("sharingPaused", "qualitySettings"); err != nil {
return nil, err
}
if !ValidOpaqueID(message.ShareGeneration) {
return nil, errors.New("shareGeneration is not an opaque id")
}
return message, nil
}
internal/server/protocol/contract_test.go
+40 -1
View File
@@ -16,6 +16,7 @@ func TestRuntimeCapabilities(t *testing.T) {
{`{}`, RuntimeCapabilities{}},
{`{"connectionAttemptProgress4":true}`, RuntimeCapabilities{ConnectionAttemptProgress4: true}},
{`{"sfu":true,"extra":{"enabled":true}}`, RuntimeCapabilities{Sfu: true}},
{`{"sfu":true,"sfuOnly":true}`, RuntimeCapabilities{Sfu: true, SfuOnly: true}},
{`{"natPrediction":true,"SFU":true}`, RuntimeCapabilities{NatPrediction: true}},
{`{"sfu":true,"natPrediction":true}`, RuntimeCapabilities{Sfu: true, NatPrediction: true}},
{`{"sfu":"ignored duplicate","sfu":true}`, RuntimeCapabilities{Sfu: true}},
@@ -26,7 +27,7 @@ func TestRuntimeCapabilities(t *testing.T) {
}
}
for _, data := range []string{
`null`, `[]`, `true`, `{"sfu":null}`, `{"sfu":"true"}`,
`null`, `[]`, `true`, `{"sfu":null}`, `{"sfu":"true"}`, `{"sfuOnly":null}`, `{"sfuOnly":"true"}`,
`{"natPrediction":null}`, `{"natPrediction":1}`,
`{"connectionAttemptProgress4":null}`, `{"connectionAttemptProgress4":4}`,
} {
@@ -698,6 +699,44 @@ func TestAuthenticateAppliesTheRoutePolicyDefault(t *testing.T) {
}
}
func TestAuthenticateAcceptsPersistentRoomSessionOptIn(t *testing.T) {
message, err := DecodeClientMessage([]byte(
`{"type":"authenticate","protocol":"piik-v23","roomId":"1234","role":"host",` +
`"token":"` + repeat("a", 43) + `","clientId":"client_12345678","roomSession":true,"roomOnly":true}`))
if err != nil {
t.Fatal(err)
}
host, ok := message.(AuthenticateHostMessage)
if !ok || !host.RoomSession || !host.RoomOnly {
t.Fatalf("decoded %#v, want room-session opt-in and room-only state", message)
}
if _, err := DecodeClientMessage([]byte(
`{"type":"authenticate","protocol":"piik-v23","roomId":"1234","role":"host",` +
`"token":"` + repeat("a", 43) + `","clientId":"client_12345678","roomOnly":true}`)); err == nil {
t.Fatal("accepted roomOnly without roomSession opt-in")
}
}
func TestStartSharingRequiresAnOpaqueGenerationAndRoutePolicy(t *testing.T) {
message, err := DecodeClientMessage([]byte(
`{"type":"start-sharing","shareGeneration":"share_12345678",` +
`"routePolicy":{"peerOnly":true,"topologyOptimization":false,"natPrediction":false}}`))
if err != nil {
t.Fatal(err)
}
if _, ok := message.(StartSharingMessage); !ok {
t.Fatalf("decoded %#v, want StartSharingMessage", message)
}
for _, raw := range []string{
`{"type":"start-sharing","shareGeneration":"short","routePolicy":{"peerOnly":true,"topologyOptimization":false,"natPrediction":false}}`,
`{"type":"start-sharing","shareGeneration":"share_12345678"}`,
} {
if _, err := DecodeClientMessage([]byte(raw)); err == nil {
t.Errorf("accepted invalid start-sharing message %s", raw)
}
}
}
// The sealed unions accept nothing outside their own message structs.
func TestDecodeRejectsUnknownEnvelopes(t *testing.T) {
for _, raw := range []string{`{}`, `[]`, `null`, `"authenticate"`, `{"type":42}`,
internal/server/protocol/fixture_test.go
+2
View File
@@ -55,6 +55,8 @@ func TestFixtureConstants(t *testing.T) {
"viewerQualityEvidenceExpiryMs": float64(ViewerQualityEvidenceExpiryMs),
"persistentNativeEdgeDegradedWindows": float64(PersistentNativeEdgeDegradedWindows),
"maxDisplayNameCodePoints": float64(MaxDisplayNameCodePoints),
"maxChatCodePoints": float64(MaxChatCodePoints),
"interactionIntervalMs": float64(InteractionIntervalMs),
"defaultViewerDisplayName": DefaultViewerDisplayName,
"defaultHostDisplayNamePrefix": DefaultHostDisplayNamePrefix,
"minViewerPasswordLength": float64(MinViewerPasswordLength),
internal/server/protocol/interactions.go
+194
View File
@@ -0,0 +1,194 @@
package protocol
import (
"errors"
"strings"
"unicode/utf8"
"golang.org/x/text/unicode/norm"
)
const MaxChatCodePoints = 280
const InteractionIntervalMs = 800
type InteractionPayload struct {
Kind string `json:"kind"`
Text string `json:"text,omitempty"`
Reaction string `json:"reaction,omitempty"`
TargetPeerID string `json:"targetPeerId,omitempty"`
}
func (payload *InteractionPayload) UnmarshalJSON(data []byte) error {
type plain InteractionPayload
var value plain
fields, err := decodeObject(data, &value)
if err != nil {
return err
}
if err = fields.require("kind"); err != nil {
return err
}
switch value.Kind {
case "chat":
if err = fields.require("text"); err != nil {
return err
}
if fields.has("reaction") || fields.has("targetPeerId") ||
hasUnpairedSurrogateEscape(fields["text"]) || !validChatText(value.Text) {
return errors.New("invalid chat payload")
}
case "reaction":
if err = fields.require("reaction"); err != nil {
return err
}
if err = fields.optional("targetPeerId"); err != nil {
return err
}
if fields.has("text") || (fields.has("targetPeerId") && !ValidOpaqueID(value.TargetPeerID)) {
return errors.New("invalid reaction payload")
}
switch value.Reaction {
case "tomato", "poop":
if !ValidOpaqueID(value.TargetPeerID) {
return errors.New("throwing a prop requires a participant")
}
case "wave", "heart", "clap", "laugh", "wow", "party", "fire", "eyes", "star", "sleep":
default:
return errors.New("unknown reaction")
}
default:
return errors.New("unknown interaction kind")
}
*payload = InteractionPayload(value)
return nil
}
func validChatText(value string) bool {
return utf8.ValidString(value) && utf8.RuneCountInString(value) >= 1 &&
utf8.RuneCountInString(value) <= MaxChatCodePoints && !forbiddenDisplayNameCharacters.MatchString(value) &&
norm.NFC.IsNormalString(value) && strings.TrimFunc(value, IsJSWhitespace) == value
}
type SubscribeRoomInteractionsMessage struct {
Type string `json:"type"`
}
type SendRoomInteractionMessage struct {
Type string `json:"type"`
RequestID string `json:"requestId"`
Payload InteractionPayload `json:"payload"`
}
func (SubscribeRoomInteractionsMessage) isClientMessage() {}
func (SendRoomInteractionMessage) isClientMessage() {}
func decodeSendRoomInteraction(data []byte) (ClientMessage, error) {
var message SendRoomInteractionMessage
present, err := decodeObject(data, &message)
if err != nil {
return nil, err
}
if err = present.require("type", "requestId", "payload"); err != nil {
return nil, err
}
if !ValidOpaqueID(message.RequestID) {
return nil, errors.New("invalid interaction requestId")
}
return message, nil
}
type InteractionSender struct {
PeerID string `json:"peerId"`
Role Role `json:"role"`
DisplayName DisplayName `json:"displayName"`
}
func (sender *InteractionSender) UnmarshalJSON(data []byte) error {
type plain InteractionSender
var value plain
present, err := decodeObject(data, &value)
if err != nil {
return err
}
if err = present.require("peerId", "role", "displayName"); err != nil {
return err
}
if !ValidOpaqueID(value.PeerID) {
return errors.New("invalid sender peerId")
}
*sender = InteractionSender(value)
return nil
}
type RoomInteractionsReadyMessage struct {
Type string `json:"type"`
ServerTime Int `json:"serverTime"`
}
type RoomInteractionMessage struct {
Type string `json:"type"`
ID string `json:"id"`
RequestID string `json:"requestId"`
OccurredAt Int `json:"occurredAt"`
Sender InteractionSender `json:"sender"`
Payload InteractionPayload `json:"payload"`
}
type RoomInteractionRejectedMessage struct {
Type string `json:"type"`
RequestID string `json:"requestId"`
Reason string `json:"reason"`
}
func (RoomInteractionsReadyMessage) isServerMessage() {}
func (RoomInteractionMessage) isServerMessage() {}
func (RoomInteractionRejectedMessage) isServerMessage() {}
func decodeRoomInteractionsReady(data []byte) (ServerMessage, error) {
var message RoomInteractionsReadyMessage
present, err := decodeObject(data, &message)
if err != nil {
return nil, err
}
if err = present.require("type", "serverTime"); err != nil {
return nil, err
}
if !inRangeInt(message.ServerTime, 0, MaxSafeInteger) {
return nil, errors.New("invalid interaction server time")
}
return message, nil
}
func decodeRoomInteraction(data []byte) (ServerMessage, error) {
var message RoomInteractionMessage
present, err := decodeObject(data, &message)
if err != nil {
return nil, err
}
if err = present.require("type", "id", "requestId", "occurredAt", "sender", "payload"); err != nil {
return nil, err
}
if !ValidOpaqueID(message.ID) || !ValidOpaqueID(message.RequestID) {
return nil, errors.New("invalid interaction identity")
}
if !inRangeInt(message.OccurredAt, 0, MaxSafeInteger) {
return nil, errors.New("invalid interaction time")
}
return message, nil
}
func decodeRoomInteractionRejected(data []byte) (ServerMessage, error) {
var message RoomInteractionRejectedMessage
present, err := decodeObject(data, &message)
if err != nil {
return nil, err
}
if err = present.require("type", "requestId", "reason"); err != nil {
return nil, err
}
if !ValidOpaqueID(message.RequestID) {
return nil, errors.New("invalid interaction requestId")
}
switch message.Reason {
case "rate-limited", "target-unavailable", "not-subscribed", "busy":
default:
return nil, errors.New("invalid rejection reason")
}
return message, nil
}
internal/server/protocol/server.go
+63
View File
@@ -57,6 +57,23 @@ type AuthenticatedViewerMessage struct {
Role Role `json:"role"`
}
// SharingStartedMessage acknowledges a new media publication on an existing Host session.
type SharingStartedMessage struct {
Type string `json:"type"`
ShareGeneration string `json:"shareGeneration"`
RoutePolicy RoutePolicy `json:"routePolicy"`
RouteRevision Int `json:"routeRevision"`
RouteAssignment ParticipantRouteAssignment `json:"routeAssignment"`
QualitySettings QualitySettings `json:"qualitySettings"`
Paused bool `json:"paused"`
}
type SharingStartFailedMessage struct {
Type string `json:"type"`
ShareGeneration string `json:"shareGeneration"`
Code string `json:"code"`
}
// SignalingChallengeResponseMessage is the challenge echo.
type SignalingChallengeResponseMessage struct {
Type string `json:"type"`
@@ -197,6 +214,8 @@ type ErrorMessage struct {
func (AuthenticatedHostMessage) isServerMessage() {}
func (AuthenticatedViewerMessage) isServerMessage() {}
func (SharingStartedMessage) isServerMessage() {}
func (SharingStartFailedMessage) isServerMessage() {}
func (SignalingChallengeResponseMessage) isServerMessage() {}
func (ServerSignalMessage) isServerMessage() {}
func (ServerRestartRequestMessage) isServerMessage() {}
@@ -230,8 +249,18 @@ func DecodeServerMessage(data []byte) (ServerMessage, error) {
return nil, err
}
switch messageType {
case "room-interactions-ready":
return decodeRoomInteractionsReady(data)
case "room-interaction":
return decodeRoomInteraction(data)
case "room-interaction-rejected":
return decodeRoomInteractionRejected(data)
case "authenticated":
return decodeAuthenticated(data)
case "sharing-started":
return decodeSharingStarted(data)
case "sharing-start-failed":
return decodeSharingStartFailed(data)
case "signaling-challenge-response":
return decodeChallengeResponse(data)
case "signal":
@@ -386,6 +415,40 @@ func decodeAuthenticated(data []byte) (ServerMessage, error) {
return nil, fmt.Errorf("unknown authenticated role %q", role)
}
func decodeSharingStarted(data []byte) (ServerMessage, error) {
var message SharingStartedMessage
present, err := decodeObject(data, &message)
if err != nil {
return nil, err
}
if err := present.require("type", "shareGeneration", "routePolicy", "routeRevision",
"routeAssignment", "qualitySettings", "paused"); err != nil {
return nil, err
}
if !ValidOpaqueID(message.ShareGeneration) {
return nil, errors.New("shareGeneration is not an opaque id")
}
if !validRevision(message.RouteRevision) {
return nil, errors.New("routeRevision is out of range")
}
return message, nil
}
func decodeSharingStartFailed(data []byte) (ServerMessage, error) {
var message SharingStartFailedMessage
present, err := decodeObject(data, &message)
if err != nil {
return nil, err
}
if err := present.require("type", "shareGeneration", "code"); err != nil {
return nil, err
}
if !ValidOpaqueID(message.ShareGeneration) || (message.Code != "FORBIDDEN" && message.Code != "SERVER_ERROR") {
return nil, errors.New("invalid sharing-start-failed response")
}
return message, nil
}
func decodeChallengeResponse(data []byte) (ServerMessage, error) {
var message SignalingChallengeResponseMessage
present, err := decodeObject(data, &message)
internal/server/protocol/types.go
+7 -2
View File
@@ -170,7 +170,10 @@ func (v *RoutePolicy) UnmarshalJSON(data []byte) error {
type RuntimeCapabilities struct {
ConnectionAttemptProgress4 bool `json:"connectionAttemptProgress4,omitempty"`
Sfu bool `json:"sfu"`
SfuOnly bool `json:"sfuOnly,omitempty"`
NatPrediction bool `json:"natPrediction"`
RoomInteractions bool `json:"roomInteractions,omitempty"`
HostRoomSession bool `json:"hostRoomSession,omitempty"`
}
// UnmarshalJSON implements json.Unmarshaler.
@@ -183,12 +186,14 @@ func (v *RuntimeCapabilities) UnmarshalJSON(data []byte) error {
if present == nil {
return errors.New("runtime capabilities must be an object")
}
if err := present.optional("sfu", "natPrediction", "connectionAttemptProgress4"); err != nil {
if err := present.optional("sfu", "sfuOnly", "roomInteractions", "hostRoomSession", "natPrediction", "connectionAttemptProgress4"); err != nil {
return err
}
for key, target := range map[string]*bool{
"sfu": &v.Sfu, "natPrediction": &v.NatPrediction,
"sfu": &v.Sfu, "sfuOnly": &v.SfuOnly, "natPrediction": &v.NatPrediction,
"connectionAttemptProgress4": &v.ConnectionAttemptProgress4,
"roomInteractions": &v.RoomInteractions, "hostRoomSession": &v.HostRoomSession,
} {
if value, ok := present[key]; ok {
if err := json.Unmarshal(value, target); err != nil {
internal/server/route/api.go
+2
View File
@@ -455,6 +455,7 @@ type Options struct {
EndpointMediaCopyCapacity int
OperationTimeoutMs int64
SfuEnabled bool
SfuOnly bool
QualityConvergenceEnabled bool
NatPredictionEnabled bool
}
@@ -470,6 +471,7 @@ type Controller struct {
endpointMediaCopyCapacity int
operationTimeoutMs int64
sfuEnabled bool
sfuOnly bool
qualityConvergenceEnabled bool
natPredictionEnabled bool
internal/server/route/candidates.go
+4 -1
View File
@@ -70,7 +70,7 @@ func (c *Controller) buildCandidatePlans(childPeerID string, sfuOnly, preferUntr
}
var candidates []CandidateTuple
switch {
case sfuOnly:
case sfuOnly || c.sfuOnly:
if sfuCandidate != nil {
candidates = []CandidateTuple{*sfuCandidate}
}
@@ -166,6 +166,9 @@ func (c *Controller) candidateValid(childPeerID string, plan CandidatePlan, att
return false
}
if tuple.Kind == UpstreamPeer {
if c.sfuOnly {
return false
}
parent, _ := c.participants.Get(tuple.ParentPeerID)
if parent == nil || parent.sessionID == "" || parent.departureConfirmed || (att != nil && parent.sessionID != att.parentSessionID) ||
slices.Contains(c.descendantsOf(childPeerID), parent.peerID) || !c.sourceUsable(parent.peerID) ||
internal/server/route/controller.go
+7 -2
View File
@@ -271,6 +271,9 @@ func New(options Options) *Controller {
if !isSafeInteger(options.OperationTimeoutMs) || options.OperationTimeoutMs <= 0 {
panic(errors.New("Route operation timeout must be a positive integer"))
}
if options.SfuOnly && !options.SfuEnabled {
panic(errors.New("SFU-only routing requires SFU availability"))
}
c := &Controller{
hostPeerID: options.HostPeerID,
debugRoomID: options.DebugRoomID,
@@ -278,8 +281,9 @@ func New(options Options) *Controller {
endpointMediaCopyCapacity: options.EndpointMediaCopyCapacity,
operationTimeoutMs: options.OperationTimeoutMs,
sfuEnabled: options.SfuEnabled,
qualityConvergenceEnabled: options.QualityConvergenceEnabled,
natPredictionEnabled: options.NatPredictionEnabled,
sfuOnly: options.SfuOnly,
qualityConvergenceEnabled: options.QualityConvergenceEnabled && !options.SfuOnly,
natPredictionEnabled: options.NatPredictionEnabled && !options.SfuOnly,
routeTimings: make(map[string]*routeTimingRecord),
qualityObservations: make(map[string]*routeQualityObservation),
senderQualityObservations: make(map[string]*senderQualityObservation),
@@ -291,6 +295,7 @@ func New(options Options) *Controller {
"endpointCapacity", options.EndpointMediaCopyCapacity,
"operationTimeoutMs", options.OperationTimeoutMs,
"sfuEnabled", options.SfuEnabled,
"sfuOnly", options.SfuOnly,
"qualityConvergenceEnabled", options.QualityConvergenceEnabled)
return c
}
internal/server/route/sfu_only_test.go
+64
View File
@@ -0,0 +1,64 @@
package route
import "testing"
func TestSFUOnlyUsesOnePublicationWithoutPeerConvergence(t *testing.T) {
routes := newController(1, Options{SfuEnabled: true, SfuOnly: true,
QualityConvergenceEnabled: true, NatPredictionEnabled: true})
for index, viewer := range []string{A, B, C} {
now := int64(index * 10)
addViewer(routes, viewer, 1, &now)
op := must(t, routes.Reconcile(now).Operation)
eq(t, len(op.Candidates), 1)
publication := PublicationReuse
input := BeginInput{NowMs: now + 1, ConnectionID: viewer + "_sfu",
Reservation: sfuReuse(viewer)}
if index == 0 {
publication = PublicationCreate
input.PublicationGeneration = "publication"
input.PublicationConnectionID = "publication_connection"
input.Reservation = sfuCreate(viewer, "publication_resource")
}
eq(t, op.Candidates[0].Tuple, sfuTuple(publication))
prepared := beginOperation(t, routes, input)
// Availability still commits only on the exact child's media-ready proof.
eq(t, hasEdge(routes, viewer), false)
eq(t, routes.CandidateReady(guardFor(viewer, prepared.Current.Revision, input.ConnectionID),
now+2, nil, CandidateProof{}).Accepted, true)
eq(t, edgeOf(t, routes, viewer).Kind, UpstreamSfu)
noOperation(t, routes.Reconcile(now+3).Operation)
}
eq(t, routes.physicalCopies(HOST), 1)
eq(t, routes.directContinuations.Len(), 0)
// A failed publication is replaced through the same bounded operation.
publication := routes.Publication()
eq(t, routes.InvalidateHostPublication(PublicationGuard{HostSessionID: "host_session",
RouteRevision: routes.Revision(), Generation: publication.Generation,
ConnectionID: publication.ConnectionID}, ms(40)), true)
op := must(t, routes.Reconcile(41).Operation)
eq(t, len(op.Candidates), 1)
eq(t, op.Candidates[0].Tuple, sfuTuple(PublicationReplace))
}
func TestSFUOnlyFailureDoesNotEscapeToPeers(t *testing.T) {
for _, admissionDenied := range []bool{false, true} {
routes := newController(2, Options{SfuEnabled: true, SfuOnly: true})
addViewer(routes, A, 2, nil)
op := must(t, routes.Reconcile(0).Operation)
if admissionDenied {
routes.SkipCurrentCandidate(cursorGuard(op), 1, RejectionSfuAdmission)
} else {
prepared := beginOperation(t, routes, BeginInput{NowMs: 1,
ConnectionID: "sfu_connection", PublicationGeneration: "publication",
PublicationConnectionID: "publication_connection", Reservation: sfuCreate("edge", "publication")})
routes.CandidateFailed(guardFor(A, prepared.Current.Revision, "sfu_connection"), 2)
}
noOperation(t, routes.Reconcile(3).Operation)
eq(t, hasEdge(routes, A), false)
// Resource release may retry SFU; it must not reopen a Peer opportunity.
routes.TouchExternalFacts()
retry := must(t, routes.Reconcile(4).Operation)
eq(t, len(retry.Candidates), 1)
eq(t, retry.Candidates[0].Tuple.Kind, UpstreamSfu)
}
}
internal/server/signal/interactions.go
+62
View File
@@ -0,0 +1,62 @@
package signal
import "github.com/TNTcraftHIM/Piik/internal/server/protocol"
// Room authority remains in the authenticated session/store. Interaction state
// ends with that connection; no message history, routing state or retry worker
// lives on the server. Server.mu serializes membership checks and broadcast.
func (s *Server) handleRoomInteraction(sess *session, actor *authenticatedSession, message protocol.SendRoomInteractionMessage) {
reject := func(reason string) {
s.send(sess, protocol.RoomInteractionRejectedMessage{
Type: "room-interaction-rejected", RequestID: message.RequestID, Reason: reason,
})
}
if !sess.roomInteractions || actor.displayName == nil {
reject("not-subscribed")
return
}
now := s.now()
if now < sess.nextInteractionAtMs {
reject("rate-limited")
return
}
sess.nextInteractionAtMs = now + protocol.InteractionIntervalMs
if targetID := message.Payload.TargetPeerID; targetID != "" {
target, exists := s.connectedPeer(actor.roomID, targetID)
targetSession := s.sessionsByID[target.SessionID]
if !exists || targetSession == nil || targetSession.revoked || !targetSession.open() || !s.isCurrentSession(targetSession) {
reject("target-unavailable")
return
}
}
// Reserve the existing signaling queue for media/authority work when a
// reader stalls. Room data must not fill it and terminate a healthy share.
const maxInteractionQueueBytes = 16 * 1024
if sess.queuedBytes >= maxInteractionQueueBytes {
reject("busy")
return
}
event := protocol.RoomInteractionMessage{
Type: "room-interaction", ID: opaqueID(), RequestID: message.RequestID,
OccurredAt: protocol.Int(now),
Sender: protocol.InteractionSender{PeerID: actor.peerID, Role: actor.role, DisplayName: protocol.DisplayName(*actor.displayName)},
Payload: message.Payload,
}
encoded, err := protocol.EncodeServerMessage(event)
if err != nil {
panic(err)
}
deliver := func(sessionID string) {
target := s.sessionsByID[sessionID]
if target != nil && target.roomInteractions && !target.revoked && target.open() &&
s.isCurrentSession(target) && target.queuedBytes < maxInteractionQueueBytes {
s.sendEncoded(target, encoded)
}
}
if host, ok := s.store.GetConnectedHost(actor.roomID); ok {
deliver(host.SessionID)
}
for _, viewer := range s.store.GetConnectedViewers(actor.roomID) {
deliver(viewer.SessionID)
}
}
internal/server/signal/interactions_test.go
+144
View File
@@ -0,0 +1,144 @@
package signal
import (
"sync/atomic"
"testing"
"time"
"github.com/TNTcraftHIM/Piik/internal/server/protocol"
)
func subscribeInteractions(t *testing.T, client *testClient) {
t.Helper()
client.sendJSON(map[string]any{"type": "subscribe-room-interactions"})
ready := client.next("room-interactions-ready")
if _, err := protocol.DecodeServerMessage(ready.raw); err != nil {
t.Fatal(err)
}
}
func TestRoomInteractionsStayInsideAuthorizedOptedInMembership(t *testing.T) {
var now atomic.Int64
now.Store(100_000)
h := startHarness(t, harnessOptions{maxViewersPerRoom: 4, now: now.Load})
host, viewer, old := openClient(t, h), openClient(t, h), openClient(t, h)
authenticate(t, host, h.room, protocol.RoleHost, "host_interaction", -1, "", presenceOptions{displayName: "房主", viewerPresence: true})
actor := authenticate(t, viewer, h.room, protocol.RoleViewer, "viewer_interaction", -1, "", presenceOptions{displayName: "小明", viewerPresence: true})
authenticate(t, old, h.room, protocol.RoleViewer, "old_page_interaction", -1, "", presenceOptions{})
otherRoom, err := h.server.CreateRoom(protocol.CodeEntryOpen, nil, "")
if err != nil {
t.Fatal(err)
}
outsider := openClient(t, h)
other := authenticate(t, outsider, otherRoom, protocol.RoleViewer, "outsider_interaction", -1, "", presenceOptions{displayName: "Elsewhere", viewerPresence: true})
for _, client := range []*testClient{host, viewer, outsider} {
subscribeInteractions(t, client)
}
viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_1", "payload": map[string]any{"kind": "chat", "text": "<b>你好 👋</b>"}})
hostEvent := host.next("room-interaction")
viewerEvent := viewer.next("room-interaction")
if string(hostEvent.raw) != string(viewerEvent.raw) {
t.Fatal("recipients received different events")
}
decoded, err := protocol.DecodeServerMessage(hostEvent.raw)
if err != nil {
t.Fatal(err)
}
event := decoded.(protocol.RoomInteractionMessage)
if event.OccurredAt != protocol.Int(now.Load()) {
t.Fatal("event time was not stamped at server acceptance")
}
if event.Sender.PeerID != actor.PeerID || event.Sender.DisplayName != "小明" {
t.Fatal("sender identity was not derived from authentication")
}
for _, client := range []*testClient{old, outsider} {
if _, ok := client.tryNext("room-interaction", 30*time.Millisecond); ok {
t.Fatal("interaction escaped its negotiated room boundary")
}
}
viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_2", "payload": map[string]any{"kind": "reaction", "reaction": "wave"}})
expectMatch(t, viewer.next("room-interaction-rejected").raw, `{"reason":"rate-limited"}`)
now.Add(protocol.InteractionIntervalMs)
viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_3", "payload": map[string]any{"kind": "reaction", "reaction": "tomato", "targetPeerId": other.PeerID}})
expectMatch(t, viewer.next("room-interaction-rejected").raw, `{"reason":"target-unavailable"}`)
// Room data and its rejections never terminate signaling or require media.
viewer.sendJSON(map[string]any{"type": "signaling-challenge", "sequence": 123})
viewer.next("signaling-challenge-response")
}
func TestRoomInteractionRequiresAuthenticationAndSubscription(t *testing.T) {
h := startHarness(t, harnessOptions{})
guest := openClient(t, h)
guest.sendJSON(map[string]any{"type": "subscribe-room-interactions"})
expectMatch(t, guest.next("error").raw, `{"code":"AUTH_REQUIRED"}`)
viewer := openClient(t, h)
authenticate(t, viewer, h.room, protocol.RoleViewer, "unsubscribed_viewer", -1, "", presenceOptions{displayName: "Friend", viewerPresence: true})
viewer.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "request_message_1", "payload": map[string]any{"kind": "chat", "text": "hello"}})
expectMatch(t, viewer.next("room-interaction-rejected").raw, `{"reason":"not-subscribed"}`)
}
func TestViewersCanInteractAfterHostDisconnects(t *testing.T) {
var now atomic.Int64
now.Store(100_000)
h := startHarness(t, harnessOptions{now: now.Load})
host, first, second := openClient(t, h), openClient(t, h), openClient(t, h)
hostActor := authenticate(t, host, h.room, protocol.RoleHost, "leaving_host", -1, "",
presenceOptions{displayName: "Host", roomSession: true})
authenticate(t, first, h.room, protocol.RoleViewer, "remaining_first", -1, "", presenceOptions{displayName: "First"})
secondActor := authenticate(t, second, h.room, protocol.RoleViewer, "remaining_second", -1, "", presenceOptions{displayName: "Second"})
subscribeInteractions(t, first)
subscribeInteractions(t, second)
h.closeClient(host)
for _, payload := range []map[string]any{
{"kind": "chat", "text": "still here"},
{"kind": "reaction", "reaction": "wave"},
{"kind": "reaction", "reaction": "tomato", "targetPeerId": secondActor.PeerID},
} {
first.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "after_host_left", "payload": payload})
if a, b := first.next("room-interaction"), second.next("room-interaction"); string(a.raw) != string(b.raw) {
t.Fatal("remaining viewers did not receive the same interaction")
}
now.Add(protocol.InteractionIntervalMs)
}
first.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "departed_target",
"payload": map[string]any{"kind": "reaction", "reaction": "tomato", "targetPeerId": hostActor.PeerID}})
expectMatch(t, first.next("room-interaction-rejected").raw, `{"reason":"target-unavailable"}`)
}
func TestRoomInteractionBackpressureDoesNotRetireMediaSignaling(t *testing.T) {
h := startHarness(t, harnessOptions{})
host, viewer := openClient(t, h), openClient(t, h)
authenticate(t, host, h.room, protocol.RoleHost, "data_host_1234", -1, "", presenceOptions{displayName: "Host", viewerPresence: true})
actor := authenticate(t, viewer, h.room, protocol.RoleViewer, "slow_viewer_1234", -1, "", presenceOptions{displayName: "Viewer", viewerPresence: true})
subscribeInteractions(t, host)
subscribeInteractions(t, viewer)
h.locked(func() {
peer, ok := h.server.store.GetConnectedViewer(h.room.RoomID, actor.PeerID)
if !ok {
t.Fatal("missing viewer")
}
slow := h.server.sessionsByID[peer.SessionID]
// Inject a stalled writer under the same lock as a real broadcast, so
// the test does not depend on OS TCP buffer sizes or the reader's timing.
previous := slow.queuedBytes
slow.queuedBytes = 20 * 1024
hostPeer, _ := h.server.store.GetConnectedHost(h.room.RoomID)
hostSession := h.server.sessionsByID[hostPeer.SessionID]
h.server.handleRoomInteraction(hostSession, hostSession.authenticated, protocol.SendRoomInteractionMessage{
Type: "send-room-interaction", RequestID: "busy_request_1234", Payload: protocol.InteractionPayload{Kind: "chat", Text: "hello"},
})
if slow.terminated || slow.closeQueued {
t.Error("room data closed the signaling session")
}
if slow.queuedBytes != 20*1024 {
t.Error("room data entered a stalled signaling queue")
}
slow.queuedBytes = previous
})
host.next("room-interaction")
if _, ok := viewer.tryNext("room-interaction", 30*time.Millisecond); ok {
t.Fatal("stalled recipient got room data")
}
viewer.sendJSON(map[string]any{"type": "signaling-challenge", "sequence": 456})
viewer.next("signaling-challenge-response")
}
internal/server/signal/router.go
+4
View File
@@ -75,6 +75,7 @@ type routerOptions struct {
store *room.Store
endpointMediaCopyCapacity int
sfu *sfuFallback
sfuOnly bool
hooks routerHooks
now func() int64
afterFunc func(time.Duration, func()) func() bool
@@ -160,6 +161,7 @@ type router struct {
store *room.Store
capacity int
sfu *sfuFallback
sfuOnly bool
hooks routerHooks
now func() int64
afterFunc func(time.Duration, func()) func() bool
@@ -187,6 +189,7 @@ func newRouter(options routerOptions) *router {
store: options.store,
capacity: options.endpointMediaCopyCapacity,
sfu: options.sfu,
sfuOnly: options.sfuOnly,
hooks: options.hooks,
now: options.now,
afterFunc: options.afterFunc,
@@ -995,6 +998,7 @@ func (r *router) createController(roomID string, rm *roomRuntime, host authentic
EndpointMediaCopyCapacity: r.capacity,
OperationTimeoutMs: operationTimeoutMs,
SfuEnabled: r.sfu != nil && !(routePolicy != nil && routePolicy.PeerOnly),
SfuOnly: r.sfuOnly,
QualityConvergenceEnabled: routePolicy != nil && routePolicy.TopologyOptimization,
NatPredictionEnabled: routePolicy != nil && routePolicy.NatPrediction,
})
internal/server/signal/server.go
+181 -31
View File
@@ -72,6 +72,8 @@ type Options struct {
EndpointMediaCopyCapacity int
// SfuFallback is nil when embedded SFU media is disabled.
SfuFallback *SfuFallback
// SfuOnly restricts the existing controller to server media candidates.
SfuOnly bool
// Ice is the derived wire ICE configuration (config.IceConfig).
Ice protocol.IceConfig
// NATPredictionEnabled is the server capability that clamps a Host's
@@ -215,6 +217,9 @@ func New(options Options) (*Server, error) {
if options.SfuFallback != nil && (options.SfuFallback.Media == nil || options.SfuFallback.Admission == nil) {
return nil, errors.New("SFU fallback requires media and resource admission")
}
if options.SfuOnly && options.SfuFallback == nil {
return nil, errors.New("SFU-only routing requires SFU media and resource admission")
}
s := &Server{
store: options.Store,
endpointMediaCopyCapacity: options.EndpointMediaCopyCapacity,
@@ -271,6 +276,7 @@ func New(options Options) (*Server, error) {
store: s.store,
endpointMediaCopyCapacity: s.endpointMediaCopyCapacity,
sfu: fallback,
sfuOnly: options.SfuOnly,
hooks: routerHooks{
sendToSession: s.sendToSession,
shareGeneration: func(roomID string) string { return s.shares[roomID].generation },
@@ -713,6 +719,8 @@ type authRequest struct {
viewerPresence bool
connectionAttemptProgress4 bool
displayName *string
roomOnly bool
roomSession bool
}
func authRequestOf(message protocol.ClientMessage) (authRequest, bool) {
@@ -730,6 +738,8 @@ func authRequestOf(message protocol.ClientMessage) (authRequest, bool) {
viewerPresence: m.ViewerPresence,
connectionAttemptProgress4: m.ConnectionAttemptProgress4,
displayName: displayNameString(m.DisplayName),
roomOnly: m.RoomOnly,
roomSession: m.RoomSession,
}, true
case protocol.AuthenticateViewerMessage:
return authRequest{
@@ -758,6 +768,14 @@ func displayNameString(name *protocol.DisplayName) *string {
// section. Password checks run in a goroutine so the reader can keep reading
// while the KDF runs.
func (s *Server) authenticate(sess *session, request authRequest) {
// Reject the conflicting preference before acquiring room or media authority,
// including reconnects from older pages. Never turn peer-only into SFU consent.
if request.role == protocol.RoleHost && !request.roomOnly && s.router.sfuOnly && request.routePolicy.PeerOnly {
s.sendError(sess, "FORBIDDEN", "This site requires server media; turn off Privacy mode or use another site")
sess.close(websocket.StatusCode(protocol.SignalCloseAuthenticationFailed), "Route policy conflict")
s.finishAuthenticating(sess)
return
}
if request.role == protocol.RoleHost && !sess.siteAccessAuthenticated {
s.sendError(sess, "AUTH_REQUIRED", "Site access is required")
sess.close(websocket.StatusCode(protocol.SignalCloseAuthenticationFailed), "Authentication failed")
@@ -889,28 +907,32 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part
displayName: displayName,
viewerPresence: request.viewerPresence,
connectionAttemptProgress4: request.connectionAttemptProgress4,
roomSession: request.roomSession,
}
sess.authenticated = authenticated
if participant.Role == protocol.RoleViewer {
s.clearViewerEvidence(roomID, participant.PeerID)
}
s.clearViewerGrace(roomID, participant.PeerID)
routeParticipant := authenticatedRouteParticipant{
roomID: roomID,
role: participant.Role,
peerID: participant.PeerID,
sessionID: sess.sessionID,
mediaActive := participant.Role == protocol.RoleHost && shareGeneration != ""
if participant.Role == protocol.RoleViewer {
mediaActive = s.activeHostShareGeneration(roomID) != ""
}
if participant.Role == protocol.RoleHost {
routeParticipant := routeParticipant(sess, authenticated)
if participant.Role == protocol.RoleHost && mediaActive {
policy := s.routePolicyOf(roomID)
routeParticipant.routePolicy = &policy
}
hybridState, assigned := s.connectRouteParticipant(routeParticipant)
if !assigned {
s.logger.Error("Media router did not assign an authenticated participant")
s.sendError(sess, "SERVER_ERROR", "Media assignment failed")
sess.close(websocket.StatusInternalError, "Media assignment failed")
return
hybridState := hybridAuthenticationState{routeAssignment: emptyAssignment()}
if participant.Role == protocol.RoleViewer || mediaActive {
var assigned bool
hybridState, assigned = s.connectRouteParticipant(routeParticipant)
if !assigned {
s.logger.Error("Media router did not assign an authenticated participant")
s.sendError(sess, "SERVER_ERROR", "Media assignment failed")
sess.close(websocket.StatusInternalError, "Media assignment failed")
return
}
}
// The viewer's media identity is derived from the committed edge, never
// stored beside it; connectParticipant has just settled the graph.
@@ -934,8 +956,11 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part
qualitySettings = *share.qualitySettings
}
var roomShareGeneration *string
if share.generation != "" {
generation := share.generation
if participant.Role == protocol.RoleHost && mediaActive {
generation := shareGeneration
roomShareGeneration = &generation
} else if participant.Role == protocol.RoleViewer && mediaActive {
generation := s.activeHostShareGeneration(roomID)
roomShareGeneration = &generation
}
if participant.Role == protocol.RoleHost {
@@ -960,15 +985,15 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part
ViewerPasswordEnabled: participant.ViewerPasswordEnabled,
})
} else {
hostPaused := participant.HostOnline &&
share.pausedGeneration == share.generation
hostOnline := participant.HostOnline && mediaActive
hostPaused := hostOnline && share.pausedGeneration == share.generation
s.send(sess, protocol.AuthenticatedViewerMessage{
Type: "authenticated",
Protocol: protocol.SignalingProtocol,
PeerID: participant.PeerID,
MaxViewers: protocol.Int(s.store.MaxViewersPerRoom()),
EndpointMediaCopyCapacity: protocol.Int(s.endpointMediaCopyCapacity),
HostOnline: participant.HostOnline,
HostOnline: hostOnline,
HostPaused: &hostPaused,
ConnectionID: connectionID,
IceConfig: s.ice,
@@ -989,12 +1014,14 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part
}
}
if participant.Role == protocol.RoleHost {
if participant.Role == protocol.RoleHost && mediaActive {
s.router.setPaused(roomID, s.shares[roomID].pausedGeneration == shareGeneration)
}
s.router.completeAuthentication(routeParticipant, hybridState)
if participant.Role == protocol.RoleViewer || mediaActive {
s.router.completeAuthentication(routeParticipant, hybridState)
}
if participant.Role == protocol.RoleHost && shareGeneration != "" &&
if participant.Role == protocol.RoleHost && mediaActive &&
s.shares[roomID].pausedGeneration == shareGeneration {
s.sendToSession(sess.sessionID, protocol.PauseSharingSourceMessage{
Type: "pause-sharing-source",
@@ -1004,7 +1031,7 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part
s.sendViewerPresence(roomID)
if participant.Role == protocol.RoleHost {
if participant.Role == protocol.RoleHost && mediaActive {
for _, viewer := range connectedViewers {
s.sendToSession(viewer.SessionID, protocol.HostStatusMessage{
Type: "host-status",
@@ -1021,6 +1048,15 @@ func (s *Server) completeAuthentication(sess *session, request authRequest, part
// with. It returns the settled generation and must run with mu held, before the
// `authenticated` message is built from the record it writes.
func (s *Server) settleHostShare(roomID, sessionID string, request authRequest) string {
if request.roomOnly {
// A room-only reconnect must not resurrect a stale publication. The
// stop path normally already cleared it; this guard covers a socket
// loss between the client cleanup and the server's stop message.
if s.shares[roomID].generation != "" {
s.stopSharing(roomID)
}
return ""
}
current := s.shares[roomID]
currentGeneration := current.generation
shareGeneration := request.shareGeneration
@@ -1111,10 +1147,33 @@ func (s *Server) routePolicyOf(roomID string) protocol.RoutePolicy {
return s.configuredRoutePolicy(protocol.DefaultRoutePolicy)
}
// activeHostShareGeneration is the media truth for a room. The room share
// record intentionally keeps its generation as a stale reconnect fence after
// stopSharing, so membership or that fence alone cannot mean "publishing".
// Callers hold Server.mu.
func (s *Server) activeHostShareGeneration(roomID string) string {
host, ok := s.store.GetConnectedHost(roomID)
if !ok {
return ""
}
hostSession := s.sessionsByID[host.SessionID]
if hostSession == nil || hostSession.authenticated == nil ||
hostSession.authenticated.role != protocol.RoleHost ||
hostSession.authenticated.roomID != roomID {
return ""
}
generation := hostSession.authenticated.shareGeneration
if generation == "" || s.shares[roomID].generation != generation {
return ""
}
return generation
}
// configuredRoutePolicy restricts share preferences to the services this runtime owns.
func (s *Server) configuredRoutePolicy(policy protocol.RoutePolicy) protocol.RoutePolicy {
policy.PeerOnly = s.router.sfu == nil || policy.PeerOnly
policy.NatPrediction = s.natPredictionEnabled && policy.NatPrediction
policy.NatPrediction = s.natPredictionEnabled && !s.router.sfuOnly && policy.NatPrediction
policy.TopologyOptimization = !s.router.sfuOnly && policy.TopologyOptimization
return policy
}
@@ -1131,6 +1190,12 @@ func routeParticipant(sess *session, authenticated *authenticatedSession) authen
func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authenticatedSession, message protocol.ClientMessage) {
switch m := message.(type) {
case protocol.SubscribeRoomInteractionsMessage:
// Explicit opt-in protects older v23 readers from unknown events.
sess.roomInteractions = true
s.send(sess, protocol.RoomInteractionsReadyMessage{Type: "room-interactions-ready", ServerTime: protocol.Int(s.now())})
case protocol.SendRoomInteractionMessage:
s.handleRoomInteraction(sess, authenticated, m)
case protocol.SignalingChallengeMessage:
if !s.sessions.Has(sess) || sess.authenticated != authenticated {
return
@@ -1165,6 +1230,10 @@ func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authen
s.sendError(sess, "FORBIDDEN", "Only the host may set quality settings")
return
}
if authenticated.shareGeneration == "" ||
s.shares[authenticated.roomID].generation != authenticated.shareGeneration {
return
}
share := s.shares[authenticated.roomID]
share.qualitySettings = &m.QualitySettings
s.shares[authenticated.roomID] = share
@@ -1246,28 +1315,39 @@ func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authen
s.shares[authenticated.roomID] = share
s.router.setPaused(authenticated.roomID, m.Paused)
s.broadcastHostStatus(authenticated.roomID, true, m.Paused)
case protocol.StartSharingMessage:
s.startSharing(sess, authenticated, m)
case protocol.StopSharingMessage:
if authenticated.role != protocol.RoleHost {
s.sendError(sess, "FORBIDDEN", "只有当前房主可以停止分享")
return
}
if authenticated.shareGeneration == "" ||
s.shares[authenticated.roomID].generation != authenticated.shareGeneration ||
if authenticated.shareGeneration == "" {
return
}
if s.shares[authenticated.roomID].generation != authenticated.shareGeneration ||
(m.ShareGeneration != "" && m.ShareGeneration != authenticated.shareGeneration) {
sess.close(websocket.StatusCode(protocol.SignalCloseSessionReplaced), "Sharing generation replaced")
return
}
// The participant leaves the store before stopSharing()
// so the sharing-stopped / host-status pair is what viewers see.
if s.sessions.Has(sess) && sess.authenticated == authenticated {
if _, err := s.store.DisconnectParticipant(authenticated.roomID, authenticated.peerID, sess.sessionID); err != nil {
// The TS threw out of the message handler; nothing caught it.
panic(fmt.Errorf("room store failed while stopping sharing: %w", err))
if !authenticated.roomSession {
// Older clients treat stopping media as leaving the Host session.
// Keep that contract until the client explicitly opts into the
// persistent room-session lifecycle.
if s.sessions.Has(sess) && sess.authenticated == authenticated {
if _, err := s.store.DisconnectParticipant(authenticated.roomID, authenticated.peerID, sess.sessionID); err != nil {
panic(fmt.Errorf("room store failed while stopping sharing: %w", err))
}
}
s.stopSharing(authenticated.roomID)
sess.close(websocket.StatusNormalClosure, "Sharing stopped")
return
}
s.stopSharing(authenticated.roomID)
// Room membership and its interaction stream outlive the media
// publication. The next start-sharing message claims a fresh generation.
authenticated.shareGeneration = ""
s.sendViewerPresence(authenticated.roomID)
sess.close(websocket.StatusNormalClosure, "Sharing stopped")
case protocol.AbandonRoomMessage:
if authenticated.role != protocol.RoleHost {
s.sendError(sess, "FORBIDDEN", "Only the host may abandon the room")
@@ -1280,6 +1360,76 @@ func (s *Server) handleAuthenticatedMessage(sess *session, authenticated *authen
}
}
// startSharing claims a new publication generation without replacing the
// authenticated Host participant. Room interactions therefore keep the same
// session while the media graph is rebuilt for the new source.
func (s *Server) startSharing(sess *session, authenticated *authenticatedSession, message protocol.StartSharingMessage) {
if authenticated.role != protocol.RoleHost {
s.sendError(sess, "FORBIDDEN", "Only the host may start sharing")
return
}
if !authenticated.roomSession {
s.sendError(sess, "FORBIDDEN", "Persistent room session is not enabled")
return
}
if !s.sessions.Has(sess) || sess.authenticated != authenticated ||
authenticated.shareGeneration != "" || message.ShareGeneration == s.shares[authenticated.roomID].generation ||
(s.router.sfuOnly && message.RoutePolicy.PeerOnly) {
s.send(sess, protocol.SharingStartFailedMessage{Type: "sharing-start-failed",
ShareGeneration: message.ShareGeneration, Code: "FORBIDDEN"})
return
}
// stop-sharing deliberately leaves the previous generation as a stale
// fence for reconnect validation. A fresh start supersedes that fence
// without emitting a second sharing-stopped event.
share := s.shares[authenticated.roomID]
share.generation = ""
share.pausedGeneration = ""
s.shares[authenticated.roomID] = share
request := authRequest{
role: protocol.RoleHost,
roomID: authenticated.roomID,
shareGeneration: message.ShareGeneration,
sharingPaused: message.SharingPaused,
qualitySettings: message.QualitySettings,
routePolicy: message.RoutePolicy,
}
shareGeneration := s.settleHostShare(authenticated.roomID, sess.sessionID, request)
authenticated.shareGeneration = shareGeneration
routeParticipant := routeParticipant(sess, authenticated)
policy := s.routePolicyOf(authenticated.roomID)
routeParticipant.routePolicy = &policy
routeState, assigned := s.connectRouteParticipant(routeParticipant)
if !assigned {
authenticated.shareGeneration = ""
s.stopSharing(authenticated.roomID)
s.send(sess, protocol.SharingStartFailedMessage{Type: "sharing-start-failed",
ShareGeneration: message.ShareGeneration, Code: "SERVER_ERROR"})
return
}
share = s.shares[authenticated.roomID]
qualitySettings := protocol.DefaultQualitySettings
if share.qualitySettings != nil {
qualitySettings = *share.qualitySettings
}
s.send(sess, protocol.SharingStartedMessage{
Type: "sharing-started",
ShareGeneration: shareGeneration,
RoutePolicy: s.routePolicyOf(authenticated.roomID),
RouteRevision: protocol.Int(routeState.routeRevision),
RouteAssignment: routeState.routeAssignment,
QualitySettings: qualitySettings,
Paused: share.pausedGeneration == shareGeneration,
})
// Acknowledge publication authority before any media effects use it.
s.router.setPaused(authenticated.roomID, share.pausedGeneration == shareGeneration)
s.router.completeAuthentication(routeParticipant, routeState)
s.sendViewerPresence(authenticated.roomID)
s.broadcastHostStatus(authenticated.roomID, true, share.pausedGeneration == shareGeneration)
}
// ---------------------------------------------------------------------------
// quality evidence
// ---------------------------------------------------------------------------
internal/server/signal/server_test.go
+157
View File
@@ -9,6 +9,7 @@ package signal
// TS went through HTTP; the site-access cookie is a stub predicate.
import (
"bytes"
"context"
"encoding/json"
"errors"
@@ -80,6 +81,7 @@ func TestStopSharingDoesNotWriteRoomAuthority(t *testing.T) {
h := startHarness(t, harnessOptions{store: store})
host := openClient(t, h)
authenticated := authenticate(t, host, h.room, protocol.RoleHost, "storage-free-host", 1, "", presenceOptions{})
host.ignore("route-update")
h.locked(func() {
if err := database.Close(); err != nil {
t.Fatal(err)
@@ -94,6 +96,151 @@ func TestStopSharingDoesNotWriteRoomAuthority(t *testing.T) {
authenticate(t, resumed, h.room, protocol.RoleHost, "storage-free-host", 1, "", presenceOptions{})
}
func TestHostKeepsRoomSessionAndInteractionsAfterSharingStops(t *testing.T) {
h := startHarness(t, harnessOptions{maxViewersPerRoom: 2})
host := openClient(t, h)
hostAuth := authenticate(t, host, h.room, protocol.RoleHost, "persistent-host", 1,
"first_share_generation_12345678", presenceOptions{displayName: "Host", viewerPresence: true, roomSession: true})
viewer := openClient(t, h)
authenticate(t, viewer, h.room, protocol.RoleViewer, "persistent-viewer", 1, "",
presenceOptions{displayName: "Viewer", viewerPresence: true})
host.ignore("route-update")
host.ignore("viewer-presence")
subscribeInteractions(t, host)
subscribeInteractions(t, viewer)
host.sendJSON(map[string]any{"type": "stop-sharing", "shareGeneration": hostAuth.ShareGeneration})
viewer.next("sharing-stopped")
viewer.next("host-status")
host.expectNone(80 * time.Millisecond)
h.locked(func() {
connected, ok := h.server.store.GetConnectedHost(h.room.RoomID)
if !ok || connected.SessionID == "" {
t.Fatal("stopping media must retain the Host room session")
}
})
host.sendJSON(map[string]any{
"type": "send-room-interaction",
"requestId": "persistent_chat_request",
"payload": map[string]any{"kind": "chat", "text": "still here"},
})
host.next("room-interaction")
viewer.next("room-interaction")
host.sendJSON(map[string]any{
"type": "start-sharing",
"shareGeneration": "second_share_generation_12345678",
"routePolicy": map[string]any{"peerOnly": true, "topologyOptimization": true, "natPrediction": false},
})
started := host.next("sharing-started")
if !bytes.Contains(started.raw, []byte(`"shareGeneration":"second_share_generation_12345678"`)) {
t.Fatalf("unexpected sharing-started: %s", started.raw)
}
viewer.next("host-status")
h.closeClient(viewer)
h.closeClient(host)
}
func TestIdleHostChatAndPublicationAcknowledgmentOrder(t *testing.T) {
h := startHarness(t, harnessOptions{})
host := openClient(t, h)
authenticate(t, host, h.room, protocol.RoleHost, "room-only-host", 1, "", presenceOptions{roomSession: true, roomOnly: true, viewerPresence: true})
host.ignore("viewer-presence")
viewer := openClient(t, h)
authenticate(t, viewer, h.room, protocol.RoleViewer, "room-only-viewer", 1, "", presenceOptions{})
subscribeInteractions(t, host)
subscribeInteractions(t, viewer)
host.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "chat_before_capture",
"payload": map[string]any{"kind": "chat", "text": "before capture"}})
host.next("room-interaction")
viewer.next("room-interaction")
host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "new_publication_generation",
"routePolicy": protocol.DefaultRoutePolicy})
// No ignored route messages: acknowledgment must precede media effects.
host.sendJSON(map[string]any{"type": "signaling-challenge", "sequence": 1})
host.next("signaling-challenge-response")
host.mu.Lock()
first := host.messages[0].typ
host.mu.Unlock()
if first != "sharing-started" {
t.Fatalf("first publication message = %s", first)
}
host.next("sharing-started")
host.next("route-update")
}
func TestRejectedPublicationRetainsHostRoomSession(t *testing.T) {
h := startHarness(t, harnessOptions{})
host := openClient(t, h)
authenticate(t, host, h.room, protocol.RoleHost, "rejected-start-host", 1, "", presenceOptions{roomSession: true, roomOnly: true, viewerPresence: true})
host.ignore("viewer-presence")
subscribeInteractions(t, host)
// Inject the existing route-construction failure, after room admission.
h.locked(func() { h.server.router.capacity = 0 })
host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "rejected_publication_generation",
"routePolicy": protocol.DefaultRoutePolicy})
expectMatch(t, host.next("sharing-start-failed").raw,
`{"shareGeneration":"rejected_publication_generation","code":"SERVER_ERROR"}`)
h.locked(func() {
h.server.router.capacity = 2
if h.server.activeHostShareGeneration(h.room.RoomID) != "" {
t.Fatal("failed publication remained active")
}
})
host.sendJSON(map[string]any{"type": "send-room-interaction", "requestId": "chat_after_rejected_start",
"payload": map[string]any{"kind": "chat", "text": "still in room"}})
host.next("room-interaction")
host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "next_publication_generation",
"routePolicy": protocol.DefaultRoutePolicy})
host.next("sharing-started")
host.next("route-update")
host.ignore("route-update")
host.sendJSON(map[string]any{"type": "stop-sharing", "shareGeneration": "next_publication_generation"})
host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "next_publication_generation",
"routePolicy": protocol.DefaultRoutePolicy})
expectMatch(t, host.next("sharing-start-failed").raw,
`{"shareGeneration":"next_publication_generation","code":"FORBIDDEN"}`)
}
func TestPersistentRoomSessionReconnectStaysMediaIdle(t *testing.T) {
h := startHarness(t, harnessOptions{maxViewersPerRoom: 2})
host := openClient(t, h)
hostAuth := authenticate(t, host, h.room, protocol.RoleHost, "idle-reconnect-host", 1,
"idle_reconnect_generation_12345678", presenceOptions{roomSession: true})
viewer := openClient(t, h)
authenticate(t, viewer, h.room, protocol.RoleViewer, "idle-reconnect-viewer", 1, "", presenceOptions{})
host.ignore("route-update")
viewer.ignore("route-update")
host.sendJSON(map[string]any{"type": "stop-sharing", "shareGeneration": hostAuth.ShareGeneration})
viewer.next("sharing-stopped")
viewer.next("host-status")
h.closeClient(viewer)
h.closeClient(host)
resumed := openClient(t, h)
resumedAuth := authenticate(t, resumed, h.room, protocol.RoleHost, "idle-reconnect-host", 1, "",
presenceOptions{roomSession: true, roomOnly: true})
if resumedAuth.ShareGeneration != nil {
t.Fatalf("room-only Host must not receive a media generation: %v", *resumedAuth.ShareGeneration)
}
h.locked(func() {
if runtime, ok := h.server.router.rooms.Get(h.room.RoomID); ok && runtime.controller != nil {
t.Fatal("room-only Host reconnect created a media controller")
}
})
lateViewer := openClient(t, h)
lateAuth := authenticate(t, lateViewer, h.room, protocol.RoleViewer, "idle-reconnect-late-viewer", 1, "", presenceOptions{})
if lateAuth.HostOnline || lateAuth.HostPaused == nil || *lateAuth.HostPaused {
t.Fatalf("idle room advertised active Host media: online=%v paused=%v", lateAuth.HostOnline, lateAuth.HostPaused)
}
if lateAuth.ShareGeneration != nil || lateAuth.RouteAssignment.Upstream.Kind != "none" {
t.Fatalf("idle room advertised a media route: generation=%v assignment=%+v", lateAuth.ShareGeneration, lateAuth.RouteAssignment)
}
h.closeClient(lateViewer)
h.closeClient(resumed)
}
// ---------------------------------------------------------------------------
// harness
// ---------------------------------------------------------------------------
@@ -113,6 +260,7 @@ type harnessOptions struct {
room *room.CreatedRoom
afterFunc func(time.Duration, func()) func() bool
sfu *SfuFallback
sfuOnly bool
}
type harness struct {
@@ -189,6 +337,7 @@ func startHarness(t *testing.T, options harnessOptions) *harness {
Store: store,
EndpointMediaCopyCapacity: orValue(options.endpointMediaCopyCapacity, 2),
SfuFallback: options.sfu,
SfuOnly: options.sfuOnly,
Ice: config.IceConfig(config.Config{
STUNURLs: options.stunURLs,
NATPredictionEnabled: options.natPredictionEnabled,
@@ -803,6 +952,8 @@ type presenceOptions struct {
sharingPaused *bool
qualitySettings map[string]any
routePolicy *protocol.RoutePolicy
roomSession bool
roomOnly bool
}
// authenticate ports the authenticate() helper. relayCapacity -1 is the TS
@@ -833,6 +984,12 @@ func authenticate(
if shareGeneration != "" {
message["shareGeneration"] = shareGeneration
}
if presence.roomSession {
message["roomSession"] = true
}
if presence.roomOnly {
message["roomOnly"] = true
}
if presence.sharingPaused != nil {
message["sharingPaused"] = *presence.sharingPaused
}
internal/server/signal/session.go
+3
View File
@@ -23,6 +23,7 @@ type authenticatedSession struct {
displayName *string
viewerPresence bool
connectionAttemptProgress4 bool
roomSession bool
}
// outbound is one queue item: a text frame or the close frame that ends the
@@ -54,6 +55,8 @@ type session struct {
challenged bool
lastSignalingChallengeAtMs int64
authenticated *authenticatedSession
roomInteractions bool
nextInteractionAtMs int64
// authStop stops the authentication deadline; authGeneration is
// bumped whenever the TS cleared the timer so a callback that lost the
internal/server/signal/sfu_only_test.go
+34
View File
@@ -0,0 +1,34 @@
package signal
import (
"testing"
"github.com/TNTcraftHIM/Piik/internal/server/protocol"
"github.com/TNTcraftHIM/Piik/internal/server/sfu"
"github.com/TNTcraftHIM/Piik/internal/server/sfu/sfutest"
)
func TestSFUOnlyRejectsPeerOnlyBeforeAcquiringShareAuthority(t *testing.T) {
h := startHarness(t, harnessOptions{sfuOnly: true, natPredictionEnabled: true,
sfu: &SfuFallback{Media: sfutest.New(), Admission: sfu.NewAdmission(sfu.AdmissionOptions{IngressCapacity: 2, EgressCapacity: 20})}})
policy := protocol.RoutePolicy{PeerOnly: true, TopologyOptimization: true, NatPrediction: true}
rejected := openClient(t, h)
rejected.sendJSON(map[string]any{"type": "authenticate", "protocol": protocol.SignalingProtocol,
"roomId": h.room.RoomID, "role": "host", "clientId": "old-peer-only-host",
"token": h.room.HostToken, "routePolicy": policy})
expectMatch(t, rejected.next("error").raw, `{"code":"FORBIDDEN"}`)
expectCloseCode(t, rejected, int(protocol.SignalCloseAuthenticationFailed))
// An idle room remains available. Refusing a conflicting share preserves it.
host := openClient(t, h)
authenticate(t, host, h.room, protocol.RoleHost, "idle-host", -1, "",
presenceOptions{roomSession: true, roomOnly: true, routePolicy: &policy})
host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "sfu_only_share_generation",
"routePolicy": policy})
expectMatch(t, host.next("sharing-start-failed").raw, `{"code":"FORBIDDEN"}`)
policy.PeerOnly = false
host.sendJSON(map[string]any{"type": "start-sharing", "shareGeneration": "sfu_only_share_generation",
"routePolicy": policy})
expectMatch(t, host.next("sharing-started").raw,
`{"routePolicy":{"peerOnly":false,"topologyOptimization":false,"natPrediction":false}}`)
}
native/capture/windows/README.md
+21 -2
View File
@@ -5,7 +5,8 @@ enumerates local displays and visible top-level windows, binds each window to
its PID and process creation time, and has independent modes for:
- capability discovery (`--probe`);
- process-tree or default-device loopback PCM (`--capture-audio`); or
- process-tree inclusion/exclusion or default-device loopback PCM
(`--capture-audio`); or
- WGC/D3D11 screen/window video with adapter-bound Media Foundation H.264 or
libvpx VP8 output (`--capture-video --codec auto|h264|vp8`).
- bounded 320x180 BMP source previews (`--preview`), delivered once per target
@@ -30,7 +31,12 @@ Video and audio run as separate bounded child processes. A source whose audio
loopback cannot be initialized keeps video available and reports audio
unavailable instead of failing the whole source. Process loopback is probed by
activation rather than inferred from a Windows build number; display sources
use the standard render-device loopback available on Windows 10 and later.
normally use the standard render-device loopback available on Windows 10 and
later. The optional `--capture-audio exclude <pid> <creation-time>` mode requires
build 20348+ and a successful exclusion probe. It captures all output devices
except the selected process tree, and retires when that process exits. The
[screen-audio contract](../../../docs/standards/media-quality.md#screen-audio)
owns selection and failure behavior.
Auto measures delivered frame cadence through the target-profile encoding
pipeline within a four-second selection budget; H.264 that sustains the target
@@ -50,6 +56,19 @@ The checks include synthetic output-worker replacement and failure cases using
WARP and a test codec, plus preview cleanup when WGC rejects a hidden test window.
They do not capture a user screen or validate a physical encoder.
On a Windows machine with a physical D3D11 adapter, check SDR and HDR-to-SDR conversion:
```powershell
powershell -NoProfile -File native/capture/windows/build.ps1 -OutputDirectory build/color-check -CheckColor
```
This sends synthetic RGB, full/limited-range BT.601/BT.709 NV12 and FP16 HDR bars
through the production converters. It checks range/matrix, highlight detail,
SDR/HDR replacement, white-level changes and retained-frame ownership without
capturing the desktop. Decoded H.264/VP8, Browser reception and real HDR displays
remain separate media acceptance; [fidelity research](../../../docs/research/media-fidelity.md)
records current evidence and limits.
The implementation follows Microsoft's MIT-licensed reference samples and
official API contracts without copying their WIL framework. The retained MF
fixture compiles the same encoder source with `PIIK_H264_FIXTURE`; there is
native/capture/windows/build.ps1
+15 -2
View File
@@ -2,7 +2,8 @@
param(
[Parameter(Mandatory = $true)]
[string]$OutputDirectory,
[switch]$Check
[switch]$Check,
[switch]$CheckColor
)
Set-StrictMode -Version Latest
@@ -34,7 +35,7 @@ $includeFlags = @('', 'third_party\abseil-cpp', 'third_party\boringssl\src\inclu
'/external:I "{0}"' -f (Join-Path $webrtc.Include $_).TrimEnd('\')
}
$compileFlags = '/nologo /c /std:c++20 /EHsc /GR /O2 /W4 /WX /MT /D_ITERATOR_DEBUG_LEVEL=0 /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 /DNTDDI_VERSION=0x0A00000A /external:W0 ' + ($includeFlags -join ' ')
$systemLibraries = 'ole32.lib mmdevapi.lib runtimeobject.lib user32.lib gdi32.lib dwmapi.lib shell32.lib mfplat.lib mf.lib mfuuid.lib d3d11.lib dxgi.lib dxguid.lib evr.lib oleaut32.lib windowsapp.lib winmm.lib ws2_32.lib strmiids.lib crypt32.lib dmoguids.lib iphlpapi.lib msdmo.lib secur32.lib wmcodecdspuuid.lib'
$systemLibraries = 'ole32.lib mmdevapi.lib runtimeobject.lib user32.lib gdi32.lib dwmapi.lib shell32.lib mfplat.lib mf.lib mfuuid.lib d3d11.lib d2d1.lib dxgi.lib dxguid.lib evr.lib oleaut32.lib windowsapp.lib winmm.lib ws2_32.lib strmiids.lib crypt32.lib dmoguids.lib iphlpapi.lib msdmo.lib secur32.lib wmcodecdspuuid.lib'
$linkCommand = '"{0}" /nologo /libpath:"{1}"' -f $webrtc.Linker,$webrtc.RuntimeLibraries
function Invoke-CaptureBuild([string]$command) {
@@ -76,4 +77,16 @@ if ($Check) {
if ($LASTEXITCODE -ne 0) { throw "Capture check failed: $name" }
}
}
if ($CheckColor) {
$source = Join-Path $helperDirectory 'frame_converter.probe.cpp'
$object = Join-Path $outputPath 'frame_converter.probe.obj'
$executable = Join-Path $outputPath 'frame_converter.probe.exe'
Invoke-CaptureBuild ('cl.exe {0} "{1}" /Fo:"{2}"' -f $compileFlags,$source,$object)
$dependencies = @('process_audio', 'capture_target', 'h264_encoder', 'adaptive_encoder') | ForEach-Object {
'"{0}"' -f (Join-Path $outputPath ($_ + '.obj'))
}
Invoke-CaptureBuild ('{0} /out:"{1}" "{2}" {3} "{4}" {5}' -f $linkCommand,$executable,$object,($dependencies -join ' '),$webrtc.Library,$systemLibraries)
& $executable
if ($LASTEXITCODE -ne 0) { throw 'Capture color check failed.' }
}
Write-Output $executablePath
native/capture/windows/capture_color.h
+34
View File
@@ -0,0 +1,34 @@
#pragma once
#include "capture_error.h"
#include <dxgicommon.h>
#include <mfapi.h>
#include <mfobjects.h>
namespace piik::capture::windows {
// Untagged WebRTC SDR uses BT.601. Keep the converter's pixels and the H264
// encoder's range/matrix declaration in agreement; VP8 has no equivalent VUI.
inline constexpr auto kSdrVideoColor = DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P601;
inline constexpr auto kDesktopColor = DXGI_COLOR_SPACE_RGB_FULL_G22_NONE_P709;
inline void SetSdrVideoColor(IMFAttributes* type) {
Check(type->SetUINT32(MF_MT_YUV_MATRIX, MFVideoTransferMatrix_BT601), "video-type-matrix");
Check(type->SetUINT32(MF_MT_VIDEO_NOMINAL_RANGE, MFNominalRange_16_235), "video-type-range");
Check(type->SetUINT32(MF_MT_VIDEO_PRIMARIES, MFVideoPrimaries_SMPTE170M), "video-type-primaries");
Check(type->SetUINT32(MF_MT_TRANSFER_FUNCTION, MFVideoTransFunc_709), "video-type-transfer");
}
// The decoder owns the incoming stream's declaration. Preserve it through the
// relay's local scale/re-encode boundary instead of guessing from resolution.
inline DXGI_COLOR_SPACE_TYPE DecodedVideoColor(IMFAttributes* type) {
const auto matrix = MFGetAttributeUINT32(type, MF_MT_YUV_MATRIX, MFVideoTransferMatrix_BT601);
const bool full = MFGetAttributeUINT32(type, MF_MT_VIDEO_NOMINAL_RANGE,
MFNominalRange_16_235) == MFNominalRange_0_255;
if (matrix == MFVideoTransferMatrix_BT709)
return full ? DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P709 : DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P709;
return full ? DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P601 : kSdrVideoColor;
}
} // namespace piik::capture::windows
native/capture/windows/capture_sdr.h
+209
View File
@@ -0,0 +1,209 @@
#pragma once
#include "capture_error.h"
#include "capture_target.h"
#include <d2d1_1.h>
#include <d2d1effects_2.h>
#include <d3d11.h>
#include <dxgi1_6.h>
#include <wrl/client.h>
#include <algorithm>
#include <cmath>
#include <utility>
#include <vector>
namespace piik::capture::windows {
struct CaptureColor final {
bool hdr = false;
float white_nits = 80.0f;
float peak_nits = 80.0f;
DXGI_FORMAT Format() const {
return hdr ? DXGI_FORMAT_R16G16B16A16_FLOAT : DXGI_FORMAT_B8G8R8A8_UNORM;
}
};
// Windows invalidates the DXGI factory when display color characteristics
// change. Window movement additionally changes the source monitor; neither
// event belongs to encoder or media-route recovery.
class CaptureDisplayColor final {
public:
CaptureDisplayColor(TargetKind kind, UINT64 source_id)
: kind_(kind), source_id_(source_id) {}
CaptureColor Resolve() {
const auto handle = static_cast<UINT_PTR>(source_id_);
const HMONITOR monitor = kind_ == TargetKind::window
? MonitorFromWindow(reinterpret_cast<HWND>(handle), MONITOR_DEFAULTTONEAREST)
: reinterpret_cast<HMONITOR>(handle);
if (factory_ && factory_->IsCurrent() && monitor == monitor_) return color_;
Microsoft::WRL::ComPtr<IDXGIFactory1> factory;
if (FAILED(CreateDXGIFactory1(IID_PPV_ARGS(&factory))))
return monitor == monitor_ ? color_ : CaptureColor{};
CaptureColor next;
for (UINT adapter_index = 0;; ++adapter_index) {
Microsoft::WRL::ComPtr<IDXGIAdapter1> adapter;
if (FAILED(factory->EnumAdapters1(adapter_index, &adapter))) break;
for (UINT output_index = 0;; ++output_index) {
Microsoft::WRL::ComPtr<IDXGIOutput> output;
if (FAILED(adapter->EnumOutputs(output_index, &output))) break;
Microsoft::WRL::ComPtr<IDXGIOutput6> extended;
DXGI_OUTPUT_DESC1 description{};
if (FAILED(output.As(&extended)) || FAILED(extended->GetDesc1(&description)) ||
description.Monitor != monitor) continue;
next.hdr = description.ColorSpace == DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020;
if (next.hdr) {
next.white_nits = ReadWhiteLevel(description.DeviceName);
// WGC has no per-frame MaxCLL. Use the source display's stable peak;
// absent usable metadata, retain the HDR10 luminance range.
const float peak = std::isfinite(description.MaxLuminance) &&
description.MaxLuminance > 0.0f
? description.MaxLuminance : 10'000.0f;
next.peak_nits = std::max(next.white_nits, peak);
}
factory_ = std::move(factory);
monitor_ = monitor;
color_ = next;
return color_;
}
}
return next;
}
private:
static float ReadWhiteLevel(const wchar_t* device_name) {
UINT32 path_count = 0, mode_count = 0;
if (GetDisplayConfigBufferSizes(QDC_ONLY_ACTIVE_PATHS, &path_count, &mode_count) !=
ERROR_SUCCESS) return 80.0f;
std::vector<DISPLAYCONFIG_PATH_INFO> paths(path_count);
std::vector<DISPLAYCONFIG_MODE_INFO> modes(mode_count);
if (QueryDisplayConfig(QDC_ONLY_ACTIVE_PATHS, &path_count, paths.data(), &mode_count,
modes.data(), nullptr) != ERROR_SUCCESS) return 80.0f;
for (UINT32 index = 0; index < path_count; ++index) {
const auto& path = paths[index];
DISPLAYCONFIG_SOURCE_DEVICE_NAME name{};
name.header = {DISPLAYCONFIG_DEVICE_INFO_GET_SOURCE_NAME, sizeof(name),
path.sourceInfo.adapterId, path.sourceInfo.id};
if (DisplayConfigGetDeviceInfo(&name.header) != ERROR_SUCCESS ||
wcscmp(name.viewGdiDeviceName, device_name) != 0) continue;
DISPLAYCONFIG_SDR_WHITE_LEVEL white{};
white.header = {DISPLAYCONFIG_DEVICE_INFO_GET_SDR_WHITE_LEVEL, sizeof(white),
path.targetInfo.adapterId, path.targetInfo.id};
if (DisplayConfigGetDeviceInfo(&white.header) == ERROR_SUCCESS && white.SDRWhiteLevel > 0)
return static_cast<float>(white.SDRWhiteLevel) * 80.0f / 1000.0f;
}
return 80.0f;
}
const TargetKind kind_;
const UINT64 source_id_;
Microsoft::WRL::ComPtr<IDXGIFactory1> factory_;
HMONITOR monitor_ = nullptr;
CaptureColor color_;
};
// Produces an owned SDR texture before preview or output fanout. Ordinary SDR
// keeps its exact copy path. HDR uses Windows effects on the capture GPU once,
// rather than duplicating tone mapping in each encoder or reading pixels to CPU.
class CaptureSdrConverter final {
public:
explicit CaptureSdrConverter(ID3D11Device* device) : device_(device) {
device_->GetImmediateContext(&context_);
}
Microsoft::WRL::ComPtr<ID3D11Texture2D> Convert(ID3D11Texture2D* source,
const CaptureColor& color) {
D3D11_TEXTURE2D_DESC description{};
source->GetDesc(&description);
if (description.Format != color.Format())
Fail("capture-color-format", "Captured frame and display color generation differ");
description.Usage = D3D11_USAGE_DEFAULT;
description.CPUAccessFlags = description.MiscFlags = 0;
description.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET;
Microsoft::WRL::ComPtr<ID3D11Texture2D> owned;
Check(device_->CreateTexture2D(&description, nullptr, &owned), "capture-owned-input");
context_->CopyResource(owned.Get(), source);
if (!color.hdr) return owned;
if (!drawing_) Initialize();
Microsoft::WRL::ComPtr<IDXGISurface> surface;
Check(owned.As(&surface), "capture-hdr-surface");
const auto input_properties = D2D1::BitmapProperties1(D2D1_BITMAP_OPTIONS_NONE,
D2D1::PixelFormat(description.Format, D2D1_ALPHA_MODE_IGNORE));
Microsoft::WRL::ComPtr<ID2D1Bitmap1> input;
Check(drawing_->CreateBitmapFromDxgiSurface(surface.Get(), &input_properties, &input),
"capture-hdr-bitmap");
Check(tone_map_->SetValue(D2D1_HDRTONEMAP_PROP_INPUT_MAX_LUMINANCE, color.peak_nits),
"capture-hdr-peak");
Check(tone_map_->SetValue(D2D1_HDRTONEMAP_PROP_OUTPUT_MAX_LUMINANCE, color.white_nits),
"capture-sdr-peak");
Check(white_level_->SetValue(D2D1_WHITELEVELADJUSTMENT_PROP_OUTPUT_WHITE_LEVEL, color.white_nits),
"capture-sdr-white");
description.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
Microsoft::WRL::ComPtr<ID3D11Texture2D> output;
Check(device_->CreateTexture2D(&description, nullptr, &output), "capture-sdr-texture");
surface.Reset();
Check(output.As(&surface), "capture-sdr-surface");
const auto output_properties = D2D1::BitmapProperties1(
D2D1_BITMAP_OPTIONS_TARGET | D2D1_BITMAP_OPTIONS_CANNOT_DRAW,
D2D1::PixelFormat(description.Format, D2D1_ALPHA_MODE_IGNORE));
Microsoft::WRL::ComPtr<ID2D1Bitmap1> target;
Check(drawing_->CreateBitmapFromDxgiSurface(surface.Get(), &output_properties, &target),
"capture-sdr-bitmap");
tone_map_->SetInput(0, input.Get());
drawing_->SetTarget(target.Get());
Microsoft::WRL::ComPtr<ID2D1Image> image;
srgb_->GetOutput(&image);
drawing_->BeginDraw();
drawing_->DrawImage(image.Get(), nullptr, nullptr,
D2D1_INTERPOLATION_MODE_NEAREST_NEIGHBOR, D2D1_COMPOSITE_MODE_SOURCE_COPY);
const HRESULT result = drawing_->EndDraw();
drawing_->SetTarget(nullptr);
tone_map_->SetInput(0, nullptr);
Check(result, "capture-hdr-tone-map");
return output;
}
private:
void Initialize() {
Microsoft::WRL::ComPtr<ID2D1Factory1> factory;
Check(D2D1CreateFactory(D2D1_FACTORY_TYPE_SINGLE_THREADED, IID_PPV_ARGS(&factory)),
"capture-d2d-factory");
Microsoft::WRL::ComPtr<IDXGIDevice> dxgi;
Check(device_.As(&dxgi), "capture-d2d-dxgi");
Microsoft::WRL::ComPtr<ID2D1Device> device;
Check(factory->CreateDevice(dxgi.Get(), &device), "capture-d2d-device");
Check(device->CreateDeviceContext(D2D1_DEVICE_CONTEXT_OPTIONS_NONE, &drawing_),
"capture-d2d-context");
const D2D1_RENDERING_CONTROLS controls{D2D1_BUFFER_PRECISION_16BPC_FLOAT, {0, 0}};
drawing_->SetRenderingControls(&controls);
Check(drawing_->CreateEffect(CLSID_D2D1HdrToneMap, &tone_map_), "capture-tone-map-effect");
Check(tone_map_->SetValue(D2D1_HDRTONEMAP_PROP_DISPLAY_MODE, D2D1_HDRTONEMAP_DISPLAY_MODE_SDR),
"capture-tone-map-mode");
Check(drawing_->CreateEffect(CLSID_D2D1WhiteLevelAdjustment, &white_level_), "capture-white-effect");
Check(white_level_->SetValue(D2D1_WHITELEVELADJUSTMENT_PROP_INPUT_WHITE_LEVEL, 80.0f),
"capture-input-white");
white_level_->SetInputEffect(0, tone_map_.Get());
Check(drawing_->CreateEffect(CLSID_D2D1ColorManagement, &srgb_), "capture-srgb-effect");
Microsoft::WRL::ComPtr<ID2D1ColorContext> linear, srgb;
Check(drawing_->CreateColorContext(D2D1_COLOR_SPACE_SCRGB, nullptr, 0, &linear), "capture-linear-color");
Check(drawing_->CreateColorContext(D2D1_COLOR_SPACE_SRGB, nullptr, 0, &srgb), "capture-srgb-color");
Check(srgb_->SetValue(D2D1_COLORMANAGEMENT_PROP_SOURCE_COLOR_CONTEXT, linear.Get()), "capture-input-color");
Check(srgb_->SetValue(D2D1_COLORMANAGEMENT_PROP_DESTINATION_COLOR_CONTEXT, srgb.Get()), "capture-output-color");
Check(srgb_->SetValue(D2D1_COLORMANAGEMENT_PROP_QUALITY, D2D1_COLORMANAGEMENT_QUALITY_BEST),
"capture-color-quality");
srgb_->SetInputEffect(0, white_level_.Get());
}
Microsoft::WRL::ComPtr<ID3D11Device> device_;
Microsoft::WRL::ComPtr<ID3D11DeviceContext> context_;
Microsoft::WRL::ComPtr<ID2D1DeviceContext> drawing_;
Microsoft::WRL::ComPtr<ID2D1Effect> tone_map_, white_level_, srgb_;
};
} // namespace piik::capture::windows
native/capture/windows/capture_target.cpp
+12 -2
View File
@@ -19,6 +19,7 @@
#include "capture_target.h"
#include "capture_border.h"
#include "capture_geometry.h"
#include "capture_sdr.h"
#include <algorithm>
#include <array>
@@ -497,8 +498,11 @@ HRESULT CaptureWithWgc(TargetKind kind, UINT64 source_id,
if (size.Width <= 0 || size.Height <= 0 || frame_ready == nullptr) {
result = HRESULT_FROM_WIN32(ERROR_INVALID_DATA);
} else {
windows::CaptureDisplayColor display_color(kind, source_id);
const auto color = display_color.Resolve();
windows::CaptureSdrConverter sdr(device.Get());
pool = Direct3D11CaptureFramePool::CreateFreeThreaded(
capture_device, DirectXPixelFormat::B8G8R8A8UIntNormalized, 1,
capture_device, static_cast<DirectXPixelFormat>(color.Format()), 1,
size);
session = pool.CreateCaptureSession(item);
frame_token = pool.FrameArrived(
@@ -530,6 +534,7 @@ HRESULT CaptureWithWgc(TargetKind kind, UINT64 source_id,
::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>();
ComPtr<ID3D11Texture2D> texture;
result = access->GetInterface(IID_PPV_ARGS(&texture));
if (SUCCEEDED(result)) texture = sdr.Convert(texture.Get(), color);
D3D11_TEXTURE2D_DESC description{};
if (SUCCEEDED(result)) texture->GetDesc(&description);
if (SUCCEEDED(result) &&
@@ -1016,9 +1021,14 @@ HRESULT ValidateProcessTarget(DWORD pid, UINT64 expected_creation_time) {
}
HANDLE process = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid);
if (process == nullptr) return HRESULT_FROM_WIN32(GetLastError());
const HRESULT result = ValidateProcessTarget(process, expected_creation_time);
CloseHandle(process);
return result;
}
HRESULT ValidateProcessTarget(HANDLE process, UINT64 expected_creation_time) {
UINT64 actual_creation_time = 0;
HRESULT result = ReadProcessCreationTime(process, &actual_creation_time);
CloseHandle(process);
if (SUCCEEDED(result) && actual_creation_time != expected_creation_time) {
result = HRESULT_FROM_WIN32(ERROR_INVALID_DATA);
}
native/capture/windows/capture_target.h
+1
View File
@@ -13,6 +13,7 @@ int WriteMicrophoneList();
HRESULT WriteSourcePreview(TargetKind kind, UINT64 source_id, DWORD pid,
UINT64 expected_creation_time);
HRESULT ValidateProcessTarget(DWORD pid, UINT64 expected_creation_time);
HRESULT ValidateProcessTarget(HANDLE process, UINT64 expected_creation_time);
HRESULT ValidateWindowTarget(UINT64 source_id, DWORD pid,
UINT64 expected_creation_time);
HRESULT ValidateDisplayTarget(UINT64 source_id);
native/capture/windows/capture_target.test.cpp
+2
View File
@@ -15,6 +15,8 @@ int main() {
assert(GetProcessTimes(GetCurrentProcess(), &created, &exited, &kernel, &user));
const UINT64 creation_time = (static_cast<UINT64>(created.dwHighDateTime) << 32) |
created.dwLowDateTime;
assert(SUCCEEDED(piik::capture::ValidateProcessTarget(GetCurrentProcess(), creation_time)));
assert(FAILED(piik::capture::ValidateProcessTarget(GetCurrentProcess(), creation_time + 1)));
HANDLE previous_output = GetStdHandle(STD_OUTPUT_HANDLE);
HANDLE sink = CreateFileW(L"NUL", GENERIC_WRITE, FILE_SHARE_WRITE, nullptr,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr);
native/capture/windows/frame_converter.probe.cpp
+218
View File
@@ -0,0 +1,218 @@
// Physical D3D11 check; opt in with build.ps1 -CheckColor. No desktop capture.
#define wmain capture_main
#include "main.cpp"
#undef wmain
#include <DirectXPackedVector.h>
namespace {
constexpr std::array<UINT32, 8> kRgb{
0xff000000, 0xff202020, 0xff808080, 0xffebebeb,
0xffffffff, 0xffff0000, 0xff00ff00, 0xff0000ff};
constexpr std::array<std::array<int, 3>, 8> kYuv{{
{16, 128, 128}, {44, 128, 128}, {126, 128, 128}, {218, 128, 128},
{235, 128, 128}, {82, 90, 240}, {145, 54, 34}, {41, 240, 110}}};
constexpr std::array<std::array<int, 3>, 8> kYuv709{{
{16, 128, 128}, {44, 128, 128}, {126, 128, 128}, {218, 128, 128},
{235, 128, 128}, {63, 102, 240}, {173, 42, 26}, {32, 240, 118}}};
constexpr std::array<std::array<int, 3>, 8> kYuvFull{{
{0, 128, 128}, {32, 128, 128}, {128, 128, 128}, {235, 128, 128},
{255, 128, 128}, {76, 85, 255}, {150, 44, 21}, {29, 255, 107}}};
constexpr std::array<std::array<int, 3>, 8> kYuv709Full{{
{0, 128, 128}, {32, 128, 128}, {128, 128, 128}, {235, 128, 128},
{255, 128, 128}, {54, 99, 255}, {182, 30, 12}, {18, 255, 116}}};
ComPtr<ID3D11Texture2D> ColorBars(const DeviceContext& device, UINT width,
UINT height, DXGI_COLOR_SPACE_TYPE color) {
const bool rgb = color == kDesktopColor;
const auto& yuv = color == DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P709 ? kYuv709
: color == DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P709 ? kYuv709Full
: color == DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P601 ? kYuvFull : kYuv;
std::vector<UINT8> pixels(static_cast<size_t>(width) * height * (rgb ? 4 : 2));
for (UINT row = 0; row < height; ++row) {
for (UINT column = 0; column < width; ++column) {
const size_t bar = column * kRgb.size() / width;
const size_t index = static_cast<size_t>(row) * width + column;
if (rgb) {
for (UINT channel = 0; channel < 4; ++channel)
pixels[index * 4 + channel] = static_cast<UINT8>(kRgb[bar] >> (channel * 8));
} else {
pixels[index] = static_cast<UINT8>(yuv[bar][0]);
if (row < height / 2)
pixels[static_cast<size_t>(width) * height + index] =
static_cast<UINT8>(yuv[bar][1 + column % 2]);
}
}
}
D3D11_TEXTURE2D_DESC description{};
description.Width = width;
description.Height = height;
description.MipLevels = description.ArraySize = description.SampleDesc.Count = 1;
description.Format = rgb ? DXGI_FORMAT_B8G8R8A8_UNORM : DXGI_FORMAT_NV12;
description.BindFlags = D3D11_BIND_RENDER_TARGET | D3D11_BIND_SHADER_RESOURCE;
D3D11_SUBRESOURCE_DATA data{};
data.pSysMem = pixels.data();
data.SysMemPitch = width * (rgb ? 4 : 1);
ComPtr<ID3D11Texture2D> texture;
Check(device.device->CreateTexture2D(&description, &data, &texture), "color-source");
return texture;
}
void CheckColors(const DeviceContext& device, ID3D11Texture2D* texture) {
D3D11_TEXTURE2D_DESC description{};
texture->GetDesc(&description);
description.Usage = D3D11_USAGE_STAGING;
description.BindFlags = 0;
description.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
ComPtr<ID3D11Texture2D> staging;
Check(device.device->CreateTexture2D(&description, nullptr, &staging), "color-readback");
device.context->CopyResource(staging.Get(), texture);
D3D11_MAPPED_SUBRESOURCE mapped{};
Check(device.context->Map(staging.Get(), 0, D3D11_MAP_READ, 0, &mapped), "color-map");
const auto* bytes = static_cast<const UINT8*>(mapped.pData);
bool correct = true;
for (size_t bar = 0; bar < kYuv.size(); ++bar) {
const auto x = description.Width * (2 * bar + 1) / (2 * kYuv.size());
const size_t y = description.Height / 2;
const size_t uv = (description.Height + y / 2) * mapped.RowPitch + (x & ~size_t{1});
const std::array<int, 3> actual{bytes[y * mapped.RowPitch + x], bytes[uv], bytes[uv + 1]};
for (size_t channel = 0; channel < 3; ++channel) {
if (std::abs(actual[channel] - kYuv[bar][channel]) > 2) {
std::cerr << "bar=" << bar << " channel=" << channel
<< " expected=" << kYuv[bar][channel] << " actual=" << actual[channel] << '\n';
correct = false;
}
}
}
device.context->Unmap(staging.Get(), 0);
if (!correct) throw std::runtime_error("SDR range or matrix changed");
}
ComPtr<ID3D11Texture2D> LinearBars(const DeviceContext& device, UINT width,
UINT height, const std::array<float, 8>& nits) {
std::vector<UINT16> pixels(static_cast<size_t>(width) * height * 4);
for (size_t pixel = 0; pixel < static_cast<size_t>(width) * height; ++pixel) {
const auto value = DirectX::PackedVector::XMConvertFloatToHalf(nits[(pixel % width) * 8 / width] / 80.0f);
for (size_t channel = 0; channel < 3; ++channel) pixels[pixel * 4 + channel] = value;
pixels[pixel * 4 + 3] = DirectX::PackedVector::XMConvertFloatToHalf(1.0f);
}
D3D11_TEXTURE2D_DESC description{};
description.Width = width;
description.Height = height;
description.MipLevels = description.ArraySize = description.SampleDesc.Count = 1;
description.Format = DXGI_FORMAT_R16G16B16A16_FLOAT;
description.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET;
D3D11_SUBRESOURCE_DATA data{};
data.pSysMem = pixels.data();
data.SysMemPitch = width * 8;
ComPtr<ID3D11Texture2D> texture;
Check(device.device->CreateTexture2D(&description, &data, &texture), "hdr-source");
return texture;
}
std::array<int, 8> ReadGrayBars(const DeviceContext& device, ID3D11Texture2D* texture) {
D3D11_TEXTURE2D_DESC description{};
texture->GetDesc(&description);
description.Usage = D3D11_USAGE_STAGING;
description.BindFlags = 0;
description.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
ComPtr<ID3D11Texture2D> staging;
Check(device.device->CreateTexture2D(&description, nullptr, &staging), "hdr-readback");
device.context->CopyResource(staging.Get(), texture);
D3D11_MAPPED_SUBRESOURCE mapped{};
Check(device.context->Map(staging.Get(), 0, D3D11_MAP_READ, 0, &mapped), "hdr-map");
const auto* bytes = static_cast<const UINT8*>(mapped.pData);
std::array<int, 8> result{};
bool neutral = true;
for (size_t bar = 0; bar < result.size(); ++bar) {
const size_t pixel = description.Height / 2 * mapped.RowPitch +
description.Width * (2 * bar + 1) / 16 * 4;
result[bar] = bytes[pixel];
neutral = neutral && std::abs(result[bar] - bytes[pixel + 1]) <= 1 &&
std::abs(result[bar] - bytes[pixel + 2]) <= 1;
}
device.context->Unmap(staging.Get(), 0);
if (!neutral) throw std::runtime_error("HDR gray acquired a color cast");
return result;
}
void CheckHdr(const DeviceContext& device) {
CaptureSdrConverter capture(device.device.Get());
VideoProfile profile;
profile.width = 256;
profile.height = 128;
FrameConverter converter(device.device.Get(), profile);
for (const UINT width : {256u, 512u, 256u}) {
// SDR must retain the old pixels after an HDR frame, with the same owner.
auto sdr = ColorBars(device, width, width / 2, kDesktopColor);
auto owned = capture.Convert(sdr.Get(), {});
CheckColors(device, converter.Convert(owned.Get(), width, width / 2,
{static_cast<LONG>(width), static_cast<LONG>(width / 2)}, kDesktopColor).Get());
auto hdr = LinearBars(device, width, width / 2, {0, 5, 20, 80, 200, 400, 700, 1000});
for (const float white : {80.0f, 240.0f, 80.0f}) {
auto mapped = capture.Convert(hdr.Get(), {true, white, 1000.0f});
const auto levels = ReadGrayBars(device, mapped.Get());
std::cout << "HDR white=" << white << " levels=";
for (const auto level : levels) std::cout << level << ',';
std::cout << '\n';
if (levels.front() > 1 || levels.back() < 245)
throw std::runtime_error("HDR black/peak mapping is incorrect");
for (size_t index = 1; index < levels.size(); ++index)
if (levels[index] <= levels[index - 1])
throw std::runtime_error("HDR luminance detail was clipped before tone mapping");
auto next = capture.Convert(sdr.Get(), {});
if (ReadGrayBars(device, mapped.Get()) != levels)
throw std::runtime_error("Capture replacement modified a retained frame");
CheckColors(device, converter.Convert(next.Get(), width, width / 2,
{static_cast<LONG>(width), static_cast<LONG>(width / 2)}, kDesktopColor).Get());
}
}
}
} // namespace
int main() {
try {
Runtime runtime;
const auto adapters = EnumerateAdapters();
if (adapters.empty()) throw std::runtime_error("A physical D3D11 adapter is required");
for (const auto& adapter : adapters) {
auto device = CreateDevice(adapter);
for (const UINT output_width : {256u, 128u}) {
VideoProfile profile;
profile.width = output_width;
profile.height = output_width / 2;
FrameConverter converter(device.device.Get(), profile);
// Reuse the same converter across RGB/NV12 and size replacements.
for (const UINT input_width : {256u, 512u, 256u}) {
for (const auto color : {kDesktopColor, kSdrVideoColor,
DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P709,
DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P601,
DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P709, kDesktopColor}) {
const UINT input_height = input_width / 2;
auto source = ColorBars(device, input_width, input_height, color);
auto output = converter.Convert(source.Get(), input_width, input_height,
{static_cast<LONG>(input_width), static_cast<LONG>(input_height)},
color);
try {
CheckColors(device, output.Get());
} catch (...) {
std::cerr << "adapter=" << adapter.index << " color=" << color
<< " input-width=" << input_width << " output-width=" << output_width << '\n';
throw;
}
}
}
}
std::cout << "SDR conversion passed: " << NarrowAscii(adapter.description.Description) << '\n';
CheckHdr(device);
std::cout << "HDR conversion passed: " << NarrowAscii(adapter.description.Description) << '\n';
}
return 0;
} catch (const std::exception& error) {
std::cerr << error.what() << '\n';
if (const auto* failure = dynamic_cast<const GateFailure*>(&error))
std::cerr << "stage=" << failure->stage() << " result=" << std::hex << failure->result() << '\n';
return 1;
}
}
native/capture/windows/h264_decoder.h
+7
View File
@@ -1,6 +1,7 @@
#pragma once
#include "capture_error.h"
#include "capture_color.h"
#include "capture_input.h"
@@ -56,6 +57,12 @@ class H264Decoder final {
H264Decoder(const H264Decoder&) = delete;
H264Decoder& operator=(const H264Decoder&) = delete;
DXGI_COLOR_SPACE_TYPE ColorSpace() const {
Microsoft::WRL::ComPtr<IMFMediaType> type;
Check(decoder_->GetOutputCurrentType(0, &type), "current-output-type");
return windows::DecodedVideoColor(type.Get());
}
void Decode(const InputEnvelope& input, const Consumer& consume) {
if (input.kind != 2 || input.data.empty() || input.data.size() > 4 * 1024 * 1024)
throw std::runtime_error("Invalid H264 decoder input");
native/capture/windows/h264_encoder.cpp
+3
View File
@@ -1,6 +1,7 @@
#include "h264_encoder.h"
#include "capture_output.h"
#include "capture_color.h"
#include <d3d10_1.h>
#include <evr.h>
@@ -327,6 +328,7 @@ ComPtr<IMFMediaType> CreateOutputType(
"output-type-major");
Check(type->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_H264),
"output-type-subtype");
SetSdrVideoColor(type.Get());
Check(type->SetUINT32(MF_MT_AVG_BITRATE, profile.bit_rate),
"output-type-bitrate");
Check(MFSetAttributeSize(type.Get(), MF_MT_FRAME_SIZE, profile.width,
@@ -354,6 +356,7 @@ ComPtr<IMFMediaType> CreateInputType(
"input-type-major");
Check(type->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_NV12),
"input-type-subtype");
SetSdrVideoColor(type.Get());
Check(MFSetAttributeSize(type.Get(), MF_MT_FRAME_SIZE, profile.width,
profile.height),
"input-type-size");
native/capture/windows/main.cpp
+60 -25
View File
@@ -3,7 +3,7 @@
#include <codecapi.h>
#include <d3d10_1.h>
#include <d3d11.h>
#include <d3d11_1.h>
#include <dxgi1_2.h>
#include <evr.h>
#include <fcntl.h>
@@ -32,6 +32,8 @@
#include "capture_target.h"
#include "capture_border.h"
#include "capture_geometry.h"
#include "capture_color.h"
#include "capture_sdr.h"
#include "process_audio.h"
#include "h264_encoder.h"
#ifndef PIIK_H264_FIXTURE
@@ -663,8 +665,9 @@ class FrameConverter final {
public:
explicit FrameConverter(
ID3D11Device* device,
VideoProfile profile = kDefaultVideoProfile)
: device_(device), profile_(profile) {
VideoProfile profile = kDefaultVideoProfile,
DXGI_COLOR_SPACE_TYPE output_color = kSdrVideoColor)
: device_(device), profile_(profile), output_color_(output_color) {
Check(device_->QueryInterface(IID_PPV_ARGS(&video_device_)),
"video-processor-device");
ComPtr<ID3D11DeviceContext> context;
@@ -673,13 +676,30 @@ class FrameConverter final {
}
ComPtr<ID3D11Texture2D> Convert(ID3D11Texture2D* source, UINT32 width,
UINT32 height, SIZE presentation) {
UINT32 height, SIZE presentation,
DXGI_COLOR_SPACE_TYPE input_color) {
if (source == nullptr || width == 0 || height == 0 ||
width > 16'384 || height > 16'384) {
Fail("capture-size", "captured window dimensions are invalid");
}
if (!enumerator_ || width != input_width_ || height != input_height_) {
Configure(width, height);
ComPtr<ID3D11Texture2D> normalized;
if (output_color_ == kSdrVideoColor && input_color != kDesktopColor &&
input_color != kSdrVideoColor) {
// YUV-to-YUV matrix/range changes are not reliable across tested drivers,
// even when reported as supported. Normalize through RGB on the GPU;
// ordinary RGB capture and canonical NV12 scaling stay single-pass.
if (!rgb_converter_)
rgb_converter_ = std::make_unique<FrameConverter>(device_.Get(), profile_, kDesktopColor);
normalized = rgb_converter_->Convert(source, width, height, presentation, input_color);
source = normalized.Get();
width = profile_.width;
height = profile_.height;
presentation = {static_cast<LONG>(width), static_cast<LONG>(height)};
input_color = kDesktopColor;
}
if (!enumerator_ || width != input_width_ || height != input_height_ ||
input_color != input_color_) {
Configure(width, height, input_color);
}
D3D11_TEXTURE2D_DESC output_description = {};
@@ -687,7 +707,8 @@ class FrameConverter final {
output_description.Height = profile_.height;
output_description.MipLevels = 1;
output_description.ArraySize = 1;
output_description.Format = DXGI_FORMAT_NV12;
output_description.Format = output_color_ == kDesktopColor
? DXGI_FORMAT_B8G8R8A8_UNORM : DXGI_FORMAT_NV12;
output_description.SampleDesc.Count = 1;
output_description.Usage = D3D11_USAGE_DEFAULT;
output_description.BindFlags = D3D11_BIND_RENDER_TARGET;
@@ -745,7 +766,7 @@ class FrameConverter final {
}
private:
void Configure(UINT32 width, UINT32 height) {
void Configure(UINT32 width, UINT32 height, DXGI_COLOR_SPACE_TYPE input_color) {
D3D11_VIDEO_PROCESSOR_CONTENT_DESC description = {};
description.InputFrameFormat = D3D11_VIDEO_FRAME_FORMAT_PROGRESSIVE;
description.InputFrameRate = {profile_.frame_rate, 1};
@@ -762,20 +783,27 @@ class FrameConverter final {
ComPtr<ID3D11VideoProcessor> processor;
Check(video_device_->CreateVideoProcessor(enumerator.Get(), 0, &processor),
"video-processor-create");
// The incoming frame owns its range/matrix. Never infer these from size.
video_context_->VideoProcessorSetStreamColorSpace1(processor.Get(), 0, input_color);
video_context_->VideoProcessorSetOutputColorSpace1(processor.Get(), output_color_);
enumerator_ = std::move(enumerator);
processor_ = std::move(processor);
input_width_ = width;
input_height_ = height;
input_color_ = input_color;
}
ComPtr<ID3D11Device> device_;
VideoProfile profile_;
const DXGI_COLOR_SPACE_TYPE output_color_;
std::unique_ptr<FrameConverter> rgb_converter_;
ComPtr<ID3D11VideoDevice> video_device_;
ComPtr<ID3D11VideoContext> video_context_;
ComPtr<ID3D11VideoContext1> video_context_;
ComPtr<ID3D11VideoProcessorEnumerator> enumerator_;
ComPtr<ID3D11VideoProcessor> processor_;
UINT32 input_width_ = 0;
UINT32 input_height_ = 0;
DXGI_COLOR_SPACE_TYPE input_color_ = kDesktopColor;
};
winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice
@@ -933,6 +961,7 @@ struct ProductArguments final {
VideoProfile profile;
std::vector<VideoProfile> outputs;
bool show_capture_border = false;
bool exclude_audio = false;
};
DegradationPreference ParseDegradationPreference(const wchar_t* value) {
@@ -1020,7 +1049,8 @@ ProductArguments ParseProductArguments(int count, wchar_t** values) {
}
if (count == 5 && std::wstring(values[1]) == L"--capture-audio") {
arguments.mode = ProductArguments::Mode::audio;
arguments.target_kind = ParseTargetKind(values[2]);
arguments.exclude_audio = std::wstring(values[2]) == L"exclude";
arguments.target_kind = arguments.exclude_audio ? piik::capture::TargetKind::window : ParseTargetKind(values[2]);
} else if (count >= 16 && count <= 11 + 5 * piik::capture::kMaxOutputs && (count - 11) % 5 == 0 &&
std::wstring(values[1]) == L"--encoded-video" &&
std::wstring(values[2]) == L"--codec" &&
@@ -1126,7 +1156,7 @@ HRESULT RunAudioCapture(const ProductArguments& arguments) {
stop.get(), should_stop, ready, pcm);
}
return piik::capture::CaptureProcessAudio(
arguments.pid, arguments.creation_time, stop.get(),
arguments.pid, arguments.creation_time, arguments.exclude_audio, stop.get(),
should_stop,
ready, pcm);
}
@@ -1153,6 +1183,7 @@ UINT32 WindowsBuild() {
void WriteCapabilityProbe() {
constexpr UINT32 kCreateForWindowMinimumBuild = 18'362;
constexpr UINT32 kProcessLoopbackMinimumBuild = 19'041;
constexpr UINT32 kProcessExclusionMinimumBuild = 20'348;
const UINT32 build = WindowsBuild();
bool window_capture = false;
if (build >= kCreateForWindowMinimumBuild) {
@@ -1167,6 +1198,8 @@ void WriteCapabilityProbe() {
build >= kProcessLoopbackMinimumBuild &&
piik::capture::ProcessAudioAvailable();
const bool system_audio = piik::capture::SystemAudioAvailable();
const bool process_audio_exclusion = build >= kProcessExclusionMinimumBuild &&
piik::capture::ProcessAudioAvailable(true);
std::vector<Adapter> adapters = EnumerateAdapters();
std::ostringstream output;
@@ -1178,6 +1211,7 @@ void WriteCapabilityProbe() {
<< ",\"microphone\":true,\"softwareVP8\":true"
<< ",\"processAudio\":"
<< (process_audio ? "true" : "false")
<< ",\"processAudioExclusion\":" << (process_audio_exclusion ? "true" : "false")
<< ",\"systemAudio\":" << (system_audio ? "true" : "false")
<< ",\"adapters\":[";
for (size_t adapter_index = 0; adapter_index < adapters.size();
@@ -1361,6 +1395,7 @@ VideoEncoderSelection SelectVideoEncoder(
struct CaptureInput final {
ComPtr<ID3D11Texture2D> texture;
DXGI_COLOR_SPACE_TYPE color_space = kDesktopColor;
UINT32 width = 0;
UINT32 height = 0;
SIZE presentation{};
@@ -1448,7 +1483,7 @@ class OutputWorker final {
converted_height = height;
}
return converter->Convert(input->texture.Get(), input->width,
input->height, input->presentation);
input->height, input->presentation, input->color_space);
}, profile_.width, profile_.height, input->timestamp, work.recovery, work.bitrate);
} catch (...) {
// Device loss is current shared evidence; an old codec failure only
@@ -1703,12 +1738,13 @@ void RunEncodedVideo(ProductArguments arguments) {
bool began = false;
UINT64 last_timestamp = 0;
auto submit = [&](ComPtr<ID3D11Texture2D> texture, UINT32 width, UINT32 height,
UINT64 timestamp, UINT64 duration) {
UINT64 timestamp, UINT64 duration, DXGI_COLOR_SPACE_TYPE color) {
if ((began && timestamp <= last_timestamp) || width < arguments.profile.width || height < arguments.profile.height) {
Fail("encoded-input-output", "decoded frame changed the source timeline or output bounds");
}
auto input = std::make_shared<CaptureInput>();
input->texture = std::move(texture);
input->color_space = color;
input->width = width;
input->height = height;
input->presentation = {static_cast<LONG>(width), static_cast<LONG>(height)};
@@ -1735,12 +1771,12 @@ void RunEncodedVideo(ProductArguments arguments) {
Check(sample->GetSampleDuration(&duration), "decoded-duration");
if (timestamp < 0 || duration <= 0) Fail("decoded-timing", "decoder did not preserve source timing");
submit(OwnDecodedTexture(device, sample, width, height, stride), width, height,
static_cast<UINT64>(timestamp), static_cast<UINT64>(duration));
static_cast<UINT64>(timestamp), static_cast<UINT64>(duration), h264->ColorSpace());
});
} else {
const auto decoded = vp8->Decode(input.data);
if (!decoded.nv12.empty()) submit(UploadDecodedNV12(device, decoded.nv12.data(), decoded.width, decoded.height),
decoded.width, decoded.height, input.timestamp, input.duration);
decoded.width, decoded.height, input.timestamp, input.duration, kSdrVideoColor);
}
}
}
@@ -1801,8 +1837,11 @@ void RunVideoCapture(ProductArguments arguments) {
if (initial_size.Width <= 0 || initial_size.Height <= 0) {
Fail("capture-size", "selected window has no capturable content");
}
CaptureDisplayColor display_color(arguments.target_kind, arguments.source_id);
CaptureSdrConverter sdr(device.device.Get());
auto pool_format = display_color.Resolve().Format();
Direct3D11CaptureFramePool pool = Direct3D11CaptureFramePool::CreateFreeThreaded(
capture_device, DirectXPixelFormat::B8G8R8A8UIntNormalized, 2,
capture_device, static_cast<DirectXPixelFormat>(pool_format), 2,
initial_size);
GraphicsCaptureSession capture_session = pool.CreateCaptureSession(item);
EnableFastCaptureUpdates(capture_session);
@@ -1964,15 +2003,17 @@ void RunVideoCapture(ProductArguments arguments) {
}
auto content_size = latest.ContentSize();
if (content_size.Width <= 0 || content_size.Height <= 0) continue;
const auto color = display_color.Resolve();
if (content_size.Width != pool_size.Width ||
content_size.Height != pool_size.Height) {
content_size.Height != pool_size.Height || color.Format() != pool_format) {
latest.Close();
latest = nullptr;
latest_input.reset();
pool.Recreate(capture_device,
DirectXPixelFormat::B8G8R8A8UIntNormalized, 2,
static_cast<DirectXPixelFormat>(color.Format()), 2,
content_size);
pool_size = content_size;
pool_format = color.Format();
continue;
}
ComPtr<ID3D11Texture2D> source = CaptureTexture(latest);
@@ -1983,13 +2024,7 @@ void RunVideoCapture(ProductArguments arguments) {
UINT32 content_height = std::min<UINT32>(
source_description.Height, static_cast<UINT32>(content_size.Height));
auto input = std::make_shared<CaptureInput>();
auto owned_description = source_description;
owned_description.Usage = D3D11_USAGE_DEFAULT;
owned_description.CPUAccessFlags = 0;
owned_description.MiscFlags = 0;
owned_description.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET;
Check(device.device->CreateTexture2D(&owned_description, nullptr, &input->texture), "capture-owned-input");
device.context->CopyResource(input->texture.Get(), source.Get());
input->texture = sdr.Convert(source.Get(), color);
input->width = content_width;
input->height = content_height;
input->timestamp = timestamp;
native/capture/windows/process_audio.cpp
+29 -24
View File
@@ -31,7 +31,8 @@ namespace {
class ActivationHandler final : public RuntimeClass<
RuntimeClassFlags<ClassicCom>, FtmBase, IActivateAudioInterfaceCompletionHandler> {
public:
explicit ActivationHandler(HANDLE completed) : completed_(completed) {}
ActivationHandler() : completed_(CreateEventW(nullptr, FALSE, FALSE, nullptr)) {}
~ActivationHandler() { if (completed_) CloseHandle(completed_); }
STDMETHODIMP ActivateCompleted(IActivateAudioInterfaceAsyncOperation* operation) override {
ComPtr<IUnknown> activated;
@@ -45,6 +46,7 @@ class ActivationHandler final : public RuntimeClass<
HRESULT Result() const { return result_; }
ComPtr<IAudioClient> Client() const { return client_; }
HANDLE Completed() const { return completed_; }
private:
HANDLE completed_ = nullptr;
@@ -52,26 +54,29 @@ class ActivationHandler final : public RuntimeClass<
ComPtr<IAudioClient> client_;
};
HRESULT ActivateProcessLoopback(DWORD pid, HANDLE completed,
HRESULT ActivateProcessLoopback(DWORD pid, bool exclude,
ComPtr<IAudioClient>* client) {
AUDIOCLIENT_ACTIVATION_PARAMS parameters{};
parameters.ActivationType = AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK;
parameters.ProcessLoopbackParams.TargetProcessId = pid;
parameters.ProcessLoopbackParams.ProcessLoopbackMode =
PROCESS_LOOPBACK_MODE_INCLUDE_TARGET_PROCESS_TREE;
exclude ? PROCESS_LOOPBACK_MODE_EXCLUDE_TARGET_PROCESS_TREE
: PROCESS_LOOPBACK_MODE_INCLUDE_TARGET_PROCESS_TREE;
PROPVARIANT variant{};
variant.vt = VT_BLOB;
variant.blob.cbSize = sizeof(parameters);
variant.blob.pBlobData = reinterpret_cast<BYTE*>(&parameters);
auto handler = Make<ActivationHandler>(completed);
// The completion can outlive a timed-out caller; it owns its event handle.
auto handler = Make<ActivationHandler>();
if (!handler) return E_OUTOFMEMORY;
if (!handler->Completed()) return HRESULT_FROM_WIN32(GetLastError());
ComPtr<IActivateAudioInterfaceAsyncOperation> operation;
HRESULT result = ActivateAudioInterfaceAsync(VIRTUAL_AUDIO_DEVICE_PROCESS_LOOPBACK,
__uuidof(IAudioClient), &variant,
handler.Get(), &operation);
if (FAILED(result)) return result;
if (WaitForSingleObject(completed, 10'000) != WAIT_OBJECT_0) return HRESULT_FROM_WIN32(WAIT_TIMEOUT);
if (WaitForSingleObject(handler->Completed(), 10'000) != WAIT_OBJECT_0) return HRESULT_FROM_WIN32(WAIT_TIMEOUT);
result = handler->Result();
if (SUCCEEDED(result)) *client = handler->Client();
return result;
@@ -114,6 +119,14 @@ HRESULT CaptureAudioFrames(ComPtr<IAudioClient> client, HANDLE process, bool loo
return result;
}
HRESULT result = S_OK;
// Bind process audio to the selected lifetime, including while draining PCM.
// Exclusion must never become unfiltered system capture after its target dies.
const auto processAlive = [&]() -> HRESULT {
if (!process) return S_OK;
const DWORD wait = WaitForSingleObject(process, 0);
if (wait == WAIT_TIMEOUT) return S_OK;
return HRESULT_FROM_WIN32(wait == WAIT_OBJECT_0 ? ERROR_PROCESS_ABORTED : GetLastError());
};
ComPtr<IAudioCaptureClient> capture;
WAVEFORMATEX format{};
format.wFormatTag = WAVE_FORMAT_PCM;
@@ -131,6 +144,7 @@ HRESULT CaptureAudioFrames(ComPtr<IAudioClient> client, HANDLE process, bool loo
if (SUCCEEDED(result)) result = client->GetService(IID_PPV_ARGS(&capture));
if (SUCCEEDED(result)) result = client->SetEventHandle(sampleReady);
if (SUCCEEDED(result)) result = client->Start();
if (SUCCEEDED(result)) result = processAlive();
if (SUCCEEDED(result)) result = ready_writer();
if (SUCCEEDED(result)) {
std::array<BYTE, kAudioBytesPerChunk> silence{};
@@ -171,6 +185,7 @@ HRESULT CaptureAudioFrames(ComPtr<IAudioClient> client, HANDLE process, bool loo
}
UINT32 frames = 0;
while (SUCCEEDED(result = capture->GetNextPacketSize(&frames)) && frames > 0) {
if (FAILED(result = processAlive())) break;
BYTE* data = nullptr;
DWORD flags = 0;
UINT64 devicePosition = 0;
@@ -198,6 +213,7 @@ HRESULT CaptureAudioFrames(ComPtr<IAudioClient> client, HANDLE process, bool loo
}
capture->ReleaseBuffer(frames);
while (pending.size() - consumed >= kAudioBytesPerChunk) {
if (FAILED(result = processAlive())) break;
result = writer(nextTimestamp, pending.data() + consumed,
kAudioBytesPerChunk);
if (FAILED(result)) break;
@@ -218,13 +234,10 @@ HRESULT CaptureAudioFrames(ComPtr<IAudioClient> client, HANDLE process, bool loo
} // namespace
bool ProcessAudioAvailable() {
HANDLE completed = CreateEventW(nullptr, FALSE, FALSE, nullptr);
if (completed == nullptr) return false;
bool ProcessAudioAvailable(bool exclude) {
ComPtr<IAudioClient> client;
const HRESULT result =
ActivateProcessLoopback(GetCurrentProcessId(), completed, &client);
CloseHandle(completed);
ActivateProcessLoopback(GetCurrentProcessId(), exclude, &client);
return SUCCEEDED(result) && client != nullptr;
}
@@ -233,32 +246,24 @@ bool SystemAudioAvailable() {
return SUCCEEDED(ActivateEndpoint(&client)) && client != nullptr;
}
HRESULT CaptureProcessAudio(DWORD pid, UINT64 expectedCreationTime,
HRESULT CaptureProcessAudio(DWORD pid, UINT64 expectedCreationTime, bool exclude,
HANDLE stop_event, const StopProbe& stop_probe,
const ReadyWriter& ready_writer,
const PCMWriter& writer) {
HRESULT com_result = CoInitializeEx(nullptr, COINIT_MULTITHREADED);
if (FAILED(com_result)) return com_result;
HRESULT result = ValidateProcessTarget(pid, expectedCreationTime);
HANDLE process = nullptr;
HANDLE completed = nullptr;
// Validate and watch the same handle so PID reuse cannot change the target.
HANDLE process = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION | SYNCHRONIZE, FALSE, pid);
HRESULT result = process ? ValidateProcessTarget(process, expectedCreationTime)
: HRESULT_FROM_WIN32(GetLastError());
ComPtr<IAudioClient> client;
if (SUCCEEDED(result)) {
process = OpenProcess(SYNCHRONIZE, FALSE, pid);
if (process == nullptr) result = HRESULT_FROM_WIN32(GetLastError());
}
if (SUCCEEDED(result)) {
completed = CreateEventW(nullptr, FALSE, FALSE, nullptr);
if (completed == nullptr) result = HRESULT_FROM_WIN32(GetLastError());
}
if (SUCCEEDED(result)) {
result = ActivateProcessLoopback(pid, completed, &client);
result = ActivateProcessLoopback(pid, exclude, &client);
}
if (SUCCEEDED(result)) {
result = CaptureAudioFrames(client, process, true, stop_event, stop_probe,
ready_writer, writer);
}
if (completed != nullptr) CloseHandle(completed);
if (process != nullptr) CloseHandle(process);
client.Reset();
CoUninitialize();
native/capture/windows/process_audio.h
+2 -2
View File
@@ -21,11 +21,11 @@ using PCMWriter =
using ReadyWriter = std::function<HRESULT()>;
using StopProbe = std::function<bool()>;
bool ProcessAudioAvailable();
bool ProcessAudioAvailable(bool exclude = false);
bool SystemAudioAvailable();
HRESULT CaptureMicrophone(const std::wstring& device_id, HANDLE stop_event, const StopProbe& stop_probe,
const ReadyWriter& ready_writer, const PCMWriter& writer);
HRESULT CaptureProcessAudio(DWORD pid, UINT64 expected_creation_time,
HRESULT CaptureProcessAudio(DWORD pid, UINT64 expected_creation_time, bool exclude,
HANDLE stop_event, const StopProbe& stop_probe,
const ReadyWriter& ready_writer,
const PCMWriter& writer);
scripts/embedded-sfu-gate.ts
+8 -2
View File
@@ -65,6 +65,7 @@ async function main(): Promise<void> {
const chromePath = process.env.CHROME_PATH?.trim();
if (!chromePath) throw new Error("CHROME_PATH is required");
const nativeArm = process.env.PIIK_EMBEDDED_SFU_NATIVE === "true";
const sfuOnly = process.env.PIIK_EMBEDDED_SFU_ONLY === "true";
const displayArm = process.env.PIIK_EMBEDDED_SFU_DISPLAY === "true";
if (displayArm && nativeArm) throw new Error("Display and Native capture are separate arms");
const twoRooms = process.env.PIIK_EMBEDDED_SFU_TWO_ROOMS === "true";
@@ -93,6 +94,7 @@ async function main(): Promise<void> {
const sourceProfile = nativeArm ? await mkdtemp(join(tmpdir(), "piik-client-media-")) : null;
const [sourcePort, sourceDebugPort] = nativeArm ? [await reservePort(), await reservePort()] : [0, 0];
const result = { passed: false, arm: `${nativeArm ? "native" : displayArm ? "display" : "browser"}-${codec}-simulcast${twoRooms ? "-two-rooms" : ""}`, stage: "start",
sfuOnly, stereoSeparationDb: null as number[] | null,
startedAt: new Date().toISOString(), finishedAt: "", processes: [] as Array<{ role: string; pid: number | null }>,
high: null as { frames: number; width: number; height: number } | null,
low: null as { frames: number; width: number; height: number } | null,
@@ -144,7 +146,8 @@ async function main(): Promise<void> {
PIIK_ENV: "development", PIIK_DEBUG: "route", LISTEN_HOST: "127.0.0.1", PORT: String(serverPort),
PIIK_LOG_DIR: join(BUILD_ROOT, "logs"),
PUBLIC_BASE_URL: origin, ALLOWED_ORIGINS: origin,
SFU_LISTEN_HOST: "127.0.0.1", SFU_UDP_PORT: String(mediaPort), SFU_PUBLIC_IP: "127.0.0.1" },
SFU_LISTEN_HOST: "127.0.0.1", SFU_UDP_PORT: String(mediaPort), SFU_PUBLIC_IP: "127.0.0.1",
SFU_ONLY: String(sfuOnly) },
});
processStarted("server", server);
server.stdout.resume();
@@ -257,6 +260,7 @@ async function main(): Promise<void> {
result.stage = "decoded-audio-energy";
await waitForSample(() => call<number>(viewer!, "gate.audioEnergy()", 3_000),
(energy) => { result.audioEnergy.high = energy; return energy > 0; }, 5_000);
if (!nativeArm) result.stereoSeparationDb = await call(viewer, "gate.audioSeparation()", 5_000);
result.host = await call(host, "gate.snapshot()");
if (displayArm) {
result.stage = "display-background";
@@ -357,7 +361,9 @@ async function main(): Promise<void> {
}
result.finishedAt = new Date().toISOString();
result.passed = !result.error && result.host?.published === true && result.host.simulcast &&
result.viewer?.decoded === true && result.viewer.committed && result.viewer.peerFailures > 0 &&
result.viewer?.decoded === true && result.viewer.committed &&
(sfuOnly ? result.viewer.peerFailures === 0 : result.viewer.peerFailures > 0) &&
(nativeArm || result.stereoSeparationDb?.every(db => db > 20) === true) &&
result.viewer.audioKbps > 0 && result.audioEnergy.high > 0 && result.audioEnergy.low > result.audioEnergy.high &&
result.high !== null && result.low !== null && result.audioEnergy.recovered > 0 &&
result.publicationRetired && result.subscriptionRecovered && result.cleanup && result.nativeShareStopped &&
scripts/embedded-sfu-page.ts
+43 -3
View File
@@ -239,18 +239,58 @@ export async function startHost(nativeSource?: { title: string; port: number },
video: { width: 1280, height: 720, frameRate: 15 }, audio: false,
}) : canvas.captureStream(15);
audio = new AudioContext();
const tone = audio.createOscillator();
const gain = audio.createGain();
gain.gain.value = 0.05;
const destination = audio.createMediaStreamDestination();
tone.connect(gain).connect(destination);
tone.start();
const channels = audio.createChannelMerger(2);
[500, 1500].forEach((frequency, channel) => {
const tone = audio!.createOscillator();
tone.frequency.value = frequency;
tone.connect(channels, 0, channel);
tone.start();
});
channels.connect(gain).connect(destination);
await audio.resume();
source.addTrack(destination.stream.getAudioTracks()[0]!);
await connect(room, true);
return room;
}
// Check decoded channels, not just /2 in SDP (Opus also uses that for mono).
export async function audioSeparation(stream = video?.srcObject): Promise<number[]> {
if (!(stream instanceof MediaStream) || !stream.getAudioTracks().length) throw new Error("No received audio");
const context = new AudioContext();
try {
const input = context.createMediaStreamSource(stream);
const splitter = context.createChannelSplitter(2);
const sink = context.createGain();
sink.gain.value = 0;
sink.connect(context.destination);
input.connect(splitter);
const analyzers = [0, 1].map(channel => {
const analyzer = context.createAnalyser();
analyzer.fftSize = 8192;
analyzer.smoothingTimeConstant = 0;
splitter.connect(analyzer, channel);
analyzer.connect(sink);
return analyzer;
});
await context.resume();
await new Promise(resolve => setTimeout(resolve, 1000));
return analyzers.map((analyzer, channel) => {
const bins = new Float32Array(analyzer.frequencyBinCount);
analyzer.getFloatFrequencyData(bins);
const level = (frequency: number) => {
const bin = Math.round(frequency * analyzer.fftSize / context.sampleRate);
return Math.max(-120, ...bins.slice(bin - 1, bin + 2));
};
return level(channel === 0 ? 500 : 1500) - level(channel === 0 ? 1500 : 500);
});
} finally {
await context.close();
}
}
export async function startViewer(room: CreateRoomResponse): Promise<void> {
await connect(room, false);
}
src/client/App.tsx
+3
View File
@@ -397,8 +397,11 @@ function SiteAccessGate({
}
return (
<HostPage
roomInteractionsAvailable={capabilities?.roomInteractions === true}
hostRoomSessionAvailable={capabilities?.hostRoomSession === true}
launchedByClient={clientLaunchBootstrap?.launchedByClient}
sfuAvailable={capabilities?.sfu === true}
sfuOnly={capabilities?.sfuOnly === true}
natPredictionAvailable={capabilities?.natPrediction === true}
connectionAttemptProgress4={capabilities?.connectionAttemptProgress4 === true}
onAuthorizationRequired={() =>
src/client/components/living/CaptureSourcePicker.tsx
+41 -3
View File
@@ -32,6 +32,7 @@ export type NativeSourceList =
sources: NativeCaptureTarget[];
processAudio: boolean;
systemAudio: boolean;
processAudioExclusion?: boolean;
captureBorderControl?: boolean;
};
@@ -51,6 +52,7 @@ export function CaptureSourcePicker({
loadCameras,
initialAudio = true,
initialShowCaptureBorder = false,
initialExcludeAudio,
audioLocked = false,
selectionDisabled = false,
}: {
@@ -60,7 +62,7 @@ export function CaptureSourcePicker({
initialCamera?: string;
activeCameraVideo?: HTMLVideoElement | null;
loadCameras?: typeof loadCameraPreviews;
onNative: (target: NativeCaptureTarget, audio: boolean, showCaptureBorder: boolean) => void;
onNative: (target: NativeCaptureTarget, audio: boolean, showCaptureBorder: boolean, excludeAudio?: NativeCaptureTarget) => void;
onPreview: (
target: NativeCaptureTarget,
signal?: AbortSignal,
@@ -72,6 +74,7 @@ export function CaptureSourcePicker({
initialTab?: SourceTab;
initialAudio?: boolean;
initialShowCaptureBorder?: boolean;
initialExcludeAudio?: NativeCaptureTarget;
audioLocked?: boolean;
selectionDisabled?: boolean;
}) {
@@ -81,6 +84,7 @@ export function CaptureSourcePicker({
const [tab, setTab] = useState<SourceTab | null>(null);
const [shareAudio, setShareAudio] = useState(initialAudio);
const [showCaptureBorder, setShowCaptureBorder] = useState(initialShowCaptureBorder);
const [excludeAudio, setExcludeAudio] = useState(initialExcludeAudio);
const appDetected = nativeSources.kind === "ready" || nativeSources.kind === "failed" ||
nativeSources.kind === "unsupported" || nativeSources.kind === "incompatible";
const tabs = SOURCE_TABS.filter((value) => appDetected || value === "browser" || value === "camera");
@@ -122,6 +126,14 @@ export function CaptureSourcePicker({
? nativeSources.processAudio
: nativeSources.systemAudio);
const anyNativeAudio = sources.some(supportsAudio);
// Source enumeration already owns process identity; do not create a second app scanner.
const audioApplications = nativeSources.kind === "ready"
? [...new Map(nativeSources.sources.filter(target => target.kind === "window")
.map(target => [`${target.pid}:${target.creationTime}`, target])).values()] : [];
const exclusionKey = excludeAudio?.kind === "window" ? `${excludeAudio.pid}:${excludeAudio.creationTime}` : "";
const canExcludeAudio = activeTab === "display" && nativeSources.kind === "ready" && nativeSources.processAudioExclusion;
const missingExclusion = !!excludeAudio && !audioApplications.some(target => `${target.pid}:${target.creationTime}` === exclusionKey);
const exclusionUnavailable = activeTab === "display" && shareAudio && !!excludeAudio && (!canExcludeAudio || missingExclusion);
const refreshing = activeTab === "camera" ? cameraSources.busy : nativeSources.kind === "loading";
const refreshLabel = t(refreshing ? "host.sourcePicker.loading" : "host.sourcePicker.refresh");
const audioAction = t(!anyNativeAudio ? "host.noAudio" : audioLocked
@@ -300,12 +312,12 @@ export function CaptureSourcePicker({
key={nativeCaptureTargetKey(target)}
target={target}
disabled={
selectionDisabled || (audioLocked && shareAudio && !supportsAudio(target))
selectionDisabled || exclusionUnavailable || (audioLocked && shareAudio && !supportsAudio(target))
}
onPreview={onPreview}
onSelect={() =>
onNative(target, shareAudio && supportsAudio(target),
supportsCaptureBorder && showCaptureBorder)
supportsCaptureBorder && showCaptureBorder, target.kind === "display" ? excludeAudio : undefined)
}
/>
))
@@ -352,6 +364,32 @@ export function CaptureSourcePicker({
</Tooltip>
</div>
) : null}
{canExcludeAudio && shareAudio && anyNativeAudio ? (
<details className="lr-source-exclusion" onToggle={event => {
if (event.currentTarget.open) event.currentTarget.scrollIntoView({ block: "nearest" });
}}>
<summary aria-label={vis ? t("host.sourcePicker.excludeAudio") : undefined}>
<Glyph name="speakerOff" size={18} />
{!vis && <span>{t("host.sourcePicker.excludeAudio")}{excludeAudio ? ` · ${excludeAudio.title}` : ""}</span>}
<Glyph name="chevron" size={14} />
</summary>
<div className="lr-capture-device">
<select id={`${pickerId}-exclude-audio`} value={exclusionKey}
aria-label={t("host.sourcePicker.excludeAudio")}
aria-describedby={`${pickerId}-exclude-hint`} disabled={selectionDisabled}
onChange={event => setExcludeAudio(audioApplications.find(target => `${target.pid}:${target.creationTime}` === event.target.value))}>
<option value="">{t("host.sourcePicker.excludeNone")}</option>
{missingExclusion && <option value={exclusionKey} disabled>{excludeAudio?.title} · {t("host.sourcePicker.excludeMissing")}</option>}
{audioApplications.map(target => <option key={`${target.pid}:${target.creationTime}`} value={`${target.pid}:${target.creationTime}`}>
{target.title}
</option>)}
</select>
<small id={`${pickerId}-exclude-hint`} className={vis ? "visually-hidden" : undefined}>
{t(excludeAudio ? "host.sourcePicker.excludeAudioHint" : "host.sourcePicker.excludeAudioChoose")}
</small>
</div>
</details>
) : null}
</div>
) : null}
src/client/components/living/Couch.tsx
+28 -17
View File
@@ -25,19 +25,27 @@ export interface CouchHostEntry {
onSelect?: () => void;
}
export interface CouchProps {
view: "host" | "viewer";
host?: CouchHostEntry | null;
entries: CouchEntry[];
selectedKey?: string | null;
onSelect?: (key: string) => void;
participantAction?: {
open: (key: string, trigger: HTMLButtonElement) => void;
target: string | null;
controls: string;
};
}
export function Couch({
view,
host,
entries,
selectedKey,
onSelect,
}: {
view: "host" | "viewer";
host?: CouchHostEntry | null;
entries: CouchEntry[];
selectedKey?: string | null;
onSelect?: (key: string) => void;
}) {
participantAction,
}: CouchProps) {
const { t } = useCopy();
const [couchRef, containerWidth] = useElementWidth();
const count = entries.length + (host ? 1 : 0);
@@ -76,15 +84,16 @@ export function Couch({
aria-label={`${t("common.host")} · ${t("common.viewers")}`}
>
{host ? (
<span className="lr-seat" style={seatStyle(0)}><Tooltip overflow={{ text: host.name, selector: ".lr-pawn-name" }}>
{host.onSelect ? (
<span className="lr-seat" data-room-peer={host.key} style={seatStyle(0)}><Tooltip overflow={{ text: host.name, selector: ".lr-pawn-name" }}>
{host.onSelect || participantAction ? (
<button
type="button"
className={`lr-pawn is-host${host.you ? " is-you" : ""}${host.selected ? " is-selected" : ""}`}
aria-label={hostLabel}
aria-pressed={host.selected}
aria-controls={host.controls}
onClick={host.onSelect}
aria-pressed={participantAction ? undefined : host.selected}
aria-expanded={participantAction ? participantAction.target === host.key : undefined}
aria-controls={participantAction?.controls ?? host.controls}
onClick={event => participantAction ? participantAction.open(host.key, event.currentTarget) : host.onSelect?.()}
>
<PawnSvg color={participantColor(host.key)} identity={host.key} host />
<span className="lr-pawn-name">{host.name}</span>
@@ -115,9 +124,9 @@ export function Couch({
</>
);
const className = `lr-pawn${entry.you ? " is-you" : ""}${
entry.selectable === false ? " is-static" : ""
entry.selectable === false && !participantAction ? " is-static" : ""
}${selectedKey === entry.key ? " is-selected" : ""}${view === "viewer" && entry.status.pulse ? " is-waiting" : ""}`;
const pawn = entry.selectable === false ? (
const pawn = entry.selectable === false && !participantAction ? (
<span
className={className}
role="img"
@@ -130,14 +139,16 @@ export function Couch({
type="button"
className={className}
aria-label={label}
aria-pressed={selectedKey === entry.key}
onClick={() => onSelect?.(entry.key)}
aria-pressed={participantAction ? undefined : selectedKey === entry.key}
aria-expanded={participantAction ? participantAction.target === entry.key : undefined}
aria-controls={participantAction?.controls}
onClick={event => participantAction ? participantAction.open(entry.key, event.currentTarget) : onSelect?.(entry.key)}
>
{inner}
</button>
);
return (
<span key={entry.key} className="lr-seat" style={seatStyle(index + (host ? 1 : 0))}><Tooltip overflow={{ text: entry.name, selector: ".lr-pawn-name" }}>
<span key={entry.key} className="lr-seat" data-room-peer={entry.key} style={seatStyle(index + (host ? 1 : 0))}><Tooltip overflow={{ text: entry.name, selector: ".lr-pawn-name" }}>
{pawn}
</Tooltip></span>
);
src/client/components/living/FloatingPanel.tsx
+222
View File
@@ -0,0 +1,222 @@
import { useLayoutEffect, useRef, useState, type PointerEvent, type ReactNode, type RefObject } from "react";
import { useCopy } from "../../ui/copy";
import { Glyph, type GlyphName } from "../../ui/icons";
import "./floating-panel.css";
const PANEL_SIZES = { default: { width: 420, height: 560 }, large: { width: 560, height: 680 } };
const COMPACT_SIZES = { default: { width: 304, height: undefined }, large: { width: 380, height: 460 } };
type PanelGesture = "move" | "nw" | "se";
type PanelChange = { x: number; y: number } | { rect: DOMRect; corner: "nw" | "se"; dx: number; dy: number };
/** Native visibility owns the shell; content owns its session, draft and reading position. */
export function FloatingPanel({ id, trigger, title, icon, compact = false, anchorOnOpen = false, className = "", children, onOpenChange }: {
id: string; trigger: string | RefObject<HTMLElement | null>; title: string; icon: GlyphName; compact?: boolean; className?: string;
anchorOnOpen?: boolean;
children: ReactNode; onOpenChange?: (open: boolean) => void;
}) {
const { t, vis } = useCopy();
const [open, setOpen] = useState(false);
const panelRef = useRef<HTMLElement>(null);
const positionMenu = useRef<HTMLDetailsElement>(null);
const sizes = compact ? COMPACT_SIZES : PANEL_SIZES;
const position = useRef({ x: 1, y: 1 });
const preferredSize = useRef(sizes.default);
const placePanel = useRef<((change?: PanelChange) => void) | null>(null);
const gesture = useRef<{ pointerId: number; x: number; y: number; rect: DOMRect; kind: PanelGesture } | null>(null);
const suppressDragClick = useRef(false);
const getTrigger = () => typeof trigger === "string" ? document.getElementById(trigger) : trigger.current;
const focusTrigger = () => {
const element = getTrigger();
if (element?.isConnected) element.focus({ preventScroll: true });
};
function startGesture(event: PointerEvent<HTMLElement>, kind: PanelGesture) {
if (!event.isPrimary || event.button !== 0 || !panelRef.current) return;
if (kind === "move" && (event.target as Element).closest("button, .lr-floating-positions")) return;
panelRef.current.getAnimations().forEach(animation => animation.finish());
gesture.current = { pointerId: event.pointerId, x: event.clientX, y: event.clientY,
rect: panelRef.current.getBoundingClientRect(), kind };
// Capture on the summary preserves its native click/keyboard activation.
const handle = (event.target as Element).closest<HTMLElement>("summary") ?? event.currentTarget;
handle.focus({ preventScroll: true });
handle.setPointerCapture(event.pointerId);
}
useLayoutEffect(() => {
const panel = panelRef.current;
if (!open || !panel) return;
// The visual viewport keeps the composer above a phone's keyboard without
// resizing the living room or taking ownership of scrolling/focus.
const viewport = window.visualViewport;
const place = (change?: PanelChange) => {
const layout = document.documentElement;
const width = viewport?.width ?? layout.clientWidth, height = viewport?.height ?? layout.clientHeight;
const maxWidth = Math.max(0, width - 24), maxHeight = Math.max(0, height - 24);
const left = (viewport?.offsetLeft ?? 0) + 12, top = (viewport?.offsetTop ?? 0) + 12;
let point = change && "x" in change ? change : undefined;
if (change && "corner" in change) {
const { rect, corner, dx, dy } = change;
const nw = corner === "nw";
// Hold the opposite corner steady; viewport changes only constrain the
// rendered size, while deliberate resizing writes the room UI preference.
const availableWidth = nw ? rect.right - left : left + maxWidth - rect.left;
const availableHeight = nw ? rect.bottom - top : top + maxHeight - rect.top;
const w = Math.min(availableWidth, Math.max(Math.min(compact ? 264 : 320, maxWidth), rect.width + (nw ? -dx : dx)));
const h = Math.min(availableHeight, Math.max(Math.min(compact ? 240 : 360, maxHeight), rect.height + (nw ? -dy : dy)));
preferredSize.current = { width: w, height: h };
point = { x: nw ? rect.right - w : rect.left, y: nw ? rect.bottom - h : rect.top };
}
panel.style.width = `${Math.min(preferredSize.current.width, maxWidth)}px`;
panel.style.maxHeight = `${maxHeight}px`;
const preferredHeight = preferredSize.current.height;
panel.style.height = preferredHeight === undefined ? "auto" : `${Math.min(preferredHeight, maxHeight)}px`;
const freeX = Math.max(0, maxWidth - panel.offsetWidth), freeY = Math.max(0, maxHeight - panel.offsetHeight);
// Keep the chosen relative position when the keyboard or viewport changes.
// Only deliberate movement writes it; a narrow viewport must not erase it.
if (point) {
if (freeX) position.current.x = Math.max(0, Math.min(1, (point.x - left) / freeX));
if (freeY) position.current.y = Math.max(0, Math.min(1, (point.y - top) / freeY));
}
panel.style.inset = `${top + freeY * position.current.y}px auto auto ${left + freeX * position.current.x}px`;
};
const resize = () => place();
const hideOnEscape = (event: KeyboardEvent) => {
// Inner native surfaces consume Escape first, even after focus moves out.
if (event.key !== "Escape" || event.defaultPrevented || event.isComposing ||
document.querySelector('[popover="auto"]:popover-open, dialog:modal')) return;
// Native top-layer order follows opening, not DOM order. Consume one
// close request even when chat and the reaction palette are both open.
const panels = [...document.querySelectorAll<HTMLElement>(".lr-floating-panel:popover-open")];
if (!panels.length) return;
const topmost = panels.reduce((top, next) => Number(next.dataset.openedAt) >= Number(top.dataset.openedAt) ? next : top);
if (topmost !== panel) return;
event.preventDefault();
const restoreFocus = panel.contains(document.activeElement);
panel.hidePopover();
if (restoreFocus) focusTrigger();
};
placePanel.current = place;
place();
// Contextual palettes start beside their invoker, then use the same free
// movement owner. Retargeting an open palette must not undo a user's drag.
const anchor = anchorOnOpen && getTrigger()?.getBoundingClientRect();
if (anchor) {
const top = (viewport?.offsetTop ?? 0) + 12;
const bottom = top + (viewport?.height ?? document.documentElement.clientHeight) - 24;
const below = anchor.bottom + 8;
const headroom = anchor.top - top >= panel.offsetHeight + 136 ? 128 : 0;
place({ x: anchor.x + anchor.width / 2 - panel.offsetWidth / 2,
y: below + panel.offsetHeight <= bottom ? below : anchor.top - panel.offsetHeight - 8 - headroom });
}
viewport?.addEventListener("resize", resize);
viewport?.addEventListener("scroll", resize);
window.addEventListener("resize", resize);
// Content-sized palettes still fit after a target, language or status change.
const observer = new ResizeObserver(resize);
observer.observe(panel);
document.addEventListener("keydown", hideOnEscape);
return () => {
placePanel.current = null;
gesture.current = null;
viewport?.removeEventListener("resize", resize);
viewport?.removeEventListener("scroll", resize);
window.removeEventListener("resize", resize);
observer.disconnect();
document.removeEventListener("keydown", hideOnEscape);
};
}, [open, trigger, compact, anchorOnOpen]);
return (<section id={id} ref={panelRef} popover="manual" role="dialog" tabIndex={-1} className={`lr-floating-surface lr-floating-panel ${className}`}
style={sizes.default}
aria-label={title}
onPointerDownCapture={() => { suppressDragClick.current = false; }}
onPointerMove={event => {
const drag = gesture.current;
if (!drag || drag.pointerId !== event.pointerId) return;
const dx = event.clientX - drag.x, dy = event.clientY - drag.y;
if (!suppressDragClick.current && Math.hypot(dx, dy) < 4) return;
suppressDragClick.current = true;
placePanel.current?.(drag.kind === "move" ? { x: drag.rect.left + dx, y: drag.rect.top + dy }
: { rect: drag.rect, corner: drag.kind, dx, dy });
}}
onLostPointerCapture={() => { gesture.current = null; }}
onPointerCancel={() => { suppressDragClick.current = false; }}
onClickCapture={event => {
if (suppressDragClick.current && event.detail > 0) { event.preventDefault(); event.stopPropagation(); }
suppressDragClick.current = false;
}}
onToggle={event => {
// React delivers nested details/popover toggles to this handler too.
// Only this panel's native visibility owns its positioning lifetime.
if (event.target !== event.currentTarget) return;
const open = event.newState === "open";
setOpen(open);
onOpenChange?.(open);
if (open) {
event.currentTarget.dataset.openedAt = String(performance.now());
// Opening to read must not summon a phone's keyboard.
event.currentTarget.focus({ preventScroll: true });
} else {
if (positionMenu.current) positionMenu.current.open = false;
if (event.currentTarget.contains(document.activeElement)) focusTrigger();
}
}}>
<header className="lr-floating-header" role="group" tabIndex={0} aria-label={t("floating.move")}
aria-describedby={`${id}-move-hint`}
onPointerDown={event => startGesture(event, "move")}
onKeyDown={event => {
if (event.target !== event.currentTarget || !event.key.startsWith("Arrow") || !panelRef.current) return;
event.preventDefault();
const rect = panelRef.current.getBoundingClientRect();
placePanel.current?.({ x: rect.left + (event.key === "ArrowLeft" ? -24 : event.key === "ArrowRight" ? 24 : 0),
y: rect.top + (event.key === "ArrowUp" ? -24 : event.key === "ArrowDown" ? 24 : 0) });
}}>
<span className="lr-floating-title"><Glyph name="grip" size={17} />
{vis ? <Glyph name={icon} size={18} /> : <b>{title}</b>}</span>
<span id={`${id}-move-hint`} className="visually-hidden">{t("floating.moveHint")}</span>
<details ref={positionMenu} className="lr-floating-position"
onBlur={event => { if (!event.currentTarget.contains(event.relatedTarget as Node)) event.currentTarget.open = false; }}
onKeyDown={event => {
if (event.key !== "Escape" || event.defaultPrevented || event.nativeEvent.isComposing || !event.currentTarget.open) return;
event.stopPropagation(); event.currentTarget.open = false;
event.currentTarget.querySelector("summary")?.focus();
}}>
<summary aria-label={t("floating.position")}><Glyph name="move" size={16} />{!vis && <span>{t("floating.position")}</span>}</summary>
<div className="lr-floating-positions" role="group" aria-label={t("floating.position")}>
{(["topLeft", "topRight", "bottomLeft", "bottomRight"] as const).map((corner, index) => {
const x = index % 2, y = Math.floor(index / 2);
return <button key={corner} type="button" aria-label={t(`floating.position.${corner}`)} onClick={() => {
position.current = { x, y }; placePanel.current?.();
if (positionMenu.current) { positionMenu.current.open = false; positionMenu.current.querySelector("summary")?.focus(); }
}}>
<svg viewBox="0 0 28 22" width="28" height="22" aria-hidden="true">
<rect x="1" y="1" width="26" height="20" rx="4" fill="none" stroke="currentColor" strokeWidth="1.5" />
<rect x={x ? 16 : 4} y={y ? 12 : 4} width="8" height="6" rx="1.5" fill="currentColor" />
</svg>{!vis && <span>{t(`floating.position.${corner}`)}</span>}
</button>;
})}
{(["default", "large"] as const).map(size => <button key={size} type="button" className="lr-floating-size"
onClick={() => {
preferredSize.current = sizes[size]; placePanel.current?.();
if (positionMenu.current) { positionMenu.current.open = false; positionMenu.current.querySelector("summary")?.focus(); }
}} aria-label={t(`floating.size.${size}`)}>
<Glyph name={size === "large" ? "expand" : "contract"} size={18} />{!vis && <span>{t(`floating.size.${size}`)}</span>}
</button>)}
</div>
</details>
<button type="button" className="lr-popover-close" aria-label={t("common.close")} onClick={() => {
panelRef.current?.hidePopover(); focusTrigger();
}}><Glyph name="x" size={16} /></button>
</header>
{children}
{(["nw", "se"] as const).map(corner => <span key={corner} className={`lr-floating-resize is-${corner}`}
role="group" tabIndex={0} aria-label={t("floating.resize")} aria-describedby={`${id}-resize-hint`}
onPointerDown={event => startGesture(event, corner)} onKeyDown={event => {
if (!event.key.startsWith("Arrow") || !panelRef.current) return;
event.preventDefault();
placePanel.current?.({ rect: panelRef.current.getBoundingClientRect(), corner,
dx: event.key === "ArrowLeft" ? -24 : event.key === "ArrowRight" ? 24 : 0,
dy: event.key === "ArrowUp" ? -24 : event.key === "ArrowDown" ? 24 : 0 });
}}><svg viewBox="0 0 16 16" width="14" height="14" aria-hidden="true"><path d="M3 12 12 3M8 12l4-4" fill="none" stroke="currentColor" strokeWidth="1.5" strokeLinecap="round" /></svg></span>)}
<span id={`${id}-resize-hint`} className="visually-hidden">{t("floating.resizeHint")}</span>
</section>);
}
src/client/components/living/LauncherForm.tsx
+1 -1
View File
@@ -112,7 +112,7 @@ export function LauncherForm({
key={choice.mode}
type="button"
role="radio"
className={`lr-client-mode${mode === choice.mode ? " is-selected" : ""}`}
className={`lr-tile lr-client-mode${mode === choice.mode ? " is-selected" : ""}`}
aria-checked={mode === choice.mode}
tabIndex={mode === choice.mode ? 0 : -1}
aria-label={t(choice.label)}
src/client/components/living/MicrophoneDeviceSelect.tsx
+1 -1
View File
@@ -3,7 +3,7 @@ import type { CaptureDevice } from "../../media/capture-devices";
import { useCopy } from "../../ui/copy";
import { Glyph } from "../../ui/icons";
// Shared presentation only: capture and replacement stay with the Host owner.
// Shared presentation only: the caller owns capture and device replacement.
export function MicrophoneDeviceSelect({ value, load, onChange, disabled, revision, browser = true }: {
value: string; load: () => Promise<CaptureDevice[]>;
onChange: (id: string) => void; disabled?: boolean; revision?: unknown; browser?: boolean;
src/client/components/living/Pawn.tsx
+2 -2
View File
@@ -7,7 +7,7 @@ export function PersonShape({ eyes = true, gaze = 0, eyeClassName, host }: {
eyes?: boolean | "closed"; gaze?: number; eyeClassName?: string; host?: boolean;
}) {
return <>
<circle cx="24" cy="14" r="8" stroke="none" />
<circle className="lr-person-head" cx="24" cy="14" r="8" stroke="none" />
<path d="M10 47c0-13 5-20 14-20s14 7 14 20q0 3-3 3H13q-3 0-3-3Z" stroke="none" />
{eyes ? <g className={`lr-person-eyes${eyeClassName ? ` ${eyeClassName}` : ""}`} fill="#263b43">
{eyes === "closed" ? <path d={`M${20.1 + gaze} 14.5q1.1-1.4 2.2 0m3.4 0q1.1-1.4 2.2 0`}
@@ -35,6 +35,6 @@ export function PawnSvg({ color, host, identity }: {
"--person-gaze-x": `${host ? 1.6 : motion.gaze}px`,
"--person-gaze-y": host ? "-1.4px" : "0px",
} as CSSProperties : undefined}>
<g className="lr-person-body" fill={color}><PersonShape host={host} /></g>
<g className="lr-reaction-body"><g className="lr-person-body" fill={color}><PersonShape host={host} /></g></g>
</svg>;
}
src/client/components/living/PlaybackControls.tsx
+4 -1
View File
@@ -1,4 +1,4 @@
import { useEffect, useRef, useState, type RefObject } from "react";
import { useEffect, useRef, useState, type RefObject, type ReactNode } from "react";
import { ViewerAudio, type ViewerAudioSnapshot } from "../../media/viewer-audio";
import { useCopy } from "../../ui/copy";
import { debugError } from "../../lib/debug";
@@ -15,6 +15,7 @@ import "./playback-controls.css";
export function PlaybackControls({
videoRef, stream, audioTrackKey, canPlay, theaterMode, onPlay,
onToggleTheater, onReconnect, reconnectAvailable,
extraActions,
}: {
videoRef: RefObject<HTMLVideoElement | null>;
stream: MediaStream | null;
@@ -25,6 +26,7 @@ export function PlaybackControls({
onToggleTheater: () => void;
onReconnect: () => void;
reconnectAvailable: boolean;
extraActions?: ReactNode;
}) {
const { t, vis } = useCopy();
const audioRef = useRef<ViewerAudio | null>(null);
@@ -146,6 +148,7 @@ export function PlaybackControls({
</span>
</span>
<span className="lr-playback-view">
{extraActions}
<Btn icon="refresh" title="viewer.reconnect" hint="hint-reconnect" draw="playback-reconnect"
disabled={!reconnectAvailable} onClick={onReconnect} />
<Btn icon={picture.active ? "pipExit" : "pip"} draw="playback-pip"
src/client/components/living/RoomChatOverlay.tsx
+55
View File
@@ -0,0 +1,55 @@
import { useEffect, useSyncExternalStore, type CSSProperties } from "react";
import { CHAT_OVERLAY_DURATION_MS, RoomInteractionSession } from "../../lib/room-interactions";
import { Btn } from "./primitives";
import "./room-chat-overlay.css";
export function RoomChatToggle({ session, caption = false }: { session: RoomInteractionSession | null; caption?: boolean }) {
return session ? <ConnectedToggle key={session.key} session={session} caption={caption} /> : null;
}
function ConnectedToggle({ session, caption }: { session: RoomInteractionSession; caption: boolean }) {
const { overlayEnabled } = useSyncExternalStore(session.subscribe, session.getSnapshot);
return <span className="lr-chat-overlay-toggle"><Btn icon="danmaku"
cap={caption ? "interaction.overlay.label" : undefined}
title={overlayEnabled ? "interaction.overlay.hide" : "interaction.overlay.show"}
hint={overlayEnabled ? "hint-chat-overlay-hide" : "hint-chat-overlay-show"}
hintText={overlayEnabled ? "interaction.overlay.hideHint" : "interaction.overlay.showHint"}
pressed={overlayEnabled} tone={overlayEnabled ? "on" : undefined}
onClick={() => session.setOverlayEnabled(!overlayEnabled)} /></span>;
}
export function RoomChatOverlay({ session, visible }: { session: RoomInteractionSession | null; visible: boolean }) {
return session ? <ConnectedOverlay key={session.key} session={session} visible={visible} /> : null;
}
function ConnectedOverlay({ session, visible }: { session: RoomInteractionSession; visible: boolean }) {
const { overlayMessages } = useSyncExternalStore(session.subscribe, session.getSnapshot);
useEffect(() => {
const sync = () => session.setOverlayVisible(visible && !document.hidden);
sync();
document.addEventListener("visibilitychange", sync);
return () => {
document.removeEventListener("visibilitychange", sync);
session.setOverlayVisible(false);
};
}, [session, visible]);
return visible ? <div className="lr-chat-overlay" aria-hidden="true">
{overlayMessages.map(message => <ChatFlight key={message.id} message={message} now={session.now} />)}
</div> : null;
}
function ChatFlight({ message, now }: {
message: ReturnType<RoomInteractionSession["getSnapshot"]>["overlayMessages"][number]; now: () => number;
}) {
return <span className="lr-chat-flight" data-lane={message.lane} style={{ "--chat-lane": message.lane,
animationDuration: `${CHAT_OVERLAY_DURATION_MS}ms` } as CSSProperties} onAnimationStart={event => {
// Mounting or returning from reduced motion uses the remaining lifetime.
for (const animation of event.currentTarget.getAnimations()) {
if (animation instanceof CSSAnimation && animation.animationName === "lr-chat-flight") {
animation.currentTime = Math.min(CHAT_OVERLAY_DURATION_MS, Math.max(0, CHAT_OVERLAY_DURATION_MS - (message.expiresAt - now())));
}
}
}}>
{message.payload.kind === "chat" && message.payload.text}
</span>;
}
src/client/components/living/RoomInteractions.tsx
+228
View File
@@ -0,0 +1,228 @@
import { useEffect, useId, useLayoutEffect, useRef, useState, useSyncExternalStore } from "react";
import { normalizeChatText, MAX_CHAT_CODE_POINTS, REACTION_IDS, isThrow } from "../../../shared/room-interactions";
import { RoomInteractionSession } from "../../lib/room-interactions";
import { useCopy } from "../../ui/copy";
import { Glyph } from "../../ui/icons";
import { Btn, SwitchItem } from "./primitives";
import { participantColor } from "./participant-color";
import { FloatingPanel } from "./FloatingPanel";
import { PawnSvg } from "./Pawn";
import { Couch, type CouchProps } from "./Couch";
import { RoomReaction, ReactionIcon } from "./RoomReaction";
import { RoomChatToggle } from "./RoomChatOverlay";
import "./room-interactions.css";
interface Participant { peerId: string; displayName: string }
type Props = CouchProps & { session: RoomInteractionSession | null };
export function RoomInteractions(props: Props) {
return props.session ? <ConnectedRoomInteractions key={props.session.key} {...props} session={props.session} /> : <Couch {...props} />;
}
type ConnectedProps = Props & { session: RoomInteractionSession };
function ConnectedRoomInteractions(props: ConnectedProps) {
const state = useSyncExternalStore(props.session.subscribe, props.session.getSnapshot);
return <InteractionPanel {...props} state={state} />;
}
function InteractionPanel({ session, state, ...couch }: ConnectedProps & {
state: ReturnType<RoomInteractionSession["getSnapshot"]>;
}) {
const { t, vis } = useCopy();
const [chatOpen, setChatOpen] = useState(false);
const [effects, setEffects] = useState(true);
const [target, setTarget] = useState<string | null>(null);
const [draft, setDraft] = useState("");
const [lastSeen, setLastSeen] = useState<string | null>(null);
const [visible, setVisible] = useState(false);
const [following, setFollowing] = useState(true);
const area = useRef<HTMLDivElement>(null);
const anchor = useRef<HTMLButtonElement | null>(null);
const log = useRef<HTMLDivElement>(null);
const id = useId();
const menuId = `${id}-person`;
const participants: Participant[] = [
...(couch.host ? [{ peerId: couch.host.key, displayName: couch.host.name }] : []),
...couch.entries.map(entry => ({ peerId: entry.key, displayName: entry.name })),
];
const latest = state.messages.at(-1)?.id ?? null;
const unseenIndex = state.messages.findIndex(message => message.id === lastSeen);
const unread = latest === lastSeen ? 0 : state.messages.length - unseenIndex - 1;
const canSend = state.ready && !state.pending && !state.coolingDown;
const person = participants.find(person => person.peerId === target);
const rosterKey = participants.map(person => person.peerId).join(",");
const inspect = target === couch.host?.key && couch.host.onSelect ? () => { if (!couch.host?.selected) couch.host?.onSelect?.(); }
: couch.entries.some(entry => entry.key === target && entry.selectable !== false) && couch.onSelect
? () => { if (target !== couch.selectedKey) couch.onSelect?.(target!); } : undefined;
useEffect(() => {
const confirmed = state.confirmed;
if (confirmed?.payload.kind === "chat") {
const text = confirmed.payload.text;
setDraft(current => normalizeChatText(current) === text ? "" : current);
}
}, [state.confirmed]);
useLayoutEffect(() => {
const popup = document.getElementById(menuId);
if (!popup) return;
if (person && anchor.current?.isConnected) popup.showPopover({ source: anchor.current });
else popup.hidePopover();
}, [target, rosterKey, menuId]);
function openReactions(key: string, trigger: HTMLButtonElement) {
const popup = document.getElementById(menuId);
if (target === key && popup?.matches(":popover-open")) { popup.hidePopover(); return; }
anchor.current = trigger;
setTarget(key);
}
useEffect(() => {
if (!chatOpen || !log.current) return;
let active = true;
let inView = false;
const sync = () => { if (active) setVisible(inView && document.visibilityState === "visible"); };
const observer = new IntersectionObserver(entries => {
const entry = entries.at(-1);
inView = Boolean(entry?.isIntersecting && entry.intersectionRatio > 0);
sync();
});
observer.observe(log.current);
document.addEventListener("visibilitychange", sync);
return () => {
active = false;
observer.disconnect();
document.removeEventListener("visibilitychange", sync);
setVisible(false);
};
}, [chatOpen]);
useEffect(() => {
if (!chatOpen || !visible || !following || document.visibilityState !== "visible" ||
!log.current?.getClientRects().length) return;
setLastSeen(latest);
log.current.scrollTop = log.current.scrollHeight;
}, [latest, chatOpen, visible, following]);
return <div className="lr-room-interactions">
<div className="visually-hidden" role="log" aria-label={t("interaction.reactions")}
aria-live="polite" aria-relevant="additions">
{state.reactions.map(({ id, sender, payload }) => payload.kind === "reaction" && <p key={id}>
{t(payload.targetPeerId ? "interaction.receivedReaction" : "interaction.expressedReaction", {
sender: sender.displayName,
reaction: t(`interaction.reaction.${payload.reaction}`),
target: participants.find(person => person.peerId === payload.targetPeerId)?.displayName ?? t("interaction.participant"),
})}
</p>)}
</div>
<FloatingPanel id={id} trigger={`${id}-toggle`} icon="chat" className="lr-interaction-panel"
title={t("interaction.title")} onOpenChange={setChatOpen}>
<div className="lr-chat-history">
<div className="lr-chat-log" role="log" aria-label={t("interaction.title")} aria-live="polite" aria-relevant="additions" tabIndex={0} ref={log}
onScroll={event => { const node = event.currentTarget; setFollowing(node.scrollHeight - node.scrollTop - node.clientHeight < 40); }}>
{state.messages.length === 0 ? <div className="lr-chat-empty">
<span className="lr-chat-welcome" aria-hidden="true">{participants.slice(0, 2).map(person =>
<PawnSvg key={person.peerId} identity={person.peerId} color={participantColor(person.peerId)} host={person.peerId === couch.host?.key} />)}
<Glyph name="chat" size={24} /></span><p>{vis ? "···" : t("interaction.empty")}</p>
</div> : state.messages.map((message, index) => {
const previous = state.messages[index - 1];
const continuation = previous?.sender.peerId === message.sender.peerId && previous.sender.displayName === message.sender.displayName;
return <div key={message.id} className={`lr-chat-message${message.isSelf ? " is-self" : ""}${continuation ? " is-continuation" : ""}`}>
{!continuation && <span className="lr-chat-avatar" aria-hidden="true"><PawnSvg
color={participantColor(message.sender.peerId)} host={message.sender.role === "host"} /></span>}
<div className="lr-chat-bubble">
<span className={continuation ? "visually-hidden" : "lr-chat-author"}>{message.sender.displayName}
{message.sender.role === "host" && <span className="visually-hidden"> · {t("common.host")}</span>}</span>
<p>{message.payload.kind === "chat" && message.payload.text}</p>
</div>
</div>;
})}
</div>
{!following && state.messages.length > 0 && <button type="button" className="lr-chat-latest" onClick={() => {
setFollowing(true); log.current?.focus({ preventScroll: true });
}}><Glyph name="arrowDown" size={15} /><span className={vis ? "visually-hidden" : undefined}>
{unread ? t("interaction.unread", { count: String(unread) }) : t("interaction.latest")}</span></button>}
</div>
<form className="lr-chat-compose" onSubmit={event => {
event.preventDefault();
const text = normalizeChatText(draft);
if (text && canSend) { setFollowing(true); session.send({ kind: "chat", text }); }
}}>
<input value={draft} onChange={event => setDraft(event.target.value)} maxLength={MAX_CHAT_CODE_POINTS * 2}
aria-label={t("interaction.placeholder")} placeholder={vis ? "···" : t("interaction.placeholder")}
autoComplete="off"
onKeyDown={event => { if (event.key === "Enter" && event.nativeEvent.isComposing) event.preventDefault(); }} />
<Btn type="submit" icon="send" title="interaction.send" hint="hint-chat-send" tone="primary" busy={state.pending?.payload.kind === "chat"}
disabled={!canSend || !normalizeChatText(draft)} />
</form>
<footer className="lr-chat-footer">
<RoomChatToggle session={session} caption />
<div className="lr-interaction-status" role="status">
{state.error ? <><Glyph name="alert" size={14} /><span className={vis ? "visually-hidden" : undefined}>{t(`interaction.error.${state.error}`)}</span></>
: !state.ready ? <><Glyph name="network" size={14} /><span className={vis ? "visually-hidden" : undefined}>{t("interaction.reconnecting")}</span></>
: !vis && <span>{t("interaction.ephemeral")}</span>}
</div>
{draft && <span className="lr-chat-count" aria-hidden="true">{Array.from(draft).length} / {MAX_CHAT_CODE_POINTS}</span>}
</footer>
</FloatingPanel>
<div className="lr-interaction-couch" ref={area}>
<Couch {...couch} participantAction={{ open: openReactions, target, controls: menuId }} />
{effects && state.reactions.map((reaction, index) => {
const payload = reaction.payload;
if (payload.kind !== "reaction") return null;
// Only one's own expression replaces one's previous expression.
// Receiving a gift never speaks for that person or hides their reaction.
const covered = !payload.targetPeerId && !isThrow(payload.reaction) && state.reactions.slice(index + 1).some(next =>
next.payload.kind === "reaction" && !next.payload.targetPeerId && !isThrow(next.payload.reaction) &&
next.sender.peerId === reaction.sender.peerId);
if (covered) return null;
return participants.some(person => person.peerId === reaction.sender.peerId) &&
(!payload.targetPeerId || participants.some(person => person.peerId === payload.targetPeerId)) &&
<RoomReaction key={reaction.id} reaction={reaction} area={area} now={session.now} />;
})}
</div>
<div className="lr-interaction-toggle lr-media-controls">
<span className="lr-interaction-chat-toggle">
<Btn id={`${id}-toggle`} icon="chat" title={chatOpen ? "interaction.close" : "interaction.open"} cap="interaction.open" expanded={chatOpen} controls={id}
hint={chatOpen ? "hint-close" : "hint-chat-open"}
tone={chatOpen ? "on" : undefined} popoverTarget={id} />
{unread > 0 && <span className="lr-interaction-unread" aria-label={t("interaction.unread", { count: String(unread) })}>{unread}</span>}
</span>
</div>
<FloatingPanel id={menuId} trigger={anchor} compact anchorOnOpen icon="smile" className="lr-person-menu"
title={t("interaction.express")} onOpenChange={open => { if (!open) setTarget(null); }}>
<div className="lr-person-menu-body">
<div className="lr-person-menu-target" role="group"
aria-label={person ? t("interaction.withPerson", { name: person.displayName }) : t("interaction.express")}>
<span className="lr-person-menu-name">
<span className="lr-person-menu-portrait">{person
? <PawnSvg color={participantColor(person.peerId)} host={person.peerId === couch.host?.key} />
: <Glyph name="smile" size={22} />}</span>
<span>{!vis && <small>{t(target === state.peerId ? "interaction.express" : "interaction.toPerson")}</small>}
<b>{person?.displayName ?? t("interaction.participant")}</b></span>
</span>
</div>
{[false, true].filter(throws => !throws || (person && target !== state.peerId)).map(throws => <div key={String(throws)} className={throws ? "lr-reaction-props" : "lr-reaction-grid"}>
{REACTION_IDS.filter(reaction => isThrow(reaction) === throws).map(reaction => <button key={reaction} type="button"
aria-label={t(`interaction.reaction.${reaction}`)} aria-disabled={!canSend || !person}
onClick={() => {
if (!canSend || !person) return;
session.send({ kind: "reaction", reaction, ...(target !== state.peerId ? { targetPeerId: person.peerId } : {}) });
}}>
<ReactionIcon reaction={reaction} />
{throws && !vis && <span>{t(reaction === "tomato" ? "interaction.prop.tomato" : "interaction.prop.poop")}</span>}
</button>)}
</div>)}
{state.error || !state.ready ? <div className="lr-interaction-status" role="status">
<Glyph name={state.error ? "alert" : "network"} size={14} />
<span className={vis ? "visually-hidden" : undefined}>{t(state.error ? `interaction.error.${state.error}` : "interaction.reconnecting")}</span>
</div> : null}
{inspect && <button type="button" className="lr-person-menu-details" onClick={() => { document.getElementById(menuId)?.hidePopover(); inspect(); }}>
<Glyph name="gauge" size={16} />{!vis && t("host.details")}
<span className="visually-hidden">{vis && t("host.details")}</span>
</button>}
{target === state.peerId && <div className="lr-interaction-effects">
<SwitchItem checked={effects} onChange={setEffects} label={t("interaction.effects")} />
{vis && <Glyph name="smile" size={17} />}
</div>}
</div>
</FloatingPanel>
</div>;
}
src/client/components/living/RoomReaction.tsx
+119
View File
@@ -0,0 +1,119 @@
import { useLayoutEffect, useRef, type CSSProperties, type RefObject } from "react";
import { REACTION_DURATION_MS, REACTION_GLYPHS, isThrow, type ReactionId } from "../../../shared/room-interactions";
import type { RoomInteraction } from "../../lib/room-interactions";
import { identityHash } from "../../lib/identity-hash";
import { Glyph } from "../../ui/icons";
/** Props use the same small silhouette in the picker and on the couch. */
export function ReactionIcon({ reaction }: { reaction: ReactionId }) {
if (!isThrow(reaction)) return <span aria-hidden="true">{REACTION_GLYPHS[reaction]}</span>;
return <svg viewBox="0 0 40 40" width="32" height="32" aria-hidden="true">
{reaction === "tomato" ? <>
<path d="M20 11C5 3 0 31 16 34c20 5 26-30 4-23Z" fill="#e96c55" />
<path d="m20 14-9-6 8 2 3-6 2 6 8-1-7 6Z" fill="#448e6b" />
<path d="M10 20q-2 5 1 7" fill="none" stroke="#ffb399" strokeWidth="3" strokeLinecap="round" />
</> : <>
<path d="M24 5c2 7 10 7 5 13 8 1 9 7 5 10 8 8-22 9-28 3-5-5 1-9 5-10-4-5 2-8 6-9 4-1 5-4 7-7Z" fill="#997554" />
<path d="M13 21q9 3 17-1M9 29q12 3 22-1" fill="none" stroke="#795b43" strokeWidth="2" strokeLinecap="round" />
<circle cx="18" cy="23" r="1.5" fill="#fff4dd" /><circle cx="25" cy="23" r="1.5" fill="#fff4dd" />
</>}
</svg>;
}
/** Fixed, event-seeded accents: no independent timer or mutable random stream. */
function ReactionAccents({ reaction, seed }: { reaction: ReactionId; seed: number }) {
return <svg className={`lr-reaction-accents is-${reaction}`} viewBox="-48 -48 96 96" width="96" height="96">
{reaction === "tomato" && <ellipse className="lr-reaction-impact" rx="18" ry="12" />}
{Array.from({ length: reaction === "sleep" ? 3 : 5 }, (_, index) => {
const angle = (-160 + index * 35 + (seed % 13)) * Math.PI / 180;
const radius = 27 + ((seed >>> (index * 4)) % 12);
return <g key={index} className="lr-reaction-spark" style={{
"--spark-x": `${Math.cos(angle) * radius}px`, "--spark-y": `${Math.sin(angle) * radius}px`,
"--spark-turn": `${index * 23 - 46}deg`,
} as CSSProperties}>
{reaction === "heart" ? <path d="M0 3C-8-1-5-7 0-4 5-7 8-1 0 3Z" />
: reaction === "sleep" ? <text x="0" y="0" textAnchor="middle">z</text>
: reaction === "poop" ? <circle r={4 + index % 2} />
: reaction === "tomato" ? <ellipse rx="2.5" ry="4.5" />
: reaction === "party" ? <rect x="-2" y="-3" width="4" height="6" rx="1" />
: <path d="M0-4 1-1 4 0 1 1 0 4-1 1-4 0-1-1Z" />}
</g>;
})}
</svg>;
}
/** Event expiry owns lifetime. This component owns only placement and its own impact. */
export function RoomReaction({ reaction, area, now }: {
reaction: RoomInteraction & { expiresAt: number }; area: RefObject<HTMLDivElement | null>; now: () => number;
}) {
const flight = useRef<HTMLDivElement>(null);
const payload = reaction.payload;
useLayoutEffect(() => {
const root = area.current;
const element = flight.current;
if (!root || !element || payload.kind !== "reaction") return;
const pawn = (peer: string) => root.querySelector<SVGSVGElement>(`[data-room-peer="${CSS.escape(peer)}"] .lr-person`);
const sender = pawn(reaction.sender.peerId);
const recipient = pawn(payload.targetPeerId ?? reaction.sender.peerId);
if (!sender || !recipient) return;
let frame = 0;
let impact: Animation | undefined;
const place = () => {
if (now() >= reaction.expiresAt || !sender.isConnected || !recipient.isConnected) {
element.style.visibility = "hidden";
return;
}
frame = requestAnimationFrame(place);
if (document.hidden || !root.getClientRects().length) {
element.style.visibility = "hidden";
impact?.cancel();
return;
}
const rect = element.getBoundingClientRect();
// Transforms do not trigger ResizeObserver. Follow the actual head for
// this event's remaining lifetime, including hover, lean and impact.
const from = sender.querySelector(".lr-person-head")?.getBoundingClientRect();
const to = recipient.querySelector(".lr-person-head")?.getBoundingClientRect();
if (!from || !to) { element.style.visibility = "hidden"; return; }
element.style.setProperty("--from-x", `${from.x - rect.x + from.width / 2}px`);
element.style.setProperty("--from-y", `${from.y - rect.y + from.height / 2}px`);
element.style.setProperty("--to-x", `${to.x - rect.x + to.width / 2}px`);
element.style.setProperty("--to-y", `${to.y - rect.y + to.height / 2}px`);
element.style.setProperty("--throw-direction", from.x > to.x ? "-1" : "1");
element.style.visibility = "visible";
};
place();
const body = recipient.querySelector<SVGGElement>(".lr-reaction-body");
if (isThrow(payload.reaction) && payload.targetPeerId && body &&
!window.matchMedia("(prefers-reduced-motion: reduce)").matches && !root.closest("[data-comic-reduced-motion]")) {
const elapsed = Math.max(0, REACTION_DURATION_MS - (reaction.expiresAt - now()));
const delay = REACTION_DURATION_MS * .32 - elapsed;
if (delay > -560) {
const direction = sender.getBoundingClientRect().x > recipient.getBoundingClientRect().x ? -1 : 1;
impact = body.animate([
{ transform: "none" },
{ transform: payload.reaction === "tomato" ? `rotate(${direction * 6}deg) scale(1.05,.95)` : `translateX(${direction * 4}px) rotate(${direction * 8}deg)` },
{ transform: `rotate(${-direction * 3}deg)` },
{ transform: "none" },
], { duration: 560, delay, easing: "ease-out" });
}
}
return () => { cancelAnimationFrame(frame); impact?.cancel(); };
}, [area, reaction, payload, now]);
if (payload.kind !== "reaction") return null;
const gift = Boolean(payload.targetPeerId && !isThrow(payload.reaction));
return <div ref={flight} className={`lr-reaction-flight ${payload.targetPeerId ? gift ? "is-gift" : "is-throw" : "is-expression"}`}
style={{ visibility: "hidden", "--reaction-duration": `${REACTION_DURATION_MS}ms` } as CSSProperties} aria-hidden="true">
<span className={`lr-reaction-projectile is-${payload.reaction}`} onAnimationStart={event => {
// CSS restarts after display:none or reduced motion. Event expiry still
// owns time, including a late mount when effects are shown again.
for (const animation of (event.target as Element).getAnimations()) {
if (animation instanceof CSSAnimation && animation.animationName.startsWith("lr-reaction-")) {
animation.currentTime = Math.min(REACTION_DURATION_MS, Math.max(0, REACTION_DURATION_MS - (reaction.expiresAt - now())));
}
}
}}>{gift && <span className="lr-reaction-from"><span>{reaction.sender.displayName}</span><Glyph name="arrowDown" size={12} /></span>}
<ReactionAccents reaction={payload.reaction} seed={identityHash(`${reaction.sender.peerId}:${reaction.id}`)} />
<span className="lr-reaction-glyph"><ReactionIcon reaction={payload.reaction} /></span></span>
</div>;
}
src/client/components/living/Tooltip.tsx
+5 -3
View File
@@ -294,15 +294,17 @@ export function Tooltip({
if (!interactionOpen) return;
// Hover does not move keyboard focus into the trigger.
const dismiss = (event: KeyboardEvent) => {
if (event.key !== "Escape") return;
if (event.key !== "Escape" || event.isComposing) return;
// Close every hint channel, including a hover panel suppressed by a
// focused neighbour. Only visible guidance consumes the outer action.
const style = tipRef.current && getComputedStyle(tipRef.current);
if (style?.visibility === "visible" && style.pointerEvents !== "none") event.preventDefault();
dismissPanel();
};
document.addEventListener("keydown", dismiss);
return () => document.removeEventListener("keydown", dismiss);
// Guidance consumes Escape before its containing window, regardless of
// which surface mounted first. Outer handlers respect defaultPrevented.
document.addEventListener("keydown", dismiss, true);
return () => document.removeEventListener("keydown", dismiss, true);
}, [interactionOpen]);
useEffect(() => {
src/client/components/living/floating-panel.css
+33
View File
@@ -0,0 +1,33 @@
.lr-floating-surface { color: var(--ink); background: var(--paper); border: 1px solid var(--control-edge); border-radius: 24px; box-shadow: 0 12px 36px rgb(16 26 44 / .2), var(--control-shadow); }
.lr-floating-panel { position: fixed; inset: auto 12px 12px auto; max-width: calc(100vw - 24px); max-height: calc(100dvh - 24px); margin: 0; padding: 16px; overflow: auto; overscroll-behavior: contain; transform-origin: right bottom; }
.lr-floating-panel:popover-open { display: flex; flex-direction: column; animation: lr-floating-enter 180ms var(--spring); }
.lr-floating-panel:focus { outline: none; }
.lr-floating-panel:focus-visible { outline: 2px solid var(--action); outline-offset: -2px; }
.lr-floating-header { display: flex; align-items: center; gap: 8px; min-height: 56px; flex-shrink: 0; margin: -16px -16px 0; padding: 6px 10px 6px 24px; border-bottom: 1px solid var(--wall-2); border-radius: 24px 24px 0 0; cursor: grab; user-select: none; touch-action: none; }
.lr-floating-header:active { cursor: grabbing; }
.lr-floating-header:focus-visible { outline: 2px solid var(--action); outline-offset: -3px; }
.lr-floating-title { display: flex; flex: 1; min-width: 0; align-items: center; gap: 7px; font-size: 14px; }
.lr-floating-title svg { flex: none; color: var(--muted); }
.lr-floating-title b { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.lr-floating-position { position: relative; flex: none; }
.lr-floating-position > summary { display: flex; align-items: center; gap: 5px; min-height: 40px; padding: 0 9px; border-radius: 12px; font-size: 12px; color: var(--muted); cursor: pointer; list-style: none; }
.lr-floating-position > summary::-webkit-details-marker { display: none; }
.lr-floating-position > summary:is(:hover, :focus-visible), .lr-floating-position[open] > summary { color: var(--ink); background: var(--control-hover); }
.lr-floating-positions { position: absolute; z-index: 2; top: calc(100% + 6px); right: 0; display: grid; grid-template-columns: 1fr 1fr; gap: 6px; width: 180px; padding: 8px; border: 1px solid var(--control-edge); border-radius: 18px; background: var(--paper); box-shadow: var(--control-shadow), 0 8px 24px #0002; }
.lr-floating-positions button { display: grid; justify-items: center; gap: 6px; min-height: 60px; padding: 8px 4px; border: 0; border-radius: 10px; color: var(--ink); background: transparent; font-size: 11px; }
.lr-floating-positions button:is(:hover, :focus-visible) { color: var(--action); background: var(--control-hover); }
.lr-floating-positions .lr-floating-size { border-top: 1px solid var(--wall-2); }
.lr-floating-resize { position: absolute; z-index: 3; display: grid; place-items: center; width: 28px; height: 28px; color: var(--muted); cursor: nwse-resize; touch-action: none; user-select: none; }
.lr-floating-resize.is-nw { top: 1px; left: 1px; transform: rotate(180deg); }
.lr-floating-resize.is-se { right: 1px; bottom: 1px; }
.lr-floating-resize:is(:hover, :focus-visible) { color: var(--action); }
.lr-floating-resize:focus-visible { outline: 2px solid var(--action); outline-offset: -4px; border-radius: 10px; }
@keyframes lr-floating-enter { from { opacity: 0; transform: translateY(8px) scale(.97); } to { opacity: 1; transform: none; } }
.lr-popover-close { display: grid; place-items: center; width: 36px; height: 36px; flex: none; background: transparent; color: var(--muted); border: 0; border-radius: 12px; cursor: pointer; }
.lr-popover-close:is(:hover, :focus-visible) { background: var(--control-hover); color: var(--ink); }
[data-comic-reduced-motion] .lr-floating-panel:popover-open { animation: none; }
@media (prefers-reduced-motion: reduce) { .lr-floating-panel:popover-open { animation: none; } }
@media (max-width: 420px) {
.lr-floating-panel { padding: 12px; }
.lr-floating-header { margin: -12px -12px 0; padding: 6px 10px 6px 24px; }
}
Loaded 100 of 141 files, more files were not shown because too many files have changed in this diff. Show more