diff --git a/.github/workflows/website.yml b/.github/workflows/website.yml index 62d8083..9d0d1ea 100644 --- a/.github/workflows/website.yml +++ b/.github/workflows/website.yml @@ -20,7 +20,10 @@ jobs: deploy: if: >- github.ref == 'refs/heads/main' && - (github.event_name == 'workflow_dispatch' || github.event.workflow_run.event == 'push') + (github.event_name == 'workflow_dispatch' || + (github.event.workflow_run.event == 'push' && + github.event.workflow_run.head_branch == 'main' && + github.event.workflow_run.conclusion == 'success')) runs-on: ubuntu-latest timeout-minutes: 5 permissions: @@ -33,7 +36,7 @@ jobs: steps: - uses: actions/checkout@v6 with: - ref: main + ref: ${{ github.event.workflow_run.head_sha || github.sha }} - uses: actions/setup-node@v6 with: node-version-file: .node-version @@ -49,5 +52,16 @@ jobs: with: path: build/site retention-days: 1 + - name: Keep older runs from replacing the current website + id: current + shell: bash + run: | + git fetch --no-tags origin main + if [ "$(git rev-parse HEAD)" = "$(git rev-parse origin/main)" ]; then + echo "deploy=true" >> "$GITHUB_OUTPUT" + else + echo "Skipped an outdated website build. Run Website again for current main." >> "$GITHUB_STEP_SUMMARY" + fi - uses: actions/deploy-pages@v4 + if: steps.current.outputs.deploy == 'true' id: deployment diff --git a/README.md b/README.md index 7e44b95..58e85d9 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ watch in their browsers. ## Features - **Watch without installing.** Friends join by invitation in a desktop or mobile browser. -- **Share from Piik App or an existing Piik site.** Capture a screen, window or browser tab; available sources and audio depend on the platform. +- **Share from Piik App or an existing Piik site.** Choose a screen, window, browser tab or camera; available sources and audio depend on the platform. - **Direct connections first.** Media travels between participants where possible (P2P). A self-hosted server can provide automatic media forwarding (SFU) as a fallback. - **Rooms you control.** Invitations, room codes and optional room passwords, for one host and up to 20 viewers. - **Flexible viewing.** Light and dark themes, playback controls, picture-in-picture and a connection topology view. @@ -55,7 +55,7 @@ you can also create a room in the online version. | Start here | What you need to do | | --- | --- | -| [Use Piik online](https://demo.piik.tv) | Open it in a desktop browser to create a room and share. | +| [Use Piik online](https://demo.piik.tv) | Share from a desktop browser, or choose Camera on a phone with browser camera support. | | [Download Piik App](https://piik.tv/#download) | Extract and open the App, choose **Public invite**, then **Open Piik** to create a temporary room. | | [Host your own site](./docs/operations/self-hosting.md) | Advanced: deploy Piik Server on a Linux x64 server with your own domain, then connect through Piik App or a browser. | @@ -64,6 +64,9 @@ you can also create a room in the online version. 1. Select **Start sharing**, then choose the picture and audio to share. 2. Select **Copy invite link** and send it to your friends. Keep the sharing tab open. +Turn on the **Microphone** for commentary; its settings let you choose the input +device and adjust your voice volume. + Piik follows your system language: Chinese or English, with English as the fallback. Use **中 / EN / ✦** in the header to switch language or try the visual mode. The App offers a local room, a temporary public link, or a connection to your own diff --git a/README.zh-CN.md b/README.zh-CN.md index a14ebf3..511d839 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -30,7 +30,7 @@ Piik 是一个免费开源的屏幕共享工具，适合游戏围观、一起看 ## 功能 - **观看无须安装。** 朋友通过邀请链接，用电脑或手机浏览器加入。 -- **用 Piik App 或已有 Piik 站点分享。** 支持屏幕、窗口和浏览器标签页；可选来源与音频能力取决于平台。 +- **用 Piik App 或已有 Piik 站点分享。** 支持屏幕、窗口、浏览器标签页和摄像头；可选来源与音频能力取决于平台。 - **优先直接连接。** 画面优先在设备之间传输（P2P），自建站点可启用媒体转发（SFU）自动兜底。 - **房间由你管理。** 支持邀请链接、房间号和可选的房间密码，一位房主、最多 20 位观众。 - **灵活的观看方式。** 支持深浅主题、播放控制、画中画和连接拓扑查看。 @@ -52,7 +52,7 @@ Piik 是一个免费开源的屏幕共享工具，适合游戏围观、一起看 | 从这里开始 | 需要做什么 | | --- | --- | -| [使用在线版](https://demo.piik.tv) | 在电脑浏览器打开，直接创建房间并分享。 | +| [使用在线版](https://demo.piik.tv) | 用电脑浏览器分享画面；手机浏览器支持摄像头访问时，也可选择摄像头分享。 | | [下载 Piik App](https://piik.tv/?lang=zh-CN#download) | 解压并启动 Piik App，选择**公网邀请**，点击**进入 Piik**，创建临时房间。 | | [部署自己的站点](./docs/operations/self-hosting.zh-CN.md) | 进阶使用：在 Linux x64 服务器上部署 Piik Server、配置自己的域名，再用 Piik App 或浏览器连接。 | @@ -61,6 +61,8 @@ Piik 是一个免费开源的屏幕共享工具，适合游戏围观、一起看 1. 点击**开始分享**，选择要分享的画面和声音。 2. 点击**复制邀请链接**，发给朋友。分享期间保持标签页打开。 +分享时还可以开启**麦克风**解说，在旁边的设置中选择设备、调整人声音量。 + Piik 默认跟随系统语言，中文系统显示简体中文，其他语言显示英文。 顶部的 **中 / EN / ✦** 可切换中英文或体验纯视觉模式。 Piik App 提供本地房间、临时公网邀请和连接已有站点三种模式，分享期间需保持 App 运行。 diff --git a/docs/README.md b/docs/README.md index 5cad289..3ef26b1 100644 --- a/docs/README.md +++ b/docs/README.md @@ -98,6 +98,7 @@ and [website preview and publishing](./operations/website.md). | Host and viewer behavior | [Presentation and lifecycle](./standards/presentation-lifecycle.md) | | Why a design was chosen | [Architecture decisions](./adr/) | | Measurements and platform limits | [Research](./research/) and [verification status](./verification-status.md) | +| Camera and Host microphone candidate | [Capture scope and limits](./research/camera-and-microphone.md) | ## Shared conventions diff --git a/docs/adr/0009-optional-nat-prediction.md b/docs/adr/0009-optional-nat-prediction.md index 98d8c76..72500b7 100644 --- a/docs/adr/0009-optional-nat-prediction.md +++ b/docs/adr/0009-optional-nat-prediction.md @@ -19,10 +19,10 @@ connection acquisition reuses the route controller's existing opportunity ledger ## Decision -1. `NAT_PREDICTION_ENABLED` defaults to `false`. A disabled deployment does not - expose the Host control and the server rejects NAT prediction in the room - policy. An enabled deployment exposes `NAT traversal`; the pre-share switch - defaults on, remains Host-controllable, locks while sharing, and is not +1. `NAT_PREDICTION_ENABLED` defaults to `false`. A disabled deployment shows the + Host control locked off and the server rejects NAT prediction in the room + policy. An enabled deployment exposes `Enhanced NAT traversal`; the pre-share + switch defaults on, remains Host-controllable, locks while sharing, and is not persisted beyond that share generation. 2. An enabled Site derives ports 3479 and 3480 from its first ordinary STUN authority on UDP 3478. Public-link App mode instead supplies one ordinary diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index cf54b67..776a68a 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -41,15 +41,19 @@ With just a four-digit room code, open the same Piik site and choose The host decides whether code entry is available; ask for an invitation link if you cannot join by code. +Hosts can turn off **Skip passwords** beside the link to copy the ordinary room +address. Visitors follow the site's and room's access checks; invite-only rooms +require the invitation credential. This switch does not revoke existing invites. + If an old link no longer works, ask the host for the current invitation. App Local rooms and temporary public invitations end when that App run stops. ## Use Piik online -1. Open [demo.piik.tv](https://demo.piik.tv) in a desktop browser that supports - screen sharing. -2. Select **Start sharing**, choose a screen, window or tab, and enable audio if - needed and offered by the browser. +1. Open [demo.piik.tv](https://demo.piik.tv) in your browser. +2. Select **Start sharing**, then **Browser** for a screen, window or tab, or + **Camera** where your computer or phone browser supports it. Enable source + audio if the picker offers it. 3. Confirm the preview and select **Copy invite link**. Open that link on a second device or send it to a friend. 4. Select **Stop sharing** when finished. @@ -67,7 +71,7 @@ restrictive networks may load the page but block video. The launcher opens in your system browser. 3. Choose **Public invite** for friends outside your local network, then select **Open Piik**. Other modes are explained below. -4. Choose **Start sharing**, then select **Browser**, **Apps / Windows**, or +4. Choose **Start sharing**, then select **Browser**, **Camera**, **Apps / Windows**, or **Screens** as available. Pick the actual source and sound option. 5. Copy and send the room invitation. Keep both the App and sharing tab open. @@ -91,8 +95,12 @@ See the [App guide](../../cmd/piik-app/README.md) for package and runtime detail | **Public invite** | Friends are elsewhere and you want a temporary invitation. | Needs Internet access. The public address lasts for this App run. | | **Connect to Site** | Your group already has a Piik site. | Enter its address; the App adds capture capabilities to the site's sharing page. | +For Local room and Public invite, expand **Site passphrase** to set an optional +access passphrase. + For **Connect to Site**, enter the full site address, such as -`https://demo.piik.tv`, add its passphrase if required, then choose **Open Piik**. +`https://demo.piik.tv`, then choose **Open Piik**. Enter the site's passphrase +on the page that opens if asked. After opening the site through the App once, the same browser remembers that entry. Keep the App running with that site configured and you can visit the address directly. **Start sharing** opens the selector; choose **Apps / Windows** @@ -114,9 +122,10 @@ or [Piik App](#share-with-piik-app). Hosting a site yourself is an advanced opti follow the [self-hosting guide](../operations/self-hosting.md) for server, HTTPS and network setup. -1. Open your group's Piik site on your computer. Enter its site passphrase if asked. -2. Choose **Start sharing**. In the browser picker, select the screen, window or - tab you want friends to see, and enable audio if offered. +1. Open your group's Piik site. Enter its site passphrase if asked. +2. Choose **Start sharing**, then **Browser** to select a screen, window or tab + in the browser picker, or **Camera** where your computer or phone browser supports it. + Enable source audio if the picker offers it. 3. Check the preview, then choose **Copy invite link** and send it to your friends. 4. Keep the sharing tab open. Use **Pause sharing**, **Switch source**, or **Stop sharing** when you need them. @@ -130,6 +139,21 @@ The site manages rooms and invitations; the App supplies native capture. If its operator has enabled SFU forwarding, turn off **Privacy mode** before sharing to allow that fallback. The project's online site uses P2P only. +## Add your voice + +While sharing, select **Microphone** below the picture and +allow microphone access. Select it again to mute. Open **Sharing settings** below the picture +to choose a microphone and adjust its input volume. +The default volume is 100%, with up to 200% available. +Source audio and your voice reach viewers together. The Host preview stays muted. + +Camera sharing does not enable the microphone automatically. App **Apps / Windows** +and **Screens** capture use the same controls. Leave the device on **Default device** +or choose a specific microphone. In the **Camera** tab, allow access if asked, +then select a picture card to start sharing. Device names and previews depend on +browser permissions; while a camera is shared, other cameras are listed by name. +Use headphones to keep speaker sound from feeding back into the microphone. + ## When something gets in the way For sound, capture, startup or playback issues, see [Troubleshooting](./troubleshooting.md). diff --git a/docs/guide/getting-started.zh-CN.md b/docs/guide/getting-started.zh-CN.md index da8654a..ef3ef23 100644 --- a/docs/guide/getting-started.zh-CN.md +++ b/docs/guide/getting-started.zh-CN.md @@ -38,13 +38,17 @@ Piik 支持一位房主分享屏幕，最多 20 位受邀观众通过浏览器 你可能需要站点口令和房间密码。能否通过房间号加入由房主决定； 无法加入时，请朋友发邀请链接。 +房主可关闭链接旁的 **免密邀请**，复制普通房间地址。访客打开地址后按站点和房间设置 +验证口令或密码；仅限邀请的房间需保留凭证。这个开关不会撤销已发出的邀请。 + 旧链接失效了，就请房主重新发一份当前邀请。 App 的本地房间和临时公网邀请会随本次 App 退出而结束。 ## 使用在线版 -1. 在支持屏幕共享的电脑浏览器中打开 [demo.piik.tv](https://demo.piik.tv)。 -2. 点击 **开始分享**，选择屏幕、窗口或标签页；如需声音且浏览器提供该选项，启用音频。 +1. 在浏览器中打开 [demo.piik.tv](https://demo.piik.tv)。 +2. 点击 **开始分享**，选择 **浏览器** 分享屏幕、窗口或标签页；也可在电脑或手机上选择 + **摄像头**（需浏览器支持访问）。如需来源声音且选择器提供该选项，启用音频。 3. 确认预览后，点击 **复制邀请链接**，用另一台设备打开，或发送给朋友。 4. 分享结束后，点击 **停止分享**。 @@ -60,7 +64,7 @@ App 的本地房间和临时公网邀请会随本次 App 退出而结束。 启动页会在系统浏览器里打开。 3. 邀请不在同一局域网的朋友时，选择 **公网邀请**，再点击 **进入 Piik**。 其他模式的用途见下表。 -4. 点击 **开始分享**，按可用选项选择 **浏览器**、**程序 / 窗口** 或 **屏幕**， +4. 点击 **开始分享**，按可用选项选择 **浏览器**、**摄像头**、**程序 / 窗口** 或 **屏幕**， 然后选定要分享的画面与声音。 5. 复制房间邀请发给朋友。分享期间保持 App 和分享标签页打开。 @@ -83,8 +87,10 @@ Linux 原生采集还需要[系统组件](../../native/capture/linux/README.md) | **公网邀请** | 朋友在别处，想临时发个邀请。 | 需要互联网连接；公网地址只在本次 App 运行期间有效。 | | **连接站点** | 你们已经有一个 Piik 站点。 | 输入站点地址，在原有分享页面使用 App 的采集能力。 | -使用 **连接站点** 时，填写完整地址，例如 `https://demo.piik.tv`；如果站点有访问口令， -也一并填写，然后点击 **进入 Piik**。通过 App 打开过一次后，同一浏览器会记住这个入口。 +本地房间和公网邀请可展开 **站点口令** 设置可选的访问口令。 + +使用 **连接站点** 时，填写完整地址，例如 `https://demo.piik.tv`，然后点击 **进入 Piik**。 +如果站点要求访问口令，在打开的站点页面输入。通过 App 打开过一次后，同一浏览器会记住这个入口。 保持 App 运行并使用该站点配置，之后就能直接访问网址。点击 **开始分享**， 在选择器里选 **程序 / 窗口** 或 **屏幕**，即可使用 App 采集。 浏览器询问本地网络权限时，请允许访问。换浏览器、使用无痕窗口或清除站点数据后， @@ -101,9 +107,9 @@ Linux 原生采集还需要[系统组件](../../native/capture/linux/README.md) 也可以使用项目提供的[在线版](#使用在线版)或 [Piik App](#用-piik-app-分享)。 自建站点属于进阶操作，服务器、HTTPS 和网络端口的配置步骤见[部署指南](../operations/self-hosting.zh-CN.md)。 -1. 在电脑上打开 Piik 站点。如果站点设置了口令，先输入口令。 -2. 点击 **开始分享**，在浏览器弹窗里选择屏幕、窗口或标签页； - 如需声音且浏览器提供该选项，启用音频。 +1. 打开 Piik 站点。如果站点设置了口令，先输入口令。 +2. 点击 **开始分享**，选择 **浏览器** 后在系统弹窗里选择屏幕、窗口或标签页； + 浏览器支持时，电脑和手机也可选择 **摄像头**。如需来源声音且选择器提供该选项，启用音频。 3. 确认预览画面后，点击 **复制邀请链接** 发给朋友。 4. 保持分享标签页打开。需要时使用 **暂停分享**、**切换来源** 或 **停止分享**。 @@ -116,6 +122,17 @@ Linux 原生采集还需要[系统组件](../../native/capture/linux/README.md) 如果管理员已启用 SFU 转发，房主可在开始分享前关闭 **隐私模式**，允许使用这条兜底线路。 项目提供的在线版仅使用 P2P。 +## 加上自己的声音 + +分享时，点击画面下方的 **麦克风**，并允许访问。 +再次点击即可静音。在画面下方的 **分享设置** 中可以选择麦克风和调节输入音量，默认 100%，最高可调至 200%。 +观众会同时听到来源声音和你的解说；房主本机预览保持静音。 + +摄像头分享不会自动开启麦克风。App 的 **程序 / 窗口** 和 **屏幕** 采集也使用同一组控件， +可以保留「默认设备」，也可以指定麦克风。在 **摄像头** 页签中允许访问后，点击对应的画面卡片开始分享。 +设备名称和可用预览取决于浏览器权限；正在分享摄像头时，其他摄像头只显示名称。 +建议佩戴耳机，避免扬声器声音被麦克风再次收进去。 + ## 遇到问题了 声音、采集、启动或播放遇到问题时，先看[问题排查](./troubleshooting.zh-CN.md)。 diff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.md index 1ad6276..fc7cc0d 100644 --- a/docs/guide/troubleshooting.md +++ b/docs/guide/troubleshooting.md @@ -9,7 +9,8 @@ collect a diagnostic report if the problem persists. | What you see | Try this | | --- | --- | -| Picture but no sound | Unmute the video. The host should choose a source with shareable audio; if sound was disabled, stop and start sharing with it enabled. App window/screen capture keeps that setting when switching sources. | +| Picture but no source sound | Unmute the video. The host should choose a source with shareable audio and enable sound in the source picker. | +| Cannot hear the host's voice | The host can enable **Microphone** below the picture, then check the selected input and its volume in **Sharing settings → Sound**. Check the device connection and microphone permission. If the control is unavailable, update the App and refresh the page. | | No screen picker | Allow the browser or App to record the screen when the OS asks. Browser capture needs HTTPS or `localhost`; try sharing from a desktop computer. | | No App windows or screens listed | Check the message in the source picker, then follow [App source troubleshooting](#app-windows-or-screens-are-missing). **Browser** → **Browser picker** also offers browser capture. | | Yellow outline around the shared window or screen | This is Windows' capture indicator. See [capture borders](#yellow-capture-border-on-windows) for Windows 11 controls and an optional Windows 10 workaround. | @@ -51,6 +52,20 @@ On supported Windows versions, the App source picker offers **Show capture borde off by default. Windows permissions or another active capture can still require the border. Windows 10 does not provide this control for its capture API. +### Windows 11: the border is still visible + +1. Select **Apps / Windows** or **Screens** in Piik and leave **Show capture border** + off. A Browser-selected source uses the browser's own capture indicator. +2. Stop other apps or tabs capturing the same window/display, then restart sharing. +3. If Windows denied borderless capture, review **Screenshot borders** in Windows + privacy settings. Its settings page can also be opened with + `ms-settings:privacy-graphicscapturewithoutborder` from **Win+R**, when available. + +Windows requires consent and can retain the border when another capture requests +it ([Microsoft's API documentation](https://learn.microsoft.com/en-us/uwp/api/windows.graphics.capture.graphicscapturesession.isborderrequired)). +If it persists, include the Windows version, selected source type and an +[App diagnostic report](../../cmd/piik-app/README.md#diagnostics) with your feedback. + ### Optional Windows 10 workaround [DWM Custom Projection Border](https://windhawk.net/mods/dwm-custom-projection-border) diff --git a/docs/guide/troubleshooting.zh-CN.md b/docs/guide/troubleshooting.zh-CN.md index f782a6f..322ccfd 100644 --- a/docs/guide/troubleshooting.zh-CN.md +++ b/docs/guide/troubleshooting.zh-CN.md @@ -9,7 +9,8 @@ | 你遇到的情况 | 可以先试什么 | | --- | --- | -| 有画面，没声音 | 先取消视频静音。房主应选择支持分享声音的来源；如果开始时关闭了声音，请停止后重新开始并打开声音。通过 App 采集窗口或屏幕时，切换来源会保留原来的声音设置。 | +| 有画面，没来源声音 | 先取消视频静音。房主应选择支持分享声音的来源，并在选源器中打开声音。 | +| 听不到房主说话 | 房主可在画面下方开启 **麦克风**，在 **分享设置 → 声音** 中检查输入设备和音量，并确认设备连接与麦克风权限。如控件不可用，请更新 App 并刷新页面。 | | 没有屏幕选择弹窗 | 按系统提示允许浏览器或 App 录制屏幕。浏览器采集需要 HTTPS 或 `localhost`，请在电脑上尝试分享。 | | App 的窗口或屏幕列表为空 | 先查看选源器中的提示，再按 [App 来源排查](#app-找不到窗口或屏幕) 操作；也可以选 **浏览器** → **浏览器选择器** 使用浏览器采集。 | | 分享的窗口或屏幕周围出现黄框 | 这是 Windows 的采集提示。查看[采集边框说明](#windows-分享时出现黄框)，了解 Windows 11 的开关和 Windows 10 的可选处理方法。 | @@ -47,6 +48,19 @@ Windows 用黄色边框标出正在采集的窗口或屏幕。在支持的 Windo App 选源器提供 **显示采集边框** 开关，默认关闭；系统权限或其他正在进行的采集 仍可能要求显示边框。Windows 10 的采集接口没有这个开关。 +### Windows 11 关闭开关后仍有黄框 + +1. 从 Piik 的 **程序 / 窗口** 或 **屏幕** 列表选源，保持 **显示采集边框** 关闭。 + 通过浏览器选源时，采集标识由浏览器管理。 +2. 停止其他软件或标签页对同一窗口、屏幕的采集，再重新开始分享。 +3. 如果之前拒绝了无边框采集权限，在 Windows 隐私设置中检查 **屏幕截图边框**。 + 支持该设置页的系统也可以按 **Win+R**，输入 `ms-settings:privacy-graphicscapturewithoutborder` 打开。 + +Windows 需要用户授权；其他采集要求显示边框时，边框仍会保留 +（[微软接口说明](https://learn.microsoft.com/en-us/uwp/api/windows.graphics.capture.graphicscapturesession.isborderrequired)，英文）。 +仍未解决时，反馈中请附上 Windows 版本、选源方式和 +[App 诊断报告](../../cmd/piik-app/README.zh-CN.md#诊断)。 + ### Windows 10 可选处理方法 [DWM Custom Projection Border](https://windhawk.net/mods/dwm-custom-projection-border) diff --git a/docs/operations/website.md b/docs/operations/website.md index 4711e25..0a73475 100644 --- a/docs/operations/website.md +++ b/docs/operations/website.md @@ -133,9 +133,10 @@ Verify the latest redirects and package checksums after releases. 3. Set the approved `piik.tv` custom domain in Pages settings and configure its DNS using GitHub's [current domain instructions](https://docs.github.com/en/pages/configuring-a-custom-domain-for-your-github-pages-site/managing-a-custom-domain-for-your-github-pages-site). Actions publishing does not need a `CNAME` file in the source tree. -4. Accepted main pushes run CI first; its completion triggers **Website**, which - checks out current main. Manual **Website** dispatch from `main` republishes - the site when needed. Download links resolve through each provider, so +4. Accepted main pushes run CI first; success triggers **Website**, which checks + out that CI run's exact commit. Manual dispatch uses its selected `main` commit. + Before publishing, both paths skip a build that is no longer current main; + dispatch again for the latest commit when needed. Download links resolve through each provider, so publishing a package or retrying its mirror needs no link refresh. Only `build/site/` is uploaded; website work does not package App/Server. Verify HTTPS, both languages, relative assets and per-platform download links. diff --git a/docs/project-memory.md b/docs/project-memory.md index ec69dff..a3e9549 100644 --- a/docs/project-memory.md +++ b/docs/project-memory.md @@ -1,10 +1,11 @@ # Project Memory -Last updated: 2026-09-11 +Last updated: 2026-09-21 Piik is private, low-latency screen sharing for one Host and up to 20 authenticated friends. The current product surface is Web Host, Web Viewer, and -Browser relay; desktop and mobile Browsers are Viewer targets. It is not a +Browser relay; desktop and mobile Browsers are Viewer targets, and camera-capable +mobile Browsers can also host. It is not a public broadcast service. Hosted and App run one shared Go core ([ADR-0012](./adr/0012-shared-go-backend-core.md)); Node and Vite build the Browser UI and are not a runtime. Source includes a cross-platform App that diff --git a/docs/research/camera-and-microphone.md b/docs/research/camera-and-microphone.md new file mode 100644 index 0000000..221a640 --- /dev/null +++ b/docs/research/camera-and-microphone.md @@ -0,0 +1,117 @@ +# Camera And Host Microphone + +Status: integrated into the ordinary Host UI and accepted for release. +Remaining device coverage belongs to [TODO](../todo.md#awaiting-device-or-reporter-evidence). + +## Scope And Ownership + +The existing source picker offers Browser, Camera, Window and Screen in that +order, defaulting to Window when an App is detected. Without an App, it shows +only Browser and Camera, defaulting to Browser. Mobile browsers can choose Camera +when screen capture is unavailable. +There is one active picture; capture, replacement, pause and stop retain their +existing Host session and route owners. Camera permission never includes an +implicit microphone request. HTTPS or localhost is required for device capture. + +The Host microphone mixes commentary with Browser source audio into one outgoing +audio track through Web Audio. The existing P2P, relay and SFU senders borrow +that stream. Microphone mute preserves the mixed track and source replacement +preserves microphone intent. Denial or unplugging leaves healthy video alone; +retiring the share closes audio resources and discards late permission results. +The Host preview stays muted. Viewers receive the combined sound with their +existing volume control and never request microphone permission. + +App native screen/window capture mixes selected-device microphone PCM with source +audio before the existing Opus encoder. Windows uses WASAPI, macOS uses +AVAudioEngine and Linux uses PulseAudio/PipeWire through GStreamer. Platform +converters supply 48 kHz stereo PCM; one 20 ms output clock consumes each input +once, with a bounded queue and silence for missing samples. This bounds backlog; +it does not replace platform resampling or establish long-term clock-drift and +echo-cancellation performance. + +New App/page pairs opt into a stable mixed audio track from share startup, even +with source sound disabled. Opening or muting the microphone and replacing the +source preserve that track, encoder, preview and route owners. Source sound and +microphone availability remain separate reported facts. Old Apps/pages retain +the existing capture contract through capability negotiation; microphone capture +starts only after an explicit action. Permission waiting is cancellable and does +not hold the control reader. Device loss retires only that audio input. + +Windows native capture and the App-to-Viewer path have local acceptance evidence. +macOS/Linux microphone compilation is checked by native-platform packaging. +Real audio-level/echo checks, macOS/Linux device acceptance and actual phone +capture remain physical evidence limits; synthetic tests do not establish +those results. The owner authorized release with those limits retained. The +Host preview remains muted on every path. + +This scope adds no room messages, ports, chat, Viewer microphone or emoji +interactions. Microphone, pause, source replacement and stop share one action +row below the picture, outside the playback controls and television status. +Their order and meaning stay the same in Chinese, English and pure-visual modes. +Source-sound details describe the raw capture input; microphone intent and +Viewer playback volume are separate facts. Pausing the share silences the mixed +output and temporarily disables microphone toggling without clearing its intent. +Sharing settings open from the action dock. Picture presets/parameters and sound +form separate groups, with microphone device/input gain under sound and a further +disclosure for connection/codec options. The panel expands in page flow, pushing +the couch down; video presets remain available before capture starts. +The dock keeps only its frequent mute action; camera selection stays with source selection. +The default follows the system; an explicit selection never silently falls back +to another device. Browser capture and native capture keep separate device IDs +for the page lifetime. Enumeration does not request permission. Browser labels +may appear only after the first grant; refresh and device-change events update +the list. A missing selected device stays visible as unavailable. +Switching an active microphone prepares the new input before retiring the old +one; failure leaves the old input running. The mixed output stays unchanged. +Camera selection belongs in the existing Camera tab and uses the same named +thumbnail cards as window/display selection. Entering the +Camera tab may request camera permission for local previews; it never requests +microphone access or starts a room. Preview inputs open serially and stop after +one bounded thumbnail. Leaving the tab or cancelling the picker stops owned +inputs and discards late permission results. Images last only for that picker. +While already sharing a camera, preview borrows the current video frame and +lists other devices without opening them, because some phones retire an active +camera when another opens. Missing thumbnails do not disable a known device. +The Host's existing source-replacement operation +owns the old camera's automatic end event until the pending capture settles; +failure with an already-ended old camera retires sharing normally. This does +not suppress a user's screen-share Stop action. Some phones require releasing +the old camera before opening another; seamless switching is not established. + +The input-volume control adjusts microphone input from +0–200% through smoothed input gain. It does not change source audio or Viewer playback volume, +reopen permission, or replace the outgoing track. Its value lasts for the Host +page lifetime; new page loads use 100%. Mute remains a separate direct action. +Volume settings use the existing settings disclosure without covering the picture +or participants. The microphone-volume comic describes outgoing +commentary, separately from the Viewer's local listening volume. + +## Local Review + +Use the ordinary development or embedded Server entry described in +[the contributor guide](../../CONTRIBUTING.md). +Start a share from the normal Host page, select a source, then join through its +ordinary invitation in another browser tab. Use headphones for same-device +microphone tests. Browser-emulated mobile layout and synthetic devices do not +establish actual iOS/Android permission, orientation, background or audio behavior. + +## Primary Technical References + +- [Google Meet audio controls](https://support.google.com/meet/answer/10409699?hl=en) + keeps microphone settings next to the frequent mute action. +- [OBS audio mixer](https://obsproject.com/kb/audio-mixer-guide) separates source + volume, mute and monitoring; Piik uses only the controls needed for commentary. + +- [Web Audio stream destination](https://www.w3.org/TR/webaudio-1.0/#MediaStreamAudioDestinationNode) + produces a single audio track from the audio graph. +- [Media Capture and Streams](https://www.w3.org/TR/mediacapture-streams/) + owns microphone permissions, device lifetime and echo-cancellation constraints. +- [Camera capture](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia) + and [screen capture](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia) + have separate permissions, capabilities and secure-context requirements. +- [WebRTC track replacement](https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpSender/replaceTrack) + describes reuse of negotiated media slots and when renegotiation is required. +- [WASAPI shared capture](https://learn.microsoft.com/en-us/windows/win32/api/audioclient/nf-audioclient-iaudioclient-initialize), + [AVAudioConverter resampling](https://developer.apple.com/documentation/technotes/tn3136-avaudioconverter-performing-sample-rate-conversions) + and [GStreamer audio mixing](https://gstreamer.freedesktop.org/documentation/audiomixer/audiomixer.html) + define the platform conversion and PCM mixing boundaries. diff --git a/docs/research/cross-platform-client-runtime.md b/docs/research/cross-platform-client-runtime.md index 606dfd2..eaf1216 100644 --- a/docs/research/cross-platform-client-runtime.md +++ b/docs/research/cross-platform-client-runtime.md @@ -158,6 +158,26 @@ totals as its portable core proof. Detailed Piik quality metrics enrich a development run when its source module is served, but their absence on a static deployment no longer erases valid native RTCStats. +### Public Invitation Startup + +The pinned cloudflared Quick Tunnel defaults to explicit QUIC, which disables +its HTTP/2 fallback. Piik requests `--protocol auto` and +`--max-edge-addr-retries 0`: try QUIC once, then let cloudflared fall back within +the existing 30-second startup budget. This affects HTTP/WebSocket control only; +media routes, tunnel identity and ordered shutdown keep their existing owners. +DNS failures, API rejection and networks blocking both transports can still fail. + +A 2026-09-21 Windows check used the packaged binary with an isolated UDP sink and +TCP forwarder. The previous arguments timed out at 30 seconds without trying TCP. +The candidate connected in about 11 seconds, served the public page and echoed a +WebSocket message, then joined the child on close. A separate ordinary-network +run registered in about five seconds but this host's public HTTPS/WSS requests +ended with EOF; end-to-end access on that path remains unverified. These checks +do not establish the cause of reports without matching diagnostics. Upstream +[Quick Tunnel setup](https://github.com/cloudflare/cloudflared/blob/2026.8.3/cmd/cloudflared/tunnel/quick_tunnel.go) +and [protocol fallback](https://github.com/cloudflare/cloudflared/blob/2026.8.3/supervisor/tunnel.go) +own the dependency behavior; Piik adds no separate retry loop. + ## Remaining Gates - Run the exact clean-revision package on macOS; Linux amd64 is proved. diff --git a/docs/research/native-client-lifecycle.md b/docs/research/native-client-lifecycle.md index 86ffa4a..d36ddb0 100644 --- a/docs/research/native-client-lifecycle.md +++ b/docs/research/native-client-lifecycle.md @@ -260,6 +260,24 @@ starts at build 20348, beyond consumer Windows 10 build 19045. The current capture helper therefore retains the system border there. Browser capture indicators remain Browser-owned. +On Windows 11 build 26200, a 2026-09-21 controlled native-window check confirmed +borderless permission was allowed and the default capture had no border. A +second session explicitly requiring a border made it visible; gracefully ending +that session cleared it while borderless capture continued. Force-killing the +bordered capture instead left the outline behind in this fixture, even after +the remaining capture ended. + +The stream owner previously connected parent cancellation directly to +`exec.CommandContext`, killing the child before its existing bounded stop +sequence could release the platform session. Parent cancellation now enters +that same `Stream.Close` sequence; an unresponsive process still has the existing +one-second kill deadline. Subprocess regression checks cover explicit stop, +parent cancellation, concurrent stop and an unresponsive child. A real WGC +capture cancelled through this Go owner exited normally and cleared the border. +This establishes a retirement defect and its local repair, not the cause of +every reported Windows 11 border: denied consent and other active captures +remain distinct Windows-owned limits. + An unofficial WGC-preserving option exists: [Windhawk's DWM Custom Projection Border](https://github.com/ramensoftware/windhawk-mods/blob/main/mods/dwm-custom-projection-border.wh.cpp) documents disabling the border on Windows 10 21H2. Source inspection confirms diff --git a/docs/standards/media-quality.md b/docs/standards/media-quality.md index 0bbd9e3..07135d7 100644 --- a/docs/standards/media-quality.md +++ b/docs/standards/media-quality.md @@ -12,12 +12,25 @@ production behavior and remaining acceptance. ## Capture And Controls -- The Web Host may share a display, application window, or Browser tab and may +- The Web Host may share a display, application window, Browser tab or camera and may stop, synchronously pause/resume audio and video, or switch source. - An App-launched Host explicitly chooses either that Browser capture path or one native screen/window enumerated by the packaged platform capture boundary. The latter uses one supported native codec path and never infers a target from a title. An ordinary Web Host does not probe localhost. +- Camera is a peer source choice in the same picker, including on phones. Only + the chosen source requests permission. Browser/camera Host microphone capture + is explicit and uses voice processing separately from screen audio; its mixer + produces one audio output for the existing media routes. Muting or losing the + microphone must not stop healthy video. Share retirement releases its devices + and rejects late permission results. Native App capture mixes source and + microphone PCM before its existing Opus encoder, preserving one output track + through mute, pause and source replacement. A capable App/page pair opts into + this output at startup; source sound and microphone state remain independent + facts. Browser preview audio never becomes the native mixing source. +- An audio-only update with the same capture video preserves the Browser video + sender, encoding state and startup evidence on both direct and SFU paths. + A new video source owns a new startup baseline; audio changes do not. - Windows native capture follows an explicitly stretched active display path for an entire display or a window covering that display, when the captured frame matches its desktop source dimensions. Other frames retain their own diff --git a/docs/standards/presentation-lifecycle.md b/docs/standards/presentation-lifecycle.md index 2dee695..52c1e86 100644 --- a/docs/standards/presentation-lifecycle.md +++ b/docs/standards/presentation-lifecycle.md @@ -46,14 +46,16 @@ versions of the product UI. A remembered App activation permits discovery when choosing a source or receiving media; it does not make App a prerequisite for sharing. Idle Host -pages hold no App control session. Ordinary Web entry opens the Browser picker -directly. Activated origins offer the shared source selector, initially on its -Browser tab unless App is already connected. Discovery never blocks Browser -selection or changes the chosen tab. Native tabs and refresh remain reachable -after absence or disconnection; pending attempts are shared, completed failures +pages hold no App control session. The shared source selector orders Browser, +Camera, Window and Screen, defaulting to Window after App detection. Without an +App it shows only Browser and Camera, defaulting to Browser. Discovery never +blocks Browser selection or overrides an explicitly chosen tab. Refresh remains +reachable after failed App discovery; pending attempts are shared, completed failures do not suppress later discovery. Cancelling the selector or ending a share releases its unused control session; an obsolete operation cannot install or -retire another operation's media. Optional native ingress uses an already +retire another operation's media. Camera thumbnails own only temporary preview +inputs; leaving their tab or picker retires them. An active shared camera is +borrowed, never reopened or stopped by preview. Optional native ingress uses an already available connection without another scan. If it is not ready at Browser startup, that share keeps Browser senders. diff --git a/docs/standards/rooms-access.md b/docs/standards/rooms-access.md index 5960e78..16b8e27 100644 --- a/docs/standards/rooms-access.md +++ b/docs/standards/rooms-access.md @@ -8,7 +8,8 @@ the storage decision; implementation detail belongs in code and tests. - A room has one Host and at most 20 authenticated Viewers. - Room codes are random free four-digit values from `1000..9999`. A code locates - a room; it is not a secret or permanent identity. + a room within its site authority; it is not globally unique, a secret or a + permanent identity. App public-link rooms belong to that App's local authority. - Every room has a 256-bit Host token. The server stores only its digest; the owning Browser keeps the raw token locally. - Each Host tab retains its current room in session storage. The origin keeps @@ -60,6 +61,12 @@ tabs or Browser sessions; tab duplication and opener initialization remain Browser behavior. The grant has no independent TTL and ends with the room or when the Host rotates or revokes it. +The Host can omit the grant from the displayed/copied link. This page-local +choice defaults to including it and does not change admission policy, revoke +existing invitations or clear a recipient's valid authorization. The ordinary +room address retains the server-provided site and room path; it follows site +access and code-entry rules. An invite-only room has no ordinary-address entry. + Code-only admission is independent of invitations: - `open` admits a site-authorized Viewer by room code; diff --git a/docs/standards/routing-transport.md b/docs/standards/routing-transport.md index 175ea94..c55a48d 100644 --- a/docs/standards/routing-transport.md +++ b/docs/standards/routing-transport.md @@ -34,10 +34,11 @@ share generation: - native-edge topology convergence is enabled by default. The Host may disable it before sharing; when disabled, quality evidence stays diagnostic and availability routing is unchanged; -- NAT traversal requires an exact three-destination STUN survey: a Site may use - self-hosted 3478/3479/3480, Public Link uses its bounded public survey, and +- NAT traversal enhancement requires an exact three-destination STUN survey: + a Site may use self-hosted 3478/3479/3480, Public Link uses its bounded public survey, and pure LAN supplies none. Its default-on Host switch augments Browser and Native - P2P edges. Native uses one media socket for IPv4 and, where available, IPv6 + P2P edges. This gate leaves ordinary ICE and Native gateway mapping enabled. + Native uses one media socket for IPv4 and, where available, IPv6 direct connections; its STUN survey and best-effort gateway mapping use IPv4. Availability and background direct acquisition share a bounded budget of three actual connection attempts per eligible parent/session opportunity. @@ -105,8 +106,8 @@ begins only after that exact candidate commits as the active route. Either preparing endpoint reports local rejection or connection-start failure to that same operation immediately; it does not wait out a silent candidate's deadline or start an independent recovery loop. -When NAT traversal is enabled, another operation may use the remaining -connection-attempt budget after rollback. The waiting display uses the actual +When NAT traversal enhancement is enabled, another operation may use the +remaining connection-attempt budget after rollback. The waiting display uses the actual server-issued attempt ordinal; it never counts time as an attempted connection. Committed Browser P2P recovery first restarts ICE on the same connection, then rebuilds that connection if necessary, within the existing two-request budget. diff --git a/docs/standards/visual-language.md b/docs/standards/visual-language.md index b9228eb..29d7ba8 100644 --- a/docs/standards/visual-language.md +++ b/docs/standards/visual-language.md @@ -116,6 +116,37 @@ on hover or keyboard focus, then compress on press. Inputs and option tiles share softer corners. Keep a distinct focus ring and a persistent selected state; disabled controls must not pretend to activate. Reduced motion keeps the colour, outline and selected position without decorative displacement. +The shared button uses the same pressed value for `aria-pressed` and its selected +treatment; active state remains distinct from hover in both themes. +Disclosure buttons keep a stable visible caption, while their expanded state and +accessible action describe opening or closing. Neutral panel borders stay quieter +than the controls inside them; nested diagnostic panels retain their softer fill. +Keep room identity, invitation and admission controls ahead of expanded topology +and connection details, so inspecting a route does not displace room actions. + +Sharing and local playback controls use the same quiet, compact button treatment. +The Host's persistent dock stays below the picture; the Viewer's playback bar +belongs to the video. Their similar appearance does not equate broadcast pause +with local playback pause. Waiting overlays use the space above the actual +playback bar, including when it wraps. Keep the primary status readable on +narrow screens; optional waiting copy yields first. Microphone mute, sharing +pause, source replacement and stop remain direct actions; stopping stays +visually separated from frequent actions. +One compact dock combines direct actions and the sharing-settings entry below +the television, above the couch. Its disclosure groups quality presets and picture +parameters with sound and microphone settings; presets remain available before +sharing starts. Separate the less frequent connection/codec options in a nested +disclosure. Expand in page flow, pushing later content down instead of covering +the picture or participants. Use two content groups on wide screens and stack +them on narrow screens; avoid another permanent row of settings cards. Keep +shared presentation separate from the existing setting/resource owners. +The microphone comic depicts Host commentary gain; +the Viewer's listening-volume comic remains a different action. + +Visual source choices (windows, screens and cameras) share named thumbnail cards, +including their focus, hover and refresh treatment. Microphone selection uses +device names in a compact selector. Shared enumeration does not make these two +selection tasks visually interchangeable. ## Cast And Objects @@ -130,6 +161,7 @@ Participant characters should leave room for that distinctive presence. | Shared or watched media | Small television, including antenna, body and feet | | Browser application | Browser chrome around content; not a second design of television | | A captured window or display | Window title bar or display stand, with the shared media metaphor kept distinct | +| A camera source or Host commentary | `camera` identifies the alternative picture source; `microphone` identifies the enabled Host voice input and `microphoneOff` its muted state. Neither replaces the speaker metaphor for source/playback sound. | | Server forwarding media | A server on the media path; opening a Site alone does not imply SFU | These are **identity/object colours**, not status. A green pawn remains green diff --git a/docs/status.md b/docs/status.md index 26d7ef4..fb7bdb2 100644 --- a/docs/status.md +++ b/docs/status.md @@ -1,6 +1,6 @@ # Current Status -Last updated: 2026-09-17 +Last updated: 2026-09-21 This is the compact execution/deployment index. Product modules own behavior, [verification status](./verification-status.md) owns unresolved physical limits, @@ -24,6 +24,11 @@ and Git/PRs own completed history. mascot, concrete media waits use their owning comic, and waiting captions are secondary to literal status. Website reduced-motion preference and player tooltip placement follow the same presentation owners. +- Camera sharing and Host microphone commentary are accepted product features. + Both Browser and native App audio use the existing media routes; device + selection and sharing settings keep their current resource owners. The + [capture assessment](./research/camera-and-microphone.md) owns platform behavior + and the remaining physical evidence limits. Website delivery includes a bilingual reader documentation center generated from the repository's guides alongside the homepage, with search and detailed diff --git a/docs/todo.md b/docs/todo.md index 1a34e0a..29db354 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -1,24 +1,46 @@ # Current TODO Ledger -Last reviewed: 2026-09-20 +Last reviewed: 2026-09-21 Only **Now** is executable. Product modules own behavior; Git/PRs own completed history. A parked idea is not implementation authority. ## Now -- [ ] **Review camera and Host microphone in the separate capture candidate.** - Keep device selection in the existing source/settings surfaces. Finish control - grouping, dark-theme feedback and microphone volume presentation; verify native - mixing, device replacement and mobile camera behavior before acceptance. - These features are excluded from the bug-fix release. +- [ ] **Complete manual accessible-name review.** Verify the empty video's + screen-reader output: Chromium exposes an unavailable-media + description despite the literal shared-picture label and no media error. Also + review accessible naming on disabled tooltip wrappers and the UI catalogue's + paired-character example; automated checks leave those for manual review. - [ ] **Refine the isolated room-interaction prototype.** Review chat, optional danmaku and participant-targeted reactions for usability, placement and intended cross-view synchronization. Preserve room authorization and media-route owners. + Accept its combined layout: sharing settings with the picture, wide-screen chat + beside participants, stacked chat on narrow screens and room actions before + diagnostics. Check its theater chat entry and participant menu on real devices; + keep chat available independently of sharing. + Retain current room creation: the first share creates a room, an existing room + can resume, and stopping media keeps the opted-in interaction session. Do not + add visit-triggered creation or a separate pre-share room-creation entry. This experiment does not authorize integration or publication. ## Awaiting Device Or Reporter Evidence +- [ ] **Camera and Host microphone device coverage.** The owner accepted the + sharing layout and authorized release with these physical limits recorded. + Verify real audio levels/echo, multiple-device replacement and native mixing on + target systems, plus phone camera permission, orientation and background use. + Keep the existing capture and route owners. The + [capture assessment](./research/camera-and-microphone.md) owns behavior and + separates bounded Windows/browser checks from untested device combinations. +- [ ] **App public-invitation startup field acceptance.** Retest the frequent + creation-failure report and the earlier + [#396 timeout](https://github.com/TNTcraftHIM/Piik/issues/396#issuecomment-5691465700) + with the updated App. A controlled UDP-blocked/TCP-available reproduction confirms + a dependency configuration defect; the existing tunnel now uses cloudflared's + bounded protocol fallback. [Runtime evidence](./research/cross-platform-client-runtime.md#public-invitation-startup) + separates this fix from DNS, provider and remote access failures. The reporting + environments still need version and diagnostic evidence before assigning a cause. - [ ] **Interruption during established viewing.** A Viewer reportedly returns to P2P connecting after watching for a while. Native receiver renegotiation, retired event delivery and SFU replacement have locally reproduced defects @@ -27,9 +49,11 @@ history. A parked idea is not implementation authority. report when present. The existing five-second Viewer membership grace during signaling loss is unchanged. This is distinct from the first-frame report below. - [ ] **Windows 11 capture border remains visible.** Identify the App/Browser - capture path, Windows build and capture-border permission result. A local - Windows 11 request succeeded; the reported failure remains unconfirmed. Check - system consent and other active captures before changing the WGC path. + capture path, Windows build and capture-border permission result. Local checks + reproduced a border surviving forced process termination; parent cancellation + now uses bounded graceful retirement. [Capture evidence](./research/native-client-lifecycle.md#windows-capture-borders) + verifies the repair and concurrent-capture behavior on Windows 11. The + reporter's cause remains unconfirmed; check consent and other active captures. - [ ] **Brief connection followed by repeated Viewer loss on v1.5.0.** One Viewer reportedly drops just after connection details appear, while other Viewers work; App public-link mode is suspected. Check signaling, first-frame @@ -37,23 +61,18 @@ history. A parked idea is not implementation authority. reproduction establishes one premature candidate replacement; the affected environment and paired diagnostics are still unavailable, so the reporting machine's cause remains unconfirmed. -- [ ] **App source-discovery field acceptance.** Retest the missing-window report - with the current App and page. Obtain OS/browser, version and paired Debug - reports to distinguish discovery, capture capability and enumeration failures. - Local permission and recovery checks do not establish the reporting machine's - cause. +- [ ] **App discovery and share-start field acceptance.** Retest the missing-window + report, an unreachable App despite its process running, and generic share-start + failure with the current App and page. OS/browser, version, selected source and + paired Debug reports are still unavailable. Distinguish site authorization, + browser permission, control capacity, enumeration and capture-start failures; + local permission and recovery checks do not establish the reporters' causes. - [ ] **Windows 32-bit candidate acceptance.** Verify the isolated `spike/windows-x86-capture` candidate's launch, capture/audio, memory pressure, source replacement and update links on a 32-bit Windows device. WOW64 Host/media checks establish only that environment. Reconcile its scoped SDK/toolchain and atomic-alignment changes with current main when accepted; the experiment is not part of this maintenance release. -- [ ] **Public invitation startup timeout.** The reporter in - [#396](https://github.com/TNTcraftHIM/Piik/issues/396#issuecomment-5691465700) - confirmed Local startup is fixed, but separately reported that the public - invitation service did not connect within 30 seconds. Trace tunnel startup, - network reachability and the existing timeout/error feedback using a current - Debug report; the Local fix does not establish this separate failure's cause. - [ ] **Windows launcher exit after opening the page.** A user reports that the mode-selection page opens, then the App console reports `Piik App could not open its launcher: exit status 0xc0000005`. @@ -129,3 +148,10 @@ not establish better connection success or speed; this note adds no retry policy and retain Gitee; do not add a self-hosted mirror. Chrome still blocks the Gitee attachment when Referer is removed. Reopen for new evidence or a provider review; the warning remains unresolved. +10. **4K in advanced sharing settings.** Deferred until after the current capture + maintenance phase. Add 3840x2160 without changing the default or recommended + presets. Extend the existing capture, decode and relay bounds together; verify + H.264 level negotiation for 4K at 60 fps. Current strict quality messages reject + `2160p`, so preserve published Browser/App/Server compatibility through explicit + receiving-end support before exposing or sending the new setting. Include + source replacement, lower outputs and resource limits in acceptance. diff --git a/internal/app/app.go b/internal/app/app.go index 54a0402..b645a17 100644 --- a/internal/app/app.go +++ b/internal/app/app.go @@ -624,6 +624,7 @@ func discoverNativeMedia(ctx context.Context, configuredPath string) nativeRunti ReceiverReuse: true, Video: summary.Video, CaptureBorderControl: summary.CaptureBorderControl, + Microphone: summary.Microphone, ProcessAudio: summary.ProcessAudio, SystemAudio: summary.SystemAudio, HardwareH264: summary.HardwareH264, diff --git a/internal/app/loopback/server.go b/internal/app/loopback/server.go index b695907..0749824 100644 --- a/internal/app/loopback/server.go +++ b/internal/app/loopback/server.go @@ -72,6 +72,7 @@ type NativeMediaCapabilities struct { ReceiverReuse bool `json:"receiverReuse,omitempty"` Video bool `json:"video"` CaptureBorderControl bool `json:"captureBorderControl,omitempty"` + Microphone bool `json:"microphone,omitempty"` ProcessAudio bool `json:"processAudio"` SystemAudio bool `json:"systemAudio"` HardwareH264 bool `json:"hardwareH264"` diff --git a/internal/app/mediaedge/audio.go b/internal/app/mediaedge/audio.go index ddfbfe8..49e840c 100644 --- a/internal/app/mediaedge/audio.go +++ b/internal/app/mediaedge/audio.go @@ -94,6 +94,10 @@ func (engine *Engine) NewRelayedAudioSource(capacity int) (*AudioSource, error) } func (source *AudioSource) WritePCM(pcm []byte, duration time.Duration) error { + return source.WritePCMWithDropped(pcm, duration, 0) +} + +func (source *AudioSource) WritePCMWithDropped(pcm []byte, duration time.Duration, dropped uint16) error { if source == nil || source.encoder == nil || duration <= 0 { return errors.New("native PCM sample is invalid") } @@ -110,7 +114,7 @@ func (source *AudioSource) WritePCM(pcm []byte, duration time.Duration) error { return err } source.bytes.Add(uint64(len(packet))) - return source.samples.WriteSample(media.Sample{Data: packet, Duration: duration}) + return source.samples.WriteSample(media.Sample{Data: packet, Duration: duration, PrevDroppedPackets: dropped}) } func (source *AudioSource) WriteRTP(packet *rtp.Packet) error { diff --git a/internal/app/nativecapture/microphones.go b/internal/app/nativecapture/microphones.go new file mode 100644 index 0000000..40db487 --- /dev/null +++ b/internal/app/nativecapture/microphones.go @@ -0,0 +1,44 @@ +package nativecapture + +import ( + "context" + "errors" + "os/exec" + "strings" + "unicode/utf8" +) + +// Device identities remain local to the authenticated App control connection. +type Microphone struct { + ID string `json:"id"` + Label string `json:"label"` +} + +func ValidDeviceID(id string) bool { + return len(id) <= 512 && utf8.ValidString(id) && !strings.ContainsRune(id, 0) +} + +func ListMicrophones(parent context.Context, executable string) ([]Microphone, error) { + ctx, cancel := context.WithTimeout(parent, probeTimeout) + defer cancel() + stdout := &boundedBuffer{limit: maxProbeOutputBytes} + command := exec.CommandContext(ctx, executable, "--list-microphones") + command.Stdout = stdout + hideWindow(command) + if err := command.Run(); err != nil { + return nil, errors.New("native microphone list is unavailable") + } + var devices []Microphone + if err := decodeStrictJSON(stdout.Bytes(), &devices); err != nil || devices == nil || len(devices) > 64 { + return nil, errors.New("native microphone list is invalid") + } + seen := make(map[string]bool, len(devices)) + for _, device := range devices { + if device.ID == "" || !ValidDeviceID(device.ID) || seen[device.ID] || + device.Label == "" || len(device.Label) > 512 || !utf8.ValidString(device.Label) || strings.ContainsRune(device.Label, 0) { + return nil, errors.New("native microphone identity is invalid") + } + seen[device.ID] = true + } + return devices, nil +} diff --git a/internal/app/nativecapture/probe.go b/internal/app/nativecapture/probe.go index d6bceb8..cc30fcb 100644 --- a/internal/app/nativecapture/probe.go +++ b/internal/app/nativecapture/probe.go @@ -47,6 +47,7 @@ type Capabilities struct { PlatformBuild string `json:"platformBuild"` VideoCapture bool `json:"videoCapture"` CaptureBorderControl bool `json:"captureBorderControl,omitempty"` + Microphone bool `json:"microphone,omitempty"` ProcessAudio bool `json:"processAudio"` SystemAudio bool `json:"systemAudio"` SoftwareVP8 bool `json:"softwareVP8"` @@ -56,6 +57,7 @@ type Capabilities struct { type Summary struct { Video bool CaptureBorderControl bool + Microphone bool ProcessAudio bool SystemAudio bool HardwareH264 bool @@ -66,6 +68,7 @@ func (capabilities Capabilities) Summary() Summary { summary := Summary{ Video: capabilities.VideoCapture, CaptureBorderControl: capabilities.VideoCapture && capabilities.CaptureBorderControl, + Microphone: capabilities.Microphone, ProcessAudio: capabilities.ProcessAudio, SystemAudio: capabilities.SystemAudio, SoftwareVP8: capabilities.SoftwareVP8, diff --git a/internal/app/nativecapture/stream.go b/internal/app/nativecapture/stream.go index e8e5611..af49cee 100644 --- a/internal/app/nativecapture/stream.go +++ b/internal/app/nativecapture/stream.go @@ -298,7 +298,22 @@ func StartSystemAudio(parent context.Context, executable string) (*Stream, error }) } +func StartMicrophone(parent context.Context, executable, deviceID string) (*Stream, error) { + if !ValidDeviceID(deviceID) { + return nil, errors.New("microphone device is invalid") + } + arguments := []string{"--capture-microphone"} + if deviceID != "" { + arguments = append(arguments, "--device", deviceID) + } + return startAudioStreamWithTimeout(parent, executable, arguments, time.Minute) +} + func startAudioStream(parent context.Context, executable string, arguments []string) (*Stream, error) { + return startAudioStreamWithTimeout(parent, executable, arguments, probeTimeout) +} + +func startAudioStreamWithTimeout(parent context.Context, executable string, arguments []string, timeout time.Duration) (*Stream, error) { if parent == nil { parent = context.Background() } @@ -315,7 +330,7 @@ func startAudioStream(parent context.Context, executable string, arguments []str } ready <- validateAudioReadyFrame(frame) }() - timer := time.NewTimer(probeTimeout) + timer := time.NewTimer(timeout) defer timer.Stop() select { case readyErr := <-ready: @@ -395,7 +410,12 @@ func startStreamWithEnvironment( if executable == "" { return nil, errors.New("native capture process is unavailable") } - ctx, cancel := context.WithCancel(parent) + if err := parent.Err(); err != nil { + return nil, err + } + // Parent cancellation requests the same bounded Q/Close sequence as an + // explicit stop. Killing first can leave platform capture state behind. + ctx, cancel := context.WithCancel(context.WithoutCancel(parent)) started := time.Now() // A diagnostic-only process label correlates stderr while old/new captures overlap. captureID := strconv.FormatInt(started.UnixNano(), 36) @@ -444,8 +464,10 @@ func startStreamWithEnvironment( logger: logger, ctx: ctx, } + stopParent := context.AfterFunc(parent, func() { _ = stream.Close() }) go func() { waitErr := command.Wait() + stopParent() _ = trace.Close() logCaptureFailure(ctx, waitErr, command.ProcessState.ExitCode(), stderr.Bytes()) logger.DebugContext(ctx, "piik-client", "event", "capture-process-ended", "mode", mode, diff --git a/internal/app/nativecapture/stream_test.go b/internal/app/nativecapture/stream_test.go index abe5576..a5f0661 100644 --- a/internal/app/nativecapture/stream_test.go +++ b/internal/app/nativecapture/stream_test.go @@ -7,10 +7,80 @@ import ( "errors" "io" "log/slog" + "os" + "path/filepath" "strings" "testing" + "time" ) +func TestCaptureStopHelper(t *testing.T) { + mode := os.Getenv("PIIK_CAPTURE_STOP_FIXTURE") + if mode == "" { + return + } + if err := writeFrame(os.Stdout, Frame{Kind: FrameStatus, Data: []byte("ready")}); err != nil { + os.Exit(2) + } + if mode == "unresponsive" { + time.Sleep(10 * time.Second) + os.Exit(3) + } + frame, err := readFrame(os.Stdin) + if err != nil || frame.Kind != FrameControl || string(frame.Data) != "Q" { + os.Exit(4) + } + // A platform capture must get time to release its session before process exit. + time.Sleep(50 * time.Millisecond) + if err := os.WriteFile(os.Getenv("PIIK_CAPTURE_STOP_MARKER"), []byte("released"), 0600); err != nil { + os.Exit(5) + } + os.Exit(0) +} + +func TestCaptureStopUsesOneBoundedRetirementPath(t *testing.T) { + executable, err := os.Executable() + if err != nil { + t.Fatal(err) + } + for _, mode := range []string{"explicit", "parent", "both", "unresponsive"} { + t.Run(mode, func(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + marker := filepath.Join(t.TempDir(), "released") + stream, err := startStreamWithEnvironment(ctx, executable, + []string{"-test.run=^TestCaptureStopHelper$"}, + []string{"PIIK_CAPTURE_STOP_FIXTURE=" + mode, "PIIK_CAPTURE_STOP_MARKER=" + marker}) + if err != nil { + t.Fatal(err) + } + defer stream.Close() + if _, err := stream.Read(); err != nil { + t.Fatal(err) + } + if mode != "explicit" { + cancel() + } + if mode == "explicit" || mode == "both" { + go stream.Close() + } + select { + case <-stream.Done(): + case <-time.After(captureStopTimeout + 3*time.Second): + t.Fatal("capture retirement exceeded its stop budget") + } + _, err = os.Stat(marker) + if mode == "unresponsive" { + if !os.IsNotExist(err) { + t.Fatalf("unresponsive capture unexpectedly released: %v", err) + } + } else if err != nil { + t.Fatalf("capture was killed before receiving its stop command: %v", err) + } + }) + } +} + func TestDiagnosticStderrOverflowDoesNotStopTheChildReader(t *testing.T) { for _, discard := range []bool{false, true} { buffer := &boundedBuffer{limit: 4, discardOverflow: discard} diff --git a/internal/app/nativecontrol/session.go b/internal/app/nativecontrol/session.go index 6a5b8c4..3396b9c 100644 --- a/internal/app/nativecontrol/session.go +++ b/internal/app/nativecontrol/session.go @@ -8,6 +8,7 @@ import ( "errors" "io" "log/slog" + "math" "net/url" "regexp" "strconv" @@ -43,7 +44,6 @@ type Session struct { ctx context.Context cancel context.CancelFunc events chan any - hostEvents chan nativehost.Event viewerEvents chan nativeviewer.Event mu sync.Mutex @@ -72,7 +72,6 @@ func New( ctx: ctx, cancel: cancel, events: make(chan any, 256), - hostEvents: make(chan nativehost.Event, 256), viewerEvents: make(chan nativeviewer.Event, 256), } go session.relayEvents() @@ -118,6 +117,22 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any, responseEnvelope: response(envelope, "capture-options"), Adapters: session.capabilities.Adapters, }, nil + case "list-microphones": + var request requestEnvelope + if err := decodeStrict(payload, &request); err != nil { + return nil, protocolViolation("native microphone list request is invalid") + } + if !session.capabilities.Microphone { + return operationFailure(envelope, errors.New("native microphone is unavailable")), nil + } + devices, err := nativecapture.ListMicrophones(ctx, session.captureProcess) + if err != nil { + return operationFailure(envelope, err), nil + } + return struct { + responseEnvelope + Devices []nativecapture.Microphone `json:"devices"` + }{response(envelope, "microphone-list"), devices}, nil case "list-sources": var request listSourcesRequest if err := decodeStrict(payload, &request); err != nil || request.Type != envelope.Type { @@ -168,6 +183,34 @@ func (session *Session) Handle(ctx context.Context, payload []byte) (result any, "share", diagnostics.ID(request.ShareID), "profile", nativeQualityProfile(request.Profile), "codec", request.Codec, "audio", request.Audio, "sourceKind", request.Source.Kind, "adapterIndex", request.AdapterIndex, "encoderIndex", request.EncoderIndex) return session.startShare(ctx, envelope, request) + case "set-microphone": + var request microphoneRequest + if err := decodeStrict(payload, &request); err != nil || !validIdentities(request.ShareID) || + (request.Enabled == nil && request.Volume == nil && request.DeviceID == nil) || + (request.DeviceID != nil && !nativecapture.ValidDeviceID(*request.DeviceID)) || + (request.Volume != nil && (math.IsNaN(*request.Volume) || math.IsInf(*request.Volume, 0) || *request.Volume < 0 || *request.Volume > 2)) { + return nil, protocolViolation("native microphone request is invalid") + } + host := session.current(request.ShareID) + if host == nil { + return nil, errors.New("native share does not exist") + } + // Commit gain in command order; asynchronous device setup must not later + // overwrite a newer slider value received while permission was pending. + if request.Volume != nil { + if err := host.SetMicrophone(nil, request.Volume, nil); err != nil { + return nil, err + } + } + if request.Enabled == nil && request.DeviceID == nil { + return response(envelope, "microphone-set"), nil + } + result = session.runHostOperation(host, envelope, func() (any, error) { + err := host.SetMicrophone(request.Enabled, nil, request.DeviceID) + return response(envelope, "microphone-set"), err + }, complete) + asynchronous = result == nil + return result, nil case "update-share": var request updateShareRequest if err := decodeStrict(payload, &request); err != nil || @@ -394,22 +437,33 @@ func (session *Session) Close() error { } func (session *Session) updateHost(host *nativehost.Session, request requestEnvelope, shareID string, profile nativehost.QualityProfile, complete func(any, error)) any { + return session.runHostOperation(host, request, func() (any, error) { + err := host.UpdateProfile(profile) + return session.shareUpdateResult(request, shareID, profile, err), err + }, complete) +} + +// Quality and microphone setup share one bounded operation slot. Stop remains +// synchronous and cancels the owning share before joining this completion. +func (session *Session) runHostOperation(host *nativehost.Session, request requestEnvelope, operation func() (any, error), complete func(any, error)) any { session.mu.Lock() if session.closed || session.host != host || session.updateDone != nil { session.mu.Unlock() - return operationFailure(request, errors.New("native quality update is unavailable or already active")) + return operationFailure(request, errors.New("native share operation is unavailable or already active")) } done := make(chan struct{}) session.updateDone = done session.mu.Unlock() go func() { - err := host.UpdateProfile(profile) + result, err := operation() session.mu.Lock() current := !session.closed && session.host == host if !current && err == nil { - err = errors.New("native share stopped during quality update") + err = errors.New("native share stopped during operation") + } + if err != nil { + result = operationFailure(request, err) } - result := session.shareUpdateResult(request, shareID, profile, err) complete(result, err) close(done) if session.closed { @@ -476,13 +530,24 @@ func (session *Session) startShare( EncoderIndex: request.EncoderIndex, Profile: profile.Video, }, - Profile: profile, - EdgeCapacity: request.EdgeCapacity, + MicrophoneMixing: request.MicrophoneMixing && session.capabilities.Microphone, + Profile: profile, + EdgeCapacity: request.EdgeCapacity, AudioEnabled: request.Audio && session.capabilities.Summary().AudioFor( request.Source.Kind, ), PortMapping: session.portMapping, - Events: session.hostEvents, + Events: func(ctx context.Context, event nativehost.Event) { + var value any = eventMessage(event) + if event.Current != nil { + value = loopback.ControlEvent{Value: value, Current: event.Current} + } + select { + case session.events <- value: + case <-ctx.Done(): + case <-session.ctx.Done(): + } + }, }) if err != nil { return nil, err @@ -503,6 +568,7 @@ func (session *Session) startShare( responseEnvelope: response(envelope, "share-started"), ShareID: request.ShareID, Audio: host.HasAudio(), + SourceAudio: host.SourceAudio(), Codec: codec, }, nil } @@ -758,12 +824,6 @@ func (session *Session) watchHost(host *nativehost.Session) { func (session *Session) relayEvents() { for { select { - case event := <-session.hostEvents: - var value any = eventMessage(event) - if event.Current != nil { - value = loopback.ControlEvent{Value: value, Current: event.Current} - } - session.emit(value) case event := <-session.viewerEvents: session.emit(loopback.ControlEvent{Value: viewerEventMessage(event), Current: event.Current}) case <-session.ctx.Done(): @@ -829,6 +889,8 @@ func eventMessage(event nativehost.Event) any { PublicationGeneration: event.PublicationGeneration, } switch event.Type { + case "audio-state": + return audioStateEvent{eventEnvelope: base, SourceAudio: event.SourceAudio, Microphone: event.Microphone, Failed: event.Failed} case "publication-quality": return publicationQualityEvent{eventEnvelope: base, PublicationQualitySample: *event.PublicationQuality} case "edge-candidate", "publication-candidate": diff --git a/internal/app/nativecontrol/session_test.go b/internal/app/nativecontrol/session_test.go index a1b08d5..8194035 100644 --- a/internal/app/nativecontrol/session_test.go +++ b/internal/app/nativecontrol/session_test.go @@ -111,6 +111,11 @@ func runQuietCaptureFixture(directory string) { _, _ = os.Stdout.Write(payload) } status := func() { payload, _ := json.Marshal(state); write(nativecapture.FrameStatus, payload, 0) } + if len(os.Args) == 2 && os.Args[1] == "--capture-microphone" { + write(nativecapture.FrameStatus, []byte(`{"state":"active","audio":true}`), 0) + _, _ = io.Copy(io.Discard, os.Stdin) + return + } status() closed := make(chan struct{}) go func() { _, _ = io.Copy(io.Discard, os.Stdin); close(closed) }() @@ -144,6 +149,65 @@ func runQuietCaptureFixture(directory string) { <-closed } +func TestMicrophoneOptInPreservesLegacyAndOrdersStateBeforeCompletion(t *testing.T) { + for _, optIn := range []bool{false, true} { + t.Run(strconv.FormatBool(optIn), func(t *testing.T) { + t.Setenv("PIIK_QUIET_CAPTURE_FIXTURE", t.TempDir()) + executable, err := os.Executable() + if err != nil { + t.Fatal(err) + } + session := New(executable, nativecapture.Capabilities{VideoCapture: true, SoftwareVP8: true, Microphone: true}, false) + defer session.Close() + request := startShareRequest{Version: 9, ID: "request_start", Type: "start-share", ShareID: "share_123456", + Source: nativecapture.CaptureTarget{Kind: "display", SourceID: "1", Title: "Fixture"}, + MicrophoneMixing: optIn, EdgeCapacity: 1, Codec: "vp8", + Profile: qualitySettings{Resolution: "1080p", MaxFramerate: 30, MaxBitrate: 5_000_000, DegradationPreference: "balanced"}} + payload, _ := json.Marshal(request) + result, err := session.Handle(t.Context(), payload) + started, ok := result.(shareStartedResponse) + if err != nil || !ok { + t.Fatalf("start failed: %v %#v", err, result) + } + encoded, _ := json.Marshal(started) + if started.Audio != optIn || bytes.Contains(encoded, []byte(`"sourceAudio"`)) != optIn { + t.Fatalf("mixed output leaked across opt-in: %s", encoded) + } + result, err = session.Handle(t.Context(), []byte(`{"version":9,"id":"request_mic","type":"set-microphone","shareId":"share_123456","enabled":true,"volume":1}`)) + if !optIn { + if _, failed := result.(requestFailedResponse); err != nil || !failed { + t.Fatalf("non-opted share admitted mic: %v %#v", err, result) + } + return + } + if err != nil || result != nil { + t.Fatalf("device setup was not asynchronous: %v %#v", err, result) + } + stateSeen := false + for { + select { + case event := <-session.Events(): + switch value := event.(type) { + case audioStateEvent: + stateSeen = value.Microphone && !value.SourceAudio + case responseEnvelope: + if value.Type == "microphone-set" { + if !stateSeen { + t.Fatal("completion overtook microphone state") + } + return + } + case requestFailedResponse: + t.Fatalf("microphone failed: %#v", value) + } + case <-time.After(2 * time.Second): + t.Fatal("microphone did not complete") + } + } + }) + } +} + func TestCaptureBorderPreferenceFollowsSourceAndProfile(t *testing.T) { for _, supported := range []bool{false, true} { t.Run(strconv.FormatBool(supported), func(t *testing.T) { diff --git a/internal/app/nativecontrol/wire.go b/internal/app/nativecontrol/wire.go index 281bd43..a68404d 100644 --- a/internal/app/nativecontrol/wire.go +++ b/internal/app/nativecontrol/wire.go @@ -48,6 +48,7 @@ type startShareRequest struct { ShareID string `json:"shareId"` Source nativecapture.CaptureTarget `json:"source"` Audio bool `json:"audio"` + MicrophoneMixing bool `json:"microphoneMixing,omitempty"` ShowCaptureBorder bool `json:"showCaptureBorder,omitempty"` AdapterIndex uint32 `json:"adapterIndex"` EncoderIndex uint32 `json:"encoderIndex"` @@ -56,6 +57,21 @@ type startShareRequest struct { Codec string `json:"codec"` } +type microphoneRequest struct { + requestEnvelope + ShareID string `json:"shareId"` + Enabled *bool `json:"enabled,omitempty"` + Volume *float64 `json:"volume,omitempty"` + DeviceID *string `json:"deviceId,omitempty"` +} + +type audioStateEvent struct { + eventEnvelope + SourceAudio bool `json:"sourceAudio"` + Microphone bool `json:"microphone"` + Failed bool `json:"failed"` +} + type updateShareRequest struct { Version int `json:"version"` ID string `json:"id"` @@ -195,9 +211,10 @@ type captureOptionsResponse struct { type shareStartedResponse struct { responseEnvelope - ShareID string `json:"shareId"` - Audio bool `json:"audio"` - Codec string `json:"codec"` + ShareID string `json:"shareId"` + Audio bool `json:"audio"` + SourceAudio *bool `json:"sourceAudio,omitempty"` + Codec string `json:"codec"` } type shareUpdatedResponse struct { diff --git a/internal/app/nativehost/audio_mix.go b/internal/app/nativehost/audio_mix.go new file mode 100644 index 0000000..b4707c3 --- /dev/null +++ b/internal/app/nativehost/audio_mix.go @@ -0,0 +1,263 @@ +package nativehost + +import ( + "context" + "encoding/binary" + "errors" + "log/slog" + "math" + "sync" + "time" + + "github.com/TNTcraftHIM/Piik/internal/app/mediaedge" + "github.com/TNTcraftHIM/Piik/internal/app/nativeaudio" + "github.com/TNTcraftHIM/Piik/internal/app/nativecapture" + "github.com/TNTcraftHIM/Piik/internal/diagnostics" +) + +const audioPeriod = 20 * time.Millisecond + +// Each device is resampled by the platform to PCM48 stereo. This bounded FIFO +// absorbs scheduling jitter; it never replays samples or builds a latency tail. +// It does not claim to eliminate long-term drift between hardware clocks. +type audioInput struct { + stream *nativecapture.Stream + frames [][]byte +} + +func (input *audioInput) push(frame []byte) { + if len(input.frames) == 4 { + input.frames = input.frames[1:] + } + input.frames = append(input.frames, frame) +} + +func (input *audioInput) pop() []byte { + if input == nil || len(input.frames) == 0 { + return nil + } + frame := input.frames[0] + input.frames[0] = nil + input.frames = input.frames[1:] + return frame +} + +// One share owns one output/encoder. Replacing an input never replaces that +// output. The session owns source capture; this owner owns microphone capture. +type audioMix struct { + ctx context.Context + output *mediaedge.AudioSource + emit func(Event) + shareID string + mu sync.Mutex + // State delivery is ordered, but backpressure never holds the audio lock. + eventMu sync.Mutex + readers sync.WaitGroup + source, microphone *audioInput + paused bool + gain, appliedGain float64 + microphoneDevice string +} + +func (mix *audioMix) stateLocked(failed bool) Event { + return Event{Type: "audio-state", ShareID: mix.shareID, + SourceAudio: mix.source != nil, Microphone: mix.microphone != nil, Failed: failed} +} + +func (mix *audioMix) setSource(stream *nativecapture.Stream) { + mix.eventMu.Lock() + defer mix.eventMu.Unlock() + mix.mu.Lock() + if mix.ctx.Err() != nil { + mix.mu.Unlock() + return + } + mix.source = nil + if stream != nil { + mix.source = &audioInput{stream: stream} + mix.readers.Add(1) + go mix.read(mix.source, false) + } + state := mix.stateLocked(false) + mix.mu.Unlock() + mix.emit(state) +} + +func (mix *audioMix) read(input *audioInput, microphone bool) { + defer mix.readers.Done() + for { + frame, err := input.stream.Read() + mix.mu.Lock() + current := mix.source + if microphone { + current = mix.microphone + } + if current != input || mix.ctx.Err() != nil { + mix.mu.Unlock() + return + } + if err != nil || frame.Kind != nativecapture.FramePCM || len(frame.Data) != nativeaudio.FrameBytes || frame.Duration != audioPeriod { + mix.mu.Unlock() + mix.inputEnded(input, microphone) + slog.DebugContext(mix.ctx, "piik-client", "event", "audio-input-ended", "microphone", microphone, diagnostics.Error(err)) + _ = input.stream.Close() + return + } + if !mix.paused { + input.push(frame.Data) + } + mix.mu.Unlock() + } +} + +func (mix *audioMix) setPaused(paused bool) { + mix.mu.Lock() + defer mix.mu.Unlock() + mix.paused = paused + for _, input := range []*audioInput{mix.source, mix.microphone} { + if input != nil { + input.frames = nil + } + } + // Never carry a pre-pause ramp into the resumed share. + mix.appliedGain = 0 +} + +func (mix *audioMix) inputEnded(input *audioInput, microphone bool) { + mix.eventMu.Lock() + defer mix.eventMu.Unlock() + mix.mu.Lock() + current := mix.source + if microphone { + current = mix.microphone + } + if current != input || mix.ctx.Err() != nil { + mix.mu.Unlock() + return + } + if microphone { + mix.microphone = nil + } else { + mix.source = nil + } + state := mix.stateLocked(microphone) + mix.mu.Unlock() + mix.emit(state) +} + +func (mix *audioMix) setMicrophone(executable string, enabled *bool, gain *float64, deviceID *string) error { + if deviceID != nil && !nativecapture.ValidDeviceID(*deviceID) { + return errors.New("microphone device is invalid") + } + if gain != nil && (math.IsNaN(*gain) || math.IsInf(*gain, 0) || *gain < 0 || *gain > 2) { + return errors.New("microphone volume is invalid") + } + mix.mu.Lock() + if gain != nil { + mix.gain = *gain + } + wanted := mix.microphone != nil + if enabled != nil { + wanted = *enabled + } + device := mix.microphoneDevice + if deviceID != nil { + device = *deviceID + } + unchanged := wanted == (mix.microphone != nil) && device == mix.microphoneDevice + mix.mu.Unlock() + if unchanged { + return nil + } + var stream *nativecapture.Stream + if wanted { + var err error + stream, err = nativecapture.StartMicrophone(mix.ctx, executable, device) + if err != nil { + return err + } + } + mix.eventMu.Lock() + defer mix.eventMu.Unlock() + mix.mu.Lock() + if mix.ctx.Err() != nil { + mix.mu.Unlock() + if stream != nil { + _ = stream.Close() + } + return mix.ctx.Err() + } + previous := mix.microphone + mix.microphoneDevice = device + mix.microphone = nil + if stream != nil { + mix.microphone = &audioInput{stream: stream} + mix.readers.Add(1) + go mix.read(mix.microphone, true) + } + state := mix.stateLocked(false) + mix.mu.Unlock() + if previous != nil { + _ = previous.stream.Close() + } + mix.emit(state) + return nil +} + +func (mix *audioMix) run() { + ticker := time.NewTicker(audioPeriod) + defer ticker.Stop() + var output [nativeaudio.FrameBytes]byte + previous := time.Now() + for { + select { + case now := <-ticker.C: + // Pion advances RTP time across missed ticks; do not burst old speech. + dropped := uint16(min(max(int(now.Sub(previous)/audioPeriod)-1, 0), 65535)) + previous = now + mix.mu.Lock() + source, microphone := mix.source.pop(), mix.microphone.pop() + gain := mix.gain + if microphone == nil { + gain = 0 + } + mixPCM(output[:], source, microphone, mix.appliedGain, gain) + mix.appliedGain = gain + if mix.paused { + clear(output[:]) + } + // A retired edge cannot end healthy sibling media. + _ = mix.output.WritePCMWithDropped(output[:], audioPeriod, dropped) + mix.mu.Unlock() + case <-mix.ctx.Done(): + mix.mu.Lock() + microphone := mix.microphone + mix.microphone = nil + mix.mu.Unlock() + if microphone != nil { + _ = microphone.stream.Close() + } + mix.readers.Wait() // Session cancellation/retirement also closes source. + return + } + } +} + +// Mix once in a wider type, then saturate. Never attenuate source sound just +// because the microphone was enabled. Gain changes span a frame, not a sample. +func mixPCM(output, source, microphone []byte, from, to float64) { + for sample := range nativeaudio.FrameSamples { + gain := from + (to-from)*float64(sample+1)/nativeaudio.FrameSamples + for channel := range nativeaudio.Channels { + offset := (sample*nativeaudio.Channels + channel) * 2 + value := 0.0 + if source != nil { + value = float64(int16(binary.LittleEndian.Uint16(source[offset:]))) + } + if microphone != nil { + value += gain * float64(int16(binary.LittleEndian.Uint16(microphone[offset:]))) + } + binary.LittleEndian.PutUint16(output[offset:], uint16(int16(max(-32768, min(32767, value))))) + } + } +} diff --git a/internal/app/nativehost/audio_mix_test.go b/internal/app/nativehost/audio_mix_test.go new file mode 100644 index 0000000..0bac324 --- /dev/null +++ b/internal/app/nativehost/audio_mix_test.go @@ -0,0 +1,242 @@ +package nativehost + +import ( + "context" + "encoding/binary" + "os" + "path/filepath" + "testing" + "time" + + "github.com/TNTcraftHIM/Piik/internal/app/nativeaudio" + "github.com/TNTcraftHIM/Piik/internal/app/nativecapture" +) + +func TestMixPCMRetainsSourceGainAndSaturatesStereo(t *testing.T) { + frame := func(value int16) []byte { + data := make([]byte, nativeaudio.FrameBytes) + for i := 0; i < len(data); i += 2 { + binary.LittleEndian.PutUint16(data[i:], uint16(value)) + } + return data + } + output := frame(0) + mixPCM(output, frame(1000), nil, 0, 0) + if int16(binary.LittleEndian.Uint16(output)) != 1000 { + t.Fatal("source was attenuated") + } + mixPCM(output, frame(1000), frame(2000), 0, 2) + if binary.LittleEndian.Uint16(output[:2]) != binary.LittleEndian.Uint16(output[2:4]) || + int16(binary.LittleEndian.Uint16(output[len(output)-2:])) != 5000 || + int16(binary.LittleEndian.Uint16(output)) >= 1100 { + t.Fatal("stereo gain did not ramp") + } + for _, value := range []int16{-30000, 30000} { + mixPCM(output, frame(value), frame(value), 2, 2) + expected := int16(32767) + if value < 0 { + expected = -32768 + } + if int16(binary.LittleEndian.Uint16(output)) != expected { + t.Fatal("sum wrapped instead of saturating") + } + } + mixPCM(output, nil, nil, 2, 0) + for _, value := range output { + if value != 0 { + t.Fatal("missing input replayed old audio") + } + } + input := &audioInput{} + for i := byte(0); i < 10; i++ { + input.push([]byte{i}) + } + for i := byte(6); i < 10; i++ { + if got := input.pop(); len(got) != 1 || got[0] != i { + t.Fatal("FIFO retained stale frames") + } + } + if input.pop() != nil { + t.Fatal("FIFO replayed the last sample") + } +} + +func startMixedFixture(t *testing.T) (*Session, nativecapture.VideoOptions, <-chan Event) { + t.Helper() + t.Setenv("PIIK_NATIVEHOST_PIPE_FIXTURE", "audio-recovery") + executable, err := os.Executable() + if err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second) + t.Cleanup(cancel) + events := make(chan Event, 64) + options := nativecapture.VideoOptions{Codec: "vp8", Profile: audioRecoveryProfile(), + Target: nativecapture.CaptureTarget{Kind: "display", SourceID: "1", Title: "Mixed fixture"}} + session, err := Start(ctx, Options{ShareID: "mixed_fixture", CaptureProcess: executable, Video: options, + Profile: QualityProfile{Video: options.Profile, AudioBitrate: 64000}, MicrophoneMixing: true, + EdgeCapacity: 1, BindAddress: "127.0.0.1:0", Events: func(_ context.Context, event Event) { + select { + case events <- event: + case <-ctx.Done(): + } + }}) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = session.Close() }) + return session, options, events +} + +func TestNativeMicrophonePreservesMediaOwnersAcrossInputChanges(t *testing.T) { + session, options, events := startMixedFixture(t) + check := func(err error) { + t.Helper() + if err != nil { + t.Fatal(err) + } + } + if !session.HasAudio() || session.SourceAudio() == nil || *session.SourceAudio() { + t.Fatal("silent mixed output was not reserved independently of source") + } + _, err := session.PrepareLocalEdge("mixed_edge") + check(err) + edge, source := session.edge("mixed_edge"), session.audioSource + enabled, gain := true, 1.0 + check(session.SetMicrophone(&enabled, &gain, nil)) + session.mixer.mu.Lock() + microphone := session.mixer.microphone + session.mixer.mu.Unlock() + if microphone == nil { + t.Fatal("microphone did not start") + } + device := "missing" + if session.SetMicrophone(nil, nil, &device) == nil { + t.Fatal("missing microphone accepted") + } + session.mixer.mu.Lock() + retained := session.mixer.microphone == microphone + session.mixer.mu.Unlock() + if !retained { + t.Fatal("failed device switch retired healthy microphone") + } + device = "headset" + check(session.SetMicrophone(nil, nil, &device)) + session.mixer.mu.Lock() + replaced := session.mixer.microphone + session.mixer.mu.Unlock() + if replaced == nil || replaced == microphone || session.audioSource != source || session.edge("mixed_edge") != edge { + t.Fatal("device change failed to replace only the input") + } + microphone = replaced + options.Target.SourceID = "2" + check(session.ReplaceSource(t.Context(), options, true)) + session.mixer.mu.Lock() + sameMicrophone := session.mixer.microphone == microphone + session.mixer.mu.Unlock() + if !sameMicrophone || session.audioSource != source || session.edge("mixed_edge") != edge || !*session.SourceAudio() { + t.Fatal("source replacement rebuilt microphone/output/edge") + } + session.SetPaused(true) + time.Sleep(60 * time.Millisecond) + session.mixer.mu.Lock() + queued := len(session.mixer.source.frames) + len(session.mixer.microphone.frames) + session.mixer.mu.Unlock() + if queued != 0 { + t.Fatal("paused speech accumulated") + } + session.SetPaused(false) + // EOF retires only the microphone; source and all negotiated media stay live. + check(microphone.stream.Close()) + timer := time.NewTimer(time.Second) + defer timer.Stop() + for { + select { + case event := <-events: + if event.Type != "audio-state" || !event.Failed { + continue + } + if event.Microphone || !event.SourceAudio || !session.HasAudio() || session.edge("mixed_edge") != edge { + t.Fatal("input loss damaged healthy media") + } + enabled = false + check(session.SetMicrophone(&enabled, nil, nil)) + return + case <-timer.C: + t.Fatal("microphone loss was not reported") + } + } +} + +func TestStopCancelsPendingMicrophoneWithoutRevivingShare(t *testing.T) { + marker := filepath.Join(t.TempDir(), "microphone-started") + t.Setenv("PIIK_MICROPHONE_WAIT", marker) + session, _, _ := startMixedFixture(t) + result := make(chan error, 1) + go func() { enabled := true; result <- session.SetMicrophone(&enabled, nil, nil) }() + deadline := time.Now().Add(2 * time.Second) + for { + if _, err := os.Stat(marker); err == nil { + break + } + if time.Now().After(deadline) { + t.Fatal("microphone fixture did not start") + } + time.Sleep(time.Millisecond) + } + stopped := make(chan error, 1) + go func() { stopped <- session.Close() }() + select { + case err := <-stopped: + if err != nil { + t.Fatal(err) + } + case <-time.After(2 * time.Second): + t.Fatal("microphone permission blocked Stop") + } + if err := <-result; err == nil { + t.Fatal("late microphone became live after Stop") + } + session.mixer.mu.Lock() + defer session.mixer.mu.Unlock() + if session.mixer.microphone != nil { + t.Fatal("retired microphone retained") + } +} + +func TestCancelledMixerCannotRegisterReplacementReader(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + cancel() + mix := &audioMix{ctx: ctx} + mix.setSource(&nativecapture.Stream{}) + if mix.source != nil { + t.Fatal("cancelled mixer installed a source") + } + mix.readers.Wait() +} + +func TestAudioNotificationBackpressureDoesNotBlockPauseOrGain(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + entered, returned := make(chan struct{}), make(chan struct{}) + input := &audioInput{} + mix := &audioMix{ctx: ctx, microphone: input, emit: func(Event) { close(entered); <-ctx.Done() }} + go func() { mix.inputEnded(input, true); close(returned) }() + <-entered + controlled := make(chan struct{}) + go func() { + mix.setPaused(true) + gain := 1.5 + if err := mix.setMicrophone("", nil, &gain, nil); err != nil { + t.Error(err) + } + close(controlled) + }() + select { + case <-controlled: + case <-time.After(time.Second): + t.Fatal("notifications blocked the control reader before Stop") + } + cancel() + <-returned +} diff --git a/internal/app/nativehost/audio_test.go b/internal/app/nativehost/audio_test.go index 92198ee..5a82770 100644 --- a/internal/app/nativehost/audio_test.go +++ b/internal/app/nativehost/audio_test.go @@ -42,7 +42,18 @@ func runAudioRecoveryCapture() { os.Exit(1) } } - audio := len(os.Args) > 1 && os.Args[1] == "--capture-audio" + microphone := len(os.Args) > 1 && os.Args[1] == "--capture-microphone" + if microphone { + if len(os.Args) == 4 && os.Args[2] == "--device" && os.Args[3] == "missing" { + os.Exit(2) + } + if marker := os.Getenv("PIIK_MICROPHONE_WAIT"); marker != "" { + _ = os.WriteFile(marker, nil, 0600) + <-stopped + return + } + } + audio := microphone || (len(os.Args) > 1 && os.Args[1] == "--capture-audio") period := time.Second / 30 status := []byte(`{"state":"active","audio":true}`) if audio { @@ -92,7 +103,12 @@ func TestAudioEOFWaitsForExplicitSourceReplacement(t *testing.T) { Target: nativecapture.CaptureTarget{Kind: "display", SourceID: "1", Title: "Audio fixture"}} session, err := Start(ctx, Options{ShareID: "audio-recovery", CaptureProcess: executable, Video: options, Profile: QualityProfile{Video: options.Profile, AudioBitrate: 64_000}, - AudioEnabled: true, EdgeCapacity: 1, BindAddress: "127.0.0.1:0", Events: events}) + AudioEnabled: true, EdgeCapacity: 1, BindAddress: "127.0.0.1:0", Events: func(_ context.Context, event Event) { + select { + case events <- event: + case <-ctx.Done(): + } + }}) check(err) defer session.Close() if !session.HasAudio() { diff --git a/internal/app/nativehost/publication_test.go b/internal/app/nativehost/publication_test.go index f6612da..d94a03e 100644 --- a/internal/app/nativehost/publication_test.go +++ b/internal/app/nativehost/publication_test.go @@ -1,6 +1,7 @@ package nativehost import ( + "context" "testing" "time" @@ -25,7 +26,12 @@ func TestPublicationGenerationFencesNativeControl(t *testing.T) { t.Fatal(err) } events := make(chan Event, 64) - session := &Session{shareID: "share_12345678", engine: engine, source: source, ctx: t.Context(), events: events} + session := &Session{shareID: "share_12345678", engine: engine, source: source, ctx: t.Context(), events: func(ctx context.Context, event Event) { + select { + case events <- event: + case <-ctx.Done(): + } + }} _, media, err := session.PreparePublication("generation_12345678", "connection_12345678", nil) if err != nil || media.Codec != "vp8" || len(media.Layers) != 1 { t.Fatalf("publication did not reuse capture metadata: %+v %v", media, err) diff --git a/internal/app/nativehost/session.go b/internal/app/nativehost/session.go index 513debf..dd49d70 100644 --- a/internal/app/nativehost/session.go +++ b/internal/app/nativehost/session.go @@ -43,6 +43,9 @@ type CaptureState struct { type Event struct { Current func() bool + SourceAudio bool + Microphone bool + Failed bool Type string ShareID string ConnectionID string @@ -57,15 +60,16 @@ type Event struct { } type Options struct { - ShareID string - CaptureProcess string - Video nativecapture.VideoOptions - Profile QualityProfile - AudioEnabled bool - EdgeCapacity int - BindAddress string - PortMapping bool - Events chan<- Event + ShareID string + CaptureProcess string + Video nativecapture.VideoOptions + Profile QualityProfile + AudioEnabled bool + MicrophoneMixing bool + EdgeCapacity int + BindAddress string + PortMapping bool + Events func(context.Context, Event) } type Session struct { @@ -79,7 +83,8 @@ type Session struct { engine *mediaedge.Engine source *mediaedge.Source audioSource *mediaedge.AudioSource - events chan<- Event + mixer *audioMix + events func(context.Context, Event) ctx context.Context cancel context.CancelFunc @@ -151,19 +156,31 @@ func Start(parent context.Context, options Options) (*Session, error) { edges: make(map[string]*mediaedge.Edge), audioChanged: make(chan struct{}, 1), } - if audioStream != nil { + if audioStream != nil || options.MicrophoneMixing { audioSource, audioErr := engine.NewAudioSource( options.EdgeCapacity, options.Profile.AudioBitrate, ) if audioErr != nil { slog.DebugContext(ctx, "piik-client", "event", "capture-audio-source-failed", "share", diagnostics.ID(options.ShareID), diagnostics.Error(audioErr)) - _ = audioStream.Close() + if audioStream != nil { + _ = audioStream.Close() + } audioStream = nil session.audioStream = nil } else { session.audioSource = audioSource } } + if options.MicrophoneMixing { + if session.audioSource == nil { + cancel() + _ = stream.Close() + _ = engine.Close() + return nil, errors.New("native mixed audio could not start") + } + session.mixer = &audioMix{ctx: ctx, output: session.audioSource, emit: session.emit, shareID: session.shareID, gain: 1} + session.mixer.setSource(audioStream) + } go session.run() var timeout <-chan time.Time var timer *time.Timer @@ -308,6 +325,9 @@ func (session *Session) SetPaused(paused bool) { return } session.paused = paused + if session.mixer != nil { + session.mixer.setPaused(paused) + } source := session.source session.mu.Unlock() if !paused { @@ -391,7 +411,7 @@ func (session *Session) ReplaceSource( options.OutputGroups = session.edgeCapacity hasAudio := session.audioSource != nil && session.audioStream != nil session.mu.Unlock() - if audioEnabled != hasAudio { + if session.mixer == nil && audioEnabled != hasAudio { return errors.New("native source audio availability cannot change while sharing") } options.Profile = profile.Video @@ -401,7 +421,7 @@ func (session *Session) ReplaceSource( return err } var replacementAudio *nativecapture.Stream - if hasAudio { + if audioEnabled { replacementAudio, err = startAudioCapture( ctx, session.captureProcess, @@ -418,7 +438,7 @@ func (session *Session) ReplaceSource( replacement, state, replacementAudio, - hasAudio, + true, ) } @@ -500,6 +520,9 @@ func (session *Session) commitCapture( session.profile = profile session.mu.Unlock() + if replaceAudio && session.mixer != nil { + session.mixer.setSource(replacementAudio) + } _ = replacement.RequestKeyFrame(-1) _ = previous.Close() if replaceAudio && previousAudio != nil { @@ -513,9 +536,6 @@ func (session *Session) commitCapture( if err := session.ctx.Err(); err != nil { return err } - session.emit(Event{ - Type: "capture-state", ShareID: session.shareID, State: "active", - }) return nil } @@ -532,6 +552,7 @@ func (session *Session) installCapture(next *nativecapture.Stream) error { session.captureApplied = nil } session.mu.Unlock() + session.emit(Event{Type: "capture-state", ShareID: session.shareID, State: "active"}) return nil } @@ -607,7 +628,9 @@ func (session *Session) run() { session.runQuality() close(qualityDone) }() - if session.audioStream == nil { + if session.mixer != nil { + go func() { session.mixer.run(); close(audioDone) }() + } else if session.audioStream == nil { close(audioDone) } else { go func() { @@ -1016,6 +1039,7 @@ func (session *Session) runQuality() { } } +// Public shares without the microphone opt-in retain their original audio shape. func (session *Session) runAudio() { current := session.currentAudioStream() for current != nil { @@ -1065,9 +1089,8 @@ func (session *Session) emit(event Event) { if session.events == nil { return } - select { - case session.events <- event: - case <-session.ctx.Done(): + if session.ctx.Err() == nil { + session.events(session.ctx, event) } } @@ -1114,3 +1137,25 @@ func validH264ProfileLevelID(value string) bool { return false } } + +// Optional facts are emitted only to a page which opted into mixed audio. +func (session *Session) SourceAudio() *bool { + if session.mixer == nil { + return nil + } + session.mixer.mu.Lock() + defer session.mixer.mu.Unlock() + available := session.mixer.source != nil + return &available +} + +func (session *Session) SetMicrophone(enabled *bool, gain *float64, deviceID *string) error { + if enabled != nil || deviceID != nil { + session.updateMu.Lock() + defer session.updateMu.Unlock() + } + if session.mixer == nil || session.ctx.Err() != nil { + return errors.New("native microphone is unavailable") + } + return session.mixer.setMicrophone(session.captureProcess, enabled, gain, deviceID) +} diff --git a/internal/app/nativehost/session_test.go b/internal/app/nativehost/session_test.go index 9918788..107a03c 100644 --- a/internal/app/nativehost/session_test.go +++ b/internal/app/nativehost/session_test.go @@ -66,7 +66,7 @@ func TestCaptureCommitWaitsForReaderMetadataOrTermination(t *testing.T) { publication, err := engine.NewPublication(source, mediaedge.EdgeOptions{ConnectionID: "metadata-fixture"}) check(err) session := &Session{ctx: ctx, cancel: cancel, engine: engine, source: source, stream: previous, - edgeCapacity: 1, ready: make(chan error, 1), done: make(chan error, 1), events: make(chan Event, 16)} + edgeCapacity: 1, ready: make(chan error, 1), done: make(chan error, 1), events: func(context.Context, Event) {}} options.Profile.Width, options.Profile.Height = 854, 480 replacement, err := nativecapture.StartVideo(ctx, executable, options) check(err) diff --git a/internal/app/publictunnel/process.go b/internal/app/publictunnel/process.go index 94a7ed5..992f53d 100644 --- a/internal/app/publictunnel/process.go +++ b/internal/app/publictunnel/process.go @@ -88,6 +88,10 @@ func Start( "--no-autoupdate", "--loglevel", "info", "--output", "json", + // Quick Tunnels otherwise force QUIC. Let cloudflared try TCP after + // the first failed edge attempt, within our bounded startup window. + "--protocol", "auto", + "--max-edge-addr-retries", "0", "--url", localOrigin, ) hideWindow(child) diff --git a/internal/app/publictunnel/process_test.go b/internal/app/publictunnel/process_test.go index 9be2aff..394ff3b 100644 --- a/internal/app/publictunnel/process_test.go +++ b/internal/app/publictunnel/process_test.go @@ -9,6 +9,7 @@ import ( "os" "os/exec" "path/filepath" + "slices" "strings" "sync" "testing" @@ -17,7 +18,18 @@ import ( func TestMain(tests *testing.M) { if mode := os.Getenv("PIIK_TUNNEL_FIXTURE"); mode != "" { - if mode == "ready" || mode == "fail-after-ready" { + if mode == "tcp-only" { + // Model the pinned Quick Tunnel's opt-in protocol fallback. Without + // both flags, an unavailable UDP edge consumes Piik's startup budget. + for name, want := range map[string]string{"--protocol": "auto", "--max-edge-addr-retries": "0"} { + index := slices.Index(os.Args, name) + if index < 0 || index+1 >= len(os.Args) || os.Args[index+1] != want { + fmt.Fprintln(os.Stderr, "UDP unavailable; TCP fallback not enabled within startup budget") + os.Exit(7) + } + } + } + if mode == "ready" || mode == "fail-after-ready" || mode == "tcp-only" { fmt.Println(`{"message":"https://test-room.trycloudflare.com"}`) fmt.Println(`{"message":"Registered tunnel connection"}`) } @@ -37,6 +49,23 @@ func TestMain(tests *testing.M) { os.Exit(tests.Run()) } +func TestStartupEnablesCloudflaredProtocolFallback(t *testing.T) { + executable, err := os.Executable() + if err != nil { + t.Fatal(err) + } + t.Setenv("PIIK_TUNNEL_FIXTURE", "tcp-only") + ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second) + defer cancel() + process, err := Start(ctx, executable, "http://127.0.0.1:8787") + if err != nil { + t.Fatalf("tunnel with TCP available: %v", err) + } + if err := process.Close(); err != nil { + t.Fatal(err) + } +} + type blockedTunnelLog struct { slog.Handler entered chan struct{} diff --git a/native/capture/darwin/Info.plist b/native/capture/darwin/Info.plist new file mode 100644 index 0000000..94901d2 --- /dev/null +++ b/native/capture/darwin/Info.plist @@ -0,0 +1,7 @@ + + + + CFBundleIdentifiertv.piik.capture + CFBundleNamePiik + NSMicrophoneUsageDescriptionInclude your voice while sharing. + diff --git a/native/capture/darwin/build.sh b/native/capture/darwin/build.sh index 6a83f87..0f300e4 100644 --- a/native/capture/darwin/build.sh +++ b/native/capture/darwin/build.sh @@ -13,10 +13,13 @@ xcrun swiftc \ -parse-as-library \ -swift-version 5 \ -target arm64-apple-macos13.0 \ + -framework AVFoundation \ -framework AudioToolbox \ + -framework CoreAudio \ -framework CoreMedia \ -framework CoreVideo \ -framework ScreenCaptureKit \ -framework VideoToolbox \ + -Xlinker -sectcreate -Xlinker __TEXT -Xlinker __info_plist -Xlinker "$(dirname "$0")/Info.plist" \ "$(dirname "$0")/main.swift" \ -o "$output/piik-capture" diff --git a/native/capture/darwin/main.swift b/native/capture/darwin/main.swift index 9af0e5e..9154c93 100644 --- a/native/capture/darwin/main.swift +++ b/native/capture/darwin/main.swift @@ -1,4 +1,6 @@ +import AVFoundation import AudioToolbox +import CoreAudio import CoreMedia import CoreVideo import Darwin @@ -76,6 +78,7 @@ private struct Probe: Codable { let platformBuild: String let videoCapture: Bool let processAudio: Bool + let microphone = true let systemAudio: Bool let softwareVP8 = false let adapters: [AdapterProbe] @@ -284,6 +287,12 @@ private final class StopSignal { semaphore.signal() } + var isStopped: Bool { + lock.lock() + defer { lock.unlock() } + return stopped + } + func wait() throws { semaphore.wait() lock.lock() @@ -1178,6 +1187,14 @@ private final class AudioCaptureOutput: NSObject, SCStreamOutput, SCStreamDelega } } + // Microphone conversion and ScreenCaptureKit audio use the same PCM framing. + func appendPCM(_ buffer: AVAudioPCMBuffer) throws { + guard !done.isStopped, let samples = buffer.int16ChannelData else { return } + pending.append(UnsafeRawPointer(samples[0]).assumingMemoryBound(to: UInt8.self), + count: Int(buffer.frameLength) * 4) + try flushFrames() + } + private func flushFrames() throws { let frameBytes = 960 * 2 * MemoryLayout.size while pending.count >= frameBytes { @@ -1697,6 +1714,141 @@ private func capture(_ arguments: [String]) async throws { try await stream.stopCapture() } +// AVAudioEngine callbacks never wait on stdout. Four copied buffers bound the +// worker; the persistent platform converter owns resampling between callbacks. +private func audioDeviceString(_ device: AudioDeviceID, _ selector: AudioObjectPropertySelector) -> String? { + var address = AudioObjectPropertyAddress(mSelector: selector, mScope: kAudioObjectPropertyScopeGlobal, mElement: kAudioObjectPropertyElementMain) + var value: CFString? + var size = UInt32(MemoryLayout.size) + guard AudioObjectGetPropertyData(device, &address, 0, nil, &size, &value) == noErr else { return nil } + return value as String? +} + +private func microphoneDevices() throws -> [(id: AudioDeviceID, uid: String, name: String)] { + var address = AudioObjectPropertyAddress(mSelector: kAudioHardwarePropertyDevices, mScope: kAudioObjectPropertyScopeGlobal, mElement: kAudioObjectPropertyElementMain) + var size: UInt32 = 0 + guard AudioObjectGetPropertyDataSize(AudioObjectID(kAudioObjectSystemObject), &address, 0, nil, &size) == noErr, + size <= 4096 else { throw CaptureFailure(description: "microphone list is unavailable") } + var devices = [AudioDeviceID](repeating: 0, count: Int(size) / MemoryLayout.size) + if devices.isEmpty { return [] } + let status = devices.withUnsafeMutableBytes { buffer in + AudioObjectGetPropertyData(AudioObjectID(kAudioObjectSystemObject), &address, 0, nil, &size, buffer.baseAddress!) + } + guard status == noErr else { throw CaptureFailure(description: "microphone list is unavailable") } + return devices.compactMap { device in + var input = AudioObjectPropertyAddress(mSelector: kAudioDevicePropertyStreams, mScope: kAudioDevicePropertyScopeInput, mElement: kAudioObjectPropertyElementMain) + var bytes: UInt32 = 0 + guard AudioObjectGetPropertyDataSize(device, &input, 0, nil, &bytes) == noErr, bytes > 0, + let uid = audioDeviceString(device, kAudioDevicePropertyDeviceUID), + let name = audioDeviceString(device, kAudioObjectPropertyName) else { return nil } + return (id: device, uid: uid, name: name) + } +} + +private func listMicrophones() throws { + let devices = try microphoneDevices().map { ["id": $0.uid, "label": $0.name] } + guard devices.count <= 64 else { throw CaptureFailure(description: "too many microphones") } + FileHandle.standardOutput.write(try JSONSerialization.data(withJSONObject: devices)) +} + +private func captureMicrophone(deviceUID: String = "") throws { + let done = StopSignal() + let control = DispatchQueue(label: "piik.microphone.control") + let worker = DispatchQueue(label: "piik.microphone.pcm") + let slots = DispatchSemaphore(value: 4) + let output = AudioCaptureOutput(writer: ProtocolWriter(), done: done) + var engine: AVAudioEngine? + var observer: NSObjectProtocol? + var tapped = false + readInput(done: done) + defer { + control.sync { + if let observer { NotificationCenter.default.removeObserver(observer) } + if tapped { engine?.inputNode.removeTap(onBus: 0) } + engine?.stop() + engine = nil + } + worker.sync {} // Queued work observes stopped; no tail audio is flushed. + } + let permitted: (Bool) -> Void = { allowed in + control.async { + guard !done.isStopped else { return } + guard allowed else { + done.signal(CaptureFailure(description: "microphone permission denied")) + return + } + do { + let capture = AVAudioEngine() + engine = capture + let input = capture.inputNode + if !deviceUID.isEmpty { + guard let selected = try microphoneDevices().first(where: { $0.uid == deviceUID }), + let unit = input.audioUnit else { throw CaptureFailure(description: "selected microphone is unavailable") } + var device = selected.id + guard AudioUnitSetProperty(unit, kAudioOutputUnitProperty_CurrentDevice, kAudioUnitScope_Global, 0, + &device, UInt32(MemoryLayout.size)) == noErr else { + throw CaptureFailure(description: "selected microphone could not open") + } + } + let format = input.outputFormat(forBus: 0) + guard format.sampleRate > 0, format.channelCount > 0, + let target = AVAudioFormat(commonFormat: .pcmFormatInt16, + sampleRate: 48_000, channels: 2, interleaved: true), + let converter = AVAudioConverter(from: format, to: target) else { + throw CaptureFailure(description: "microphone format is unavailable") + } + converter.channelMap = [0, format.channelCount > 1 ? 1 : 0] + observer = NotificationCenter.default.addObserver( + forName: .AVAudioEngineConfigurationChange, object: capture, queue: nil + ) { _ in done.signal(CaptureFailure(description: "microphone device changed")) } + input.installTap(onBus: 0, bufferSize: 960, format: format) { buffer, _ in + guard !done.isStopped, buffer.frameLength > 0, + slots.wait(timeout: .now()) == .success else { return } + guard let copy = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: buffer.frameLength) else { + slots.signal() + return + } + copy.frameLength = buffer.frameLength + let source = UnsafeMutableAudioBufferListPointer(buffer.mutableAudioBufferList) + let destination = UnsafeMutableAudioBufferListPointer(copy.mutableAudioBufferList) + for index in 0.. 1, arguments[1] == "--capture-audio" { try await captureAudio(arguments) } else if arguments.count > 1, arguments[1] == "--encoded-video" { diff --git a/native/capture/linux/main.c b/native/capture/linux/main.c index 464b8f5..38143b6 100644 --- a/native/capture/linux/main.c +++ b/native/capture/linux/main.c @@ -316,10 +316,10 @@ static int write_probe(void) { g_string_append_printf( json, "{\"protocol\":%d,\"platform\":\"linux\",\"platformBuild\":%s," - "\"videoCapture\":%s,\"softwareVP8\":false,\"processAudio\":false,\"systemAudio\":%s," + "\"videoCapture\":%s,\"softwareVP8\":false,\"processAudio\":false,\"systemAudio\":%s,\"microphone\":%s," "\"adapters\":[", kCaptureProtocol, build, video ? "true" : "false", - audio ? "true" : "false"); + audio ? "true" : "false", audio ? "true" : "false"); g_free(build); if (encoders->len > 0) { g_string_append(json, @@ -1234,27 +1234,69 @@ cleanup: return result; } +static int write_microphones(void) { + GstDeviceMonitor *monitor = gst_device_monitor_new(); + gst_device_monitor_add_filter(monitor, "Audio/Source", NULL); + if (!gst_device_monitor_start(monitor)) { gst_object_unref(monitor); return 2; } + GList *devices = gst_device_monitor_get_devices(monitor); + GString *json = g_string_new("["); + guint count = 0; + for (GList *item = devices; item != NULL && count < 64; item = item->next) { + GstElement *source = gst_device_create_element(GST_DEVICE(item->data), NULL); + GstElementFactory *factory = source == NULL ? NULL : gst_element_get_factory(source); + if (factory != NULL && strcmp(gst_plugin_feature_get_name(GST_PLUGIN_FEATURE(factory)), "pulsesrc") == 0) { + gchar *device = NULL; + g_object_get(source, "device", &device, NULL); + if (device != NULL && device[0] != '\0') { + gchar *name = gst_device_get_display_name(GST_DEVICE(item->data)); + gchar *id_json = json_string(device), *name_json = json_string(name); + g_string_append_printf(json, "%s{\"id\":%s,\"label\":%s}", count++ == 0 ? "" : ",", id_json, name_json); + g_free(id_json); g_free(name_json); g_free(name); + } + g_free(device); + } + if (source != NULL) gst_object_unref(source); + } + g_string_append_c(json, ']'); + fputs(json->str, stdout); + g_string_free(json, TRUE); + g_list_free_full(devices, gst_object_unref); + gst_device_monitor_stop(monitor); + gst_object_unref(monitor); + return ferror(stdout) ? 2 : 0; +} + static int capture_audio(int count, char **values) { - if (count != 5 || strcmp(values[1], "--capture-audio") != 0 || + gboolean microphone = (count == 2 || (count == 4 && strcmp(values[2], "--device") == 0 && + values[3][0] != '\0' && strlen(values[3]) <= 512)) && strcmp(values[1], "--capture-microphone") == 0; + if (!microphone && (count != 5 || strcmp(values[1], "--capture-audio") != 0 || (strcmp(values[2], "picker") != 0 && strcmp(values[2], "display") != 0) || strcmp(values[3], "0") != 0 || - strcmp(values[4], "0") != 0 || !audio_stack_available()) { + strcmp(values[4], "0") != 0)) { return 2; } GError *error = NULL; - GstElement *pipeline = gst_parse_launch( - "pulsesrc device=@DEFAULT_MONITOR@ do-timestamp=true ! " + if (!audio_stack_available()) return 2; + gchar *description = g_strconcat( + microphone ? "pulsesrc name=microphone do-timestamp=true ! " : "pulsesrc device=@DEFAULT_MONITOR@ do-timestamp=true ! ", "queue max-size-buffers=4 max-size-bytes=0 max-size-time=0 " "leaky=downstream ! audioconvert ! audioresample ! " "audio/x-raw,format=S16LE,rate=48000,channels=2,layout=interleaved ! " - "appsink name=output emit-signals=true sync=false max-buffers=4 drop=true", - &error); + "appsink name=output emit-signals=true sync=false max-buffers=4 drop=true", NULL); + GstElement *pipeline = gst_parse_launch(description, &error); + g_free(description); if (pipeline == NULL || error != NULL) { g_clear_error(&error); if (pipeline != NULL) gst_object_unref(pipeline); return 2; } + if (microphone && count == 4) { + GstElement *input = gst_bin_get_by_name(GST_BIN(pipeline), "microphone"); + if (input == NULL) { gst_object_unref(pipeline); return 2; } + g_object_set(input, "device", values[3], NULL); + gst_object_unref(input); + } CaptureRun run = { .loop = g_main_loop_new(NULL, FALSE), .pipeline = pipeline, @@ -1276,13 +1318,15 @@ static int capture_audio(int count, char **values) { } int main(int argc, char **argv) { - gst_init(&argc, &argv); + // Piik owns this CLI. Opaque device IDs must never become GStreamer flags. + gst_init(NULL, NULL); if (argc == 2 && strcmp(argv[1], "--probe") == 0) return write_probe(); if (argc == 2 && strcmp(argv[1], "--list") == 0) return write_sources(); + if (argc == 2 && strcmp(argv[1], "--list-microphones") == 0) return write_microphones(); if (argc > 1 && (strcmp(argv[1], "--capture-video") == 0 || strcmp(argv[1], "--encoded-video") == 0)) { return capture_video(argc, argv); } - if (argc > 1 && strcmp(argv[1], "--capture-audio") == 0) { + if (argc > 1 && (strcmp(argv[1], "--capture-audio") == 0 || strcmp(argv[1], "--capture-microphone") == 0)) { return capture_audio(argc, argv); } fprintf(stderr, "Piik capture unavailable: unsupported command\n"); diff --git a/native/capture/windows/README.md b/native/capture/windows/README.md index 59dc7e7..7bd87e6 100644 --- a/native/capture/windows/README.md +++ b/native/capture/windows/README.md @@ -46,7 +46,8 @@ npm run check:native ``` The checks include synthetic output-worker replacement and failure cases using -WARP and a test codec. They do not capture a screen or validate a physical encoder. +WARP and a test codec, plus preview cleanup when WGC rejects a hidden test window. +They do not capture a user screen or validate a physical encoder. The implementation follows Microsoft's MIT-licensed reference samples and official API contracts without copying their WIL framework. The retained MF diff --git a/native/capture/windows/capture_target.cpp b/native/capture/windows/capture_target.cpp index 3b1de55..9f74c3c 100644 --- a/native/capture/windows/capture_target.cpp +++ b/native/capture/windows/capture_target.cpp @@ -1,6 +1,8 @@ #define NOMINMAX #include #include +#include +#include #include #include #include @@ -921,6 +923,41 @@ HRESULT CapturePreview(TargetKind kind, UINT64 source_id, } // namespace +int WriteMicrophoneList() { + if (FAILED(CoInitializeEx(nullptr, COINIT_MULTITHREADED))) return 2; + const int result = []() { + ComPtr enumerator; + ComPtr devices; + if (FAILED(CoCreateInstance(__uuidof(MMDeviceEnumerator), nullptr, CLSCTX_ALL, IID_PPV_ARGS(&enumerator))) || + FAILED(enumerator->EnumAudioEndpoints(eCapture, DEVICE_STATE_ACTIVE, &devices))) return 2; + UINT count = 0; + if (FAILED(devices->GetCount(&count)) || count > 64) return 2; + std::cout << '['; + bool first = true; + for (UINT index = 0; index < count; ++index) { + ComPtr device; + ComPtr properties; + if (FAILED(devices->Item(index, &device)) || FAILED(device->OpenPropertyStore(STGM_READ, &properties))) continue; + LPWSTR id = nullptr; + PROPVARIANT name; + PropVariantInit(&name); + if (SUCCEEDED(device->GetId(&id)) && SUCCEEDED(properties->GetValue(PKEY_Device_FriendlyName, &name)) && + name.vt == VT_LPWSTR && name.pwszVal != nullptr) { + if (!first) std::cout << ','; + first = false; + std::cout << "{\"id\":" << JsonString(Utf8(id)) + << ",\"label\":" << JsonString(Utf8(name.pwszVal)) << '}'; + } + CoTaskMemFree(id); + PropVariantClear(&name); + } + std::cout << ']'; + return std::cout.good() ? 0 : 2; + }(); + CoUninitialize(); + return result; +} + int WriteSourceList() { std::vector targets; if (!EnumDisplayMonitors(nullptr, nullptr, CollectDisplay, diff --git a/native/capture/windows/capture_target.h b/native/capture/windows/capture_target.h index 94898b1..e1ccf2c 100644 --- a/native/capture/windows/capture_target.h +++ b/native/capture/windows/capture_target.h @@ -9,6 +9,7 @@ namespace piik::capture { enum class TargetKind { window, display }; int WriteSourceList(); +int WriteMicrophoneList(); HRESULT WriteSourcePreview(TargetKind kind, UINT64 source_id, DWORD pid, UINT64 expected_creation_time); HRESULT ValidateProcessTarget(DWORD pid, UINT64 expected_creation_time); diff --git a/native/capture/windows/main.cpp b/native/capture/windows/main.cpp index a37142c..92eb483 100644 --- a/native/capture/windows/main.cpp +++ b/native/capture/windows/main.cpp @@ -920,9 +920,10 @@ piik::capture::TargetKind ParseTargetKind(const wchar_t* value) { } struct ProductArguments final { - enum class Mode { list, probe, preview, audio, video, encoded } mode = Mode::list; + enum class Mode { list, microphones, probe, preview, audio, microphone, video, encoded } mode = Mode::list; piik::capture::TargetKind target_kind = piik::capture::TargetKind::window; + std::wstring microphone_device; DWORD pid = 0; UINT64 creation_time = 0; UINT64 source_id = 0; @@ -1005,6 +1006,18 @@ ProductArguments ParseProductArguments(int count, wchar_t** values) { ParseNonNegativeUint64(values[5], "argument-creation-time"); return arguments; } + if (count == 2 && std::wstring(values[1]) == L"--list-microphones") { + arguments.mode = ProductArguments::Mode::microphones; + return arguments; + } + if ((count == 2 || (count == 4 && std::wstring(values[2]) == L"--device")) && std::wstring(values[1]) == L"--capture-microphone") { + if (count == 4) { + arguments.microphone_device = values[3]; + if (arguments.microphone_device.empty() || arguments.microphone_device.size() > 512) Fail("argument-device", "microphone device is invalid"); + } + arguments.mode = ProductArguments::Mode::microphone; + return arguments; + } if (count == 5 && std::wstring(values[1]) == L"--capture-audio") { arguments.mode = ProductArguments::Mode::audio; arguments.target_kind = ParseTargetKind(values[2]); @@ -1105,6 +1118,9 @@ HRESULT RunAudioCapture(const ProductArguments& arguments) { piik::capture::kAudioChunkDuration100ns, data, size); }; + if (arguments.mode == ProductArguments::Mode::microphone) { + return piik::capture::CaptureMicrophone(arguments.microphone_device, stop.get(), should_stop, ready, pcm); + } if (arguments.target_kind == piik::capture::TargetKind::display) { return piik::capture::CaptureSystemAudio( stop.get(), should_stop, ready, pcm); @@ -1159,7 +1175,7 @@ void WriteCapabilityProbe() { << ",\"videoCapture\":" << (window_capture ? "true" : "false") << ",\"captureBorderControl\":" << (window_capture && piik::capture::CaptureBorder::Supported() ? "true" : "false") - << ",\"softwareVP8\":true" + << ",\"microphone\":true,\"softwareVP8\":true" << ",\"processAudio\":" << (process_audio ? "true" : "false") << ",\"systemAudio\":" << (system_audio ? "true" : "false") @@ -2063,6 +2079,7 @@ int wmain(int argc, wchar_t** argv) { int wmain(int argc, wchar_t** argv) { try { ProductArguments arguments = ParseProductArguments(argc, argv); + if (arguments.mode == ProductArguments::Mode::microphones) return piik::capture::WriteMicrophoneList(); if (arguments.mode == ProductArguments::Mode::list) { return piik::capture::WriteSourceList(); } @@ -2081,7 +2098,7 @@ int wmain(int argc, wchar_t** argv) { "capture-preview"); return 0; } - if (arguments.mode == ProductArguments::Mode::audio) { + if (arguments.mode == ProductArguments::Mode::audio || arguments.mode == ProductArguments::Mode::microphone) { Check(RunAudioCapture(arguments), "process-audio-capture"); return 0; } diff --git a/native/capture/windows/process_audio.cpp b/native/capture/windows/process_audio.cpp index d6d6b40..ca84706 100644 --- a/native/capture/windows/process_audio.cpp +++ b/native/capture/windows/process_audio.cpp @@ -77,13 +77,22 @@ HRESULT ActivateProcessLoopback(DWORD pid, HANDLE completed, return result; } -HRESULT ActivateSystemLoopback(ComPtr* client) { +HRESULT ActivateEndpoint(ComPtr* client, bool microphone = false, const std::wstring& device_id = L"") { ComPtr enumerator; HRESULT result = CoCreateInstance(__uuidof(MMDeviceEnumerator), nullptr, CLSCTX_ALL, IID_PPV_ARGS(&enumerator)); ComPtr device; if (SUCCEEDED(result)) { - result = enumerator->GetDefaultAudioEndpoint(eRender, eConsole, &device); + result = device_id.empty() + ? enumerator->GetDefaultAudioEndpoint(microphone ? eCapture : eRender, microphone ? eCommunications : eConsole, &device) + : enumerator->GetDevice(device_id.c_str(), &device); + if (SUCCEEDED(result) && microphone) { + ComPtr endpoint; + EDataFlow flow = eAll; + result = device.As(&endpoint); + if (SUCCEEDED(result)) result = endpoint->GetDataFlow(&flow); + if (SUCCEEDED(result) && flow != eCapture) result = E_INVALIDARG; + } } if (SUCCEEDED(result)) { result = device->Activate(__uuidof(IAudioClient), CLSCTX_ALL, nullptr, @@ -92,7 +101,7 @@ HRESULT ActivateSystemLoopback(ComPtr* client) { return result; } -HRESULT CaptureLoopbackAudio(ComPtr client, HANDLE process, +HRESULT CaptureAudioFrames(ComPtr client, HANDLE process, bool loopback, HANDLE stop_event, const StopProbe& stop_probe, const ReadyWriter& ready_writer, const PCMWriter& writer) { @@ -115,7 +124,7 @@ HRESULT CaptureLoopbackAudio(ComPtr client, HANDLE process, format.nAvgBytesPerSec = kAudioSampleRate * format.nBlockAlign; if (SUCCEEDED(result)) { result = client->Initialize(AUDCLNT_SHAREMODE_SHARED, - AUDCLNT_STREAMFLAGS_LOOPBACK | AUDCLNT_STREAMFLAGS_EVENTCALLBACK | + (loopback ? AUDCLNT_STREAMFLAGS_LOOPBACK : 0) | AUDCLNT_STREAMFLAGS_EVENTCALLBACK | AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM | AUDCLNT_STREAMFLAGS_SRC_DEFAULT_QUALITY, 0, 0, &format, nullptr); } @@ -221,7 +230,7 @@ bool ProcessAudioAvailable() { bool SystemAudioAvailable() { ComPtr client; - return SUCCEEDED(ActivateSystemLoopback(&client)) && client != nullptr; + return SUCCEEDED(ActivateEndpoint(&client)) && client != nullptr; } HRESULT CaptureProcessAudio(DWORD pid, UINT64 expectedCreationTime, @@ -246,7 +255,7 @@ HRESULT CaptureProcessAudio(DWORD pid, UINT64 expectedCreationTime, result = ActivateProcessLoopback(pid, completed, &client); } if (SUCCEEDED(result)) { - result = CaptureLoopbackAudio(client, process, stop_event, stop_probe, + result = CaptureAudioFrames(client, process, true, stop_event, stop_probe, ready_writer, writer); } if (completed != nullptr) CloseHandle(completed); @@ -262,9 +271,9 @@ HRESULT CaptureSystemAudio(HANDLE stop_event, const StopProbe& stop_probe, HRESULT com_result = CoInitializeEx(nullptr, COINIT_MULTITHREADED); if (FAILED(com_result)) return com_result; ComPtr client; - HRESULT result = ActivateSystemLoopback(&client); + HRESULT result = ActivateEndpoint(&client); if (SUCCEEDED(result)) { - result = CaptureLoopbackAudio(client, nullptr, stop_event, stop_probe, + result = CaptureAudioFrames(client, nullptr, true, stop_event, stop_probe, ready_writer, writer); } client.Reset(); @@ -272,4 +281,18 @@ HRESULT CaptureSystemAudio(HANDLE stop_event, const StopProbe& stop_probe, return result; } +HRESULT CaptureMicrophone(const std::wstring& device_id, HANDLE stop_event, const StopProbe& stop_probe, + const ReadyWriter& ready_writer, const PCMWriter& writer) { + HRESULT result = CoInitializeEx(nullptr, COINIT_MULTITHREADED); + if (FAILED(result)) return result; + ComPtr client; + result = ActivateEndpoint(&client, true, device_id); + if (SUCCEEDED(result)) { + result = CaptureAudioFrames(client, nullptr, false, stop_event, stop_probe, ready_writer, writer); + } + client.Reset(); + CoUninitialize(); + return result; +} + } // namespace piik::capture diff --git a/native/capture/windows/process_audio.h b/native/capture/windows/process_audio.h index 692b991..55e0215 100644 --- a/native/capture/windows/process_audio.h +++ b/native/capture/windows/process_audio.h @@ -4,6 +4,7 @@ #include #include +#include namespace piik::capture { @@ -22,6 +23,8 @@ using StopProbe = std::function; bool ProcessAudioAvailable(); bool SystemAudioAvailable(); +HRESULT CaptureMicrophone(const std::wstring& device_id, HANDLE stop_event, const StopProbe& stop_probe, + const ReadyWriter& ready_writer, const PCMWriter& writer); HRESULT CaptureProcessAudio(DWORD pid, UINT64 expected_creation_time, HANDLE stop_event, const StopProbe& stop_probe, const ReadyWriter& ready_writer, diff --git a/scripts/access-privacy-gate.ts b/scripts/access-privacy-gate.ts index 001cc21..e2ffc1e 100644 --- a/scripts/access-privacy-gate.ts +++ b/scripts/access-privacy-gate.ts @@ -329,7 +329,7 @@ async function main(): Promise { throw new Error("Room preparation did not enable the Viewer password"); } report.roomPrepared = true; - chrome = launchChrome(chromePath, debugPort, profile, [ + chrome = await launchChrome(chromePath, debugPort, profile, [ "--headless=new", "--no-first-run", "--disable-extensions", "--disable-logging", ], { cwd: root }); diff --git a/scripts/app-browser-fanout-gate.ts b/scripts/app-browser-fanout-gate.ts index b38b986..44d38ea 100644 --- a/scripts/app-browser-fanout-gate.ts +++ b/scripts/app-browser-fanout-gate.ts @@ -1,4 +1,5 @@ import { spawn, spawnSync, type ChildProcessWithoutNullStreams } from "node:child_process"; +import { once } from "node:events"; import { mkdtemp, mkdir, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join, resolve } from "node:path"; @@ -6,7 +7,7 @@ import { CdpConnection, cleanupRun, createPage, evaluate, fetchJsonBefore, launchChrome, reservePort, waitForSample, waitForVersion, type PageHandle, } from "./browser-gate-harness"; -import { decodeAppEndpoint } from "./app-gate-endpoint"; +import { readAppEndpoint } from "./app-gate-endpoint"; const ROOT = resolve(import.meta.dirname, ".."); const BUILD_ROOT = join(ROOT, "build/go-check"); @@ -96,7 +97,8 @@ const probe = String.raw`(() => { })()`; function build(command: string, args: string[]): void { - const result = spawnSync(command, args, { cwd: ROOT, encoding: "utf8", windowsHide: true }); + const result = spawnSync(command, args, { cwd: ROOT, encoding: "utf8", windowsHide: true, + timeout: 120_000, killSignal: "SIGKILL" }); if (result.error) throw result.error; if (result.status !== 0) { throw new Error((result.stderr || result.stdout).trim() || command + " failed"); @@ -136,12 +138,18 @@ async function startRoomServer( }); child.stdout.resume(); child.stderr.resume(); - await waitForSample( - (deadline) => fetchJsonBefore<{ status: string }>(`http://127.0.0.1:${port}/healthz`, deadline), - (value) => value.status === "ok", - 20000, - ); - return child; + try { + await once(child, "spawn"); + await waitForSample( + (deadline) => fetchJsonBefore<{ status: string }>(`http://127.0.0.1:${port}/healthz`, deadline), + (value) => value.status === "ok", + 20000, + ); + return child; + } catch (error) { + await cleanupRun({ server: child, chrome: null, native: null, cdp: null, profile: null, ports: [port] }); + throw error; + } } async function main(): Promise { @@ -167,14 +175,11 @@ async function main(): Promise { "--site", origin, "--config", join(profile, "client.json"), "--capture-process", join(BUILD_ROOT, "piik-capture.exe"), ], { windowsHide: true, stdio: "pipe", env: { ...process.env, PIIK_CLIENT_GATE_NO_BROWSER: "true" } }); + await once(app, "spawn"); app.stderr.resume(); - let output = ""; - app.stdout.on("data", (chunk: Buffer) => { - output += chunk.toString(); - if (!nativePort && output.includes("\n")) nativePort = decodeAppEndpoint(output.split("\n")[0]!).port; - }); - await waitForSample(async () => nativePort, (value) => value > 0, 15000); - chrome = launchChrome(chromePath, debugPort, profile, [ + nativePort = (await readAppEndpoint(app)).port; + app.stdout.resume(); + chrome = await launchChrome(chromePath, debugPort, profile, [ "--headless=new", "--no-first-run", "--no-default-browser-check", "--no-proxy-server", "--autoplay-policy=no-user-gesture-required", "--disable-background-timer-throttling", @@ -215,12 +220,10 @@ async function main(): Promise { "--site", relayOrigin, "--config", join(profile,"relay-client.json"), "--capture-process", join(BUILD_ROOT,"piik-capture.exe"), ], { windowsHide:true, stdio:"pipe", env:{...process.env,PIIK_CLIENT_GATE_NO_BROWSER:"true"} }); - relayApp.stderr.resume(); let relayOutput=""; - relayApp.stdout.on("data", (chunk:Buffer) => { - relayOutput+=chunk.toString(); - if (!relayPort && relayOutput.includes("\n")) relayPort=decodeAppEndpoint(relayOutput.split("\n")[0]!).port; - }); - await waitForSample(async()=>relayPort,(value)=>value>0,15000); + await once(relayApp, "spawn"); + relayApp.stderr.resume(); + relayPort = (await readAppEndpoint(relayApp)).port; + relayApp.stdout.resume(); const viewers: PageHandle[] = []; for (let index=0; index<2; index++) { stage = `viewer-${index+1}`; diff --git a/scripts/app-icons.mjs b/scripts/app-icons.mjs index fd46cb3..5c6af9e 100644 --- a/scripts/app-icons.mjs +++ b/scripts/app-icons.mjs @@ -139,6 +139,7 @@ function writeMacAssets(packageRoot, version, revision, entries) { "CFBundlePackageTypeAPPL", `CFBundleShortVersionString${bundleVersion}`, "NSScreenCaptureUsageDescriptionShare a screen or application selected by you.", + "NSMicrophoneUsageDescriptionInclude your voice while sharing.", "NSAudioCaptureUsageDescriptionShare sound from the selected screen or application.", `CFBundleVersion${bundleVersion}`, `PiikSourceRevision${revision}`, diff --git a/scripts/app-link-gate.ts b/scripts/app-link-gate.ts index 7ca830e..815b16b 100644 --- a/scripts/app-link-gate.ts +++ b/scripts/app-link-gate.ts @@ -1,4 +1,5 @@ import { spawn, spawnSync, type ChildProcessWithoutNullStreams } from "node:child_process"; +import { once } from "node:events"; import { mkdir, mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join, resolve } from "node:path"; @@ -66,12 +67,16 @@ function run( args: string[], cwd = ROOT, environment: NodeJS.ProcessEnv = process.env, + timeoutMs = 120_000, ): string { const result = spawnSync(command, args, { cwd, env: environment, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"], + timeout: timeoutMs, + killSignal: "SIGKILL", + windowsHide: true, }); if (result.error) throw result.error; if (result.status !== 0) { @@ -121,8 +126,9 @@ async function waitForRemotePage( const result = spawnSync(ssh, [ ...transport, destination, - "curl", "--fail", "--silent", url, - ], { encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }); + "curl", "--fail", "--silent", "--connect-timeout", "5", "--max-time", "8", url, + ], { encoding: "utf8", stdio: ["ignore", "pipe", "pipe"], windowsHide: true, + timeout: Math.max(1, Math.min(10_000, deadline - Date.now())), killSignal: "SIGKILL" }); if (result.status === 0 && result.stdout.includes('
')) { return result.stdout; } @@ -134,12 +140,14 @@ async function waitForRemotePage( async function createInvitation(port: number, password: string): Promise { const origin = `http://localhost:${port}`; const access = await fetch(`${origin}/api/site-access`, { + signal: AbortSignal.timeout(5_000), method: "POST", headers: { Authorization: `Bearer ${password}`, Origin: origin }, }); const cookie = access.headers.get("set-cookie")?.split(";", 1)[0]; if (!access.ok) throw new Error("Local access authentication failed"); const room = await fetch(`${origin}/api/rooms`, { + signal: AbortSignal.timeout(5_000), method: "POST", headers: { "Content-Type": "application/json", @@ -156,15 +164,15 @@ async function createInvitation(port: number, password: string): Promise } async function stopApp(child: ChildProcessWithoutNullStreams | null): Promise { - if (!child || child.exitCode !== null || child.signalCode !== null) return true; - child.stdin.write("\n"); + if (!child?.pid || child.exitCode !== null || child.signalCode !== null) return true; + const exited = once(child, "exit"); + child.stdin.once("error", () => {}); // Exit may race the stop request (EPIPE). try { - await withDeadline( - () => new Promise((resolveExit) => child.once("exit", () => resolveExit())), - Date.now() + 8_000, - ); + child.stdin.end("\n"); + await withDeadline(() => exited, Date.now() + 8_000); } catch { - child.kill(); + child.kill("SIGKILL"); + await withDeadline(() => exited, Date.now() + 3_000).catch(() => undefined); } return child.exitCode !== null || child.signalCode !== null; } @@ -172,11 +180,13 @@ async function stopApp(child: ChildProcessWithoutNullStreams | null): Promise { const deadline = Date.now() + 15_000; while (Date.now() < deadline) { + const signal = AbortSignal.timeout(Math.max(1, Math.min(5_000, deadline - Date.now()))); try { - const response = await fetch(`${origin}/healthz`, { cache: "no-store" }); + const response = await fetch(`${origin}/healthz`, { cache: "no-store", signal }); if (!response.ok) return true; } catch { - return true; + // A silent request is not proof that the invitation was retired. + if (!signal.aborted) return true; } await new Promise((resolveWait) => setTimeout(resolveWait, 250)); } @@ -239,6 +249,7 @@ async function main(): Promise { stdio: "pipe", windowsHide: true, }); + await once(app, "spawn"); const output = new LineCapture(app); const accessLine = await output.wait( (line) => line.startsWith("Local access password: ") || line === "Local access: open", @@ -283,7 +294,7 @@ async function main(): Promise { ...transport, destination, websocketProbe, - ]); + ], ROOT, process.env, Math.max(1, Math.min(10_000, websocketDeadline - Date.now()))); result.remoteWebSocket = upgrade.includes("101 Switching Protocols"); if (!result.remoteWebSocket) { await new Promise((resolveWait) => setTimeout(resolveWait, 250)); diff --git a/scripts/app-local-gate.ts b/scripts/app-local-gate.ts index 2434709..ab616a3 100644 --- a/scripts/app-local-gate.ts +++ b/scripts/app-local-gate.ts @@ -124,7 +124,7 @@ async function main(): Promise { "piik-client": "1", }).toString(); - browser = launchChrome(browserPath, debugPort, profile, [ + browser = await launchChrome(browserPath, debugPort, profile, [ "--headless=new", ...(process.env.PIIK_CLIENT_GATE_NO_SANDBOX === "true" ? ["--no-sandbox"] diff --git a/scripts/app-loopback-gate.ts b/scripts/app-loopback-gate.ts index e0fd9f0..f7982ab 100644 --- a/scripts/app-loopback-gate.ts +++ b/scripts/app-loopback-gate.ts @@ -199,7 +199,7 @@ async function main(): Promise { portServer.close((error) => error ? rejectPort(error) : resolvePort(address.port)); }); }); - browser = launchChrome(browserPath, debugPort, profile, [ + browser = await launchChrome(browserPath, debugPort, profile, [ "--headless=new", ...(disposableNoSandbox ? ["--no-sandbox"] : []), "--no-first-run", diff --git a/scripts/app-media-gate.ts b/scripts/app-media-gate.ts index 958ab02..27b1ad6 100644 --- a/scripts/app-media-gate.ts +++ b/scripts/app-media-gate.ts @@ -743,7 +743,7 @@ async function main(): Promise { } server = await startPageServer(pagePort); - chrome = launchChrome(chromePath, debugPort, profile, [ + chrome = await launchChrome(chromePath, debugPort, profile, [ "--no-first-run", "--no-default-browser-check", "--disable-extensions", diff --git a/scripts/app-native-host-gate.ts b/scripts/app-native-host-gate.ts index 0cf1a83..ca281f8 100644 --- a/scripts/app-native-host-gate.ts +++ b/scripts/app-native-host-gate.ts @@ -126,12 +126,16 @@ function run( args: string[], cwd = ROOT, environment: NodeJS.ProcessEnv = process.env, + timeoutMs = 120_000, ): string { const result = spawnSync(command, args, { cwd, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"], env: environment, + timeout: timeoutMs, + killSignal: "SIGKILL", + windowsHide: true, }); if (result.error) throw result.error; if (result.status !== 0) { @@ -163,6 +167,8 @@ async function stopChild(child: ChildProcessWithoutNullStreams | ChildProcess | if (process.platform === "win32" && child.pid !== undefined) { spawnSync("taskkill.exe", ["/PID", String(child.pid), "/T", "/F"], { stdio: "ignore", + timeout: 5_000, + windowsHide: true, }); try { await waitForChild(child, 2_000); @@ -299,8 +305,8 @@ async function runRemotePeerGate( const remotePath = `/tmp/piik-peer-gate-${randomBytes(8).toString("hex")}`; const transportOptions = remoteTransportOptions(options); try { - run(scp, [...transportOptions, binary, `${destination}:${remotePath}`]); - run(ssh, [...transportOptions, destination, "chmod", "700", remotePath]); + run(scp, [...transportOptions, binary, `${destination}:${remotePath}`], ROOT, process.env, 30_000); + run(ssh, [...transportOptions, destination, "chmod", "700", remotePath], ROOT, process.env, 15_000); const remote = spawn( ssh, [ @@ -367,7 +373,7 @@ async function runRemotePeerGate( }; } finally { try { - run(ssh, [...transportOptions, destination, "rm", "-f", remotePath]); + run(ssh, [...transportOptions, destination, "rm", "-f", remotePath], ROOT, process.env, 10_000); } catch { // The temporary binary is harmless if an already-closed SSH session // prevents cleanup; the gate never places it in the repository. @@ -425,7 +431,7 @@ export async function startSourceBrowser( child: ChildProcessWithoutNullStreams; cdp: CdpConnection; }> { - const child = launchChrome(chromePath, debugPort, profile, [ + const child = await launchChrome(chromePath, debugPort, profile, [ "--no-first-run", "--no-default-browser-check", "--disable-extensions", "--disable-logging", "--disable-background-timer-throttling", @@ -435,12 +441,17 @@ export async function startSourceBrowser( ]); child.stdout.resume(); child.stderr.resume(); - const version = await waitForVersion(debugPort, child); - const cdp = await CdpConnection.connect( - version.webSocketDebuggerUrl, - Date.now() + 10_000, - ); - return { child, cdp }; + try { + const version = await waitForVersion(debugPort, child); + const cdp = await CdpConnection.connect( + version.webSocketDebuggerUrl, + Date.now() + 10_000, + ); + return { child, cdp }; + } catch (error) { + await cleanupRun({ chrome: child, cdp: null, native: null, server: null, profile: null, ports: [debugPort] }); + throw error; + } } export async function closeSourceBrowser( @@ -461,7 +472,7 @@ export async function closeSourceBrowser( export async function waitForCaptureWindow(captureBinary: string): Promise { await waitForValue( - async () => JSON.parse(run(captureBinary, ["--list"])) as Array<{ + async () => JSON.parse(run(captureBinary, ["--list"], ROOT, process.env, 5_000)) as Array<{ title?: unknown; }>, (targets) => targets.some((target) => @@ -746,7 +757,7 @@ async function main(): Promise { 15_000, ); stage = "host-browser"; - chrome = launchChrome(chromePath, debugPort, profile, [ + chrome = await launchChrome(chromePath, debugPort, profile, [ "--no-first-run", "--no-default-browser-check", "--disable-extensions", "--disable-logging", "--disable-background-timer-throttling", @@ -801,6 +812,7 @@ async function main(): Promise { 5_000, ); stage = "pre-share-codec"; + await clickHostControl(cdp, host, `document.querySelector('.lr-sharing-technical > summary')`); await clickHostControl(cdp, host, `([...document.querySelectorAll('button.lr-chip')].find((button) => button.textContent?.trim() === ${JSON.stringify(requestedCodec.toUpperCase())}))`, @@ -956,6 +968,9 @@ async function main(): Promise { Date.now() + 5_000, ); stage = "native-quality-presets"; + await clickHostControl(cdp, host, + `document.querySelector('button[aria-controls="host-advanced-door"]')`, + ); for (const [index, width, height] of [[0, 1280, 720], [1, 1920, 1080]] as const) { await clickHostControl(cdp, host, `document.querySelectorAll('button.lr-tile')[${index}]`); await waitForValue( @@ -974,12 +989,6 @@ async function main(): Promise { result.livePresetChanges += 1; } stage = "native-quality-controls"; - await evaluate( - cdp, - host, - `document.querySelector('button[aria-controls="host-advanced-door"]')?.click()`, - Date.now() + 5_000, - ); await waitForValue( (deadline) => evaluate( cdp!, @@ -1190,7 +1199,7 @@ async function main(): Promise { Date.now() + 5_000, ); await waitForValue((deadline) => evaluate(cdp!, host, - `Boolean(document.querySelector('#host-advanced-door .lr-door-body[aria-busy="true"]'))`, + `Boolean(document.querySelector('#host-advanced-door .lr-sharing-panel[aria-busy="true"]'))`, deadline, ), Boolean, 5_000); } finally { @@ -1219,7 +1228,7 @@ async function main(): Promise { })()`, deadline, ), Boolean, 20_000); await waitForValue((deadline) => evaluate(cdp!, host, - `Boolean(document.querySelector('#host-advanced-door .lr-door-body[aria-busy="false"]'))`, + `Boolean(document.querySelector('#host-advanced-door .lr-sharing-panel[aria-busy="false"]'))`, deadline, ), Boolean, 5_000); } @@ -1261,11 +1270,8 @@ async function main(): Promise { 10_000, ); const framesBeforeSourceChange = resumed.frames; - await evaluate( - cdp, - host, - `document.querySelector('[role="dialog"] button[data-native-source]')?.click()`, - Date.now() + 5_000, + await selectNativeSource(cdp, host, sourceKind, + sourceKind === "window" ? SOURCE_TITLE : undefined, ); stage = "native-source-replaced"; result.nativeSourceChanged = await waitForValue( diff --git a/scripts/browser-gate-harness.ts b/scripts/browser-gate-harness.ts index 4b7ec3b..72b8c0a 100644 --- a/scripts/browser-gate-harness.ts +++ b/scripts/browser-gate-harness.ts @@ -1,4 +1,5 @@ import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process"; +import { once } from "node:events"; import { lstat, readdir, rm } from "node:fs/promises"; import { createConnection, createServer } from "node:net"; import { tmpdir } from "node:os"; @@ -142,14 +143,14 @@ export class CdpConnection { } // launchChrome starts an isolated Chrome process with the shared CDP flags. -export function launchChrome( +export async function launchChrome( executable: string, debugPort: number, profile: string, flags: string[] = [], options: { cwd?: string; env?: NodeJS.ProcessEnv } = {}, -): ChildProcessWithoutNullStreams { - return spawn(executable, [ +): Promise { + const child = spawn(executable, [ `--remote-debugging-port=${debugPort}`, `--user-data-dir=${profile}`, ...flags, @@ -160,6 +161,8 @@ export function launchChrome( stdio: "pipe", windowsHide: true, }); + await once(child, "spawn"); + return child; } export function withDeadline( @@ -361,6 +364,7 @@ export async function cleanupRun( ? await stopProcessTree(resources.chrome) : true; if (resources.native && processRunning(resources.native)) { + resources.native.stdin.once("error", () => {}); // Exit may race the stop request (EPIPE). try { resources.native.stdin.write("\n"); } catch {} @@ -396,7 +400,7 @@ export async function cleanupRun( } function processRunning(child: ChildProcessWithoutNullStreams): boolean { - return child.exitCode === null && child.signalCode === null; + return child.pid !== undefined && child.exitCode === null && child.signalCode === null; } async function stopProcessTree( diff --git a/scripts/browser-local-pool-probe.ts b/scripts/browser-local-pool-probe.ts index a44070d..0bf7874 100644 --- a/scripts/browser-local-pool-probe.ts +++ b/scripts/browser-local-pool-probe.ts @@ -66,7 +66,7 @@ const server = createServer((request, response) => { await new Promise(r => server.listen(0, '127.0.0.1', r)); const port = (server.address() as { port: number }).port, debugPort = await reservePort(); const profile = await mkdtemp(join(tmpdir(), 'piik-client-media-')); -const chrome = launchChrome(process.env.CHROME_PATH || 'C:/Program Files/Google/Chrome/Application/chrome.exe', debugPort, profile, +const chrome = await launchChrome(process.env.CHROME_PATH || 'C:/Program Files/Google/Chrome/Application/chrome.exe', debugPort, profile, ['--headless=new', '--no-first-run', '--autoplay-policy=no-user-gesture-required', ...(background ? [] : ['--disable-background-timer-throttling', '--disable-renderer-backgrounding']), ...(nativeSource ? ['--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream'] : [])]); diff --git a/scripts/embedded-sfu-gate.ts b/scripts/embedded-sfu-gate.ts index 4647b05..81a75e9 100644 --- a/scripts/embedded-sfu-gate.ts +++ b/scripts/embedded-sfu-gate.ts @@ -180,7 +180,7 @@ async function main(): Promise { })()`, Date.now() + 5_000); if (audioState !== "running") throw new Error("Native source tone did not start"); } - chrome = launchChrome(chromePath, debugPort, profile, [ + chrome = await launchChrome(chromePath, debugPort, profile, [ "--headless=new", "--disable-gpu", "--no-first-run", "--no-default-browser-check", "--disable-extensions", "--disable-background-networking", "--autoplay-policy=no-user-gesture-required", "--disable-background-timer-throttling", "--disable-backgrounding-occluded-windows", diff --git a/src/client/components/living/CameraSources.tsx b/src/client/components/living/CameraSources.tsx new file mode 100644 index 0000000..6359141 --- /dev/null +++ b/src/client/components/living/CameraSources.tsx @@ -0,0 +1,49 @@ +import { useEffect, useState } from "react"; +import { loadCameraPreviews, type CameraPreview } from "../../media/camera-previews"; +import { useCopy } from "../../ui/copy"; +import { Glyph } from "../../ui/icons"; +import { CaptureSourceCard } from "./CaptureSourceCard"; +import { HintComic } from "./hints"; +import { Pill } from "./primitives"; + +export function useCameraSources(active: boolean, activeVideo?: HTMLVideoElement | null, + load = loadCameraPreviews) { + const [sources, setSources] = useState([]); + const [busy, setBusy] = useState(true); + const [error, setError] = useState<"host.camera.denied" | "host.camera.unavailable" | null>(null); + const [revision, refresh] = useState(0); + useEffect(() => { + if (!active) return; + const request = new AbortController(); + setBusy(true); setError(null); setSources([]); + void load(request.signal, result => { if (!request.signal.aborted) setSources(result); }, activeVideo).catch(error => { + if (!request.signal.aborted) setError(error instanceof DOMException && error.name === "NotAllowedError" + ? "host.camera.denied" : "host.camera.unavailable"); + }).finally(() => { if (!request.signal.aborted) setBusy(false); }); + return () => request.abort(); + }, [active, revision, activeVideo, load]); + return { sources, busy, error, refresh: () => refresh(value => value + 1) }; +} + +export function CameraSources({ sources, busy, error, selected, disabled, onSelect }: + Pick, "sources" | "busy" | "error"> & { + selected: string; disabled: boolean; onSelect: (deviceId: string) => void; + }) { + const { t, vis } = useCopy(); + return <> +
+ {sources.map((source, index) => onSelect(source.id)} />)} +
+ {busy && + {!vis && t("host.sourcePicker.loading")} + } + {error ? + : !busy && !sources.length ? + {!vis && t("host.sourcePicker.empty")} + : null} + ; +} diff --git a/src/client/components/living/CaptureSourceCard.tsx b/src/client/components/living/CaptureSourceCard.tsx new file mode 100644 index 0000000..1624a99 --- /dev/null +++ b/src/client/components/living/CaptureSourceCard.tsx @@ -0,0 +1,24 @@ +import type { Ref } from "react"; +import { Glyph, type GlyphName } from "../../ui/icons"; +import { Tooltip } from "./Tooltip"; +import type { HintKind } from "./hints"; + +/** One visual source card; its caller owns discovery and preview resources. */ +export function CaptureSourceCard({ title, action, icon, hint, preview, disabled, selected, + nativeKey, cameraKey, buttonRef, onPreview, onSelect }: { + title: string; action: string; icon: GlyphName; hint: HintKind; preview: string | null; + disabled?: boolean; selected?: boolean; nativeKey?: string; cameraKey?: string; + buttonRef?: Ref; onPreview?: () => void; onSelect: () => void; +}) { + return + + ; +} diff --git a/src/client/components/living/CaptureSourcePicker.tsx b/src/client/components/living/CaptureSourcePicker.tsx index ea133d7..be4e1c6 100644 --- a/src/client/components/living/CaptureSourcePicker.tsx +++ b/src/client/components/living/CaptureSourcePicker.tsx @@ -7,17 +7,21 @@ import { Glyph, type GlyphName } from "../../ui/icons"; import { Tooltip } from "./Tooltip"; import { HintComic } from "./hints"; import { Pill } from "./primitives"; -import { WaitingCaption } from "./WaitingStatus"; +import { CameraSources, useCameraSources } from "./CameraSources"; +import { CaptureSourceCard } from "./CaptureSourceCard"; +import type { loadCameraPreviews } from "../../media/camera-previews"; -const SOURCE_TABS = ["browser", "window", "display"] as const; +const SOURCE_TABS = ["browser", "camera", "window", "display"] as const; type SourceTab = (typeof SOURCE_TABS)[number]; const SOURCE_ICONS = { browser: "globe", window: "window", display: "display", + camera: "camera", } satisfies Record; export type NativeSourceList = + | { kind: "browser" } | { kind: "loading" } | { kind: "unavailable" } | { kind: "incompatible" } @@ -34,12 +38,17 @@ export type NativeSourceList = export function CaptureSourcePicker({ nativeSources, onBrowser, + onCamera, onNative, onPreview, onRefresh, onCancel, browserAvailable = true, + cameraAvailable = !!onCamera, initialTab = "window", + initialCamera = "", + activeCameraVideo, + loadCameras, initialAudio = true, initialShowCaptureBorder = false, audioLocked = false, @@ -47,6 +56,10 @@ export function CaptureSourcePicker({ }: { nativeSources: NativeSourceList; onBrowser: () => void; + onCamera?: (deviceId: string) => void; + initialCamera?: string; + activeCameraVideo?: HTMLVideoElement | null; + loadCameras?: typeof loadCameraPreviews; onNative: (target: NativeCaptureTarget, audio: boolean, showCaptureBorder: boolean) => void; onPreview: ( target: NativeCaptureTarget, @@ -55,6 +68,7 @@ export function CaptureSourcePicker({ onRefresh: () => void; onCancel: () => void; browserAvailable?: boolean; + cameraAvailable?: boolean; initialTab?: SourceTab; initialAudio?: boolean; initialShowCaptureBorder?: boolean; @@ -64,14 +78,24 @@ export function CaptureSourcePicker({ const { vis, t } = useCopy(); const pickerId = useId(); const panelRef = useRef(null); - const [tab, setTab] = useState(initialTab); + const [tab, setTab] = useState(null); const [shareAudio, setShareAudio] = useState(initialAudio); const [showCaptureBorder, setShowCaptureBorder] = useState(initialShowCaptureBorder); - const activeTab = tab === "browser" && !browserAvailable ? "window" : tab; - const supportsCaptureBorder = nativeSources.kind === "ready" && nativeSources.captureBorderControl === true; + const appDetected = nativeSources.kind === "ready" || nativeSources.kind === "failed" || + nativeSources.kind === "unsupported" || nativeSources.kind === "incompatible"; + const tabs = SOURCE_TABS.filter((value) => appDetected || value === "browser" || value === "camera"); + const preferredTab = tab ?? initialTab; + const tabAvailable = (value: SourceTab) => value === "camera" ? cameraAvailable + : value === "browser" || nativeSources.kind === "browser" ? browserAvailable : true; + const activeTab = tabs.includes(preferredTab) && tabAvailable(preferredTab) + ? preferredTab : tabs.find(tabAvailable) ?? "browser"; + const nativeTab = activeTab === "window" || activeTab === "display"; + const browserCapture = activeTab === "browser"; + const cameraSources = useCameraSources(activeTab === "camera", activeCameraVideo, loadCameras); + const supportsCaptureBorder = nativeTab && nativeSources.kind === "ready" && nativeSources.captureBorderControl === true; const issueKey = nativeSources.kind === "incompatible" ? "native.incompatible" - : activeTab !== "browser" && (nativeSources.kind === "unavailable" || - nativeSources.kind === "unsupported" || nativeSources.kind === "failed") + : nativeSources.kind === "unavailable" ? "host.sourcePicker.unavailable" : !nativeTab ? null + : (nativeSources.kind === "unsupported" || nativeSources.kind === "failed") ? `host.sourcePicker.${nativeSources.kind}` as const : null; @@ -87,7 +111,7 @@ export function CaptureSourcePicker({ }, [onCancel]); const sources = - nativeSources.kind === "ready" + nativeTab && nativeSources.kind === "ready" ? nativeSources.sources.filter( (target) => target.kind === activeTab || target.kind === "picker", ) @@ -98,7 +122,8 @@ export function CaptureSourcePicker({ ? nativeSources.processAudio : nativeSources.systemAudio); const anyNativeAudio = sources.some(supportsAudio); - const refreshLabel = t(nativeSources.kind === "loading" ? "host.sourcePicker.loading" : "host.sourcePicker.refresh"); + const refreshing = activeTab === "camera" ? cameraSources.busy : nativeSources.kind === "loading"; + const refreshLabel = t(refreshing ? "host.sourcePicker.loading" : "host.sourcePicker.refresh"); const audioAction = t(!anyNativeAudio ? "host.noAudio" : audioLocked ? "host.sourcePicker.audioLocked" : shareAudio ? "host.sourcePicker.audioOff" : "host.sourcePicker.audioOn"); @@ -112,10 +137,10 @@ export function CaptureSourcePicker({ type="button" className="lr-source-picker-refresh" aria-label={refreshLabel} - disabled={nativeSources.kind === "loading"} - onClick={onRefresh} + disabled={refreshing} + onClick={activeTab === "camera" ? cameraSources.refresh : onRefresh} > - + ); const closeButton = ( @@ -158,12 +183,12 @@ export function CaptureSourcePicker({ {t("host.sourcePicker.title")} )} - {refreshButton} - + : null} {closeButton} @@ -171,14 +196,14 @@ export function CaptureSourcePicker({
{ if (!["ArrowLeft", "ArrowRight", "Home", "End"].includes(event.key)) return; - const available = SOURCE_TABS.filter( - (value) => value !== "browser" || browserAvailable, - ); + const available = tabs.filter(tabAvailable); + if (!available.length) return; const index = available.indexOf(activeTab); const next = event.key === "Home" @@ -199,7 +224,7 @@ export function CaptureSourcePicker({ ?.focus(); }} > - {SOURCE_TABS.map((value) => { + {tabs.map((value) => { const button = ( ) : null} - {activeTab !== "browser" + {nativeTab && !browserCapture ? sources.map((target) => ( + } - {activeTab !== "browser" && + {nativeTab && nativeSources.kind === "ready" && (supportsCaptureBorder || sources.length > 0) ? (
@@ -329,17 +357,7 @@ export function CaptureSourcePicker({ {issueKey ? ( - ) : activeTab === "browser" ? null : nativeSources.kind === "loading" ? ( - - - {vis ? null : {t("host.sourcePicker.loading")}} - - - ) : sources.length === 0 ? ( + ) : browserCapture || activeTab === "camera" ? null : sources.length === 0 ? ( - - - ); + return ; } diff --git a/src/client/components/living/HostMicrophone.tsx b/src/client/components/living/HostMicrophone.tsx new file mode 100644 index 0000000..0748efe --- /dev/null +++ b/src/client/components/living/HostMicrophone.tsx @@ -0,0 +1,49 @@ +import { useId } from "react"; +import { Btn } from "./primitives"; +import { Tooltip } from "./Tooltip"; +import { Glyph } from "../../ui/icons"; +import { useCopy } from "../../ui/copy"; +import type { CaptureDevice } from "../../media/capture-devices"; +import { MicrophoneDeviceSelect } from "./MicrophoneDeviceSelect"; + +export function HostMicrophone({ enabled, pending, disabled, paused, unavailable, volume, onToggle }: { + enabled: boolean; pending?: boolean; disabled?: boolean; + paused?: boolean; + volume: number; + unavailable?: boolean; onToggle: () => void; +}) { + return ; +} + +export function HostMicrophoneSettings({ enabled, disabled, volume, onVolume, deviceId, onDevice, loadDevices, native }: { + enabled: boolean; disabled?: boolean; native?: boolean; + volume: number; onVolume: (volume: number) => void; + deviceId: string; onDevice: (id: string) => void; loadDevices: () => Promise; +}) { + const { t, vis } = useCopy(); + const id = useId(); + const percent = Math.round(volume * 100); + return
+ + + + onVolume(Number(event.target.value) / 100)} /> + +
; +} diff --git a/src/client/components/living/LauncherForm.tsx b/src/client/components/living/LauncherForm.tsx index 53e94af..d268587 100644 --- a/src/client/components/living/LauncherForm.tsx +++ b/src/client/components/living/LauncherForm.tsx @@ -1,4 +1,4 @@ -import type { FormEvent, ReactNode } from "react"; +import { useId, type FormEvent, type ReactNode } from "react"; import { BrandMark } from "./BrandMark"; import { Tooltip } from "./Tooltip"; import { WelcomeLine } from "./WelcomeLine"; @@ -7,6 +7,7 @@ import type { HintKind } from "./hints"; import { Glyph, type GlyphName } from "../../ui/icons"; import { useCopy, type CopyKey } from "../../ui/copy"; export type AppMode = "local" | "link" | "site"; +export const SUGGESTED_SITE = "https://demo.piik.tv"; const MODES: Array<{ mode: AppMode; @@ -65,6 +66,7 @@ export function LauncherForm({ children?: ReactNode; }) { const { vis, t } = useCopy(); + const accessHintId = useId(); const accessField = (
{invalid ? : null} @@ -129,7 +174,7 @@ export function ControlsPreview() {
06

{en ? "Choose a picture" : "挑一块画面。"}

-

{en ? "The actual source selector, with sample windows. Selection only updates this preview." : "正式的画面选择器，放入了几个示例窗口；选择只影响这张预览。"}

+

{en ? "The actual source selector, with sample windows and cameras. Selection only updates this preview." : "正式的画面选择器，放入了示例窗口和摄像头；选择只影响这张预览。"}

{(["ready", "loading", "unavailable", "incompatible", "unsupported", "failed", "empty"] as const).map((value, index) => { setSourceState(value); setSourceOpen(true); }}> {en ? ["Available", "Loading", "App unavailable", "Update needed", "Capture unavailable", "Read failed", "Empty list"][index] : ["正常", "读取中", "App 未连接", "需要更新", "无法采集", "读取失败", "空列表"][index]})} @@ -142,6 +187,8 @@ export function ControlsPreview() { : sourceState === "ready" ? { ...SOURCES, captureBorderControl: captureBorderAvailable } : { kind: sourceState }} initialShowCaptureBorder={showCaptureBorder} onBrowser={() => { setSourceOpen(false); notify(); }} + onCamera={() => { setSourceOpen(false); notify(); }} + loadCameras={previewCameras} onNative={(_target, _audio, showBorder) => { setShowCaptureBorder(showBorder); setSourceOpen(false); notify(); }} onPreview={previewSource} onRefresh={() => setSourceState("ready")} onCancel={() => setSourceOpen(false)} /> :
setSourceOpen(true)} />
} @@ -156,5 +203,14 @@ export function ControlsPreview() {

{en ? "Open the full playback preview" : "进入完整播放预览"} →

+
+
08

{t("client.launch.title")}

+
+ event.preventDefault()} /> +
+
; } diff --git a/src/client/pages/HostPage.tsx b/src/client/pages/HostPage.tsx index ae4140b..7417ed3 100644 --- a/src/client/pages/HostPage.tsx +++ b/src/client/pages/HostPage.tsx @@ -1,4 +1,5 @@ import { + useCallback, useEffect, useLayoutEffect, useMemo, @@ -21,9 +22,13 @@ import { type CodeEntryPolicy, type RoutePolicy, } from "../../shared/protocol"; +import { browserCaptureDevices } from "../media/capture-devices"; import { AppHeader, LedStrip } from "../components/living/Header"; import { WelcomeLine } from "../components/living/WelcomeLine"; import { Couch, type CouchEntry } from "../components/living/Couch"; +import { SharingSettings } from "../components/living/SharingSettings"; +import { HostMicrophone, HostMicrophoneSettings } from "../components/living/HostMicrophone"; +import { HostAudio } from "../media/host-audio"; import { CaptureSourcePicker, type NativeSourceList, @@ -110,7 +115,8 @@ import { import { labelParticipantSnapshot } from "../lib/viewer-presence"; import { applyCaptureProfile, - captureDisplay, + captureBrowserSource, + type BrowserCaptureSource, matchingQualityProfileId, QUALITY_PROFILES, DEGRADATION_PREFERENCE_KEYS, @@ -235,10 +241,10 @@ interface CaptureDetails { * mode never see a baked-in string from capture time. */ resolution: string | null; frameRate: number | null; - hasAudio: boolean; + hasSourceAudio: boolean; } -function captureDetails(stream: MediaStream, native = false): CaptureDetails { +function captureDetails(stream: MediaStream, native = false, sourceAudio?: boolean): CaptureDetails { // A Native preview is a received track, not an observation of raw capture. const settings = native ? undefined : stream.getVideoTracks()[0]?.getSettings(); return { @@ -247,7 +253,7 @@ function captureDetails(stream: MediaStream, native = false): CaptureDetails { ? `${settings.width}x${settings.height}` : null, frameRate: settings?.frameRate ?? null, - hasAudio: stream.getAudioTracks().length > 0, + hasSourceAudio: sourceAudio ?? stream.getAudioTracks().length > 0, }; } @@ -331,7 +337,7 @@ interface HostPageProps { } type ShareSourceSelection = - | { kind: "browser" } + | { kind: "browser"; source?: BrowserCaptureSource; deviceId?: string } | { kind: "native"; client: NativeClient; @@ -371,6 +377,16 @@ export function HostPage({ const [signalStatus, setSignalStatus] = useState("offline"); const [stream, setStream] = useState(null); + const hostAudioRef = useRef(null); + const [microphoneEnabled, setMicrophoneEnabled] = useState(false); + const [microphoneVolume, setMicrophoneVolume] = useState(1); + const [microphonePending, setMicrophonePending] = useState(false); + const [microphoneDevices, setMicrophoneDevices] = useState({ browser: "", native: "" }); + const [cameraDevice, setCameraDevice] = useState(""); + const loadMicrophones = useCallback(() => { + const client = nativeModeRef.current ? nativeClientRef.current : null; + return client ? client.microphones() : browserCaptureDevices("audioinput"); + }, []); const [nativeActive, setNativeActive] = useState(false); const [showCaptureBorder, setShowCaptureBorder] = useState(false); const [nativeSources, setNativeSources] = @@ -487,10 +503,13 @@ export function HostPage({ ): void { setNoticeKey(key, comic, tone, vars); } - const [copiedInviteUrl, setCopiedInviteUrl] = useState(null); - const copied = copiedInviteUrl !== null && copiedInviteUrl === room?.inviteUrl; + const [includeInviteCredential, setIncludeInviteCredential] = useState(true); + const roomLink = includeInviteCredential ? room?.inviteUrl : room?.canonicalUrl; + const roomLinkBlocked = !includeInviteCredential && room?.codeEntryPolicy === "private" && !viewerPasswordEnabled; + const [copiedRoomLink, setCopiedRoomLink] = useState(null); + const copied = copiedRoomLink !== null && copiedRoomLink === roomLink; const copiedResetTimerRef = useRef(null); - const copyInviteRequestRef = useRef(null); + const copyRoomLinkRequestRef = useRef(null); const [switchingSource, setSwitchingSource] = useState(false); const [changingQuality, setChangingQuality] = useState(false); const [sharingPaused, setSharingPaused] = useState(false); @@ -540,7 +559,7 @@ export function HostPage({ const activeRouteRevisionRef = useRef(0); const generationRef = useRef(0); const activeGenerationRef = useRef(null); - const sourceSwitchRef = useRef(null); + const sourceSwitchRef = useRef<{ replacingVideo?: MediaStreamTrack } | null>(null); const qualityChangeRef = useRef(null); const pendingQualityChangeRef = useRef(null); const qualitySettingsRef = useRef(DEFAULT_QUALITY_SETTINGS); @@ -563,6 +582,7 @@ export function HostPage({ const nativeEventCleanupRef = useRef<(() => void) | null>(null); const nativeClientCloseCleanupRef = useRef<(() => void) | null>(null); const nativeModeRef = useRef(false); + const nativeSourceAudioRef = useRef(undefined); const nativeSourceRequestRef = useRef(null); const nativePreviewTailRef = useRef>(Promise.resolve()); const nativeSourcePathRef = useRef(null); @@ -670,7 +690,7 @@ export function HostPage({ window.clearTimeout(copiedResetTimerRef.current); copiedResetTimerRef.current = null; } - copyInviteRequestRef.current = null; + copyRoomLinkRequestRef.current = null; viewerQualityEvidenceStore.clear(); cancelViewerQualityEvidenceRender(); activeRouteRevisionRef.current = 0; @@ -684,6 +704,8 @@ export function HostPage({ nativeClientConnectRef.current = null; nativeClientRef.current?.close(); nativeClientRef.current = null; + hostAudioRef.current?.dispose(); + hostAudioRef.current = null; streamRef.current?.getTracks().forEach((track) => track.stop()); retiringStreamRef.current?.getTracks().forEach((track) => track.stop()); streamRef.current = null; @@ -847,6 +869,10 @@ export function HostPage({ hostPeerIdRef.current = null; void hostSfuRouteRef.current?.disconnect(); hostSfuRouteRef.current = null; + hostAudioRef.current?.dispose(); + hostAudioRef.current = null; + setMicrophoneEnabled(false); + setMicrophonePending(false); streamRef.current?.getTracks().forEach((track) => track.stop()); retiringStreamRef.current?.getTracks().forEach((track) => track.stop()); streamRef.current = null; @@ -882,7 +908,7 @@ export function HostPage({ clearHostRoom(keepResumeHint); roomRef.current = null; setRoom(null); - setCopiedInviteUrl(null); + setCopiedRoomLink(null); setViewerPasswordDraft(creationProfileRef.current.roomPassword ?? ""); setViewerPasswordVisible(false); return true; @@ -970,7 +996,7 @@ export function HostPage({ writePreferredRoom(replacement.roomId); roomRef.current = replacement; setRoom(replacement); - setCopiedInviteUrl(null); + setCopiedRoomLink(null); setViewerPasswordEnabled(profile.roomPassword !== null); setViewerPasswordDraft(profile.roomPassword ?? ""); setViewerPasswordVisible(false); @@ -1010,7 +1036,7 @@ export function HostPage({ writePreferredRoom(replacement.roomId); roomRef.current = replacement; setRoom(replacement); - setCopiedInviteUrl(null); + setCopiedRoomLink(null); setViewerPasswordEnabled(profile.roomPassword !== null); setViewerPasswordDraft(profile.roomPassword ?? ""); setViewerPasswordVisible(false); @@ -1106,12 +1132,14 @@ export function HostPage({ captured: MediaStream, generation: number, ): void { - captured.getVideoTracks()[0]?.addEventListener( + const track = captured.getVideoTracks()[0]; + track?.addEventListener( "ended", () => { if ( isCurrentGeneration(generation) && - streamRef.current === captured + streamRef.current?.getVideoTracks()[0] === track && + sourceSwitchRef.current?.replacingVideo !== track ) { endSharing({ key: "host.stopNotice" }); } @@ -1128,11 +1156,31 @@ export function HostPage({ const { client, target, audio, showCaptureBorder, path } = selection; let bridge: NativeMediaBridge | null = null; let shareStarted = false; + let nativeEventCleanup: (() => void) | null = null; try { await nativeShareCleanupRef.current; await nativePreviewTailRef.current; if (!isCurrentShare(generation, shareGeneration)) return null; if (nativeClientRef.current !== client) throw new Error("Piik App is unavailable"); + nativeSourceAudioRef.current = undefined; + nativeEventCleanup = client.onEvent((event) => { + if (event.shareId !== shareGeneration || !isCurrentShare(generation, shareGeneration) || nativeClientRef.current !== client) return; + if (event.type === "audio-state") { + nativeSourceAudioRef.current = event.sourceAudio; + setMicrophoneEnabled(event.microphone); + setDetails(previous => previous ? { ...previous, hasSourceAudio: event.sourceAudio } : previous); + if (event.failed) setNoticeValue({ kind: "key", key: "host.microphone.unavailable", target: "operation", comic: "warning", tone: "warn" }); + } + if (event.type === "share-ended") { + endSharing( + { key: event.failed ? "host.shareEnded" : "host.stopNotice" }, + true, + event.failed ? "source-failed" : "share-ended", + event.failed ? "bad" : "off", + ); + } + }); + nativeEventCleanupRef.current = nativeEventCleanup; const started = await client.startShare({ shareId: shareGeneration, source: target, @@ -1150,6 +1198,7 @@ export function HostPage({ return null; } if (nativeClientRef.current !== client) throw new Error("Piik App is unavailable"); + nativeSourceAudioRef.current ??= started.sourceAudio ?? started.audio; videoCodecRef.current = manualVideoCodecPreference(started.codec); bridge = new NativeMediaBridge( shareGeneration, @@ -1171,22 +1220,6 @@ export function HostPage({ nativeMediaBridgeRef.current = bridge; nativeModeRef.current = true; setNativeActive(true); - const nativeEventCleanup = client.onEvent((event) => { - if ( - event.type === "share-ended" && - event.shareId === shareGeneration && - isCurrentShare(generation, shareGeneration) && - nativeClientRef.current === client - ) { - endSharing( - { key: event.failed ? "host.shareEnded" : "host.stopNotice" }, - true, - event.failed ? "source-failed" : "share-ended", - event.failed ? "bad" : "off", - ); - } - }); - nativeEventCleanupRef.current = nativeEventCleanup; const stream = await bridge.start(); if (!isCurrentShare(generation, shareGeneration)) { if (nativeMediaBridgeRef.current === bridge) disposeNativeShare(); @@ -1195,6 +1228,8 @@ export function HostPage({ } return stream; } catch (error) { + nativeEventCleanup?.(); + if (nativeEventCleanupRef.current === nativeEventCleanup) nativeEventCleanupRef.current = null; if ( nativeClientRef.current === client && nativeShareGenerationRef.current === shareGeneration @@ -1295,6 +1330,10 @@ export function HostPage({ const request = {}; nativeSourceRequestRef.current = request; nativeSourcePathRef.current = null; + if (!launchedByClient || (phase === "live" && !nativeModeRef.current)) { + setNativeSources({ kind: "browser" }); + return; + } setNativeSources({ kind: "loading" }); let client: NativeClient | null; @@ -1402,15 +1441,12 @@ export function HostPage({ function requestSharing(): void { setJoiningRoom(false); - if (!launchedByClient) { - void startSharing({ kind: "browser" }); - return; - } void openCaptureSourcePicker(); } - function startBrowserShareFromPicker(): void { - void startSharing({ kind: "browser" }); + function startBrowserShareFromPicker(source: BrowserCaptureSource, deviceId = ""): void { + if (phase === "live") void switchSource(source, deviceId); + else void startSharing({ kind: "browser", source, deviceId }); } async function loadNativeSourcePreview( @@ -1467,6 +1503,7 @@ export function HostPage({ nativeEventCleanupRef.current = null; nativeShareGenerationRef.current = null; nativeModeRef.current = false; + nativeSourceAudioRef.current = undefined; setNativeActive(false); if (!client || !shareGeneration) { releaseUnusedNativeClient(); @@ -1614,9 +1651,9 @@ export function HostPage({ debugEvent("quality", "committed", { generation, applied: appliedProfile }); outcome = "applied"; if (nativeUpdate) { - setDetails(captureDetails(activeStream, true)); + setDetails(captureDetails(activeStream, true, nativeSourceAudioRef.current)); } else if (captureChanged) { - setDetails(captureDetails(activeStream)); + setDetails(captureDetails(hostAudioRef.current?.sourceStream ?? activeStream)); } signalRef.current?.setHostQualitySettings(appliedProfile); const ingress = nativeMediaIngressRef.current; @@ -1787,6 +1824,9 @@ export function HostPage({ return; } sharingPausedRef.current = false; + // Before the mixer existed, source audio itself was the paused output. + // Restore it only on an accepted resume, never during permission handoff. + if (hostAudioRef.current) setMediaPaused(hostAudioRef.current.sourceStream, false); setSharingPaused(false); setNoticeValue(null); return; @@ -2395,7 +2435,7 @@ export function HostPage({ shareGenerationRef.current = shareGeneration; closeCaptureSourcePicker(); setNoticeValue(null); - setCopiedInviteUrl(null); + setCopiedRoomLink(null); setPhase("starting"); let captured: MediaStream | null = null; @@ -2410,7 +2450,7 @@ export function HostPage({ nativeStarted = true; } else { // This must remain the first awaited operation in the button gesture. - captured = await captureDisplay(qualitySettingsRef.current); + captured = await captureBrowserSource(qualitySettingsRef.current, selection.source ?? "browser", selection.deviceId); } } catch (error) { if (!isCurrentShare(generation, shareGeneration)) { @@ -2419,7 +2459,7 @@ export function HostPage({ } activeGenerationRef.current = null; shareGenerationRef.current = null; - setNoticeError(error, "capture", "television"); + setCaptureError(error, selection.kind === "browser" ? selection.source : undefined, "capture"); setPhase(isCapturePermissionFailure(error, "capture") ? "idle" : "error"); return; } @@ -2430,12 +2470,16 @@ export function HostPage({ return; } if (captured) { + if (selection.kind === "browser") { + hostAudioRef.current = new HostAudio(captured, setMicrophoneEnabled, selection.source ?? "browser"); + if (selection.source === "camera") setCameraDevice(selection.deviceId ?? ""); + } streamRef.current = captured; setStream(captured); watchCaptureEnd(captured, generation); if (selection.kind === "native") { setDetails( - captureDetails(captured, true), + captureDetails(captured, true, nativeSourceAudioRef.current), ); } else { setDetails(captureDetails(captured)); @@ -2718,7 +2762,7 @@ export function HostPage({ const activeStream = streamRef.current; if (activeStream) { setDetails( - captureDetails(activeStream, true), + captureDetails(activeStream, true, nativeSourceAudioRef.current), ); } const sfuUpdated = await hostSfuRouteRef.current?.updateProfile(qualitySettingsRef.current) ?? true; @@ -2745,7 +2789,7 @@ export function HostPage({ } } - async function switchSource(): Promise { + async function switchSource(source?: BrowserCaptureSource, deviceId = ""): Promise { const generation = activeGenerationRef.current; if ( phase !== "live" || @@ -2756,26 +2800,35 @@ export function HostPage({ ) { return; } - if (nativeModeRef.current) { + if (!source) { await openCaptureSourcePicker(); return; } + if (nativeModeRef.current) return; - const token = {}; + // Some browsers retire the old camera while opening the new one. That + // transition belongs to this replacement, not the share-ended observer. + const token = { replacingVideo: source === "camera" && hostAudioRef.current?.sourceKind === "camera" + ? streamRef.current?.getVideoTracks()[0] : undefined }; sourceSwitchRef.current = token; + closeCaptureSourcePicker(); setSwitchingSource(true); setNoticeValue(null); let captured: MediaStream; try { // Like initial capture, changing source must begin in this button gesture. - captured = await captureDisplay(qualitySettingsRef.current); + captured = await captureBrowserSource(qualitySettingsRef.current, source, deviceId); } catch (error) { if ( isCurrentGeneration(generation) && sourceSwitchRef.current === token ) { - setNoticeError(error, "source"); + if (token.replacingVideo?.readyState === "ended") { + endSharing({ key: "host.shareEnded" }, true, "source-failed", "bad"); + } else { + setCaptureError(error, source, "source"); + } } finishSourceSwitch(token); return; @@ -2789,24 +2842,90 @@ export function HostPage({ return; } + try { + if (source === "camera") setCameraDevice(deviceId); + captured = hostAudioRef.current?.attach(captured, source) ?? captured; + await replaceBrowserStream(captured, generation, token); + } catch (error) { + if (isCurrentGeneration(generation)) setNoticeError(error, "source"); + } finally { + finishSourceSwitch(token); + } + } + + function setCaptureError(error: unknown, source: BrowserCaptureSource | undefined, action: "source" | "capture") { + const target = action === "source" ? "operation" : "television"; + if (source !== "camera") { + setNoticeError(error, action, target); + return; + } + setNoticeValue({ kind: "key", key: error instanceof DOMException && error.name === "NotAllowedError" + ? "host.camera.denied" : "host.camera.unavailable", target, comic: "source-failed", tone: "warn" }); + } + + async function changeMicrophone(enabled: boolean, deviceId: string): Promise { + const audio = hostAudioRef.current; + const generation = activeGenerationRef.current; + const client = nativeModeRef.current ? nativeClientRef.current : null; + const shareId = nativeShareGenerationRef.current; + if ((!audio && !(client?.health.nativeMedia.microphone && shareId)) || generation === null || sourceSwitchRef.current || qualityChangeRef.current || sharingPausedRef.current) return; + const token = {}; + sourceSwitchRef.current = token; + setMicrophonePending(true); + setNoticeValue(null); + try { + if (client && shareId) { + await client.setMicrophone(shareId, enabled, microphoneVolume, deviceId); + if (isCurrentGeneration(generation) && nativeClientRef.current === client && sourceSwitchRef.current === token) { + setMicrophoneDevices(previous => ({ ...previous, native: deviceId })); + } + return; + } + if (!audio) return; + audio.setMicrophoneVolume(microphoneVolume); + const mixed = await audio.setMicrophone(enabled, deviceId); + if (!isCurrentGeneration(generation) || hostAudioRef.current !== audio) return; + setMicrophoneDevices(previous => ({ ...previous, browser: deviceId })); + if (mixed) await replaceBrowserStream(mixed, generation, token); + } catch (error) { + if (isCurrentGeneration(generation) && sourceSwitchRef.current === token) { + debugError("capture", "microphone-failed", error); + setNoticeValue({ kind: "key", key: error instanceof DOMException && error.name === "NotAllowedError" + ? "host.microphone.denied" : "host.microphone.unavailable", target: "operation", comic: "warning", tone: "warn" }); + } + } finally { + if (sourceSwitchRef.current === token) { + setMicrophonePending(false); + finishSourceSwitch(token); + } + } + } + + async function replaceBrowserStream(captured: MediaStream, generation: number, token: object): Promise { + // The caller retires its operation; this function owns only stream resources. const previousStream = streamRef.current; if (!previousStream) { captured.getTracks().forEach((track) => track.stop()); setNoticeKey("host.shareEnded", "share-ended", "off"); - finishSourceSwitch(token); return; } retiringStreamRef.current = previousStream; - invalidateSenderQualityEvidence(); - if (routePolicyRef.current.topologyOptimization) { + const videoChanged = captured.getVideoTracks()[0] !== previousStream.getVideoTracks()[0]; + // HostAudio owns raw inputs and mixed output. Adding a mixer must not stop + // the old stream's audio: that track is still feeding the new output. + const retirePrevious = () => previousStream.getVideoTracks().forEach((track) => { + if (!captured.getTracks().includes(track)) track.stop(); + }); + if (videoChanged) invalidateSenderQualityEvidence(); + if (videoChanged && routePolicyRef.current.topologyOptimization) { signalRef.current?.send({ type: "reset-sender-quality" }); } setMediaPaused(captured, sharingPausedRef.current); streamRef.current = captured; setStream(captured); - setDetails(captureDetails(captured)); - watchCaptureEnd(captured, generation); + setDetails(captureDetails(hostAudioRef.current?.sourceStream ?? captured)); + if (videoChanged) watchCaptureEnd(captured, generation); try { const ingress = nativeMediaIngressRef.current; @@ -2884,7 +3003,7 @@ export function HostPage({ activeSfuRoute && hostSfuRouteRef.current === activeSfuRoute ? syncHostSfuQualityWarning(activeSfuRoute, generation) : null; - previousStream.getTracks().forEach((track) => track.stop()); + retirePrevious(); if (retiringStreamRef.current === previousStream) { retiringStreamRef.current = null; } @@ -2912,7 +3031,7 @@ export function HostPage({ ); if ( - isCurrentGeneration(generation) && + videoChanged && isCurrentGeneration(generation) && sourceSwitchRef.current === token ) { const sourceNotice = @@ -2932,28 +3051,28 @@ export function HostPage({ ); } } finally { - previousStream.getTracks().forEach((track) => track.stop()); + retirePrevious(); if (retiringStreamRef.current === previousStream) { retiringStreamRef.current = null; } - finishSourceSwitch(token); } } - async function copyInvite(): Promise { + async function copyRoomLink(): Promise { const activeRoom = roomRef.current; - const inviteUrl = activeRoom?.inviteUrl; - if (!inviteUrl) { + const link = includeInviteCredential ? activeRoom?.inviteUrl : activeRoom?.canonicalUrl; + if (!activeRoom || !link || roomLinkBlocked || roomMutating) { return; } const request = {}; - copyInviteRequestRef.current = request; - const current = () => copyInviteRequestRef.current === request && - isCurrentRoomAuthority(activeRoom) && roomRef.current?.inviteUrl === inviteUrl; + copyRoomLinkRequestRef.current = request; + const current = () => copyRoomLinkRequestRef.current === request && + isCurrentRoomAuthority(activeRoom) && + (includeInviteCredential ? roomRef.current?.inviteUrl : roomRef.current?.canonicalUrl) === link; try { - await navigator.clipboard.writeText(inviteUrl); + await navigator.clipboard.writeText(link); if (!current()) return; - setCopiedInviteUrl(inviteUrl); + setCopiedRoomLink(link); setNoticeValue((current) => current?.kind === "key" && current.key === "host.invite.copyFailed" ? null : current); // One owner for the confirmation window: a second copy restarts it // instead of inheriting the first click's expiry. @@ -2962,11 +3081,11 @@ export function HostPage({ } copiedResetTimerRef.current = window.setTimeout(() => { copiedResetTimerRef.current = null; - setCopiedInviteUrl(null); + setCopiedRoomLink(null); }, 1_500); } catch { if (!current()) return; - setCopiedInviteUrl(null); + setCopiedRoomLink(null); setNoticeErrorKey("host.invite.copyFailed", "copy-failed"); } } @@ -3318,20 +3437,23 @@ export function HostPage({ {nativeSources ? ( startBrowserShareFromPicker("browser")} + onCamera={deviceId => startBrowserShareFromPicker("camera", deviceId)} + initialCamera={cameraDevice} + activeCameraVideo={hostAudioRef.current?.sourceKind === "camera" ? videoRef.current : null} onNative={startNativeShareFromPicker} onPreview={loadNativeSourcePreview} onRefresh={openCaptureSourcePicker} onCancel={closeCaptureSourcePicker} - browserAvailable={!nativeActive} + browserAvailable={!nativeActive && !!navigator.mediaDevices?.getDisplayMedia} + cameraAvailable={!nativeActive && !!navigator.mediaDevices?.getUserMedia} selectionDisabled={roomMutating || switchingSource || changingQuality} initialAudio={ nativeActive - ? (streamRef.current?.getAudioTracks().length ?? 0) > 0 + ? nativeSourceAudioRef.current ?? false : true } - audioLocked={nativeActive} + audioLocked={nativeActive && !nativeClientRef.current?.health.nativeMedia.microphone} initialShowCaptureBorder={showCaptureBorder} /> ) : !stream && @@ -3381,6 +3503,7 @@ export function HostPage({ style={{ gap: 12 }} onSubmit={joinRoomFromStage} > + {!vis && {t("join.hint", { site: window.location.host })}} { setJoinRoomCode(value); setJoinRejectedAttempt(0); }} /> @@ -3426,9 +3549,67 @@ export function HostPage({ /> ) : null} +
+ {phase === "live" ? <> + void changeMicrophone(!microphoneEnabled, microphoneDevices[nativeActive ? "native" : "browser"])} /> + + void switchSource()} + /> + : null} + setShowAdvanced((current) => !current)} + /> + {phase === "live" ? ( + endSharing({ key: "host.stopNotice" })} + /> + ) : phase === "starting" ? ( + endSharing({ key: "host.startCancelled" })} + /> + ) : null} +
- {!details?.hasAudio && stream ? ( - + {!details?.hasSourceAudio && stream ? ( + ) : null} {hostSfuWarningText ? ( @@ -3439,6 +3620,308 @@ export function HostPage({ motion={noticeValue.comic === "signal-recovering" || noticeValue.comic === "recovering" || noticeValue.comic === "connecting-sfu" ? "progress" : "still"} /> ) : null}
+ void changeQuality({ ...QUALITY_PROFILES[id], + screenAudioQuality: resolveScreenAudioQuality(advancedQualityRef.current.screenAudioQuality), + })} /> + } + picture={<> +
+ + + + +
+ {( + Object.keys(QUALITY_RESOLUTIONS) as QualityResolution[] + ).map((resolution) => ( + + changeAdvancedQuality({ resolution }) + } + > + {QUALITY_RESOLUTIONS[resolution].label} + + ))} +
+
+
+
+ + + + + + + + changeAdvancedQuality({ + maxFramerate: Number(event.target.value), + }) + } + /> + {advancedQuality.maxFramerate} fps + + +
+
+ + + + + + + + changeAdvancedQuality({ + maxBitrate: Number(event.target.value), + }) + } + /> + + {(advancedQuality.maxBitrate / 1_000_000).toFixed(1)}{" "} + Mbps + + + +
+
+
+ + + + +
+ {( + Object.keys( + PREFERENCE_PRESENTATION, + ) as DegradationPreference[] + ).map((preference) => ( + + changeAdvancedQuality({ + degradationPreference: preference, + }) + } + > + + + + ))} +
+
+ } + audio={<> + {phase === "live" && (!nativeActive || nativeClientRef.current?.health.nativeMedia.microphone) ? ( + { + setMicrophoneVolume(volume); + hostAudioRef.current?.setMicrophoneVolume(volume); + const client = nativeClientRef.current; + const shareId = nativeShareGenerationRef.current; + if (nativeModeRef.current && client && shareId) { + void client.setMicrophoneVolume(shareId, volume).catch(error => { + if (nativeClientRef.current === client && nativeShareGenerationRef.current === shareId) { + debugError("capture", "microphone-volume-failed", error); + setNoticeValue({ kind: "key", key: "host.microphone.unavailable", target: "operation", comic: "warning", tone: "warn" }); + } + }); + } + }} + native={nativeActive} loadDevices={loadMicrophones} + deviceId={microphoneDevices[nativeActive ? "native" : "browser"]} + onDevice={deviceId => void changeMicrophone(microphoneEnabled, deviceId)} + /> + ) : null} +
+ + + + +
+ {( + Object.keys( + AUDIO_QUALITY_CAPTIONS, + ) as ScreenAudioQuality[] + ).map((audioQuality) => ( + changeScreenAudioQuality(audioQuality)} + > + + + {SCREEN_AUDIO_BITRATES[audioQuality] / 1_000} + + + ))} +
+
+ } + technical={<> +
+ + + + +
+ + changeRoutePolicy({ topologyOptimization: checked }) + } + label={t("host.advanced.route.topo")} + note={t("host.advanced.route.topoHint")} + hint="hint-topology" + /> + + changeRoutePolicy({ natPrediction: checked }) + } + label={t("host.advanced.route.natPrediction")} + note={t( + natPredictionAvailable + ? "host.advanced.route.natPredictionHint" + : "host.advanced.route.natPredictionUnavailable", + )} + hint={ + natPredictionAvailable ? "hint-nat-prediction" : "hint-nat-unavailable" + } + /> + + changeRoutePolicy({ peerOnly: checked }) + } + label={t("host.advanced.route.peerOnly")} + note={t( + sfuAvailable + ? "host.advanced.route.peerOnlyHint" + : "host.advanced.route.peerOnlyRequired", + )} + hint={sfuAvailable ? "hint-route-p2p" : "hint-route-p2p-required"} + /> +
+
+
+ + + + +
+ {(["vp8", "auto", "h264"] as const).map((mode) => ( + changeVideoCodecMode(mode)} + > + {mode.toUpperCase()} + + ))} +
+
+ } + /> setShowTopology((current) => !current)} /> - {phase === "live" || phase === "starting" ? ( -
- {phase === "live" ? ( - <> - - void switchSource()} - /> - endSharing({ key: "host.stopNotice" })} - /> - - ) : ( - endSharing({ key: "host.startCancelled" })} - /> - )} -
- ) : null} + - {showConnectionDetails && details && (stream || nativeActive) ? ( - -
- - - {vis ? null : ( - {t("stats.capture")} - )} - -
- - - - - - -
-
- - ) : null} - {showConnectionDetails && viewerOverviewEntries.length > 0 ? ( - - setSelectedPawn((current) => - current === peerId ? null : peerId, - ) - } - /> - ) : null} - {selectedViewer && selectedDetail ? ( - setSelectedPawn(null)} - /> - ) : null} - {showTopology ? ( - - - setSelectedPawn((current) => - current === peerId ? null : peerId, - ) - } - /> - - ) : null} - {room ? ( void copyInvite()} + disabled={!roomLink || roomMutating || roomLinkBlocked} + onClick={() => void copyRoomLink()} /> void changeViewerGrant("revoke")} /> - {room.inviteUrl ? ( - - event.currentTarget.select()} - /> - - ) : null} +
+ {roomLink ? ( + + event.currentTarget.select()} + /> + + ) : null} + + + { + copyRoomLinkRequestRef.current = null; + setCopiedRoomLink(null); + setIncludeInviteCredential(checked); + }} /> + +
+ {roomLinkBlocked ? : null}