AAA
a9b7507488
## Problem and outcome / 问题与结果 Add camera sharing and Host microphone commentary to the existing Host workflow, with shared source cards, device selection, input gain and grouped sharing settings. Browser and native capture keep one outgoing audio track and the existing room/media owners. Include optional password-free invitations, public-tunnel protocol fallback, capture retirement and App-discovery feedback fixes. This is the complete capture maintenance phase; interaction, x86 capture and 4K experiments are excluded. ## Release notes ### 中文 这次更新让分享多了两种表达方式：用摄像头展示现场，用麦克风边分享边讲解。 分享设置也重新整理，并修复了公网邀请、音视频切换和采集退出中的一些问题。 #### 新功能与使用改进 - **摄像头分享**：在原有选源器中选择摄像头，支持设备选择和画面预览；支持摄像头访问的手机浏览器也可使用。 - **房主麦克风**：浏览器分享和 App 原生采集均可混入房主声音，支持选择麦克风、静音和输入音量调节。观众仍使用原有音量控件收听。 - **更集中的分享设置**：画质、声音和详细连接选项集中在画面下方，常用操作保持直接可用，并调整了窄屏播控与等待提示的布局。 - **邀请链接可选择是否免密加入**：关闭后，链接不再附带免密邀请；接收者按当前站点和房间的准入设置加入。 - **更清晰的启动设置**：可选站点口令默认收起，已有站点地址提供 `demo.piik.tv` 建议，并澄清 NAT 增强选项的含义。 #### 修复 - 公网邀请在 UDP 受限但 TCP 可用的网络中，可使用隧道组件原有的协议回退，减少这类启动失败。 - 调整音频时保留健康的视频发送；修复旧采集事件干扰当前分享的情况。 - 停止或取消原生采集时，先正常释放采集会话，超时才强制退出；修复本地复现的强制终止后 Windows 黄框残留问题。 - 从 App 打开的页面无法连接 App 时，选源器保留明确提示，不再只剩浏览器和摄像头选项而缺少原因说明。 #### 使用说明 摄像头和音频能力取决于系统、浏览器与设备。Windows 仍可能因权限或其他采集要求显示边框。 使用新增的 App 麦克风功能，请更新 App；连接自部署站点时，也请更新站点以获得对应页面功能。 ### English This update adds camera sharing and Host microphone commentary, alongside a clearer sharing-settings layout. It also fixes several public-invitation, audio/video transition and capture-shutdown issues. #### Features and usability - **Camera sharing:** choose a camera in the existing source picker, with device selection and thumbnail previews. Phone browsers with camera access can share too. - **Host microphone:** add commentary to Browser sharing or App-native capture, with device selection, mute and input gain. Viewers listen through their existing volume controls. - **Grouped sharing settings:** picture quality, sound and detailed connection options sit below the picture, while frequent actions remain directly available. Narrow-screen playback and waiting layouts are refined. - **Optional password-free invitations:** turn this off to copy a link without an invitation credential. Recipients then follow the site's and room's current admission settings. - **Clearer startup settings:** optional site-password settings are collapsed, the existing-site field suggests `demo.piik.tv`, and the NAT enhancement option has a clearer description. #### Fixes - Public invitations can use the tunnel component's existing protocol fallback when UDP is blocked but TCP is available, reducing startup failures on those networks. - Audio changes preserve healthy video sending. Retired capture events no longer affect an active share. - Stopping or cancelling native capture releases the capture session before enforcing the shutdown deadline. This fixes a locally reproduced case where forced termination left a Windows capture border behind. - Pages opened from the App keep a clear connection warning in the source picker when the App cannot be reached, instead of showing only Browser and Camera without an explanation. #### Usage notes Camera and audio support depend on the OS, browser and device. Windows may still require a border because of permissions or another active capture. Update the App to use its new microphone feature; update self-hosted sites as well to receive the matching page controls. ## Verification / 验证 - npm run check: 74 files / 1054 tests passed, TypeScript and production Web build passed. - npm run check:go: Go tests, vet, target builds and Windows native compile/regressions passed. - npm run gate:app-native-host: actual Windows H264 Host/Viewer, live/paused profile changes, quiet-source recovery, source replacement/failure, subsequent share and cleanup passed. - Windows 11 capture-border check: parent cancellation retires the actual WGC session and clears its border; subprocess tests verify normal, concurrent and timed-out retirement. - Website build, scripts typecheck, documentation and repository hygiene passed. - Native macOS/Linux packaging and Linux race checks run in CI before publication. Physical phone camera and real audio-level/echo/multiple-device coverage remain recorded limits; the owner authorized release with those limits. No universal NAT/border fix is claimed. ## Impact / 影响 Compatible feature release **v1.6.0**, explicitly approved by the owner. Browser/Server v23, Native control v9 and capture v7 stay unchanged; new App microphone commands use capability negotiation. No new dependency, port, media route, room lifecycle or storage migration. Published Server cutover retains configuration and room storage and uses bounded health checks with the previous executable available for rollback. Source, tests and owned documentation ship together; exact artifact/deployment identities remain outside the repo.
7.6 KiB
Camera And Host Microphone
Status: integrated into the ordinary Host UI and accepted for release. Remaining device coverage belongs to TODO.
Scope And Ownership
The existing source picker offers Browser, Camera, Window and Screen in that order, defaulting to Window when an App is detected. Without an App, it shows only Browser and Camera, defaulting to Browser. Mobile browsers can choose Camera when screen capture is unavailable. There is one active picture; capture, replacement, pause and stop retain their existing Host session and route owners. Camera permission never includes an implicit microphone request. HTTPS or localhost is required for device capture.
The Host microphone mixes commentary with Browser source audio into one outgoing audio track through Web Audio. The existing P2P, relay and SFU senders borrow that stream. Microphone mute preserves the mixed track and source replacement preserves microphone intent. Denial or unplugging leaves healthy video alone; retiring the share closes audio resources and discards late permission results. The Host preview stays muted. Viewers receive the combined sound with their existing volume control and never request microphone permission.
App native screen/window capture mixes selected-device microphone PCM with source audio before the existing Opus encoder. Windows uses WASAPI, macOS uses AVAudioEngine and Linux uses PulseAudio/PipeWire through GStreamer. Platform converters supply 48 kHz stereo PCM; one 20 ms output clock consumes each input once, with a bounded queue and silence for missing samples. This bounds backlog; it does not replace platform resampling or establish long-term clock-drift and echo-cancellation performance.
New App/page pairs opt into a stable mixed audio track from share startup, even with source sound disabled. Opening or muting the microphone and replacing the source preserve that track, encoder, preview and route owners. Source sound and microphone availability remain separate reported facts. Old Apps/pages retain the existing capture contract through capability negotiation; microphone capture starts only after an explicit action. Permission waiting is cancellable and does not hold the control reader. Device loss retires only that audio input.
Windows native capture and the App-to-Viewer path have local acceptance evidence. macOS/Linux microphone compilation is checked by native-platform packaging. Real audio-level/echo checks, macOS/Linux device acceptance and actual phone capture remain physical evidence limits; synthetic tests do not establish those results. The owner authorized release with those limits retained. The Host preview remains muted on every path.
This scope adds no room messages, ports, chat, Viewer microphone or emoji interactions. Microphone, pause, source replacement and stop share one action row below the picture, outside the playback controls and television status. Their order and meaning stay the same in Chinese, English and pure-visual modes. Source-sound details describe the raw capture input; microphone intent and Viewer playback volume are separate facts. Pausing the share silences the mixed output and temporarily disables microphone toggling without clearing its intent. Sharing settings open from the action dock. Picture presets/parameters and sound form separate groups, with microphone device/input gain under sound and a further disclosure for connection/codec options. The panel expands in page flow, pushing the couch down; video presets remain available before capture starts. The dock keeps only its frequent mute action; camera selection stays with source selection. The default follows the system; an explicit selection never silently falls back to another device. Browser capture and native capture keep separate device IDs for the page lifetime. Enumeration does not request permission. Browser labels may appear only after the first grant; refresh and device-change events update the list. A missing selected device stays visible as unavailable. Switching an active microphone prepares the new input before retiring the old one; failure leaves the old input running. The mixed output stays unchanged. Camera selection belongs in the existing Camera tab and uses the same named thumbnail cards as window/display selection. Entering the Camera tab may request camera permission for local previews; it never requests microphone access or starts a room. Preview inputs open serially and stop after one bounded thumbnail. Leaving the tab or cancelling the picker stops owned inputs and discards late permission results. Images last only for that picker. While already sharing a camera, preview borrows the current video frame and lists other devices without opening them, because some phones retire an active camera when another opens. Missing thumbnails do not disable a known device. The Host's existing source-replacement operation owns the old camera's automatic end event until the pending capture settles; failure with an already-ended old camera retires sharing normally. This does not suppress a user's screen-share Stop action. Some phones require releasing the old camera before opening another; seamless switching is not established.
The input-volume control adjusts microphone input from 0–200% through smoothed input gain. It does not change source audio or Viewer playback volume, reopen permission, or replace the outgoing track. Its value lasts for the Host page lifetime; new page loads use 100%. Mute remains a separate direct action. Volume settings use the existing settings disclosure without covering the picture or participants. The microphone-volume comic describes outgoing commentary, separately from the Viewer's local listening volume.
Local Review
Use the ordinary development or embedded Server entry described in the contributor guide. Start a share from the normal Host page, select a source, then join through its ordinary invitation in another browser tab. Use headphones for same-device microphone tests. Browser-emulated mobile layout and synthetic devices do not establish actual iOS/Android permission, orientation, background or audio behavior.
Primary Technical References
-
Google Meet audio controls keeps microphone settings next to the frequent mute action.
-
OBS audio mixer separates source volume, mute and monitoring; Piik uses only the controls needed for commentary.
-
Web Audio stream destination produces a single audio track from the audio graph.
-
Media Capture and Streams owns microphone permissions, device lifetime and echo-cancellation constraints.
-
Camera capture and screen capture have separate permissions, capabilities and secure-context requirements.
-
WebRTC track replacement describes reuse of negotiated media slots and when renegotiation is required.
-
WASAPI shared capture, AVAudioConverter resampling and GStreamer audio mixing define the platform conversion and PCM mixing boundaries.