0 Fork 0

Files

T

AAA 95cc11f2c1 fix(media): restore capture recovery and forwarding cadence

Browser display capture could remain low-resolution after a temporary bandwidth reduction because sender adaptation also reduced the shared capture input. The encoded-output carrier could separately reduce frame rate after the real producer had already adapted, causing repeated queue recovery.

Keep display capture and sender feedback separate through bounded native frame streams, retain capture-owned constraints and metadata, and preserve carrier cadence. Existing WebRTC picture adaptation and transport ownership remain in place. Unsupported frame APIs retain ordinary sender tracks.

Real display acceptance also exposed an existing VP8 forwarding defect: a 15 fps L1T3 stream reached the SFU subscriber at about 4 fps because both shared forwarding owners overrode LiveKit's temporal limit with T0. Removing those overrides restores the library's layer selection for SFU and Native forwarding.

## Validation

- Web type checking, 1,186 tests and production build passed.
- Go tests, formatting, vet, Windows/Linux builds and Windows native capture checks passed; macOS packaging runs in the existing native CI job.
- H.264/VP8 real-tab first-frame, live/paused settings, source replacement, failure retirement and two-Viewer shaped-network checks passed.
- Real display through embedded SFU passed H.264 and VP8: hidden Host, two decoded spatial layers, resumed delivery, track replacement, profile updates, subscription recovery and full cleanup. VP8 decoded cadence recovered from about 4 to 15 fps. Track replacement uses the same visual source, and these checks do not measure A/V synchronization or physical mobile backgrounding.
- Shared RTP tests cover all three VP8 temporal layers through Transport and Publication; restoring the old T0 caps makes both tests fail.
- Documentation and repository hygiene checks passed.

## Release notes

### 中文

这是一次浏览器分享稳定性修复更新，重点改善网络波动后的画质恢复和编码复用时的画面流畅度。

- 修复部分 Chromium 浏览器屏幕共享降画质后，采集输入持续停留在低分辨率的问题。
- 修复编码复用路径重复降低帧率，导致画面卡顿和反复请求恢复帧的问题。
- 修复部分 VP8 画面经过媒体服务器或 App 转发时，帧率被错误限制在最低时间层的问题。
- 网页用户请在站点更新后刷新页面并重新开始分享；App 用户请更新并重启 App。

复杂画面的恢复速度和不同设备、网络下的短时清晰度波动仍在继续验证，本次更新不代表所有画质波动都已解决。

### English

This maintenance update improves browser sharing stability, focusing on picture recovery after network fluctuations and smoother delivery when encoded frames are reused.

- Fixed an issue in some Chromium browsers where display capture could remain at a low resolution after the sender reduced picture quality.
- Fixed repeated frame-rate reduction in the encoding-reuse path that could cause stuttering and repeated recovery-frame requests.
- Fixed an unintended frame-rate limit when VP8 video with temporal layers passed through the media server or Piik App's forwarding path.
- Web users should refresh the page and restart sharing after their site is updated. App users should update and restart Piik App.

Recovery with complex content and brief quality fluctuations across different devices and networks remain under investigation. This update does not claim to eliminate every quality fluctuation.

2026-09-30 20:15:52 +08:00

4.7 KiB

Raw Blame History

Windows Capture Process

This isolated Windows process provides native capture for Piik App. It enumerates local displays and visible top-level windows, binds each window to its PID and process creation time, and has independent modes for:

capability discovery (--probe);
process-tree or default-device loopback PCM (--capture-audio); or
WGC/D3D11 screen/window video with adapter-bound Media Foundation H.264 or libvpx VP8 output (--capture-video --codec auto|h264|vp8).
bounded 320x180 BMP source previews (--preview), delivered once per target over the local control connection rather than the media route.

The video command receives the current product resolution, frame-rate, bitrate, and quality preference. A replacement process applies live changes while the Go session retains its Pion source and connections.

Sharing and source previews request borderless access by default when Windows exposes IsBorderRequired and GraphicsCaptureAccess (build 20348+, including Windows 11). The probe advertises API availability; the source-picker preference passes --show-capture-border when the Host wants the sharing border visible. One capture-local owner applies completed access without waiting for consent, and cancels pending access on retirement. Unsupported APIs, denial or failure retain ordinary capture; another capture session may still require a border. The property value is a preference, not proof that the border is invisible. Consumer Windows 10 through build 19045 lacks this API; the current WGC capture path keeps its system border there.

Video and audio run as separate bounded child processes. A source whose audio loopback cannot be initialized keeps video available and reports audio unavailable instead of failing the whole source. Process loopback is probed by activation rather than inferred from a Windows build number; display sources use the standard render-device loopback available on Windows 10 and later.

Auto measures delivered frame cadence through the target-profile encoding pipeline within a four-second selection budget; H.264 that sustains the target needs no software comparison. The selected codec remains fixed across profile and source changes. VP8 reads the existing NV12 surface through one staging texture and uses the same encoded-frame boundary. The process has no network fallback. The App consumes the selected process or system-audio stream through its native media edge when the capability probe reports support. Build to the stable project build/go-check directory for a bounded capability run:

npm run check:native

The checks include synthetic output-worker replacement and failure cases using WARP and a test codec, plus preview cleanup when WGC rejects a hidden test window. They do not capture a user screen or validate a physical encoder.

The implementation follows Microsoft's MIT-licensed reference samples and official API contracts without copying their WIL framework. The retained MF fixture compiles the same encoder source with PIIK_H264_FIXTURE; there is not a second product MFT implementation. Shared local outputs use stock WebRTC source/encoder adaptation. H264 keeps native NV12 surfaces, while WebRTC VP8 reads back into I420. The existing capture and output mailboxes retain source/generation ownership; adapted sizes and timestamps cross the same media envelope. Hardware low-latency mode stays on; the old fixed one-frame VBV override is removed so later bitrate increases are not trapped by the original buffer limit.

The build caches the pinned WebRTC SDK and matching official MSVC linker/runtime libraries under build/encoder-pool. The SDK supplies libvpx, replacing the separate source build. Only statically linked code and required notices enter the package, not the SDK/toolchain or another runtime service.

4.7 KiB Raw Blame History

Windows Capture Process

4.7 KiB

Raw Blame History