fix(media): restore capture recovery and forwarding cadence
Browser display capture could remain low-resolution after a temporary bandwidth reduction because sender adaptation also reduced the shared capture input. The encoded-output carrier could separately reduce frame rate after the real producer had already adapted, causing repeated queue recovery. Keep display capture and sender feedback separate through bounded native frame streams, retain capture-owned constraints and metadata, and preserve carrier cadence. Existing WebRTC picture adaptation and transport ownership remain in place. Unsupported frame APIs retain ordinary sender tracks. Real display acceptance also exposed an existing VP8 forwarding defect: a 15 fps L1T3 stream reached the SFU subscriber at about 4 fps because both shared forwarding owners overrode LiveKit's temporal limit with T0. Removing those overrides restores the library's layer selection for SFU and Native forwarding. ## Validation - Web type checking, 1,186 tests and production build passed. - Go tests, formatting, vet, Windows/Linux builds and Windows native capture checks passed; macOS packaging runs in the existing native CI job. - H.264/VP8 real-tab first-frame, live/paused settings, source replacement, failure retirement and two-Viewer shaped-network checks passed. - Real display through embedded SFU passed H.264 and VP8: hidden Host, two decoded spatial layers, resumed delivery, track replacement, profile updates, subscription recovery and full cleanup. VP8 decoded cadence recovered from about 4 to 15 fps. Track replacement uses the same visual source, and these checks do not measure A/V synchronization or physical mobile backgrounding. - Shared RTP tests cover all three VP8 temporal layers through Transport and Publication; restoring the old T0 caps makes both tests fail. - Documentation and repository hygiene checks passed. ## Release notes ### 中文 这是一次浏览器分享稳定性修复更新，重点改善网络波动后的画质恢复和编码复用时的画面流畅度。 - 修复部分 Chromium 浏览器屏幕共享降画质后，采集输入持续停留在低分辨率的问题。 - 修复编码复用路径重复降低帧率，导致画面卡顿和反复请求恢复帧的问题。 - 修复部分 VP8 画面经过媒体服务器或 App 转发时，帧率被错误限制在最低时间层的问题。 - 网页用户请在站点更新后刷新页面并重新开始分享；App 用户请更新并重启 App。 复杂画面的恢复速度和不同设备、网络下的短时清晰度波动仍在继续验证，本次更新不代表所有画质波动都已解决。 ### English This maintenance update improves browser sharing stability, focusing on picture recovery after network fluctuations and smoother delivery when encoded frames are reused. - Fixed an issue in some Chromium browsers where display capture could remain at a low resolution after the sender reduced picture quality. - Fixed repeated frame-rate reduction in the encoding-reuse path that could cause stuttering and repeated recovery-frame requests. - Fixed an unintended frame-rate limit when VP8 video with temporal layers passed through the media server or Piik App's forwarding path. - Web users should refresh the page and restart sharing after their site is updated. App users should update and restart Piik App. Recovery with complex content and brief quality fluctuations across different devices and networks remain under investigation. This update does not claim to eliminate every quality fluctuation.
This commit is contained in:
23 files changed
+1006
-133
No files matched your search
@@ -1,8 +1,8 @@
|
||||
# Browser Node-Local Encoding Pool
|
||||
|
||||
Reviewed and executed 2026-09-09 on Windows, Chrome 152.0.7977.82.
|
||||
Initial baseline: 2026-09-09 on Windows, Chrome 152.0.7977.82.
|
||||
[ADR-0014](../adr/0014-browser-node-local-encoding-pool.md) owns the selected
|
||||
design. The observations below belong to that measured baseline;
|
||||
design. Dated follow-ups identify their own Browser and workload boundaries;
|
||||
[status](../status.md) owns current adoption and release state, and
|
||||
[verification status](../verification-status.md) owns remaining physical limits.
|
||||
|
||||
@@ -263,6 +263,206 @@ models `requestFrame()` as a pending request, not a counted queue of clock ticks
|
||||
Do not infer one delivered carrier frame per call or enlarge the encoded queue
|
||||
to hide lost timing.
|
||||
|
||||
### Sustained H264 Recovery
|
||||
|
||||
A v1.6.8 field observation on 2026-09-29 confirmed minutes of 180p / low FPS
|
||||
under 1080p30, balanced and a 5 Mbps ceiling. Server summaries and receiver
|
||||
RTCStats agree. A new Viewer briefly restored the incumbent to 1080p without
|
||||
changing its parent; reconnecting the observer did not restore quality. An SFU
|
||||
candidate initially decoded well, then stalled; replay through the production
|
||||
quality probe correctly withheld approval. These observations establish the
|
||||
symptom, not its network or encoder cause. Host producer/carrier histories were
|
||||
unavailable; server summaries cannot reconstruct those local owners.
|
||||
|
||||
Serial comparisons on 2026-09-29/30 used synthetic motion, played audio,
|
||||
1080p30 / balanced / 5 Mbps and a test-only UDP bridge. They distinguish
|
||||
Canvas input from real `getDisplayMedia` capture. Each case is bounded evidence,
|
||||
not a public-network reproduction or a population result.
|
||||
|
||||
| Input / Browser | Constraint and recovery result |
|
||||
| --- | --- |
|
||||
| Canvas bars / Edge 154.0.4258.37 | Ordinary and pooled senders recovered from 360p to 1080p after 400 kbps, 4% forward loss and 200 ms RTT; about 21 / 17 seconds after rate/loss release |
|
||||
| Textured Canvas / Chrome 153.0.8010.53 and 154.0.8037.58 | Ordinary and pooled sending could retain 540p after allocation recovered; the longer Chrome 154 pair remained there after 145 seconds |
|
||||
| Browser tab / Chrome 154.0.8037.58 | Ordinary and bare WebRTC retained 148p / about 10 fps after 65 recovery seconds; the bare sender's outgoing estimate was 5.6 Mbps and video target 5 Mbps |
|
||||
| Chrome's own window / Chrome 154.0.8037.58 | Bare and pooled sending retained 180p / about 10 fps after 65 recovery seconds, with roughly 4.5–4.7 Mbps video targets |
|
||||
| Non-Browser Windows test window / Chrome 154.0.8037.58 | All 2,758 observed source frames remained 1866x1080; bare sending recovered from 180p through 270/360p to 540p during the 65-second window |
|
||||
|
||||
The display cases used a 45-second 400 kbps / 4% loss interval followed by
|
||||
removing both constraints; 300 ms configured RTT remained. The non-Browser
|
||||
fixture captured about 21 fps and does not establish 30 fps performance. Bare
|
||||
WebRTC used one cloned track, codec selection and initial sender parameters,
|
||||
without Piik's pool, startup guard, quality controller or repeated parameter
|
||||
writes. It also reproduced the tab restriction with VP8. Keeping the original
|
||||
preview playing and using `resizeMode:none` did not remove it.
|
||||
|
||||
**Short disturbances can enter the persistent state.** Two one-second rate/loss
|
||||
pulses, each followed by 40 unconstrained seconds, took the bare tab sender from
|
||||
778p to 418p and then 238p; the actual Piik pool comparison ended at 148p.
|
||||
Targets recovered to about 5 Mbps and actual captured frames remained reduced.
|
||||
These single runs establish reachability in both paths, not that the pool is
|
||||
worse or that every brief blur has this cause. The fixture retained 300 ms RTT.
|
||||
|
||||
A standalone HTML reproduction removed Piik, audio and the network shaper
|
||||
entirely. It connected two local PeerConnections, captured its own moving tab,
|
||||
selected H264 / balanced, reduced only `maxBitrate` from 5 Mbps to 400 kbps for
|
||||
45 seconds, then restored it for 65 seconds. Source frames stayed 270x148 and
|
||||
encoded output stayed at 148p / 10 fps, despite a 5 Mbps target, an approximately
|
||||
13 Mbps outgoing estimate and 0–1 ms measured RTT. A preceding latency-only comparison
|
||||
retained 238p. Network loss is therefore not necessary for this local lock;
|
||||
these are synthetic Chrome 154 results, not the field room's diagnosed cause.
|
||||
|
||||
**Capture lock and encoder hysteresis remain distinct.** Canvas recovery could follow content
|
||||
complexity: changing to simple bars restored 1080p, retained after restoring
|
||||
the textured scene. WebRTC's [quality scaler](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/modules/video_coding/utility/quality_scaler.cc)
|
||||
uses QP thresholds and sampling history as well as bandwidth-related adaptation.
|
||||
Its default two-second checks and longer sampling after down-adaptation can
|
||||
outlast a short disturbance. Native Chrome 154 logging also confirmed a stable
|
||||
QP deadband with the NVIDIA H264 MFT: after one 1048p-to-786p reduction, smoothed
|
||||
QP stayed between the recorded 24/37 thresholds for the remaining 90-second
|
||||
observation. Capture remained full size, but no low-QP upscale was requested.
|
||||
This is not merely old samples taking time to expire; restored bandwidth alone
|
||||
does not require this scaler to restore resolution.
|
||||
|
||||
A matched three-arm comparison used fixed 5 Mbps, alternating 5.0/4.9 Mbps
|
||||
every 500 ms, and repeated identical 5 Mbps writes. Native reconfiguration
|
||||
counts were 2/122/2; actual encoded rates were 4.914/4.114/4.907 Mbps. Changed
|
||||
budgets repeatedly returned the encoder target to the
|
||||
[bitrate adjuster's conservative startup state](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/video/encoder_bitrate_adjuster.cc),
|
||||
whereas identical writes did not. All three stayed at 786p and about 29 fps,
|
||||
without new keyframes during the experiment. These single-device runs show a
|
||||
framework reconfiguration cost, not additional resolution oscillation or a
|
||||
perceptual blur measurement. They do not justify delaying real demand, bitrate
|
||||
compensation or periodic resets; equal product updates are already deduplicated.
|
||||
The field's repeated brief blur still lacks matching Host histories.
|
||||
|
||||
The tab case exposed a more specific capture-feedback problem. A Chromium trace
|
||||
kept the source geometry at 1904x1049 while capture supplied a 268x148 content
|
||||
rectangle. Feedback rose to 66,600 pixels, but capture did not grow.
|
||||
[Chrome 154's source adapter](https://raw.githubusercontent.com/chromium/chromium/154.0.8037.58/third_party/blink/renderer/platform/peerconnection/webrtc_video_track_source.cc)
|
||||
puts the adapter's target pixels into a maximum-pixel feedback field. Its
|
||||
[capture oracle](https://raw.githubusercontent.com/chromium/chromium/154.0.8037.58/media/capture/content/video_capture_oracle.cc)
|
||||
then chooses a size below that limit using
|
||||
[90-row capture steps](https://raw.githubusercontent.com/chromium/chromium/154.0.8037.58/media/capture/content/capture_resolution_chooser.cc).
|
||||
For this geometry the next step above 270x149 (40,230 pixels) is 433x239
|
||||
(103,487 pixels), so 66,600 cannot advance it.
|
||||
|
||||
This matches WebRTC's [up-adaptation distinction](https://webrtc.googlesource.com/src/+/3a8b0c76e6b7bf0e72141047f82eb2c3f4862c8d/call/adaptation/video_stream_adapter.cc)
|
||||
between a soft target and a larger hard maximum intended to accommodate source
|
||||
sizes. The inspected libwebrtc revision is the exact revision pinned by
|
||||
[Chrome 154's DEPS](https://raw.githubusercontent.com/chromium/chromium/154.0.8037.58/DEPS),
|
||||
retrieved through Gerrit revision content. At 270x148, the soft recovery target
|
||||
is 66,600 pixels; its intended hard maximum is 159,840. Forwarding only the soft
|
||||
target prevents the next capture step. Without larger input, the next recovery
|
||||
request cannot exceed the already installed maximum and is rejected. This
|
||||
strongly supports a feedback/size-quantization lock. Actual `VideoFrame`
|
||||
dimensions confirm reduced capture independently of stale `getSettings()`.
|
||||
Reapplying exact dimensions to the original track did not unblock it.
|
||||
|
||||
The Piik sender boundary now forwards bounded raw display frames using
|
||||
[Chromium's track streams](https://developer.chrome.com/docs/capabilities/web-apis/mediastreamtrack-insertable-media-processing),
|
||||
without resizing, re-encoding or generated cadence. Camera/received tracks and
|
||||
unsupported Browsers keep ordinary clones. In the textured-tab comparison,
|
||||
all but the initial source frame stayed 1904x1048; output escaped 192p through
|
||||
262/390p to 524p. It still remained 524p after 125 recovery seconds. This removes
|
||||
the reproduced capture lock, not content-dependent encoder hysteresis or every
|
||||
reported blur. The separate non-Browser window case also retained full input and
|
||||
recovered gradually; do not generalize the tab mechanism to all desktop capture.
|
||||
|
||||
Static-source testing caught a necessary lifecycle detail: unlike an RTC sink,
|
||||
a Processor does not request idle refresh frames. A static tab delivered no first
|
||||
encoded frame without them. The input retains the RTC sink's
|
||||
[1 fps refresh requirement](https://raw.githubusercontent.com/chromium/chromium/154.0.8037.58/third_party/blink/renderer/modules/peerconnection/media_stream_video_webrtc_sink.h)
|
||||
through the native
|
||||
[capture constraint handler](https://raw.githubusercontent.com/chromium/chromium/154.0.8037.58/third_party/blink/renderer/modules/mediastream/media_stream_video_track.cc).
|
||||
No frame is synthesized by Piik and no sender quality floor is added.
|
||||
H264 pooling and ordinary VP8 passed static first-frame, live/quiet/paused profile
|
||||
updates, resume, source replacement and retirement with actual tab capture.
|
||||
Fault checks cover pending replacement and active-input failure without retiring
|
||||
the shared source. A fake-device input exercising the same raw-frame path retained
|
||||
20 fps during a verified hidden-page interval; self-tab capture could not enter
|
||||
hidden state in that fixture. Physical background/device coverage remains open.
|
||||
A matched full-resolution pair consumed 12.30 versus 12.46 whole-fixture CPU-seconds
|
||||
over 15 seconds. This single-machine sample is not a low-end performance guarantee.
|
||||
|
||||
**Carrier cadence must not adapt the picture twice.** Two one-second rate/loss
|
||||
pulses exposed a separate starvation loop: after the second pulse the producer
|
||||
delivered about 15 fps, but its synthetic carrier fell to 1-3 fps. The four-frame
|
||||
queue overflowed and repeatedly requested recovery keys. Keeping only the
|
||||
carrier's native preference at `maintain-framerate` retained about 15 decoded fps;
|
||||
the last 31 seconds had four queue-overflow recoveries instead of 69. The real
|
||||
producer remained balanced and source frames remained full size in both runs.
|
||||
This implements the existing producer/clock ownership boundary; it adds no
|
||||
queue growth, quality floor or network estimator. It does not remove ordinary
|
||||
WebRTC adaptation: output still ended at 390p in that textured two-pulse case.
|
||||
|
||||
The bounded two-Viewer tab probe passed with both H264 and VP8: after one Viewer
|
||||
was limited to 400 kbps for 14 seconds, it returned to 1904x1048 at about 30 fps
|
||||
within the 40-second recovery observation. The healthy sibling retained full size,
|
||||
and source-frame observations confirmed no capture shrink. This simpler scene
|
||||
does not supersede the textured-scene recovery boundary above.
|
||||
|
||||
Actual tab capture through embedded SFU exposed a separate forwarding defect:
|
||||
VP8's two spatial encodings each sent 15 fps with `L1T3`, but the subscriber
|
||||
decoded about 4 fps. Both shared forwarding owners forced temporal layer zero,
|
||||
discarding enhancement frames even with ample budget. The ordinary-clone control
|
||||
had the same failure. Removing those overrides retains
|
||||
[LiveKit's supported temporal bound](https://github.com/livekit/livekit/blob/v1.13.6/pkg/sfu/forwarder.go#L293-L295)
|
||||
and allocation from real incoming layers. The repaired real-tab check decoded
|
||||
15 fps with the Host hidden, including original/half-size switching, resumed
|
||||
delivery, track replacement and retirement. RTP regression tests cover both
|
||||
Transport and Publication; restoring either old cap makes its test fail.
|
||||
This does not identify the field room's H264 degradation cause.
|
||||
|
||||
**Upstream repair boundary.** Chromium's
|
||||
[feedback forwarding change](https://chromium-review.googlesource.com/c/chromium/src/+/2386743)
|
||||
dates to 2020; the relevant logic remained in inspected upstream main on
|
||||
2026-09-30. Reviewed historical fixes for
|
||||
[scale-boundary equality](https://codereview.webrtc.org/2713683002),
|
||||
[preference-switch waits](https://webrtc-review.googlesource.com/c/src/+/174805)
|
||||
and [requested-resolution reconfiguration](https://webrtc-review.googlesource.com/c/src/+/360200)
|
||||
do not supply a matching unshipped fix for this reproduction. Preserving the
|
||||
hard maximum separately from the soft target lets a numerical reduction escape
|
||||
the observed 148p, 180p and 238p fixed points. This is a narrow upstream repair
|
||||
candidate, not a compiled or accepted Chromium patch; full capture/adapter
|
||||
tests are still required. Updating a Piik package does not replace the user's
|
||||
Browser implementation.
|
||||
|
||||
**The existing preference control can recover this reproduction.** After 65
|
||||
seconds stuck at 148p / 10 fps despite restored allocation, changing from
|
||||
balanced to `maintain-resolution` restored 1048p within the next two-second
|
||||
sample and reached about 30 fps. Switching back to balanced after 30 seconds
|
||||
retained that output for another 30 seconds. Bare WebRTC changed only the
|
||||
sender's degradation preference; a separate Piik run used `HostPeer`'s existing
|
||||
profile update through the pool. Both kept their connection and source; the
|
||||
pool retained one producer. Actual source frames recovered too. This provides
|
||||
a locally verified manual recovery through the existing clarity preference,
|
||||
not a field guarantee or justification for automatic preference toggling.
|
||||
The pinned `VideoStreamAdapter` clears adaptation restrictions when switching
|
||||
to or from balanced, explaining why this control escapes the reproduced state.
|
||||
|
||||
**A new Viewer can help, but need not.** In a Canvas comparison, a degraded
|
||||
incumbent adopted a newcomer's fresh healthy producer and returned to 1080p in
|
||||
about 1.5 seconds on the same connection. In the tab comparison, the newcomer
|
||||
reused the existing producer and both stayed at 148p. Neither result establishes
|
||||
which local producer transition occurred in the field room.
|
||||
|
||||
**Regression and repair boundary.** v1.6.5 (`04cbb2c9`)'s pool, `HostPeer` and
|
||||
quality owners showed the same Canvas recovery patterns in the current fixture.
|
||||
This compares those owners, not the whole older release or Browser. The
|
||||
[sender-owned clone and five-frame protections](./realtime-quality-adaptation.md)
|
||||
remain implemented; they address connection generations and startup, not every
|
||||
mid-share adaptation state. Field server evidence contains same-parent quality
|
||||
attempts that failed proof. There is no confirmed removed protection or recent
|
||||
Piik regression. The capture and carrier repairs above address their reproduced
|
||||
defects, not every content-dependent adaptation state.
|
||||
Do not weaken candidate proof, impose output floors or reset encoders periodically.
|
||||
|
||||
Use actual source frames, producer/carrier histories and receiver RTCStats
|
||||
together; settings and configured ceilings are not frame evidence. Some headless
|
||||
runs lost presentation callbacks while decoding continued. Raw traces and fixture
|
||||
variants remain ignored. Matched field Host evidence is still needed to separate
|
||||
capture feedback, encoder adaptation and a continuing transport limitation.
|
||||
|
||||
## Cost And Accounting
|
||||
|
||||
Before the CPU-carrier refinement, a matched actual-product VP8 1080p30 comparison's ordinary pair made
|
||||
@@ -300,6 +500,9 @@ npx tsx scripts/browser-local-pool-probe.ts carrier --1080 --av --network --auto
|
||||
npx tsx scripts/browser-local-pool-probe.ts carrier --1080 --av --lifecycle
|
||||
npx tsx scripts/browser-local-pool-probe.ts carrier --1080 --av --relay
|
||||
npx tsx scripts/browser-local-pool-probe.ts carrier --1080 --background
|
||||
npx tsx scripts/browser-local-pool-probe.ts carrier --display --quiet-start --1080 --h264 --av --lifecycle
|
||||
npx tsx scripts/browser-local-pool-probe.ts ordinary --display --quiet-start --1080 --av --lifecycle
|
||||
npx tsx scripts/browser-local-pool-probe.ts carrier --display --1080 --h264 --av --network --auto --rate=400000 --preference=balanced
|
||||
```
|
||||
|
||||
Add `--h264`, `--single` or `--late` for the corresponding case. Set
|
||||
@@ -310,6 +513,19 @@ recovery observation; product code owns all adaptation. Results go to ignored
|
||||
`build/browser-local-pool`; summarize with
|
||||
`node scripts/browser-local-pool-summary.mjs <result.json>`.
|
||||
A successful process exit does not establish visual/performance parity.
|
||||
`--display` captures only the isolated probe tab. It records actual source-frame
|
||||
sizes and checks that sender adaptation does not shrink capture; its receiver
|
||||
observations establish frame progress, not barcode-based glass-to-glass latency
|
||||
or audiovisual offset. The summary suppresses those unmeasured timing values.
|
||||
`--quiet-start` requires a decoded frame before the source animation starts.
|
||||
|
||||
For real tab capture through the shared forwarding path, set `CHROME_PATH`,
|
||||
`PIIK_EMBEDDED_SFU_GATE=true`, `PIIK_EMBEDDED_SFU_DISPLAY=true` and
|
||||
`PIIK_EMBEDDED_SFU_CODEC=h264` or `vp8`, then run `npm run gate:embedded-sfu`.
|
||||
This separate arm verifies a hidden Host, decoded original/half-size layers,
|
||||
resumed delivery, video-track replacement, profile changes and retirement.
|
||||
Track replacement uses a clone of the same visual source; it does not establish
|
||||
different-device switching or audio/video synchronization.
|
||||
|
||||
## Primary References And Remaining Boundaries
|
||||
|
||||
|
||||
@@ -1,12 +1,17 @@
|
||||
# Native App Media Evidence
|
||||
|
||||
- Reviewed: 2026-09-05
|
||||
- Scope: platform capture, one shared H.264/Opus source, Pion transport, Browser
|
||||
decode, and Browser-mediated SFU fallback
|
||||
- Reviewed: 2026-09-29
|
||||
- Scope: platform capture, shared encoded sources, Pion transport and Browser
|
||||
decode; current behavior belongs to [media quality](../standards/media-quality.md)
|
||||
- Status: Windows physical native Host and Viewer gates passed; macOS and Linux adapters
|
||||
compile and package but still require physical media gates
|
||||
|
||||
## Result
|
||||
## Initial Physical Evidence (2026-09-05)
|
||||
|
||||
These checks established the original native media paths. The Browser-mediated
|
||||
SFU and loopback v8 observations below are historical. [ADR-0013](../adr/0013-embedded-node-local-media.md)
|
||||
owns the SFU replacement; [versioning](../standards/versioning.md) owns the current
|
||||
wire contract.
|
||||
|
||||
Piik App can own one process-isolated Windows capture and hardware H.264
|
||||
encoder, feed its Annex-B access units into one Pion source, and deliver that
|
||||
@@ -74,12 +79,11 @@ requires Go 1.25; App CI uses Go 1.26.6.
|
||||
|
||||
## Current Boundary
|
||||
|
||||
Windows now supports hardware H264 and software VP8 behind the same capture,
|
||||
source and Pion transport boundary. libvpx 1.17.0 accepts the existing NV12 data
|
||||
with row pitch preserved, without another color-conversion library. In a local
|
||||
CPU-only 720p30 trial its mean encoding work was 6.112 ms per frame; this excludes
|
||||
GPU readback and is not game-quality evidence. Separate real Browser gates
|
||||
proved manual VP8 and Auto-selected H264, including live presets, pause/resume,
|
||||
Windows supports hardware H264 and software VP8 behind the same capture,
|
||||
source and Pion transport boundary. The pinned WebRTC encoder pipeline supplies
|
||||
libvpx; VP8 reads NV12 through a reusable staging texture and converts to I420.
|
||||
Separate real Browser gates proved manual VP8 and Auto-selected H264, including
|
||||
live presets, pause/resume,
|
||||
source replacement and exact RTP codec preservation. Auto uses a bounded
|
||||
synthetic target-profile cadence check, not a score or a runtime codec switch.
|
||||
GPU-heavy selection and cross-device startup remain open acceptance work.
|
||||
@@ -87,12 +91,11 @@ GPU-heavy selection and cross-device startup remain open acceptance work.
|
||||
A 2026-09-28 check on Ryzen 7 9700X / RTX 4070 SUPER / Windows 11 26200
|
||||
used the animated Browser gate source at a 1080p, 8 Mbps ceiling. With asynchronous
|
||||
MFT readiness and no `timeBeginPeriod(1)` request, Auto selected NVIDIA H264 at
|
||||
30 and 60 fps, reaching active in 1.851 and 1.855 s respectively. Over roughly
|
||||
6.8-second steady windows, source and output both averaged 27.9 fps at 30 and
|
||||
54.1 fps at 60; helper CPU averaged 3.7% and 6.9% of one logical core. The source
|
||||
did not supply 60 fps, so this is not proof of 60-fps game capture or a measured
|
||||
speedup over the previous implementation. Separate full Browser gates passed
|
||||
Auto, manual H264 and VP8 with source/profile replacement and retirement.
|
||||
30 and 60 fps, reaching active in 1.851 and 1.855 s respectively. Full Browser
|
||||
gates passed Auto, manual H264 and VP8 with source/profile replacement and
|
||||
retirement. The helper-cost measurement below uses asynchronous CPU sampling:
|
||||
blocking the probe's encoded-output reader while querying process statistics
|
||||
distorts sustained cadence and must not be used for performance comparison.
|
||||
|
||||
Native Auto now observes the live pipeline's output, including the frame dropper
|
||||
implicated by the [Browser AMD evidence](./realtime-quality-adaptation.md#h264-root-cause-and-gate).
|
||||
@@ -111,14 +114,11 @@ switch. The result does not yet prove macOS/Linux physical capture or endurance.
|
||||
Native Host media is exposed only through an explicit App-launched Host
|
||||
selection; an ordinary Web Host retains Browser capture.
|
||||
|
||||
Native code does not publish directly to LiveKit. One local Pion edge gives the
|
||||
system Browser a remote `MediaStreamTrack`; WebRTC requires that remote track to
|
||||
reject capture constraints, so native capture keeps source ownership while the
|
||||
existing SFU publisher owns sender parameters, simulcast, Dynacast, and recovery.
|
||||
The bridge adds one local decode for Host preview and a Browser encode only when
|
||||
SFU publication is active. Direct P2P children continue to reuse the one native
|
||||
H.264 encode. This preserves the accepted SFU behavior without another LiveKit
|
||||
SDK or media policy.
|
||||
Native Host publication now sends its encoded source directly to the embedded
|
||||
SFU. The local Browser bridge supplies preview and Browser quality candidates;
|
||||
it does not supply the ordinary Native SFU publication. Received tracks still
|
||||
cannot own capture constraints. [Media quality](../standards/media-quality.md#framework-owned-adaptation)
|
||||
owns the current output and adaptation contract.
|
||||
|
||||
A shared encode cannot independently adapt one bitstream for unequal paths.
|
||||
Rather than lower every Native child, a quality operation originating from a
|
||||
@@ -129,17 +129,11 @@ edge to Native. Healthy Native edges keep sharing the hardware encode. This is
|
||||
implemented without a new route reason, timer, score, or representation ladder;
|
||||
controlled weak-path physical acceptance remains open.
|
||||
|
||||
Native P2P edges negotiate transport-wide feedback and use Pion's send-side GCC
|
||||
with its immediate no-op pacer. The pacer neither queues nor applies one edge's
|
||||
estimate to the shared encoder. Once real feedback and source frames exist, the
|
||||
edge compares GCC's target payload bitrate with the H.264 plus Opus payload
|
||||
actually produced in the same window. A lower target is `degraded/bandwidth`, a
|
||||
sufficient target is `healthy/none`, and absent feedback or source progress is
|
||||
`unknown`. The existing two-second evidence cadence and three-window route rule
|
||||
own persistence. This is a direct capacity relation, not a loss/RTT score or a
|
||||
new adaptation ladder. The known Pion no-op-pacer issue concerns separately
|
||||
negotiated RTX SSRCs; the current native H.264 contract has no RTX codec and must
|
||||
reopen that choice before adding one.
|
||||
Native edges now use the shared Pion/LiveKit forwarding adapter for transport
|
||||
feedback, allocation, pacing and recovery. The former no-op-pacer and
|
||||
source-payload-versus-GCC comparison are no longer the product implementation.
|
||||
[Observable truth](../standards/media-quality.md#observable-truth) owns current
|
||||
quality evidence, including committed allocation deficiency and unknown feedback.
|
||||
|
||||
No-Site Internet control can use the one-link mode owned by ADR-0010. Its remote
|
||||
Pion gate proves public signaling plus direct media, but not decoded Browser
|
||||
@@ -151,10 +145,11 @@ Native shares use the same Pion UDP socket for all edges. Site and one-link
|
||||
shares make one bounded, best-effort PCP, UPnP, or NAT-PMP mapping through
|
||||
NetBird's standalone Go NAT package; pure LAN Local mode does not. A physical
|
||||
router created and removed an ephemeral UPnP mapping; the Apache-2.0 dependency
|
||||
added about 0.38 MiB to the stripped Windows App. Mapping begins
|
||||
with the share, is awaited before the first PeerConnection, refreshes only when
|
||||
a later edge arrives after half the requested lease, and is removed with the
|
||||
engine. Failure is cached for that share and leaves ordinary ICE/STUN unchanged.
|
||||
added about 0.38 MiB to the stripped Windows App. Mapping and the supplemental
|
||||
STUN survey now run alongside ordinary candidate gathering; neither delays the
|
||||
offer or ordinary candidates. End-of-candidates waits for both owners. Mapping
|
||||
refresh and retirement remain engine-owned; [NAT evidence](./nat-traversal.md)
|
||||
records the current dependency and reachability limits.
|
||||
The returned port is advertised once per observed public address as a
|
||||
lower-priority candidate. Only the three explicit same-socket survey candidates
|
||||
feed NAT prediction; the mapped candidate does not. This proves lifecycle and
|
||||
@@ -170,8 +165,8 @@ With the failing AMD candidate preferred, bounded product selection now reaches
|
||||
the working NVIDIA H264 encoder without changing codec. A local native Host gate
|
||||
also passes decoded Viewer output, quality changes and source replacement. This
|
||||
establishes the selection defect and repair, not the unknown reporter's cause.
|
||||
The product therefore observes GCC for routing but does not yet apply one edge's
|
||||
target globally to the shared encoder.
|
||||
This earlier fixture proves dynamic hardware rate changes, not the current
|
||||
multi-output allocation policy or cross-vendor performance.
|
||||
|
||||
The Windows source boundary now enumerates displays and windows and returns a
|
||||
bounded best-effort preview. A display uses WGC plus default render-device
|
||||
@@ -225,17 +220,68 @@ physical Linux gate. GStreamer stays a system dependency: bundling another RTC
|
||||
or an 80+ MB media runtime would defeat the thin-adapter boundary, while an
|
||||
unavailable dependency cleanly leaves Browser capture available.
|
||||
|
||||
## Windows Helper Cost (2026-09-29)
|
||||
|
||||
At `89a03e64` on Ryzen 7 9700X / RTX 4070 SUPER / Windows 11 26200,
|
||||
an instrumented Release helper captured a controlled animated Browser window
|
||||
and an offline CS2 scene window. Each run used one 1920x1080 output with an
|
||||
8 Mbps ceiling; CPU sampling did not block the encoded-output reader. CPU below
|
||||
is a percentage of **one logical core**, not total-machine use. The scene runs
|
||||
are sequential observations, not matched gameplay or a cross-device benchmark.
|
||||
|
||||
| Source / codec | Target fps | WGC interval | Sample | Admitted / encoded fps | Helper CPU |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| Browser / H264 | 30 | 4 ms | 15.8 s | 30.0 / 30.1 | 4.6% |
|
||||
| Browser / H264 | 60 | 4 ms | 15.8 s | 60.1 / 60.0 | 9.6% |
|
||||
| Browser / VP8 | 60 | 4 ms | 15.7 s | 60.0 / 60.0 | 75.6% |
|
||||
| CS2 / H264 | 60 | 4 ms | 45.8 s | 54.8 / 52.7 | 5.7% |
|
||||
| CS2 / H264 | 60 | 8 ms | 45.8 s | 53.0 / 51.4 | 5.3% |
|
||||
| CS2 / VP8 | 60 | 4 ms | 20.9 s | 54.9 / 53.3 | 82.9% |
|
||||
|
||||
Across these runs, NV12 conversion API work averaged 0.060–0.226 ms per frame,
|
||||
including texture/view creation; owned-input allocation/copy submission averaged
|
||||
0.044–0.048 ms. These are CPU-side call durations, not GPU completion timings.
|
||||
They do not justify adding a texture pool. In particular, MFT input samples may
|
||||
remain referenced after submission; one mutable reused surface would discard
|
||||
the current ownership guarantee. The 8 ms trial establishes no worthwhile gain
|
||||
and does not supersede the earlier 144 Hz cadence evidence or the existing 4 ms
|
||||
setting, also used by Sunshine.
|
||||
|
||||
At 60 fps, explicitly requesting recovery every 250 ms retained 59.9 encoded
|
||||
fps on the Browser source and 56.9 on the CS2 source, with normal capture exits.
|
||||
A separate unchanged window stayed active for over 20 seconds in each codec;
|
||||
recovery requests produced fresh H264 keyframes from its retained image. These
|
||||
are one-output stress/idle checks, not proof of multi-output recovery cost or
|
||||
endurance. Code tracing also confirms membership/demand changes use recovery to
|
||||
wake a quiet source and apply its next output plan; SFU demand emits only on
|
||||
change. Removing these requests solely as redundant keyframes is unsafe.
|
||||
|
||||
H264 frame-rate bitrate compensation matches Chromium's MFT approach: compensate
|
||||
for the fixed configured frame rate without restarting the encoder. An adjusted
|
||||
API value above the requested bitrate is not by itself wire-rate overshoot.
|
||||
The measured aggregate H264 payload stayed below 8 Mbps, including recovery
|
||||
stress; this does not establish every short-window burst or driver behavior.
|
||||
|
||||
No automatic capture termination was reproduced. Admitted capture input stayed
|
||||
below 60 fps during the CS2 runs; these observations do not separate game render
|
||||
cadence, WGC delivery and capture sampling. They also do not measure game FPS
|
||||
impact, exclusive fullscreen/display changes, saturated GPU behavior, end-to-end
|
||||
decoded media or other GPUs. Software encoding costs more CPU here, but neither
|
||||
that result nor the allocation timings establish the reported system-wide lag's
|
||||
cause. Retain the existing implementation; reopen optimization from a matched
|
||||
bottleneck.
|
||||
|
||||
## Implementation Boundary
|
||||
|
||||
- `nativecapture` owns the child process, source identity, and bounded frame protocol.
|
||||
- `mediaedge` owns the stable Pion API, one UDP mux, shared H.264/Opus sources,
|
||||
- `mediaedge` owns the stable Pion API, one UDP mux, shared encoded sources,
|
||||
and independent PeerConnections.
|
||||
- `nativehost` owns the current capture generation and its bounded stable edges.
|
||||
- `nativeviewer` owns one native inbound H.264/Opus source and its encoded child
|
||||
- `nativeviewer` owns one native inbound media source and its encoded child
|
||||
edges; it does not own room or route state.
|
||||
- `nativecontrol` maps local source/share/receive/edge commands and exact native
|
||||
sender quality windows, live profile updates, and source replacement to the
|
||||
loopback v8 wire.
|
||||
current loopback wire.
|
||||
|
||||
The deleted sender application, UI, room client, and old wire are not
|
||||
compatibility inputs. Historical measurements remain in the separately marked
|
||||
@@ -250,11 +296,10 @@ compatibility inputs. Historical measurements remain in the separately marked
|
||||
- [WASAPI process loopback](https://learn.microsoft.com/en-us/samples/microsoft/windows-classic-samples/applicationloopbackaudio-sample/)
|
||||
- [OBS application-audio capture guide](https://obsproject.com/kb/application-audio-capture-guide/)
|
||||
- [Media Foundation hardware MFTs](https://learn.microsoft.com/en-us/windows/win32/medfound/hardware-mfts)
|
||||
- [MFT input sample ownership](https://learn.microsoft.com/en-us/windows/win32/api/mftransform/nf-mftransform-imftransform-processinput)
|
||||
- [Chromium MFT frame-rate compensation](https://chromium.googlesource.com/chromium/src/+/master/media/gpu/windows/media_foundation_video_encode_accelerator_win.cc)
|
||||
- [Pion v4.2.18 stats implementation](https://github.com/pion/webrtc/blob/v4.2.18/stats.go)
|
||||
- [Pion Google congestion control](https://github.com/pion/interceptor/tree/v0.1.47/pkg/gcc)
|
||||
- [Pion bandwidth-estimation example](https://github.com/pion/webrtc/tree/v4.2.18/examples/bandwidth-estimation-from-disk)
|
||||
- [WebRTC Stats target bitrate and limitation semantics](https://www.w3.org/TR/webrtc-stats/)
|
||||
- [Pion no-op pacer RTX issue](https://github.com/pion/interceptor/issues/406)
|
||||
- [WebRTC remote-track constraints](https://www.w3.org/TR/webrtc/#mediastreamtrack-network-use)
|
||||
- [Pion single-port ICE](https://github.com/pion/webrtc/tree/master/examples/ice-single-port)
|
||||
- [Pion broadcast relay](https://github.com/pion/webrtc/tree/master/examples/broadcast)
|
||||
|
||||
@@ -160,12 +160,18 @@ quality/source changes retain it. Other native platform encoders remain H264.
|
||||
|
||||
## Framework-Owned Adaptation
|
||||
|
||||
Each Browser direct, relay, or SFU video sender owns one clone of its capture or received
|
||||
source track. The original track remains a source and local presentation track;
|
||||
it is never attached directly to an outbound sender. Replacing or retiring a
|
||||
sender also stops its clone, so native adaptation state cannot survive by being
|
||||
inherited through the original track. Host pause and live capture constraints
|
||||
are propagated to current Host-owned clones.
|
||||
Each Browser direct, relay, or SFU video sender owns a separate track. The original
|
||||
track remains a source and local presentation track; it is never attached directly
|
||||
to an outbound sender. Display capture with raw-frame stream support forwards a
|
||||
bounded input clone into a generated sender track, without resizing or encoding.
|
||||
This prevents encoder feedback from shrinking its own capture recovery input;
|
||||
[the Chromium capture lock](../research/browser-local-encoding-pool.md#sustained-h264-recovery)
|
||||
owns the evidence. Camera/received tracks and unsupported Browsers retain ordinary
|
||||
clones. Capture constraints and capture metadata belong to the input. Its native
|
||||
idle refresh is retained through a 1 fps capture constraint; sender adaptation has
|
||||
no new FPS or resolution floor. Pause gates the outgoing track. Replacement and
|
||||
retirement cancel the frame pipe and stop both owned ends; an unexpected input
|
||||
end belongs to the current sending operation, never the shared source or siblings.
|
||||
Connection retirement also settles its pending sender and negotiation waits;
|
||||
closing a Browser PeerConnection alone does not guarantee that its promises
|
||||
settle. Cancellation starts before joining work that depends on it, while a
|
||||
@@ -182,7 +188,9 @@ Producer startup protection begins at actual outgoing publication, excluding
|
||||
local warmup and paused frames. The synthetic carrier uses screen-content
|
||||
transport probing; the real producer keeps the Host's picture adaptation.
|
||||
Capture constraints and the picture's sender FPS ceiling belong to the real
|
||||
source and producer, not the producer-driven carrier clock. Outgoing transport
|
||||
source and producer, not the producer-driven carrier clock. The carrier
|
||||
preserves its frame rate instead of adapting picture cadence a second time.
|
||||
The producer retains the Host's degradation preference. Outgoing transport
|
||||
bitrate still follows the Host ceiling; ordinary fallback restores its picture
|
||||
sender limits. Preparation and applied-budget completion re-evaluate pending
|
||||
membership immediately; accepting a recovery frame still commits the handoff.
|
||||
@@ -191,9 +199,9 @@ pooling use ordinary senders. Prepared ordinary quality candidates, Native
|
||||
ingress and Browser SFU remain independent compositions.
|
||||
|
||||
Each ordinary Browser PeerConnection owns stock WebRTC congestion control and
|
||||
sender adaptation. Clones share one underlying media source and therefore do not provide
|
||||
complete simultaneous isolation: framework source-wants aggregation may still
|
||||
partially reduce frames available to sibling clones. Sibling outputs may differ;
|
||||
sender adaptation. Ordinary clones share one underlying media source and therefore
|
||||
do not provide complete simultaneous isolation: framework source-wants aggregation
|
||||
may still partially reduce frames available to sibling clones. Sibling outputs may differ;
|
||||
Piik does not impose a room-wide minimum. The Host's one Browser SFU
|
||||
publication uses the share-generation codec and selected ceilings. Its ordinary
|
||||
simulcast outputs follow pinned LiveKit screen-share construction: original and
|
||||
|
||||
+20
-7
@@ -1,6 +1,6 @@
|
||||
# Current TODO Ledger
|
||||
|
||||
Last reviewed: 2026-09-29
|
||||
Last reviewed: 2026-09-30
|
||||
|
||||
Only **Now** is executable. Product modules own behavior; Git/PRs own completed
|
||||
history. A parked idea is not implementation authority.
|
||||
@@ -12,10 +12,19 @@ history. A parked idea is not implementation authority.
|
||||
admission, and narrow/touch UI. Verify confirmed fixes through the existing
|
||||
owners before resuming feature polish. Keep field reports without matched
|
||||
evidence separate from locally reproduced defects.
|
||||
- [ ] **Windows native helper cost.** Measure GPU texture/view allocation,
|
||||
WGC update cadence, source-wide keyframe demand and frame-rate bitrate
|
||||
compensation under game load before changing them. VP8 CPU cost and these
|
||||
candidate costs do not establish the reported system-wide lag's cause.
|
||||
- [ ] **Sustained Browser degradation.** The H264 field cause remains open.
|
||||
[Comparisons and Chromium traces](./research/browser-local-encoding-pool.md#sustained-h264-recovery)
|
||||
distinguish encoder recovery hysteresis from a tab-capture feedback/size-step
|
||||
lock reproduced in bare WebRTC. Verify the upstream soft-target/hard-maximum
|
||||
repair candidate against a patched Browser, which remains unbuilt. The Piik
|
||||
sender boundary now isolates display frames while preserving native idle refresh,
|
||||
capture intent and operation retirement. Real hidden-tab SFU checks pass;
|
||||
broader device/background coverage remains. Native logs confirm content-driven
|
||||
QP hysteresis and a cost from bitrate reconfiguration, but not repeated brief
|
||||
blur. Match field Host/Viewer evidence
|
||||
before assigning the field cause or declaring repeated brief blur fixed.
|
||||
Retain existing clone/startup protections and candidate proof; no periodic
|
||||
reset or quality floor. A recent Piik regression is not established.
|
||||
|
||||
## Deferred Feature Work
|
||||
|
||||
@@ -145,8 +154,12 @@ is active. Retain these decisions for resumption, not as parallel work.
|
||||
- [ ] **System becomes very laggy after starting a share.** Obtain mode, actual
|
||||
codec, profile, display refresh rate and CPU/GPU use, distinguishing startup
|
||||
from sustained lag. Auto's local selection checks and NVIDIA success do not
|
||||
establish this reporter's cause; software encoding and the helper costs under
|
||||
Now remain candidate explanations.
|
||||
establish this reporter's cause. The bounded
|
||||
[helper-cost checks](./research/native-client-media.md#windows-helper-cost-2026-09-29)
|
||||
found no justified allocation/cadence change and no capture termination on the
|
||||
tested NVIDIA machine. Saturated GPU/game-FPS impact, multi-output recovery
|
||||
cost and the reporter's environment remain unmeasured; software encoding cost
|
||||
alone is not a diagnosis.
|
||||
|
||||
## Next: P2P Connection And Feedback Evidence
|
||||
|
||||
|
||||
@@ -148,7 +148,6 @@ func NewPublication(options TransportOptions) (_ *Publication, err error) {
|
||||
return nil, trackErr
|
||||
}
|
||||
publication.tracks = append(publication.tracks, track)
|
||||
track.SetMaxTemporalLayer(0)
|
||||
track.SetMaxSpatialLayer(int32(index))
|
||||
local := &publicationTrack{DownTrack: track, id: options.Source.TrackID(), rid: PublicationRID(index, publication.activeCount), pacer: publication.pacer, counters: &publication.counters[index]}
|
||||
if transceiver == nil {
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
package forwarding
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/livekit/livekit-server/pkg/sfu"
|
||||
"github.com/pion/rtp"
|
||||
"github.com/pion/rtp/codecs"
|
||||
"github.com/pion/webrtc/v4"
|
||||
)
|
||||
|
||||
// Browser VP8 can produce L1T3 even with only one spatial encoding. Both
|
||||
// forwarding owners must retain its enhancement frames, not silently cap T0.
|
||||
// This exercises RTP forwarding; the Browser gate separately checks decoding.
|
||||
func TestForwardingPreservesVP8TemporalLayers(t *testing.T) {
|
||||
for _, name := range []string{"transport", "publication"} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
|
||||
defer cancel()
|
||||
check := func(err error) {
|
||||
t.Helper()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
codec := webrtc.RTPCodecParameters{RTPCodecCapability: webrtc.RTPCodecCapability{
|
||||
MimeType: webrtc.MimeTypeVP8, ClockRate: 90_000,
|
||||
}, PayloadType: 96}
|
||||
source, err := NewSource(SourceOptions{ID: "screen", StreamID: "temporal", Codec: codec,
|
||||
Formats: []LayerFormat{{Width: 8, Height: 8, Bitrate: 300_000}}})
|
||||
check(err)
|
||||
defer source.Close()
|
||||
check(source.BindLayer(0, 1000, webrtc.RTPParameters{Codecs: []webrtc.RTPCodecParameters{codec}}))
|
||||
settings := webrtc.SettingEngine{}
|
||||
settings.SetNetworkTypes([]webrtc.NetworkType{webrtc.NetworkTypeUDP4})
|
||||
settings.SetIncludeLoopbackCandidate(true)
|
||||
settings.SetIPFilter(func(ip net.IP) bool { return ip.IsLoopback() })
|
||||
options := TransportOptions{Source: source, Settings: settings, ConnectionID: name, InitialBitrate: 1_000_000}
|
||||
var pc *webrtc.PeerConnection
|
||||
var connect func() error
|
||||
if name == "publication" {
|
||||
publication, err := NewPublication(options)
|
||||
check(err)
|
||||
defer publication.Close()
|
||||
pc, connect = publication.PC, publication.SetConnected
|
||||
} else {
|
||||
transport, err := NewTransport(options)
|
||||
check(err)
|
||||
defer transport.Close()
|
||||
pc, connect = transport.PC, transport.SetConnected
|
||||
}
|
||||
receiver, err := webrtc.NewAPI(webrtc.WithSettingEngine(settings)).NewPeerConnection(webrtc.Configuration{})
|
||||
check(err)
|
||||
defer receiver.Close()
|
||||
var received [3]atomic.Uint32
|
||||
receiver.OnTrack(func(track *webrtc.TrackRemote, _ *webrtc.RTPReceiver) {
|
||||
for {
|
||||
packet, _, err := track.ReadRTP()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
var vp8 codecs.VP8Packet
|
||||
if _, err := vp8.Unmarshal(packet.Payload); err == nil && vp8.T == 1 && vp8.TID < 3 {
|
||||
received[vp8.TID].Add(1)
|
||||
}
|
||||
}
|
||||
})
|
||||
offer, err := pc.CreateOffer(nil)
|
||||
check(err)
|
||||
gather := webrtc.GatheringCompletePromise(pc)
|
||||
check(pc.SetLocalDescription(offer))
|
||||
select {
|
||||
case <-gather:
|
||||
case <-ctx.Done():
|
||||
t.Fatal(ctx.Err())
|
||||
}
|
||||
check(receiver.SetRemoteDescription(*pc.LocalDescription()))
|
||||
answer, err := receiver.CreateAnswer(nil)
|
||||
check(err)
|
||||
gather = webrtc.GatheringCompletePromise(receiver)
|
||||
check(receiver.SetLocalDescription(answer))
|
||||
select {
|
||||
case <-gather:
|
||||
case <-ctx.Done():
|
||||
t.Fatal(ctx.Err())
|
||||
}
|
||||
check(pc.SetRemoteDescription(*receiver.LocalDescription()))
|
||||
for pc.ConnectionState() != webrtc.PeerConnectionStateConnected {
|
||||
select {
|
||||
case <-time.After(5 * time.Millisecond):
|
||||
case <-ctx.Done():
|
||||
t.Fatal(ctx.Err())
|
||||
}
|
||||
}
|
||||
check(connect())
|
||||
for frame := uint16(0); received[0].Load() < 4 || received[1].Load() < 4 || received[2].Load() < 4; frame++ {
|
||||
tid := []byte{0, 2, 1, 2}[frame%4]
|
||||
// RFC 7741: I/L/T descriptor, PictureID, TL0PICIDX, TID/Y.
|
||||
payload := append([]byte{0x90, 0xe0, byte(frame & 0x7f), byte(frame / 4), tid<<6 | 0x20}, sfu.VP8KeyFrame8x8...)
|
||||
if frame%20 != 0 {
|
||||
payload[5] |= 1 // Delta-frame header; only T0 carries keyframes.
|
||||
}
|
||||
check(source.WriteRTP(0, &rtp.Packet{Header: rtp.Header{Version: 2, Marker: true,
|
||||
PayloadType: 96, SSRC: 1000, SequenceNumber: frame + 1, Timestamp: uint32(frame+1) * 3000}, Payload: payload}))
|
||||
select {
|
||||
case <-time.After(time.Second / 30):
|
||||
case <-ctx.Done():
|
||||
t.Fatalf("forwarding discarded temporal layers: T0=%d T1=%d T2=%d", received[0].Load(), received[1].Load(), received[2].Load())
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -167,8 +167,6 @@ func NewTransport(options TransportOptions) (_ *Transport, err error) {
|
||||
transport.driveProbeLocked()
|
||||
}
|
||||
})
|
||||
// Spatial selection arms acquisition; its temporal bound must already be valid.
|
||||
transport.Output.SetMaxTemporalLayer(0)
|
||||
transport.Output.SetMaxSpatialLayer(int32(len(options.Source.TrackInfo().Layers) - 1))
|
||||
transceiver, err := transport.PC.AddTransceiverFromTrack(track,
|
||||
webrtc.RTPTransceiverInit{Direction: webrtc.RTPTransceiverDirectionSendonly})
|
||||
|
||||
@@ -32,9 +32,10 @@ unavailable instead of failing the whole source. Process loopback is probed by
|
||||
activation rather than inferred from a Windows build number; display sources
|
||||
use the standard render-device loopback available on Windows 10 and later.
|
||||
|
||||
Auto compares target-profile encoding work within a four-second selection
|
||||
budget; H.264 that meets the target needs no software comparison. The selected
|
||||
codec remains fixed across profile and source changes. VP8 reads the existing
|
||||
Auto measures delivered frame cadence through the target-profile encoding
|
||||
pipeline within a four-second selection budget; H.264 that sustains the target
|
||||
needs no software comparison. The selected codec remains fixed across profile
|
||||
and source changes. VP8 reads the existing
|
||||
NV12 surface through one staging texture and uses the same encoded-frame
|
||||
boundary. The process has no network fallback. The App
|
||||
consumes the selected process or system-audio stream through its native media
|
||||
|
||||
@@ -13,16 +13,28 @@ const nativeSource = settings.get('nativeSource') === '1', background = settings
|
||||
const lifecycle = settings.get('lifecycle') === '1', shapedRate = Number(settings.get('rate') || 120000);
|
||||
const relay = settings.get('relay') === '1';
|
||||
const shortPulse = settings.get('shortPulse') === '1';
|
||||
const displaySource = settings.get('displaySource') === '1';
|
||||
const quietStart = settings.get('quietStart') === '1';
|
||||
const selectedProfile = () => ({ resolution: high ? '1080p' : '480p', maxFramerate: 30,
|
||||
maxBitrate: high ? 5000000 : 1700000,
|
||||
degradationPreference: settings.get('preference') || (high ? 'maintain-resolution' : 'balanced'),
|
||||
screenAudioQuality: 'music' });
|
||||
const productPool = mode === 'carrier' ? new ProbePool() : null;
|
||||
const productPeers = [], peers = [], tracks = [], edges = [], monitors = [], audioMonitors = [];
|
||||
const errors = [], sourcePulses = [], controls = [], visibility = [], productTrace = [];
|
||||
const errors = [], sourcePulses = [], controls = [], visibility = [], productTrace = [], sourceFrames = [];
|
||||
let sourceReader, sourceReading;
|
||||
const displayClones = [];
|
||||
if (displaySource) {
|
||||
const clone = MediaStreamTrack.prototype.clone;
|
||||
MediaStreamTrack.prototype.clone = function () {
|
||||
const track = clone.call(this);
|
||||
if (this.getSettings().displaySurface) displayClones.push(track);
|
||||
return track;
|
||||
};
|
||||
}
|
||||
const sourceTimes = new Map(), seenProductGroups = new Map();
|
||||
let rawSource, drawing, productTraceTimer, audioContext, audioTrack, audioGain, upstreamPeer;
|
||||
let producing = true, sourceFrame = 0, phase = 'startup';
|
||||
let producing = !quietStart, sourceFrame = 0, phase = 'startup';
|
||||
const wait = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
const poolState = () => productPool ? {
|
||||
members: [...productPool.members].map(m => ({ id: m.id, group: m.current?.producer.id,
|
||||
@@ -44,7 +56,7 @@ function watch(video, role) {
|
||||
let stopped = false;
|
||||
const next = () => video.requestVideoFrameCallback((_time, metadata) => {
|
||||
if (stopped) return;
|
||||
if (nativeSource) {
|
||||
if (nativeSource || displaySource) {
|
||||
samples.push({ at: performance.now(), phase, id: metadata.presentedFrames, age: null,
|
||||
width: video.videoWidth, height: video.videoHeight });
|
||||
} else {
|
||||
@@ -222,10 +234,25 @@ async function run() {
|
||||
if (!nativeSource) document.body.prepend(source);
|
||||
const ctx = source.getContext('2d');
|
||||
if (high) ctx.scale(3, 3);
|
||||
rawSource = nativeSource ? (await navigator.mediaDevices.getUserMedia({
|
||||
if (displaySource) {
|
||||
document.body.style.margin = '0';
|
||||
source.style.cssText = 'position:fixed;inset:0;width:100vw;height:100vh;z-index:10';
|
||||
}
|
||||
rawSource = displaySource ? await new Promise((resolve, reject) => {
|
||||
window.acquireDisplay = () => navigator.mediaDevices.getDisplayMedia({ audio: false, preferCurrentTab: true,
|
||||
selfBrowserSurface: 'include', video: { width: source.width, height: source.height, frameRate: 30 } })
|
||||
.then(stream => resolve(stream.getVideoTracks()[0]), reject);
|
||||
}) : nativeSource ? (await navigator.mediaDevices.getUserMedia({
|
||||
video: { width: source.width, height: source.height, frameRate: 30 }, audio: false
|
||||
})).getVideoTracks()[0] : source.captureStream(0).getVideoTracks()[0];
|
||||
rawSource.contentHint = 'motion'; tracks.push(rawSource);
|
||||
if (displaySource) {
|
||||
sourceReader = new MediaStreamTrackProcessor({ track: rawSource, maxBufferSize: 1 }).readable.getReader();
|
||||
sourceReading = (async () => { for (;;) {
|
||||
const { done, value } = await sourceReader.read(); if (done) break;
|
||||
sourceFrames.push({ at: performance.now(), phase, width: value.displayWidth, height: value.displayHeight }); value.close();
|
||||
} })().catch(error => errors.push(String(error)));
|
||||
}
|
||||
await probeApplyProfile(rawSource, selectedProfile());
|
||||
const main = rawSource;
|
||||
const connectChildren = async () => {
|
||||
@@ -247,7 +274,7 @@ async function run() {
|
||||
for (let bit = 0; bit < 12; bit++) {
|
||||
ctx.fillStyle = id & (1 << bit) ? '#fff' : '#000'; ctx.fillRect(bit * 50, 0, 50, 30);
|
||||
}
|
||||
(relay ? main : rawSource).requestFrame();
|
||||
(relay ? main : rawSource).requestFrame?.();
|
||||
}, 1000 / (lifecycle ? 60 : 30));
|
||||
if (productPool) productTraceTimer = setInterval(() => productTrace.push({ at: performance.now(), phase, ...poolState() }), 500);
|
||||
if (relay) {
|
||||
@@ -272,7 +299,7 @@ async function run() {
|
||||
else activated = null;
|
||||
if (activated !== null && monitors.every(m => {
|
||||
const last = m.samples.at(-1);
|
||||
return last?.width > 16 && (nativeSource ? last.at >= activated :
|
||||
return last?.width > 16 && (nativeSource || displaySource ? last.at >= activated :
|
||||
last.age !== null && (sourceTimes.get(last.id) ?? -1) >= activated);
|
||||
})) break;
|
||||
if (performance.now() >= deadline) throw Error('Product shared output did not become fresh');
|
||||
@@ -283,6 +310,15 @@ async function run() {
|
||||
extensions: e.send.localDescription?.sdp.match(/^a=extmap:.*$/gm),
|
||||
parameters: e.sender.getParameters().encodings }));
|
||||
const snapshots = [await snapshot()];
|
||||
if (quietStart) {
|
||||
const deadline = performance.now() + 10000;
|
||||
while (!monitors.every(m => m.samples.some(s => s.width > 16))) {
|
||||
if (performance.now() >= deadline) throw Error('Static display produced no first frame');
|
||||
await wait(30);
|
||||
}
|
||||
controls.push({ action: 'static-first-frame', at: performance.now(), sourceFrame });
|
||||
producing = true;
|
||||
}
|
||||
phase = 'healthy'; await wait(shortPulse ? 45000 : 6000); snapshots.push(await snapshot());
|
||||
if (lifecycle) {
|
||||
let appliedProfile = selectedProfile();
|
||||
@@ -299,7 +335,7 @@ async function run() {
|
||||
if (name.startsWith('quiet-') || name.startsWith('paused-')) return;
|
||||
for (const monitor of monitors) {
|
||||
const last = monitor.samples.at(-1), ceiling = { '480p': 480, '720p': 720, '1080p': 1080 }[appliedProfile.resolution];
|
||||
if (last?.phase !== name || last.age === null || last.age >= 1000 ||
|
||||
if (last?.phase !== name || (!displaySource && (last.age === null || last.age >= 1000)) ||
|
||||
performance.now() - last.at >= 1000 || last.height > ceiling)
|
||||
throw Error(name + ': receiver did not deliver current frames within the requested ceiling');
|
||||
}
|
||||
@@ -346,13 +382,19 @@ async function run() {
|
||||
await fetch('/shaper?rate=0');
|
||||
// --auto extends recovery observation; adaptation always belongs to the product.
|
||||
phase = 'released'; await wait(automatic ? 40000 : 10000); snapshots.push(await snapshot());
|
||||
if (displaySource) {
|
||||
const healthy = sourceFrames.findLast(frame => frame.phase === 'healthy');
|
||||
if (!healthy || sourceFrames.some(frame => ['limited', 'released'].includes(frame.phase) &&
|
||||
(frame.width < healthy.width - 2 || frame.height < healthy.height - 2)))
|
||||
throw Error('Sender adaptation reduced its display-capture input');
|
||||
}
|
||||
}
|
||||
return { descriptions, snapshots, browser: navigator.userAgent, sourcePulses,
|
||||
return { descriptions, snapshots, browser: navigator.userAgent, sourcePulses, sourceFrames,
|
||||
audioMonitors: audioMonitors.map(({ role, pulses, attached }) => ({ role, pulses, attached })),
|
||||
monitors: monitors.map(({ role, samples }) => ({ role, samples })) };
|
||||
}
|
||||
const configuration = { mode, codec, product: true, network, automatic, av, high, single, late, shortPulse,
|
||||
background, nativeSource, lifecycle, relay, shapedRate, profile: selectedProfile() };
|
||||
background, nativeSource, displaySource, quietStart, lifecycle, relay, shapedRate, profile: selectedProfile() };
|
||||
try {
|
||||
window.probeResult = { ...configuration, ...await run(), controls, visibility, productTrace, errors };
|
||||
if (browserDebugEnabled) {
|
||||
@@ -370,7 +412,10 @@ try {
|
||||
monitors.forEach(m => m.stop()); audioMonitors.forEach(m => m.stop());
|
||||
productPeers.forEach(peer => peer.dispose()); productPool?.dispose();
|
||||
upstreamPeer?.dispose();
|
||||
await sourceReader?.cancel();
|
||||
await sourceReading;
|
||||
tracks.forEach(t => t.stop()); peers.forEach(pc => pc.close());
|
||||
await audioContext?.close();
|
||||
if (displayClones.some(track => track.readyState !== 'ended')) errors.push('Sender retirement left a live display-capture clone');
|
||||
window.probeDone = true;
|
||||
}
|
||||
@@ -9,13 +9,15 @@ import { createPoolShaper } from './browser-pool-shaper.mjs';
|
||||
|
||||
const mode = process.argv[2] || 'carrier', args = process.argv.slice(3);
|
||||
const flags = new Set(['--product', '--1080', '--h264', '--single', '--av', '--late', '--background',
|
||||
'--native-source', '--lifecycle', '--network', '--auto', '--relay', '--debug', '--short-pulse']);
|
||||
'--native-source', '--lifecycle', '--network', '--auto', '--relay', '--debug', '--short-pulse', '--display', '--quiet-start']);
|
||||
if (!['ordinary', 'carrier'].includes(mode)) throw Error('Expected ordinary|carrier');
|
||||
for (const arg of args) if (!flags.has(arg) && !arg.startsWith('--rate=') && !arg.startsWith('--preference='))
|
||||
throw Error('Unsupported option: ' + arg);
|
||||
const codec = args.includes('--h264') ? 'H264' : 'VP8', network = args.includes('--network');
|
||||
const automatic = args.includes('--auto'), high = args.includes('--1080'), av = args.includes('--av');
|
||||
const single = args.includes('--single'), late = args.includes('--late'), background = args.includes('--background');
|
||||
const displaySource = args.includes('--display');
|
||||
const quietStart = args.includes('--quiet-start');
|
||||
const nativeSource = background || args.includes('--native-source'), lifecycle = args.includes('--lifecycle');
|
||||
const relay = args.includes('--relay');
|
||||
const shortPulse = args.includes('--short-pulse');
|
||||
@@ -23,9 +25,11 @@ const rate = Number(args.find(arg => arg.startsWith('--rate='))?.split('=')[1] |
|
||||
const preference = args.find(arg => arg.startsWith('--preference='))?.split('=')[1] || (high ? 'maintain-resolution' : 'balanced');
|
||||
if (!['balanced', 'maintain-resolution', 'maintain-framerate'].includes(preference)) throw Error('Unsupported degradation preference');
|
||||
if (!Number.isInteger(rate) || rate <= 0) throw Error('Rate must be a positive integer in bps');
|
||||
if (lifecycle && (nativeSource || network || relay)) throw Error('Lifecycle uses direct canvas capture without network shaping');
|
||||
if (lifecycle && (nativeSource || network || relay)) throw Error('Lifecycle uses direct canvas/display capture without network shaping');
|
||||
if (single && late) throw Error('Single-viewer control cannot also late-join B');
|
||||
if (shortPulse && (!network || !automatic)) throw Error('Short pulse requires --network --auto');
|
||||
if (displaySource && nativeSource) throw Error('Display and fake-device sources are separate probes');
|
||||
if (quietStart && !displaySource) throw Error('Quiet start requires real display capture');
|
||||
const root = resolve(import.meta.dirname, '..');
|
||||
const bundle = await build({ entryPoints: [join(root, 'scripts/browser-local-pool-page.js')],
|
||||
bundle: true, write: false, format: 'esm', platform: 'browser', logLevel: 'error' });
|
||||
@@ -70,20 +74,26 @@ const port = (server.address() as { port: number }).port, debugPort = await rese
|
||||
const profile = await mkdtemp(join(tmpdir(), 'piik-client-media-'));
|
||||
const chrome = await launchChrome(process.env.CHROME_PATH || 'C:/Program Files/Google/Chrome/Application/chrome.exe', debugPort, profile,
|
||||
['--headless=new', '--no-first-run', '--autoplay-policy=no-user-gesture-required',
|
||||
...(displaySource ? ['--window-size=1920,1200', '--force-device-scale-factor=1',
|
||||
'--auto-accept-this-tab-capture', '--auto-select-tab-capture-source-by-title=Browser local pool'] : []),
|
||||
...(background ? [] : ['--disable-background-timer-throttling', '--disable-renderer-backgrounding']),
|
||||
...(nativeSource ? ['--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream'] : [])]);
|
||||
chrome.stdout.resume(); chrome.stderr.resume();
|
||||
let result: Record<string, unknown> = { mode, codec, product: true, network, automatic, high, av, single, late, shortPulse,
|
||||
background, nativeSource, lifecycle, relay, shapedRate: rate, preference };
|
||||
background, nativeSource, displaySource, quietStart, lifecycle, relay, shapedRate: rate, preference };
|
||||
try {
|
||||
const version = await waitForVersion(debugPort, chrome);
|
||||
cdp = await CdpConnection.connect(version.webSocketDebuggerUrl, Date.now() + 5000);
|
||||
page = await createPage(cdp, 'about:blank');
|
||||
const query = new URLSearchParams({ mode, codec, rate: String(rate), preference });
|
||||
if (process.argv.includes('--debug')) query.set('debug', '1');
|
||||
for (const [key, value] of Object.entries({ network, automatic, high, av, single, late, background, nativeSource, lifecycle, relay, shortPulse }))
|
||||
for (const [key, value] of Object.entries({ network, automatic, high, av, single, late, background, nativeSource, displaySource, quietStart, lifecycle, relay, shortPulse }))
|
||||
query.set(key, value ? '1' : '0');
|
||||
await cdp.call('Page.navigate', { url: 'http://127.0.0.1:' + port + '/?' + query }, page.sessionId, Date.now() + 5000);
|
||||
if (displaySource) {
|
||||
await waitForSample(() => evaluate<boolean>(cdp!, page!, '!!window.acquireDisplay', Date.now() + 1000), Boolean, 5000);
|
||||
await cdp.call('Runtime.evaluate', { expression: 'void window.acquireDisplay()', userGesture: true }, page.sessionId, Date.now() + 3000);
|
||||
}
|
||||
await waitForSample(() => evaluate<boolean>(cdp!, page!, 'window.probeDone===true', Date.now() + 2000),
|
||||
Boolean, automatic || lifecycle ? 110000 : network || background ? 60000 : 25000);
|
||||
result = await evaluate<Record<string, unknown>>(cdp, page, 'window.probeResult', Date.now() + 3000);
|
||||
@@ -98,8 +108,8 @@ try {
|
||||
result.probeSha256 = createHash('sha256').update(body).digest('hex');
|
||||
result.cleanup = clean;
|
||||
await mkdir(join(root, 'build/browser-local-pool'), { recursive: true });
|
||||
const name = [mode, codec.toLowerCase(), network && 'network', automatic && 'auto', shortPulse && 'pulse', high && '1080',
|
||||
single && 'single', late && 'late', background && 'background', lifecycle && 'lifecycle', relay && 'relay', Date.now()].filter(Boolean).join('-');
|
||||
const name = [mode, codec.toLowerCase(), displaySource && 'display', network && 'network', automatic && 'auto', shortPulse && 'pulse', high && '1080',
|
||||
single && 'single', late && 'late', quietStart && 'quiet-start', background && 'background', lifecycle && 'lifecycle', relay && 'relay', Date.now()].filter(Boolean).join('-');
|
||||
const output = join(root, 'build/browser-local-pool', name + '.json');
|
||||
await writeFile(output, JSON.stringify(result, null, 2) + '\n');
|
||||
console.log(JSON.stringify({ mode, codec, network, error: result.error ?? null, output, cleanup: clean }));
|
||||
|
||||
@@ -6,6 +6,7 @@ for (const filename of process.argv.slice(2)) {
|
||||
const configuration = { file: filename, probeSha256: result.probeSha256, mode: result.mode, codec: result.codec,
|
||||
product: result.product, high: result.high, single: result.single, late: result.late, network: result.network,
|
||||
automatic: result.automatic, shortPulse: result.shortPulse, av: result.av, background: result.background, nativeSource: result.nativeSource,
|
||||
displaySource: result.displaySource, quietStart: result.quietStart,
|
||||
lifecycle: result.lifecycle, relay: result.relay, shapedRate: result.shapedRate, profile: result.profile,
|
||||
completed: !!completed, error: result.error, errors: result.errors, cleanup: result.cleanup };
|
||||
if (!completed) process.exitCode = 1;
|
||||
@@ -41,19 +42,23 @@ for (const filename of process.argv.slice(2)) {
|
||||
duplicates++;
|
||||
}
|
||||
if (samples.length) maxGapMs = Math.max(maxGapMs, next.at - samples.at(-1).at);
|
||||
row.viewers[monitor.role] = { rendered: samples.length, fresh: result.nativeSource ? null : samples.filter(s => s.age !== null && s.age < 1000).length, backwards, duplicates, maxGapMs,
|
||||
const sourceIds = !result.nativeSource && !result.displaySource;
|
||||
row.viewers[monitor.role] = { rendered: samples.length, fresh: sourceIds ? samples.filter(s => s.age !== null && s.age < 1000).length : null, backwards, duplicates, maxGapMs,
|
||||
ageP50: ages[Math.floor(ages.length * .5)] ?? null, ageP95: ages[Math.floor(ages.length * .95)] ?? null };
|
||||
if (result.av) {
|
||||
const sound = result.audioMonitors.find(m => m.role === monitor.role);
|
||||
const offsets = [];
|
||||
let missed = 0;
|
||||
for (const pulse of result.sourcePulses.filter(p => p.at >= prev.at && p.at < next.at - 500)) {
|
||||
// Native/display samples count presented frames; they do not decode
|
||||
// source IDs and therefore cannot be paired with audio pulses.
|
||||
for (const pulse of sourceIds ? result.sourcePulses.filter(p => p.at >= prev.at && p.at < next.at - 500) : []) {
|
||||
const video = samples.find(s => s.id >= pulse.id && s.id < pulse.id + 6);
|
||||
const audio = sound?.pulses.find(p => p.at >= pulse.at && p.at < pulse.at + 1500);
|
||||
if (video && audio) offsets.push(video.at - audio.at);
|
||||
else missed++;
|
||||
}
|
||||
row.viewers[monitor.role].av = { attached: sound?.attached ?? false, videoMinusAudioMs: offsets, missed };
|
||||
row.viewers[monitor.role].av = { attached: sound?.attached ?? false,
|
||||
videoMinusAudioMs: sourceIds ? offsets : null, missed: sourceIds ? missed : null };
|
||||
}
|
||||
}
|
||||
phases.push(row);
|
||||
|
||||
@@ -22,6 +22,11 @@ const AUDIO_PROBE = `(() => {
|
||||
window.RTCPeerConnection = class extends BrowserPeer {
|
||||
constructor(...args) { super(...args); peers.push(this); }
|
||||
};
|
||||
window.__piikGateMediaStats = async () => Promise.all(peers.map(async peer => ({
|
||||
state: peer.connectionState,
|
||||
stats: Array.from((await peer.getStats()).values()).filter(row =>
|
||||
['media-source','outbound-rtp','inbound-rtp','candidate-pair'].includes(row.type))
|
||||
})));
|
||||
window.__piikGateAudioEnergy = async () => {
|
||||
const trackId = document.querySelector('video')?.srcObject?.getAudioTracks()[0]?.id;
|
||||
const diagnostic = window.__piikGateAudioStats = { hasAudioTrack: !!trackId, peerCount: peers.length, inbound: [] };
|
||||
@@ -60,6 +65,8 @@ async function main(): Promise<void> {
|
||||
const chromePath = process.env.CHROME_PATH?.trim();
|
||||
if (!chromePath) throw new Error("CHROME_PATH is required");
|
||||
const nativeArm = process.env.PIIK_EMBEDDED_SFU_NATIVE === "true";
|
||||
const displayArm = process.env.PIIK_EMBEDDED_SFU_DISPLAY === "true";
|
||||
if (displayArm && nativeArm) throw new Error("Display and Native capture are separate arms");
|
||||
const twoRooms = process.env.PIIK_EMBEDDED_SFU_TWO_ROOMS === "true";
|
||||
if (twoRooms && !nativeArm) throw new Error("Two-room arm requires Native publication");
|
||||
const codec = process.env.PIIK_EMBEDDED_SFU_CODEC?.trim() || "vp8";
|
||||
@@ -85,7 +92,7 @@ async function main(): Promise<void> {
|
||||
const profile = await mkdtemp(join(tmpdir(), "piik-client-media-"));
|
||||
const sourceProfile = nativeArm ? await mkdtemp(join(tmpdir(), "piik-client-media-")) : null;
|
||||
const [sourcePort, sourceDebugPort] = nativeArm ? [await reservePort(), await reservePort()] : [0, 0];
|
||||
const result = { passed: false, arm: `${nativeArm ? "native" : "browser"}-${codec}-simulcast${twoRooms ? "-two-rooms" : ""}`, stage: "start",
|
||||
const result = { passed: false, arm: `${nativeArm ? "native" : displayArm ? "display" : "browser"}-${codec}-simulcast${twoRooms ? "-two-rooms" : ""}`, stage: "start",
|
||||
startedAt: new Date().toISOString(), finishedAt: "", processes: [] as Array<{ role: string; pid: number | null }>,
|
||||
high: null as { frames: number; width: number; height: number } | null,
|
||||
low: null as { frames: number; width: number; height: number } | null,
|
||||
@@ -93,6 +100,8 @@ async function main(): Promise<void> {
|
||||
audioDiagnostics: null as unknown,
|
||||
secondRoom: twoRooms ? { connected: false, framesAfterFirstStopped: 0, audioBefore: 0, audioAfter: 0, stopped: false } : null,
|
||||
host: null as Snapshot | null, viewer: null as Snapshot | null,
|
||||
displayLifecycle: null as { hiddenFrames: number; hiddenElapsedMs: number; lowLayerDecoded: boolean; resumed: boolean; replaced: boolean } | null,
|
||||
mediaDiagnostics: null as unknown,
|
||||
publicationRetired: false, subscriptionRecovered: false, nativeShareStopped: !nativeArm, udpReleased: false, cleanup: false, error: null as string | null };
|
||||
let server: ChildProcessWithoutNullStreams | null = null;
|
||||
let chrome: ChildProcessWithoutNullStreams | null = null;
|
||||
@@ -147,6 +156,15 @@ async function main(): Promise<void> {
|
||||
proxy: { "/api": { target: backend }, "/signal": { target: backend, ws: true } } },
|
||||
plugins: [{ name: "embedded-sfu-gate", configureServer(instance) {
|
||||
instance.middlewares.use((request, response, next) => {
|
||||
if (request.url === "/embedded-sfu-source") {
|
||||
response.setHeader("Content-Type", "text/html; charset=utf-8");
|
||||
response.end(`<!doctype html><title>Piik SFU captured source</title><style>body{margin:0}canvas{width:100vw;height:100vh}</style>
|
||||
<canvas width="1280" height="720"></canvas><script>
|
||||
const canvas=document.querySelector('canvas'),ctx=canvas.getContext('2d');let n=0;
|
||||
function draw(){ctx.fillStyle='#167d8d';ctx.fillRect(0,0,1280,720);ctx.fillStyle='#f2d748';
|
||||
ctx.fillRect((n++*17)%1280,60,100,100);requestAnimationFrame(draw)}draw();</script>`);
|
||||
return;
|
||||
}
|
||||
if (request.url !== "/embedded-sfu-gate") return next();
|
||||
response.setHeader("Content-Type", "text/html; charset=utf-8");
|
||||
response.end("<!doctype html><html><head><title>Embedded SFU acceptance</title></head><body></body></html>");
|
||||
@@ -184,8 +202,9 @@ async function main(): Promise<void> {
|
||||
chrome = await launchChrome(chromePath, debugPort, profile, [
|
||||
"--headless=new", "--disable-gpu", "--no-first-run", "--no-default-browser-check",
|
||||
"--disable-extensions", "--disable-background-networking", "--autoplay-policy=no-user-gesture-required",
|
||||
"--disable-background-timer-throttling", "--disable-backgrounding-occluded-windows",
|
||||
"--disable-renderer-backgrounding", "--disable-features=WebRtcHideLocalIpsWithMdns",
|
||||
...(displayArm ? ["--auto-select-tab-capture-source-by-title=Piik SFU captured source"] : [
|
||||
"--disable-background-timer-throttling", "--disable-backgrounding-occluded-windows", "--disable-renderer-backgrounding"]),
|
||||
"--disable-features=WebRtcHideLocalIpsWithMdns",
|
||||
]);
|
||||
processStarted("decoder-browser", chrome);
|
||||
chrome.stdout.resume();
|
||||
@@ -200,12 +219,17 @@ async function main(): Promise<void> {
|
||||
if (!gatePage.ok || !(await gatePage.text()).includes("Embedded SFU acceptance")) {
|
||||
throw new Error("Local SFU gate page is unavailable");
|
||||
}
|
||||
host = await createPage(cdp, `${origin}/embedded-sfu-gate`);
|
||||
if (displayArm) {
|
||||
const sourcePage = await createPage(cdp, `${origin}/embedded-sfu-source`);
|
||||
await cdp.call("Emulation.setDeviceMetricsOverride", { width: 1280, height: 720,
|
||||
deviceScaleFactor: 1, mobile: false }, sourcePage.sessionId, Date.now() + 3_000);
|
||||
}
|
||||
host = await createPage(cdp, `${origin}/embedded-sfu-gate`, AUDIO_PROBE);
|
||||
viewer = await createPage(cdp, `${origin}/embedded-sfu-gate`, AUDIO_PROBE);
|
||||
result.stage = "authenticate";
|
||||
const room = await call<CreateRoomResponse>(host, nativeArm
|
||||
? `gate.startHost(${JSON.stringify({ title: SOURCE_TITLE, port: nativePort })}, ${JSON.stringify(codec)})`
|
||||
: `gate.startHost(undefined, ${JSON.stringify(codec)})`);
|
||||
: `gate.startHost(undefined, ${JSON.stringify(codec)}, ${displayArm})`);
|
||||
await until(host, (value) => value.authenticated);
|
||||
await call(viewer, `gate.startViewer(${JSON.stringify(room)})`);
|
||||
result.stage = "simulcast-first-frame";
|
||||
@@ -234,6 +258,34 @@ async function main(): Promise<void> {
|
||||
await waitForSample(() => call<number>(viewer!, "gate.audioEnergy()", 3_000),
|
||||
(energy) => { result.audioEnergy.high = energy; return energy > 0; }, 5_000);
|
||||
result.host = await call(host, "gate.snapshot()");
|
||||
if (displayArm) {
|
||||
result.stage = "display-background";
|
||||
if (result.host!.capture?.displaySurface !== "browser" || result.host!.visibility !== "hidden") {
|
||||
throw new Error("Gate must capture a separate real tab with the Host hidden");
|
||||
}
|
||||
const hiddenStart = Date.now();
|
||||
const hidden = await call<{ frames: number }>(viewer, "gate.waitForFrames(1280, 720, 150)", 32_000);
|
||||
result.displayLifecycle = { hiddenFrames: hidden.frames, hiddenElapsedMs: Date.now() - hiddenStart,
|
||||
lowLayerDecoded: false, resumed: false, replaced: false };
|
||||
result.mediaDiagnostics = await Promise.all([host, viewer].map(page =>
|
||||
evaluate(cdp!, page, "window.__piikGateMediaStats()", Date.now() + 3_000)));
|
||||
result.stage = "display-layer-demand";
|
||||
await call(host, "gate.demandLayers(1)");
|
||||
await call(viewer, "gate.waitForFrames(640, 360)");
|
||||
result.displayLifecycle.lowLayerDecoded = true;
|
||||
await call(host, "gate.demandLayers(2)");
|
||||
await call(viewer, "gate.waitForFrames(1280, 720)");
|
||||
result.stage = "display-pause-resume";
|
||||
await call(host, "gate.pause(true)");
|
||||
await new Promise(resolve => setTimeout(resolve, 500));
|
||||
await call(host, "gate.pause(false)");
|
||||
await call(viewer, "gate.waitForFrames(1280, 720)");
|
||||
result.displayLifecycle.resumed = true;
|
||||
result.stage = "display-source-replacement";
|
||||
if (!await call<boolean>(host, "gate.replaceSource()")) throw new Error("Display source replacement failed");
|
||||
await call(viewer, "gate.waitForFrames(1280, 720)");
|
||||
result.displayLifecycle.replaced = true;
|
||||
}
|
||||
result.stage = "live-profile";
|
||||
if (!await call<boolean>(host, "gate.lowerProfile()")) {
|
||||
result.host = await call(host, "gate.snapshot()");
|
||||
@@ -278,6 +330,8 @@ async function main(): Promise<void> {
|
||||
}
|
||||
} catch (error) {
|
||||
result.error = error instanceof Error ? error.message : String(error);
|
||||
if (cdp) result.mediaDiagnostics = await Promise.all([host, viewer].map(page => page ?
|
||||
evaluate(cdp!, page, "window.__piikGateMediaStats?.()", Date.now() + 3_000).catch(() => null) : null));
|
||||
if (viewer && cdp) result.viewer = await call<Snapshot>(viewer, "gate.snapshot()", 3_000).catch(() => null);
|
||||
if (viewer && cdp && result.stage === "decoded-audio-energy") result.audioDiagnostics = await evaluate(
|
||||
cdp, viewer, "window.__piikGateAudioStats ?? null", Date.now() + 3_000).catch(() => null);
|
||||
@@ -307,6 +361,8 @@ async function main(): Promise<void> {
|
||||
result.viewer.audioKbps > 0 && result.audioEnergy.high > 0 && result.audioEnergy.low > result.audioEnergy.high &&
|
||||
result.high !== null && result.low !== null && result.audioEnergy.recovered > 0 &&
|
||||
result.publicationRetired && result.subscriptionRecovered && result.cleanup && result.nativeShareStopped &&
|
||||
(!displayArm || result.displayLifecycle?.hiddenFrames === 150 && result.displayLifecycle.lowLayerDecoded &&
|
||||
result.displayLifecycle.resumed && result.displayLifecycle.replaced) &&
|
||||
(!result.secondRoom || result.secondRoom.connected && result.secondRoom.framesAfterFirstStopped >= 30 &&
|
||||
result.secondRoom.audioBefore > 0 && result.secondRoom.audioAfter > result.secondRoom.audioBefore && result.secondRoom.stopped) &&
|
||||
(!nativeArm || result.host.nativeShareStarted && result.host.nativeFramesPerSecond > 0 && result.host.nativeBitrateKbps > 0 && result.host.nativeEncodingCount === 2);
|
||||
|
||||
@@ -194,7 +194,7 @@ async function connect(room: CreateRoomResponse, host: boolean): Promise<void> {
|
||||
: { ...common, role: "viewer", viewerGrant: new URLSearchParams(new URL(room.inviteUrl).hash.slice(1)).get("v")! });
|
||||
}
|
||||
|
||||
export async function startHost(nativeSource?: { title: string; port: number }, selectedCodec: "vp8" | "h264" = "vp8"): Promise<CreateRoomResponse> {
|
||||
export async function startHost(nativeSource?: { title: string; port: number }, selectedCodec: "vp8" | "h264" = "vp8", display = false): Promise<CreateRoomResponse> {
|
||||
codec = selectedCodec;
|
||||
const response = await fetch("/api/rooms", {
|
||||
method: "POST", headers: { "Content-Type": "application/json" },
|
||||
@@ -235,7 +235,9 @@ export async function startHost(nativeSource?: { title: string; port: number },
|
||||
context.font = "48px sans-serif";
|
||||
context.fillText(String(frame), 40, 240);
|
||||
}, 1000 / 15);
|
||||
source = canvas.captureStream(15);
|
||||
source = display ? await navigator.mediaDevices.getDisplayMedia({
|
||||
video: { width: 1280, height: 720, frameRate: 15 }, audio: false,
|
||||
}) : canvas.captureStream(15);
|
||||
audio = new AudioContext();
|
||||
const tone = audio.createOscillator();
|
||||
const gain = audio.createGain();
|
||||
@@ -263,6 +265,7 @@ export async function reconnectViewer(room: CreateRoomResponse): Promise<void> {
|
||||
|
||||
export function snapshot() {
|
||||
return { authenticated, published, simulcast, decoded, committed, peerFailures,
|
||||
visibility: document.visibilityState, capture: source?.getVideoTracks()[0]?.getSettings(),
|
||||
routeEvents, routeStatus,
|
||||
connectionId: configuration?.connectionId,
|
||||
audioKbps, audioCodec, publicationRetired, sharingStopped, nativeShareStarted, nativeShareStopped,
|
||||
@@ -321,6 +324,33 @@ export async function lowerProfile(): Promise<boolean> {
|
||||
return await hostRoute!.updateProfile(low);
|
||||
}
|
||||
|
||||
export function pause(paused: boolean): void {
|
||||
source?.getTracks().forEach(track => { track.enabled = !paused; });
|
||||
hostRoute?.setPaused(paused);
|
||||
}
|
||||
|
||||
// Exercise the existing server-to-publisher demand contract independently of
|
||||
// the local machine's unconstrained subscriber bandwidth.
|
||||
export async function demandLayers(activeCount: number): Promise<void> {
|
||||
if (!publisher || !configuration || native) throw new Error("Browser publisher is unavailable");
|
||||
const { revision, publicationGeneration, connectionId } = configuration;
|
||||
await publisher.acceptSignal({ type: "sfu-signal", kind: "layers", activeCount,
|
||||
revision, publicationGeneration, connectionId });
|
||||
}
|
||||
|
||||
export async function replaceSource(): Promise<boolean> {
|
||||
if (!source || !hostRoute || native) throw new Error("Browser source is unavailable");
|
||||
const previous = source;
|
||||
source = previous.clone();
|
||||
if (!await hostRoute.replaceStream(source)) {
|
||||
source.getTracks().forEach(track => track.stop());
|
||||
source = previous;
|
||||
return false;
|
||||
}
|
||||
previous.getTracks().forEach(track => track.stop());
|
||||
return true;
|
||||
}
|
||||
|
||||
export function stopSharing(): void {
|
||||
stopping = true;
|
||||
send({ type: "stop-sharing", shareGeneration });
|
||||
|
||||
@@ -4,10 +4,11 @@ import { debugRtcFailure, debugRtcStats, debugTrack, observeDebugConnection } fr
|
||||
import { addRemoteIceCandidate } from "../webrtc/nat-prediction";
|
||||
import { encodedStreams } from "./browser-encoding-output";
|
||||
import {
|
||||
applyVideoCaptureProfile, cloneSenderVideoTrack, configureVideoSender,
|
||||
applyVideoCaptureProfile, configureVideoSender,
|
||||
needsStartupVideoProfile, startupVideoProfile, STARTUP_VIDEO_ENCODED_FRAMES,
|
||||
videoQualitySettingsEqual, type QualityProfile,
|
||||
} from "./quality";
|
||||
import { cloneSenderVideoTrack, stopSenderVideoTrack } from "./sender-video-track";
|
||||
|
||||
// Leave time for ordinary encoding before the outer Viewer/route deadline.
|
||||
// Bound local transport setup, not frame production from a quiet source.
|
||||
@@ -53,7 +54,7 @@ export class BrowserEncodingProducer {
|
||||
this.budget = initialBudget;
|
||||
try {
|
||||
if (this.source.readyState !== "live") throw new Error("Browser encoding source ended");
|
||||
const track = this.input = cloneSenderVideoTrack(this.source);
|
||||
const track = this.input = cloneSenderVideoTrack(this.source, () => this.fail());
|
||||
track.enabled = !this.paused;
|
||||
this.source.addEventListener("ended", this.fail);
|
||||
track.addEventListener("ended", this.fail);
|
||||
@@ -181,7 +182,7 @@ export class BrowserEncodingProducer {
|
||||
debugEvent("encoding-pool", "producer-retired", { producerId: this.id });
|
||||
this.source.removeEventListener("ended", this.fail);
|
||||
this.input?.removeEventListener("ended", this.fail);
|
||||
this.input?.stop();
|
||||
stopSenderVideoTrack(this.input);
|
||||
this.streamAbort.abort();
|
||||
void this.streamWriter?.abort().catch(() => undefined);
|
||||
this.streamWriter = null;
|
||||
|
||||
@@ -11,6 +11,7 @@ import type { MediaFailure } from "../ui/media-failure";
|
||||
import { displayMediaOptions } from "./audio-capture";
|
||||
import { browserDebugEnabled, debugOperation } from "../lib/debug";
|
||||
import { debugTrack } from "../lib/debug-webrtc";
|
||||
import { applySenderCaptureConstraints, senderCaptureTrack } from "./sender-video-track";
|
||||
|
||||
export type {
|
||||
DegradationPreference,
|
||||
@@ -249,15 +250,6 @@ export async function applyCaptureProfile(
|
||||
await applyVideoCaptureProfile(videoTrack, profile);
|
||||
}
|
||||
|
||||
export function cloneSenderVideoTrack(
|
||||
source: MediaStreamTrack,
|
||||
): MediaStreamTrack {
|
||||
const clone = source.clone();
|
||||
clone.contentHint = source.contentHint || "motion";
|
||||
clone.enabled = source.enabled;
|
||||
return clone;
|
||||
}
|
||||
|
||||
export async function applyVideoCaptureProfile(
|
||||
track: MediaStreamTrack,
|
||||
profile: QualityProfile,
|
||||
@@ -267,7 +259,7 @@ export async function applyVideoCaptureProfile(
|
||||
}
|
||||
const complete = debugOperation("quality", "capture", { requested: profile, trackId: track.id });
|
||||
try {
|
||||
await track.applyConstraints(captureConstraints(profile));
|
||||
await applySenderCaptureConstraints(track, captureConstraints(profile));
|
||||
complete("applied", { trackId: track.id });
|
||||
debugTrack(track, { event: "profile-applied" });
|
||||
} catch (error) {
|
||||
@@ -277,6 +269,7 @@ export async function applyVideoCaptureProfile(
|
||||
}
|
||||
|
||||
function videoTrackOwnsCaptureConstraints(track: MediaStreamTrack): boolean {
|
||||
track = senderCaptureTrack(track);
|
||||
const capabilities = track.getCapabilities?.();
|
||||
return !capabilities ||
|
||||
"width" in capabilities ||
|
||||
@@ -305,7 +298,7 @@ function requestedScaleResolutionDownBy(
|
||||
}
|
||||
const source =
|
||||
sender.track && typeof sender.track.getSettings === "function"
|
||||
? sender.track.getSettings()
|
||||
? senderCaptureTrack(sender.track).getSettings()
|
||||
: undefined;
|
||||
const ceiling = QUALITY_RESOLUTIONS[profile.resolution];
|
||||
if (!source?.width || !source.height) {
|
||||
@@ -390,7 +383,10 @@ export async function configureVideoSender(
|
||||
if (parameters.encodings.length === 1) {
|
||||
encoding.scaleResolutionDownBy = carrierScale ?? requestedScaleResolutionDownBy(sender, profile);
|
||||
}
|
||||
parameters.degradationPreference = profile.degradationPreference;
|
||||
// The carrier transports already-adapted frames. Native FPS adaptation on
|
||||
// that clock would discard them a second time and repeatedly break the queue.
|
||||
parameters.degradationPreference = carrierScale === undefined
|
||||
? profile.degradationPreference : "maintain-framerate";
|
||||
|
||||
const requested = readVideoSenderParameters(parameters, false, encodingIndex);
|
||||
await sender.setParameters(parameters);
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
import { debugError } from "../lib/debug";
|
||||
|
||||
type FrameTrack = MediaStreamTrack & { writable: WritableStream<VideoFrame> };
|
||||
type FrameAPIs = {
|
||||
MediaStreamTrackProcessor?: new (options: { track: MediaStreamTrack; maxBufferSize: number }) => {
|
||||
readable: ReadableStream<VideoFrame>;
|
||||
};
|
||||
MediaStreamTrackGenerator?: new (options: { kind: "video" }) => FrameTrack;
|
||||
};
|
||||
const inputs = new WeakMap<MediaStreamTrack, { track: MediaStreamTrack; abort: AbortController }>();
|
||||
|
||||
/** The sender owns this track, including its optional display-capture input. */
|
||||
export function cloneSenderVideoTrack(
|
||||
source: MediaStreamTrack,
|
||||
onEnded: (track: MediaStreamTrack) => void,
|
||||
): MediaStreamTrack {
|
||||
const input = source.clone();
|
||||
input.contentHint = source.contentHint || "motion";
|
||||
input.enabled = source.enabled;
|
||||
const { MediaStreamTrackProcessor: Processor, MediaStreamTrackGenerator: Generator } = globalThis as typeof globalThis & FrameAPIs;
|
||||
if (!Processor || !Generator || !source.getSettings().displaySurface || input.readyState !== "live") return input;
|
||||
|
||||
let output: FrameTrack | undefined;
|
||||
let readable: ReadableStream<VideoFrame> | undefined;
|
||||
try {
|
||||
output = new Generator({ kind: "video" });
|
||||
readable = new Processor({ track: input, maxBufferSize: 1 }).readable;
|
||||
} catch (error) {
|
||||
output?.stop();
|
||||
debugError("capture", "sender-frame-isolation-unavailable", error);
|
||||
return input;
|
||||
}
|
||||
// Chromium forwards an encoder's soft pixel target as a capture hard limit.
|
||||
// Forward frames without transforming them so sender adaptation cannot shrink
|
||||
// its own recovery input. No new encoder, frame clock or resolution policy.
|
||||
const abort = new AbortController();
|
||||
const track = output;
|
||||
const frames = readable;
|
||||
input.enabled = true;
|
||||
track.contentHint = input.contentHint;
|
||||
track.enabled = source.enabled;
|
||||
inputs.set(track, { track: input, abort });
|
||||
void applySenderCaptureConstraints(track, input.getConstraints())
|
||||
.then(() => frames.pipeTo(track.writable, { signal: abort.signal }))
|
||||
.catch((error) => {
|
||||
if (!abort.signal.aborted) debugError("capture", "sender-frame-isolation-failed", error);
|
||||
}).finally(() => {
|
||||
inputs.delete(track);
|
||||
// Also cancel when initial constraints failed before pipeTo acquired a reader.
|
||||
void frames.cancel().catch(() => undefined);
|
||||
input.stop();
|
||||
track.stop();
|
||||
if (!abort.signal.aborted) onEnded(track);
|
||||
});
|
||||
return track;
|
||||
}
|
||||
|
||||
/** Capture settings belong to the input, before sender-owned adaptation. */
|
||||
export function senderCaptureTrack(track: MediaStreamTrack): MediaStreamTrack {
|
||||
return inputs.get(track)?.track ?? track;
|
||||
}
|
||||
|
||||
export function applySenderCaptureConstraints(track: MediaStreamTrack, constraints: MediaTrackConstraints): Promise<void> {
|
||||
const input = inputs.get(track)?.track;
|
||||
if (!input) return track.applyConstraints(constraints);
|
||||
const rate = constraints.frameRate;
|
||||
// A Processor does not request idle refresh frames as an RTC sender does.
|
||||
// Preserve the native display-capture refresh with a standard constraint;
|
||||
// this is not a sender FPS floor or an application-generated frame clock.
|
||||
return input.applyConstraints({ ...constraints, frameRate: {
|
||||
...(typeof rate === "object" ? rate : { ideal: rate }), min: 1,
|
||||
} });
|
||||
}
|
||||
|
||||
export function stopSenderVideoTrack(track: MediaStreamTrack | null): void {
|
||||
if (!track) return;
|
||||
const input = inputs.get(track);
|
||||
input?.abort.abort();
|
||||
input?.track.stop();
|
||||
track.stop();
|
||||
}
|
||||
@@ -5,7 +5,6 @@ import { debugRtcFailure, debugRtcStats, debugTrack } from "../lib/debug-webrtc"
|
||||
import {
|
||||
audioSenderParameterWarning,
|
||||
applyVideoCaptureProfile,
|
||||
cloneSenderVideoTrack,
|
||||
configureScreenAudioSender,
|
||||
configureVideoSender,
|
||||
needsStartupVideoProfile,
|
||||
@@ -18,6 +17,7 @@ import {
|
||||
type QualityProfile,
|
||||
type VideoSenderParameterReadback,
|
||||
} from "../media/quality";
|
||||
import { cloneSenderVideoTrack, senderCaptureTrack, stopSenderVideoTrack } from "../media/sender-video-track";
|
||||
import type { ConnectionMetrics } from "../types";
|
||||
import {
|
||||
applyVideoCodecPreference,
|
||||
@@ -146,17 +146,18 @@ export class SfuPublisher {
|
||||
try {
|
||||
this.failureStage = "source";
|
||||
const sourceVideo = requiredVideo(stream);
|
||||
video = this.ownTrack(cloneSenderVideoTrack(sourceVideo));
|
||||
video = this.cloneVideoTrack(sourceVideo);
|
||||
await peer.waitForOperation(() => applyVideoCaptureProfile(video!, profile));
|
||||
if (this.peer !== peer) {
|
||||
this.releaseTrack(video);
|
||||
return false;
|
||||
}
|
||||
if (video.readyState === "ended") throw new Error("SFU video track ended");
|
||||
this.video = video;
|
||||
this.sourceVideo = sourceVideo;
|
||||
this.audio = this.ownTrack(stream.getAudioTracks()[0]?.clone() ?? null);
|
||||
this.setPaused(this.paused);
|
||||
const dimensions = video.getSettings();
|
||||
const dimensions = senderCaptureTrack(video).getSettings();
|
||||
this.layerCount =
|
||||
Math.max(dimensions.width ?? 0, dimensions.height ?? 0) >= 480
|
||||
? 2
|
||||
@@ -230,7 +231,7 @@ export class SfuPublisher {
|
||||
const sourceVideo = requiredVideo(stream);
|
||||
const videoChanged = sourceVideo !== this.sourceVideo;
|
||||
const nextVideo = videoChanged
|
||||
? this.ownTrack(cloneSenderVideoTrack(sourceVideo)) : previousVideo;
|
||||
? this.cloneVideoTrack(sourceVideo) : previousVideo;
|
||||
const nextAudio = this.ownTrack(
|
||||
stream.getAudioTracks()[0]?.clone() ?? null,
|
||||
);
|
||||
@@ -249,6 +250,7 @@ export class SfuPublisher {
|
||||
if (videoChanged) await peer.waitForOperation(() => videoSender.replaceTrack(nextVideo));
|
||||
await peer.waitForOperation(() => audioSender.replaceTrack(nextAudio));
|
||||
if (this.peer !== peer) return false;
|
||||
if (nextVideo.readyState !== "live") throw new Error("SFU replacement video track ended");
|
||||
this.video = nextVideo;
|
||||
this.audio = nextAudio;
|
||||
if (videoChanged) {
|
||||
@@ -257,6 +259,7 @@ export class SfuPublisher {
|
||||
}
|
||||
await this.configure(profile, videoChanged);
|
||||
if (this.peer !== peer) return false;
|
||||
if (nextVideo.readyState !== "live") throw new Error("SFU replacement video track ended");
|
||||
if (!peer.send({ kind: "media", media: this.media(profile) }))
|
||||
throw new Error("SFU signaling is unavailable");
|
||||
retained = true;
|
||||
@@ -377,13 +380,21 @@ export class SfuPublisher {
|
||||
this.events.onDisconnected?.();
|
||||
}
|
||||
|
||||
private cloneVideoTrack(source: MediaStreamTrack): MediaStreamTrack {
|
||||
return this.ownTrack(cloneSenderVideoTrack(source, (track) => {
|
||||
if (this.video === track && this.sourceVideo === source) this.fail("source");
|
||||
}));
|
||||
}
|
||||
|
||||
private ownTrack<T extends MediaStreamTrack | null>(track: T): T {
|
||||
if (track) this.ownedTracks.add(track);
|
||||
return track;
|
||||
}
|
||||
|
||||
private releaseTrack(track: MediaStreamTrack | null): void {
|
||||
if (track && this.ownedTracks.delete(track)) track.stop();
|
||||
if (!track || !this.ownedTracks.delete(track)) return;
|
||||
if (track.kind === "video") stopSenderVideoTrack(track);
|
||||
else track.stop();
|
||||
}
|
||||
|
||||
private restartIce(): Promise<boolean> {
|
||||
@@ -413,7 +424,7 @@ export class SfuPublisher {
|
||||
}
|
||||
|
||||
private encodings(profile: QualityProfile): RTCRtpEncodingParameters[] {
|
||||
const settings = this.video?.getSettings();
|
||||
const settings = this.video ? senderCaptureTrack(this.video).getSettings() : undefined;
|
||||
const ceiling = QUALITY_RESOLUTIONS[profile.resolution];
|
||||
const width = settings?.width ?? ceiling.width;
|
||||
const height = settings?.height ?? ceiling.height;
|
||||
@@ -442,7 +453,7 @@ export class SfuPublisher {
|
||||
}
|
||||
|
||||
private media(profile: QualityProfile): SfuMedia {
|
||||
const settings = this.video!.getSettings();
|
||||
const settings = senderCaptureTrack(this.video!).getSettings();
|
||||
const ceiling = QUALITY_RESOLUTIONS[profile.resolution];
|
||||
return {
|
||||
codec: this.codec,
|
||||
|
||||
@@ -8,7 +8,6 @@ import { debugError, debugEvent } from "../lib/debug";
|
||||
import { debugTrack, observeDebugConnection } from "../lib/debug-webrtc";
|
||||
import {
|
||||
applyVideoCaptureProfile,
|
||||
cloneSenderVideoTrack,
|
||||
configureScreenAudioSender,
|
||||
configureVideoSender,
|
||||
needsStartupVideoProfile,
|
||||
@@ -21,6 +20,7 @@ import {
|
||||
type QualityProfile,
|
||||
type ScreenAudioQuality,
|
||||
} from "../media/quality";
|
||||
import { cloneSenderVideoTrack, stopSenderVideoTrack } from "../media/sender-video-track";
|
||||
import { EMPTY_METRICS, type PeerSnapshot } from "../types";
|
||||
import {
|
||||
captureMetrics,
|
||||
@@ -130,7 +130,7 @@ export class HostPeer {
|
||||
const sourceVideoTrack = stream.getVideoTracks()[0] ?? null;
|
||||
this.paused = sourceVideoTrack?.enabled === false;
|
||||
this.senderVideoTrack = sourceVideoTrack
|
||||
? cloneSenderVideoTrack(sourceVideoTrack)
|
||||
? this.cloneVideoTrack(sourceVideoTrack)
|
||||
: null;
|
||||
this.startupVideoProfilePending = needsStartupVideoProfile(desiredProfile);
|
||||
this.encodedStreamsEnabled = !!videoPool && supportsBrowserEncoding();
|
||||
@@ -144,7 +144,7 @@ export class HostPeer {
|
||||
),
|
||||
});
|
||||
} catch (error) {
|
||||
this.senderVideoTrack?.stop();
|
||||
stopSenderVideoTrack(this.senderVideoTrack);
|
||||
throw error;
|
||||
}
|
||||
observeDebugConnection(this.connection, { connectionId, peerId, role: "send" });
|
||||
@@ -245,7 +245,7 @@ export class HostPeer {
|
||||
}
|
||||
|
||||
const videoChanged = this.stream.getVideoTracks()[0] !== nextSourceVideoTrack;
|
||||
const nextVideoTrack = videoChanged ? cloneSenderVideoTrack(nextSourceVideoTrack) : previousVideoTrack;
|
||||
const nextVideoTrack = videoChanged ? this.cloneVideoTrack(nextSourceVideoTrack) : previousVideoTrack;
|
||||
let retainedNextVideoTrack = false;
|
||||
const nextAudioTrack = nextStream.getAudioTracks()[0] ?? null;
|
||||
const nextAudioDirection = nextAudioTrack ? "sendonly" : "inactive";
|
||||
@@ -265,6 +265,7 @@ export class HostPeer {
|
||||
try {
|
||||
if (videoChanged) await this.waitForOperation(() => videoSender.replaceTrack(nextVideoTrack));
|
||||
await this.waitForOperation(() => audioSender.replaceTrack(nextAudioTrack));
|
||||
if (nextVideoTrack.readyState === "ended") throw new Error("Replacement video track ended");
|
||||
if (audioDirectionChanged) {
|
||||
audioTransceiver.direction = nextAudioDirection;
|
||||
}
|
||||
@@ -296,7 +297,7 @@ export class HostPeer {
|
||||
this.pooledVideo?.dispose();
|
||||
this.pooledVideo = null;
|
||||
this.attachVideoPool(nextSourceVideoTrack);
|
||||
if (previousVideoTrack !== this.encodedOutput?.track) previousVideoTrack.stop();
|
||||
if (previousVideoTrack !== this.encodedOutput?.track) stopSenderVideoTrack(previousVideoTrack);
|
||||
this.startupVideoProfilePending = needsStartupVideoProfile(this.desiredProfile);
|
||||
this.startupFramesBaseline = null;
|
||||
this.snapshot = { ...this.snapshot, metrics: { ...EMPTY_METRICS } };
|
||||
@@ -313,7 +314,7 @@ export class HostPeer {
|
||||
return !audioDirectionChanged || (await this.createOffer());
|
||||
} finally {
|
||||
if (videoChanged && !retainedNextVideoTrack) {
|
||||
nextVideoTrack.stop();
|
||||
stopSenderVideoTrack(nextVideoTrack);
|
||||
}
|
||||
if (this.replacementVideoTrack === nextVideoTrack) {
|
||||
this.replacementVideoTrack = null;
|
||||
@@ -525,7 +526,7 @@ export class HostPeer {
|
||||
this.encodedOutput?.dispose();
|
||||
this.encodedOutput = null;
|
||||
this.localIceCandidates.discard();
|
||||
this.senderVideoTrack?.stop();
|
||||
stopSenderVideoTrack(this.senderVideoTrack);
|
||||
this.senderVideoTrack = null;
|
||||
this.audioTransceiver = null;
|
||||
}
|
||||
@@ -564,6 +565,12 @@ export class HostPeer {
|
||||
this.connection.addEventListener("iceconnectionstatechange", () => this.emit());
|
||||
}
|
||||
|
||||
private cloneVideoTrack(source: MediaStreamTrack): MediaStreamTrack {
|
||||
return cloneSenderVideoTrack(source, (track) => {
|
||||
if (this.senderVideoTrack === track) this.fail();
|
||||
});
|
||||
}
|
||||
|
||||
private attachVideoPool(source: MediaStreamTrack): void {
|
||||
const sender = this.videoSender;
|
||||
if (!sender || !this.videoPool || !this.encodedOutput) return;
|
||||
@@ -580,7 +587,7 @@ export class HostPeer {
|
||||
() => this.enqueueSenderMutation(async () => {
|
||||
if (!owns()) return false;
|
||||
const previous = this.senderVideoTrack!;
|
||||
const next = binding?.carrierScale() === undefined ? cloneSenderVideoTrack(source) : this.encodedOutput!.track;
|
||||
const next = binding?.carrierScale() === undefined ? this.cloneVideoTrack(source) : this.encodedOutput!.track;
|
||||
try {
|
||||
this.applyPausedState(next, null);
|
||||
// The producer already limits real frames. Filtering its requested
|
||||
@@ -588,21 +595,22 @@ export class HostPeer {
|
||||
if (next !== this.encodedOutput?.track) {
|
||||
await this.waitForOperation(() => applyVideoCaptureProfile(next, this.desiredProfile));
|
||||
}
|
||||
if (!owns()) { if (next !== this.encodedOutput?.track) next.stop(); return false; }
|
||||
if (!owns()) { if (next !== this.encodedOutput?.track) stopSenderVideoTrack(next); return false; }
|
||||
if (next !== previous) await this.waitForOperation(() => sender.replaceTrack(next));
|
||||
await this.waitForOperation(() => configureVideoSender(sender,
|
||||
this.startupVideoProfilePending ? startupVideoProfile(this.desiredProfile) : this.desiredProfile,
|
||||
binding?.carrierScale()));
|
||||
if (!owns()) throw new DOMException("Retired Browser pool binding", "AbortError");
|
||||
if (next.readyState === "ended") throw new Error("Replacement video track ended");
|
||||
this.senderVideoTrack = next;
|
||||
if (previous !== next && previous !== this.encodedOutput?.track) previous.stop();
|
||||
if (previous !== next && previous !== this.encodedOutput?.track) stopSenderVideoTrack(previous);
|
||||
this.statsAccumulator = createStatsAccumulator();
|
||||
return true;
|
||||
} catch (error) {
|
||||
if (!this.disposed && sender.track !== previous) {
|
||||
try { await this.waitForOperation(() => sender.replaceTrack(previous)); } catch (rollbackError) { this.fail(rollbackError); }
|
||||
}
|
||||
if (next !== previous && next !== this.encodedOutput?.track) next.stop();
|
||||
if (next !== previous && next !== this.encodedOutput?.track) stopSenderVideoTrack(next);
|
||||
if (this.disposed) return false;
|
||||
debugError("encoding-pool", "carrier-attachment-failed", error, { connectionId: this.connectionId });
|
||||
return false;
|
||||
|
||||
@@ -6,6 +6,7 @@ import { packetLossPercentFromDeltas } from "../../shared/packet-loss";
|
||||
import { deriveVideoCodecEvidence } from "../../shared/video-codec-evidence";
|
||||
import { isPredictedCandidateFoundation } from "../../shared/nat-candidate";
|
||||
import { debugRtcFailure, debugRtcStats } from "../lib/debug-webrtc";
|
||||
import { senderCaptureTrack } from "../media/sender-video-track";
|
||||
|
||||
type StatsRecord = Record<string, unknown> & {
|
||||
id: string;
|
||||
@@ -208,7 +209,7 @@ export function captureMetrics(
|
||||
"captureWidth" | "captureHeight" | "captureFramesPerSecond"
|
||||
> {
|
||||
try {
|
||||
const settings = track.getSettings();
|
||||
const settings = senderCaptureTrack(track).getSettings();
|
||||
const finite = (value: unknown): number | null =>
|
||||
typeof value === "number" && Number.isFinite(value) ? value : null;
|
||||
return {
|
||||
|
||||
@@ -20,6 +20,7 @@ import {
|
||||
} from "../src/client/webrtc/viewer-relay.ts";
|
||||
import { setCopy } from "../src/client/ui/copy.ts";
|
||||
import * as encodingOutput from "../src/client/media/browser-encoding-output";
|
||||
import * as senderTracks from "../src/client/media/sender-video-track";
|
||||
import type { BrowserEncodingPool } from "../src/client/media/browser-encoding-pool";
|
||||
import type {
|
||||
IceConfig,
|
||||
@@ -558,6 +559,44 @@ function encodedPeerFixture() {
|
||||
failAudio: () => audio.error(new Error("audio transform failed")) };
|
||||
}
|
||||
|
||||
describe("HostPeer sender-frame lifetime", () => {
|
||||
it.each(["current", "replaced", "pending"])("handles a %s frame input ending in its owning operation", async (phase) => {
|
||||
const endings: Array<{ track: MediaStreamTrack; end: () => void }> = [];
|
||||
vi.spyOn(senderTracks, "cloneSenderVideoTrack").mockImplementation((source, onEnded) => {
|
||||
const track = source.clone();
|
||||
endings.push({ track, end() {
|
||||
Object.defineProperty(track, "readyState", { value: "ended" });
|
||||
onEnded(track);
|
||||
} });
|
||||
return track;
|
||||
});
|
||||
const source = createTrack("video", "source"), next = createTrack("video", "next");
|
||||
const peer = createPeer(createStream(source, null));
|
||||
try {
|
||||
await peer.start();
|
||||
const pc = FakePeerConnection.latest!, original = pc.senders[0]!.track;
|
||||
if (phase === "current") endings[0]!.end();
|
||||
else if (phase === "replaced") {
|
||||
expect(await peer.replaceStream(createStream(next, null))).toBe(true);
|
||||
endings[0]!.end();
|
||||
expect(pc.senders[0]!.track).toBe(endings[1]!.track);
|
||||
} else {
|
||||
pc.senders[1]!.deferReplaceCall = 1;
|
||||
const replacing = peer.replaceStream(createStream(next, null));
|
||||
await vi.waitFor(() => expect(pc.senders[1]!.replaceTrack).toHaveBeenCalled());
|
||||
endings[1]!.end();
|
||||
pc.senders[1]!.releaseDeferredReplaceTrack();
|
||||
expect(await replacing).toBe(false);
|
||||
expect(pc.senders[0]!.track).toBe(original);
|
||||
expect(endings[1]!.track.stop).toHaveBeenCalled();
|
||||
}
|
||||
expect(pc.connectionState === "closed").toBe(phase === "current");
|
||||
expect(source.stop).not.toHaveBeenCalled();
|
||||
expect(next.stop).not.toHaveBeenCalled();
|
||||
} finally { peer.dispose(); }
|
||||
});
|
||||
});
|
||||
|
||||
describe("HostPeer encoded output ownership", () => {
|
||||
it.each(["initial", "replacement"])("owns output pause when the pool declines the %s source", async (phase) => {
|
||||
const fixture = encodedPeerFixture();
|
||||
|
||||
@@ -253,7 +253,7 @@ describe("realtime quality controls", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it("removes only the carrier's duplicate FPS ceiling and restores ordinary sender limits", async () => {
|
||||
it("leaves picture cadence with the producer and restores ordinary sender adaptation", async () => {
|
||||
let applied = { encodings: [{ maxFramerate: 30 }] } as RTCRtpSendParameters;
|
||||
const sender = {
|
||||
getParameters: () => applied,
|
||||
@@ -261,10 +261,12 @@ describe("realtime quality controls", () => {
|
||||
} as unknown as RTCRtpSender;
|
||||
const carrier = await configureVideoSender(sender, QUALITY_PROFILES["1080p60"], 1);
|
||||
expect(applied.encodings[0]).not.toHaveProperty("maxFramerate");
|
||||
expect(carrier).toMatchObject({ requested: { maxFramerate: null, maxBitrate: 8_000_000 }, mismatches: [] });
|
||||
expect(carrier).toMatchObject({ requested: { maxFramerate: null, maxBitrate: 8_000_000,
|
||||
degradationPreference: "maintain-framerate" }, mismatches: [] });
|
||||
for (const id of ["720p30", "1080p60"] as const) {
|
||||
const readback = await configureVideoSender(sender, QUALITY_PROFILES[id]);
|
||||
expect(applied.encodings[0]!.maxFramerate).toBe(QUALITY_PROFILES[id].maxFramerate);
|
||||
expect(applied.degradationPreference).toBe(QUALITY_PROFILES[id].degradationPreference);
|
||||
expect(readback.mismatches).toEqual([]);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { cloneSenderVideoTrack, senderCaptureTrack, stopSenderVideoTrack } from "../src/client/media/sender-video-track";
|
||||
import { applyVideoCaptureProfile, configureVideoSender, QUALITY_PROFILES } from "../src/client/media/quality";
|
||||
import { captureMetrics } from "../src/client/webrtc/stats";
|
||||
|
||||
class Track extends EventTarget {
|
||||
readonly kind = "video";
|
||||
enabled = true;
|
||||
contentHint = "motion";
|
||||
readyState = "live";
|
||||
settings: MediaTrackSettings = { displaySurface: "browser", width: 1920, height: 1080 };
|
||||
readonly stop = vi.fn(() => { this.readyState = "ended"; });
|
||||
readonly applyConstraints = vi.fn(async (_constraints: MediaTrackConstraints) => undefined);
|
||||
getSettings() { return this.settings; }
|
||||
getConstraints() { return { width: { max: 1920 }, height: { max: 1080 }, frameRate: { ideal: 30, max: 30 } }; }
|
||||
getCapabilities() { return { width: { min: 1, max: 1920 } }; }
|
||||
clone = vi.fn(() => {
|
||||
const clone = new Track();
|
||||
clone.settings = { ...this.settings };
|
||||
clone.enabled = this.enabled;
|
||||
return clone;
|
||||
});
|
||||
}
|
||||
const asTrack = (track: Track) => track as unknown as MediaStreamTrack;
|
||||
const flush = async () => { for (let i = 0; i < 20; i++) await Promise.resolve(); };
|
||||
const owned: MediaStreamTrack[] = [];
|
||||
afterEach(async () => {
|
||||
owned.splice(0).forEach(stopSenderVideoTrack);
|
||||
await flush();
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
function platform() {
|
||||
const cancel = vi.fn(), write = vi.fn();
|
||||
let frames!: ReadableStreamDefaultController<VideoFrame>;
|
||||
const processor = vi.fn(function () {
|
||||
return { readable: new ReadableStream<VideoFrame>({ start(c) { frames = c; }, cancel }) };
|
||||
});
|
||||
class Generator extends Track {
|
||||
static instances: Generator[] = [];
|
||||
override settings = {};
|
||||
writable = new WritableStream<VideoFrame>({ write(frame) { write(frame); frame.close(); } });
|
||||
constructor() { super(); Generator.instances.push(this); }
|
||||
}
|
||||
vi.stubGlobal("MediaStreamTrackProcessor", processor);
|
||||
vi.stubGlobal("MediaStreamTrackGenerator", Generator);
|
||||
return { cancel, write, processor, frames: () => frames, Generator };
|
||||
}
|
||||
function clone(source: Track, onEnded = vi.fn()) {
|
||||
const track = cloneSenderVideoTrack(asTrack(source), onEnded);owned.push(track);return track;
|
||||
}
|
||||
|
||||
describe("sender display-frame ownership", () => {
|
||||
it("cancels a quiet input on retirement without stopping the shared source", async () => {
|
||||
const p = platform(), source = new Track();
|
||||
source.enabled = false;
|
||||
const ended = vi.fn(), track = clone(source, ended), input = source.clone.mock.results[0]!.value;
|
||||
expect(track.enabled).toBe(false);
|
||||
expect(input.enabled).toBe(true);
|
||||
expect(senderCaptureTrack(track)).toBe(input);
|
||||
expect(p.processor).toHaveBeenCalledWith({ track: input, maxBufferSize: 1 });
|
||||
expect(input.applyConstraints).toHaveBeenCalledWith(expect.objectContaining({ frameRate: { min: 1, ideal: 30, max: 30 } }));
|
||||
stopSenderVideoTrack(track);
|
||||
expect(input.readyState).toBe("ended");
|
||||
expect(track.readyState).toBe("ended");
|
||||
await flush();
|
||||
expect(p.cancel).toHaveBeenCalledOnce();
|
||||
expect(source.readyState).toBe("live");
|
||||
expect(source.stop).not.toHaveBeenCalled();
|
||||
expect(ended).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("forwards frames unchanged and leaves profile constraints with the capture input", async () => {
|
||||
const p = platform(), source = new Track(), track = clone(source);
|
||||
const input = source.clone.mock.results[0]!.value;
|
||||
const frame = { close: vi.fn(), timestamp: 123, displayWidth: 1920, displayHeight: 1080 } as unknown as VideoFrame;
|
||||
p.frames().enqueue(frame);
|
||||
await flush();
|
||||
expect(p.write).toHaveBeenCalledWith(frame);
|
||||
expect(frame.close).toHaveBeenCalledOnce();
|
||||
await applyVideoCaptureProfile(track, QUALITY_PROFILES["720p30"]);
|
||||
expect(input.applyConstraints).toHaveBeenCalledWith(expect.objectContaining({ width: { ideal: 1280, max: 1280 } }));
|
||||
expect(input.applyConstraints).toHaveBeenLastCalledWith(expect.objectContaining({ frameRate: { min: 1, ideal: 30, max: 30 } }));
|
||||
expect((track as unknown as Track).applyConstraints).not.toHaveBeenCalled();
|
||||
|
||||
// Generated-track settings can lag by a frame. Do not apply the profile's
|
||||
// scale twice after the input accepted a smaller capture size.
|
||||
input.settings = { displaySurface: "browser", width: 1280, height: 720 };
|
||||
(track as unknown as Track).settings = { width: 1920, height: 1080 };
|
||||
expect(captureMetrics(track)).toEqual({ captureWidth: 1280, captureHeight: 720, captureFramesPerSecond: null });
|
||||
let parameters = { encodings: [{}] } as RTCRtpSendParameters;
|
||||
const sender = { track, getParameters: () => parameters, setParameters: async (next: RTCRtpSendParameters) => { parameters = next; } };
|
||||
await configureVideoSender(sender as unknown as RTCRtpSender, QUALITY_PROFILES["720p30"]);
|
||||
expect(parameters.encodings[0]!.scaleResolutionDownBy).toBe(1);
|
||||
});
|
||||
|
||||
it.each(["ended", "failed"])("retires the generated output when its input %s", async (end) => {
|
||||
const p = platform(), source = new Track(), ended = vi.fn(), track = clone(source, ended);
|
||||
if (end === "ended") p.frames().close();
|
||||
else p.frames().error(new Error("capture ended unexpectedly"));
|
||||
await flush();
|
||||
expect(track.readyState).toBe("ended");
|
||||
expect(source.clone.mock.results[0]!.value.readyState).toBe("ended");
|
||||
expect(source.readyState).toBe("live");
|
||||
expect(ended).toHaveBeenCalledExactlyOnceWith(track);
|
||||
});
|
||||
|
||||
it("keeps camera and received tracks on the ordinary clone path", () => {
|
||||
const p = platform(), source = new Track();source.settings = { width: 640, height: 480 };
|
||||
expect(clone(source)).toBe(source.clone.mock.results[0]!.value);
|
||||
expect(p.processor).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("retires both ends if idle-refresh configuration fails before piping", async () => {
|
||||
const p = platform(), source = new Track(), input = new Track(), ended = vi.fn();
|
||||
source.clone.mockReturnValue(input);
|
||||
input.applyConstraints.mockRejectedValueOnce(new Error("constraints rejected"));
|
||||
const track = clone(source, ended);
|
||||
await flush();
|
||||
expect(p.cancel).toHaveBeenCalledOnce();
|
||||
expect(ended).toHaveBeenCalledExactlyOnceWith(track);
|
||||
expect(input.readyState).toBe("ended");
|
||||
expect(track.readyState).toBe("ended");
|
||||
expect(source.readyState).toBe("live");
|
||||
});
|
||||
|
||||
it("retains the ordinary clone if frame APIs are unavailable or construction fails", () => {
|
||||
vi.stubGlobal("MediaStreamTrackProcessor", undefined);
|
||||
const source = new Track();expect(clone(source)).toBe(source.clone.mock.results[0]!.value);
|
||||
const p = platform();
|
||||
vi.stubGlobal("MediaStreamTrackProcessor", class { constructor() { throw new Error("unavailable"); } });
|
||||
const other = new Track();const track = clone(other);
|
||||
expect(track).toBe(other.clone.mock.results[0]!.value);
|
||||
expect(track.readyState).toBe("live");
|
||||
expect(p.Generator.instances[0]!.readyState).toBe("ended");
|
||||
});
|
||||
});
|
||||
@@ -2,6 +2,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { QUALITY_PROFILES } from "../src/client/media/quality.ts";
|
||||
import { SfuPublisher } from "../src/client/sfu/publisher.ts";
|
||||
import { SfuSubscriber } from "../src/client/sfu/subscriber.ts";
|
||||
import * as senderTracks from "../src/client/media/sender-video-track";
|
||||
import {
|
||||
clientMessageSchema,
|
||||
type SfuSignalMessage,
|
||||
@@ -24,6 +25,7 @@ class FakeTrack extends EventTarget {
|
||||
readonly applyConstraints = vi.fn(
|
||||
async (_constraints: MediaTrackConstraints) => undefined,
|
||||
);
|
||||
getConstraints() { return {}; }
|
||||
readonly clone = vi.fn(() => {
|
||||
const next = new FakeTrack(this.kind);
|
||||
this.clones.push(next);
|
||||
@@ -190,6 +192,7 @@ beforeEach(() => {
|
||||
afterEach(async () => {
|
||||
for (const cleanup of cleanups.splice(0)) await cleanup();
|
||||
vi.useRealTimers();
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
@@ -241,6 +244,57 @@ function feedVideoStats(
|
||||
}
|
||||
|
||||
describe("embedded SFU browser transport", () => {
|
||||
it("uses capture dimensions for simulcast before a generated sender has its first frame", async () => {
|
||||
class Generator extends FakeTrack {
|
||||
writable = new WritableStream<VideoFrame>();
|
||||
constructor() { super("video"); this.getSettings.mockReturnValue({} as never); }
|
||||
}
|
||||
let frames!: ReadableStreamDefaultController<VideoFrame>;
|
||||
vi.stubGlobal("MediaStreamTrackGenerator", Generator);
|
||||
vi.stubGlobal("MediaStreamTrackProcessor", class {
|
||||
readable = new ReadableStream<VideoFrame>({ start(controller) { frames = controller; } });
|
||||
});
|
||||
const video = new FakeTrack("video"), send = vi.fn(() => true), onDisconnected = vi.fn();
|
||||
video.getSettings.mockReturnValue({ ...video.getSettings(), displaySurface: "browser" } as never);
|
||||
const host = new SfuPublisher({ send, onDisconnected });
|
||||
cleanups.push(() => host.disconnect());
|
||||
await host.connect(config);
|
||||
expect(await host.activate(stream(video), QUALITY_PROFILES["1080p30"], "h264")).toBe(true);
|
||||
const sender = FakePc.instances[0]!.transceivers[0]!.sender;
|
||||
expect(sender.track!.getSettings()).toEqual({});
|
||||
expect(sender.parameters.encodings.map(e => e.scaleResolutionDownBy)).toEqual([2, 1]);
|
||||
expect(send).toHaveBeenCalledWith(expect.objectContaining({ media: expect.objectContaining({ layers: [
|
||||
expect.objectContaining({ width: 960, height: 540 }), expect.objectContaining({ width: 1920, height: 1080 }),
|
||||
] }) }));
|
||||
frames.error(new Error("display frame input failed"));
|
||||
for (let i = 0; i < 30; i++) await Promise.resolve();
|
||||
expect(onDisconnected).toHaveBeenCalledOnce();
|
||||
expect(video.stop).not.toHaveBeenCalled();
|
||||
expect(video.clones[0]!.readyState).toBe("ended");
|
||||
});
|
||||
|
||||
it.each(["current", "replaced", "pending", "configuring"])("handles a %s frame input ending without affecting the source", async (phase) => {
|
||||
const endings: Array<{ track: FakeTrack; end: () => void }> = [];
|
||||
vi.spyOn(senderTracks, "cloneSenderVideoTrack").mockImplementation((source, onEnded) => {
|
||||
const track = source.clone() as unknown as FakeTrack;
|
||||
endings.push({ track, end() { track.stop(); onEnded(track as unknown as MediaStreamTrack); } });
|
||||
return track as unknown as MediaStreamTrack;
|
||||
});
|
||||
const { publisher: host, pc, onDisconnected, video } = await publisher();
|
||||
if (phase === "current") endings[0]!.end();
|
||||
else {
|
||||
const next = new FakeTrack("video");
|
||||
if (phase === "pending") pc.transceivers[1]!.sender.replaceTrack.mockImplementationOnce(async () => endings[1]!.end());
|
||||
if (phase === "configuring") pc.transceivers[0]!.sender.setParameters.mockImplementationOnce(async () => endings[1]!.end());
|
||||
expect(await host.replaceStream(stream(next))).toBe(phase === "replaced");
|
||||
if (phase === "replaced") endings[0]!.end();
|
||||
expect(pc.transceivers[0]!.sender.track).toBe(endings[phase === "replaced" ? 1 : 0]!.track);
|
||||
expect(next.stop).not.toHaveBeenCalled();
|
||||
}
|
||||
expect(onDisconnected).toHaveBeenCalledTimes(phase === "current" ? 1 : 0);
|
||||
expect(video.stop).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["queued", "active"])("preserves publication after a %s candidate rejection", async (phase) => {
|
||||
const { publisher: host, pc, onDisconnected, video } = await publisher();
|
||||
const rejected = signal({ kind: "candidate", candidate: { candidate: "candidate:1 1 UDP 1 127.0.0.1 4100 typ host" } });
|
||||
|
||||
Reference in new issue
Block a user