okooooooo/Piik
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(media): validate codec and recovery boundaries

Browse Source
This commit is contained in:
AAA committed 2026-08-26 15:18:00 +08:00
1 parent e23f367313
commit 2d3e82e1f1
6 files changed
+548 -33
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/README.md
+2 -1
View File
@@ -17,7 +17,8 @@
- [Native shared-encode sender](./research/native-shared-encode-sender.md): consolidated Draft #16/#18/#22/#23/#25/#28 ladder, unclassified product-gate failure, staged revalidation, and stop line.
- [Native H.264 hardware decision](./research/native-h264-hardware-decision.md): bounded WebCodecs no-go, proved Media Foundation/NVIDIA hardware encode, the default-fmtp `42c01f` stop, and the retained interop boundary.
- [Native H.264 opt-in path](./research/native-h264-opt-in-path.md): explicit H.264 sender mode, exact Pion profile registration, and one bounded Host/Viewer loopback with hardware caveats.
- [Realtime quality adaptation](./research/realtime-quality-adaptation.md): fixed Browser VP8/no-video-hint evidence, manual quality ceilings, diagnostic-only capture/send/receive correlation, and LiveKit-owned SFU dual-representation adaptation evidence.
- [Realtime quality adaptation](./research/realtime-quality-adaptation.md): Browser codec and `motion` evidence, manual quality ceilings, path-quality authority, and LiveKit-owned SFU representation adaptation.
- [Cross-restart room recovery](./research/cross-restart-room-recovery.md): pending lineage, authority-epoch, invitation-replay, and bounded media-rebuild evidence without reopening durable room storage.
- [Browser background capture diagnostics](./research/browser-background-capture.md): browser lifecycle evidence, a privacy-bounded capture/send/receive experiment, interpretation gates, and non-goal keepalive mechanisms.
- [Browser screen-audio quality](./research/browser-screen-audio-quality.md): capture/source compatibility, live sender-ceiling readback, Opus negotiation boundaries, and remaining audible/device evidence.
- [Display name and presence boundaries](./research/display-name-presence.md): browser-local storage failure behavior, Unicode normalization, control-character policy, and the Web-only presence capability boundary.
docs/research/cross-restart-room-recovery.md
+100
View File
@@ -0,0 +1,100 @@
# Cross-Restart Room Recovery
Status: Research evidence for an unresolved product decision. ADR-0002 continues
to own the accepted behavior: application restart loses every room, credential,
lease, invitation, route, and media authority.
## Current Boundary
The application process owns the complete `RoomStore`, while the Host browser
keeps its raw Host token, creation profile, and best-effort preferred room code.
After restart the old token and Viewer grant cannot authenticate; replaying the
preferred code and profile creates a new room incarnation. LiveKit rooms and the
Screener SFU resource ledger are also generation-scoped and process-resident.
LiveKit resume covers a transient signaling or network interruption while the
same server-side session exists. A full reconnect rebuilds participants, tracks,
publications, and subscriptions. The current deployment additionally deletes
managed LiveKit rooms before admitting a new application generation. Neither
WebRTC nor LiveKit can restore application authorization or route ownership from
surviving media packets.
## Minimal Recoverable Model
The smallest candidate is Host-led lineage recovery, not persistent room-state
reconstruction:
- a random stable `lineageId` identifies the Host-owned room across application
generations;
- every successful creation or recovery allocates a fresh random
`authorityEpoch`;
- the server signs a bounded Host recovery capsule containing the lineage,
Host-token digest, preferred code hint, server-owned expiry, and purpose;
- the Host retains the capsule and raw Host token locally; a Viewer can never
create, recover, reserve, or renew a room;
- recovery atomically verifies site access, capsule, Host token, expiry, code
availability, and replayed policy/password before publishing the new room;
- session, share, connection, route, authorization, and SFU generations are
fenced by the fresh authority epoch. Old asynchronous results fail closed.
The room code remains a best-effort locator. If another lineage already owns it,
recovery must not evict that Host; the recovering lineage receives another free
code or fails according to the future accepted contract. Client clocks, local
lease extensions, roster, graph, first-frame proof, and resource counts never
become server truth.
## Invitation And Revocation Boundary
A signature proves that the server issued a capsule; it does not prove that the
capsule is the newest one. Restoring the old Viewer-grant digest and authorization
generation from a still-valid capsule can replay a grant that the Host rotated or
revoked before the crash. Rejecting that stale checkpoint requires a server-side
latest-generation high-water mark or accepting replay until the capsule expires.
There is no stateless third option.
An old invitation must always bind to its lineage and authorization generation.
It cannot fall through to code-only admission, enter a different lineage that
reused the same four-digit code, or create a missing room. A private room whose
local password material is unavailable must fail closed rather than become open.
The lowest-risk first phase would recover only Host lineage, control, lease
ceiling, and a free preferred code, then issue a new invitation. Stable old
invitations are a separate decision that must settle revocation high-water and
code relocation first.
## Media Boundary
Control recovery can be automatic while the Host tab and capture track remain
alive, but a page or browser restart still requires a new user gesture for screen
capture. Every recovered route must close or fence old P2P/SFU senders, rebuild
the graph and resource ledger, and recommit on a newly decoded frame. Media may
briefly survive a process crash, but it has no current authority and cannot be
advertised as uninterrupted service.
True multi-node LiveKit continuity requires its distributed Redis deployment and
draining model. That is a different infrastructure and availability decision,
not a consequence of Host lineage recovery.
## Decision Needed
1. Is Host lineage/control recovery sufficient, with new invitations and a
bounded media gap?
2. If old invitations must survive, does the product accept bounded stale replay
or one small durable latest-authorization-generation owner?
3. May a recovered lineage move to another room code when its hint is occupied,
and should a lineage-bound invitation resolve that new code?
4. Is a same-browser bearer recovery capsule an acceptable ownership boundary?
5. Are application and LiveKit restart targets both bounded rebuild, or is
LiveKit high availability a separately funded requirement?
## Primary Sources
- [W3C Capability URLs](https://www.w3.org/TR/capability-urls/)
- [RFC 6750 bearer-token threats](https://www.rfc-editor.org/rfc/rfc6750.html#section-5.2)
- [RFC 7519 expiration and token IDs](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1)
- [RFC 9246 signed-URI replay boundary](https://www.rfc-editor.org/rfc/rfc9246.html)
- [Kubernetes names and UIDs](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/)
- [The Chubby lock service](https://storage.googleapis.com/gweb-research2023-media/pubtools/4444.pdf)
- [WebRTC ICE restart](https://www.w3.org/TR/webrtc/#dom-rtcpeerconnection-restartice)
- [LiveKit reconnect behavior](https://docs.livekit.io/intro/basics/connect/#network-changes-and-reconnection)
- [LiveKit distributed deployment and draining](https://docs.livekit.io/transport/self-hosting/distributed/)
docs/research/realtime-quality-adaptation.md
+56 -19
View File
@@ -241,27 +241,45 @@ The Browser candidates therefore have narrower potential than their names imply:
| AV1 | Modern hardware can provide the strongest compression candidate. | Chrome 151 and 153 keep Windows WebRTC AV1 hardware encode disabled by default, so codec support does not provide a usable hardware contract. |
| H.265 | Supported hardware can provide efficient H.264-class low-CPU encode at lower bitrate. | It is an advanced codec not supported by every Browser client; LiveKit compatibility requires regression or a backup publication, and the current cross-Browser relay matrix is absent. |
No candidate currently provides a product-wide Browser improvement with VP8's
coverage. Browser media therefore remains VP8 while one bounded preflight is
open. It reproduces the accepted `motion` capture semantics and compares VP8
with H.264 constrained baseline on exact Chrome 153. The default browser path is
the product evidence; forcing AMD software BRC is a diagnostic arm only.
No candidate yet provides a product-wide Browser improvement with VP8's
coverage, so Browser media remains VP8. A bounded preflight nevertheless proved
that an actual sender can distinguish the previously bad and a newly good H.264
path without naming a GPU or trusting capability advertisement. All samples used
one real tab `getDisplayMedia()` source at 1904x928@30, `contentHint = "motion"`,
a 5 Mbps ceiling, five seconds of settling, and a ten-second measurement window.
The gate covers Host-to-Viewer and Browser-relay-to-child paths with one and two
outbound senders. Codec capability is insufficient: each endpoint must prove the
negotiated codec/profile, encoder and decoder implementation when exposed,
power-efficiency evidence, encoded and decoded cadence, bitrate, resolution,
encode/decode time, quality-limitation reason, and process CPU/GPU video-engine
attribution. This determines whether a P2P connection can select H.264 for an
efficient sender and receiver and otherwise retain VP8, including the relay's
fresh encode cost.
| Browser and path | Actual implementation | Source / encoded / decoded cadence | Result |
| --- | --- | --- | --- |
| Chrome 151 direct H.264 | AMD Media Foundation hardware encoder; D3D11 decoder | 30.0 / 12.5 / 12.4 fps | Failed despite `powerEfficientEncoder = true`; encoded/source ratio 0.42. |
| Chrome 153 direct VP8 | `libvpx` software encoder and decoder | about 29.9 / 29.9 / 29.9 fps | Control sustained source cadence. |
| Chrome 153 direct H.264 | NVIDIA Media Foundation hardware encoder; D3D11 decoder | 29.9 / 29.9 / 30.0 fps | Full-cadence constrained baseline, ratio 1.00. |
| Chrome 153 LiveKit 1.13.5 H.264 | Simulcast adapter with two NVIDIA MFT encoders; D3D11 decoder | 29.5 / 29.1 / 29.1 fps | One `backupCodec: false` HIGH+LOW publication delivered the HIGH stream, ratio 0.99. |
A single SFU publication cannot supply subscriber-specific H.264 and VP8 without
another encoded publication. A VP8-first viewing path followed by background
H.264 confirmation must therefore separately prove its transition behavior and
the temporary dual-encode or republish cost before it can be accepted as
seamless fallback. The spike adds no VP9, AV1, or H.265 runtime probing, codec
selector, product integration, or deployment.
Exact Chrome 153 also sustained about 30 fps on two simultaneous direct H.264
senders. A Host-to-Browser-relay-to-child chain sustained 29.6-29.7 fps at full
source resolution, and both encode stages remained NVIDIA MFT while both decode
stages remained D3D11. The dual-direct sample was still inside native bandwidth
ramp-up and therefore proves encoder cadence, not steady resolution quality.
Process identities changed during startup, so those whole-browser CPU deltas are
not retained as a codec cost claim.
`powerEfficientEncoder` alone cannot be the gate because the cadence-failing AMD
path also reported true. The portable evidence is the actual negotiated H.264
profile and implementation, plus identity-stable source-frame and encoded-frame
progress; the successful cohorts kept their encoded/source deficit bounded while
the bad path dropped frames continuously. The measurement window and acceptable
deficit still require multi-device and real-game calibration before product use.
For direct peers, native SDP negotiation can order H.264 before VP8 in one offer
and choose the first codec supported by both endpoints; it does not require a
VP8 connection and a parallel H.264 connection. A research candidate is one
share-scoped Host sender preflight and one received-source-scoped Browser-relay
preflight: a proved sender prefers H.264 for future edges, otherwise it retains
VP8, and already active edges are not churned. A single SFU publication still
cannot provide per-subscriber H.264 and VP8 without a second encoded publication.
LiveKit backup codec is therefore not accepted, and an H.264 SFU publication
requires a separate decision on the supported Browser decode floor. No VP9,
AV1, H.265, product codec selector, or deployment is accepted by this evidence.
HEVC, AV1, custom WebCodecs pipelines, and application packetization do not
replace the browser WebRTC sender without a new capture, RTP/RTCP, feedback,
@@ -352,6 +370,25 @@ distort comparison. It necessarily permits a temporarily worse candidate and a
second transition on restore; that behavior, the freeze SLO, inconclusive result,
and permission to create or prioritize SFU resources remain product decisions.
The current quality-evidence wire is not yet eligible for that shadow authority.
WebRTC defines freeze and pause from rendered frames, while the current payload
has decoded progress but no rendered-frame or standard pause evidence. It also
caps one recovered `freezeDuration` delta to one report window even though the
cumulative counter can legitimately jump by several windows after rendering
resumes. Presentation eligibility must bind visibility, page lifecycle, local
playback/autoplay, Host pause, current frame proof, and an identity epoch; the
first sample after any epoch change establishes a baseline only.
The minimal active state machine remains `observe -> qualified -> queued ->
trial prepare -> first-frame commit -> probation -> keep | restore once ->
cooldown`. It uses the existing reservation and room-serial child operation,
permits no bounded-gap quality trial, and yields to availability work. Relay
ingress evidence may reparent that relay while retaining its subtree, but one or
several child reports cannot infer that the relay is globally bad. Phase one has
no quality-driven relay-wide abdication. Freeze entry/exit, probation evidence,
improvement margin, cooldown, and room disruption budget must be calibrated from
real annotated sessions; paper and library defaults are not Screener thresholds.
## Current Verification Gaps
- Real games under competing CPU/GPU load at 720p30, 1080p30, and 1080p60.
docs/todo.md
+5 -13
View File
@@ -7,21 +7,12 @@ later topic is not implementation authority by itself.
## Now
1. **Browser codec preflight.** Keep the product on VP8 while an isolated Chrome
153 experiment compares VP8 with H.264 constrained baseline under the current
`motion` capture semantics. Measure Host send, Viewer decode, Browser relay,
one/two outbound senders, and a VP8-first background transition using actual
codec/profile, implementation and power-efficiency evidence when exposed,
cadence, encode/decode time, and CPU/GPU attribution. The default browser path
is product evidence; an AMD forced-SW-BRC arm is diagnostic only. Do not add
product codec selection, VP9/AV1/H.265 probing, a custom encoder, or deploy a
codec change before this gate is accepted.
2. **Mobile Viewer lifecycle.** Run Android Chrome and iOS Safari matrices for
1. **Mobile Viewer lifecycle.** Run Android Chrome and iOS Safari matrices for
autoplay gesture, foreground/background audio, foreground video recovery,
lock/page reclamation, rotation, Wi-Fi/cellular migration, and assigned-relay
survival or controller recovery. Web does not promise background video or
relay execution after OS suspension.
3. **Finish real-network route and media acceptance.** Exercise direct peer,
2. **Finish real-network route and media acceptance.** Exercise direct peer,
Browser relay, SFU, relay-ingress recovery with subtree retention,
disconnect/capacity drain, Pause/Resume, source replacement, screen-audio
continuity, and real-game A/V sync across representative IPv4/IPv6,
@@ -66,8 +57,9 @@ later topic is not implementation authority by itself.
| Item | Decision |
| --- | --- |
| Repository and distribution license | Decide before public release or package distribution. GPL/AGPL implementations remain research-only until then. |
| Cross-restart room identity and media continuity | Decide whether stable invitations and uninterrupted control justify durable or signed room lineage plus exact route/resource recovery. Preferred code/password replay alone remains a new room incarnation. |
| Quality-driven route exploration | Confirm the trigger SLO, whether one post-commit trial plus one restore may temporarily worsen playback, how inconclusive probation settles, and whether quality work may prioritize an existing SFU publication or create a new one. No active route change is authorized until then. |
| Adaptive Browser codec gate | Decide whether to accept one share/source-scoped H.264 sender preflight plus native H.264-to-VP8 negotiation. Chrome 153 passed direct, dual-sender, Browser-relay, and LiveKit paths on the measured NVIDIA/AMD machine while Chrome 151's selected AMD MFT failed. Probe calibration, real-game/60 fps, heterogeneous endpoints, and the single-SFU-publication codec policy remain unaccepted. |
| Cross-restart room identity and media continuity | Decide whether the target stops at Host lineage/control recovery with a fresh authority epoch or also preserves old invitations. A signed Host capsule can restore lineage but cannot combine stable old grants with strong cross-restart rotate/revoke without a small durable latest-generation truth. Media can be rebuilt with a bounded gap, not promised uninterrupted. |
| Quality-driven route exploration | First repair rendered/pause evidence, long recovered-freeze deltas, and presentation eligibility. Then confirm the freeze SLO, whether one post-commit trial plus one best-effort restore may temporarily worsen playback, how inconclusive probation settles, and whether quality work may prioritize an existing SFU publication. No active route change or relay-wide quality abdication is authorized. |
## Evidence Boundaries
scripts/browser-codec-preflight.ts
+384
View File
@@ -0,0 +1,384 @@
import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process";
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
import { createServer, type Server } from "node:http";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { AccessToken, TrackSource } from "livekit-server-sdk";
import {
CdpConnection,
cleanupRun,
createPage,
reservePort,
waitForVersion,
type PageHandle,
} from "./native-one-viewer-gate";
const WIDTH = 1_920;
const HEIGHT = 1_080;
const FRAME_RATE = 30;
const MAX_BITRATE = 5_000_000;
const SETTLE_MS = 5_000;
const SAMPLE_MS = 10_000;
interface RuntimeResult<T> {
result: { value?: T };
exceptionDetails?: {
text?: string;
exception?: { description?: string };
};
}
interface ProcessInfo {
type: string;
id: number;
cpuTime: number;
}
interface ProcessSnapshot {
processInfo: ProcessInfo[];
}
async function evaluate<T>(
cdp: CdpConnection,
page: PageHandle,
expression: string,
timeoutMs: number,
): Promise<T> {
const value = await cdp.call<RuntimeResult<T>>(
"Runtime.evaluate",
{
expression,
awaitPromise: true,
returnByValue: true,
userGesture: true,
},
page.sessionId,
Date.now() + timeoutMs,
);
if (value.exceptionDetails) {
throw new Error(
value.exceptionDetails.exception?.description ??
value.exceptionDetails.text ??
"Browser evaluation failed",
);
}
return value.result.value as T;
}
function sourceHtml(): string {
return `<!doctype html><title>Codec Preflight Source</title>
<style>html,body{margin:0;overflow:hidden;background:#000}canvas{display:block;width:100vw;height:100vh}</style>
<canvas width="${WIDTH}" height="${HEIGHT}"></canvas><script>
const c=document.querySelector('canvas'),x=c.getContext('2d',{alpha:false});let f=0;
function draw(){f+=1;for(let r=0;r<14;r+=1)for(let q=0;q<24;q+=1){
x.fillStyle='hsl('+((f*29+r*71+q*43)%360)+' 80% '+(35+(f+r+q)%30)+'%)';
x.fillRect(q*80,r*78,80,78)}x.fillStyle='#fff';x.fillRect(f*13%1920,0,12,1080);
x.fillStyle='#000';x.font='60px monospace';x.fillText(String(f).padStart(8,'0'),24,72)}
draw();setInterval(draw,1000/${FRAME_RATE});</script>`;
}
async function startServer(port: number): Promise<Server> {
const livekitBundle = await readFile(
resolve("node_modules/livekit-client/dist/livekit-client.umd.js"),
);
const server = createServer((request, response) => {
const path = new URL(request.url ?? "/", `http://127.0.0.1:${port}`).pathname;
if (path === "/livekit.js") {
response.setHeader("content-type", "text/javascript; charset=utf-8");
response.end(livekitBundle);
return;
}
response.setHeader("content-type", "text/html; charset=utf-8");
response.end(path === "/source"
? sourceHtml()
: "<!doctype html><title>Codec Preflight</title><script src='/livekit.js'></script><video autoplay muted playsinline style='width:960px'></video>");
});
await new Promise<void>((resolveListen, rejectListen) => {
server.once("error", rejectListen);
server.listen(port, "127.0.0.1", resolveListen);
});
return server;
}
function browserProbe(): string {
return `(() => {
const WIDTH=${WIDTH},HEIGHT=${HEIGHT},FRAME_RATE=${FRAME_RATE},MAX_BITRATE=${MAX_BITRATE};
const SETTLE_MS=${SETTLE_MS},SAMPLE_MS=${SAMPLE_MS};let stream=null,track=null;
const delay=(ms)=>new Promise((resolve)=>setTimeout(resolve,ms));
async function waitFor(predicate,timeout,label){const deadline=performance.now()+timeout;
while(performance.now()<deadline){if(await predicate())return;await delay(50)}
throw new Error('Timed out waiting for '+label)}
function preferences(codec){const all=RTCRtpSender.getCapabilities('video')?.codecs||[];
const mime=codec==='h264'?'video/h264':'video/vp8';const primary=all.filter((item)=>item.mimeType.toLowerCase()===mime);
if(!primary.length)throw new Error(mime+' unavailable');
const repair=new Set(['video/rtx','video/red','video/ulpfec','video/flexfec-03']);
return [...primary,...all.filter((item)=>repair.has(item.mimeType.toLowerCase()))]}
async function gathered(pc){if(pc.iceGatheringState==='complete')return;
await new Promise((resolve)=>{const changed=()=>{if(pc.iceGatheringState!=='complete')return;
pc.removeEventListener('icegatheringstatechange',changed);resolve()};pc.addEventListener('icegatheringstatechange',changed)})}
async function negotiate(a,b){await a.setLocalDescription(await a.createOffer());await gathered(a);
await b.setRemoteDescription(a.localDescription);await b.setLocalDescription(await b.createAnswer());
await gathered(b);await a.setRemoteDescription(b.localDescription)}
function statsValue(item){if(!item)return null;const keys=['id','timestamp','bytesSent','bytesReceived',
'framesEncoded','framesDecoded','framesDropped','framesPerSecond','frameWidth','frameHeight','totalEncodeTime',
'totalDecodeTime','qualityLimitationReason','qualityLimitationDurations','encoderImplementation',
'decoderImplementation','powerEfficientEncoder','powerEfficientDecoder','frames'];const value={};
for(const key of keys)value[key]=item[key]??null;return value}
async function snapshot(sender,receiver,rendered){const [sendReport,receiveReport]=await Promise.all([sender.getStats(),receiver.getStats()]);
let outbound=null,inbound=null;sendReport.forEach((item)=>{if(item.type==='outbound-rtp'&&item.kind==='video'&&!item.isRemote)outbound=item});
receiveReport.forEach((item)=>{if(item.type==='inbound-rtp'&&item.kind==='video'&&!item.isRemote)inbound=item});
const sendCodec=outbound?.codecId?sendReport.get(outbound.codecId):null;
const receiveCodec=inbound?.codecId?receiveReport.get(inbound.codecId):null;
const mediaSource=outbound?.mediaSourceId?sendReport.get(outbound.mediaSourceId):null;
const codec=(item)=>item?{mimeType:item.mimeType||null,sdpFmtpLine:item.sdpFmtpLine||null}:null;
return {rendered:rendered(),sender:statsValue(outbound),receiver:statsValue(inbound),
mediaSource:statsValue(mediaSource),senderCodec:codec(sendCodec),receiverCodec:codec(receiveCodec)}}
function delta(after,before,key){const a=after?.[key],b=before?.[key];return Number.isFinite(a)&&Number.isFinite(b)&&a>=b?a-b:null}
function summarize(before,after,sampleMs){const encoded=delta(after.sender,before.sender,'framesEncoded');
const decoded=delta(after.receiver,before.receiver,'framesDecoded');const bytes=delta(after.sender,before.sender,'bytesSent');
const sourceFrames=delta(after.mediaSource,before.mediaSource,'frames');
const encodeTime=delta(after.sender,before.sender,'totalEncodeTime');const decodeTime=delta(after.receiver,before.receiver,'totalDecodeTime');
return {senderCodec:after.senderCodec,receiverCodec:after.receiverCodec,sender:after.sender,receiver:after.receiver,
measured:{sampleMs,encodedFps:encoded===null?null:encoded*1000/sampleMs,
decodedFps:decoded===null?null:decoded*1000/sampleMs,bitrateKbps:bytes===null?null:bytes*8/sampleMs,
encodeMsPerFrame:encodeTime!==null&&encoded>0?encodeTime*1000/encoded:null,
decodeMsPerFrame:decodeTime!==null&&decoded>0?decodeTime*1000/decoded:null,
sourceFps:sourceFrames===null?after.mediaSource?.framesPerSecond:sourceFrames*1000/sampleMs,
encodedSourceRatio:sourceFrames>0&&encoded!==null?encoded/sourceFrames:null,
renderedFrames:delta(after,before,'rendered')}}}
async function setup(){stream=await navigator.mediaDevices.getDisplayMedia({video:{
width:{ideal:WIDTH,max:WIDTH},height:{ideal:HEIGHT,max:HEIGHT},frameRate:{ideal:FRAME_RATE,max:FRAME_RATE}},audio:false});
track=stream.getVideoTracks()[0];if(!track)throw new Error('No display video track');track.contentHint='motion';
return {settings:track.getSettings(),constraints:track.getConstraints(),contentHint:track.contentHint,label:track.label}}
function capabilities(){const simple=(codec)=>({mimeType:codec.mimeType,sdpFmtpLine:codec.sdpFmtpLine||null});
return {sender:(RTCRtpSender.getCapabilities('video')?.codecs||[]).map(simple),
receiver:(RTCRtpReceiver.getCapabilities('video')?.codecs||[]).map(simple)}}
async function edge(codec,sourceTrack){
const sendPc=new RTCPeerConnection({iceServers:[]}),receivePc=new RTCPeerConnection({iceServers:[]});
const transceiver=sendPc.addTransceiver(sourceTrack,{direction:'sendonly'});transceiver.setCodecPreferences(preferences(codec));
let receiver=null,remoteTrack=null;receivePc.addEventListener('track',(event)=>{receiver=event.receiver;remoteTrack=event.track});
await negotiate(sendPc,receivePc);await waitFor(()=>sendPc.connectionState==='connected'&&receivePc.connectionState==='connected',10000,'connection');
await waitFor(()=>receiver&&remoteTrack,5000,'remote track');const video=document.createElement('video');let rendered=0;
video.autoplay=true;video.muted=true;video.playsInline=true;document.body.append(video);
video.srcObject=new MediaStream([remoteTrack]);if(video.requestVideoFrameCallback){const frame=()=>{rendered+=1;video.requestVideoFrameCallback(frame)};video.requestVideoFrameCallback(frame)}
await video.play();const parameters=transceiver.sender.getParameters();if(!parameters.encodings.length)parameters.encodings=[{}];
parameters.encodings[0].maxBitrate=MAX_BITRATE;parameters.encodings[0].maxFramerate=FRAME_RATE;
parameters.encodings[0].scaleResolutionDownBy=1;parameters.degradationPreference='balanced';await transceiver.sender.setParameters(parameters);
await waitFor(async()=>{const report=await receiver.getStats();let frames=0;report.forEach((item)=>{if(item.type==='inbound-rtp'&&item.kind==='video')frames=Math.max(frames,item.framesDecoded||0)});return frames>0&&rendered>0},10000,'decoded frame');
return {sendPc,receivePc,sender:transceiver.sender,receiver,remoteTrack,rendered:()=>rendered}}
async function run(codec,count,topology){if(!track)throw new Error('Capture not initialized');const edges=[];
try{edges.push(await edge(codec,track));
if(topology==='relay')edges.push(await edge(codec,edges[0].remoteTrack));
else for(let index=1;index<count;index+=1)edges.push(await edge(codec,track));await delay(SETTLE_MS);
const before=await Promise.all(edges.map((item)=>snapshot(item.sender,item.receiver,item.rendered))),started=performance.now();
await delay(SAMPLE_MS);const sampleMs=performance.now()-started;
const after=await Promise.all(edges.map((item)=>snapshot(item.sender,item.receiver,item.rendered)));
return edges.map((_,index)=>summarize(before[index],after[index],sampleMs))}
finally{for(const item of edges){item.sendPc.close();item.receivePc.close()}await delay(300)}}
async function runLivekit(codec,url,publisherToken,subscriberToken){if(!track)throw new Error('Capture not initialized');
const sdk=globalThis.LivekitClient,publisher=new sdk.Room({dynacast:true}),subscriber=new sdk.Room();let remoteTrack=null,rendered=0;
subscriber.on(sdk.RoomEvent.TrackSubscribed,(nextTrack)=>{if(nextTrack.kind==='video')remoteTrack=nextTrack});
try{await publisher.connect(url,publisherToken,{autoSubscribe:false,rtcConfig:{iceServers:[]}});
await subscriber.connect(url,subscriberToken,{rtcConfig:{iceServers:[]}});
const publication=await publisher.localParticipant.publishTrack(track,{source:sdk.Track.Source.ScreenShare,
videoCodec:codec,backupCodec:false,screenShareEncoding:{maxBitrate:MAX_BITRATE,maxFramerate:FRAME_RATE},
degradationPreference:'balanced'});
await waitFor(()=>publication.videoTrack?.sender&&remoteTrack,15000,'LiveKit publication');
const video=document.createElement('video');video.autoplay=true;video.muted=true;video.playsInline=true;document.body.append(video);
video.srcObject=new MediaStream([remoteTrack.mediaStreamTrack]);if(video.requestVideoFrameCallback){const frame=()=>{rendered+=1;video.requestVideoFrameCallback(frame)};video.requestVideoFrameCallback(frame)}
await video.play();const receiver={getStats:async()=>{const report=await remoteTrack.getRTCStatsReport();if(!report)throw new Error('LiveKit receiver stats unavailable');return report}};
await waitFor(async()=>{const report=await receiver.getStats();let frames=0;report?.forEach((item)=>{if(item.type==='inbound-rtp'&&item.kind==='video')frames=Math.max(frames,item.framesDecoded||0)});return frames>0&&rendered>0},15000,'LiveKit decoded frame');
await delay(SETTLE_MS);const sender=publication.videoTrack.sender;
const before=await snapshot(sender,receiver,()=>rendered),started=performance.now();await delay(SAMPLE_MS);
const sampleMs=performance.now()-started,after=await snapshot(sender,receiver,()=>rendered);
return {...summarize(before,after,sampleMs),senderParameters:sender.getParameters()}}
finally{await Promise.allSettled([publisher.disconnect(false),subscriber.disconnect(false)]);await delay(300)}}
function stop(){stream?.getTracks().forEach((item)=>item.stop())}
Object.defineProperty(globalThis,'__CODEC_PREFLIGHT__',{value:{setup,capabilities,run,runLivekit,stop}});
})()`;
}
async function processSnapshot(cdp: CdpConnection): Promise<ProcessSnapshot> {
return cdp.call<ProcessSnapshot>(
"SystemInfo.getProcessInfo",
{},
undefined,
Date.now() + 5_000,
);
}
function processCpu(
before: ProcessSnapshot,
after: ProcessSnapshot,
wallMs: number,
): Record<string, unknown> {
const baseline = new Map(
before.processInfo.map((item) => [`${item.type}:${item.id}`, item.cpuTime]),
);
let cpuSeconds = 0;
let stableProcesses = 0;
for (const item of after.processInfo) {
const previous = baseline.get(`${item.type}:${item.id}`);
if (previous === undefined || item.cpuTime < previous) continue;
cpuSeconds += item.cpuTime - previous;
stableProcesses += 1;
}
const wallSeconds = wallMs / 1_000;
return {
wallSeconds,
cpuSeconds,
logicalCoreEquivalent: cpuSeconds / wallSeconds,
stableProcesses,
processIdentityChanged:
stableProcesses !== before.processInfo.length ||
stableProcesses !== after.processInfo.length,
};
}
async function main(): Promise<void> {
const chromePath = process.env.CHROME_PATH?.trim();
if (!chromePath) throw new Error("CHROME_PATH must point to Chrome");
const codec = process.env.CODEC_PREFLIGHT_CODEC?.trim().toLowerCase();
if (codec !== "vp8" && codec !== "h264") {
throw new Error("CODEC_PREFLIGHT_CODEC must be vp8 or h264");
}
const edges = Number(process.env.CODEC_PREFLIGHT_EDGES ?? "1");
if (edges !== 1 && edges !== 2) {
throw new Error("CODEC_PREFLIGHT_EDGES must be 1 or 2");
}
const topology = process.env.CODEC_PREFLIGHT_TOPOLOGY?.trim() || "direct";
if (topology !== "direct" && topology !== "relay" && topology !== "livekit") {
throw new Error("CODEC_PREFLIGHT_TOPOLOGY must be direct, relay, or livekit");
}
const [debugPort, pagePort] = await Promise.all([reservePort(), reservePort()]);
const profile = await mkdtemp(join(tmpdir(), "screener-codec-preflight-"));
let chrome: ChildProcessWithoutNullStreams | null = null;
let cdp: CdpConnection | null = null;
let server: Server | null = null;
try {
server = await startServer(pagePort);
chrome = spawn(chromePath, [
`--remote-debugging-port=${debugPort}`,
`--user-data-dir=${profile}`,
"--no-first-run",
"--no-default-browser-check",
"--disable-extensions",
"--disable-background-timer-throttling",
"--disable-backgrounding-occluded-windows",
"--disable-renderer-backgrounding",
"--autoplay-policy=no-user-gesture-required",
"--auto-select-tab-capture-source-by-title=Codec Preflight Source",
"--window-size=1920,1080",
"about:blank",
], { stdio: "pipe", windowsHide: true });
chrome.stdout.resume();
chrome.stderr.resume();
const version = await waitForVersion(debugPort, chrome);
cdp = await CdpConnection.connect(version.webSocketDebuggerUrl, Date.now() + 5_000);
const gpuInfo = await cdp.call<unknown>("SystemInfo.getInfo", {}, undefined, Date.now() + 5_000);
await createPage(cdp, `http://127.0.0.1:${pagePort}/source`, undefined, true);
const page = await createPage(cdp, `http://127.0.0.1:${pagePort}/test`, undefined, true);
await evaluate<void>(cdp, page, browserProbe(), 5_000);
const capabilities = await evaluate<unknown>(
cdp,
page,
"globalThis.__CODEC_PREFLIGHT__.capabilities()",
5_000,
);
const capture = await evaluate<unknown>(
cdp,
page,
"globalThis.__CODEC_PREFLIGHT__.setup()",
15_000,
);
const before = await processSnapshot(cdp);
const started = performance.now();
let media: unknown;
if (topology === "livekit") {
const livekitUrl = process.env.CODEC_PREFLIGHT_LIVEKIT_URL?.trim();
const apiKey = process.env.CODEC_PREFLIGHT_LIVEKIT_KEY?.trim();
const apiSecret = process.env.CODEC_PREFLIGHT_LIVEKIT_SECRET?.trim();
if (!livekitUrl || !apiKey || !apiSecret) {
throw new Error("LiveKit preflight credentials are missing");
}
const room = `codec-preflight-${Date.now()}`;
const token = async (identity: string, publish: boolean) => {
const access = new AccessToken(apiKey, apiSecret, { identity, ttl: 60 });
access.addGrant({
roomJoin: true,
room,
canPublish: publish,
canPublishSources: publish ? [TrackSource.SCREEN_SHARE] : undefined,
canSubscribe: !publish,
canPublishData: false,
});
return access.toJwt();
};
const [publisherToken, subscriberToken] = await Promise.all([
token("host", true),
token("viewer", false),
]);
media = await evaluate<unknown>(
cdp,
page,
`globalThis.__CODEC_PREFLIGHT__.runLivekit(${JSON.stringify(codec)}, ${JSON.stringify(livekitUrl)}, ${JSON.stringify(publisherToken)}, ${JSON.stringify(subscriberToken)})`,
60_000,
);
} else {
media = await evaluate<unknown>(
cdp,
page,
`globalThis.__CODEC_PREFLIGHT__.run(${JSON.stringify(codec)}, ${edges}, ${JSON.stringify(topology)})`,
45_000,
);
}
const wallMs = performance.now() - started;
const after = await processSnapshot(cdp);
await evaluate<void>(cdp, page, "globalThis.__CODEC_PREFLIGHT__.stop()", 5_000);
const report = {
schemaVersion: 1,
browser: version.Browser,
configuration: {
width: WIDTH,
height: HEIGHT,
frameRate: FRAME_RATE,
maxBitrate: MAX_BITRATE,
contentHint: "motion",
settleMs: SETTLE_MS,
sampleMs: SAMPLE_MS,
edges,
topology,
},
gpuInfo,
capabilities,
capture,
codec,
media,
cpu: processCpu(before, after, wallMs),
};
const outputDirectory = resolve("benchmark-results");
await mkdir(outputDirectory, { recursive: true });
const output = join(
outputDirectory,
`browser-codec-preflight-${codec}-${topology}-${edges}.json`,
);
await writeFile(output, `${JSON.stringify(report, null, 2)}\n`, "utf8");
console.log(output);
} finally {
await cleanupRun({
cdp,
native: null,
chrome,
server: null,
profile,
ports: [debugPort],
});
if (server) {
await new Promise<void>((resolveClose) => server!.close(() => resolveClose()));
}
}
}
await main();
tsconfig.tools.json
+1
View File
@@ -21,6 +21,7 @@
"vitest.config.ts",
"scripts/native-one-viewer-gate.ts",
"scripts/native-one-viewer-ledger.ts",
"scripts/browser-codec-preflight.ts",
"tests"
]
}