diff --git a/docs/README.md b/docs/README.md index f0c146b..f8c1190 100644 --- a/docs/README.md +++ b/docs/README.md @@ -17,7 +17,8 @@ - [Native shared-encode sender](./research/native-shared-encode-sender.md): consolidated Draft #16/#18/#22/#23/#25/#28 ladder, unclassified product-gate failure, staged revalidation, and stop line. - [Native H.264 hardware decision](./research/native-h264-hardware-decision.md): bounded WebCodecs no-go, proved Media Foundation/NVIDIA hardware encode, the default-fmtp `42c01f` stop, and the retained interop boundary. - [Native H.264 opt-in path](./research/native-h264-opt-in-path.md): explicit H.264 sender mode, exact Pion profile registration, and one bounded Host/Viewer loopback with hardware caveats. -- [Realtime quality adaptation](./research/realtime-quality-adaptation.md): fixed Browser VP8/no-video-hint evidence, manual quality ceilings, diagnostic-only capture/send/receive correlation, and LiveKit-owned SFU dual-representation adaptation evidence. +- [Realtime quality adaptation](./research/realtime-quality-adaptation.md): Browser codec and `motion` evidence, manual quality ceilings, path-quality authority, and LiveKit-owned SFU representation adaptation. +- [Cross-restart room recovery](./research/cross-restart-room-recovery.md): pending lineage, authority-epoch, invitation-replay, and bounded media-rebuild evidence without reopening durable room storage. - [Browser background capture diagnostics](./research/browser-background-capture.md): browser lifecycle evidence, a privacy-bounded capture/send/receive experiment, interpretation gates, and non-goal keepalive mechanisms. - [Browser screen-audio quality](./research/browser-screen-audio-quality.md): capture/source compatibility, live sender-ceiling readback, Opus negotiation boundaries, and remaining audible/device evidence. - [Display name and presence boundaries](./research/display-name-presence.md): browser-local storage failure behavior, Unicode normalization, control-character policy, and the Web-only presence capability boundary. diff --git a/docs/research/cross-restart-room-recovery.md b/docs/research/cross-restart-room-recovery.md new file mode 100644 index 0000000..5531ab1 --- /dev/null +++ b/docs/research/cross-restart-room-recovery.md @@ -0,0 +1,100 @@ +# Cross-Restart Room Recovery + +Status: Research evidence for an unresolved product decision. ADR-0002 continues +to own the accepted behavior: application restart loses every room, credential, +lease, invitation, route, and media authority. + +## Current Boundary + +The application process owns the complete `RoomStore`, while the Host browser +keeps its raw Host token, creation profile, and best-effort preferred room code. +After restart the old token and Viewer grant cannot authenticate; replaying the +preferred code and profile creates a new room incarnation. LiveKit rooms and the +Screener SFU resource ledger are also generation-scoped and process-resident. + +LiveKit resume covers a transient signaling or network interruption while the +same server-side session exists. A full reconnect rebuilds participants, tracks, +publications, and subscriptions. The current deployment additionally deletes +managed LiveKit rooms before admitting a new application generation. Neither +WebRTC nor LiveKit can restore application authorization or route ownership from +surviving media packets. + +## Minimal Recoverable Model + +The smallest candidate is Host-led lineage recovery, not persistent room-state +reconstruction: + +- a random stable `lineageId` identifies the Host-owned room across application + generations; +- every successful creation or recovery allocates a fresh random + `authorityEpoch`; +- the server signs a bounded Host recovery capsule containing the lineage, + Host-token digest, preferred code hint, server-owned expiry, and purpose; +- the Host retains the capsule and raw Host token locally; a Viewer can never + create, recover, reserve, or renew a room; +- recovery atomically verifies site access, capsule, Host token, expiry, code + availability, and replayed policy/password before publishing the new room; +- session, share, connection, route, authorization, and SFU generations are + fenced by the fresh authority epoch. Old asynchronous results fail closed. + +The room code remains a best-effort locator. If another lineage already owns it, +recovery must not evict that Host; the recovering lineage receives another free +code or fails according to the future accepted contract. Client clocks, local +lease extensions, roster, graph, first-frame proof, and resource counts never +become server truth. + +## Invitation And Revocation Boundary + +A signature proves that the server issued a capsule; it does not prove that the +capsule is the newest one. Restoring the old Viewer-grant digest and authorization +generation from a still-valid capsule can replay a grant that the Host rotated or +revoked before the crash. Rejecting that stale checkpoint requires a server-side +latest-generation high-water mark or accepting replay until the capsule expires. +There is no stateless third option. + +An old invitation must always bind to its lineage and authorization generation. +It cannot fall through to code-only admission, enter a different lineage that +reused the same four-digit code, or create a missing room. A private room whose +local password material is unavailable must fail closed rather than become open. + +The lowest-risk first phase would recover only Host lineage, control, lease +ceiling, and a free preferred code, then issue a new invitation. Stable old +invitations are a separate decision that must settle revocation high-water and +code relocation first. + +## Media Boundary + +Control recovery can be automatic while the Host tab and capture track remain +alive, but a page or browser restart still requires a new user gesture for screen +capture. Every recovered route must close or fence old P2P/SFU senders, rebuild +the graph and resource ledger, and recommit on a newly decoded frame. Media may +briefly survive a process crash, but it has no current authority and cannot be +advertised as uninterrupted service. + +True multi-node LiveKit continuity requires its distributed Redis deployment and +draining model. That is a different infrastructure and availability decision, +not a consequence of Host lineage recovery. + +## Decision Needed + +1. Is Host lineage/control recovery sufficient, with new invitations and a + bounded media gap? +2. If old invitations must survive, does the product accept bounded stale replay + or one small durable latest-authorization-generation owner? +3. May a recovered lineage move to another room code when its hint is occupied, + and should a lineage-bound invitation resolve that new code? +4. Is a same-browser bearer recovery capsule an acceptable ownership boundary? +5. Are application and LiveKit restart targets both bounded rebuild, or is + LiveKit high availability a separately funded requirement? + +## Primary Sources + +- [W3C Capability URLs](https://www.w3.org/TR/capability-urls/) +- [RFC 6750 bearer-token threats](https://www.rfc-editor.org/rfc/rfc6750.html#section-5.2) +- [RFC 7519 expiration and token IDs](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1) +- [RFC 9246 signed-URI replay boundary](https://www.rfc-editor.org/rfc/rfc9246.html) +- [Kubernetes names and UIDs](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/) +- [The Chubby lock service](https://storage.googleapis.com/gweb-research2023-media/pubtools/4444.pdf) +- [WebRTC ICE restart](https://www.w3.org/TR/webrtc/#dom-rtcpeerconnection-restartice) +- [LiveKit reconnect behavior](https://docs.livekit.io/intro/basics/connect/#network-changes-and-reconnection) +- [LiveKit distributed deployment and draining](https://docs.livekit.io/transport/self-hosting/distributed/) diff --git a/docs/research/realtime-quality-adaptation.md b/docs/research/realtime-quality-adaptation.md index 14e693b..8620892 100644 --- a/docs/research/realtime-quality-adaptation.md +++ b/docs/research/realtime-quality-adaptation.md @@ -241,27 +241,45 @@ The Browser candidates therefore have narrower potential than their names imply: | AV1 | Modern hardware can provide the strongest compression candidate. | Chrome 151 and 153 keep Windows WebRTC AV1 hardware encode disabled by default, so codec support does not provide a usable hardware contract. | | H.265 | Supported hardware can provide efficient H.264-class low-CPU encode at lower bitrate. | It is an advanced codec not supported by every Browser client; LiveKit compatibility requires regression or a backup publication, and the current cross-Browser relay matrix is absent. | -No candidate currently provides a product-wide Browser improvement with VP8's -coverage. Browser media therefore remains VP8 while one bounded preflight is -open. It reproduces the accepted `motion` capture semantics and compares VP8 -with H.264 constrained baseline on exact Chrome 153. The default browser path is -the product evidence; forcing AMD software BRC is a diagnostic arm only. +No candidate yet provides a product-wide Browser improvement with VP8's +coverage, so Browser media remains VP8. A bounded preflight nevertheless proved +that an actual sender can distinguish the previously bad and a newly good H.264 +path without naming a GPU or trusting capability advertisement. All samples used +one real tab `getDisplayMedia()` source at 1904x928@30, `contentHint = "motion"`, +a 5 Mbps ceiling, five seconds of settling, and a ten-second measurement window. -The gate covers Host-to-Viewer and Browser-relay-to-child paths with one and two -outbound senders. Codec capability is insufficient: each endpoint must prove the -negotiated codec/profile, encoder and decoder implementation when exposed, -power-efficiency evidence, encoded and decoded cadence, bitrate, resolution, -encode/decode time, quality-limitation reason, and process CPU/GPU video-engine -attribution. This determines whether a P2P connection can select H.264 for an -efficient sender and receiver and otherwise retain VP8, including the relay's -fresh encode cost. +| Browser and path | Actual implementation | Source / encoded / decoded cadence | Result | +| --- | --- | --- | --- | +| Chrome 151 direct H.264 | AMD Media Foundation hardware encoder; D3D11 decoder | 30.0 / 12.5 / 12.4 fps | Failed despite `powerEfficientEncoder = true`; encoded/source ratio 0.42. | +| Chrome 153 direct VP8 | `libvpx` software encoder and decoder | about 29.9 / 29.9 / 29.9 fps | Control sustained source cadence. | +| Chrome 153 direct H.264 | NVIDIA Media Foundation hardware encoder; D3D11 decoder | 29.9 / 29.9 / 30.0 fps | Full-cadence constrained baseline, ratio 1.00. | +| Chrome 153 LiveKit 1.13.5 H.264 | Simulcast adapter with two NVIDIA MFT encoders; D3D11 decoder | 29.5 / 29.1 / 29.1 fps | One `backupCodec: false` HIGH+LOW publication delivered the HIGH stream, ratio 0.99. | -A single SFU publication cannot supply subscriber-specific H.264 and VP8 without -another encoded publication. A VP8-first viewing path followed by background -H.264 confirmation must therefore separately prove its transition behavior and -the temporary dual-encode or republish cost before it can be accepted as -seamless fallback. The spike adds no VP9, AV1, or H.265 runtime probing, codec -selector, product integration, or deployment. +Exact Chrome 153 also sustained about 30 fps on two simultaneous direct H.264 +senders. A Host-to-Browser-relay-to-child chain sustained 29.6-29.7 fps at full +source resolution, and both encode stages remained NVIDIA MFT while both decode +stages remained D3D11. The dual-direct sample was still inside native bandwidth +ramp-up and therefore proves encoder cadence, not steady resolution quality. +Process identities changed during startup, so those whole-browser CPU deltas are +not retained as a codec cost claim. + +`powerEfficientEncoder` alone cannot be the gate because the cadence-failing AMD +path also reported true. The portable evidence is the actual negotiated H.264 +profile and implementation, plus identity-stable source-frame and encoded-frame +progress; the successful cohorts kept their encoded/source deficit bounded while +the bad path dropped frames continuously. The measurement window and acceptable +deficit still require multi-device and real-game calibration before product use. + +For direct peers, native SDP negotiation can order H.264 before VP8 in one offer +and choose the first codec supported by both endpoints; it does not require a +VP8 connection and a parallel H.264 connection. A research candidate is one +share-scoped Host sender preflight and one received-source-scoped Browser-relay +preflight: a proved sender prefers H.264 for future edges, otherwise it retains +VP8, and already active edges are not churned. A single SFU publication still +cannot provide per-subscriber H.264 and VP8 without a second encoded publication. +LiveKit backup codec is therefore not accepted, and an H.264 SFU publication +requires a separate decision on the supported Browser decode floor. No VP9, +AV1, H.265, product codec selector, or deployment is accepted by this evidence. HEVC, AV1, custom WebCodecs pipelines, and application packetization do not replace the browser WebRTC sender without a new capture, RTP/RTCP, feedback, @@ -352,6 +370,25 @@ distort comparison. It necessarily permits a temporarily worse candidate and a second transition on restore; that behavior, the freeze SLO, inconclusive result, and permission to create or prioritize SFU resources remain product decisions. +The current quality-evidence wire is not yet eligible for that shadow authority. +WebRTC defines freeze and pause from rendered frames, while the current payload +has decoded progress but no rendered-frame or standard pause evidence. It also +caps one recovered `freezeDuration` delta to one report window even though the +cumulative counter can legitimately jump by several windows after rendering +resumes. Presentation eligibility must bind visibility, page lifecycle, local +playback/autoplay, Host pause, current frame proof, and an identity epoch; the +first sample after any epoch change establishes a baseline only. + +The minimal active state machine remains `observe -> qualified -> queued -> +trial prepare -> first-frame commit -> probation -> keep | restore once -> +cooldown`. It uses the existing reservation and room-serial child operation, +permits no bounded-gap quality trial, and yields to availability work. Relay +ingress evidence may reparent that relay while retaining its subtree, but one or +several child reports cannot infer that the relay is globally bad. Phase one has +no quality-driven relay-wide abdication. Freeze entry/exit, probation evidence, +improvement margin, cooldown, and room disruption budget must be calibrated from +real annotated sessions; paper and library defaults are not Screener thresholds. + ## Current Verification Gaps - Real games under competing CPU/GPU load at 720p30, 1080p30, and 1080p60. diff --git a/docs/todo.md b/docs/todo.md index 5b36029..5686b8a 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -7,21 +7,12 @@ later topic is not implementation authority by itself. ## Now -1. **Browser codec preflight.** Keep the product on VP8 while an isolated Chrome - 153 experiment compares VP8 with H.264 constrained baseline under the current - `motion` capture semantics. Measure Host send, Viewer decode, Browser relay, - one/two outbound senders, and a VP8-first background transition using actual - codec/profile, implementation and power-efficiency evidence when exposed, - cadence, encode/decode time, and CPU/GPU attribution. The default browser path - is product evidence; an AMD forced-SW-BRC arm is diagnostic only. Do not add - product codec selection, VP9/AV1/H.265 probing, a custom encoder, or deploy a - codec change before this gate is accepted. -2. **Mobile Viewer lifecycle.** Run Android Chrome and iOS Safari matrices for +1. **Mobile Viewer lifecycle.** Run Android Chrome and iOS Safari matrices for autoplay gesture, foreground/background audio, foreground video recovery, lock/page reclamation, rotation, Wi-Fi/cellular migration, and assigned-relay survival or controller recovery. Web does not promise background video or relay execution after OS suspension. -3. **Finish real-network route and media acceptance.** Exercise direct peer, +2. **Finish real-network route and media acceptance.** Exercise direct peer, Browser relay, SFU, relay-ingress recovery with subtree retention, disconnect/capacity drain, Pause/Resume, source replacement, screen-audio continuity, and real-game A/V sync across representative IPv4/IPv6, @@ -66,8 +57,9 @@ later topic is not implementation authority by itself. | Item | Decision | | --- | --- | | Repository and distribution license | Decide before public release or package distribution. GPL/AGPL implementations remain research-only until then. | -| Cross-restart room identity and media continuity | Decide whether stable invitations and uninterrupted control justify durable or signed room lineage plus exact route/resource recovery. Preferred code/password replay alone remains a new room incarnation. | -| Quality-driven route exploration | Confirm the trigger SLO, whether one post-commit trial plus one restore may temporarily worsen playback, how inconclusive probation settles, and whether quality work may prioritize an existing SFU publication or create a new one. No active route change is authorized until then. | +| Adaptive Browser codec gate | Decide whether to accept one share/source-scoped H.264 sender preflight plus native H.264-to-VP8 negotiation. Chrome 153 passed direct, dual-sender, Browser-relay, and LiveKit paths on the measured NVIDIA/AMD machine while Chrome 151's selected AMD MFT failed. Probe calibration, real-game/60 fps, heterogeneous endpoints, and the single-SFU-publication codec policy remain unaccepted. | +| Cross-restart room identity and media continuity | Decide whether the target stops at Host lineage/control recovery with a fresh authority epoch or also preserves old invitations. A signed Host capsule can restore lineage but cannot combine stable old grants with strong cross-restart rotate/revoke without a small durable latest-generation truth. Media can be rebuilt with a bounded gap, not promised uninterrupted. | +| Quality-driven route exploration | First repair rendered/pause evidence, long recovered-freeze deltas, and presentation eligibility. Then confirm the freeze SLO, whether one post-commit trial plus one best-effort restore may temporarily worsen playback, how inconclusive probation settles, and whether quality work may prioritize an existing SFU publication. No active route change or relay-wide quality abdication is authorized. | ## Evidence Boundaries diff --git a/scripts/browser-codec-preflight.ts b/scripts/browser-codec-preflight.ts new file mode 100644 index 0000000..6cc2c34 --- /dev/null +++ b/scripts/browser-codec-preflight.ts @@ -0,0 +1,384 @@ +import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process"; +import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { createServer, type Server } from "node:http"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { AccessToken, TrackSource } from "livekit-server-sdk"; + +import { + CdpConnection, + cleanupRun, + createPage, + reservePort, + waitForVersion, + type PageHandle, +} from "./native-one-viewer-gate"; + +const WIDTH = 1_920; +const HEIGHT = 1_080; +const FRAME_RATE = 30; +const MAX_BITRATE = 5_000_000; +const SETTLE_MS = 5_000; +const SAMPLE_MS = 10_000; + +interface RuntimeResult { + result: { value?: T }; + exceptionDetails?: { + text?: string; + exception?: { description?: string }; + }; +} + +interface ProcessInfo { + type: string; + id: number; + cpuTime: number; +} + +interface ProcessSnapshot { + processInfo: ProcessInfo[]; +} + +async function evaluate( + cdp: CdpConnection, + page: PageHandle, + expression: string, + timeoutMs: number, +): Promise { + const value = await cdp.call>( + "Runtime.evaluate", + { + expression, + awaitPromise: true, + returnByValue: true, + userGesture: true, + }, + page.sessionId, + Date.now() + timeoutMs, + ); + if (value.exceptionDetails) { + throw new Error( + value.exceptionDetails.exception?.description ?? + value.exceptionDetails.text ?? + "Browser evaluation failed", + ); + } + return value.result.value as T; +} + +function sourceHtml(): string { + return `Codec Preflight Source + +`; +} + +async function startServer(port: number): Promise { + const livekitBundle = await readFile( + resolve("node_modules/livekit-client/dist/livekit-client.umd.js"), + ); + const server = createServer((request, response) => { + const path = new URL(request.url ?? "/", `http://127.0.0.1:${port}`).pathname; + if (path === "/livekit.js") { + response.setHeader("content-type", "text/javascript; charset=utf-8"); + response.end(livekitBundle); + return; + } + response.setHeader("content-type", "text/html; charset=utf-8"); + response.end(path === "/source" + ? sourceHtml() + : "Codec Preflight"); + }); + await new Promise((resolveListen, rejectListen) => { + server.once("error", rejectListen); + server.listen(port, "127.0.0.1", resolveListen); + }); + return server; +} + +function browserProbe(): string { + return `(() => { + const WIDTH=${WIDTH},HEIGHT=${HEIGHT},FRAME_RATE=${FRAME_RATE},MAX_BITRATE=${MAX_BITRATE}; + const SETTLE_MS=${SETTLE_MS},SAMPLE_MS=${SAMPLE_MS};let stream=null,track=null; + const delay=(ms)=>new Promise((resolve)=>setTimeout(resolve,ms)); + async function waitFor(predicate,timeout,label){const deadline=performance.now()+timeout; + while(performance.now()item.mimeType.toLowerCase()===mime); + if(!primary.length)throw new Error(mime+' unavailable'); + const repair=new Set(['video/rtx','video/red','video/ulpfec','video/flexfec-03']); + return [...primary,...all.filter((item)=>repair.has(item.mimeType.toLowerCase()))]} + async function gathered(pc){if(pc.iceGatheringState==='complete')return; + await new Promise((resolve)=>{const changed=()=>{if(pc.iceGatheringState!=='complete')return; + pc.removeEventListener('icegatheringstatechange',changed);resolve()};pc.addEventListener('icegatheringstatechange',changed)})} + async function negotiate(a,b){await a.setLocalDescription(await a.createOffer());await gathered(a); + await b.setRemoteDescription(a.localDescription);await b.setLocalDescription(await b.createAnswer()); + await gathered(b);await a.setRemoteDescription(b.localDescription)} + function statsValue(item){if(!item)return null;const keys=['id','timestamp','bytesSent','bytesReceived', + 'framesEncoded','framesDecoded','framesDropped','framesPerSecond','frameWidth','frameHeight','totalEncodeTime', + 'totalDecodeTime','qualityLimitationReason','qualityLimitationDurations','encoderImplementation', + 'decoderImplementation','powerEfficientEncoder','powerEfficientDecoder','frames'];const value={}; + for(const key of keys)value[key]=item[key]??null;return value} + async function snapshot(sender,receiver,rendered){const [sendReport,receiveReport]=await Promise.all([sender.getStats(),receiver.getStats()]); + let outbound=null,inbound=null;sendReport.forEach((item)=>{if(item.type==='outbound-rtp'&&item.kind==='video'&&!item.isRemote)outbound=item}); + receiveReport.forEach((item)=>{if(item.type==='inbound-rtp'&&item.kind==='video'&&!item.isRemote)inbound=item}); + const sendCodec=outbound?.codecId?sendReport.get(outbound.codecId):null; + const receiveCodec=inbound?.codecId?receiveReport.get(inbound.codecId):null; + const mediaSource=outbound?.mediaSourceId?sendReport.get(outbound.mediaSourceId):null; + const codec=(item)=>item?{mimeType:item.mimeType||null,sdpFmtpLine:item.sdpFmtpLine||null}:null; + return {rendered:rendered(),sender:statsValue(outbound),receiver:statsValue(inbound), + mediaSource:statsValue(mediaSource),senderCodec:codec(sendCodec),receiverCodec:codec(receiveCodec)}} + function delta(after,before,key){const a=after?.[key],b=before?.[key];return Number.isFinite(a)&&Number.isFinite(b)&&a>=b?a-b:null} + function summarize(before,after,sampleMs){const encoded=delta(after.sender,before.sender,'framesEncoded'); + const decoded=delta(after.receiver,before.receiver,'framesDecoded');const bytes=delta(after.sender,before.sender,'bytesSent'); + const sourceFrames=delta(after.mediaSource,before.mediaSource,'frames'); + const encodeTime=delta(after.sender,before.sender,'totalEncodeTime');const decodeTime=delta(after.receiver,before.receiver,'totalDecodeTime'); + return {senderCodec:after.senderCodec,receiverCodec:after.receiverCodec,sender:after.sender,receiver:after.receiver, + measured:{sampleMs,encodedFps:encoded===null?null:encoded*1000/sampleMs, + decodedFps:decoded===null?null:decoded*1000/sampleMs,bitrateKbps:bytes===null?null:bytes*8/sampleMs, + encodeMsPerFrame:encodeTime!==null&&encoded>0?encodeTime*1000/encoded:null, + decodeMsPerFrame:decodeTime!==null&&decoded>0?decodeTime*1000/decoded:null, + sourceFps:sourceFrames===null?after.mediaSource?.framesPerSecond:sourceFrames*1000/sampleMs, + encodedSourceRatio:sourceFrames>0&&encoded!==null?encoded/sourceFrames:null, + renderedFrames:delta(after,before,'rendered')}}} + async function setup(){stream=await navigator.mediaDevices.getDisplayMedia({video:{ + width:{ideal:WIDTH,max:WIDTH},height:{ideal:HEIGHT,max:HEIGHT},frameRate:{ideal:FRAME_RATE,max:FRAME_RATE}},audio:false}); + track=stream.getVideoTracks()[0];if(!track)throw new Error('No display video track');track.contentHint='motion'; + return {settings:track.getSettings(),constraints:track.getConstraints(),contentHint:track.contentHint,label:track.label}} + function capabilities(){const simple=(codec)=>({mimeType:codec.mimeType,sdpFmtpLine:codec.sdpFmtpLine||null}); + return {sender:(RTCRtpSender.getCapabilities('video')?.codecs||[]).map(simple), + receiver:(RTCRtpReceiver.getCapabilities('video')?.codecs||[]).map(simple)}} + async function edge(codec,sourceTrack){ + const sendPc=new RTCPeerConnection({iceServers:[]}),receivePc=new RTCPeerConnection({iceServers:[]}); + const transceiver=sendPc.addTransceiver(sourceTrack,{direction:'sendonly'});transceiver.setCodecPreferences(preferences(codec)); + let receiver=null,remoteTrack=null;receivePc.addEventListener('track',(event)=>{receiver=event.receiver;remoteTrack=event.track}); + await negotiate(sendPc,receivePc);await waitFor(()=>sendPc.connectionState==='connected'&&receivePc.connectionState==='connected',10000,'connection'); + await waitFor(()=>receiver&&remoteTrack,5000,'remote track');const video=document.createElement('video');let rendered=0; + video.autoplay=true;video.muted=true;video.playsInline=true;document.body.append(video); + video.srcObject=new MediaStream([remoteTrack]);if(video.requestVideoFrameCallback){const frame=()=>{rendered+=1;video.requestVideoFrameCallback(frame)};video.requestVideoFrameCallback(frame)} + await video.play();const parameters=transceiver.sender.getParameters();if(!parameters.encodings.length)parameters.encodings=[{}]; + parameters.encodings[0].maxBitrate=MAX_BITRATE;parameters.encodings[0].maxFramerate=FRAME_RATE; + parameters.encodings[0].scaleResolutionDownBy=1;parameters.degradationPreference='balanced';await transceiver.sender.setParameters(parameters); + await waitFor(async()=>{const report=await receiver.getStats();let frames=0;report.forEach((item)=>{if(item.type==='inbound-rtp'&&item.kind==='video')frames=Math.max(frames,item.framesDecoded||0)});return frames>0&&rendered>0},10000,'decoded frame'); + return {sendPc,receivePc,sender:transceiver.sender,receiver,remoteTrack,rendered:()=>rendered}} + async function run(codec,count,topology){if(!track)throw new Error('Capture not initialized');const edges=[]; + try{edges.push(await edge(codec,track)); + if(topology==='relay')edges.push(await edge(codec,edges[0].remoteTrack)); + else for(let index=1;indexsnapshot(item.sender,item.receiver,item.rendered))),started=performance.now(); + await delay(SAMPLE_MS);const sampleMs=performance.now()-started; + const after=await Promise.all(edges.map((item)=>snapshot(item.sender,item.receiver,item.rendered))); + return edges.map((_,index)=>summarize(before[index],after[index],sampleMs))} + finally{for(const item of edges){item.sendPc.close();item.receivePc.close()}await delay(300)}} + async function runLivekit(codec,url,publisherToken,subscriberToken){if(!track)throw new Error('Capture not initialized'); + const sdk=globalThis.LivekitClient,publisher=new sdk.Room({dynacast:true}),subscriber=new sdk.Room();let remoteTrack=null,rendered=0; + subscriber.on(sdk.RoomEvent.TrackSubscribed,(nextTrack)=>{if(nextTrack.kind==='video')remoteTrack=nextTrack}); + try{await publisher.connect(url,publisherToken,{autoSubscribe:false,rtcConfig:{iceServers:[]}}); + await subscriber.connect(url,subscriberToken,{rtcConfig:{iceServers:[]}}); + const publication=await publisher.localParticipant.publishTrack(track,{source:sdk.Track.Source.ScreenShare, + videoCodec:codec,backupCodec:false,screenShareEncoding:{maxBitrate:MAX_BITRATE,maxFramerate:FRAME_RATE}, + degradationPreference:'balanced'}); + await waitFor(()=>publication.videoTrack?.sender&&remoteTrack,15000,'LiveKit publication'); + const video=document.createElement('video');video.autoplay=true;video.muted=true;video.playsInline=true;document.body.append(video); + video.srcObject=new MediaStream([remoteTrack.mediaStreamTrack]);if(video.requestVideoFrameCallback){const frame=()=>{rendered+=1;video.requestVideoFrameCallback(frame)};video.requestVideoFrameCallback(frame)} + await video.play();const receiver={getStats:async()=>{const report=await remoteTrack.getRTCStatsReport();if(!report)throw new Error('LiveKit receiver stats unavailable');return report}}; + await waitFor(async()=>{const report=await receiver.getStats();let frames=0;report?.forEach((item)=>{if(item.type==='inbound-rtp'&&item.kind==='video')frames=Math.max(frames,item.framesDecoded||0)});return frames>0&&rendered>0},15000,'LiveKit decoded frame'); + await delay(SETTLE_MS);const sender=publication.videoTrack.sender; + const before=await snapshot(sender,receiver,()=>rendered),started=performance.now();await delay(SAMPLE_MS); + const sampleMs=performance.now()-started,after=await snapshot(sender,receiver,()=>rendered); + return {...summarize(before,after,sampleMs),senderParameters:sender.getParameters()}} + finally{await Promise.allSettled([publisher.disconnect(false),subscriber.disconnect(false)]);await delay(300)}} + function stop(){stream?.getTracks().forEach((item)=>item.stop())} + Object.defineProperty(globalThis,'__CODEC_PREFLIGHT__',{value:{setup,capabilities,run,runLivekit,stop}}); + })()`; +} + +async function processSnapshot(cdp: CdpConnection): Promise { + return cdp.call( + "SystemInfo.getProcessInfo", + {}, + undefined, + Date.now() + 5_000, + ); +} + +function processCpu( + before: ProcessSnapshot, + after: ProcessSnapshot, + wallMs: number, +): Record { + const baseline = new Map( + before.processInfo.map((item) => [`${item.type}:${item.id}`, item.cpuTime]), + ); + let cpuSeconds = 0; + let stableProcesses = 0; + for (const item of after.processInfo) { + const previous = baseline.get(`${item.type}:${item.id}`); + if (previous === undefined || item.cpuTime < previous) continue; + cpuSeconds += item.cpuTime - previous; + stableProcesses += 1; + } + const wallSeconds = wallMs / 1_000; + return { + wallSeconds, + cpuSeconds, + logicalCoreEquivalent: cpuSeconds / wallSeconds, + stableProcesses, + processIdentityChanged: + stableProcesses !== before.processInfo.length || + stableProcesses !== after.processInfo.length, + }; +} + +async function main(): Promise { + const chromePath = process.env.CHROME_PATH?.trim(); + if (!chromePath) throw new Error("CHROME_PATH must point to Chrome"); + const codec = process.env.CODEC_PREFLIGHT_CODEC?.trim().toLowerCase(); + if (codec !== "vp8" && codec !== "h264") { + throw new Error("CODEC_PREFLIGHT_CODEC must be vp8 or h264"); + } + const edges = Number(process.env.CODEC_PREFLIGHT_EDGES ?? "1"); + if (edges !== 1 && edges !== 2) { + throw new Error("CODEC_PREFLIGHT_EDGES must be 1 or 2"); + } + const topology = process.env.CODEC_PREFLIGHT_TOPOLOGY?.trim() || "direct"; + if (topology !== "direct" && topology !== "relay" && topology !== "livekit") { + throw new Error("CODEC_PREFLIGHT_TOPOLOGY must be direct, relay, or livekit"); + } + const [debugPort, pagePort] = await Promise.all([reservePort(), reservePort()]); + const profile = await mkdtemp(join(tmpdir(), "screener-codec-preflight-")); + let chrome: ChildProcessWithoutNullStreams | null = null; + let cdp: CdpConnection | null = null; + let server: Server | null = null; + try { + server = await startServer(pagePort); + chrome = spawn(chromePath, [ + `--remote-debugging-port=${debugPort}`, + `--user-data-dir=${profile}`, + "--no-first-run", + "--no-default-browser-check", + "--disable-extensions", + "--disable-background-timer-throttling", + "--disable-backgrounding-occluded-windows", + "--disable-renderer-backgrounding", + "--autoplay-policy=no-user-gesture-required", + "--auto-select-tab-capture-source-by-title=Codec Preflight Source", + "--window-size=1920,1080", + "about:blank", + ], { stdio: "pipe", windowsHide: true }); + chrome.stdout.resume(); + chrome.stderr.resume(); + const version = await waitForVersion(debugPort, chrome); + cdp = await CdpConnection.connect(version.webSocketDebuggerUrl, Date.now() + 5_000); + const gpuInfo = await cdp.call("SystemInfo.getInfo", {}, undefined, Date.now() + 5_000); + await createPage(cdp, `http://127.0.0.1:${pagePort}/source`, undefined, true); + const page = await createPage(cdp, `http://127.0.0.1:${pagePort}/test`, undefined, true); + await evaluate(cdp, page, browserProbe(), 5_000); + const capabilities = await evaluate( + cdp, + page, + "globalThis.__CODEC_PREFLIGHT__.capabilities()", + 5_000, + ); + const capture = await evaluate( + cdp, + page, + "globalThis.__CODEC_PREFLIGHT__.setup()", + 15_000, + ); + const before = await processSnapshot(cdp); + const started = performance.now(); + let media: unknown; + if (topology === "livekit") { + const livekitUrl = process.env.CODEC_PREFLIGHT_LIVEKIT_URL?.trim(); + const apiKey = process.env.CODEC_PREFLIGHT_LIVEKIT_KEY?.trim(); + const apiSecret = process.env.CODEC_PREFLIGHT_LIVEKIT_SECRET?.trim(); + if (!livekitUrl || !apiKey || !apiSecret) { + throw new Error("LiveKit preflight credentials are missing"); + } + const room = `codec-preflight-${Date.now()}`; + const token = async (identity: string, publish: boolean) => { + const access = new AccessToken(apiKey, apiSecret, { identity, ttl: 60 }); + access.addGrant({ + roomJoin: true, + room, + canPublish: publish, + canPublishSources: publish ? [TrackSource.SCREEN_SHARE] : undefined, + canSubscribe: !publish, + canPublishData: false, + }); + return access.toJwt(); + }; + const [publisherToken, subscriberToken] = await Promise.all([ + token("host", true), + token("viewer", false), + ]); + media = await evaluate( + cdp, + page, + `globalThis.__CODEC_PREFLIGHT__.runLivekit(${JSON.stringify(codec)}, ${JSON.stringify(livekitUrl)}, ${JSON.stringify(publisherToken)}, ${JSON.stringify(subscriberToken)})`, + 60_000, + ); + } else { + media = await evaluate( + cdp, + page, + `globalThis.__CODEC_PREFLIGHT__.run(${JSON.stringify(codec)}, ${edges}, ${JSON.stringify(topology)})`, + 45_000, + ); + } + const wallMs = performance.now() - started; + const after = await processSnapshot(cdp); + await evaluate(cdp, page, "globalThis.__CODEC_PREFLIGHT__.stop()", 5_000); + const report = { + schemaVersion: 1, + browser: version.Browser, + configuration: { + width: WIDTH, + height: HEIGHT, + frameRate: FRAME_RATE, + maxBitrate: MAX_BITRATE, + contentHint: "motion", + settleMs: SETTLE_MS, + sampleMs: SAMPLE_MS, + edges, + topology, + }, + gpuInfo, + capabilities, + capture, + codec, + media, + cpu: processCpu(before, after, wallMs), + }; + const outputDirectory = resolve("benchmark-results"); + await mkdir(outputDirectory, { recursive: true }); + const output = join( + outputDirectory, + `browser-codec-preflight-${codec}-${topology}-${edges}.json`, + ); + await writeFile(output, `${JSON.stringify(report, null, 2)}\n`, "utf8"); + console.log(output); + } finally { + await cleanupRun({ + cdp, + native: null, + chrome, + server: null, + profile, + ports: [debugPort], + }); + if (server) { + await new Promise((resolveClose) => server!.close(() => resolveClose())); + } + } +} + +await main(); diff --git a/tsconfig.tools.json b/tsconfig.tools.json index 5f0a3b5..7c85324 100644 --- a/tsconfig.tools.json +++ b/tsconfig.tools.json @@ -21,6 +21,7 @@ "vitest.config.ts", "scripts/native-one-viewer-gate.ts", "scripts/native-one-viewer-ledger.ts", + "scripts/browser-codec-preflight.ts", "tests" ] }