diff --git a/docs/README.md b/docs/README.md index 289de11..6b88993 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,7 +10,7 @@ - [Peer-assisted media research](./research/peer-assisted-media.md): browser shared-encode limits, standard relay re-encoding, deterministic sticky topology, runtime relay-capacity extension, and abandon gates. - [Low-server-cost media routes](./research/low-server-media-routes.md): traffic accounting, route screening, token-free SFU standby prewarm, conditional native encoded-RTP relay, and automatic final fallback. - [Advanced peer distribution](./research/advanced-peer-distribution.md): multi-tree/SVC, encoded-object relay, native RTP forwarding, FEC/network-coding, MoQ, and measurable go/no-go gates. -- [Native shared-encode sender](./research/native-shared-encode-sender.md): bounded libwebrtc encoder-proxy risk spike, Draft PR #16's transport-only Pion fanout oracle, and hard failure gates. +- [Native shared-encode sender](./research/native-shared-encode-sender.md): consolidated Draft #16/#18/#22/#23/#25/#28 evidence ladder, current Pion/WebCodecs verdicts, and product stop line. - [Realtime quality adaptation](./research/realtime-quality-adaptation.md): browser degradation policy, live profile changes, offline-encoding boundaries, and measurement gates. - [Browser screen-audio quality](./research/browser-screen-audio-quality.md): capture/source compatibility, standard readback limits, Opus negotiation boundaries, and the retained no-control decision. - [Agent context governance research](./research/agent-context-governance.md): official Codex, Claude Code, Hermes Agent, and GitHub practices adopted by this repository. diff --git a/docs/project-memory.md b/docs/project-memory.md index 438894b..3fd2a93 100644 --- a/docs/project-memory.md +++ b/docs/project-memory.md @@ -37,9 +37,10 @@ Last updated: 2026-08-19 - With `ROOM_DATABASE_PATH` and the site password, SQLite stores only room ID and host-token digest. Links persist and stopping leaves viewers waiting. Room `1` survived the `769de201f7cc` deployment; without the path, rooms are random and temporary. - Production runs behind nginx with Node.js 24.19.0. Authenticated coturn 4.17.2 at `turn.bonfire.icu:3478` has verified public STUN, TURN/UDP, TURN/TCP, and relay-only traffic. TURN/TLS is intentionally disabled. - The deployed binary contains the default-off peer relay and ADR-0005 controller, but production has neither `PEER_ASSISTED_MEDIA` nor a LiveKit tuple. It therefore exposes no standby URL, imports no SFU SDK at runtime, and uses only the existing P2P/TURN path. -- When enabled later, peer recovery precedes an allowlisted SFU root; session-bound revisions prepare/commit/abort, and an active SFU gets one token refresh before share-local failback. Browser relays re-encode; native shared encoding remains unimplemented. +- When enabled later, peer recovery precedes an allowlisted SFU root; session-bound revisions prepare/commit/abort, and an active SFU gets one token refresh before share-local failback. Browser relays re-encode; native shared encoding remains outside product code. - Corrected Chrome 151/LiveKit 1.13.5 same-leaf/two-root cold/standby A/B reduced failure-to-active from 1.481 seconds to 200 ms and failure-to-render from 2.257 seconds to 319.7 ms. The leaf decoded/rendered 31 new frames and 25 ms sampling kept host edge peak two. It is localhost, headless, synthetic 720p30 video only; the roughly 86% result combines SDK download/parse and network prewarm and must not be extrapolated to public networks. -- Draft PR #16 (`5b09f0a`) proves one Pion RTP write fans equivalent data to two transports with independent SSRCs. It has no encoder and proves neither physical shared encode nor browser readiness. +- Draft native ladder #16/#18/#22/#23/#25/#28 reaches one bounded Chrome 720p30 loop: one WebCodecs object feeds two Pion/browser legs, applies their lower stock-GCC target, and recovers one isolated loss via primary-SSRC replay. This is not proof of one physical or hardware encode. +- Stock Pion GCC plus negotiated RTX is a no-go. The no-RTX candidate can distort loss statistics and stays outside product code until audio, heterogeneous estimates, broader loss, TURN/reconnect, and browser-diversity gates pass; do not add custom congestion control to bypass them. - Chrome 151 short synthetic `1/3/5/8` runs kept host edges at two, relay edges at one, and all viewers decoding; a three-viewer relay close recovered in 5.32 seconds. This proves topology/control only, not quality, load, endurance, or silent partitions. - The current quality stack uses one strict, non-persistent `QualitySettings` value: 720p/1080p/1440p, integer 15-60 fps, 2-12 Mbps, and clarity/balanced/fluid preference. Host-only updates are last-wins for current/future relays and optional SFU; ordinary P2P wire stays unchanged. Initial, replaced, and updated senders share serialized configure/readback; rejection, rewriting, or three identical native limitation samples produce one visible warning without automatic action. Successful replacement rollback retains the original failure warning. - Chrome 151 with one host, three viewers, and synthetic 720p30 passed balanced and clarity propagation to every participant. Every baseline active video sender showed matching preference readback, peer fingerprints stayed stable, and every viewer decoded and rendered new frames after each change; host fanout was two and relay fanout one. This is control evidence, not a quality, load, public-network, or endurance claim. diff --git a/docs/research/native-shared-encode-sender.md b/docs/research/native-shared-encode-sender.md index 42f2f15..ccd033a 100644 --- a/docs/research/native-shared-encode-sender.md +++ b/docs/research/native-shared-encode-sender.md @@ -3,8 +3,9 @@ - Research date: 2026-08-19 - Scope: one Windows game-capture sender, one encoded video stream, and at most two independent standard WebRTC media edges -- Status: Pion transport-fanout sub-oracle passed in Draft PR #16; physical - shared encoding and browser end-to-end behavior remain unproven +- Status: stacked Draft PRs #16/#18/#22/#23/#25/#28 pass one bounded live + two-leg WebCodecs/Pion candidate; product and physical shared encoding remain + unproven ## Decision Input @@ -17,6 +18,13 @@ must prove one narrow property before adding capture, audio, UI, or packaging: > one physical encoder invocation produces the compatible encoded access unit > consumed by both sessions. +The completed Pion/WebCodecs research ladder proves a narrower property: one +JavaScript `VideoEncoder` object emitted the encoded chunks consumed by two +browser sessions. WebCodecs does not expose physical encoder-instance counts, +and Chrome used `hardwareAcceleration: "no-preference"` after rejecting the +single allowed `prefer-hardware` attempt. The result therefore must not be +described as one hardware or physical encode. + The preferred first experiment is a standalone Windows C++ sender using a fixed libwebrtc revision. It keeps libwebrtc's mature PeerConnection, RTP/RTCP, pacing, NACK/RTX, DTLS-SRTP, ICE, and TURN behavior on each edge. Electron is not part of @@ -85,10 +93,10 @@ Audio, per-process WASAPI capture, WGC/DDA fallback, source selection, Electron, installers, updating, and cross-platform abstractions stay outside the first spike. -## Pion Fallback +## Pion Validation Route -Pion is the fallback when public libwebrtc encoder APIs cannot satisfy the -shared property without internal changes. `TrackLocalStaticSample` owns one +Pion is the released-public-API validation route while the larger libwebrtc +encoder-proxy experiment remains unbuilt. `TrackLocalStaticSample` owns one packetizer and writes through a `TrackLocalStaticRTP` that can bind to multiple PeerConnections, making encoded-payload fanout explicit. @@ -99,41 +107,55 @@ aggregation of both edge targets, encoder bitrate control, PLI handling, and bounded RTP/RTX queues. A Pion route must use the same minimum-edge rate rule and must not invent a custom SRTP, ICE, or congestion protocol. -### Draft PR #16 Transport Oracle +## Stacked Validation Ladder -[Draft PR #16](https://github.com/TNTcraftHIM/Screener/pull/16), branch -`spike/native-rtp-fanout-oracle` at commit `5b09f0a`, passes CI for one narrower -public-API question. With Pion WebRTC v4.2.18, one shared -`TrackLocalStaticRTP.WriteRTP` call reaches two independent PeerConnections. -Both receivers observe the same semantic payload, RTP sequence number, and -timestamp, while each binding receives its own transport SSRC and the caller's -packet is not mutated. +These Draft PRs are deliberately stacked research, not six product features: -This is an in-process RTP transport-fanout oracle, not the shared-encode -acceptance gate above. It contains no physical encoder and therefore cannot -prove one encoder invocation per input frame. It also does not prove browser -interoperability, capture, decoding, RTCP/PLI aggregation, congestion-control -fairness, retransmission, pacing, sustained throughput, latency, or production -readiness. The next independent gate is one browser publisher through the -native relay to two unmodified browser viewers with bounded RTCP, PLI, queue, -and bandwidth-estimation behavior. +| PR / commit | Bounded gate | Result | +| --- | --- | --- | +| [#16](https://github.com/TNTcraftHIM/Screener/pull/16) / `5b09f0a` | One Pion RTP write across two independent transports | Equivalent payload reached two transport-local SSRCs. There was no encoder or browser. | +| [#18](https://github.com/TNTcraftHIM/Screener/pull/18) / `80f65d0` | Synthetic WebCodecs VP8 chunks through Pion to two Chrome receivers | One encoder object produced 120 inputs/outputs and both browsers decoded; this was an offline fixture, not physical-encode evidence. | +| [#22](https://github.com/TNTcraftHIM/Screener/pull/22) / `8121b69` | Continuous 320x180@30 live bridge for about 12 seconds | One encoder object sent 360 chunks; each viewer decoded and presented 331 frames, with encoder/native queue peaks of one. | +| [#23](https://github.com/TNTcraftHIM/Screener/pull/23) / `eb9aabe` | Merged PLI/FIR policy, min-of-two target policy, and independent 512-packet NACK/RTX state | Policy traces passed, but stock Pion GCC plus negotiated RTX failed with `unknown ssrc: 2001`. This composition is no-go. | +| [#25](https://github.com/TNTcraftHIM/Screener/pull/25) / `a6c3456` | No-RTX primary-SSRC replay under one controlled leg loss | Leg 1 emitted one NACK and one replay with fresh TWCC, then decoded another 278 frames; leg 2 stayed clean. This can distort RTCP loss accounting. | +| [#28](https://github.com/TNTcraftHIM/Screener/pull/28) / `aad560e` | Live two-leg stock-GCC target applied to the same encoder object, then one controlled loss | The bounded gate passed. Both estimates happened to be 600 kbps, so heterogeneous-estimate behavior remains unproven. | -## Acceptance And Failure Gates +### Final Live Gate Evidence -- Two current unmodified browsers decode continuously. -- Unique physical encode calls equal unique input frames, not twice that count. -- Direct and TURN edges can coexist; rebuilding one edge does not create a - second physical encoder or interrupt the other edge. -- Limiting one edge lowers the common encoder target without an unbounded pacer - queue; removing that weak edge lets the remaining target recover. -- One viewer's PLI produces one physical keyframe and both viewers recover. -- Measurements include encode count/time, target and actual bitrate, PLI/NACK, - RTT/loss, queue growth, CPU, GPU, and host upload. +The sole authorized #28 browser run used Headless Chrome 151 and Pion WebRTC +v4.2.18 on Windows amd64. It ran 1280x720@30 for 12.972 seconds with 390 +scheduled inputs and 390 outputs from one `VideoEncoder` object. The encoder +queue peaked at 1/4, the native queue at 1/8, and the WebSocket buffer at +12,467/524,288 bytes; no encoder or socket drop occurred. -Reject a libwebrtc fork, deprecated internal-source encoder APIs, SVC/simulcast, -a second codec, FFmpeg/x264, a third host edge, custom RTP/SRTP or congestion -control, automatic topology switching, audio, UI, and packaging in the risk -spike. These are separate decisions after the shared property is proven. +Both legs returned real Transport-CC before target selection. Their final stock +GCC estimates were both 600 kbps. The coordinator took the minimum, published +one 600 kbps target, and the host serialized `flush()`, `configure()` on the +same encoder object, and a forced key frame to acknowledge the change from the +initial 1.2 Mbps ceiling. + +After acknowledgment, the boundary dropped one leg-1 primary packet. Chrome +sent one NACK and Pion replayed the same primary RTP identity with a new TWCC +sequence; leg 2 recorded no drop, NACK, or replay. Both viewers decoded another +359 frames after the shared recovery marker. This is a go result only for the +bounded live target plus primary-SSRC recovery candidate. + +## Product Stop Line + +The stock Pion GCC plus negotiated RFC 4588 RTX composition is +`no-go-stock-pion-gcc-rtx`; do not work around it with a custom pacer, +congestion controller, interceptor fork, or private transport. The no-RTX +primary-SSRC path stays research-only because it sacrifices retransmission- +specific receiver statistics and can distort RTP/RTCP loss accounting. + +No stacked PR is connected to Screener's product controller, capture path, or +audio path. Before product consideration, a target native sender must still +prove one physical encoder invocation on representative hardware, live +PLI/FIR-to-encoder control, heterogeneous downstream estimates, bounded burst +and sustained loss, audio/A-V synchronization, mixed direct/TURN edges, +reconnect isolation, browser diversity, lifecycle, and sustained CPU/GPU, +memory, latency, quality, and upload measurements. A third host edge, custom +RTP/SRTP, or a custom congestion-control framework remains out of scope. ## Primary Sources And License Boundary @@ -153,6 +175,12 @@ spike. These are separate decisions after the shared property is proven. - [Pion track fanout implementation](https://github.com/pion/webrtc/blob/main/track_local_static.go) - [Pion send-side bandwidth estimator](https://github.com/pion/interceptor/blob/main/pkg/gcc/send_side_bwe.go) - [Pion WebRTC license](https://github.com/pion/webrtc/blob/main/LICENSE) +- [WebCodecs](https://www.w3.org/TR/webcodecs/) +- [Pion WebRTC v4.2.18](https://github.com/pion/webrtc/tree/v4.2.18) +- [Pion Interceptor v0.1.47](https://github.com/pion/interceptor/tree/v0.1.47) +- [RFC 4585 RTP/AVPF feedback](https://www.rfc-editor.org/rfc/rfc4585.html) +- [RFC 4588 RTP retransmission](https://www.rfc-editor.org/rfc/rfc4588.html) +- [RFC 8888 congestion-control feedback](https://www.rfc-editor.org/rfc/rfc8888.html) - [Electron desktop capture](https://www.electronjs.org/docs/latest/api/desktop-capturer/) libwebrtc uses a BSD-style license and Pion uses MIT, but distribution still diff --git a/docs/status.md b/docs/status.md index 692ec0d..c9ed1c8 100644 --- a/docs/status.md +++ b/docs/status.md @@ -30,6 +30,7 @@ deployed binary still uses one host peer connection per viewer and P2P/TURN. - Chrome 151 CDP checks at 320/375/390 CSS px keep the two idle-stage actions equal, on one row, 44 px high, and free of horizontal overflow. The Viewer waiting page also has no overflow; its details checkbox starts false, changes locally, and resets after navigation. These checks cover idle/waiting states, not live media. - Local diagnostics use adjacent non-overlapping `getStats()` deltas; empty, changed-stream, and reset intervals rebase instead of publishing lifetime averages. - Production HTTPS/WSS, access cookie, room/WebSocket authorization, certificate renewal, public STUN, and authenticated TURN/UDP and TURN/TCP relay-only bidirectional paths are verified. TURN/TLS is intentionally disabled. +- Draft native ladder #16/#18/#22/#23/#25/#28 passed a bounded 720p30 two-leg loop: one WebCodecs object accepted the lower stock-GCC target, one isolated loss recovered, and leg 2 stayed clean. Research-only. ## Unverified Boundaries @@ -38,8 +39,8 @@ deployed binary still uses one host peer connection per viewer and P2P/TURN. - Silent partitions can wait 30 to 60 seconds for heartbeat detection before the default 5-second grace; this remains unverified. - Real screen/game audio, heterogeneous machines/networks, mobile lifecycle, the new production quality/pause cycle, room `1` stop/re-publish, and sustained profiles remain unverified. - Earlier production showed severe degradation across all profiles. The new controls permit clarity/balanced comparison, but a controlled 1/2/3-viewer and TURN sample must still isolate capture, CPU, path, and receiver limits before any quality claim or automatic controller. -- Browser relays do not share encoding. Encoded objects, custom congestion control, multiple trees, and network coding remain separate measured candidates. -- The endpoint budget is host/relay at most two downstream edges; a native engine must prove shared encoding. The browser path remains host two/viewer one and re-encodes per relay. Striping and multi-parent assembly are only recorded. +- Browser relays re-encode. The native ladder proves one WebCodecs object, not one physical/hardware encode. Stock GCC plus RTX is no-go; no-RTX passed one controlled loss but weakens statistics. Audio, heterogeneous estimates, broader loss, direct/TURN, reconnect, browser diversity, product wiring, striping, and multi-parent assembly remain unverified. +- The endpoint budget remains host/relay at most two downstream edges; the browser path is host two/viewer one. - The corrected standby smoke is localhost/headless/video-only; public DNS/TLS reuse, transport, audio, shaping, load, mobile, endurance, and sub-25 ms overlap remain open. ## Next Milestone @@ -58,6 +59,9 @@ picture, about 8 seconds total. Silent partitions include their detection delay. ADR-0004 fails closed. Native shared encode and packet/layer striping remain separate experiments, not ways to relabel a failed browser route. +The native ladder stops at #28. Do not bypass stock GCC/RTX with custom +transport/control; no-RTX remains research-only. + ## Blockers And Decisions Production enablement of peer assistance and automatic routing is No-Go until