diff --git a/docs/adr/0005-automatic-hybrid-media-routing.md b/docs/adr/0005-automatic-hybrid-media-routing.md index f8ae986..3d32941 100644 --- a/docs/adr/0005-automatic-hybrid-media-routing.md +++ b/docs/adr/0005-automatic-hybrid-media-routing.md @@ -84,7 +84,9 @@ TURN config, issuer, capability, refresh wire, or client propagation. Stale `PEER_ICE_TURN_*` keys fail startup even when blank. The source candidate now implements a complete selected-edge tuple and one relay-only rebuild for either the initial pending Host publication or an active Host/SFU route; production -configures the tuple, but real TURN/media evidence remains open. +configures the tuple. A bounded exact-Web-source canary against the production +LiveKit/coturn tuple proves the active Host/SFU relay route; initial Host ingress +and `peer-selected` relay media remain open. The wire has two explicit edge kinds: `peer-selected` for the last-mile failed peer edge and `host-sfu-ingress` for a restricted Host-to-SFU retry. Ordinary Peer ICE remains STUN-only. Each room admits at most one pending or answered @@ -97,7 +99,7 @@ production. The tracked LiveKit sample explicitly sets `tcp_port: 0`, disables TCP fallback, supplies the deployment-owned STUN server, and configures no TURN service. The SFU controller still activates only after a peer edge exhausts recovery. Public participant entry has -been observed, but retained media and route admission remain unverified. The old +been observed, but public-room retained media and route admission remain unverified. The old `769de201f7cc` release and coturn relay are isolated rollback resources, not a permanent compatibility branch or advertised current transport. @@ -431,9 +433,25 @@ relay-only LiveKit publisher `RTCConfiguration`, including during the initial pending prepare. Pinned LiveKit client 2.22.0 accepts `rtcConfig` on `Room.connect`; its engine clones that override before creating the publisher PeerConnection and does not replace explicitly supplied ICE servers. This path -is configured in production but has no real relay-media evidence. +is configured in production, and its active Host-ingress relay function is now +proven; the initial-ingress and `peer-selected` variants remain open. LiveKit participant-wide embedded/external TURN remains a separate ICE domain. +On 2026-08-21 a standalone relay-only Chrome allocation gate issued a credential +through the production parser/issuer and received one UDP relay candidate. One +subsequent bounded Chrome 151 canary ran the exact deployed Web source in a local +Screener room against the production LiveKit/coturn tuple. A temporary DEV-only +hook forced the active SFU publisher to fail once and was removed afterward. The +application then issued `host-sfu-ingress`, connected a relay-policy publisher +whose only ICE server was TURN, selected a succeeded/nominated pair with +`relayProtocol=udp`, sent video bytes and frames, and kept Viewer decode/render +counters advancing. Ordinary peer PCs remained STUN-only, Host outbound media +edges peaked at one, and explicit stop returned them to zero. This proves the +active functional transition only; it is not latency, quality, capacity, expiry, +mobile, or resource evidence. Chrome reported `candidateType=prflx` for the same +local candidate; the standards-bounded interpretation and raw-counter summary +are retained in the linked research rather than normalized away. + ### Deferred Optimization These items are not current runtime behavior: diff --git a/docs/deployment.md b/docs/deployment.md index f0fccf1..f3b8ec9 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -35,9 +35,12 @@ and the old endpoint remains 404. Exact `22119b907d3cf03ce8b06d6fb4596ce1a26fedd7` is the immediate rollback target; exact `fd76277b05d491af8840b28f3132b7ff445d3cbe` remains a secondary rollback. -The selected-edge UDP tuple is configured with TTL 120, but no real TURN/SFU -media canary was triggered; this remains a bounded configuration smoke, not -broad-rollout media acceptance. +The selected-edge UDP tuple is configured with TTL 120. The cutover itself +triggered no TURN/SFU media and remains configuration evidence. A separate +post-deploy local Screener canary using this exact Web source and the production +LiveKit/coturn tuple later proved active Host-ingress relay function without +using the production application room or database; it is not broad-rollout or +performance acceptance. ## Topology and prerequisites @@ -213,6 +216,16 @@ receive a short-lived grant. Roll back by removing the application tuple before changing coturn or firewall state. Credentials never enter URLs, logs, browser persistence, room rows, or SQLite. +Before a media canary, `npm run gate:turn-allocation` provides the bounded +allocation check. Set `CHROME_PATH` in the process environment and supply the +complete selected-edge tuple either in that environment or as newline-delimited +`KEY=value` records on stdin. Values containing `=` are preserved. The script +uses an isolated Chrome profile and loopback-only in-memory credential response, +then emits only status, relay-candidate count, transport protocol, and coarse ICE +error-code buckets. Do not echo, log, or persist raw production environment +input; never add TURN URLs, usernames, credentials, candidate addresses, or raw +errors to the result. + `PEER_ASSISTED_ROOM_IDS` is retired. Supplying it, even blank, fails startup so that a stale room-1 deployment cannot silently retain the old scope. With `PEER_ASSISTED_MEDIA=true`, every normal room receives peer-assisted routing, @@ -515,7 +528,8 @@ active/running with their observed `NRestarts=0`. Backup `/opt/screener/backups/16f6eab27bdf-precutover-20260820T234457Z` retains the prior environment, database, and exact current target. This deploys the one-use initial/active Host-ingress selected retry; no browser, SFU media, TURN relay, -or performance canary ran, so TURN transport remains unproved. +or performance canary ran during that cutover. The later post-deploy functional +canary is recorded above and does not change this cutover record. Enabling persistence does not migrate rooms that existed only in memory. The deployment restart invalidates those temporary links; the first subsequently @@ -632,7 +646,8 @@ state. That host retains an older authenticated-relay configuration and TCP/UDP 3478 plus UDP 49152-49251 firewall range. In this template Screener advertises no TURN credential and the canary audit found zero allocations; production selected-edge issuance remains limited to the controller's one configured edge -and still lacks a real forced-relay media acceptance. +and has one active Host-ingress forced-relay functional acceptance. Initial +Host ingress and `peer-selected` last mile remain open. Copy [`deploy/coturn/turnserver.conf.example`](../deploy/coturn/turnserver.conf.example) to an untracked service-owned location and use the tracked @@ -674,9 +689,10 @@ Run these checks from real external networks before calling the deployment usabl 3. On a normal room, exhaust a peer route and verify the host plus at most two necessary roots select LiveKit UDP 7882. Peer descendants stay on ordinary direct UDP and host/relay downstream caps remain two/one. -4. Block all UDP on one test client. Verify bounded ICE recovery ends in a clear - connection failure, without an ICE/TCP, TURN/TCP, or long pseudo-connected - path. No force-relay product branch exists in this candidate. +4. Force one controller-eligible edge past SFU/UDP and verify only that edge gets + the short-lived TURN server plus relay policy and selects TURN/UDP. Ordinary + peer PCs must stay STUN-only. Then block all UDP and verify bounded recovery + ends clearly without ICE/TCP, TURN/TCP, or a long pseudo-connected path. 5. Exercise root departure, reconnect, SFU unavailable, route prepare rollback, stop, and source/profile changes. Unaffected subtrees must not migrate. 6. Repeat at 1, 3, 5, and 8 viewers across representative consumer networks. diff --git a/docs/project-memory.md b/docs/project-memory.md index 5f7fe84..5210aaf 100644 --- a/docs/project-memory.md +++ b/docs/project-memory.md @@ -25,7 +25,7 @@ Last updated: 2026-08-21 - Keep experiments bounded: the standard representation sequence is below; native RTP relay, encoded-object striping, and FEC remain separate. - ADR-0006's historical browser-bridge canary remains no-go; VP8 stays default. Browser H.264 rendered 298/299; Native WGC/MF rendered 203 with PID/LUID-correlated `VideoEncode`. Downloaded use, multi-viewer/endurance/public proof remain open. - Mobile endpoints are Web Viewer-only. Current Android/iOS browsers do not expose Web Host capture; AirPlay/system mirroring is Viewer-local output. -- ADR-0005 accepts direct/peer UDP, bounded SFU roots, then selected-edge TURN. Source admits one pending or answered `peer-selected` attempt per room while `host-sfu-ingress` remains independent. Production grants one revision-bound, one-use retry for initial/active Host ingress. The deployed tuple/control path have no production SFU/TURN media canary; participant-wide TURN stays rejected and ordinary peers stay STUN-only. +- ADR-0005: direct/peer UDP -> bounded SFU roots -> selected-edge TURN. Source caps pending/answered `peer-selected` at one per room; Host ingress is independent. An exact-source/production-media canary proves active Host ingress only. Ordinary peers stay STUN-only. - The controller is process-enabled, not room-allowlisted: `PEER_ASSISTED_MEDIA=true` gives every normal room the same direct/peer -> SFU -> selected-edge path with per-room state. Room `1` is historical smoke. Preserve sticky P2P, mobile leaves and break-before-make; recovery gets one attempt per layer (ICE restart, same-parent rebuild, alternate peer, then SFU), never three identical retries. - C+B quality reparenting is edge-local and cooldown-bound; no score, timer, or global parent penalty. - Flagship media is UDP; HTTPS/WSS stays TLS/TCP; the old release remains rollback-only. @@ -45,13 +45,13 @@ Last updated: 2026-08-21 - Production is exact `16f6eab27bdfb1c15cdbd814a35864f4f18be767` at `https://share.bonfire.icu`; its 1,027,423-byte artifact SHA-256 is `74e0274ab11a162cb9dd4be1c34bb6b639e2ea76005969c30ae3a1daa656742f`. The 2026-08-20T23:44:57Z cutover took 10,238 ms lock/641 ms stop-health/570 ms switch-health. Immediate rollback is exact `22119b907d3cf03ce8b06d6fb4596ce1a26fedd7`; exact `fd76277b05d491af8840b28f3132b7ff445d3cbe` remains secondary. - `main` is ahead only in Native/test evidence; the Windows Native sender stays source-only. Host-ingress retry is deployed. - The four services are active/running with observed `NRestarts=0`; local/public health are 200. SQLite v3 has five rooms including room `1`, with DB owner/mode `screener:screener`/0600. -- Production ICE remains ordinary STUN-only. `PEER_ASSISTED_MEDIA=true` enables the controller for every normal room, and selected-edge TURN is configured for the single UDP tuple with TTL 120; the coturn daemon and nginx/LiveKit/firewall/listener baselines were unchanged. No production SFU/TURN media canary or quality claim is implied. +- Production ICE is ordinary STUN-only. The all-room controller has one selected-edge UDP tuple at TTL 120. A local exact-source canary used production LiveKit/coturn without production app/DB writes and proved active Host-ingress function only. - Production requires the independent site-access secret through `SITE_ACCESS_PASSWORD`. Its stateless cookie authorizes creation/Host and permits code-only Viewer attempts; valid private fragment grants remain direct. Public-watch accepts site access plus code, while private code-only entry additionally requires the room password. The env, endpoint, cookie, and nginx limiter naming migrated atomically; anonymous failures stay neutral, and there are no accounts/JWT/session rows. - Production uses nginx, Node.js 24.19.0, coturn 4.17.2, and LiveKit 1.13.5 on UDP 7882 (TCP fallback off). Ordinary peer ICE receives no TURN; the selected-edge tuple is configured only for the current controller edge, with no credential pre-advertised to ordinary peers. Coturn retains the old authenticated-relay ports. Services are healthy with zero restarts. - ADR-0005 rejects `PEER_ASSISTED_ROOM_IDS`; `PEER_ASSISTED_MEDIA=true` enables every room and ordinary peers stay STUN-only. Initial or active Host ingress may consume one bound relay-only grant; ready/abort clears it and failure restores peer baseline. Stale room-ID config fails startup; `screener-v2` is the only deployed wire. - Chrome 151/LiveKit localhost A/B cut failure-to-active/render from 1.481/2.257 seconds to 0.200/0.320; 31 new frames and 25 ms sampling kept host edges at two. It is headless synthetic 720p30 and includes SDK/network prewarm, not public-network evidence. - Chrome 151 synthetic topology/quality runs kept fanout 2/1 and decoding; one relay close recovered in 5.32 seconds. Control evidence only. -- Deployed Web source uses LiveKit 2.22.0's two-layer `q,h` layout. Local Chrome 151/LiveKit 1.13.5 kept one SFU/UDP root with growing inbound/decoded/rendered counters, clean leaves and no TURN. Production SFU media, zero-child BWE, and calibrated C+B loss remain open. +- Deployed Web uses LiveKit 2.22.0 `q,h`. Chrome 151 proves local SFU/UDP and, separately, active selected TURN/UDP Host ingress against production media services with Viewer frame progress and clean stop. Initial/peer-selected relay, BWE, and C+B remain open. - C+B uses three hard-bad pairs, one-use samples and guarded intent/cooldown. Deployed admission rescue moves the oldest childless zero-capacity Host leaf below an unassigned one-slot relay in one revision; no score/timer/global rebalance. Deployment passed, but no real room triggered it. - Web names/window audio are deployed; source adds full-display system audio, music hint, interval AV loss% and audio codec/bitrate/jitter details. Native WGC/MF remains source-only: 203 rendered frames, 500 Opus packets, matched PID/LUID `VideoEncode`. New Web audio work is undeployed; VP8 remains default. @@ -68,7 +68,7 @@ These are measurement gates, not performance claims. - Sustainable count by hardware, quality, network, and route: finish instrumented `1/3/5/8`, then pass a 20-viewer matrix before changing the accepted target default to 20. - Whether ADR-0004 passes fanout, re-encoding, depth-four latency, reparenting, silent-partition, and mobile-leaf gates. -- Whether selected-edge passes forced relay, expiry/failure, mobile, 1-GiB resource and bandwidth gates. Coturn's bearer is non-revocable until expiry; application guards and TTL only bound reuse. Performance comparison and the retained matrix follow functional landing; Media TCP is out of scope. +- Selected-edge: active Host-ingress function passes; initial ingress, `peer-selected`, expiry/failure, mobile, resource and bandwidth gates remain. The bearer is non-revocable until expiry; performance follows later and Media TCP is out. - Exact mobile Viewer lifecycle behavior across autoplay, rotation, backgrounding and network changes. - Project license and distribution model, which determines whether GPL/AGPL sources can move beyond study-only use. - Initial deployment regions and expected mainland China, Hong Kong, and overseas network mix. diff --git a/docs/research/low-server-media-routes.md b/docs/research/low-server-media-routes.md index e7e8639..712af80 100644 --- a/docs/research/low-server-media-routes.md +++ b/docs/research/low-server-media-routes.md @@ -1,14 +1,15 @@ # Low-Server-Cost Media Routes -- Research date: 2026-08-19 +- Research date: 2026-08-21 - Scope: one broadcaster, at most eight trusted viewers, low latency, and host media fanout at most two - Status: ADR-0005 accepts STUN-only direct/peer UDP, bounded SFU/UDP roots, then optional authenticated TURN/UDP for one controller-selected exceptional edge. Production enables the controller for all rooms and has a configured - selected-edge TURN tuple; ordinary peer connections remain STUN-only, while - production SFU/TURN media remains unverified. Latest source passed one local - SFU/UDP functional path; TURN remains unverified. + selected-edge TURN tuple; ordinary peer connections remain STUN-only. A + post-deploy exact-Web-source canary against the production LiveKit/coturn tuple + passed active Host SFU -> selected TURN/UDP ingress. Initial ingress, + peer-selected last mile, external cohorts, and performance remain unverified. ## Current Route Ladder @@ -44,9 +45,10 @@ Chrome 151, LiveKit 1.13.5, and pinned client 2.22.0. LiveKit produced two sende RIDs in `q,h` order; Screener's former `q,f` guard caused the Host publisher to fail closed at `sender-config`. After correcting that contract, Viewer inbound packets and decoded/rendered frames increased, endpoint edges stayed bounded, -and both clients left cleanly. TURN did not participate. This is functional, -not performance, evidence; production/public transport and browser validation -remain open. +and both clients left cleanly. TURN did not participate in that first run. A +later exact-source canary against the production media tuple proved the active +Host-ingress selected relay described below. Both are functional, not +performance, evidence; public-room and browser-diversity validation remain open. ## Token-Free SFU Standby Prewarm @@ -207,20 +209,51 @@ without advertising that credential to ordinary peer PCs. Ordinary peer ICE remains STUN-only. The participant-wide TURN config, capability and refresh wire are removed; any stale `PEER_ICE_TURN_*` key, including an empty value, fails startup. Selected-edge TURN is deployed as -a configured, controller-issued exceptional transport; its real media path is -not yet canary-proven. Every ordinary Web peer and Native-shaped client remains -STUN-only. The tracked coturn example is UDP +a configured, controller-issued exceptional transport. Its active Host-ingress +media path is canary-proven; initial ingress and `peer-selected` remain open. +Every ordinary Web peer and Native-shaped client remains STUN-only. The tracked coturn example is UDP `stun-only`; the LiveKit example exposes only ICE/UDP mux 7882, explicitly sets `tcp_port: 0` and `allow_tcp_fallback: false`, supplies the self-hosted STUN endpoint, and configures no external or embedded TURN. Candidate validation and rollback use isolated instances rather than a process-wide old-release compatibility branch. HTTPS/WSS remains TLS/TCP. + +### Active Host-Ingress Functional Evidence + +On 2026-08-21 the tracked standalone gate reused the production selected-edge +tuple parser and credential issuer, supplied the short-lived credential only to +an isolated loopback Chrome page, and returned a sanitized result: one UDP relay +candidate and an otherwise non-blocking `701` error bucket. No URL, username, +credential, candidate address, or raw error text was emitted or persisted. + +The canonical Chrome 151 product canary then used exact deployed Web source +`16f6eab27bdfb1c15cdbd814a35864f4f18be767` in a local Screener room while +connecting to the production LiveKit 1.13.5 and coturn tuple. After direct media +and active SFU media, a temporary DEV-only hook failed the refreshed publisher +once; the normal application failure path requested `host-sfu-ingress` and the +hook was removed after the run. The relay-policy publisher had only a TURN +server, reached connected/ICE-connected, and selected a succeeded/nominated UDP +pair with `relayProtocol=udp`. Pair counters reached 1,054,606 bytes sent and +4,934 received; outbound RTP reached 837,103 bytes and 101 frames. Viewer decode +and render counters both advanced during the selected stage. Ordinary peer PCs +kept policy `all` with STUN-only servers, Host active outbound edges stayed at +one, and explicit stop reduced them to zero. The production Screener app and +database were not used or modified. + +Chrome reported the selected local candidate's `candidateType` as `prflx` while +also reporting `relayProtocol=udp`. That tuple is internally inconsistent with +the W3C model: `iceTransportPolicy="relay"` permits only media-relay candidates, +and `relayProtocol` is defined for a local relay candidate. The evidence is +therefore retained verbatim and classified from the relay-only configuration, +TURN-only server set, `relayProtocol`, and transmitted media together; the +`candidateType` field is not rewritten. This proves one active functional route, +not latency, quality, capacity, expiry, mobile, resource, or bandwidth behavior. Coturn 4.17.2 documents `stun-only` as ignoring TURN requests and provides `no-tcp` and `no-tls`; it marks `no-dtls` deprecated, so the tracked temporary template does not use that switch. The shared production host instead retains its old authenticated-relay daemon and TCP/UDP 3478 plus UDP 49152-49251 rules. The -application advertises no TURN credential, and the post-canary audit found zero -allocations. This baseline is neither the accepted selected-edge rollout nor -proof that relay media works. +rejected participant-wide candidate advertised no TURN credential, and that +historical post-canary audit found zero allocations. This baseline is not the +later selected-edge functional evidence above. No public port is selected by this decision. LiveKit documents ICE/UDP mux as optional and its pinned sample recommends a multi-port UDP mux range at least @@ -300,7 +333,7 @@ lost. | Fixed two-chain browser relay | Host emits at most two copies; each relay emits at most one | Ordinary browser, but every relay decodes and re-encodes and adds a hop | Current default-off, maximum-eight-viewer spike | | Native shared-encode host | Host targets one encode for at most two standard WebRTC edges | libwebrtc public-API proxy risk spike, with Pion as fallback | Planned separate sender phase; still pays per-edge upload | | Native volunteer encoded-RTP relay | Each volunteer forwards one encoded copy | Native install, RTP/RTCP forwarding, packaging, and opt-in relay policy | Conditional experiment only if relay re-encoding is the sole browser-spike failure | -| SFU virtual parent | SFU normally emits one or two root copies; roots keep peer descendants | Service pays measured root egress; host sends one publication | Accepted primary central fallback after direct/peer UDP; latest source passes one local SFU/UDP root, while production evidence remains open | +| SFU virtual parent | SFU normally emits one or two root copies; roots keep peer descendants | Service pays measured root egress; host sends one publication | Accepted primary central fallback after direct/peer UDP; local SFU/UDP and active Host-ingress relay function pass, while public-room/performance evidence remains open | | Exceptional server-fed viewers | SFU/TURN emits necessary copies that no healthy root can distribute | Additional capped central egress | Explicit compatibility exception only; never unbounded whole-room fanout | | SVC plus multiple trees | Peers emit striped layer copies across several trees | Layer scheduling, reassembly, redundancy, and more churn state | Separate conditional spike; target endpoint upload near `B` | | Network coding | Peers or servers emit coded blocks | Generations, buffering, decoding, integrity, and a custom media plane | Trace/FEC spike only; optimize loss recovery, not clean bandwidth | @@ -387,8 +420,8 @@ Primary sources checked on 2026-08-19 and 2026-08-21: - [WebRTC transports, RFC 8835](https://www.rfc-editor.org/rfc/rfc8835.html) - browser transport capability requirements do not require an application to advertise every supported fallback on every connection. -- [WebRTC](https://www.w3.org/TR/webrtc/), - [WebRTC Statistics](https://www.w3.org/TR/webrtc-stats/), +- [WebRTC](https://w3c.github.io/webrtc-pc/), + [WebRTC Statistics](https://w3c.github.io/webrtc-stats/), [WebRTC Encoded Transform](https://w3c.github.io/webrtc-encoded-transform/#stream-processing), and [WebRTC SVC](https://www.w3.org/TR/webrtc-svc/) - W3C specifications; SVC is an encoding control, not a distribution topology. diff --git a/docs/status.md b/docs/status.md index f3e893c..8cf5028 100644 --- a/docs/status.md +++ b/docs/status.md @@ -6,7 +6,7 @@ Last updated: 2026-08-21 Production at `https://share.bonfire.icu` is exact `16f6eab27bdfb1c15cdbd814a35864f4f18be767`; artifact 1,027,423 bytes, SHA-256 `74e0274ab11a162cb9dd4be1c34bb6b639e2ea76005969c30ae3a1daa656742f`. Cutover: 10,238 ms lock/641 ms stop-health/570 ms switch-health; SQLite v3/five rooms/four active services, observed `NRestarts=0`. Initial Host-ingress retry is deployed; mobile endpoints are Web Viewer-only. -Ordinary ICE is STUN-only; all rooms use the bounded peer/SFU-UDP controller and selected-edge UDP TTL 120. Room `1` is historical smoke; production SFU/TURN media remains unverified. +Ordinary ICE is STUN-only; all rooms use bounded peer/SFU-UDP plus selected-edge UDP TTL 120. Room `1` is historical. An exact-source/production-media canary proves active Host-ingress relay only. Web names/audio hint are deployed. Native WGC/MF/package is source-only; Web defaults VP8. ## Execution Principle @@ -21,7 +21,7 @@ Flagship; parallel; min run/smoke/rollback; benchmark later. Active UI/config/lo - Native v2 is source-only (memory rooms, 300s reclaim); it follows authenticated direct-child assignments up to two and fails unsupported SFU/selected ingress boundedly. - PR #44's STUN-only/SFU-UDP router and token-free fallback prewarm are process-enabled for all normal rooms: roots <=2, browser relay <=1, bounded failure. Room `1` is historical smoke; mobile/iPad viewers are leaves, healthy edges sticky, and HTTPS/WSS stays TLS/TCP. - After an answer, a generation-bound 15s initial-connect deadline enters ICE restart; success/replacement/disposal cancels it. It is deployed but not mobile-verified. -- Production includes LiveKit's two-layer `q,h` sender guard; its functional media proof remains local, not production. +- Production `q,h` is deployed; an exact-source/production-media canary proves active Host ingress without production app/DB. - Room `1` C+B reparenting remains uncalibrated. Deployed admission rescue promotes an unassigned one-slot relay over the oldest childless zero-capacity Host leaf, with no scores or periodic optimization. - Web roster names are deployed. Source window/system audio, music hint, interval AV loss% and audio codec/bitrate/jitter details await deploy. @@ -36,13 +36,13 @@ Flagship; parallel; min run/smoke/rollback; benchmark later. Active UI/config/lo - Native Win11 audio is source-only/default-off: target isolation was 4018x and one Viewer got 495 Opus packets. Download, game sync, Win10, and other routes remain open. - Access protocol/config/HTTP/storage/SQLite/signaling focused tests pass, including commit-first teardown and v1 rollback. - Native wire/session tests enforce at most two authoritative children, no presence-created edge, stale-revision ignore, and one bounded unsupported-route failure per revision. -- Production includes initial/active selected Host ingress and peer last-mile; source caps pending plus answered `peer-selected` attempts at one per room without counting Host ingress. Focused tests cover admission, release, independent ingress, exactly-once retry, cleanup, rollback, and ordinary STUN-only. No relay media ran. +- Source caps pending/answered `peer-selected` at one per room, excluding Host ingress. Focused tests cover admission/release/rollback/STUN-only; exact-source Chrome proves active TURN/UDP media, Viewer progress, Host edge one and stop zero. Functional only. - Host names/window audio/all-room routing/Host selected ingress are active. New AV stats pass 42 focused tests/typecheck; this is not audible proof. - Exact `16f6eab` passed typecheck, both builds and 30 files/445 tests. Deploy preserved all recorded config/network/SQLite hashes; DB, routes, services, endpoints, site access, and old-endpoint 404 passed. It triggered no TURN/SFU session. Rollback is exact `22119b907d3cf03ce8b06d6fb4596ce1a26fedd7`; `fd76277b05d491af8840b28f3132b7ff445d3cbe` remains secondary. ## Unverified Boundaries -- The `16f6eab` deploy proves process/config health only; production SFU frames, initial/active retry/failback, relay pair, admission rescue, caps, quality, and resource evidence remain open. Game-share load/blur is unclassified. +- The cutover is config evidence; the separate canary closes active Host-ingress relay/frame/cap/stop function. Initial ingress, peer last mile, external cohorts, admission rescue, quality/resources/performance, and game blur remain open. - Mobile remains Viewer leaf-only. AirPlay/system mirroring is local output; live WebRTC `srcObject` has no portable in-app TV-output contract. - Silent partitions can wait 30 to 60 seconds for heartbeat detection before the default 5-second grace; this remains unverified. - The former release reportedly sustained bandwidth-limited blur on a capable LAN; Host refresh recovered while Viewer refresh did not. The new deployment has not yet reproduced or cleared it. @@ -54,10 +54,10 @@ Flagship; parallel; min run/smoke/rollback; benchmark later. Active UI/config/lo ## Next Milestone -Run one bounded production room `1` canary that retains UDP, a rendered frame, edge caps, and clean stop. TURN/performance follow later. +Next: representative production-room UDP/frame/cap/stop canary; room `1` is history. TURN performance follows later. Recovery uses one ICE restart, one same-parent rebuild, one alternate peer, then SFU. Next gate exactly-two/Dynacast-off BWE on a zero-child SFU leaf, then root evacuation. Native WGC/MF passed one-Viewer hardware; package download, Viewer2/FIFO and game A/V remain. Benchmark later. -Ordinary peers stay STUN-only. Initial/active Host ingress needs forced-relay canaries before any media claim. +Ordinary peers stay STUN-only; initial ingress and `peer-selected` still need forced-relay proof. ADR-0004 still needs the 30-minute `1/3/5/8` matrix; a separate 20-viewer gate must pass before changing the default eight. ## Blockers And Decisions diff --git a/package.json b/package.json index 0f5ee1d..a98618d 100644 --- a/package.json +++ b/package.json @@ -14,6 +14,7 @@ "build:server": "tsc -p tsconfig.server.json", "start": "node dist/server/server/index.js", "benchmark:peer-assisted": "tsx scripts/peer-assisted-benchmark.ts", + "gate:turn-allocation": "tsx scripts/turn-allocation-gate.ts", "gate:native-one-viewer": "tsx scripts/native-one-viewer-gate.ts", "typecheck": "tsc -b --pretty false", "test": "vitest run", diff --git a/scripts/turn-allocation-gate.ts b/scripts/turn-allocation-gate.ts new file mode 100644 index 0000000..2d94725 --- /dev/null +++ b/scripts/turn-allocation-gate.ts @@ -0,0 +1,126 @@ +import { execFile } from "node:child_process"; +import { once } from "node:events"; +import { readFileSync } from "node:fs"; +import { mkdtemp, rm } from "node:fs/promises"; +import { createServer } from "node:http"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { pathToFileURL } from "node:url"; + +import { parseSelectedEdgeTurn } from "../src/server/config"; +import { issueSelectedEdgeTurnCredential } from "../src/server/selected-edge-turn"; + +const allowedProtocols = new Set(["udp", "tcp", "unknown"]); +const allowedErrorBuckets = new Set(["3xx", "4xx", "5xx", "6xx", "701", "other"]); + +export interface TurnAllocationResult { + status: "passed" | "failed"; + relayCandidateCount: number; + protocols: string[]; + errorCodeBuckets: string[]; +} + +export function parseGateEnvironment(input: string): NodeJS.ProcessEnv { + const environment: NodeJS.ProcessEnv = {}; + for (const line of input.split(/\r?\n/)) { + const separator = line.indexOf("="); + if (separator > 0) environment[line.slice(0, separator)] = line.slice(separator + 1); + } + return environment; +} + +export function parseGateResult(input: string): TurnAllocationResult { + let value: unknown; + try { value = JSON.parse(input); } catch { throw new Error("Chrome returned an invalid TURN allocation result"); } + if (!value || typeof value !== "object") throw new Error("Chrome returned an invalid TURN allocation result"); + const raw = value as Record; + if ((raw.status !== "passed" && raw.status !== "failed") || + !Number.isInteger(raw.relayCandidateCount) || Number(raw.relayCandidateCount) < 0 || + !Array.isArray(raw.protocols) || !raw.protocols.every((item) => allowedProtocols.has(item)) || + !Array.isArray(raw.errorCodeBuckets) || !raw.errorCodeBuckets.every((item) => allowedErrorBuckets.has(item))) { + throw new Error("Chrome returned an invalid TURN allocation result"); + } + const protocols = [...new Set(raw.protocols as string[])]; + if (raw.status === "passed" && (Number(raw.relayCandidateCount) === 0 || + protocols.length !== 1 || protocols[0] !== "udp")) { + throw new Error("Chrome returned an invalid TURN allocation result"); + } + return { status: raw.status, relayCandidateCount: Number(raw.relayCandidateCount), + protocols, + errorCodeBuckets: [...new Set(raw.errorCodeBuckets as string[])] }; +} + +const page = ``; + +export async function main(): Promise { + const chromePath = process.env.CHROME_PATH?.trim(); + if (!chromePath) throw new Error("CHROME_PATH is required"); + const stdin = process.stdin.isTTY ? "" : readFileSync(0, "utf8"); + const config = parseSelectedEdgeTurn({ ...parseGateEnvironment(stdin), ...process.env }); + if (!config) throw new Error("The selected-edge TURN tuple is required"); + const grant = issueSelectedEdgeTurnCredential(config, { edgeKind: "host-sfu-ingress", + roomId: "allocation-gate", shareGeneration: "gate-share", revision: 1, + hostPeerId: "gate-host", hostSessionId: "gate-session", + publicationGeneration: "gate-publication", oldConnectionId: "gate-publication", + newConnectionId: "gate-connection" }, Date.now()); + let settle!: (result: TurnAllocationResult) => void; + const resultPromise = new Promise((resolve) => { settle = resolve; }); + const server = createServer((request, response) => { + response.setHeader("Cache-Control", "no-store"); + if (request.url === "/credential") response.end(JSON.stringify(grant.iceServer)); + else if (request.url === "/") response.end(page); + else if (request.url === "/result" && request.method === "POST") { + let body = ""; + request.setEncoding("utf8"); + request.on("data", (chunk) => { if (body.length < 4_096) body += chunk; }); + request.on("end", () => { + try { settle(parseGateResult(body)); response.end(); } + catch { response.statusCode = 400; response.end(); } + }); + } + else { response.statusCode = 404; response.end(); } + }); + const profile = await mkdtemp(join(tmpdir(), "screener-turn-gate-")); + let chrome: ReturnType | undefined; + let chromeClosed: Promise = Promise.resolve(); + let timeout: NodeJS.Timeout | undefined; + try { + server.listen(0, "127.0.0.1"); await once(server, "listening"); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("TURN gate listener failed"); + chrome = execFile(chromePath, ["--headless=new", "--no-first-run", + "--no-default-browser-check", "--disable-extensions", `--user-data-dir=${profile}`, + `http://127.0.0.1:${address.port}/`], { timeout: 20_000, windowsHide: true }, () => {}); + chromeClosed = new Promise((resolve) => { chrome!.once("close", () => resolve()); }); + const chromeFailure = new Promise((_, reject) => { + chrome!.once("error", () => reject(new Error("Chrome failed to start"))); + chrome!.once("exit", () => reject(new Error("Chrome exited before TURN allocation completed"))); + }); + const result = await Promise.race([resultPromise, chromeFailure, + new Promise((_, reject) => { timeout = setTimeout(() => reject(new Error("Chrome did not complete the TURN allocation gate")), 15_000); })]); + process.stdout.write(`${JSON.stringify({ schemaVersion: 1, ...result })}\n`); + return result.status === "passed" ? 0 : 1; + } finally { + if (timeout) clearTimeout(timeout); + if (chrome?.exitCode === null) chrome.kill(); + const serverClose = server.listening + ? new Promise((resolve) => { server.close(() => resolve()); }) + : Promise.resolve(); + await Promise.allSettled([chromeClosed, serverClose]); + await rm(profile, { recursive: true, force: true }); + } +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + main().then((code) => { process.exitCode = code; }).catch((error: Error) => { + process.stderr.write(`${error.message}\n`); process.exitCode = 2; + }); +} diff --git a/src/server/config.ts b/src/server/config.ts index a892d08..7a0ccde 100644 --- a/src/server/config.ts +++ b/src/server/config.ts @@ -202,7 +202,7 @@ function parseStunUrlList(value: string | undefined): string[] { }); } -function parseSelectedEdgeTurn( +export function parseSelectedEdgeTurn( environment: NodeJS.ProcessEnv, ): SelectedEdgeTurnConfig | undefined { const names = [ diff --git a/tests/turn-allocation-gate.test.ts b/tests/turn-allocation-gate.test.ts new file mode 100644 index 0000000..3aae0fc --- /dev/null +++ b/tests/turn-allocation-gate.test.ts @@ -0,0 +1,48 @@ +import { describe, expect, it } from "vitest"; + +import { parseGateEnvironment, parseGateResult } from "../scripts/turn-allocation-gate"; + +describe("TURN allocation gate", () => { + it("parses credential input without truncating values containing equals", () => { + expect( + parseGateEnvironment( + "SELECTED_EDGE_TURN_URLS=turn:relay.example:3478?transport=udp\n" + + "SELECTED_EDGE_TURN_SHARED_SECRET=secret=value==\n", + ), + ).toEqual({ + SELECTED_EDGE_TURN_URLS: "turn:relay.example:3478?transport=udp", + SELECTED_EDGE_TURN_SHARED_SECRET: "secret=value==", + }); + }); + + it("returns only allowlisted allocation fields from Chrome output", () => { + const secret = "must-not-leak"; + const result = parseGateResult( + '{"status":"passed","relayCandidateCount":1,"protocols":["udp","udp"],' + + `"errorCodeBuckets":[],"credential":"${secret}"}`, + ); + + expect(result).toEqual({ + status: "passed", + relayCandidateCount: 1, + protocols: ["udp"], + errorCodeBuckets: [], + }); + expect(JSON.stringify(result)).not.toContain(secret); + }); + + it("rejects pending or raw Chrome output with a fixed message", () => { + expect(() => parseGateResult('"pending"')).toThrow( + "Chrome returned an invalid TURN allocation result", + ); + expect(() => parseGateResult("credential=must-not-leak")).toThrow( + "Chrome returned an invalid TURN allocation result", + ); + expect(() => parseGateResult( + '{"status":"passed","relayCandidateCount":0,"protocols":[],"errorCodeBuckets":[]}', + )).toThrow("Chrome returned an invalid TURN allocation result"); + expect(() => parseGateResult( + '{"status":"passed","relayCandidateCount":1,"protocols":["tcp"],"errorCodeBuckets":[]}', + )).toThrow("Chrome returned an invalid TURN allocation result"); + }); +}); diff --git a/tsconfig.tools.json b/tsconfig.tools.json index 96706ed..3bc13e9 100644 --- a/tsconfig.tools.json +++ b/tsconfig.tools.json @@ -20,6 +20,7 @@ "vitest.config.ts", "scripts/native-one-viewer-gate.ts", "scripts/native-one-viewer-ledger.ts", + "scripts/turn-allocation-gate.ts", "tests" ] }