diff --git a/docs/adr/0005-automatic-hybrid-media-routing.md b/docs/adr/0005-automatic-hybrid-media-routing.md index ec88c2e..7e90ea9 100644 --- a/docs/adr/0005-automatic-hybrid-media-routing.md +++ b/docs/adr/0005-automatic-hybrid-media-routing.md @@ -2,7 +2,7 @@ - Status: Accepted - Date: 2026-08-20 -- Last updated: 2026-08-28 +- Last updated: 2026-08-30 ## Context @@ -270,6 +270,13 @@ are degraded, one child moves and all remaining native states are observed again; relief can cancel the remaining work without a parent score or explicit capacity penalty. +If a persistently limited edge has no eligible different Peer parent, the same +quality operation appends one same-parent regeneration candidate at the end of +its existing cursor. It creates a new connection and sender generation while +retaining the old edge until first-frame and comparative proof; the candidate +uses the existing steady or one-slot overlap reservation. This is connection +regeneration, not an in-place sender reset and not a new operation reason. + The Host capture/source is a separate fact. A stalled source cannot be repaired by topology. One limited Host-origin edge may try the same measured Peer move. SFU becomes a quality suffix only when every current Host-origin Peer edge, with a @@ -292,9 +299,13 @@ Host-to-SFU ingress proof and each exact SFU-to-Viewer subscription proof remain separate. A successful Viewer canary cannot authorize any other Viewer, and a new publication generation invalidates all prior ingress authority. -After a quality- or root-convergence commit, the new active identity starts from -unknown and must establish a fresh healthy delta before another degradation can -trigger a move. There is no periodic wake, weighted prediction, global +After a quality- or root-convergence reparent commit, the new active identity +starts from unknown and must establish a fresh healthy delta before another +degradation can trigger a move. A same-parent regeneration commit instead +records the exact new sender identity as a same-edge regeneration latch. A +healthy sample or an identity/session change clears that latch; while it is set, +only another same-edge regeneration is suppressed, while different-parent +candidates remain eligible. There is no periodic wake, weighted prediction, global optimizer, persistent parent blacklist, custom congestion controller, or manual SFU layer selection. The result converges gradually to a local stable topology: no current degraded edge has a proved healthy candidate under the current graph diff --git a/docs/product/media-quality.md b/docs/product/media-quality.md index 75640bf..9bd033e 100644 --- a/docs/product/media-quality.md +++ b/docs/product/media-quality.md @@ -85,6 +85,10 @@ representations; Dynacast aggregates demand and server send-side BWE selects subscriber forwarding. AdaptiveStream stays disabled because any Viewer may relay its received track. +When a live native sender remains persistently limited after capacity recovers, +the route controller may rebuild that exact edge with a fresh connection and +clone; media adaptation still stays entirely framework-owned. + Screener does not maintain an application bitrate/resolution ladder, scene detector, periodic quality controller, manual SFU layer selector, or application-defined whole-room lowest-common-denominator target. diff --git a/docs/product/routing-transport.md b/docs/product/routing-transport.md index a54084a..dd13b3c 100644 --- a/docs/product/routing-transport.md +++ b/docs/product/routing-transport.md @@ -122,8 +122,11 @@ retained sender closes. A complete zero-frame current window counts as zero delivery only when an overlapping candidate actually decodes clean video; two zero-frame paths remain unknown. A missing or limited exact candidate sender cannot commit; an expired one-shot -relative proof rejects that candidate. Ordinary quality moves remain -Peer-to-Peer. SFU may + relative proof rejects that candidate. When no different Peer parent is + eligible, the same operation may append one same-parent connection regeneration + candidate; it uses the same first-frame, overlap, and proof rules and is damped + by its exact post-commit sender identity. Ordinary quality moves remain + Peer-to-Peer. SFU may join quality work only as Host fanout relief when every current Host-origin Peer edge, with at least two such edges, is persistently degraded; one Viewer moves before the controller observes the new topology again. During that overlap, the diff --git a/docs/project-memory.md b/docs/project-memory.md index f858cac..17cda0e 100644 --- a/docs/project-memory.md +++ b/docs/project-memory.md @@ -41,8 +41,11 @@ implementation and routine UI detail. layers. Current source defaults persistent native-edge local P2P convergence on, with a pre-share Host opt-out. SFU quality work is limited to multi-root Host fanout relief and each Viewer must prove its own candidate non-regression - before commit. No weighted score, all-pairs probe, general parent-wide - prediction, or periodic rebalance is accepted. + before commit. A persistently limited edge with no different Peer parent may + use one same-parent connection regeneration through that same operation; its + exact post-commit sender identity damps only another same-edge regeneration. + No weighted score, all-pairs probe, general parent-wide prediction, or + periodic rebalance is accepted. - Browser video uses `motion`; each share chooses H.264 with VP8 fallback through an actual sender probe unless the Host explicitly selects VP8 or H264. Screen audio uses 64/128/192 kbps ceilings with 128 default. diff --git a/docs/research/realtime-quality-adaptation.md b/docs/research/realtime-quality-adaptation.md index 76f692f..3eca831 100644 --- a/docs/research/realtime-quality-adaptation.md +++ b/docs/research/realtime-quality-adaptation.md @@ -110,6 +110,27 @@ with the controlled experiments, this supports partial shared-source coupling and track-generation retention; it does not establish a universal all-senders minimum or quantify its share relative to uplink contention. +### Same-edge generation recovery + +On 2026-08-30, a production single-Viewer room provided a controlled comparison. +The Host capture stayed at about 30 fps while one live P2P sender remained at +480x270 and 14-16 fps after its available-outgoing estimate returned above +5 Mbps. The route controller observed three persistent native limited windows, +but had no different Peer candidate and therefore could not start a quality +operation. A Viewer page refresh created a new connection and sender-owned clone +on the same physical path; it climbed from the same source to 1920x1080 and +30 fps within the following windows. This confirms that a live sender generation +can remain adapted after network capacity recovers and that full connection +replacement is the observed escape. The degradation onset occurred while +application evidence was unavailable, so this record does not assign a trigger +to a specific preceding event. + +The accepted route change reuses that existing connection lifecycle: a persistent +limited edge may try one same-parent candidate after ordinary Peer candidates, +retaining the old edge until first-frame and comparative proof. It adds no media +setting, score, periodic rebalance, or new transport. Physical public-network +success rates and long-run retry behavior remain open evidence. + ## H.264 Root Cause And Gate Chromium's `has_trusted_rate_controller` is encoder coordination, not a Windows diff --git a/docs/status.md b/docs/status.md index 074de0e..f74c166 100644 --- a/docs/status.md +++ b/docs/status.md @@ -76,7 +76,10 @@ This is the current execution index. Git history owns completed timelines; clean receive windows that do not regress resolution or rounded FPS. A zero- frame incumbent is zero delivery only against an overlapping candidate that decodes video. Peer candidates stay first; bounded SFU remains only the whole- - Host-root suffix. The pre-share gate defaults on and locks while sharing. + Host-root suffix; when no different Peer is eligible, the same operation may + append one same-parent connection-regeneration candidate. Its old edge remains + until proof, and its exact post-commit sender identity damps only another + same-edge regeneration. The pre-share gate defaults on and locks while sharing. - Every direct, Browser-relay, and Host SFU video sender owns one clone of its source track; the original remains presentation and source authority only. Replacement, rollback, unpublish, physical LiveKit sender recreation, and @@ -112,9 +115,9 @@ This is the current execution index. Git history owns completed timelines; screening did not reproduce a Host-page drop; accepted Viewer lifecycle work prevents frozen JavaScript wall time from becoming an immediate route failure but is not capture keepalive. -- Current-path quality can drive local convergence in source and production. - Both default it on with a pre-share Host opt-out. Weighted/global optimization - remains parked. +- Current-path quality can drive local convergence in source and production, + including bounded same-edge connection regeneration. Both default it on with a + pre-share Host opt-out. Weighted/global optimization remains parked. - Repository simplification does not change or deploy product behavior beyond deleting proven dead private surfaces. - Native/executable senders, shared encode, and distribution packages remain diff --git a/docs/todo.md b/docs/todo.md index 92b6cae..6e44e85 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -12,7 +12,8 @@ until their stated boundary is met. ## Parked Product Work -1. **Broader quality optimization.** Weighted/global optimization, +1. **Broader quality optimization.** Same-edge connection regeneration is now + part of the bounded local convergence path. Weighted/global optimization, parent-wide prediction, startup-limited inference, complete cross-clone source isolation, and alternative SFU policies remain unaccepted. Reopen only from exact path evidence; do not add a score, all-pairs probe, periodic diff --git a/docs/verification-status.md b/docs/verification-status.md index 7d0f647..baa71c1 100644 --- a/docs/verification-status.md +++ b/docs/verification-status.md @@ -1,6 +1,6 @@ # Verification Status -Last updated: 2026-08-27 +Last updated: 2026-08-30 This file owns only cross-module physical evidence that still changes how the current product may be described. [Status](./status.md) owns exact source and @@ -42,9 +42,11 @@ Still required: admission at the 20-Viewer bound. Current controlled Browser evidence supports the H.264/VP8 gate, `motion` -content intent, startup-quality workaround, LiveKit-owned SFU adaptation, and -the native-edge convergence invariants. It does not establish their quality or -resource benefit on heterogeneous public networks. See [media quality](./product/media-quality.md) +content intent, startup-quality workaround, LiveKit-owned SFU adaptation, native +edge convergence, and same-edge connection regeneration invariants. The +2026-08-30 production comparison confirms recovery on a fresh same-route sender, +but does not establish its quality or resource benefit on heterogeneous public +networks. See [media quality](./product/media-quality.md) and [realtime quality research](./research/realtime-quality-adaptation.md). ## Browser Lifecycle diff --git a/src/server/hybrid-media-router.ts b/src/server/hybrid-media-router.ts index 8b421a5..459815e 100644 --- a/src/server/hybrid-media-router.ts +++ b/src/server/hybrid-media-router.ts @@ -2159,7 +2159,9 @@ export class HybridMediaRouter { private debugTuple(roomId: string, tuple: CandidateTuple): string { return tuple.kind === "peer" - ? `p2p:${this.debugPeer(roomId, tuple.parentPeerId)}` + ? `p2p:${this.debugPeer(roomId, tuple.parentPeerId)}${ + tuple.regenerate ? ":regenerate" : "" + }` : `sfu:${tuple.publication}`; } diff --git a/src/server/room-route-controller.ts b/src/server/room-route-controller.ts index 3c4e9cf..e1a80f3 100644 --- a/src/server/room-route-controller.ts +++ b/src/server/room-route-controller.ts @@ -14,7 +14,13 @@ import { import { assertEndpointMediaCopyCapacity } from "../shared/media-copy-accounting.js"; export type CandidateTuple = - | { kind: "peer"; parentPeerId: string; transport: "direct" } + | { + kind: "peer"; + parentPeerId: string; + transport: "direct"; + /** Rebuild this exact edge with a fresh connection and media generation. */ + regenerate?: true; + } | { kind: "sfu"; publication: "reuse" | "create" | "replace" }; export type CandidateReservation = @@ -264,6 +270,14 @@ interface SenderQualityObservation { consumedAtFactVersion?: number; } +interface SenderQualityRegenerationBlock { + childSessionId: string; + parentPeerId: string; + parentSessionId: string; + connectionId: string; + senderIdentity: string; +} + export interface SenderQualityEvidenceInput { parentPeerId: string; parentSessionId: string; @@ -353,6 +367,10 @@ export class RoomRouteController { SenderQualityObservation >(); private readonly senderQualityBaselines = new Map(); + private readonly senderQualityRegenerationBlocks = new Map< + string, + SenderQualityRegenerationBlock + >(); private sfuPublisherQualityObservation?: SfuPublisherQualityObservation; private readonly qualityBaselinesPending = new Set(); private readonly directContinuations = new Map(); @@ -587,6 +605,7 @@ export class RoomRouteController { this.qualityObservations.clear(); this.senderQualityObservations.clear(); this.senderQualityBaselines.clear(); + this.senderQualityRegenerationBlocks.clear(); this.qualityBaselinesPending.clear(); this.sfuPublisherQualityObservation = undefined; this.directContinuations.clear(); @@ -902,6 +921,7 @@ export class RoomRouteController { this.qualityBaselinesPending.clear(); this.senderQualityObservations.clear(); this.senderQualityBaselines.clear(); + this.senderQualityRegenerationBlocks.clear(); this.sfuPublisherQualityObservation = undefined; } this.touchFacts(); @@ -1149,6 +1169,24 @@ export class RoomRouteController { released: [], }; } + const regenerationBlock = this.senderQualityRegenerationBlocks.get( + input.childPeerId, + ); + if ( + regenerationBlock && + !senderQualityRegenerationBlockMatches( + regenerationBlock, + child.sessionId, + input, + ) + ) { + // A new connection or sender identity is a new generation. Do not carry + // a same-edge regeneration latch across that identity fence. + this.senderQualityRegenerationBlocks.delete(input.childPeerId); + } + if (input.state === "healthy") { + this.senderQualityRegenerationBlocks.delete(input.childPeerId); + } const sameFreshSenderIdentity = sameSenderIdentity && previous !== undefined && @@ -1553,6 +1591,7 @@ export class RoomRouteController { // below; a visibility or source reset must not deadlock a persistently // limited edge waiting for a healthy sample. this.senderQualityBaselines.delete(childPeerId); + this.senderQualityRegenerationBlocks.delete(childPeerId); } if (parentPeerId === this.options.hostPeerId) { this.sfuPublisherQualityObservation = undefined; @@ -2292,6 +2331,7 @@ export class RoomRouteController { this.qualityBaselinesPending.clear(); this.senderQualityObservations.clear(); this.senderQualityBaselines.clear(); + this.senderQualityRegenerationBlocks.clear(); this.sfuPublisherQualityObservation = undefined; this.hostPublication = null; this.participants.clear(); @@ -2574,6 +2614,12 @@ export class RoomRouteController { private commitAttempt(operation: ChildOperation, attempt: Attempt): Resource[] { const beforeResources = new Set(this.committedResources()); const old = this.upstreamByViewer.get(operation.childPeerId); + const sameParentRegeneration = + operation.reason === "quality-convergence" && + attempt.tuple.kind === "peer" && + attempt.tuple.regenerate === true && + old?.kind === "peer" && + old.parentPeerId === attempt.tuple.parentPeerId; const createsHostRoot = attempt.tuple.kind === "peer" && attempt.tuple.parentPeerId === this.options.hostPeerId && @@ -2666,9 +2712,27 @@ export class RoomRouteController { this.requireSenderQualityBaseline(operation.childPeerId, true); this.senderQualityBaselines.set( operation.childPeerId, - operation.reason === "quality-convergence" || - operation.reason === "root-convergence", + sameParentRegeneration + ? false + : operation.reason === "quality-convergence" || + operation.reason === "root-convergence", ); + if ( + sameParentRegeneration && + attempt.tuple.kind === "peer" && + attempt.parentSessionId && + attempt.senderQualityIdentity + ) { + this.senderQualityRegenerationBlocks.set(operation.childPeerId, { + childSessionId: operation.childSessionId, + parentPeerId: attempt.tuple.parentPeerId, + parentSessionId: attempt.parentSessionId, + connectionId: attempt.connectionId, + senderIdentity: attempt.senderQualityIdentity, + }); + } else { + this.senderQualityRegenerationBlocks.delete(operation.childPeerId); + } this.revision = attempt.revision; this.operation = undefined; this.pruneRetiringSfuAnchors(); @@ -2867,6 +2931,31 @@ export class RoomRouteController { : "unknown"; } + private senderQualityRegenerationBlocked( + childPeerId: string, + edge: Extract, { kind: "peer" }>, + ): boolean { + const block = this.senderQualityRegenerationBlocks.get(childPeerId); + if (!block) { + return false; + } + const observation = this.senderQualityObservations.get(childPeerId); + if (observation && observation.senderIdentity !== block.senderIdentity) { + this.senderQualityRegenerationBlocks.delete(childPeerId); + return false; + } + if ( + block.childSessionId !== edge.childSessionId || + block.parentPeerId !== edge.parentPeerId || + block.parentSessionId !== edge.parentSessionId || + block.connectionId !== edge.connectionId + ) { + this.senderQualityRegenerationBlocks.delete(childPeerId); + return false; + } + return true; + } + private candidateSenderPersistentlyDegraded( attempt: Attempt, nowMs: number, @@ -2991,7 +3080,23 @@ export class RoomRouteController { this.hostFanoutNeedsSfuRelief(nowMs) ? candidates.filter((candidate) => candidate.tuple.kind === "sfu") : []; - return [...clearPeers, ...remainingPeers, ...sfu]; + const regenerationTuple: CandidateTuple = { + kind: "peer", + parentPeerId: current.parentPeerId, + transport: "direct", + regenerate: true, + }; + const regenerationPlan = + this.senderQualityRegenerationBlocked(childPeerId, current) + ? null + : this.planCandidate(childPeerId, regenerationTuple); + const regeneration = + regenerationPlan && + regenerationPlan.endpointTransition.kind !== "bounded-gap" && + this.candidateValid(childPeerId, regenerationPlan) + ? [regenerationPlan] + : []; + return [...clearPeers, ...remainingPeers, ...sfu, ...regeneration]; } private selectQualityChild(nowMs: number): string | undefined { @@ -3803,7 +3908,16 @@ export class RoomRouteController { operation.current.tuple.kind === "peer" ? operation.current.tuple.parentPeerId : this.options.hostPeerId; + const regeneratingChild = + operation.current.tuple.kind === "peer" && + operation.current.tuple.regenerate === true; for (const childPeerId of this.childrenOf(producerPeerId)) { + if (regeneratingChild && childPeerId === operation.childPeerId) { + // A failed same-edge trial must not turn the retained sender into a + // global baseline block. The operation's consumed fact version + // already supplies one-shot retry damping. + continue; + } this.senderQualityObservations.delete(childPeerId); this.senderQualityBaselines.set(childPeerId, true); } @@ -4241,6 +4355,7 @@ export class RoomRouteController { this.qualityObservations.delete(childPeerId); this.senderQualityObservations.delete(childPeerId); this.senderQualityBaselines.delete(childPeerId); + this.senderQualityRegenerationBlocks.delete(childPeerId); if (this.upstreamByViewer.has(childPeerId)) { this.qualityBaselinesPending.add(childPeerId); this.senderQualityBaselines.set(childPeerId, true); @@ -4588,7 +4703,9 @@ export class RoomRouteController { private debugTuple(tuple: CandidateTuple): string { return tuple.kind === "peer" - ? `p2p:${this.debugPeer(tuple.parentPeerId)}` + ? `p2p:${this.debugPeer(tuple.parentPeerId)}${ + tuple.regenerate ? ":regenerate" : "" + }` : `sfu:${tuple.publication}`; } @@ -4661,7 +4778,23 @@ function safeAdd(total: number, delta: number): number { } function tupleKey(tuple: CandidateTuple): string { - return tuple.kind === "peer" ? `peer:${tuple.parentPeerId}` : `sfu:${tuple.publication}`; + return tuple.kind === "peer" + ? `peer:${tuple.parentPeerId}${tuple.regenerate ? ":regenerate" : ""}` + : `sfu:${tuple.publication}`; +} + +function senderQualityRegenerationBlockMatches( + block: SenderQualityRegenerationBlock, + childSessionId: string, + input: SenderQualityEvidenceInput, +): boolean { + return ( + block.childSessionId === childSessionId && + block.parentPeerId === input.parentPeerId && + block.parentSessionId === input.parentSessionId && + block.connectionId === input.connectionId && + block.senderIdentity === input.senderIdentity + ); } function edgeTupleKey(edge: CommittedEdge): string { diff --git a/tests/host-peer-tracks.test.ts b/tests/host-peer-tracks.test.ts index 6f681f4..d3511fa 100644 --- a/tests/host-peer-tracks.test.ts +++ b/tests/host-peer-tracks.test.ts @@ -1843,6 +1843,53 @@ describe("ViewerRelay downstream ownership", () => { relay.dispose(); }); + it("replaces one active child with a same-parent prepared connection", async () => { + const signals: Array<{ peerId: string; connectionId: string }> = []; + const relay = new ViewerRelay( + { iceServers: [] }, + QUALITY_PROFILES["720p30"], + { + sendSignal: (peerId, payload) => { + signals.push({ peerId, connectionId: payload.connectionId }); + return true; + }, + }, + ); + relay.setStream(createStream(createTrack("video", "same-parent-video"), null)); + expect(relay.prepareChild(1, routeCandidate(1, "same-child"), ["same-child"])).toBe(true); + await vi.waitFor(() => expect(signals).toHaveLength(1)); + const firstConnection = FakePeerConnection.latest!; + await relay.acceptSignal("same-child", { + kind: "description", + connectionId: signals[0]!.connectionId, + description: { type: "answer", sdp: "first-answer" }, + }, 1); + firstConnection.connectionState = "connected"; + relay.activateChildren(1, ["same-child"]); + expect(relay.getSnapshot("same-child")?.connectionId).toBe( + signals[0]!.connectionId, + ); + + expect(relay.prepareChild(2, routeCandidate(2, "same-child"), ["same-child"])).toBe(true); + await vi.waitFor(() => expect(signals).toHaveLength(2)); + const secondConnection = FakePeerConnection.latest!; + expect(firstConnection.connectionState).toBe("connected"); + await relay.acceptSignal("same-child", { + kind: "description", + connectionId: signals[1]!.connectionId, + description: { type: "answer", sdp: "second-answer" }, + }, 2); + secondConnection.connectionState = "connected"; + relay.activateChildren(2, ["same-child"]); + + expect(firstConnection.connectionState).toBe("closed"); + expect(secondConnection.connectionState).toBe("connected"); + expect(relay.getSnapshot("same-child")?.connectionId).toBe( + signals[1]!.connectionId, + ); + relay.dispose(); + }); + it("cleans prepared children on failure, replacement, rollback, session reset, and share stop", async () => { const relay = new ViewerRelay( { iceServers: [] }, diff --git a/tests/hybrid-media-router.test.ts b/tests/hybrid-media-router.test.ts index 22f3b68..d60c073 100644 --- a/tests/hybrid-media-router.test.ts +++ b/tests/hybrid-media-router.test.ts @@ -899,6 +899,87 @@ describe("HybridMediaRouter v9 runtime", () => { } }); + it("prepares a same-parent connection regeneration after persistent degradation", async () => { + const { store, sent, router } = harness(2); + try { + const room = await store.createRoom(); + const host = connectHost(store, room, { + peerOnly: false, + topologyOptimization: true, + }); + complete(router, host); + const viewer = connectViewer(store, room, "quality-regeneration"); + complete(router, viewer); + await vi.waitFor(() => + expect(preparedFor(sent, viewer.sessionId)).toBeDefined(), + ); + const initial = preparedFor(sent, viewer.sessionId)!; + expect(initial.assignment.upstream).toEqual({ + kind: "peer", + peerId: host.peerId, + }); + router.handleRouteReady(viewer, { + type: "route-ready", + revision: initial.revision, + phase: "prepare", + }); + const edge = router.resolveActiveViewerMediaEdge( + room.roomId, + viewer.peerId, + )!; + + router.observeSenderQualityEvidence(host, { + type: "sender-quality-evidence", + childPeerId: viewer.peerId, + connectionId: edge.connectionId, + rtpStatsId: "old-sender-rtp", + trackIdentifier: "old-track", + sampleTimestampMs: 100, + routeRevision: edge.revision, + state: "healthy", + diagnostics: senderDiagnostics("healthy"), + }); + for (const sampleTimestampMs of [101, 102, 103]) { + router.observeSenderQualityEvidence(host, { + type: "sender-quality-evidence", + childPeerId: viewer.peerId, + connectionId: edge.connectionId, + rtpStatsId: "old-sender-rtp", + trackIdentifier: "old-track", + sampleTimestampMs, + routeRevision: edge.revision, + state: "degraded", + diagnostics: senderDiagnostics("degraded"), + }); + } + + await vi.waitFor(() => { + const prepared = preparedFor(sent, viewer.sessionId); + expect(prepared?.revision).toBeGreaterThan(edge.revision); + }); + const regeneration = preparedFor(sent, viewer.sessionId)!; + expect(regeneration.candidate).toMatchObject({ + transport: "direct", + qualityProbe: true, + }); + expect(regeneration.assignment.upstream).toEqual({ + kind: "peer", + peerId: host.peerId, + }); + expect(preparedFor(sent, host.sessionId)?.candidate.connectionId).toBe( + regeneration.candidate.connectionId, + ); + expect( + router.resolveActiveViewerMediaEdge(room.roomId, viewer.peerId), + ).toMatchObject({ + connectionId: edge.connectionId, + upstream: { kind: "peer", peerId: host.peerId }, + }); + } finally { + await router.close(); + } + }); + it("keeps peer-only shares out of configured SFU fallback", async () => { const { store, sent, router } = harness(1, true); try { diff --git a/tests/room-route-controller.test.ts b/tests/room-route-controller.test.ts index 3f53682..9a392b4 100644 --- a/tests/room-route-controller.test.ts +++ b/tests/room-route-controller.test.ts @@ -563,6 +563,201 @@ describe("RoomRouteController", () => { }); }); + it("regenerates a persistently degraded edge on the same parent", () => { + const routes = controller(2, { qualityConvergenceEnabled: true }); + addViewer(routes, A, 2); + routes.hydrateEdge(A, peerEdge(HOST, "a_from_host")); + observePersistentDegraded(routes, A, "a_from_host", 10); + + const operation = routes.reconcile(14).operation!; + expect(operation.candidates).toEqual([ + { + tuple: { + kind: "peer", + parentPeerId: HOST, + transport: "direct", + regenerate: true, + }, + endpointTransition: { kind: "none", producerPeerId: HOST }, + }, + ]); + + const begin = beginCandidate(routes, { + nowMs: 15, + connectionId: "a_regenerated", + reservation: { kind: "direct" }, + }).operation!; + expect(routes.snapshot().upstreamByViewer.get(A)).toMatchObject({ + parentPeerId: HOST, + connectionId: "a_from_host", + usable: true, + physicalActive: true, + }); + const guard: CandidateGuard = { + childPeerId: A, + childSessionId: `${A}_session`, + revision: begin.current!.revision, + connectionId: "a_regenerated", + }; + expect( + routes.candidateReady(guard, 16, undefined, { + relativeQualityApproved: true, + }), + ).toMatchObject({ accepted: true, committed: false }); + + expect( + routes.observeSenderQualityEvidence({ + parentPeerId: HOST, + parentSessionId: "host_session", + childPeerId: A, + routeRevision: begin.current!.revision, + connectionId: "a_regenerated", + senderIdentity: "a-regenerated-rtp\u0000track", + sampleTimestampMs: 17, + state: "healthy", + acceptedAtMs: 17, + }).committed, + ).toBe(true); + expect(routes.snapshot().upstreamByViewer.get(A)).toMatchObject({ + kind: "peer", + parentPeerId: HOST, + connectionId: "a_regenerated", + }); + expect( + observeSenderState(routes, { + childPeerId: A, + connectionId: "a_regenerated", + senderIdentity: "a-regenerated-rtp\u0000track", + state: "healthy", + acceptedAtMs: 18, + }).accepted, + ).toBe(true); + for (const acceptedAtMs of [19, 20, 21]) { + expect( + observeSenderState(routes, { + childPeerId: A, + connectionId: "a_regenerated", + senderIdentity: "a-regenerated-rtp\u0000track", + state: "degraded", + acceptedAtMs, + }).accepted, + ).toBe(true); + } + expect(routes.reconcile(22).operation?.candidates[0]?.tuple).toMatchObject({ + kind: "peer", + parentPeerId: HOST, + regenerate: true, + }); + }); + + it("keeps same-edge regeneration damped while allowing a different parent", () => { + const routes = controller(2, { qualityConvergenceEnabled: true }); + addViewer(routes, A, 2); + routes.hydrateEdge(A, peerEdge(HOST, "a_from_host")); + observePersistentDegraded(routes, A, "a_from_host", 10); + + routes.reconcile(14); + const begin = beginCandidate(routes, { + nowMs: 15, + connectionId: "a_regenerated", + reservation: { kind: "direct" }, + }).operation!; + const guard: CandidateGuard = { + childPeerId: A, + childSessionId: `${A}_session`, + revision: begin.current!.revision, + connectionId: "a_regenerated", + }; + routes.candidateReady(guard, 16, undefined, { + relativeQualityApproved: true, + }); + expect( + routes.observeSenderQualityEvidence({ + parentPeerId: HOST, + parentSessionId: "host_session", + childPeerId: A, + routeRevision: begin.current!.revision, + connectionId: "a_regenerated", + senderIdentity: "a-regenerated-rtp\u0000track", + sampleTimestampMs: 17, + state: "healthy", + acceptedAtMs: 17, + }).committed, + ).toBe(true); + + for (const acceptedAtMs of [18, 19, 20]) { + expect( + observeSenderState(routes, { + childPeerId: A, + connectionId: "a_regenerated", + state: "degraded", + senderIdentity: "a-regenerated-rtp\u0000track", + acceptedAtMs, + }).accepted, + ).toBe(true); + } + expect(routes.reconcile(21).operation).toBeUndefined(); + + addViewer(routes, B, 2); + routes.hydrateEdge(B, peerEdge(HOST, "b_from_host")); + expect( + observeSenderState(routes, { + childPeerId: B, + connectionId: "b_from_host", + state: "healthy", + acceptedAtMs: 22, + }).accepted, + ).toBe(true); + const withAlternate = routes.reconcile(23).operation; + expect(withAlternate).toMatchObject({ + childPeerId: A, + reason: "quality-convergence", + }); + expect(withAlternate?.candidates.map((candidate) => candidate.tuple)).toEqual([ + { kind: "peer", parentPeerId: B, transport: "direct" }, + ]); + }); + + it("uses the existing one-slot overlap for same-parent regeneration", () => { + const routes = controller(1, { qualityConvergenceEnabled: true }); + addViewer(routes, A, 1); + routes.hydrateEdge(A, peerEdge(HOST, "a_from_host")); + observePersistentDegraded(routes, A, "a_from_host", 10); + + const operation = routes.reconcile(14).operation!; + expect(operation.candidates[0]?.endpointTransition).toEqual({ + kind: "overlap", + producerPeerId: HOST, + }); + const begin = beginCandidate(routes, { + nowMs: 15, + connectionId: "a_regenerated", + reservation: { kind: "direct", overlap: "a_overlap" }, + }).operation!; + const guard: CandidateGuard = { + childPeerId: A, + childSessionId: `${A}_session`, + revision: begin.current!.revision, + connectionId: "a_regenerated", + }; + routes.candidateReady(guard, 16, undefined, { + relativeQualityApproved: true, + }); + const committed = routes.observeSenderQualityEvidence({ + parentPeerId: HOST, + parentSessionId: "host_session", + childPeerId: A, + routeRevision: begin.current!.revision, + connectionId: "a_regenerated", + senderIdentity: "a-regenerated-rtp\u0000track", + sampleTimestampMs: 17, + state: "healthy", + acceptedAtMs: 17, + }); + expect(committed.committed).toBe(true); + expect(committed.released).toContain("a_overlap"); + }); + it("keeps isolated limitation windows diagnostic", () => { const routes = controller(2, { qualityConvergenceEnabled: true }); addViewer(routes, A, 2); @@ -650,7 +845,14 @@ describe("RoomRouteController", () => { routes.snapshot().revision, 103, ); - expect(routes.reconcile(104).operation).toBeUndefined(); + const operation = routes.reconcile(104).operation!; + expect(operation.reason).toBe("quality-convergence"); + expect(operation.candidates).toHaveLength(1); + expect(operation.candidates[0]?.tuple).toMatchObject({ + kind: "peer", + parentPeerId: HOST, + regenerate: true, + }); expect(routes.snapshot().upstreamByViewer.get(A)).toMatchObject({ kind: "peer", parentPeerId: HOST, @@ -1093,7 +1295,10 @@ describe("RoomRouteController", () => { acceptedAtMs: 5_111, }); expect(settled).toMatchObject({ accepted: true, committed: false }); - expect(routes.snapshot().operation).toBeUndefined(); + expect(routes.snapshot().operation).toMatchObject({ + reason: "quality-convergence", + cursor: 1, + }); expect(routes.snapshot().upstreamByViewer.get(A)).toMatchObject({ parentPeerId: HOST, connectionId: "a_from_host", @@ -1517,6 +1722,14 @@ describe("RoomRouteController", () => { { tuple: { kind: "peer", parentPeerId: B, transport: "direct" }, }, + { + tuple: { + kind: "peer", + parentPeerId: HOST, + transport: "direct", + regenerate: true, + }, + }, ], }); }); @@ -1660,13 +1873,40 @@ describe("RoomRouteController", () => { routes.hydrateEdge(A, peerEdge(HOST, "a_from_host")); routes.hydrateEdge(B, peerEdge(HOST, "b_from_host")); observePersistentDegraded(routes, A, "a_from_host", 99); - expect(routes.reconcile(103).operation).toBeUndefined(); + const firstRegeneration = routes.reconcile(103).operation!; + expect(firstRegeneration.candidates[0]?.tuple).toMatchObject({ + kind: "peer", + parentPeerId: HOST, + regenerate: true, + }); + const firstAttempt = beginCandidate(routes, { + nowMs: 104, + connectionId: "a_regeneration_failed", + reservation: { kind: "direct", overlap: "a_regeneration_overlap" }, + }).operation!; + expect( + routes.candidateFailed( + { + childPeerId: A, + childSessionId: `${A}_session`, + revision: firstAttempt.current!.revision, + connectionId: "a_regeneration_failed", + }, + 105, + ).accepted, + ).toBe(true); observePersistentDegraded(routes, B, "b_from_host", 104); const operation = routes.reconcile(108).operation!; const childPeerId = operation.childPeerId; expect(operation.candidates.map((candidate) => candidate.tuple)).toEqual([ { kind: "peer", parentPeerId: A, transport: "direct" }, { kind: "sfu", publication: "create" }, + { + kind: "peer", + parentPeerId: HOST, + transport: "direct", + regenerate: true, + }, ]); const direct = beginCandidate(routes, { nowMs: 109, @@ -1752,7 +1992,28 @@ describe("RoomRouteController", () => { routes.hydrateEdge(A, peerEdge(HOST, "a_from_host")); routes.hydrateEdge(B, peerEdge(HOST, "b_from_host")); observePersistentDegraded(routes, A, "a_from_host", 99); - expect(routes.reconcile(103).operation).toBeUndefined(); + const firstRegeneration = routes.reconcile(103).operation!; + expect(firstRegeneration.candidates[0]?.tuple).toMatchObject({ + kind: "peer", + parentPeerId: HOST, + regenerate: true, + }); + const firstAttempt = beginCandidate(routes, { + nowMs: 104, + connectionId: "a_regeneration_failed", + reservation: { kind: "direct", overlap: "a_regeneration_overlap" }, + }).operation!; + expect( + routes.candidateFailed( + { + childPeerId: A, + childSessionId: `${A}_session`, + revision: firstAttempt.current!.revision, + connectionId: "a_regeneration_failed", + }, + 105, + ).accepted, + ).toBe(true); observePersistentDegraded(routes, B, "b_from_host", 104); const operation = routes.reconcile(108).operation!; const childPeerId = operation.childPeerId; @@ -1808,7 +2069,7 @@ describe("RoomRouteController", () => { acceptedAtMs: 113, }), ).toMatchObject({ accepted: true, committed: false }); - expect(routes.snapshot().operation).toBeUndefined(); + expect(routes.reconcile(114).operation).toBeUndefined(); expect(routes.snapshot().upstreamByViewer.get(childPeerId)).toMatchObject({ kind: "peer", parentPeerId: HOST,