feat(nat): add opt-in NAT candidate experiment (#366)

This commit is contained in:
AAA authored and GitHub committed 2026-09-01 13:18:05 +08:00
1 parent ede14d417a
commit 76a6eb7f85
17 files changed
+807 -65

No files matched your search

+2
View File
@@ -45,6 +45,8 @@ ordinary implementation and UI detail.
owned media adaptation. owned media adaptation.
- [ADR-0008](./adr/0008-window-scoped-audio-capture.md): Browser/native window - [ADR-0008](./adr/0008-window-scoped-audio-capture.md): Browser/native window
audio boundary. audio boundary.
- [ADR-0009](./adr/0009-connection-local-nat-prediction-experiment.md): opt-in
connection-local NAT candidate experiment.
## Evidence ## Evidence
@@ -0,0 +1,67 @@
# ADR-0009: Connection-Local NAT Candidate Experiment
- Status: accepted as an opt-in Browser experiment; default off
- Date: 2026-09-01
## Context
The self-hosted deployment has two auxiliary STUN-only listeners for controlled
NAT measurements. A disposable namespace lab showed that a sequential,
endpoint-dependent mapping can sometimes be reached when the remote peer
receives a small set of adjacent server-reflexive candidates. The result is
network- and generation-specific; it is not a reliable NAT type or participant
capability.
The current WebRTC signaling contract already carries ICE candidates. A safe
experiment therefore needs no new message shape, route state, or media path.
## Decision
1. Host Advanced settings exposes `NAT traversal experiment`, disabled by
default and locked while sharing. The choice is local to that page and is
read when each new connection is created; it is not persisted or broadcast
as a room policy.
2. When enabled, Host-originated direct `HostPeer` connections add ports 3479
and 3480 for the same STUN authority already supplied by the server. The
server-provided STUN entries remain unchanged. SFU, ViewerPeer, and Viewer
relay connections keep the stock ICE configuration.
3. For one ICE gathering generation, the adapter observes only UDP `srflx`
candidates. If three or more distinct candidates for one media section and
public address form an arithmetic port sequence, it appends at most eight
bounded candidates around the latest observation (four in each direction).
Every ordinary candidate is retained; no ordinary candidate is rejected or
replaced. A failed or inconclusive experiment uses the existing ICE and SFU
fallback unchanged.
4. Candidate observations remain in memory for that connection only. No NAT
label, raw address, port, score, hard candidate skip, periodic probe, or
route-controller branch is introduced.
## Consequences
Positive:
- operators can test the measured hole-punching mechanism without changing the
default media path;
- normal ICE remains available, so an unsuccessful prediction cannot remove
the existing direct or SFU fallback; and
- the experiment is bounded to one connection generation and one small
candidate set.
Negative:
- the Browser may still reject synthetic candidates or the NAT may not have a
matching mapping; and
- the current implementation covers Host-originated direct edges only. It does
not claim to improve Viewer relay edges or two endpoint-dependent NATs.
The controlled lab result (14/15 sequential-to-restricted runs with a bounded
prediction set) is mechanism evidence, not a production success guarantee.
Target-network prevalence, candidate direction, and long-run resource impact
remain acceptance work.
## Primary Sources
- [RFC 8445: Interactive Connectivity Establishment](https://www.rfc-editor.org/rfc/rfc8445)
- [RFC 4787: UDP NAT behavioral requirements](https://www.rfc-editor.org/rfc/rfc4787)
- [RFC 5780: NAT Behavior Discovery Using STUN](https://www.rfc-editor.org/rfc/rfc5780)
- [W3C WebRTC candidate model](https://www.w3.org/TR/webrtc/)
+6
View File
@@ -33,6 +33,12 @@ share generation:
it before sharing; when disabled, quality evidence stays diagnostic and it before sharing; when disabled, quality evidence stays diagnostic and
availability routing is unchanged. availability routing is unchanged.
Host Advanced settings also contain an independent, default-off NAT traversal
experiment. It only augments Host-originated direct ICE candidates for the
current share generation; it is not a route policy, participant capability, or
SFU preference. [ADR-0009](../adr/0009-connection-local-nat-prediction-experiment.md)
owns its bounded behavior and evidence boundary.
These are route-policy gates, not new routing algorithms. Changing either These are route-policy gates, not new routing algorithms. Changing either
policy requires stopping the current share and starting another generation. policy requires stopping the current share and starting another generation.
+6 -4
View File
@@ -1,6 +1,6 @@
# Project Memory # Project Memory
Last updated: 2026-08-28 Last updated: 2026-09-01
Screener is private, low-latency game screen sharing for one Host and up to 20 Screener is private, low-latency game screen sharing for one Host and up to 20
authenticated friends. The current product surface is Web Host, Web Viewer, and authenticated friends. The current product surface is Web Host, Web Viewer, and
@@ -30,9 +30,11 @@ implementation and routine UI detail.
lease. Production enables optional SQLite stable room authority; live lease. Production enables optional SQLite stable room authority; live
participants, routes, and media remain process-only. participants, routes, and media remain process-only.
- Media is automatic and distributed. Ordinary peers are STUN-only and prefer - Media is automatic and distributed. Ordinary peers are STUN-only and prefer
direct/peer UDP. The only application fallback is one bounded LiveKit SFU/UDP direct/peer UDP. Host may opt into a bounded, connection-local NAT candidate
publication; Screener configures no TURN or TCP media route. A pre-share experiment; the default path remains stock ICE. The only application
peer-only policy can exclude that SFU suffix for one share generation. fallback is one bounded LiveKit SFU/UDP publication; Screener configures no
TURN or TCP media route. A pre-share peer-only policy can exclude that SFU
suffix for one share generation.
- Every endpoint shares one steady outbound-copy cap, default `2` and limited to - Every endpoint shares one steady outbound-copy cap, default `2` and limited to
`1..3`. One room controller owns one committed graph, one reconcile loop, and `1..3`. One room controller owns one committed graph, one reconcile loop, and
one serial child operation. First decoded frame commits availability work; one serial child operation. First decoded frame commits availability work;
+28 -26
View File
@@ -1,6 +1,6 @@
# Browser NAT Traversal # Browser NAT Traversal
Last reviewed: 2026-08-31 Last reviewed: 2026-09-01
This document owns evidence for improving direct Browser ICE without adding a This document owns evidence for improving direct Browser ICE without adding a
new relay or a custom transport. The current product contract remains standard new relay or a custom transport. The current product contract remains standard
@@ -45,24 +45,28 @@ second proves candidate syntax only; it does not prove that a predicted NAT
mapping receives an authenticated connectivity check. The probe retained no raw mapping receives an authenticated connectivity check. The probe retained no raw
address or port. address or port.
This preflight also leaves the proposed field mapper without an ordered port This preflight also leaves the Browser-only mapper without an ordered port
delta. Candidate events expose response completion, while configured STUN order delta. Candidate events expose response completion, while configured STUN order
does not specify the NAT allocation order. A Browser-only mapper may retain a does not specify the NAT allocation order. A disposable namespace matrix then
positive varies-by-destination observation and unordered port gaps, but cannot provided the controlled gate: its four-by-four baseline matched the expected
label a signed `P + d` sequence without controlled external evidence. Trying reachability classes, and bounded prediction improved sequential-to-restricted
both signs consumes more checklist budget and still needs the same lab gate. connectivity in 14/15 runs. Sequential-to-sequential pairs still depended on
candidate ordering and remained sensitive to intervening mappings. The result
The full prediction gate therefore remains unrun. It requires a disposable supports an opt-in mechanism, not a default or a participant-wide classifier.
Linux runner with network namespaces and packet-filter authority, controlled
STUN destinations, and two real Browser ICE agents. The production host is not
that runner, and an unexecuted nftables script is not accepted evidence.
## Current Decision ## Current Decision
Do not change the flagship STUN configuration in this phase. A second endpoint The flagship STUN configuration remains unchanged. The self-hosted endpoint is
does not itself traverse a hard NAT and does not justify a permanent external still the ordinary discovery service; ports 3479 and 3480 are optional
metadata dependency. The self-hosted STUN endpoint remains the production STUN-only listeners for an explicitly enabled Host experiment.
discovery service.
The accepted experiment is deliberately connection-local and additive. It
derives the two auxiliary ports from the existing STUN authority, waits for a
clear three-point arithmetic `srflx` shape in one ICE generation, and appends a
small two-sided candidate window. All ordinary candidates remain available;
there is no NAT label, hard candidate skip, route-controller input, or SFU
preference. The UI switch is default off and locks while sharing. Its exact
scope is recorded in [ADR-0009](../adr/0009-connection-local-nat-prediction-experiment.md).
The Browser-only alternatives do not yet have an accepted implementation: The Browser-only alternatives do not yet have an accepted implementation:
@@ -95,12 +99,11 @@ Consequently Browser observations can support a positive `varies` diagnostic,
but cannot produce the proposed `eim | edm-sequential | edm-random` participant but cannot produce the proposed `eim | edm-sequential | edm-random` participant
fact. They cannot safely remove a Peer candidate or save its five-second window. fact. They cannot safely remove a Peer candidate or save its five-second window.
Port prediction remains the only reachable Browser experiment that might add Port prediction remains an experiment rather than a default path. The
direct paths beyond stock ICE. It requires a controlled sequential/random NAT controlled namespace matrix supports the bounded mechanism, but field
matrix and two real Browser ICE agents before any product design. No such prevalence, direction, and resource impact are not established. The experiment
controlled Browser matrix is currently available, and the production server is therefore stays opt-in, Host-only, and additive until target-network evidence
not an acceptable network-emulation lab. No candidate injection ships without justifies a wider decision.
that gate.
## Rejected Product Inference ## Rejected Product Inference
@@ -118,11 +121,10 @@ type:
Therefore NAT samples must not skip a P2P candidate, alter the committed graph, Therefore NAT samples must not skip a P2P candidate, alter the committed graph,
or become a persistent participant capability. Successful `addIceCandidate()` or become a persistent participant capability. Successful `addIceCandidate()`
parsing would prove only candidate syntax; it would not prove that a predicted parsing proves only candidate syntax; it does not prove that a predicted mapping
mapping receives authenticated ICE checks. Port prediction and synthetic receives authenticated ICE checks. The accepted experiment keeps this boundary:
candidate injection remain experiments until controlled Browser and target- it adds no protocol fields or controller state and falls back to ordinary ICE
network measurements prove both benefit and bounded false rejection. They do when its shape test is inconclusive.
not enter the current protocol or controller.
QUIC on UDP 443 does not change this Browser boundary. WebRTC's ICE agent owns QUIC on UDP 443 does not change this Browser boundary. WebRTC's ICE agent owns
its UDP sockets and candidate checks; the Web application cannot replace that its UDP sockets and candidate checks; the Web application cannot replace that
+4 -1
View File
@@ -1,6 +1,6 @@
# Current Status # Current Status
Last updated: 2026-08-31 Last updated: 2026-09-01
This is the current execution index. Git history owns completed timelines; This is the current execution index. Git history owns completed timelines;
[verification status](./verification-status.md) owns evidence boundaries. [verification status](./verification-status.md) owns evidence boundaries.
@@ -75,6 +75,9 @@ This is the current execution index. Git history owns completed timelines;
sender windows retain the bounded Peer operation; failed exact candidates stay sender windows retain the bounded Peer operation; failed exact candidates stay
consumed until the sender or candidate opportunity changes. The pre-share gate consumed until the sender or candidate opportunity changes. The pre-share gate
defaults on and locks while sharing. defaults on and locks while sharing.
- Host Advanced settings include a default-off, Host-originated NAT traversal
experiment. It adds only bounded, connection-local predicted ICE candidates;
ordinary candidates and the existing SFU fallback remain unchanged.
- Every direct, Browser-relay, and Host SFU video sender owns one clone of its - Every direct, Browser-relay, and Host SFU video sender owns one clone of its
source track; the original remains presentation and source authority only. source track; the original remains presentation and source authority only.
Replacement, rollback, unpublish, physical LiveKit sender recreation, and Replacement, rollback, unpublish, physical LiveKit sender recreation, and
+8 -12
View File
@@ -81,18 +81,14 @@ until their stated boundary is met.
`MediaStream` output contract. Reopen only for a registered custom receiver `MediaStream` output contract. Reopen only for a registered custom receiver
acting as an ordinary Viewer after a named Browser and physical receiver acting as an ordinary Viewer after a named Browser and physical receiver
pass the [platform-output gate](./research/platform-output.md). pass the [platform-output gate](./research/platform-output.md).
14. **NAT inference and predictive candidates.** Keep multi-STUN observations 14. **NAT inference and predictive candidates.** The default-off Host-only
diagnostic until controlled Browser and target-network evidence proves a connection-local experiment is accepted under
connection-scoped classifier with bounded false rejection. Do not hard-skip [ADR-0009](./adr/0009-connection-local-nat-prediction-experiment.md). Keep
Peer candidates from a participant-wide NAT label or inject predicted ports multi-STUN observations and injected candidates additive and diagnostic;
into production ICE without a new accepted decision and a measured gate. See do not make them a participant-wide NAT label, hard candidate skip, route
[Browser NAT traversal](./research/nat-traversal.md). The Browser preflight score, or SFU preference. Validate target-network success, candidate
proves only positive mapping variation and synthetic-candidate syntax; the direction, and bounded resource impact before enabling it by default or
isolated Linux namespace lab now covers the controlled Browser gate. A extending it to Viewer relay edges.
self-hosted same-IP two/three-port endpoint is available for an approved
aggregate-only field survey; keep observations diagnostic until stable
sequential allocation is common enough to justify a new decision. Do not
inject predicted candidates yet.
15. **Unresolved route-state ownership claims.** Reopen lower-revision 15. **Unresolved route-state ownership claims.** Reopen lower-revision
reauthentication, active SFU failure during an unrelated prepare, and reauthentication, active SFU failure during an unrelated prepare, and
multi-child relay-evidence ownership only from an exact current-wire multi-child relay-evidence ownership only from an exact current-wire
+3 -1
View File
@@ -51,7 +51,8 @@ export type Set4Kind =
| "hint-theater" | "hint-theater"
| "hint-theater-exit" | "hint-theater-exit"
| "hint-route-p2p" | "hint-route-p2p"
| "hint-route-sfu"; | "hint-route-sfu"
| "hint-nat-prediction";
export type HintKind = Set1Kind | Set2Kind | Set3Kind | Set4Kind; export type HintKind = Set1Kind | Set2Kind | Set3Kind | Set4Kind;
@@ -88,6 +89,7 @@ export const HINT_KINDS: readonly HintKind[] = [
"hint-theater-exit", "hint-theater-exit",
"hint-route-p2p", "hint-route-p2p",
"hint-route-sfu", "hint-route-sfu",
"hint-nat-prediction",
]; ];
export type HintScene = (props: { theme: ComicTheme }) => ReactNode; export type HintScene = (props: { theme: ComicTheme }) => ReactNode;
@@ -492,6 +492,52 @@ ${rmBlock(
</> </>
); );
/* hint-nat-prediction: [one direct path] -> [a small bounded fan of
alternate direct paths]. The dots stay lightweight so the scene describes
extra ICE chances without suggesting a media relay. */
const HintNatPrediction: HintScene = ({ theme }) => (
<>
<style>{`
.vls-nat-path{stroke-dasharray:1;animation:vlsNatPath 3.2s ease-in-out infinite}
.vls-nat-dots{animation:vlsNatDots 3.2s ease-in-out infinite}
@keyframes vlsNatPath{0%,8%{stroke-dashoffset:1}30%,100%{stroke-dashoffset:0}}
@keyframes vlsNatDots{0%,32%,100%{opacity:.35}44%{opacity:1}}
${rmBlock(
["vls-nat-path", "vls-nat-dots"],
[[".vls-nat-path", "stroke-dashoffset:0"], [".vls-nat-dots", "opacity:1"]],
)}
`}</style>
<Frame x={4} w={152} theme={theme} />
<Frame x={164} w={152} theme={theme} accent={LIVE} />
<Pawn x={42} yb={76} s={9} />
<Pawn x={118} yb={76} s={9} color={SKY} />
<path
d="M51 60 Q80 42 109 60"
stroke={FAINT}
strokeWidth={2.5}
strokeLinecap="round"
fill="none"
/>
<g className="vls-nat-dots" fill={FAINT}>
<circle cx={71} cy={52} r={2} />
<circle cx={82} cy={48} r={2} />
<circle cx={93} cy={52} r={2} />
</g>
<Pawn x={202} yb={76} s={9} eyes />
<Pawn x={278} yb={76} s={9} color={SKY} eyes />
<g className="vls-nat-path" fill="none" stroke={LIVE} strokeWidth={2.5} strokeLinecap="round">
<path d="M211 60 Q240 35 269 60" pathLength={1} />
<path d="M211 64 Q240 52 269 64" pathLength={1} />
</g>
<g className="vls-nat-dots" fill={STAR_GOLD}>
<circle cx={226} cy={48} r={2} />
<circle cx={240} cy={39} r={2.5} />
<circle cx={254} cy={48} r={2} />
</g>
<Spark x={240} y={26} />
</>
);
export const SET4_SCENES: Record<Set4Kind, HintScene> = { export const SET4_SCENES: Record<Set4Kind, HintScene> = {
"hint-topology": HintTopology, "hint-topology": HintTopology,
"hint-close": HintClose, "hint-close": HintClose,
@@ -504,4 +550,5 @@ export const SET4_SCENES: Record<Set4Kind, HintScene> = {
"hint-theater-exit": HintTheaterExit, "hint-theater-exit": HintTheaterExit,
"hint-route-p2p": HintRouteP2p, "hint-route-p2p": HintRouteP2p,
"hint-route-sfu": HintRouteSfu, "hint-route-sfu": HintRouteSfu,
"hint-nat-prediction": HintNatPrediction,
}; };
+3
View File
@@ -157,6 +157,9 @@ export const en: Record<CopyKey, string> = {
"host.advanced.route.peerOnly": "Privacy mode", "host.advanced.route.peerOnly": "Privacy mode",
"host.advanced.route.peerOnlyHint": "host.advanced.route.peerOnlyHint":
"Use P2P routes only; viewers without a P2P path cannot watch", "Use P2P routes only; viewers without a P2P path cannot watch",
"host.advanced.route.natPrediction": "NAT traversal experiment",
"host.advanced.route.natPredictionHint":
"Try extra self-hosted STUN candidates for direct links; ordinary ICE remains available",
"host.advanced.codec": "Video codec", "host.advanced.codec": "Video codec",
"host.advanced.codec.auto": "Auto", "host.advanced.codec.auto": "Auto",
"host.advanced.codec.autoHint": "Pick automatically", "host.advanced.codec.autoHint": "Pick automatically",
+3
View File
@@ -155,6 +155,9 @@ export const zh = {
"host.advanced.route.peerOnly": "隐私模式", "host.advanced.route.peerOnly": "隐私模式",
"host.advanced.route.peerOnlyHint": "host.advanced.route.peerOnlyHint":
"仅使用 P2P 路线,不经过媒体服务器;无可用 P2P 路线时无法观看", "仅使用 P2P 路线,不经过媒体服务器;无可用 P2P 路线时无法观看",
"host.advanced.route.natPrediction": "NAT 打洞实验",
"host.advanced.route.natPredictionHint":
"为直连尝试额外的自托管 STUN 候选;失败时仍使用普通连接",
"host.advanced.codec": "视频编码", "host.advanced.codec": "视频编码",
"host.advanced.codec.auto": "自动", "host.advanced.codec.auto": "自动",
"host.advanced.codec.autoHint": "自动选择", "host.advanced.codec.autoHint": "自动选择",
@@ -23,6 +23,7 @@ interface HostProvisionalPrepareInput extends HostProvisionalInput {
stream: MediaStream; stream: MediaStream;
profile: QualityProfile; profile: QualityProfile;
videoCodec: BrowserVideoCodecPreference; videoCodec: BrowserVideoCodecPreference;
natPredictionEnabled?: boolean;
} }
interface HostProvisionalChildEvents { interface HostProvisionalChildEvents {
@@ -96,6 +97,7 @@ export class HostProvisionalChild {
input: Pick< input: Pick<
HostProvisionalPrepareInput, HostProvisionalPrepareInput,
"iceConfig" | "stream" | "profile" | "videoCodec" "iceConfig" | "stream" | "profile" | "videoCodec"
| "natPredictionEnabled"
>, >,
): void { ): void {
let peer: HostPeer; let peer: HostPeer;
@@ -126,6 +128,7 @@ export class HostProvisionalChild {
}, },
input.videoCodec, input.videoCodec,
candidate.connectionId, candidate.connectionId,
input.natPredictionEnabled ?? false,
); );
this.signalingPeer = peer; this.signalingPeer = peer;
this.prepared = { this.prepared = {
+21
View File
@@ -384,6 +384,7 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
const [routePolicy, setRoutePolicy] = useState<RoutePolicy>( const [routePolicy, setRoutePolicy] = useState<RoutePolicy>(
DEFAULT_ROUTE_POLICY, DEFAULT_ROUTE_POLICY,
); );
const [natPredictionEnabled, setNatPredictionEnabled] = useState(false);
const [videoCodecMode, setVideoCodecMode] = const [videoCodecMode, setVideoCodecMode] =
useState<BrowserVideoCodecMode>("auto"); useState<BrowserVideoCodecMode>("auto");
const [resolvedVideoCodec, setResolvedVideoCodec] = const [resolvedVideoCodec, setResolvedVideoCodec] =
@@ -514,6 +515,7 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
const pendingQualityChangeRef = useRef<QualitySettings | null>(null); const pendingQualityChangeRef = useRef<QualitySettings | null>(null);
const qualitySettingsRef = useRef<QualitySettings>(DEFAULT_QUALITY_SETTINGS); const qualitySettingsRef = useRef<QualitySettings>(DEFAULT_QUALITY_SETTINGS);
const routePolicyRef = useRef<RoutePolicy>(DEFAULT_ROUTE_POLICY); const routePolicyRef = useRef<RoutePolicy>(DEFAULT_ROUTE_POLICY);
const natPredictionEnabledRef = useRef(false);
const advancedQualityRef = useRef<QualitySettings>(advancedQuality); const advancedQualityRef = useRef<QualitySettings>(advancedQuality);
const videoCodecModeRef = useRef<BrowserVideoCodecMode>(videoCodecMode); const videoCodecModeRef = useRef<BrowserVideoCodecMode>(videoCodecMode);
const roomMutationRef = useRef<object | null>(null); const roomMutationRef = useRef<object | null>(null);
@@ -1172,6 +1174,14 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
setRoutePolicy(next); setRoutePolicy(next);
} }
function changeNatPrediction(enabled: boolean): void {
if (phase === "starting" || phase === "live") {
return;
}
natPredictionEnabledRef.current = enabled;
setNatPredictionEnabled(enabled);
}
function changeAdvancedQuality( function changeAdvancedQuality(
patch: Partial<QualitySettings>, patch: Partial<QualitySettings>,
): void { ): void {
@@ -1453,6 +1463,7 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
stream, stream,
profile: qualitySettingsRef.current, profile: qualitySettingsRef.current,
videoCodec: videoCodecRef.current, videoCodec: videoCodecRef.current,
natPredictionEnabled: natPredictionEnabledRef.current,
}); });
} }
@@ -1529,6 +1540,8 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
}, },
}, },
videoCodecRef.current, videoCodecRef.current,
undefined,
natPredictionEnabledRef.current,
); );
peersRef.current.set(peerId, peer); peersRef.current.set(peerId, peer);
let started: boolean; let started: boolean;
@@ -3820,6 +3833,14 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
note={t("host.advanced.route.peerOnlyHint")} note={t("host.advanced.route.peerOnlyHint")}
hint="hint-route-p2p" hint="hint-route-p2p"
/> />
<SwitchItem
checked={natPredictionEnabled}
disabled={phase === "starting" || phase === "live"}
onChange={changeNatPrediction}
label={t("host.advanced.route.natPrediction")}
note={t("host.advanced.route.natPredictionHint")}
hint="hint-nat-prediction"
/>
</div> </div>
</div> </div>
<div className="lr-door-group"> <div className="lr-door-group">
+85 -18
View File
@@ -37,13 +37,14 @@ import {
type BrowserVideoCodecPreference, type BrowserVideoCodecPreference,
VP8_ONLY_VIDEO_CODEC, VP8_ONLY_VIDEO_CODEC,
} from "./video-codec"; } from "./video-codec";
import {
iceServersWithNatPrediction,
NatPredictionCandidateBatch,
type SignalCandidate,
} from "./nat-prediction";
const MAX_PENDING_CANDIDATES = 64; const MAX_PENDING_CANDIDATES = 64;
type PeerIceConfig = Pick<RTCConfiguration, "iceServers">; type PeerIceConfig = Pick<RTCConfiguration, "iceServers">;
type SignalCandidate = Extract<
SignalPayload,
{ kind: "candidate" }
>["candidate"];
interface HostPeerEvents { interface HostPeerEvents {
sendSignal: (peerId: string, payload: SignalPayload) => boolean; sendSignal: (peerId: string, payload: SignalPayload) => boolean;
@@ -55,6 +56,10 @@ export class HostPeer {
private readonly connection: RTCPeerConnection; private readonly connection: RTCPeerConnection;
private readonly pendingCandidates: SignalCandidate[] = []; private readonly pendingCandidates: SignalCandidate[] = [];
private readonly natPredictionEnabled: boolean;
private natCandidateBatch: NatPredictionCandidateBatch | null = null;
private natCandidateUsernameFragment: string | null = null;
private natCandidateEndSent = false;
private statsAccumulator = createStatsAccumulator(); private statsAccumulator = createStatsAccumulator();
private senderVideoTrack: MediaStreamTrack | null; private senderVideoTrack: MediaStreamTrack | null;
private replacementVideoTrack: MediaStreamTrack | null = null; private replacementVideoTrack: MediaStreamTrack | null = null;
@@ -94,8 +99,10 @@ export class HostPeer {
private readonly videoCodec: BrowserVideoCodecPreference = private readonly videoCodec: BrowserVideoCodecPreference =
VP8_ONLY_VIDEO_CODEC, VP8_ONLY_VIDEO_CODEC,
connectionId = createOpaqueId(), connectionId = createOpaqueId(),
natPredictionEnabled = false,
) { ) {
this.connectionId = connectionId; this.connectionId = connectionId;
this.natPredictionEnabled = natPredictionEnabled;
const sourceVideoTrack = stream.getVideoTracks()[0] ?? null; const sourceVideoTrack = stream.getVideoTracks()[0] ?? null;
this.paused = sourceVideoTrack?.enabled === false; this.paused = sourceVideoTrack?.enabled === false;
this.senderVideoTrack = sourceVideoTrack this.senderVideoTrack = sourceVideoTrack
@@ -103,7 +110,10 @@ export class HostPeer {
: null; : null;
this.startupVideoProfilePending = needsStartupVideoProfile(desiredProfile); this.startupVideoProfilePending = needsStartupVideoProfile(desiredProfile);
this.connection = new RTCPeerConnection({ this.connection = new RTCPeerConnection({
iceServers: iceConfig.iceServers, iceServers: iceServersWithNatPrediction(
iceConfig.iceServers,
this.natPredictionEnabled,
),
}); });
this.snapshot = { this.snapshot = {
peerId, peerId,
@@ -399,7 +409,10 @@ export class HostPeer {
} }
try { try {
this.connection.setConfiguration({ this.connection.setConfiguration({
iceServers: iceConfig.iceServers, iceServers: iceServersWithNatPrediction(
iceConfig.iceServers,
this.natPredictionEnabled,
),
}); });
} catch (error) { } catch (error) {
this.setError(error, say("host.err.createConnection")); this.setError(error, say("host.err.createConnection"));
@@ -418,24 +431,61 @@ export class HostPeer {
this.statsTimer = null; this.statsTimer = null;
} }
this.connection.close(); this.connection.close();
this.natCandidateBatch?.discard();
this.natCandidateBatch = null;
this.natCandidateUsernameFragment = null;
this.natCandidateEndSent = false;
this.senderVideoTrack?.stop(); this.senderVideoTrack?.stop();
this.senderVideoTrack = null; this.senderVideoTrack = null;
} }
private bindConnectionEvents(): void { private bindConnectionEvents(): void {
this.connection.addEventListener("icecandidate", (event) => { this.connection.addEventListener("icecandidate", (event) => {
this.events.sendSignal(this.peerId, { const candidate = event.candidate
kind: "candidate", ? {
connectionId: this.connectionId, candidate: event.candidate.candidate,
candidate: event.candidate sdpMid: event.candidate.sdpMid,
? { sdpMLineIndex: event.candidate.sdpMLineIndex,
candidate: event.candidate.candidate, usernameFragment: event.candidate.usernameFragment,
sdpMid: event.candidate.sdpMid, }
sdpMLineIndex: event.candidate.sdpMLineIndex, : null;
usernameFragment: event.candidate.usernameFragment, if (!this.natPredictionEnabled) {
} this.sendIceCandidate(candidate);
: null, return;
}); }
if (candidate === null) {
this.completeNatCandidateBatch();
if (!this.natCandidateEndSent) {
this.sendIceCandidate(null);
this.natCandidateEndSent = true;
}
return;
}
if (
this.natCandidateBatch &&
this.natCandidateUsernameFragment &&
candidate.usernameFragment &&
candidate.usernameFragment !== this.natCandidateUsernameFragment
) {
this.natCandidateBatch.complete();
this.natCandidateBatch = null;
}
if (!this.natCandidateBatch) {
this.natCandidateUsernameFragment = candidate.usernameFragment ?? null;
this.natCandidateEndSent = false;
this.natCandidateBatch = new NatPredictionCandidateBatch((next) =>
this.sendIceCandidate(next),
);
}
this.natCandidateBatch.add(candidate);
});
this.connection.addEventListener("icegatheringstatechange", () => {
if (
this.natPredictionEnabled &&
this.connection.iceGatheringState === "complete"
) {
this.completeNatCandidateBatch();
}
}); });
this.connection.addEventListener("connectionstatechange", () => { this.connection.addEventListener("connectionstatechange", () => {
const state = this.connection.connectionState; const state = this.connection.connectionState;
@@ -457,6 +507,23 @@ export class HostPeer {
this.connection.addEventListener("iceconnectionstatechange", () => this.emit()); this.connection.addEventListener("iceconnectionstatechange", () => this.emit());
} }
private sendIceCandidate(candidate: SignalCandidate | null): void {
this.events.sendSignal(this.peerId, {
kind: "candidate",
connectionId: this.connectionId,
candidate,
});
}
private completeNatCandidateBatch(): void {
if (this.natCandidateBatch) {
this.natCandidateBatch.complete();
this.natCandidateBatch = null;
this.natCandidateUsernameFragment = null;
this.natCandidateEndSent = true;
}
}
private enqueueSenderMutation<T>(operation: () => Promise<T>): Promise<T> { private enqueueSenderMutation<T>(operation: () => Promise<T>): Promise<T> {
const result = this.senderMutationTail.then(operation); const result = this.senderMutationTail.then(operation);
this.senderMutationTail = result.then( this.senderMutationTail = result.then(
+300
View File
@@ -0,0 +1,300 @@
import type { SignalPayload } from "../../shared/protocol";
export type SignalCandidate = Extract<
SignalPayload,
{ kind: "candidate" }
>["candidate"];
const BASE_STUN_PORT = 3478;
const AUXILIARY_STUN_PORTS = [3479, 3480] as const;
const MIN_PREDICTABLE_PORT = 1;
const MAX_PREDICTABLE_PORT = 65_535;
/** Keep the experiment bounded so normal ICE candidates retain their budget. */
export const NAT_PREDICTION_STEPS = 4;
export const MAX_NAT_PREDICTION_CANDIDATES = NAT_PREDICTION_STEPS * 2;
interface SrflxObservation {
candidate: SignalCandidate;
fields: string[];
address: string;
port: number;
groupKey: string;
}
function urlsOf(server: RTCIceServer): readonly string[] {
return Array.isArray(server.urls) ? server.urls : [server.urls];
}
function cloneIceServer(server: RTCIceServer): RTCIceServer {
return {
...server,
urls: Array.isArray(server.urls) ? [...server.urls] : server.urls,
};
}
function auxiliaryUrlsFor(url: string): readonly string[] {
if (!/^stun:/i.test(url)) {
return [];
}
const authority = url.slice(url.indexOf(":") + 1);
let parsed: URL;
try {
parsed = new URL(`http://${authority}`);
} catch {
return [];
}
if (
!parsed.hostname ||
parsed.username ||
parsed.password ||
parsed.pathname !== "/" ||
parsed.search ||
parsed.hash
) {
return [];
}
const port = parsed.port ? Number(parsed.port) : BASE_STUN_PORT;
if (port !== BASE_STUN_PORT) {
return [];
}
const hostname = parsed.hostname.startsWith("[")
? parsed.hostname
: parsed.hostname.includes(":")
? `[${parsed.hostname}]`
: parsed.hostname;
return AUXILIARY_STUN_PORTS.map(
(auxiliaryPort) => `stun:${hostname}:${auxiliaryPort}`,
);
}
/**
* Add the two optional same-host survey listeners without changing the
* server-provided STUN list. The caller opts into this per connection.
*/
export function iceServersWithNatPrediction(
iceServers: readonly RTCIceServer[] | undefined,
enabled: boolean,
): RTCIceServer[] {
const base = (iceServers ?? []).map(cloneIceServer);
if (!enabled) {
return base;
}
const additions = new Set<string>();
const existing = new Set(
(iceServers ?? []).flatMap((server) => urlsOf(server)),
);
for (const server of iceServers ?? []) {
for (const url of urlsOf(server)) {
for (const auxiliaryUrl of auxiliaryUrlsFor(url)) {
if (!existing.has(auxiliaryUrl)) {
additions.add(auxiliaryUrl);
}
}
if (additions.size >= AUXILIARY_STUN_PORTS.length) {
break;
}
}
if (additions.size >= AUXILIARY_STUN_PORTS.length) {
break;
}
}
return [
...base,
...Array.from(additions, (url) => ({ urls: url })),
];
}
function parseSrflxCandidate(
candidate: SignalCandidate,
): SrflxObservation | null {
if (!candidate || !candidate.candidate.trim()) {
return null;
}
const fields = candidate.candidate.trim().split(/\s+/);
const typeIndex = fields.findIndex(
(field) => field.toLowerCase() === "typ",
);
if (
fields.length < 8 ||
!fields[0]?.toLowerCase().startsWith("candidate:") ||
fields[2]?.toLowerCase() !== "udp" ||
typeIndex < 0 ||
fields[typeIndex + 1]?.toLowerCase() !== "srflx"
) {
return null;
}
const address = fields[4];
const port = Number(fields[5]);
if (
!address ||
!Number.isInteger(port) ||
port < MIN_PREDICTABLE_PORT ||
port > MAX_PREDICTABLE_PORT
) {
return null;
}
const groupKey = [
candidate.sdpMid ?? "",
candidate.sdpMLineIndex ?? "",
address,
].join("/");
return { candidate, fields, address, port, groupKey };
}
function distinctObservations(
observations: readonly SrflxObservation[],
): SrflxObservation[] {
const seen = new Set<string>();
return observations.filter((observation) => {
const key = `${observation.groupKey}/${observation.port}`;
if (seen.has(key)) {
return false;
}
seen.add(key);
return true;
});
}
function stableStep(observations: readonly SrflxObservation[]): number | null {
if (observations.length < 3) {
return null;
}
const sorted = [...observations].sort((left, right) => left.port - right.port);
const step = sorted[1]!.port - sorted[0]!.port;
if (
step <= 0 ||
sorted.some(
(observation, index) =>
index > 0 && observation.port - sorted[index - 1]!.port !== step,
)
) {
return null;
}
return step;
}
function predictedFrom(
anchor: SrflxObservation,
step: number,
occupiedPorts: ReadonlySet<number>,
): SignalCandidate[] {
const predicted: SignalCandidate[] = [];
const seenPorts = new Set(occupiedPorts);
for (const direction of [1, -1] as const) {
for (let index = 1; index <= NAT_PREDICTION_STEPS; index += 1) {
const port = anchor.port + direction * step * index;
if (port < MIN_PREDICTABLE_PORT || port > MAX_PREDICTABLE_PORT) {
continue;
}
if (seenPorts.has(port)) {
continue;
}
seenPorts.add(port);
const fields = [...anchor.fields];
fields[0] = `candidate:s${direction > 0 ? "p" : "m"}${index}`;
fields[5] = String(port);
predicted.push({
...anchor.candidate,
candidate: fields.join(" "),
});
}
}
return predicted;
}
/**
* Return a bounded, connection-local prediction only for a clear arithmetic
* srflx shape. No participant classification or candidate rejection happens
* here; callers append the result to the ordinary ICE candidate set.
*/
export function predictSrflxCandidates(
candidates: readonly SignalCandidate[],
): SignalCandidate[] {
const groups = new Map<string, SrflxObservation[]>();
for (const candidate of candidates) {
const observation = parseSrflxCandidate(candidate);
if (!observation) {
continue;
}
const group = groups.get(observation.groupKey) ?? [];
group.push(observation);
groups.set(observation.groupKey, group);
}
for (const group of groups.values()) {
const distinct = distinctObservations(group);
const step = stableStep(distinct);
if (step === null) {
continue;
}
const anchor = distinct.at(-1)!;
return predictedFrom(
anchor,
step,
new Set(distinct.map((observation) => observation.port)),
).slice(0, MAX_NAT_PREDICTION_CANDIDATES);
}
return [];
}
/**
* Holds only srflx candidates until the mapping shape is known, then emits
* predictions before the held ordinary candidates. Non-srflx candidates keep
* their normal trickle timing. A new instance belongs to one ICE generation.
*/
export class NatPredictionCandidateBatch {
private readonly pendingSrflx: SignalCandidate[] = [];
private predictionsSent = false;
private completed = false;
constructor(private readonly send: (candidate: SignalCandidate | null) => void) {}
add(candidate: SignalCandidate): void {
if (this.completed) {
return;
}
if (!parseSrflxCandidate(candidate)) {
this.send(candidate);
return;
}
this.pendingSrflx.push(candidate);
this.trySendPredictions();
}
complete(): void {
if (this.completed) {
return;
}
this.trySendPredictions();
for (const candidate of this.pendingSrflx.splice(0)) {
this.send(candidate);
}
this.send(null);
this.completed = true;
}
discard(): void {
this.completed = true;
this.pendingSrflx.length = 0;
}
private trySendPredictions(): void {
if (this.predictionsSent) {
return;
}
const predictions = predictSrflxCandidates(this.pendingSrflx);
if (predictions.length === 0) {
return;
}
for (const candidate of predictions) {
this.send(candidate);
}
this.predictionsSent = true;
for (const candidate of this.pendingSrflx.splice(0)) {
this.send(candidate);
}
}
}
+82 -3
View File
@@ -17,6 +17,7 @@ import { ViewerRelay } from "../src/client/webrtc/viewer-relay.ts";
import type { import type {
IceConfig, IceConfig,
ParticipantRouteAssignment, ParticipantRouteAssignment,
SignalPayload,
} from "../src/shared/protocol.ts"; } from "../src/shared/protocol.ts";
const codecPreflight = vi.hoisted(() => ({ const codecPreflight = vi.hoisted(() => ({
@@ -109,12 +110,16 @@ class FakePeerConnection {
private releaseRemoteDescription: (() => void) | null = null; private releaseRemoteDescription: (() => void) | null = null;
connectionState: RTCPeerConnectionState = "new"; connectionState: RTCPeerConnectionState = "new";
iceConnectionState: RTCIceConnectionState = "new"; iceConnectionState: RTCIceConnectionState = "new";
iceGatheringState: RTCIceGatheringState = "new";
signalingState: RTCSignalingState = "stable"; signalingState: RTCSignalingState = "stable";
localDescription: RTCSessionDescription | null = null; localDescription: RTCSessionDescription | null = null;
remoteDescription: RTCSessionDescription | null = null; remoteDescription: RTCSessionDescription | null = null;
readonly addedIceCandidates: Array<RTCIceCandidateInit | null> = []; readonly addedIceCandidates: Array<RTCIceCandidateInit | null> = [];
readonly statsReports: Array<RTCStatsReport | Promise<RTCStatsReport>> = []; readonly statsReports: Array<RTCStatsReport | Promise<RTCStatsReport>> = [];
private readonly eventListeners = new Map<string, Array<() => void>>(); private readonly eventListeners = new Map<
string,
Array<(event: Event) => void>
>();
constructor(configuration?: RTCConfiguration) { constructor(configuration?: RTCConfiguration) {
FakePeerConnection.latest = this; FakePeerConnection.latest = this;
@@ -160,7 +165,7 @@ class FakePeerConnection {
} as unknown as RTCRtpTransceiver; } as unknown as RTCRtpTransceiver;
} }
addEventListener(type: string, listener: () => void): void { addEventListener(type: string, listener: (event: Event) => void): void {
const listeners = this.eventListeners.get(type) ?? []; const listeners = this.eventListeners.get(type) ?? [];
listeners.push(listener); listeners.push(listener);
this.eventListeners.set(type, listeners); this.eventListeners.set(type, listeners);
@@ -168,7 +173,7 @@ class FakePeerConnection {
dispatchEvent(event: Event): boolean { dispatchEvent(event: Event): boolean {
for (const listener of this.eventListeners.get(event.type) ?? []) { for (const listener of this.eventListeners.get(event.type) ?? []) {
listener(); listener(event);
} }
return true; return true;
} }
@@ -417,6 +422,7 @@ function createPeer(
iceConfig: IceConfig = { iceServers: [] }, iceConfig: IceConfig = { iceServers: [] },
profile: QualityProfile = QUALITY_PROFILES["720p30"], profile: QualityProfile = QUALITY_PROFILES["720p30"],
videoCodec: BrowserVideoCodecPreference = VP8_ONLY_VIDEO_CODEC, videoCodec: BrowserVideoCodecPreference = VP8_ONLY_VIDEO_CODEC,
natPredictionEnabled = false,
): HostPeer { ): HostPeer {
return new HostPeer( return new HostPeer(
"viewer-peer", "viewer-peer",
@@ -428,6 +434,8 @@ function createPeer(
onUpdate, onUpdate,
}, },
videoCodec, videoCodec,
undefined,
natPredictionEnabled,
); );
} }
@@ -712,6 +720,77 @@ describe("HostPeer source replacement", () => {
} }
}); });
it("keeps ordinary candidates while opting into bounded NAT predictions", () => {
const signals: SignalPayload[] = [];
const peer = new HostPeer(
"viewer-peer",
{ iceServers: [{ urls: "stun:share.example.test:3478" }] },
createStream(createTrack("video", "video"), null),
QUALITY_PROFILES["720p30"],
{
sendSignal: (_peerId, payload) => {
signals.push(payload);
return true;
},
onUpdate: () => undefined,
},
VP8_ONLY_VIDEO_CODEC,
undefined,
true,
);
const connection = FakePeerConnection.latest!;
expect(connection.configurations[0]?.iceServers).toEqual([
{ urls: "stun:share.example.test:3478" },
{ urls: "stun:share.example.test:3479" },
{ urls: "stun:share.example.test:3480" },
]);
const emitCandidate = (port: number | null): void => {
const event = new Event("icecandidate");
Object.defineProperty(event, "candidate", {
value:
port === null
? null
: {
candidate:
`candidate:base 1 udp 2122260223 203.0.113.7 ${port} ` +
"typ srflx raddr 192.0.2.7 rport 50000 generation 0 ufrag test",
sdpMid: "0",
sdpMLineIndex: 0,
usernameFragment: "test",
},
});
connection.dispatchEvent(event);
};
emitCandidate(40_000);
emitCandidate(40_003);
emitCandidate(40_006);
connection.iceGatheringState = "complete";
connection.dispatchEvent(new Event("icegatheringstatechange"));
emitCandidate(null);
const candidateSignals = signals.filter(
(signal): signal is Extract<SignalPayload, { kind: "candidate" }> =>
signal.kind === "candidate",
);
const predictionIndex = candidateSignals.findIndex((signal) =>
signal.candidate?.candidate.includes("candidate:s"),
);
const ordinaryIndex = candidateSignals.findIndex((signal) =>
signal.candidate?.candidate.startsWith("candidate:base"),
);
expect(predictionIndex).toBeGreaterThanOrEqual(0);
expect(predictionIndex).toBeLessThan(ordinaryIndex);
expect(
candidateSignals.filter((signal) =>
signal.candidate?.candidate.startsWith("candidate:base"),
),
).toHaveLength(3);
expect(candidateSignals.at(-1)?.candidate).toBeNull();
expect(candidateSignals.filter((signal) => signal.candidate === null)).toHaveLength(1);
peer.dispose();
});
it("reserves send-only video and audio senders and replaces both tracks", async () => { it("reserves send-only video and audio senders and replaces both tracks", async () => {
const oldVideo = createTrack("video", "old-video"); const oldVideo = createTrack("video", "old-video");
const oldAudio = createTrack("audio", "old-audio"); const oldAudio = createTrack("audio", "old-audio");
+139
View File
@@ -0,0 +1,139 @@
import { describe, expect, it } from "vitest";
import {
iceServersWithNatPrediction,
MAX_NAT_PREDICTION_CANDIDATES,
NatPredictionCandidateBatch,
predictSrflxCandidates,
type SignalCandidate,
} from "../src/client/webrtc/nat-prediction.ts";
function candidate(port: number, address = "203.0.113.7"): SignalCandidate {
return {
candidate:
`candidate:base 1 udp 2122260223 ${address} ${port} ` +
"typ srflx raddr 192.0.2.7 rport 50000 generation 0 ufrag test",
sdpMid: "0",
sdpMLineIndex: 0,
usernameFragment: "test",
};
}
function hostCandidate(): SignalCandidate {
return {
candidate:
"candidate:host 1 udp 2122260223 192.0.2.7 50000 typ host generation 0 ufrag test",
sdpMid: "0",
sdpMLineIndex: 0,
usernameFragment: "test",
};
}
function portOf(value: SignalCandidate): number {
return Number(value?.candidate.trim().split(/\s+/)[5]);
}
describe("NAT prediction ICE adapter", () => {
it("leaves the configured servers unchanged when disabled", () => {
const servers = [
{ urls: ["stun:share.example.test:3478"] },
{ urls: "stun:other.example.test:5349" },
];
expect(iceServersWithNatPrediction(servers, false)).toEqual(servers);
});
it("derives only the two optional same-host survey listeners", () => {
const servers = [
{ urls: ["stun:share.example.test:3478", "stun:share.example.test:3478"] },
{ urls: "stun:other.example.test:3478" },
];
expect(iceServersWithNatPrediction(servers, true)).toEqual([
...servers,
{ urls: "stun:share.example.test:3479" },
{ urls: "stun:share.example.test:3480" },
]);
});
it("does not duplicate already configured auxiliary listeners", () => {
const servers = [
{ urls: "stun:share.example.test:3478" },
{ urls: "stun:share.example.test:3479" },
];
expect(iceServersWithNatPrediction(servers, true)).toEqual([
...servers,
{ urls: "stun:share.example.test:3480" },
]);
});
it("predicts a bounded two-sided window for a stable arithmetic shape", () => {
const predictions = predictSrflxCandidates([
candidate(40_000),
candidate(40_003),
candidate(40_006),
]);
expect(predictions.length).toBeLessThanOrEqual(
MAX_NAT_PREDICTION_CANDIDATES,
);
expect(predictions.map(portOf)).toEqual(
expect.arrayContaining([40_009, 39_997]),
);
expect(predictions.every((value) => value?.sdpMid === "0")).toBe(true);
});
it("does not predict an unstable or incomplete mapping shape", () => {
expect(predictSrflxCandidates([candidate(40_000), candidate(40_001)])).toEqual(
[],
);
expect(
predictSrflxCandidates([
candidate(40_000),
candidate(40_003),
candidate(40_011),
]),
).toEqual([]);
});
it("sends predictions before held srflx candidates while retaining every one", () => {
const sent: Array<SignalCandidate | null> = [];
const batch = new NatPredictionCandidateBatch((value) => sent.push(value));
const first = candidate(40_000);
const second = candidate(40_003);
const third = candidate(40_006);
batch.add(hostCandidate());
batch.add(first);
batch.add(second);
batch.add(third);
batch.complete();
batch.complete();
expect(sent[0]).toEqual(hostCandidate());
const firstOriginal = sent.indexOf(first);
const firstPrediction = sent.findIndex(
(value) => value?.candidate.includes("candidate:s") ?? false,
);
expect(firstPrediction).toBeGreaterThanOrEqual(1);
expect(firstPrediction).toBeLessThan(firstOriginal);
expect(sent.filter((value) => value === first)).toHaveLength(1);
expect(sent.filter((value) => value === second)).toHaveLength(1);
expect(sent.filter((value) => value === third)).toHaveLength(1);
expect(sent.at(-1)).toBeNull();
});
it("flushes ordinary candidates without inventing a prediction", () => {
const sent: Array<SignalCandidate | null> = [];
const batch = new NatPredictionCandidateBatch((value) => sent.push(value));
const first = candidate(40_000);
const second = candidate(40_007);
batch.add(first);
batch.add(second);
batch.complete();
expect(sent).toEqual([first, second, null]);
});
});