feat(web): request window-scoped display audio

Merge browser window-audio hint and honest scope status; retain native WASAPI boundary in ADR-0008.
This commit is contained in:
AAA authored and GitHub committed 2026-08-21 01:01:22 +08:00
1 parent e87498d414
commit 3738db6e51
7 files changed
+184 -5

No files matched your search

+1
View File
@@ -24,5 +24,6 @@
- [ADR-0005](./adr/0005-automatic-hybrid-media-routing.md): accepted direct/peer UDP -> SFU-root -> optional exceptional-edge TURN direction, with the room-`1` exact smoke and broader migration gates.
- [ADR-0006](./adr/0006-fixed-high-native-sender-canary.md): proposed fixed-`HIGH` native canary, current no-go result, and staged revalidation boundary.
- [ADR-0007](./adr/0007-demand-driven-dual-representation-quality.md): accepted path-isolated `HIGH + at most one LOW` policy, optional idle-layer stop, evidence contract, and SVC boundary.
- [ADR-0008](./adr/0008-window-scoped-audio-capture.md): browser window-audio hint and the Windows WASAPI process-loopback boundary.
Documentation and project memory are part of the product source of truth. Update them in the same change that alters the corresponding requirement or architecture.
@@ -0,0 +1,79 @@
# ADR-0008: Window-Scoped Audio Capture
Date: 2026-08-21
Status: Accepted for the browser hint; native Windows implementation is a
planned P1 input.
## Context
The Web Host currently receives an optional audio track from
`getDisplayMedia({ audio: true })`. A browser may return no audio, and the Web
API does not expose a portable readback identifying whether that track is a
tab, window, monitor, or whole-system mix. Capturing a system mix while a voice
application is open can leak that conversation back to Viewers.
Windows has a stricter primitive: WASAPI application loopback through
`ActivateAudioInterfaceAsync` with
`AUDIOCLIENT_PROCESS_LOOPBACK_PARAMS.INCLUDE_TARGET_PROCESS_TREE`. It can be
paired with a selected window's PID and a WGC/DXGI video capture. The API is
available on Windows 11 build 20348 and later; this is not a Windows 10
fallback.
## Decision
1. The Web sender requests Chromium's best-effort window-audio hints,
`windowAudio: "window"` and `systemAudio: "exclude"`. The latter avoids
offering the whole-system source in Chromium's picker. Older browsers may
ignore these dictionary members. The UI labels a returned track as
**window requested, scope unconfirmed** and warns that system audio may
still be present. It never retries with a wider source after a capture
failure and never claims isolation from a track's presence alone.
2. The default Web route remains unchanged in topology, codecs, and relay
behavior. No Web Audio mixer, SDP rewrite, or app-owned audio bitrate knob
is introduced by this ADR.
3. The native sender's next audio slice is an explicit opt-in
`window-process-audio` mode on Windows 11. The local capture session owns
one selected target PID/process tree, one WGC/DXGI video source, and one
WASAPI loopback audio source. Process IDs, titles, paths, device identity,
and PCM stay local; they do not enter signaling, URLs, logs, or storage.
4. If the target process exits, produces no render stream, or the loopback
activation is denied, the sender reports `audio unavailable` or `audio
silent` and stops/asks the user. It never widens to whole-system audio.
Windows 10 remains explicitly unsupported until a separate API passes the
same isolation gate.
## Minimal Native Boundary
This is an interface boundary, not a new framework or a current wire change:
```text
NativeCaptureTarget { pid: uint32, includeProcessTree: true }
NativeCaptureSession.start(target)
-> VideoFrames(WGC/DXGI) + AudioPcm(WASAPI process loopback)
-> existing native sender timeline/encoder/fanout
NativeCaptureSession.stop()
```
The first fixture should prove only parent-game plus child-game audio, an
independent voice process, a notification, process restart, no render stream,
and A/V sync on one Viewer. It should use Microsoft's Application Loopback
sample and process-loopback parameter contract rather than a custom mixer.
## Consequences
- Current Web users get a useful Chromium hint and an honest status without a
privacy claim the platform cannot support.
- Strict per-window audio is a native Windows 11 feature, not a browser promise
or a silent system-audio fallback.
- Android `AudioPlaybackCapture` and packaged cross-platform UI remain later
candidates; they do not block this slice.
## Follow-Up TODO
- Add a Windows-only WGC + WASAPI process-loopback fixture beside the existing
Media Foundation H.264 fixture.
- Feed its timestamped PCM/video pair into the existing native sender only
after the isolation and sync checks pass.
- Retain a short matrix result in `docs/research/`; do not enable the mode by
default or deploy it from this ADR alone.
+44
View File
@@ -0,0 +1,44 @@
/**
* Best-effort display-audio hints for Chromium. The Screen Capture API does
* not expose a portable readback for the selected audio source, so callers
* must keep the requested scope separate from the observed track presence.
*/
export type BrowserAudioScope = "window-requested-unconfirmed" | "none";
export interface BrowserAudioCaptureStatus {
hasTrack: boolean;
scope: BrowserAudioScope;
}
type DisplayMediaAudioHints = DisplayMediaStreamOptions & {
systemAudio?: "include" | "exclude";
windowAudio?: "exclude" | "window" | "system";
};
/**
* Ask Chromium to prefer audio from the selected display surface while
* excluding the whole-system source. Unknown dictionary members are ignored
* by older browsers; no fallback is added because silently widening to
* whole-system audio would leak sound.
*/
export function displayMediaOptions(
video: MediaTrackConstraints,
): DisplayMediaStreamOptions {
return {
video,
audio: true,
systemAudio: "exclude",
windowAudio: "window",
} as DisplayMediaAudioHints;
}
export function browserAudioCaptureStatus(
stream: MediaStream,
): BrowserAudioCaptureStatus {
const hasTrack = stream.getAudioTracks().length > 0;
return {
hasTrack,
scope: hasTrack ? "window-requested-unconfirmed" : "none",
};
}
+4 -4
View File
@@ -5,6 +5,7 @@ import {
type QualityResolution,
type QualitySettings,
} from "../../shared/protocol";
import { displayMediaOptions } from "./audio-capture";
export type {
DegradationPreference,
@@ -127,10 +128,9 @@ export async function captureDisplay(
throw new Error("当前浏览器不支持屏幕共享");
}
const stream = await navigator.mediaDevices.getDisplayMedia({
video: captureConstraints(profile),
audio: true,
});
const stream = await navigator.mediaDevices.getDisplayMedia(
displayMediaOptions(captureConstraints(profile)),
);
const videoTrack = stream.getVideoTracks()[0];
if (!videoTrack) {
+14 -1
View File
@@ -60,6 +60,10 @@ import {
} from "../lib/session";
import { SignalingClient } from "../lib/signaling";
import { labelViewerPresence } from "../lib/viewer-presence";
import {
browserAudioCaptureStatus,
type BrowserAudioCaptureStatus,
} from "../media/audio-capture";
import {
applyCaptureProfile,
captureDisplay,
@@ -106,17 +110,20 @@ interface CaptureDetails {
resolution: string;
frameRate: number | null;
hasAudio: boolean;
audio: BrowserAudioCaptureStatus;
}
function captureDetails(stream: MediaStream): CaptureDetails {
const settings = stream.getVideoTracks()[0]?.getSettings();
const audio = browserAudioCaptureStatus(stream);
return {
resolution:
settings?.width && settings.height
? `${settings.width}x${settings.height}`
: "未知",
frameRate: settings?.frameRate ?? null,
hasAudio: stream.getAudioTracks().length > 0,
hasAudio: audio.hasTrack,
audio,
};
}
@@ -1631,6 +1638,12 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
{!details?.hasAudio && stream && (
<WarningBanner>当前来源没有可共享音频</WarningBanner>
)}
{details?.audio.scope === "window-requested-unconfirmed" && stream && (
<WarningBanner>
已请求所选窗口音频,但浏览器没有提供来源范围确认;当前音频仍可能包含系统混音。
需要严格窗口独占时请使用 Windows 原生发送器。
</WarningBanner>
)}
{qualityLimitation && (
<WarningBanner>{qualityLimitation}</WarningBanner>
)}
+40
View File
@@ -0,0 +1,40 @@
import { describe, expect, it } from "vitest";
import {
browserAudioCaptureStatus,
displayMediaOptions,
} from "../src/client/media/audio-capture.ts";
describe("browser display audio capture", () => {
it("requests the selected window audio without claiming source readback", () => {
const options = displayMediaOptions({ width: { ideal: 1280 } });
expect(options).toMatchObject({
audio: true,
systemAudio: "exclude",
windowAudio: "window",
});
});
it("reports an audio track while keeping its scope unconfirmed", () => {
const stream = {
getAudioTracks: () => [{}],
} as unknown as MediaStream;
expect(browserAudioCaptureStatus(stream)).toEqual({
hasTrack: true,
scope: "window-requested-unconfirmed",
});
});
it("reports a missing audio track explicitly", () => {
const stream = {
getAudioTracks: () => [],
} as unknown as MediaStream;
expect(browserAudioCaptureStatus(stream)).toEqual({
hasTrack: false,
scope: "none",
});
});
});
+2
View File
@@ -47,6 +47,8 @@ describe("realtime quality controls", () => {
frameRate: { ideal: 30, max: 30 },
},
audio: true,
systemAudio: "exclude",
windowAudio: "window",
});
expect(videoTrack.contentHint).toBe("motion");
});