feat(web): request window-scoped display audio
Merge browser window-audio hint and honest scope status; retain native WASAPI boundary in ADR-0008.
This commit is contained in:
1 parent
e87498d414
commit
3738db6e51
7 files changed
+184
-5
No files matched your search
@@ -24,5 +24,6 @@
|
||||
- [ADR-0005](./adr/0005-automatic-hybrid-media-routing.md): accepted direct/peer UDP -> SFU-root -> optional exceptional-edge TURN direction, with the room-`1` exact smoke and broader migration gates.
|
||||
- [ADR-0006](./adr/0006-fixed-high-native-sender-canary.md): proposed fixed-`HIGH` native canary, current no-go result, and staged revalidation boundary.
|
||||
- [ADR-0007](./adr/0007-demand-driven-dual-representation-quality.md): accepted path-isolated `HIGH + at most one LOW` policy, optional idle-layer stop, evidence contract, and SVC boundary.
|
||||
- [ADR-0008](./adr/0008-window-scoped-audio-capture.md): browser window-audio hint and the Windows WASAPI process-loopback boundary.
|
||||
|
||||
Documentation and project memory are part of the product source of truth. Update them in the same change that alters the corresponding requirement or architecture.
|
||||
@@ -0,0 +1,79 @@
|
||||
# ADR-0008: Window-Scoped Audio Capture
|
||||
|
||||
Date: 2026-08-21
|
||||
|
||||
Status: Accepted for the browser hint; native Windows implementation is a
|
||||
planned P1 input.
|
||||
|
||||
## Context
|
||||
|
||||
The Web Host currently receives an optional audio track from
|
||||
`getDisplayMedia({ audio: true })`. A browser may return no audio, and the Web
|
||||
API does not expose a portable readback identifying whether that track is a
|
||||
tab, window, monitor, or whole-system mix. Capturing a system mix while a voice
|
||||
application is open can leak that conversation back to Viewers.
|
||||
|
||||
Windows has a stricter primitive: WASAPI application loopback through
|
||||
`ActivateAudioInterfaceAsync` with
|
||||
`AUDIOCLIENT_PROCESS_LOOPBACK_PARAMS.INCLUDE_TARGET_PROCESS_TREE`. It can be
|
||||
paired with a selected window's PID and a WGC/DXGI video capture. The API is
|
||||
available on Windows 11 build 20348 and later; this is not a Windows 10
|
||||
fallback.
|
||||
|
||||
## Decision
|
||||
|
||||
1. The Web sender requests Chromium's best-effort window-audio hints,
|
||||
`windowAudio: "window"` and `systemAudio: "exclude"`. The latter avoids
|
||||
offering the whole-system source in Chromium's picker. Older browsers may
|
||||
ignore these dictionary members. The UI labels a returned track as
|
||||
**window requested, scope unconfirmed** and warns that system audio may
|
||||
still be present. It never retries with a wider source after a capture
|
||||
failure and never claims isolation from a track's presence alone.
|
||||
2. The default Web route remains unchanged in topology, codecs, and relay
|
||||
behavior. No Web Audio mixer, SDP rewrite, or app-owned audio bitrate knob
|
||||
is introduced by this ADR.
|
||||
3. The native sender's next audio slice is an explicit opt-in
|
||||
`window-process-audio` mode on Windows 11. The local capture session owns
|
||||
one selected target PID/process tree, one WGC/DXGI video source, and one
|
||||
WASAPI loopback audio source. Process IDs, titles, paths, device identity,
|
||||
and PCM stay local; they do not enter signaling, URLs, logs, or storage.
|
||||
4. If the target process exits, produces no render stream, or the loopback
|
||||
activation is denied, the sender reports `audio unavailable` or `audio
|
||||
silent` and stops/asks the user. It never widens to whole-system audio.
|
||||
Windows 10 remains explicitly unsupported until a separate API passes the
|
||||
same isolation gate.
|
||||
|
||||
## Minimal Native Boundary
|
||||
|
||||
This is an interface boundary, not a new framework or a current wire change:
|
||||
|
||||
```text
|
||||
NativeCaptureTarget { pid: uint32, includeProcessTree: true }
|
||||
NativeCaptureSession.start(target)
|
||||
-> VideoFrames(WGC/DXGI) + AudioPcm(WASAPI process loopback)
|
||||
-> existing native sender timeline/encoder/fanout
|
||||
NativeCaptureSession.stop()
|
||||
```
|
||||
|
||||
The first fixture should prove only parent-game plus child-game audio, an
|
||||
independent voice process, a notification, process restart, no render stream,
|
||||
and A/V sync on one Viewer. It should use Microsoft's Application Loopback
|
||||
sample and process-loopback parameter contract rather than a custom mixer.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Current Web users get a useful Chromium hint and an honest status without a
|
||||
privacy claim the platform cannot support.
|
||||
- Strict per-window audio is a native Windows 11 feature, not a browser promise
|
||||
or a silent system-audio fallback.
|
||||
- Android `AudioPlaybackCapture` and packaged cross-platform UI remain later
|
||||
candidates; they do not block this slice.
|
||||
|
||||
## Follow-Up TODO
|
||||
|
||||
- Add a Windows-only WGC + WASAPI process-loopback fixture beside the existing
|
||||
Media Foundation H.264 fixture.
|
||||
- Feed its timestamped PCM/video pair into the existing native sender only
|
||||
after the isolation and sync checks pass.
|
||||
- Retain a short matrix result in `docs/research/`; do not enable the mode by
|
||||
default or deploy it from this ADR alone.
|
||||
@@ -0,0 +1,44 @@
|
||||
/**
|
||||
* Best-effort display-audio hints for Chromium. The Screen Capture API does
|
||||
* not expose a portable readback for the selected audio source, so callers
|
||||
* must keep the requested scope separate from the observed track presence.
|
||||
*/
|
||||
|
||||
export type BrowserAudioScope = "window-requested-unconfirmed" | "none";
|
||||
|
||||
export interface BrowserAudioCaptureStatus {
|
||||
hasTrack: boolean;
|
||||
scope: BrowserAudioScope;
|
||||
}
|
||||
|
||||
type DisplayMediaAudioHints = DisplayMediaStreamOptions & {
|
||||
systemAudio?: "include" | "exclude";
|
||||
windowAudio?: "exclude" | "window" | "system";
|
||||
};
|
||||
|
||||
/**
|
||||
* Ask Chromium to prefer audio from the selected display surface while
|
||||
* excluding the whole-system source. Unknown dictionary members are ignored
|
||||
* by older browsers; no fallback is added because silently widening to
|
||||
* whole-system audio would leak sound.
|
||||
*/
|
||||
export function displayMediaOptions(
|
||||
video: MediaTrackConstraints,
|
||||
): DisplayMediaStreamOptions {
|
||||
return {
|
||||
video,
|
||||
audio: true,
|
||||
systemAudio: "exclude",
|
||||
windowAudio: "window",
|
||||
} as DisplayMediaAudioHints;
|
||||
}
|
||||
|
||||
export function browserAudioCaptureStatus(
|
||||
stream: MediaStream,
|
||||
): BrowserAudioCaptureStatus {
|
||||
const hasTrack = stream.getAudioTracks().length > 0;
|
||||
return {
|
||||
hasTrack,
|
||||
scope: hasTrack ? "window-requested-unconfirmed" : "none",
|
||||
};
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import {
|
||||
type QualityResolution,
|
||||
type QualitySettings,
|
||||
} from "../../shared/protocol";
|
||||
import { displayMediaOptions } from "./audio-capture";
|
||||
|
||||
export type {
|
||||
DegradationPreference,
|
||||
@@ -127,10 +128,9 @@ export async function captureDisplay(
|
||||
throw new Error("当前浏览器不支持屏幕共享");
|
||||
}
|
||||
|
||||
const stream = await navigator.mediaDevices.getDisplayMedia({
|
||||
video: captureConstraints(profile),
|
||||
audio: true,
|
||||
});
|
||||
const stream = await navigator.mediaDevices.getDisplayMedia(
|
||||
displayMediaOptions(captureConstraints(profile)),
|
||||
);
|
||||
|
||||
const videoTrack = stream.getVideoTracks()[0];
|
||||
if (!videoTrack) {
|
||||
|
||||
@@ -60,6 +60,10 @@ import {
|
||||
} from "../lib/session";
|
||||
import { SignalingClient } from "../lib/signaling";
|
||||
import { labelViewerPresence } from "../lib/viewer-presence";
|
||||
import {
|
||||
browserAudioCaptureStatus,
|
||||
type BrowserAudioCaptureStatus,
|
||||
} from "../media/audio-capture";
|
||||
import {
|
||||
applyCaptureProfile,
|
||||
captureDisplay,
|
||||
@@ -106,17 +110,20 @@ interface CaptureDetails {
|
||||
resolution: string;
|
||||
frameRate: number | null;
|
||||
hasAudio: boolean;
|
||||
audio: BrowserAudioCaptureStatus;
|
||||
}
|
||||
|
||||
function captureDetails(stream: MediaStream): CaptureDetails {
|
||||
const settings = stream.getVideoTracks()[0]?.getSettings();
|
||||
const audio = browserAudioCaptureStatus(stream);
|
||||
return {
|
||||
resolution:
|
||||
settings?.width && settings.height
|
||||
? `${settings.width}x${settings.height}`
|
||||
: "未知",
|
||||
frameRate: settings?.frameRate ?? null,
|
||||
hasAudio: stream.getAudioTracks().length > 0,
|
||||
hasAudio: audio.hasTrack,
|
||||
audio,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1631,6 +1638,12 @@ export function HostPage({ onAuthorizationRequired }: HostPageProps = {}) {
|
||||
{!details?.hasAudio && stream && (
|
||||
<WarningBanner>当前来源没有可共享音频</WarningBanner>
|
||||
)}
|
||||
{details?.audio.scope === "window-requested-unconfirmed" && stream && (
|
||||
<WarningBanner>
|
||||
已请求所选窗口音频,但浏览器没有提供来源范围确认;当前音频仍可能包含系统混音。
|
||||
需要严格窗口独占时请使用 Windows 原生发送器。
|
||||
</WarningBanner>
|
||||
)}
|
||||
{qualityLimitation && (
|
||||
<WarningBanner>{qualityLimitation}</WarningBanner>
|
||||
)}
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
browserAudioCaptureStatus,
|
||||
displayMediaOptions,
|
||||
} from "../src/client/media/audio-capture.ts";
|
||||
|
||||
describe("browser display audio capture", () => {
|
||||
it("requests the selected window audio without claiming source readback", () => {
|
||||
const options = displayMediaOptions({ width: { ideal: 1280 } });
|
||||
|
||||
expect(options).toMatchObject({
|
||||
audio: true,
|
||||
systemAudio: "exclude",
|
||||
windowAudio: "window",
|
||||
});
|
||||
});
|
||||
|
||||
it("reports an audio track while keeping its scope unconfirmed", () => {
|
||||
const stream = {
|
||||
getAudioTracks: () => [{}],
|
||||
} as unknown as MediaStream;
|
||||
|
||||
expect(browserAudioCaptureStatus(stream)).toEqual({
|
||||
hasTrack: true,
|
||||
scope: "window-requested-unconfirmed",
|
||||
});
|
||||
});
|
||||
|
||||
it("reports a missing audio track explicitly", () => {
|
||||
const stream = {
|
||||
getAudioTracks: () => [],
|
||||
} as unknown as MediaStream;
|
||||
|
||||
expect(browserAudioCaptureStatus(stream)).toEqual({
|
||||
hasTrack: false,
|
||||
scope: "none",
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -47,6 +47,8 @@ describe("realtime quality controls", () => {
|
||||
frameRate: { ideal: 30, max: 30 },
|
||||
},
|
||||
audio: true,
|
||||
systemAudio: "exclude",
|
||||
windowAudio: "window",
|
||||
});
|
||||
expect(videoTrack.contentHint).toBe("motion");
|
||||
});
|
||||
|
||||
Reference in new issue
Block a user