okooooooo/Piik
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
Piik/docs/research/cross-platform-client-runtime.md
T
AAAandPedro Antonio c18502fd18 feat: improve microphone, chat and platform support 
This release adds microphone processing controls and local chat controls, expands platform packages, and synchronizes the website and guides. It also fixes native H.264 input/format handling, signaling recovery, fresh STUN discovery, and UDP ownership during overlapping connections.

This is **v1.8.0**. Browser/Server v23, Native control v9 and capture v7 remain unchanged, with the existing 1440p ceiling. Additional resolutions, capability negotiation and Magicsock are separate follow-up work.

## Validation

- Full local Web check: 79 files / 1,317 tests, TypeScript and production build.
- Full Go checks, platform builds and Windows native capture regressions/probe.
- Native H.264 Host lifecycle, Browser-to-App fanout and App-exit recovery, and Native H.264 through embedded SFU: all passed with owned process/port/profile cleanup.
- Website build, documentation and repository hygiene passed. Existing UI acceptance covers narrow screens, both languages/themes, architecture downloads and all 23 film checks; final packaging and Go race checks run on the merged revision in CI.
- Physical macOS/Linux capture, representative public networks and unmatched field reports retain their documented evidence limits. No claim that every blur/disconnection report is resolved.

## Release notes

### 中文

这是一次功能与兼容性更新：麦克风和聊天增加了实用的小选项，App 与 Server 支持更多设备，同时修复了原生分享和连接恢复中的若干问题。现有 1.x 协议保持不变。

#### 新增与改进

- 浏览器麦克风新增「降噪与回声消除」开关，默认开启回声消除、降噪和自动增益；分享音乐或使用虚拟音频输入时，可按需关闭。
- 可调整弹幕字号和透明度，并将当前保留的聊天记录导出为 TXT。当前会话最多保留 1,000 条文字消息，刷新页面后清空；需要保存时请先导出。
- 新增 Intel Mac、Linux ARM64 App，以及 Linux ARM64 Server 和容器镜像。官网下载区可直接选择对应架构，并显示最新版本和发布时间。
- 更新官网演示、README 截图和中英文教程，明确浏览器分享也可选择窗口或屏幕；开始分享前显示房间号占位，保持底栏控件位置稳定。
- 官网新增直达问题排查的入口，补充浏览器 WebRTC 限制、校园网和 Windows 显卡相关的排查步骤。
- App 模式选择页显示免费开源说明、作者与贡献者署名，以及官方网站和源码链接。

#### 修复

- 修正部分 H.264 原生分享的格式兼容性校验，以及 Windows 编码器输入长度和输出缓冲区对齐问题。
- 清理已失效的重连任务和 Native 接收信令，避免旧连接继续影响当前会话。
- 修复 App 重连时可能复用过期公网地址的问题；重新探测使用当前地址，并处理探测丢包和过期响应，保留其他正常连接。
- 修复 App 新旧媒体连接重叠时共用端口造成的冲突，避免准备或撤销新连接打断仍在使用的画面；关闭连接时及时取消未完成的端口映射请求。
- 无站点密码时，连接恢复不再误引导用户输入站点密码。
- 房间失效或会话终止时，开播前后的原因提示统一显示在电视状态图标中，避免额外提示条。
- 修正 Linux 原生采集的时间戳与关键帧时间基准。感谢 Pedro Antonio 在 [#449](https://github.com/TNTcraftHIM/Piik/pull/449) 的贡献。

#### 使用说明

下载对应平台的完整压缩包，退出旧 App 后解压使用。Windows 继续提供 x64；macOS 提供 Apple 芯片与 Intel 版本，原生采集需要 macOS 13 或更新版本。Linux 提供 x64 与 ARM64，原生采集还需安装[系统组件](https://github.com/TNTcraftHIM/Piik/blob/main/native/capture/linux/README.md)。macOS 和 Linux 的实机覆盖仍有限，欢迎反馈。

### English

This update adds useful microphone and chat controls, brings Piik to more devices, and fixes several native-sharing and connection-recovery issues. The existing 1.x protocols remain unchanged.

#### New and improved

- Browser microphones now have a “Noise & echo reduction” toggle. Echo cancellation, noise suppression and automatic gain control are on by default; turn them off when appropriate for music or virtual audio inputs.
- Adjust chat-overlay text size and opacity, and export retained messages as TXT. The current session keeps up to 1,000 text messages and clears on page refresh; export first if you want to keep them.
- Added Intel Mac and Linux ARM64 App packages, plus Linux ARM64 Server and container images. The website provides architecture selectors and shows the latest release and publication date.
- Updated the website demonstration, README screenshots and bilingual guides, clarifying that browser capture can also select windows or screens. A room-code placeholder keeps the bottom bar stable before sharing starts.
- Added direct website links to troubleshooting, with steps for browser WebRTC restrictions, campus networks and Windows graphics issues.
- The App mode-selection page now shows its free and open-source status, author and contributor credit, and links to the official website and source.

#### Fixes

- Corrected H.264 format compatibility checks and Windows encoder input-length and output-buffer alignment handling.
- Retired stale reconnect tasks and Native receive signals so they cannot continue affecting the current session.
- Fixed App reconnection reusing an expired public address. Fresh discovery handles lost probes and stale responses while preserving other working connections.
- Fixed UDP socket conflicts between overlapping App media connections so preparing or canceling a replacement preserves the active picture. Closing a connection now cancels unfinished gateway mapping requests.
- Connection recovery on passwordless sites no longer incorrectly asks for a site passphrase.
- Expired-room and session-termination reasons now use the same TV status indicator before and during sharing, without an extra notice strip.
- Corrected Linux native capture timestamps and keyframe time bases. Thanks to Pedro Antonio for [#449](https://github.com/TNTcraftHIM/Piik/pull/449).

#### Updating

Download the complete archive for your platform, quit the old App, and extract the new package. Windows remains x64-only. macOS packages support Apple silicon and Intel; native capture requires macOS 13 or newer. Linux supports x64 and ARM64 and requires additional [system components](https://github.com/TNTcraftHIM/Piik/blob/main/native/capture/linux/README.md) for native capture. Physical-device coverage on macOS and Linux remains limited; feedback is welcome.

## Attribution

Co-authored-by: Pedro Antonio <wrenchwastaken@gmail.com>
2026-10-04 16:21:01 +08:00

245 lines
15 KiB
Markdown
Raw Permalink Blame History

# Cross-Platform App Runtime Research
- Reviewed: 2026-09-04
- Scope: loopback control and self-contained packaging boundary
- Status: App and Local composition implemented; Windows native Host video
and a cross-NAT media gate proved
## Reusable Evidence
| Project | Observed boundary | Applicable lesson |
| --- | --- | --- |
| Discord RPC | Bounded local endpoints and an explicit handshake | Discovery and session negotiation can stay small |
| OBS WebSocket | Versioned `Hello`/`Identify` before commands | Reject unknown protocol shapes instead of adding compatibility readers |
| Syncthing | Go process with a loopback Browser GUI by default | A native entry and system-Browser UI are a practical cross-platform pair |
| Sunshine | Native streaming host configured through a localhost Browser UI | Native media does not require a second desktop UI |
| LocalSend | One peer exposes a local HTTP service without an external server | LAN operation can be self-contained, while network reachability remains physical |
These are architecture references only. No implementation or copyleft source is
copied.
A read-only inventory of locally installed streaming clients adds one practical
constraint. GameViewer separates its UI, supervised media service, codec
capability detector, streamer, health process, and updater. Discord keeps its
Chromium UI separate from a native voice/media module and updater. Oopz ships
Flutter, a second WebView, FFmpeg, Agora RTC, and Tencent LiteAV together; that
stack is substantially larger and duplicates media ownership. Piik follows
the first two products' process separation but not their UI runtimes, and avoids
Oopz's parallel RTC stacks: one Go process, one Pion media core, and one thin
system capture sidecar per platform remain sufficient. No binary code or private
application data was copied or inspected.
## Current Control Boundary
The App binds IPv4 loopback on a bounded range (`39721`-`39730`). `/health`
returns the protocol version, service identity, selected port, and an
`instanceToken`. `/control` admits one WebSocket session whose subprotocol
contains that same value. The first frame must be `hello`; subsequent requests
are limited to capability discovery, source preview, and one generation-fenced
share's media controls.
The instance token distinguishes a newly discovered process from a stale or
unrelated listener. It is visible through health and stdout and is not an
authentication secret. Host and Origin checks plus Chromium's Local Network
Access permission form the Browser boundary.
The current protocol advertises only the native capabilities and
generation-fenced media commands that have consumers. It carries no room
credentials or route policy; those stay in the Browser/server protocol.
## App Composition
The App and Hosted Piik run the same server core, so the local package is
one executable carrying the built Browser assets plus the optional capture and
tunnel sidecars; [the App README](../../cmd/piik-app/README.md) owns
that layout. In Local mode the App runs the server in its own process with
explicit local configuration. In Site mode it starts no server, and the system
Browser opens the saved Site while the same process remains its loopback
native-media owner. The system Browser remains the UI.
Official Go process APIs require every started child to be waited and provide a
bounded `WaitDelay` for cancellation and stuck I/O. That guidance now covers the
capture and cloudflared sidecars only: the room authority is no longer a child
process, and Local shutdown is one bounded in-process sequence that ends rooms
before closing signaling and HTTP. Windows is physically exercised; macOS and
Linux remain package gates.
## Local And Hosted Reachability
A local page completed discovery and the full control exchange in Chromium
151.0.7922.138. The hosted Piik origin completed it after Chromium received
its `loopback-network` permission; without that permission the cross-origin
request was blocked. The gate checks both expected outcomes.
Run `npm run probe:app-loopback` with `PIIK_CLIENT_LOOPBACK_GATE=true`,
`CHROME_PATH`, `PIIK_CLIENT_EXE`, and `PIIK_CLIENT_GATE_SITE_URL` pointing to
the reachable HTTPS Site being tested. Use `PIIK_CLIENT_GATE_GRANT_LOOPBACK=true`
for the permission-granted run; omit it for the permission-blocked run. The gate
uses an isolated App configuration and Browser profile, and rejects navigation
failures or redirects to another origin as acceptance evidence.
A self-contained LAN room uses a localhost Host URL and a selected LAN IPv4
Viewer URL backed by the same local App process. The App persists an
optional user-chosen access password and passes it only to its Host page in a
consumed fragment; a blank value leaves the Local site open. Explicit `--link`
mode launches the packaged Cloudflare Quick
Tunnel against that same local HTTP server and supplies its random HTTPS origin
to the local server configuration before it starts listening. The public path
therefore reuses the exact frontend,
WebSocket signaling, RoomStore, and Viewer grant instead of adding a rendezvous
or second client protocol. Cloudflare carries control traffic; WebRTC media uses
public STUN and remains P2P-only.
## UI Runtime Boundary
The current App opens the system Browser rather than embedding another Web
runtime. A lightweight launcher chooses the authority mode, then navigates to
the same Host page, where an App-launched Host can select an exact native
window. This keeps the existing UI, Site cookies, WebRTC behavior, Browser
updates, and permission model intact. Syncthing and Sunshine establish this as a
practical native-process plus Browser-UI deployment shape.
An embedded shell remains a decision gate rather than a rejected category.
Current WebView2 exposes `getDisplayMedia` screen-capture events and its
Evergreen runtime updates independently; WebKitGTK exposes display capture and
uses GStreamer for WebRTC. Electron instead bundles one Chromium and Node but
makes the App responsible for shipping those updates and adds substantial
package size. These facts do not prove equivalent Piik media behavior.
Only a reproduced Browser-launch failure should open a comparative physical
gate for Electron, Tauri, Wails, Neutralino, or another shell. That gate must
compare actual capture, audio, H.264/VP8 sender implementation, 1080p cadence,
relay, background behavior, and resource use on every target platform. The
result selects one UI runtime; it does not create parallel products.
## Current Evidence
The runs below are pre-port evidence: they exercised the package that bundled a
Node runtime and an extracted application tree. Their capture, discovery, media,
and network results still apply; every package composition, startup, and
shutdown claim must be re-verified on the single-binary App before that build
is accepted.
On Windows with Chrome for Testing 151.0.7922.138:
- the Local App started its built static application and reached `/healthz`;
- the Host fragment was removed, existing SiteAccess became authenticated, and
the ordinary Host page loaded;
- the room API generated an invitation from the selected LAN origin;
- one synthetic Host and three Viewers formed a P2P-only tree with one relay,
every Viewer advanced decoded frames, and no SFU publication appeared; and
- Enter-driven App shutdown released the Browser, App, Node, loopback and
application ports, and disposable profile.
- the immutable App release was assembled with Node 24.19.0 and the Go App
from the same full revision; that packaged directory passed the Local and Site
gates, while a deliberately mismatched `app/REVISION` was rejected before a
listener started.
- the explicit target assembler produced Windows amd64, Linux amd64, and macOS
arm64 outputs from one clean revision. The Linux package ran its bundled Node
Local authority on Ubuntu, and its full package exposed and then closed a
public link reachable from another network. The macOS App and Node inputs
are both Mach-O arm64; execution there remains unproved.
- the native Host path created a Local room and a remote Pion Viewer received
30 packets over a selected `srflx`-to-`srflx` pair; the reverse SSH link in
that gate carried signaling only.
- a Windows App created an accountless Quick Tunnel, generated its ordinary
invitation on that origin, and served the Viewer page plus `/signal` WebSocket
upgrade to an independent Linux host; a native Host then repeatedly delivered
30+ H.264 RTP packets to a Linux Pion Viewer over selected direct paths using
a reflexive candidate. Stopping the App closed the URL.
The one-link gate used the official Windows amd64 `cloudflared` 2026.8.3 asset
with SHA-256 `83e726ed18ea78c5ad5213c4c3a3a27051393950d2bc8ed4de69bec12d14eaae`.
The sidecar remains an explicit package input rather than a linked App
dependency.
The peer topology gate now treats native `getStats()` RTP identity and frame
totals as its portable core proof. Detailed Piik quality metrics enrich a
development run when its source module is served, but their absence on a static
deployment no longer erases valid native RTCStats.
### Public Invitation Startup
The pinned cloudflared Quick Tunnel defaults to explicit QUIC, which disables
its HTTP/2 fallback. Piik requests `--protocol auto` and
`--max-edge-addr-retries 0`: try QUIC once, then let cloudflared fall back within
the existing 30-second startup budget. This affects HTTP/WebSocket control only;
media routes, tunnel identity and ordered shutdown keep their existing owners.
DNS failures, API rejection and networks blocking both transports can still fail.
The preceding Quick Tunnel allocation is a single HTTPS request with a 15-second
dependency timeout; failure exits before transport fallback applies. The App's
pre-readiness exit message and exit code do not identify that cause. Match the
same run's `public-tunnel` dependency diagnostics before classifying a report.
Piik retries a child that exits before readiness, up to four starts with a
one-second pause between starts, all within the same 30-second acquisition
deadline. The previous child, output readers and temporary config retire before
the next start. Cancellation and local setup errors stop acquisition; a ready
tunnel retains its original identity and explicit ordered-close owner.
A 2026-09-21 Windows check used the packaged binary with an isolated UDP sink and
TCP forwarder. The previous arguments timed out at 30 seconds without trying TCP.
The candidate connected in about 11 seconds, served the public page and echoed a
WebSocket message, then joined the child on close. The ordinary-network App link
gate also serves the invitation page and upgrades its WebSocket from an external
US host, then confirms App exit and link retirement. These bounded checks do not
establish regional reachability, uninterrupted connector availability or the
cause of reports without matching diagnostics. Upstream
[Quick Tunnel setup](https://github.com/cloudflare/cloudflared/blob/2026.8.3/cmd/cloudflared/tunnel/quick_tunnel.go)
and [protocol fallback](https://github.com/cloudflare/cloudflared/blob/2026.8.3/supervisor/tunnel.go)
own the dependency behavior. An early-exit fixture confirms fourth-start recovery
and bounded exhaustion; these checks do not identify a reporter's API or network
failure or guarantee four starts when earlier attempts consume the deadline.
[HTTP 1033](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/error-1033/)
means Cloudflare cannot find a healthy tunnel connector. App readiness requires
both the assigned origin and a registered connection; later reconnection belongs
to cloudflared, while process exit ends that App's public-link runtime. This
HTTP failure precedes WebRTC candidate exchange and is distinct from media-route
exhaustion on an already reachable site.
The pinned [supervisor](https://github.com/cloudflare/cloudflared/blob/2026.8.3/supervisor/supervisor.go)
uses [indefinite retries with bounded exponential backoff](https://github.com/cloudflare/cloudflared/blob/2026.8.3/retry/backoffhandler.go)
for recoverable edge disconnects. Piik's four-start acquisition limit is not a
four-reconnect limit after readiness. Restarting the Quick Tunnel process would
allocate a new public address rather than repair the old invitation.
## Remaining Gates
- Run the exact clean-revision package on macOS; Linux amd64 is proved.
- Prove a physical second-device LAN Viewer on desktop, Android, and iOS.
- Prove first-run Browser local-network permission outside CDP automation.
- Record no-STUN mDNS behavior on ordinary and AP-isolated LANs.
- Exercise the native Host on a configured Site and representative Browsers;
the local SFU bridge is proved in isolation, while full Site recovery,
representative audio, and non-Windows capture remain undefined.
- Exercise one-link Browser media on a physical second device; the current gate
proves the complete public control path and independent Pion media, not
decoded Browser frames.
## Sources
- [Go `os/exec`](https://pkg.go.dev/os/exec)
- [WebRTC peer connections](https://webrtc.org/getting-started/peer-connections)
- [Cloudflare public STUN endpoint](https://developers.cloudflare.com/realtime/turn/)
- [Cloudflare Quick Tunnels](https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/do-more-with-tunnels/trycloudflare/)
- [Cloudflare Tunnel WebSockets](https://developers.cloudflare.com/cloudflare-one/faq/cloudflare-tunnels-faq/)
- [cloudflared releases](https://github.com/cloudflare/cloudflared/releases/tag/2026.8.3)
- [RFC 6455](https://www.rfc-editor.org/rfc/rfc6455.html)
- [W3C Screen Capture](https://www.w3.org/TR/screen-capture/)
- [Microsoft Windows Graphics Capture](https://learn.microsoft.com/en-us/windows/apps/develop/media-authoring-processing/screen-capture)
- [Apple SCContentSharingPicker](https://developer.apple.com/documentation/screencapturekit/sccontentsharingpicker)
- [XDG ScreenCast portal](https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.ScreenCast.html)
- [OBS window-capture source](https://github.com/obsproject/obs-studio/blob/master/plugins/win-capture/window-capture.c)
- [Chrome Local Network Access](https://developer.chrome.com/blog/local-network-access)
- [Discord RPC](https://docs.discord.com/developers/topics/rpc)
- [OBS WebSocket](https://github.com/obsproject/obs-websocket/blob/master/docs/generated/protocol.md)
- [LocalSend protocol](https://github.com/localsend/protocol/blob/main/README.md)
- [Sunshine](https://docs.lizardbyte.dev/projects/sunshine/latest/)
- [Syncthing GUI configuration](https://docs.syncthing.net/users/config.html)
- [GameViewer product](https://uuyc.163.com/)
- [Discord desktop architecture](https://discord.com/blog/how-discord-maintains-performance-while-adding-features)
- [WebView2 Evergreen runtime](https://learn.microsoft.com/en-us/microsoft-edge/webview2/concepts/evergreen-vs-fixed-version)
- [WebView2 screen capture](https://learn.microsoft.com/en-us/microsoft-edge/webview2/reference/win32/icorewebview2screencapturestartingeventargs)
- [WebKitGTK multimedia](https://docs.webkit.org/Ports/WebKitGTK%20and%20WPE%20WebKit/Multimedia.html)
- [WebKitGTK display capture](https://webkitgtk.org/reference/webkit2gtk/stable/class.WebView.html)
- [Electron architecture](https://www.electronjs.org/docs/latest/why-electron)
Reference in new issue View Git Blame Copy Permalink