okooooooo/esp32-iphone-agent
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs, CI: README for the no-computer version, legacy notes, app + firmware CI

Browse Source 
server/ stays as the legacy v0.1.0 setup (tag v0.1.0, branch legacy).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
HenryandClaude Opus 5.5 committed 2026-10-15 09:55:35 +08:00
1 parent f911d841c0
commit e035e424f4
12 files changed
+290 -402
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/ci.yml
+29 -15
View File
@@ -6,7 +6,35 @@ on:
pull_request:
jobs:
server:
ios:
runs-on: macos-26
defaults:
run:
working-directory: ios
steps:
- uses: actions/checkout@v5
- name: Unit tests (simulator, unsigned)
run: |
sim=$(xcrun simctl list devices available | grep -Eo 'iPhone [0-9]+[A-Za-z ]*\(' | head -1 | sed 's/ ($//')
echo "simulator: $sim"
xcodebuild test -scheme IPhoneAgent -destination "platform=iOS Simulator,name=$sim" \
-derivedDataPath build-sim CODE_SIGNING_ALLOWED=NO
firmware:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: arduino/setup-arduino-cli@v2
- run: |
arduino-cli core update-index --additional-urls https://espressif.github.io/arduino-esp32/package_esp32_index.json
arduino-cli core install esp32:esp32@3.3.12 --additional-urls https://espressif.github.io/arduino-esp32/package_esp32_index.json
arduino-cli lib install "NimBLE-Arduino@2.5.1"
- name: Build
run: hardware/flash.sh --build-only
- name: Host tests (bplist, GOP buffer)
run: make -C hardware/tests
server: # legacy v0.1.0 architecture, kept working
runs-on: ubuntu-latest
strategy:
matrix:
@@ -22,17 +50,3 @@ jobs:
- run: pip install -e ".[dev]"
- run: ruff check .
- run: pytest -q
firmware:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: arduino/setup-arduino-cli@v2
- run: |
arduino-cli core update-index --additional-urls https://espressif.github.io/arduino-esp32/package_esp32_index.json
arduino-cli core install esp32:esp32 --additional-urls https://espressif.github.io/arduino-esp32/package_esp32_index.json
- name: Build
working-directory: hardware
run: |
cp hid_bridge/secrets.example.h hid_bridge/secrets.h
arduino-cli compile -b "esp32:esp32:esp32s3:USBMode=default,CDCOnBoot=cdc,FlashSize=8M" hid_bridge
.gitignore
+5
View File
@@ -17,3 +17,8 @@ hardware/hid_bridge/build/
.DS_Store
.idea/
.vscode/
# Host test output
hardware/tests/test
hardware/tests/test.dSYM/
hardware/tools/out/
CHANGELOG.md
+14
View File
@@ -0,0 +1,14 @@
# Changelog
## Unreleased (main)
- No computer: the ESP32 is the AirPlay receiver (ported from UxPlay) with its own hotspot, and talks to a new
iPhone app over BLE. The app is the agent: planner/actor loop, pointer calibration, run replay, Live Activity,
widgets, Siri and Shortcuts, self-healing when the board, hotspot or mirroring drops.
- Prompts moved to `prompts/`.
- Product name: ESP32 Agent.
- `server/` is legacy and no longer needed.
## v0.1.0
- Computer-side version: UxPlay + Python server + web UI; the ESP32 is a USB HID bridge over Wi-Fi.
CONTRIBUTING.md
+19
View File
@@ -0,0 +1,19 @@
# Contributing
Issues and pull requests are welcome. Keep changes small and focused, and match the surrounding code.
Before sending a change, run the checks for what you touched:
```bash
# app (Xcode 26+)
cd ios && xcodegen && xcodebuild test -scheme IPhoneAgent -destination 'platform=iOS Simulator,name=iPhone 17 Pro'
# firmware
hardware/flash.sh --build-only && make -C hardware/tests
# legacy server
cd server && pip install -e ".[dev]" && ruff check . && pytest
```
Never commit secrets: `ios/Config/Local.xcconfig` and `hardware/hid_bridge/secrets.h` are git-ignored on purpose.
Run data exported from the app is redacted, but check it before attaching it to an issue.
Code under `hardware/hid_bridge/src/airplay/` is GPL-3.0 (ported from UxPlay); contributions there are GPL-3.0 too.
README.md
+100 -199
View File
@@ -1,8 +1,10 @@
<div align="center">
<img src="docs/images/logo.png" alt="ESP32 Agent logo" width="160">
# ESP32 iPhone Agent
**Let an AI model use a real iPhone — with a $7 ESP32. No Mac, no developer account, no jailbreak.**
**Let an AI model use a real iPhone with a $7 ESP32. Just the phone and the board: no computer, no jailbreak.**
English · [简体中文](README.zh-CN.md)
@@ -10,263 +12,162 @@ Demo video and updates: [X @iamhenryyuan](https://x.com/iamhenryyuan) · [YouTub
[![CI](https://github.com/onehumanbeing/esp32-iphone-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/onehumanbeing/esp32-iphone-agent/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
![Python](https://img.shields.io/badge/python-3.10%2B-blue)
![iOS](https://img.shields.io/badge/iOS-18%2B-black)
![Board](https://img.shields.io/badge/board-ESP32--S3-red)
</div>
<p align="center"><img src="docs/images/demo.png" alt="Example 1: the web UI after drawing a smiley face in Whiteboard, with the live screen, live log, plan and step screenshots" width="100%"></p>
> **New: the computer is gone.** The board is now the AirPlay receiver and the mouse, and the agent runs in an iPhone
> app on the same phone. The original version (a Mac/Linux computer running UxPlay and a Python server) is kept as
> tag [`v0.1.0`](https://github.com/onehumanbeing/esp32-iphone-agent/tree/v0.1.0) and branch
> [`legacy`](https://github.com/onehumanbeing/esp32-iphone-agent/tree/legacy). See [docs/legacy.md](docs/legacy.md).
**Example 1 — draw on a canvas.** *"Open Whiteboard and draw a smiley face"*: the agent searches for the app, opens it and draws the face with continuous strokes. Every step's screenshot, with its tap or stroke drawn on top, stays on the step wall.
<p align="center"><img src="docs/images/example-alarm.png" alt="An iPhone with a XIAO ESP32S3 plugged in, setting an alarm" width="100%"></p>
<p align="center"><img src="docs/images/example-alarm.png" alt="Example 2: an iPhone with a XIAO ESP32S3 plugged in, setting an alarm that repeats every day, next to the web UI" width="100%"></p>
**Example 2 — set an alarm.** *"Set an alarm for 7:00 AM that repeats every day, then save it"*: the agent opens Clock, types 0700 into the time wheel, ticks the days under Repeat several taps at a time, and saves. Left: the real iPhone with the ESP32 plugged in; right: the web UI.
The agent sees the phone through AirPlay screen mirroring and touches it through a tiny ESP32-S3 board that
plugs into the phone's USB port as a mouse and keyboard. Any OpenAI-compatible vision model can be the brain —
a cloud API, or a model you run yourself.
## Why
Most iPhone automation drives the phone through XCTest / WebDriverAgent. That needs a Mac, Xcode, a signed
helper app and an Apple developer account (a free account's signature expires every 7 days). This project
uses the same paths a person uses — the screen and a pointer — so the phone needs nothing installed.
| | ESP32 iPhone Agent | XCTest / WebDriverAgent tools |
|----------------------------------|---------------------------|-------------------------------|
| Apple developer account | **Not needed** | Needed (to sign the runner) |
| Xcode | **Not needed** | Needed |
| Computer | Mac **or Linux** | Mac |
| App installed on the phone | **None** | Test runner app |
| Jailbreak | Not needed | Not needed |
| Extra hardware | ESP32-S3 board (~$7) | USB cable |
| Works in any app, incl. games | Yes (it only sees pixels) | Mostly (depends on the accessibility tree) |
*"Set an alarm for 8:00 AM that repeats every Monday and Tuesday, then save it"*: the agent leaves its own app,
opens Clock, types the time into the wheel, ticks the days and saves. (Photo from v0.1.0; the task is the same.)
## How it works
- **Eyes — AirPlay mirroring.** The phone mirrors its screen to [UxPlay](https://github.com/FDH2/UxPlay) running
on your computer; a GStreamer pipeline turns the stream into JPEG frames for the agent and the web page.
- **Hands — a USB HID bridge.** An ESP32-S3 plugged into the phone shows up as a standard USB mouse and keyboard.
With iOS AssistiveTouch on, the mouse moves an on-screen pointer, so a click is a tap and a drag is a swipe.
The board connects to the computer over Wi-Fi and executes simple line commands (`move`, `click`, `type`, …);
all logic stays on the computer.
- **Brain — a planner and an actor.** A planner model writes a short plan from the goal and the current screen.
An actor model then looks at each new screenshot and picks the next action: tap, swipe, draw a path, type, press
a key, go home, or finish. Both roles can be the same model.
- **Closing the loop.** The pointer is relative, so the server tracks where it is. Before each tap it wiggles the
pointer, finds it on the mirrored screen and corrects any miss, refining the calibration as it goes. After each
action it checks whether the screen actually changed and tells the model, so a missed tap is noticed and retried
instead of built upon. Repeated actions trigger a re-plan; a dropped link or a bad reply is recovered from.
```
iPhone ──Screen Mirroring (Wi-Fi)──▶ ESP32-S3: AirPlay receiver ──BLE──▶ ESP32 Agent app (same iPhone)
▲ own hotspot, H.264 passthrough decode → vision model → action
└──────── USB HID mouse + keyboard ◀──────────── BLE commands ◀────────────────────┘
```
- **Eyes — AirPlay to the board.** The board opens its own Wi-Fi hotspot and appears in the phone's Screen Mirroring
list as *ESP32 Agent*. It keeps the H.264 stream (no decoding on the board) and forwards it to the app over BLE.
The phone keeps using cellular data for the model while it is on the board's hotspot.
- **Hands — USB HID.** The board is a standard USB mouse and keyboard. With AssistiveTouch on, the mouse moves an
on-screen pointer, so a click is a tap and a drag is a swipe.
- **Brain — the app.** The app decodes the stream, sends screenshots to any OpenAI-compatible vision model, and runs
a planner and an actor: tap, swipe, draw, type, press keys, go home, finish. It runs in the background while it
works; the Dynamic Island shows what it is doing.
- **Closing the loop.** The pointer is relative, so the app tracks it and re-homes it in a corner; before precise taps
it wiggles the pointer, finds it on screen and corrects the miss. After each action it waits for the screen to
settle and tells the model whether anything changed. A lost board, hotspot or mirroring link pauses the run and
resumes it from the same step.
- **Review.** Every run is kept on the phone: replay it step by step with the taps and swipes drawn on top.
## Hardware
| Part | Notes |
|---|---|
| **ESP32-S3 board** | Tested with a [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/xiao_esp32s3_getting_started/), plain version without pre-soldered headers (not the Sense camera version, not the Plus). About $7 from Seeed, or about ¥40-50 on Taobao in China (search "XIAO ESP32S3"). Any ESP32-S3 board whose USB-C port is wired to the S3's native USB works. |
| **iPhone** | USB-C models plug in directly. Lightning models need a Lightning to USB Camera Adapter. The phone powers the board. |
| **Computer** | macOS or Linux, on the same network as the phone; runs UxPlay and the server. |
| **Vision model** | Any OpenAI-compatible chat API that accepts images. Running a model locally is optional. |
<p align="center"><img src="docs/images/setup.png" alt="A XIAO ESP32S3 plugged into an iPhone, next to the web UI" width="100%"></p>
Use a USB-C hub with power pass-through if the phone should keep charging while the board is plugged in.
| **ESP32-S3 board** | Tested with a [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/xiao_esp32s3_getting_started/) (plain version, 8 MB PSRAM; not Sense, not Plus). About $7. Any ESP32-S3 with native USB on its USB-C port and PSRAM should work. |
| **iPhone** | iOS 18 or later, with cellular data (the Wi-Fi is taken by the board's hotspot). USB-C models plug in directly; Lightning models need a Lightning to USB Camera Adapter. The phone powers the board. |
| **Vision model** | Any OpenAI-compatible chat API that accepts images: a cloud API or your own server. |
| **Mac with Xcode** | Only to flash the board and build the app once. Not needed at run time. |
## Quick start
> **Set up with an AI coding agent.** Paste this into Claude Code, Codex or a similar agent on the computer the
> board will connect to:
>
> ```text
> Clone https://github.com/onehumanbeing/esp32-iphone-agent and follow its README to set it up on this machine:
> flash the ESP32-S3 plugged in over USB, install UxPlay and the server, write server/.env for my model API, start
> the server and give me the web UI address. Ask me for the Wi-Fi name/password and the model API key.
> ```
### 1. Flash the board (plugged into the Mac)
### 1. Flash the board
Install [arduino-cli](https://arduino.github.io/arduino-cli/) and the ESP32 core (3.x), then:
Install [arduino-cli](https://arduino.github.io/arduino-cli/), the ESP32 core 3.3.x and NimBLE-Arduino 2.5.x, then:
```bash
cp hardware/hid_bridge/secrets.example.h hardware/hid_bridge/secrets.h
# edit secrets.h: 2.4 GHz Wi-Fi name/password and the IP address of the computer running the server
hardware/flash.sh # optional arguments: [serial port] [--debug]
arduino-cli lib install "NimBLE-Arduino@2.5.1"
hardware/flash.sh # optional: [serial port] [--debug]
```
### 2. Install and start the server
Always build with `flash.sh`: it applies the BLE buffer sizes in `hardware/hid_bridge/nimble_cfg.h`.
### 2. Install the app
```bash
# UxPlay — macOS:
brew install uxplay
# UxPlay — Debian / Ubuntu:
sudo apt install uxplay gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-libav
cd server
python3 -m venv .venv && .venv/bin/pip install -e .
cp .env.example .env # set LLM_BASE_URL / LLM_MODEL / LLM_API_KEY
set -a && . ./.env && set +a && .venv/bin/python -m hidagent
cp ios/Config/Local.xcconfig.example ios/Config/Local.xcconfig # set DEVELOPMENT_TEAM and IA_BUNDLE_ID
open ios/IPhoneAgent.xcodeproj # run the IPhoneAgent scheme on your iPhone
```
Open `http://<computer-ip>:8890` in a browser.
The project is generated with [XcodeGen](https://github.com/yonaskolb/XcodeGen) from `ios/project.yml`
(`cd ios && xcodegen`). A free Apple ID can sign it (re-sign every 7 days). The one-tap hotspot join needs the
Hotspot Configuration capability (a paid account); without it the app shows the hotspot password to join by hand.
### 3. Set up the phone (once)
### 3. Set up (once, in the app)
1. **Settings → Accessibility → Touch → AssistiveTouch**: turn it on.
2. Plug the board into the phone. A round pointer appears; the web page shows *controller connected*.
3. **Control Center → Screen Mirroring → iPhone Agent**. The live screen appears in the web page.
4. In the web page, open **Manual controls → Calibrate**. The server moves the pointer, finds it on the live screen
and calibrates itself in a few seconds; if it cannot see the pointer, it asks you to click exactly on it.
Plug the board into the iPhone and open **ESP32 Agent**. Setup walks through four steps and checks each one itself:
1. **Plug Me In** — pair the board over Bluetooth.
2. **Let Me See the Screen** — join the board's Wi-Fi, then Control Center → Screen Mirroring → *ESP32 Agent*.
3. **Let Me Use the Pointer** — turn on AssistiveTouch; the app calibrates the pointer by itself.
4. **Give Me a Brain** — enter your model endpoint and key (stored in the Keychain).
### 4. Give it a task
Type a goal (for example *"Open Settings and go to Wi-Fi"*) and press **Start**, or click one of the examples.
Type a goal (for example *"Open Settings and go to Wi-Fi"*) and press **Start**. Stop it from the Dynamic Island,
or come back to the app. Siri, Shortcuts, widgets and Control Center can run saved tasks.
## Web UI
## Model and prompts
- **Live phone** — click to tap, drag to swipe, scroll to scroll; manual Home, Back, typing and keys.
- **Live log** — every plan, thought and action as it happens, with timings.
- **Step wall** — the screenshot the model decided on for every step, with its tap or swipe drawn on top, plus the
model's reasoning and how long thinking and acting took. Click a card to zoom in.
- **Taps** are marked with a red ring and a spreading ripple, swipes and strokes with a yellow path; the **Motion** switch
turns the animation off. Taps the model marked `precise` (pointer checked before tapping) are labelled in the log.
- **Emergency stop** (or <kbd>Esc</kbd>) stops the agent and releases the pointer. **Clear history** empties the log.
- English and Simplified Chinese interface.
## Configuration
Set in `server/.env` (see [`.env.example`](server/.env.example)):
| Variable | Default | Meaning |
|---|---|---|
| `LLM_BASE_URL` | `http://localhost:8000/v1` | OpenAI-compatible endpoint |
| `LLM_MODEL` / `LLM_API_KEY` | — | Model name and API key |
| `LLM_EXTRA_BODY` | `{}` | JSON merged into every request (e.g. to turn off a model's thinking mode) |
| `LLM_TOKEN_PARAM` | `auto` | Token-limit field: `max_tokens`, `max_completion_tokens` (newer OpenAI models), or `auto` to detect |
| `ACTOR_*` / `PLANNER_*` | — | Per-role overrides of any `LLM_*` variable |
| `AIRPLAY_NAME` | `iPhone Agent` | Receiver name in the phone's Screen Mirroring list |
| `PORT` / `DEVICE_PORT` | `8890` / `7300` | Web UI port / port the board connects to |
| `HIDAGENT_DATA` | `~/.hidagent` | Calibration and debug traces |
| `HIDAGENT_DEBUG` | `0` | Debug mode, see below |
### Local model (optional)
A cloud API is enough. If you want everything on your own hardware, this is the setup the project was developed
and timed with:
- Model: [nvidia/Qwen3.8-Flash-Next-NVFP4](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4)
- Serving: [Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold](https://github.com/MiaAI-Lab/Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold)
— two NVIDIA DGX Sparks with tensor parallelism (TP=2), OpenAI-compatible API
- Speed: about 1.3 s per action and 2 s per plan, with thinking mode off:
```bash
LLM_BASE_URL=http://<dgx-spark-ip>:8888/v1
LLM_MODEL=Qwen3.8-Flash-Next
LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'
```
Any other OpenAI-compatible server (vLLM, SGLang, llama.cpp, Ollama, …) with a vision model works the same way.
## Debug mode
Set `HIDAGENT_DEBUG=1` to log every board command with its latency and to save one folder per agent run:
```
~/.hidagent/traces/20261012-005207/
├── steps.jsonl # every model call (prompt, raw reply, latency) and every action
├── 002.jpg # the screenshot each model call saw
└── ...
```
`hardware/flash.sh --debug` builds firmware that echoes each command and its duration over USB serial.
## Board protocol
The board holds no logic. On connecting it sends `hello <last reset reason> <uptime ms>`; then the server sends one
command per line over TCP and the board answers `ok` or `err <reason>`.
| Command | Effect |
|---|---|
| `ping` | Health check |
| `home` | Push the pointer into the top-left corner (a known position) |
| `move <dx> <dy>` | Relative pointer move, at a constant pace so iOS pointer acceleration stays predictable |
| `press` / `release` / `click` | Left button down / up / both |
| `scroll <v> <h>` | Scroll wheel |
| `type <text>` | Type ASCII text |
| `key <combo>` | Press keys, e.g. `enter`, `cmd+h`, `cmd+space` |
If the Wi-Fi link drops, the board releases all buttons and reconnects on its own. While the phone is locked, its USB
port sleeps: the board then tries to wake it and otherwise answers `err phone usb asleep` instead of waiting. A watchdog
restarts the board if it ever stops responding.
- Any OpenAI-compatible vision endpoint works (vLLM, SGLang, llama.cpp, Ollama, cloud APIs). The app lists the
endpoint's models and picks one; you can override it, and set a separate planner and actor model.
- The prompts are plain text in [`prompts/`](prompts/) and bundled with the app: edit them, rebuild, done.
- The project was developed with [nvidia/Qwen3.8-Flash-Next-NVFP4](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4)
served on two DGX Sparks ([TensorFold](https://github.com/MiaAI-Lab/Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold)).
## Project layout
```
hardware/ ESP32-S3 firmware (Arduino) and flash script
server/ agent, web UI, screen capture and board link (Python, FastAPI)
serve/: optional minimal server to host a Hugging Face vision model yourself
ios/ the app (SwiftUI): BLE link, H.264 decode, agent, setup, replay, Live Activity, widgets; tests
prompts/ planner / actor prompts used by the app
hardware/ ESP32-S3 firmware (Arduino): BLE GATT, hotspot, AirPlay receiver, USB HID; flash script,
host tests, Mac BLE test harness (hardware/tools)
docs/ BLE protocol, legacy version notes
server/ legacy v0.1.0 computer-side server; not used by this version (see server/README.md)
```
Debugging without the phone app: `hardware/tools/board_test.py` plays the app over BLE from a Mac
([hardware/tools/README.md](hardware/tools/README.md)). Board ↔ app protocol: [docs/ble-protocol.md](docs/ble-protocol.md).
## Roadmap
- [ ] Test a fully local setup with DiffusionGemma (`server/serve/` has an experimental OpenAI-compatible server)
- [ ] Landscape apps (screen coordinates and calibration assume portrait)
- [x] Drop the computer: the board mirrors and clicks, the app on the phone thinks
- [ ] App Store / TestFlight build (leave the app with the HID Home key instead of a private API)
- [ ] Landscape apps (coordinates and calibration assume portrait)
- [ ] Non-ASCII text input (e.g. Chinese through the iOS pinyin keyboard)
- [ ] More example tasks across popular apps
### Exploring: drop the computer
Right now the board is just the hands. The brain and the eyes run on a separate computer.
The next idea is to drop the computer. Swap the ESP32 for a small board that can run Linux, and run the whole loop on it: see the screen, decide, tap. The phone would only need that one board, which you could carry in your pocket. The board could talk to the phone over USB or Bluetooth, and call a local or cloud model.
These are directions, not promises. If you're interested, or you've tried something similar, open an issue.
- [ ] Faster steps: several actions per model call, pointer checks only when needed
## Limitations
- iOS does not mirror some screens (passcode entry, Apple Pay, DRM video); the agent cannot see them.
- Mirroring stops when the phone locks; keep auto-lock off while the agent works.
- Keyboard input is ASCII only.
- Re-calibrate after changing AssistiveTouch tracking speed.
- Landscape apps are not supported yet: drawings and taps there land in the wrong place.
- On Linux machines with several network interfaces, limit Avahi (mDNS) to the interface on the phone's network
if the phone does not list the receiver.
- BLE carries about 300-500 kbps, so the screen is mirrored small (480×1040, ≤5 fps); each step takes a few seconds,
most of it the model.
- iOS does not mirror some screens (passcode entry, Apple Pay, DRM video); the agent cannot see them. Mirroring stops
when the phone locks.
- While the agent runs, the phone's Wi-Fi is the board's hotspot (no internet over it); the model is reached over
cellular data.
- The app leaves itself with a private iOS API (`suspend`); fine for a self-built app, to be replaced before the App Store.
- Keyboard input is ASCII only. Landscape apps are not supported yet.
## Responsible use
Only automate devices and accounts you own or are allowed to use, and respect the terms of the apps involved. The
agent acts with your hands: watch it, and keep the emergency stop within reach.
agent acts with your hands: watch it, and keep the stop button within reach.
## Contributing
Issues and pull requests are welcome. Run the checks before sending a change:
```bash
cd server && pip install -e ".[dev]" && ruff check . && pytest
```
Issues and pull requests are welcome; see [CONTRIBUTING.md](CONTRIBUTING.md). Security reports: [SECURITY.md](SECURITY.md).
## Related projects
Other ways people drive an iPhone with code or AI, and how this project differs. Corrections welcome.
| Project | Approach | Difference from this project |
|---|---|---|
| [VersaiOS](https://github.com/zhaodashuaibi/VersaiOS) | UxPlay mirroring + ESP32-S3 Bluetooth mouse + vision model | Needs a computer; one tap per instruction |
| [ja1chen/actell](https://github.com/ja1chen/actell) | Agent app on the iPhone via Apple's CoreDevice debug channel | Needs Developer Mode and a Developer Disk Image |
| [AidenAI-IO/aiden-firmware](https://github.com/AidenAI-IO/aiden-firmware) | HDMI capture + USB keyboard/mouse + multimodal model | Capture hardware; mainly Android |
| [yoyicue/glassbox](https://github.com/yoyicue/glassbox) | PicoKVM HDMI capture + USB HID | macOS controller and PicoKVM |
| [boboabw/phone-agent](https://github.com/boboabw/phone-agent) | ESP32-S3 Bluetooth HID + AssistiveTouch, agent as an iPhone app | Closest to this version; this one also brings the eyes (AirPlay on the board) |
| [VKCOM/devicehub](https://github.com/VKCOM/devicehub) | ESP32 Bluetooth mouse for iOS device farms | Device-farm tooling, no AI agent |
| Project | Approach | Needs on the phone / host | Difference from this project |
|---|---|---|---|
| [VersaiOS](https://github.com/zhaodashuaibi/VersaiOS) | UxPlay mirroring + ESP32-S3 Bluetooth mouse + vision model | ESP32 tethered to a computer over USB serial; desktop GUI, mainly Windows | Closest in spirit. One tap per instruction, no planning step or before/after screen check |
| [ja1chen/actell](https://github.com/ja1chen/actell) | Agent app on the iPhone, screenshots and taps via Apple's CoreDevice debug channel | Developer Mode, pairing + Developer Disk Image, Xcode self-signing | This project needs no Developer Mode, no app on the phone and no signing |
| [AidenAI-IO/aiden-firmware](https://github.com/AidenAI-IO/aiden-firmware) | Dedicated board: HDMI capture + USB keyboard/mouse + multimodal model | HDMI capture chip and dock; mainly Android | Uses AirPlay instead of HDMI capture, so no capture hardware |
| [yoyicue/glassbox](https://github.com/yoyicue/glassbox) | PicoKVM HDMI capture + USB HID, observe/decide/act/verify loop | macOS controller and PicoKVM | Runs on Linux with a single ESP32 |
| [boboabw/phone-agent](https://github.com/boboabw/phone-agent) | ESP32-S3 Bluetooth HID + AssistiveTouch, agent runs as an iPhone app | Mac + Xcode to build and sign the app | Nothing to install on the phone |
| [dovydaskel/Iphone-control-for-social-media](https://github.com/dovydaskel/Iphone-control-for-social-media) | ESP32 Bluetooth mouse with OpenCV pointer correction, scripted posting | Mac | Similar closed-loop pointer idea; this project adds a vision-model planner/executor |
| [VKCOM/devicehub](https://github.com/VKCOM/devicehub) | ESP32 Bluetooth mouse for iOS device farms | Farm infrastructure | Device-farm tooling, no AI agent |
| [NolanCotter/AirBridge](https://github.com/NolanCotter/AirBridge) / [gbulog/pcairplay](https://github.com/gbulog/pcairplay) | UxPlay on Linux + the PC's own Bluetooth as keyboard/mouse | Linux PC with Bluetooth | Manual remote control, no AI |
Other routes: WebDriverAgent/XCUITest-based tools such as [mobile-mcp](https://github.com/mobile-next/mobile-mcp) and [Midscene](https://github.com/web-infra-dev/midscene) need a Mac and code signing; tools built on macOS iPhone Mirroring need a Mac.
WebDriverAgent/XCUITest-based tools such as [mobile-mcp](https://github.com/mobile-next/mobile-mcp) and
[Midscene](https://github.com/web-infra-dev/midscene) need a Mac and code signing.
## Acknowledgements
[UxPlay](https://github.com/FDH2/UxPlay) (AirPlay receiver) ·
[Arduino ESP32 core](https://github.com/espressif/arduino-esp32) and [TinyUSB](https://github.com/hathach/tinyusb) (USB HID) ·
[FastAPI](https://fastapi.tiangolo.com/)
[UxPlay](https://github.com/FDH2/UxPlay) (the AirPlay receiver the firmware's is ported from) ·
[NimBLE-Arduino](https://github.com/h2zero/NimBLE-Arduino) ·
[Arduino ESP32 core](https://github.com/espressif/arduino-esp32) and [TinyUSB](https://github.com/hathach/tinyusb)
## License
[MIT](LICENSE)
[MIT](LICENSE), except `hardware/hid_bridge/src/airplay/` (ported from UxPlay), which is
[GPL-3.0](hardware/hid_bridge/src/airplay/LICENSE-GPL.md); the firmware binary that links it is therefore
distributed under GPL-3.0.
README.zh-CN.md
+79 -188
View File
@@ -1,8 +1,10 @@
<div align="center">
<img src="docs/images/logo.png" alt="ESP32 Agent logo" width="160">
# ESP32 iPhone Agent
**让 AI 操作一部真实的 iPhone：只要一块约 40 多元的 ESP32，不需要开发者账号、不需要 Xcode、不用越狱。**
**让 AI 操作一部真实的 iPhone：只要手机和一块约 40 多元的 ESP32，不需要电脑，不用越狱。**
[English](README.md) · 简体中文
@@ -10,250 +12,139 @@
[![CI](https://github.com/onehumanbeing/esp32-iphone-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/onehumanbeing/esp32-iphone-agent/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
![Python](https://img.shields.io/badge/python-3.10%2B-blue)
![iOS](https://img.shields.io/badge/iOS-18%2B-black)
![Board](https://img.shields.io/badge/board-ESP32--S3-red)
</div>
<p align="center"><img src="docs/images/demo.png" alt="示例 1：在白板里画完笑脸后的网页界面，包括实时画面、实时日志、计划和每一步的截图" width="100%"></p>
> **新版：电脑去掉了。** 板子自己当 AirPlay 接收端和鼠标，Agent 跑在同一台 iPhone 上的 App 里。
> 旧版（Mac/Linux 电脑跑 UxPlay + Python 服务端）保留在 tag [`v0.1.0`](https://github.com/onehumanbeing/esp32-iphone-agent/tree/v0.1.0)
> 和分支 [`legacy`](https://github.com/onehumanbeing/esp32-iphone-agent/tree/legacy)，见 [docs/legacy.md](docs/legacy.md)。
**示例 1：在画布上画画。** *"打开白板画一个笑脸"*：Agent 搜索并打开白板 App，用连续的笔画画出笑脸。每一步的截图都会留在截图墙上，上面画出了点按或笔画的位置。
<p align="center"><img src="docs/images/example-alarm.png" alt="插着 XIAO ESP32S3 的 iPhone 正在设置闹钟" width="100%"></p>
<p align="center"><img src="docs/images/example-alarm.png" alt="示例 2：插着 XIAO ESP32S3 的 iPhone 正在设置每天重复的闹钟，旁边是网页界面" width="100%"></p>
**示例 2：设置闹钟。** *"设置早上 7:00 的闹钟，每天重复，然后保存"*：Agent 打开时钟，在时间滚轮里直接输入 0700，在「重复」里一次连续勾选多天，然后保存。左边是插着 ESP32 的真实 iPhone，右边是网页界面。
Agent 通过 AirPlay 屏幕镜像"看"手机，通过一块插在手机 USB 口上的 ESP32-S3 小板子"摸"手机：板子对手机来说就是一个
USB 鼠标加键盘。大脑可以是任何兼容 OpenAI 接口的视觉模型，云端 API 或自己部署的模型都行。
## 为什么做这个
大多数 iPhone 自动化方案通过 XCTest / WebDriverAgent 控制手机，需要 Mac、Xcode、一个签过名的辅助 App，还要 Apple
开发者账号（免费账号的签名 7 天就过期）。本项目走的是人用手机的路径：看屏幕、动指针，所以手机上什么都不用装。
| | ESP32 iPhone Agent | XCTest / WebDriverAgent 类工具 |
|--------------------------|------------------------|--------------------------------|
| Apple 开发者账号 | **不需要** | 需要（给测试程序签名） |
| Xcode | **不需要** | 需要 |
| 电脑 | Mac **或 Linux** | Mac |
| 手机上安装 App | **不需要** | 需要装测试程序 |
| 越狱 | 不需要 | 不需要 |
| 额外硬件 | ESP32-S3 开发板（约 40 多元） | USB 线 |
| 任意 App（包括游戏） | 可以（只看像素） | 大多可以（取决于无障碍控件树） |
*"设置早上 8:00 的闹钟，每周一和周二重复，然后保存"*：Agent 退出自己的 App，打开时钟，在滚轮里输入时间，勾选日期并保存。（照片拍自 v0.1.0，任务相同。）
## 工作原理
- **眼：AirPlay 镜像。** 手机把屏幕镜像到电脑上运行的 [UxPlay](https://github.com/FDH2/UxPlay)，GStreamer 把视频流
转成一帧帧 JPEG，供 Agent 和网页使用。
- **手：USB HID 桥。** 插在手机上的 ESP32-S3 被识别为标准的 USB 鼠标和键盘。打开 iOS 的辅助触控（AssistiveTouch）后，
鼠标会控制屏幕上的一个指针：点击就是点按，拖动就是滑动。板子通过 Wi-Fi 连接电脑，只执行简单的行命令
（`move`、`click`、`type` 等），所有逻辑都在电脑上。
- **脑：规划模型 + 执行模型。** 规划模型根据目标和当前屏幕写一个简短的计划；执行模型每次看最新截图，选出下一个动作：
点按、滑动、画一笔、输入、按键、回主屏幕或结束。两个角色可以用同一个模型。
- **闭环校验。** 鼠标是相对移动的，所以服务端会记录指针位置。每次点按前，它会轻推一下指针，在镜像画面里找到它的真实位置，
偏了就先纠正，同时修正校准。每个动作之后检查屏幕是否真的变化了，并告诉模型。点空了会被发现并重试，而不是将错就错。
同一个动作重复多次会触发重新规划；断线或模型回复出错都能自动恢复。
```
iPhone ──屏幕镜像（Wi-Fi）──▶ ESP32-S3：AirPlay 接收端 ──蓝牙──▶ ESP32 Agent App（同一台 iPhone）
▲ 自带热点，H.264 原样转发 解码 → 视觉模型 → 动作
└──────── USB HID 鼠标 + 键盘 ◀──────────── 蓝牙命令 ◀──────────────────┘
```
- **眼睛：AirPlay 投到板子上。** 板子开自己的 Wi-Fi 热点，在手机的「屏幕镜像」里显示为 *ESP32 Agent*。它不解码，只把 H.264 码流经蓝牙转给 App。手机连着板子热点时，调用模型走蜂窝数据。
- **手：USB HID。** 板子是标准 USB 鼠标和键盘。打开辅助触控（AssistiveTouch）后，鼠标控制屏幕上的指针，点击就是轻点，拖动就是滑动。
- **大脑：App。** App 解码画面，把截图发给任意兼容 OpenAI 接口的视觉模型，由规划器和执行器决定动作：点、滑、画、输入、按键、回主屏、结束。工作时 App 在后台运行，灵动岛显示它在做什么。
- **闭环。** 指针是相对移动的，App 记录位置并定期回角落归零；精细点击前晃一下指针、在画面里找到它并修正偏差。每个动作后等画面稳定，再告诉模型画面有没有变化。板子、热点或镜像断开时暂停，恢复后从同一步继续。
- **复盘。** 每次运行都存在手机上，可以逐步回放，点按和滑动画在截图上。
## 硬件
| 部件 | 说明 |
|---|---|
| **ESP32-S3 开发板** | 实测使用 [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/cn/xiao_esp32s3_getting_started/) 普通版、未焊排针（不是带摄像头的 Sense 版，也不是 Plus 版）。国内在淘宝搜“XIAO ESP32S3”就能买到，约 40 多元。任何 USB-C 口直连 S3 原生 USB 的 ESP32-S3 板子都可以。 |
| **iPhone** | USB-C 机型直接插；Lightning 机型用 Lightning 转 USB 相机转换器。板子由手机供电。 |
| **电脑** | macOS 或 Linux，和手机在同一个局域网；运行 UxPlay 和服务端。 |
| **视觉模型** | 任何支持图片输入、兼容 OpenAI 接口的对话模型。本地部署是可选的。 |
<p align="center"><img src="docs/images/setup.png" alt="插在 iPhone 上的 XIAO ESP32S3，旁边是网页界面" width="100%"></p>
如果希望插着板子时手机还能充电，用一个支持 PD 充电直通的 USB-C 扩展坞。
| **ESP32-S3 板子** | 测试用的是 [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/xiao_esp32s3_getting_started/)（普通版，8 MB PSRAM；不是 Sense，也不是 Plus），淘宝搜 "XIAO ESP32S3"，约 40-50 元。USB-C 口接 S3 原生 USB、带 PSRAM 的 ESP32-S3 板子都应该可以。 |
| **iPhone** | iOS 18 及以上，需要蜂窝数据（Wi-Fi 被板子热点占用）。USB-C 机型直接插；Lightning 机型需要 Lightning 转 USB 相机转换器。板子由手机供电。 |
| **视觉模型** | 任何支持图片输入、兼容 OpenAI 接口的模型：云端 API 或自己部署。 |
| **装了 Xcode 的 Mac** | 只在刷板子、编译 App 时用一次，运行时不需要。 |
## 快速开始
> **让 AI 编程助手帮你装。** 在板子要连接的那台电脑上，把下面这段话发给 Claude Code、Codex 或类似的 Agent：
>
> ```text
> Clone https://github.com/onehumanbeing/esp32-iphone-agent and follow its README to set it up on this machine:
> flash the ESP32-S3 plugged in over USB, install UxPlay and the server, write server/.env for my model API, start
> the server and give me the web UI address. Ask me for the Wi-Fi name/password and the model API key.
> ```
### 1. 刷板子（插在 Mac 上）
### 1. 刷固件
安装 [arduino-cli](https://arduino.github.io/arduino-cli/) 和 ESP32 core（3.x），然后：
安装 [arduino-cli](https://arduino.github.io/arduino-cli/)、ESP32 core 3.3.x 和 NimBLE-Arduino 2.5.x，然后：
```bash
cp hardware/hid_bridge/secrets.example.h hardware/hid_bridge/secrets.h
# 编辑 secrets.h：填 2.4 GHz Wi-Fi 名称和密码，以及运行服务端的电脑的 IP
arduino-cli lib install "NimBLE-Arduino@2.5.1"
hardware/flash.sh # 可选参数：[串口] [--debug]
```
### 2. 安装并启动服务端
一定要用 `flash.sh` 编译：它会带上 `hardware/hid_bridge/nimble_cfg.h` 里的蓝牙缓冲区配置。
### 2. 安装 App
```bash
# UxPlay，macOS：
brew install uxplay
# UxPlay，Debian / Ubuntu：
sudo apt install uxplay gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-libav
cd server
python3 -m venv .venv && .venv/bin/pip install -e .
cp .env.example .env # 填 LLM_BASE_URL / LLM_MODEL / LLM_API_KEY
set -a && . ./.env && set +a && .venv/bin/python -m hidagent
cp ios/Config/Local.xcconfig.example ios/Config/Local.xcconfig # 填 DEVELOPMENT_TEAM 和 IA_BUNDLE_ID
open ios/IPhoneAgent.xcodeproj # 在 iPhone 上运行 IPhoneAgent scheme
```
浏览器打开 `http://<电脑 IP>:8890`。
工程由 [XcodeGen](https://github.com/yonaskolb/XcodeGen) 从 `ios/project.yml` 生成（`cd ios && xcodegen`）。免费 Apple ID 就能签名（每 7 天重签一次）。一键加入热点需要 Hotspot Configuration 能力（付费账号）；没有的话 App 会显示热点密码让你手动加入。
### 3. 设置手机（只需一次）
### 3. 首次设置（在 App 里）
1. **设置 → 辅助功能 → 触控 → 辅助触控**：打开。
2. 把板子插到手机上，屏幕上会出现一个圆形指针，网页显示控制器已连接。
3. **控制中心 → 屏幕镜像 → iPhone Agent**，网页里出现实时画面。
4. 在网页里打开 **手动操控 → 校准**。服务端会移动指针、在实时画面里找到它，几秒内自动完成校准；
如果找不到指针，会提示你在画面里精确点一下指针。
把板子插到 iPhone 上，打开 **ESP32 Agent**。引导共四步，每一步 App 自己检测：
### 4. 下达任务
1. **Plug Me In**：蓝牙配对板子。
2. **Let Me See the Screen**：加入板子的 Wi-Fi，然后 控制中心 → 屏幕镜像 → *ESP32 Agent*。
3. **Let Me Use the Pointer**：打开辅助触控，App 自动校准指针。
4. **Give Me a Brain**：填模型地址和密钥（存在钥匙串里）。
输入一个目标（例如"打开设置，进入 Wi-Fi"），点 **开始**；也可以直接点下面的示例。
### 4. 给它一个任务
## 网页界面
输入目标（比如 *"打开设置，进入无线局域网"*），点 **Start**。可以从灵动岛停止，或回到 App 停止。Siri、快捷指令、小组件和控制中心可以运行保存的任务。
- **实时手机画面**：点击 = 点按，拖动 = 滑动，滚轮 = 滚动；也可以手动回主屏幕、返回、输入文字和按键。
- **实时日志**：每一个计划、思考和动作实时显示，带耗时。
- **步骤截图墙**：每一步模型做决定时看到的截图，上面画出点按或滑动的位置，附带模型的理由和思考、执行各用了多久；
点卡片可以放大。
- **点按**用红色圆圈和扩散的波纹标出，滑动和笔画用黄色轨迹；**动效**开关可以关掉动画。模型标为 `precise`（点之前先确认指针位置）的点按会在日志里注明。
- **紧急停止**（或按 <kbd>Esc</kbd>）：立即停止 Agent 并松开指针。**清空记录**：清空日志和截图墙。
- 界面支持英文和简体中文。
## 模型和提示词
## 配置
写在 `server/.env` 里（参考 [`.env.example`](server/.env.example)）：
| 变量 | 默认值 | 说明 |
|---|---|---|
| `LLM_BASE_URL` | `http://localhost:8000/v1` | 兼容 OpenAI 的接口地址 |
| `LLM_MODEL` / `LLM_API_KEY` | — | 模型名称和 API Key |
| `LLM_EXTRA_BODY` | `{}` | 合并进每次请求的 JSON（例如关闭模型的思考模式） |
| `LLM_TOKEN_PARAM` | `auto` | 输出长度参数名：`max_tokens`、`max_completion_tokens`（新版 OpenAI 模型），或 `auto` 自动识别 |
| `ACTOR_*` / `PLANNER_*` | — | 单独覆盖执行模型 / 规划模型的任意 `LLM_*` 配置 |
| `AIRPLAY_NAME` | `iPhone Agent` | 手机"屏幕镜像"列表里显示的名字 |
| `PORT` / `DEVICE_PORT` | `8890` / `7300` | 网页端口 / 板子连接的端口 |
| `HIDAGENT_DATA` | `~/.hidagent` | 校准数据和调试记录 |
| `HIDAGENT_DEBUG` | `0` | 调试模式，见下文 |
### 本地模型（可选）
用云端 API 就够了。如果想全部跑在自己的硬件上，下面是本项目开发和测速时用的配置：
- 模型：[nvidia/Qwen3.8-Flash-Next-NVFP4](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4)
- 部署：[Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold](https://github.com/MiaAI-Lab/Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold)，
两台 NVIDIA DGX Spark 张量并行（TP=2），提供 OpenAI 兼容接口
- 速度：关闭思考模式后，每个动作约 1.3 秒，每次规划约 2 秒：
```bash
LLM_BASE_URL=http://<dgx-spark-ip>:8888/v1
LLM_MODEL=Qwen3.8-Flash-Next
LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'
```
其他兼容 OpenAI 接口的服务（vLLM、SGLang、llama.cpp、Ollama 等）配上视觉模型，用法完全一样。
## 调试模式
设置 `HIDAGENT_DEBUG=1` 后，每条板子命令及其耗时都会写进日志，每次 Agent 运行会单独保存一个文件夹：
```
~/.hidagent/traces/20261012-005207/
├── steps.jsonl # 每次模型调用（提示词、原始回复、耗时）和每个动作
├── 002.jpg # 每次模型调用看到的截图
└── ...
```
`hardware/flash.sh --debug` 会编译一个调试版固件，通过 USB 串口回显每条命令和耗时。
## 板子协议
板子本身不含任何逻辑。连上时先发送 `hello <上次重启原因> <开机毫秒数>`，之后服务端通过 TCP 每行发一条命令，板子回复 `ok` 或 `err <原因>`。
| 命令 | 作用 |
|---|---|
| `ping` | 健康检查 |
| `home` | 把指针推到左上角（一个已知位置） |
| `move <dx> <dy>` | 相对移动指针，匀速移动，让 iOS 的指针加速度可预测 |
| `press` / `release` / `click` | 左键按下 / 松开 / 点击 |
| `scroll <v> <h>` | 滚轮 |
| `type <text>` | 输入 ASCII 文字 |
| `key <combo>` | 按键，例如 `enter`、`cmd+h`、`cmd+space` |
Wi-Fi 断开时，板子会松开所有按键并自动重连。手机锁屏时 USB 会休眠：板子会先尝试唤醒手机，唤醒不了就立刻回复 `err phone usb asleep`，不会卡住等待。万一板子没有响应，看门狗会让它自动重启。
- 任何兼容 OpenAI 接口的视觉模型都能用（vLLM、SGLang、llama.cpp、Ollama、云端 API）。App 会列出接口上的模型并自动选择，也可以手动指定，规划器和执行器可以用不同模型。
- 提示词是 [`prompts/`](prompts/) 里的纯文本，打包进 App：改完重新编译即可。
- 开发时用的是 [nvidia/Qwen3.8-Flash-Next-NVFP4](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4)，部署在两台 DGX Spark 上（[TensorFold](https://github.com/MiaAI-Lab/Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold)）。
## 目录结构
```
hardware/ ESP32-S3 固件（Arduino）和刷写脚本
server/ Agent、网页界面、屏幕采集和板子连接（Python，FastAPI）
serve/：可选的极简服务，用来自己部署 Hugging Face 视觉模型
ios/ App（SwiftUI）：蓝牙连接、H.264 解码、Agent、引导、回放、实时活动、小组件；单元测试
prompts/ App 用的规划器 / 执行器提示词
hardware/ ESP32-S3 固件（Arduino）：蓝牙 GATT、热点、AirPlay 接收端、USB HID；刷写脚本、
主机测试、Mac 蓝牙调试工具（hardware/tools）
docs/ 蓝牙协议、旧版说明
server/ 旧版 v0.1.0 的电脑端服务，新版不使用（见 server/README.md）
```
不用手机 App 调试：`hardware/tools/board_test.py` 在 Mac 上通过蓝牙扮演 App（[hardware/tools/README.md](hardware/tools/README.md)）。板子与 App 的协议：[docs/ble-protocol.md](docs/ble-protocol.md)。
## 路线图
- [ ] 测试基于 DiffusionGemma 的纯本地方案（`server/serve/` 里有实验性的 OpenAI 兼容服务）
- [ ] 支持横屏 App（目前坐标和校准都按竖屏计算）
- [ ] 非 ASCII 文字输入（例如通过 iOS 拼音键盘输入中文）
- [ ] 更多常用 App 的示例任务
### 探索方向：把电脑也去掉
现在板子只负责"手"，"大脑"和"眼睛"都在另一台电脑上。
下一步想把电脑也去掉：把 ESP32 换成能跑 Linux 的小板子，看屏幕、做决策、点按全在板子上完成。手机只需要接这一块板子，放兜里就能带走。板子和手机之间可以走 USB 或蓝牙，模型可以用本地的，也可以用云端的。
这些是探索方向，不是承诺。感兴趣或者做过类似尝试的，欢迎开 issue 聊。
- [x] 去掉电脑：板子负责镜像和点击，手机上的 App 负责思考
- [ ] App Store / TestFlight 版本（用 HID Home 键退出 App，替换私有 API）
- [ ] 横屏 App（坐标和校准目前按竖屏）
- [ ] 非 ASCII 输入（比如用 iOS 拼音键盘输入中文）
- [ ] 更快：一次模型调用多个动作，只在需要时校正指针
## 限制
- iOS 不镜像部分界面（输入密码、Apple Pay、有版权保护的视频），Agent 看不到这些画面。
- 手机锁屏后镜像会断开，Agent 工作时请关闭自动锁定。
- 键盘只支持 ASCII 输入。
- 修改辅助触控的跟踪速度后需要重新校准。
- 暂不支持横屏 App：在横屏界面里的点按和绘图位置会偏。
- 在有多个网卡的 Linux 机器上，如果手机列表里看不到接收端，把 Avahi（mDNS）限制在手机所在网络的网卡上。
- 蓝牙约 300-500 kbps，所以镜像画面很小（480×1040，≤5 fps）；每一步要几秒，主要花在模型上。
- iOS 不镜像部分界面（输入密码、Apple Pay、DRM 视频），Agent 看不到。手机锁屏后镜像会停止。
- Agent 运行时，手机的 Wi-Fi 是板子的热点（没有外网），模型走蜂窝数据。
- App 用 iOS 私有 API（`suspend`）退出自己；自己编译的版本没问题，上架前会替换。
- 键盘输入只支持 ASCII。暂不支持横屏 App。
## 合理使用
## 负责任地使用
只自动化你自己拥有或被允许使用的设备和账号，并遵守相关 App 的条款。Agent 是在替你动手：请看着它，紧急停止按钮随时备好。
只自动化你拥有或有权使用的设备和账号，遵守相关 App 的条款。Agent 是在用你的手操作：看着它，随时准备停止。
## 参与贡献
欢迎提 Issue 和 Pull Request。提交前请先跑一遍检查：
```bash
cd server && pip install -e ".[dev]" && ruff check . && pytest
```
欢迎提 Issue 和 PR，见 [CONTRIBUTING.md](CONTRIBUTING.md)。安全问题：[SECURITY.md](SECURITY.md)。
## 相关项目
其他用代码或 AI 操作 iPhone 的方案，以及与本项目的区别。如有错误欢迎指正。
| 项目 | 方式 | 与本项目的区别 |
|---|---|---|
| [VersaiOS](https://github.com/zhaodashuaibi/VersaiOS) | UxPlay 镜像 + ESP32-S3 蓝牙鼠标 + 视觉模型 | 需要电脑；一条指令一次点击 |
| [ja1chen/actell](https://github.com/ja1chen/actell) | iPhone 上的 Agent App，走 Apple CoreDevice 调试通道 | 需要开发者模式和开发者磁盘映像 |
| [AidenAI-IO/aiden-firmware](https://github.com/AidenAI-IO/aiden-firmware) | HDMI 采集 + USB 键鼠 + 多模态模型 | 需要采集硬件；主要面向 Android |
| [yoyicue/glassbox](https://github.com/yoyicue/glassbox) | PicoKVM HDMI 采集 + USB HID | 需要 macOS 控制端和 PicoKVM |
| [boboabw/phone-agent](https://github.com/boboabw/phone-agent) | ESP32-S3 蓝牙 HID + 辅助触控，Agent 是 iPhone App | 最接近新版；本项目还把"眼睛"也放进了板子（AirPlay） |
| [VKCOM/devicehub](https://github.com/VKCOM/devicehub) | ESP32 蓝牙鼠标，用于 iOS 设备农场 | 设备农场工具，没有 AI Agent |
| 项目 | 做法 | 手机 / 主机需要 | 与本项目的区别 |
|---|---|---|---|
| [VersaiOS](https://github.com/zhaodashuaibi/VersaiOS) | UxPlay 镜像 + ESP32-S3 蓝牙鼠标 + 视觉模型 | ESP32 通过 USB 串口连电脑；桌面 GUI，主要是 Windows | 思路最接近。一条指令点一次，没有规划步骤，也不比对前后画面 |
| [ja1chen/actell（执言）](https://github.com/ja1chen/actell) | iPhone 上装 agent App，通过苹果 CoreDevice 调试通道截屏和点按 | 开发者模式、配对 + Developer Disk Image、Xcode 自签 | 本项目不需要开发者模式，不用在手机上装 App，也不用签名 |
| [AidenAI-IO/aiden-firmware](https://github.com/AidenAI-IO/aiden-firmware) | 专用板：HDMI 采集 + USB 键鼠 + 多模态模型 | HDMI 采集芯片和扩展坞；以 Android 为主 | 本项目用 AirPlay 代替 HDMI 采集，不需要采集硬件 |
| [yoyicue/glassbox](https://github.com/yoyicue/glassbox) | PicoKVM HDMI 采集 + USB HID，观察/决策/执行/验证循环 | macOS 控制端和 PicoKVM | 本项目在 Linux 上运行，只要一块 ESP32 |
| [boboabw/phone-agent](https://github.com/boboabw/phone-agent) | ESP32-S3 蓝牙 HID + 辅助触控，agent 是 iPhone App | 需要 Mac + Xcode 编译签名 | 本项目手机上什么都不用装 |
| [dovydaskel/Iphone-control-for-social-media](https://github.com/dovydaskel/Iphone-control-for-social-media) | ESP32 蓝牙鼠标 + OpenCV 指针校正，脚本化发帖 | Mac | 指针闭环思路相近；本项目多了视觉模型的规划/执行 |
| [VKCOM/devicehub](https://github.com/VKCOM/devicehub) | ESP32 蓝牙鼠标，用于 iOS 设备农场 | 设备农场基础设施 | 设备农场工具，没有 AI agent |
| [NolanCotter/AirBridge](https://github.com/NolanCotter/AirBridge) / [gbulog/pcairplay](https://github.com/gbulog/pcairplay) | Linux 上的 UxPlay + 电脑自带蓝牙模拟键鼠 | 带蓝牙的 Linux 电脑 | 人工远程操作，没有 AI |
其他路线：基于 WebDriverAgent/XCUITest 的工具（如 [mobile-mcp](https://github.com/mobile-next/mobile-mcp)、[Midscene](https://github.com/web-infra-dev/midscene)）需要 Mac 和代码签名；基于 macOS iPhone 镜像的工具需要 Mac。
基于 WebDriverAgent/XCUITest 的工具，如 [mobile-mcp](https://github.com/mobile-next/mobile-mcp) 和 [Midscene](https://github.com/web-infra-dev/midscene)，需要 Mac 和代码签名。
## 致谢
[UxPlay](https://github.com/FDH2/UxPlay)（AirPlay 接收端） ·
[Arduino ESP32 core](https://github.com/espressif/arduino-esp32) 和 [TinyUSB](https://github.com/hathach/tinyusb)（USB HID） ·
[FastAPI](https://fastapi.tiangolo.com/)
[UxPlay](https://github.com/FDH2/UxPlay)（固件里的 AirPlay 接收端移植自它）·
[NimBLE-Arduino](https://github.com/h2zero/NimBLE-Arduino) ·
[Arduino ESP32 core](https://github.com/espressif/arduino-esp32) 和 [TinyUSB](https://github.com/hathach/tinyusb)
## 许可证
[MIT](LICENSE)
[MIT](LICENSE)；`hardware/hid_bridge/src/airplay/`（移植自 UxPlay）除外，为 [GPL-3.0](hardware/hid_bridge/src/airplay/LICENSE-GPL.md)，因此链接了它的固件二进制按 GPL-3.0 分发。
SECURITY.md
+8
View File
@@ -0,0 +1,8 @@
# Security
This project drives a real phone, so bugs can have real effects. Please report security issues privately via
GitHub's "Report a vulnerability" (Security tab) instead of a public issue. Include steps to reproduce and the
version (commit) you tested.
Scope includes: the board's BLE service and hotspot (pairing, the `net` characteristic), the AirPlay receiver, and
anything that could make the agent act without the user starting it.
docs/images/demo.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.3 MiB

docs/images/logo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 154 KiB

docs/images/setup.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 834 KiB

docs/legacy.md
+23
View File
@@ -0,0 +1,23 @@
# The legacy version (v0.1.0)
v0.1.0 needed a computer next to the phone:
| | v0.1.0 (`legacy` branch) | current (`main`) |
|---|---|---|
| Eyes | UxPlay on a Mac/Linux computer | AirPlay receiver on the ESP32 itself |
| Brain | Python server + web UI on the computer | ESP32 Agent app on the same iPhone |
| Board ↔ brain | Wi-Fi TCP, line commands | BLE GATT ([ble-protocol.md](ble-protocol.md)) |
| Network | phone and computer on one Wi-Fi | board's own hotspot; model over cellular |
| On the phone | nothing installed | the app (built with Xcode) |
Why the change: the computer was the hard part to carry around and set up. Now the phone and a $7 board are the
whole system.
Use the old version:
```bash
git clone -b legacy https://github.com/onehumanbeing/esp32-iphone-agent # or: git checkout v0.1.0
```
and follow that README. The legacy firmware and the current one are not interchangeable: flash the one matching the
branch you use.
server/README.md
+13
View File
@@ -0,0 +1,13 @@
# server/ (legacy)
This is the computer-side server of the original version (tag `v0.1.0`, branch `legacy`): UxPlay on a Mac or Linux
machine receives the screen, this Python server runs the agent and a web UI, and the ESP32 is only a USB mouse that
connects to it over Wi-Fi.
**The current version does not use it.** The board receives AirPlay itself and the agent runs in the iPhone app
(`ios/`), with its prompts in `prompts/`. This folder is kept so the legacy setup keeps working and its tests still
run in CI; it gets fixes, not new features. Its prompts are the older ones in `hidagent/agent.py` and are not synced
with `prompts/`.
Note: the current firmware speaks BLE, not the TCP protocol this server expects. To run the legacy setup, check out
`legacy` (or `v0.1.0`) and follow its README.