diff --git a/README.md b/README.md index d235db8..9876ffe 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,10 @@ Demo video and updates: [X @iamhenryyuan](https://x.com/iamhenryyuan) · [YouTub *"Set an alarm for 8:00 AM that repeats every Monday and Tuesday, then save it"*: the agent leaves its own app, opens Clock, types the time into the wheel, ticks the days and saves. (Photo from v0.1.0; the task is the same.) +

ESP32 Agent app: real phone vs. what the agent sees, and pointer calibration

+ +*Left: the real phone. Right: what the agent sees through the board's AirPlay mirror. The second shot is the pointer calibration.* + ## How it works ``` @@ -121,10 +125,10 @@ Debugging without the phone app: `hardware/tools/board_test.py` plays the app ov ## Roadmap - [x] Drop the computer: the board mirrors and clicks, the app on the phone thinks -- [ ] App Store / TestFlight build (leave the app with the HID Home key instead of a private API) - [ ] Landscape apps (coordinates and calibration assume portrait) - [ ] Non-ASCII text input (e.g. Chinese through the iOS pinyin keyboard) - [ ] Faster steps: several actions per model call, pointer checks only when needed +- [ ] A curated list of models that work best with this setup, shown inside the iOS app ([suggested by @asharthinks](https://x.com/asharthinks/status/2110740775030125019)) ## Limitations diff --git a/README.zh-CN.md b/README.zh-CN.md index ef995b1..345c1ad 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -25,6 +25,10 @@ *"设置早上 8:00 的闹钟,每周一和周二重复,然后保存"*:Agent 退出自己的 App,打开时钟,在滚轮里输入时间,勾选日期并保存。(照片拍自 v0.1.0,任务相同。) +

ESP32 Agent app: real phone vs. what the agent sees, and pointer calibration

+ +*左:真机;右:Agent 通过板子的 AirPlay 镜像看到的画面。第二张是指针校准。* + ## 工作原理 ``` @@ -105,10 +109,10 @@ server/ 旧版 v0.1.0 的电脑端服务,新版不使用(见 server/READ ## 路线图 - [x] 去掉电脑:板子负责镜像和点击,手机上的 App 负责思考 -- [ ] App Store / TestFlight 版本(用 HID Home 键退出 App,替换私有 API) - [ ] 横屏 App(坐标和校准目前按竖屏) - [ ] 非 ASCII 输入(比如用 iOS 拼音键盘输入中文) - [ ] 更快:一次模型调用多个动作,只在需要时校正指针 +- [ ] 在 iOS App 内整理一份效果最好的模型推荐列表([@asharthinks 的建议](https://x.com/asharthinks/status/2110740775030125019)) ## 限制 diff --git a/docs/images/app-screens.png b/docs/images/app-screens.png new file mode 100644 index 0000000..35fb4e1 Binary files /dev/null and b/docs/images/app-screens.png differ