Merge main before OpenRouter and reliability release

This commit is contained in:
Henry committed 2026-10-16 11:55:38 +08:00
commit d7ebad5fa7
3 files changed
+10 -2

No files matched your search

+5 -1
View File
@@ -27,6 +27,10 @@ Demo video and updates: [X @iamhenryyuan](https://x.com/iamhenryyuan) · [YouTub
*"Set an alarm for 8:00 AM that repeats every Monday and Tuesday, then save it"*: the agent leaves its own app,
opens Clock, types the time into the wheel, ticks the days and saves. (Photo from v0.1.0; the task is the same.)
<p align="center"><img src="docs/images/app-screens.png" alt="ESP32 Agent app: real phone vs. what the agent sees, and pointer calibration" width="70%"></p>
*Left: the real phone. Right: what the agent sees through the board's AirPlay mirror. The second shot is the pointer calibration.*
## How it works
```
@@ -121,10 +125,10 @@ Debugging without the phone app: `hardware/tools/board_test.py` plays the app ov
## Roadmap
- [x] Drop the computer: the board mirrors and clicks, the app on the phone thinks
- [ ] App Store / TestFlight build (leave the app with the HID Home key instead of a private API)
- [ ] Landscape apps (coordinates and calibration assume portrait)
- [ ] Non-ASCII text input (e.g. Chinese through the iOS pinyin keyboard)
- [ ] Faster steps: several actions per model call, pointer checks only when needed
- [ ] A curated list of models that work best with this setup, shown inside the iOS app ([suggested by @asharthinks](https://x.com/asharthinks/status/2110740775030125019))
## Limitations
+5 -1
View File
@@ -25,6 +25,10 @@
*"设置早上 8:00 的闹钟,每周一和周二重复,然后保存"*:Agent 退出自己的 App,打开时钟,在滚轮里输入时间,勾选日期并保存。(照片拍自 v0.1.0,任务相同。)
<p align="center"><img src="docs/images/app-screens.png" alt="ESP32 Agent app: real phone vs. what the agent sees, and pointer calibration" width="70%"></p>
*左:真机;右:Agent 通过板子的 AirPlay 镜像看到的画面。第二张是指针校准。*
## 工作原理
```
@@ -105,10 +109,10 @@ server/ 旧版 v0.1.0 的电脑端服务,新版不使用(见 server/READ
## 路线图
- [x] 去掉电脑:板子负责镜像和点击,手机上的 App 负责思考
- [ ] App Store / TestFlight 版本(用 HID Home 键退出 App,替换私有 API)
- [ ] 横屏 App(坐标和校准目前按竖屏)
- [ ] 非 ASCII 输入(比如用 iOS 拼音键盘输入中文)
- [ ] 更快:一次模型调用多个动作,只在需要时校正指针
- [ ] 在 iOS App 内整理一份效果最好的模型推荐列表([@asharthinks 的建议](https://x.com/asharthinks/status/2110740775030125019))
## 限制
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB