Merge main before OpenRouter and reliability release
This commit is contained in:
commit
d7ebad5fa7
3 files changed
+10
-2
No files matched your search
@@ -27,6 +27,10 @@ Demo video and updates: [X @iamhenryyuan](https://x.com/iamhenryyuan) · [YouTub
|
||||
*"Set an alarm for 8:00 AM that repeats every Monday and Tuesday, then save it"*: the agent leaves its own app,
|
||||
opens Clock, types the time into the wheel, ticks the days and saves. (Photo from v0.1.0; the task is the same.)
|
||||
|
||||
<p align="center"><img src="docs/images/app-screens.png" alt="ESP32 Agent app: real phone vs. what the agent sees, and pointer calibration" width="70%"></p>
|
||||
|
||||
*Left: the real phone. Right: what the agent sees through the board's AirPlay mirror. The second shot is the pointer calibration.*
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
@@ -121,10 +125,10 @@ Debugging without the phone app: `hardware/tools/board_test.py` plays the app ov
|
||||
## Roadmap
|
||||
|
||||
- [x] Drop the computer: the board mirrors and clicks, the app on the phone thinks
|
||||
- [ ] App Store / TestFlight build (leave the app with the HID Home key instead of a private API)
|
||||
- [ ] Landscape apps (coordinates and calibration assume portrait)
|
||||
- [ ] Non-ASCII text input (e.g. Chinese through the iOS pinyin keyboard)
|
||||
- [ ] Faster steps: several actions per model call, pointer checks only when needed
|
||||
- [ ] A curated list of models that work best with this setup, shown inside the iOS app ([suggested by @asharthinks](https://x.com/asharthinks/status/2110740775030125019))
|
||||
|
||||
## Limitations
|
||||
|
||||
|
||||
+5
-1
@@ -25,6 +25,10 @@
|
||||
|
||||
*"设置早上 8:00 的闹钟,每周一和周二重复,然后保存"*:Agent 退出自己的 App,打开时钟,在滚轮里输入时间,勾选日期并保存。(照片拍自 v0.1.0,任务相同。)
|
||||
|
||||
<p align="center"><img src="docs/images/app-screens.png" alt="ESP32 Agent app: real phone vs. what the agent sees, and pointer calibration" width="70%"></p>
|
||||
|
||||
*左:真机;右:Agent 通过板子的 AirPlay 镜像看到的画面。第二张是指针校准。*
|
||||
|
||||
## 工作原理
|
||||
|
||||
```
|
||||
@@ -105,10 +109,10 @@ server/ 旧版 v0.1.0 的电脑端服务,新版不使用(见 server/READ
|
||||
## 路线图
|
||||
|
||||
- [x] 去掉电脑:板子负责镜像和点击,手机上的 App 负责思考
|
||||
- [ ] App Store / TestFlight 版本(用 HID Home 键退出 App,替换私有 API)
|
||||
- [ ] 横屏 App(坐标和校准目前按竖屏)
|
||||
- [ ] 非 ASCII 输入(比如用 iOS 拼音键盘输入中文)
|
||||
- [ ] 更快:一次模型调用多个动作,只在需要时校正指针
|
||||
- [ ] 在 iOS App 内整理一份效果最好的模型推荐列表([@asharthinks 的建议](https://x.com/asharthinks/status/2110740775030125019))
|
||||
|
||||
## 限制
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.3 MiB |
Reference in new issue
Block a user