docs: rewrite README — product-focused, bilingual (zh default / en)
This commit is contained in:
Kellyvv committed
1 parent
150964c660
commit
e177bc06f3
2 files changed
+183
-585
No files matched your search
@@ -1,369 +1,145 @@
|
||||
<div align="center">
|
||||
|
||||
# PhoneClaw
|
||||
|
||||
[English](./README_EN.md)
|
||||
**你的手机，你的 AI，完全在本地。**
|
||||
|
||||
PhoneClaw 是一个运行在 iPhone 上的本地 AI Agent。
|
||||
它使用 **Gemma 4 + MLX + Metal GPU** 在设备端完成推理，不依赖云端，不上传聊天内容，适合做一个真正离线、可调用原生能力的私人助手。
|
||||
[English](README_EN.md) · [报告问题](https://github.com/kellyvv/phoneclaw/issues) · [功能建议](https://github.com/kellyvv/phoneclaw/issues)
|
||||
|
||||
## 这是什么
|
||||
</div>
|
||||
|
||||
PhoneClaw 不是普通聊天壳，它是一个 `LLM + SKILL` 的本地 Agent：
|
||||
---
|
||||
|
||||
- LLM 负责理解用户意图
|
||||
- `SKILL.md` 负责描述某项能力该怎么做
|
||||
- 原生工具负责真正执行设备内操作
|
||||
PhoneClaw 是一个运行在 iPhone 上的本地 AI 助手。它不联网、不上传数据、不依赖任何云服务——所有推理完全在设备内完成。
|
||||
|
||||
当前已经打通的原生能力包括：
|
||||
你说一句话，它可以同时帮你安排日历、存联系人、创建提醒。没有弹窗，没有授权确认，静默完成。
|
||||
|
||||
- 剪贴板读写
|
||||
- 设备信息读取
|
||||
- 文本处理
|
||||
- 日历事件创建
|
||||
- 提醒事项创建
|
||||
- 通讯录联系人创建/更新
|
||||
## 能做什么
|
||||
|
||||
## 项目特点
|
||||
### 🗓️ 日历 · 提醒 · 通讯录
|
||||
|
||||
- 完全离线运行，默认不联网
|
||||
- 支持图片输入
|
||||
- 基于文件的 Skill 系统，新增能力不需要重写整个 Agent
|
||||
- 支持多轮工具调用
|
||||
- 已内置权限管理、System Prompt 编辑、模型切换
|
||||
- 针对 iPhone 内存上限做了缓存清理和历史裁剪
|
||||
一句话，三件事：
|
||||
|
||||
## 运行要求
|
||||
> "明天下午两点在高新园见字节的王总，他电话是 138xxxx，今晚八点提醒我发架构图"
|
||||
|
||||
- macOS + Xcode 16 或更新版本
|
||||
- iOS 17.0 或更新版本
|
||||
- CocoaPods
|
||||
- 真机调试账号
|
||||
PhoneClaw 自动拆解并执行——日历里多一条日程，通讯录里静默存入联系人，提醒事项准时触发。
|
||||
|
||||
模型建议：
|
||||
### 📋 剪贴板 · 设备信息 · 文本处理
|
||||
|
||||
- `Gemma 4 E2B`：更稳，更适合默认分发
|
||||
- `Gemma 4 E4B`：效果更强，但更吃内存，建议高端机型使用
|
||||
快速读写剪贴板、查询设备信息、执行文本转换，无需切换 App。
|
||||
|
||||
## 5 分钟快速开始
|
||||
### 🖼️ 看图理解
|
||||
|
||||
### 1. 安装依赖
|
||||
拍照或选图，直接询问图片内容。
|
||||
|
||||
### 🧩 自定义能力（Skill 系统）
|
||||
|
||||
用一个 Markdown 文件定义新能力，无需改代码，应用内热更新即刻生效。
|
||||
|
||||
---
|
||||
|
||||
## 隐私承诺
|
||||
|
||||
| | PhoneClaw | 云端 AI |
|
||||
|--|--|--|
|
||||
| 联网请求 | ❌ 从不 | ✅ 必须 |
|
||||
| 数据上传 | ❌ 从不 | ✅ 每次对话 |
|
||||
| 离线可用 | ✅ 完全 | ❌ 不行 |
|
||||
| 数据归属 | 你自己 | 服务商 |
|
||||
|
||||
---
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 环境要求
|
||||
|
||||
- iPhone（A16 芯片及以上；E4B 模型需要 A17 Pro）
|
||||
- Xcode 16+，iOS 17.0+
|
||||
- CocoaPods：`gem install cocoapods`
|
||||
|
||||
### 第一步：选择并下载模型
|
||||
|
||||
模型放在项目根目录的 `Models/` 文件夹下，**放什么就支持什么，不需要两个都放**。
|
||||
|
||||
**推荐 · E2B（约 1.5 GB，适合所有 A16+ 设备）**
|
||||
```
|
||||
Models/
|
||||
└── gemma-4-e2b-it-4bit/ ← 从 Hugging Face 下载 mlx-community/gemma-4-2b-it-4bit
|
||||
```
|
||||
|
||||
**进阶 · E4B（约 3 GB，需要 iPhone 15 Pro 及以上）**
|
||||
```
|
||||
Models/
|
||||
└── gemma-4-e4b-it-4bit/ ← 从 Hugging Face 下载 mlx-community/gemma-4-4b-it-4bit
|
||||
```
|
||||
|
||||
> `Models/` 已加入 `.gitignore`，模型文件不会上传到仓库。
|
||||
|
||||
### 第二步：安装依赖并打开
|
||||
|
||||
```bash
|
||||
pod install
|
||||
```
|
||||
|
||||
### 2. 下载模型
|
||||
|
||||
先选择你要打包的模型。PhoneClaw 当前识别的目录名必须和下面保持一致。
|
||||
|
||||
推荐先安装 Hugging Face CLI：
|
||||
|
||||
```bash
|
||||
brew install hf
|
||||
```
|
||||
|
||||
或者：
|
||||
|
||||
```bash
|
||||
pip install -U "huggingface_hub"
|
||||
```
|
||||
|
||||
只下载 `E2B`：
|
||||
|
||||
```bash
|
||||
mkdir -p ./Models/gemma-4-e2b-it-4bit
|
||||
hf download mlx-community/gemma-4-e2b-it-4bit --local-dir ./Models/gemma-4-e2b-it-4bit
|
||||
```
|
||||
|
||||
只下载 `E4B`：
|
||||
|
||||
```bash
|
||||
mkdir -p ./Models/gemma-4-e4b-it-4bit
|
||||
hf download mlx-community/gemma-4-e4b-it-4bit --local-dir ./Models/gemma-4-e4b-it-4bit
|
||||
```
|
||||
|
||||
两个都下载：
|
||||
|
||||
```bash
|
||||
mkdir -p ./Models/gemma-4-e2b-it-4bit
|
||||
mkdir -p ./Models/gemma-4-e4b-it-4bit
|
||||
hf download mlx-community/gemma-4-e2b-it-4bit --local-dir ./Models/gemma-4-e2b-it-4bit
|
||||
hf download mlx-community/gemma-4-e4b-it-4bit --local-dir ./Models/gemma-4-e4b-it-4bit
|
||||
```
|
||||
|
||||
下载后目录大致如下：
|
||||
|
||||
```text
|
||||
Models/
|
||||
├── gemma-4-e2b-it-4bit/
|
||||
│ ├── config.json
|
||||
│ ├── tokenizer.json
|
||||
│ ├── processor_config.json
|
||||
│ ├── chat_template.jinja
|
||||
│ ├── model.safetensors
|
||||
│ └── model.safetensors.index.json
|
||||
└── gemma-4-e4b-it-4bit/
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- `Models/` 已在 `.gitignore` 中忽略，不会被提交到仓库
|
||||
- 当前 Hugging Face 模型页显示的仓库体积大约是：`E2B 3.58 GB`、`E4B 5.22 GB`
|
||||
- 如果你不想用 CLI，也可以直接从模型页面手动下载后放到对应目录
|
||||
|
||||
### 3. 打开工程
|
||||
|
||||
```bash
|
||||
open PhoneClaw.xcworkspace
|
||||
```
|
||||
|
||||
不要打开 `.xcodeproj`，请始终打开 `.xcworkspace`。
|
||||
> ⚠️ 必须打开 `.xcworkspace`，不要直接打开 `.xcodeproj`
|
||||
|
||||
### 4. 配置签名并运行
|
||||
### 第三步：签名并运行
|
||||
|
||||
在 Xcode 中：
|
||||
1. Xcode 选择 **PhoneClaw** Target → **Signing & Capabilities**
|
||||
2. 填入你的 **Apple ID Team**
|
||||
3. 修改 **Bundle Identifier**（如 `com.yourname.phoneclaw`）
|
||||
4. USB 连接 iPhone，按 **⌘R**
|
||||
|
||||
1. 选择 `PhoneClaw` target
|
||||
2. 打开 `Signing & Capabilities`
|
||||
3. 选择你的 `Team`
|
||||
4. 把 `Bundle Identifier` 改成你自己的唯一值
|
||||
5. 连接 iPhone
|
||||
6. 按 `⌘R`
|
||||
首次安装需信任证书：**设置 → 通用 → VPN 与设备管理 → 信任**
|
||||
|
||||
首次安装后，如果系统要求信任开发者证书，请在手机里完成信任。
|
||||
---
|
||||
|
||||
### 5. 首次使用
|
||||
## 内置能力
|
||||
|
||||
打开 App 后：
|
||||
| 能力 | 说明 |
|
||||
|------|------|
|
||||
| 📅 日历 | 创建日历事项，支持标题、时间、地点 |
|
||||
| ⏰ 提醒事项 | 创建提醒，到时间自动推送通知 |
|
||||
| 👤 通讯录 | 新建或更新联系人，按手机号自动去重 |
|
||||
| 📋 剪贴板 | 读取和写入系统剪贴板 |
|
||||
| 📱 设备信息 | 查询设备名称、系统版本、内存等 |
|
||||
| 🔤 文本工具 | 哈希计算、文本翻转等 |
|
||||
|
||||
- 右上角拼图按钮：Skill 管理
|
||||
- 右上角滑杆按钮：模型设置 / 系统提示词 / 权限
|
||||
---
|
||||
|
||||
建议先在权限页开启：
|
||||
## 添加自定义能力
|
||||
|
||||
- 日历
|
||||
- 提醒事项
|
||||
- 通讯录
|
||||
在设备的 App 数据目录下创建一个 `SKILL.md` 文件即可，应用内重载后立即生效：
|
||||
|
||||
然后直接试下面几句：
|
||||
|
||||
- `这台手机的设备信息是什么`
|
||||
- `提醒我今晚八点发文件`
|
||||
- `帮我存一下王总的电话 13812345678`
|
||||
|
||||
## 如何只打包一个模型
|
||||
|
||||
这是最常见的发布方式，尤其是你只想发 `E2B` 时。
|
||||
|
||||
### 方案 A：只打包 `E2B`
|
||||
|
||||
保留：
|
||||
|
||||
```text
|
||||
Models/gemma-4-e2b-it-4bit
|
||||
```
|
||||
|
||||
移除：
|
||||
|
||||
```text
|
||||
Models/gemma-4-e4b-it-4bit
|
||||
ApplicationSupport/PhoneClaw/skills/<能力名>/SKILL.md
|
||||
```
|
||||
|
||||
然后做两步：
|
||||
|
||||
1. 在 Xcode 的 Project Navigator 里删除不用的模型 folder reference，选择 `Remove Reference`
|
||||
2. 在 `PhoneClaw > Build Phases > Copy Bundle Resources` 里确认只剩你要打包的模型
|
||||
|
||||
最后，修改 [LLM/MLXLocalLLMService.swift](./LLM/MLXLocalLLMService.swift) 里的 `availableModels`，只保留实际会随 App 分发的模型。
|
||||
否则配置页仍然会显示不存在的模型选项。
|
||||
|
||||
### 方案 B：同时打包 `E2B + E4B`
|
||||
|
||||
保留两个目录和两个 Xcode 资源引用即可。
|
||||
用户可在 App 的 `模型设置 / Model Settings` 页里切换。
|
||||
|
||||
## 内置 Skill
|
||||
|
||||
| Skill | 工具 |
|
||||
| --- | --- |
|
||||
| 剪贴板 | `clipboard-read`, `clipboard-write` |
|
||||
| 设备 | `device-info`, `device-name`, `device-model`, `device-system-version`, `device-memory`, `device-processor-count` |
|
||||
| 文本 | `calculate-hash`, `text-reverse` |
|
||||
| 日历 | `calendar-create-event` |
|
||||
| 提醒事项 | `reminders-create` |
|
||||
| 通讯录 | `contacts-upsert` |
|
||||
|
||||
## 自定义 Skill
|
||||
|
||||
新增一个 Skill 的最小成本方式，是在应用目录里增加一个 `SKILL.md`。
|
||||
|
||||
示例结构：
|
||||
|
||||
```text
|
||||
Application Support/PhoneClaw/skills/<skill-id>/SKILL.md
|
||||
```
|
||||
|
||||
示例：
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: MySkill
|
||||
name-zh: 我的能力
|
||||
description: 这个 Skill 的作用
|
||||
name: 我的能力
|
||||
description: '简单描述这个能力做什么'
|
||||
version: "1.0.0"
|
||||
icon: star
|
||||
disabled: false
|
||||
|
||||
triggers:
|
||||
- 关键词1
|
||||
- 关键词
|
||||
|
||||
allowed-tools:
|
||||
- my-tool-name
|
||||
|
||||
examples:
|
||||
- query: "用户会怎么说"
|
||||
scenario: "什么场景会触发"
|
||||
---
|
||||
|
||||
# Skill 指令
|
||||
# 能力说明
|
||||
|
||||
告诉模型何时调用工具、如何组织参数、何时直接回答。
|
||||
告诉 AI 在什么情况下使用这个能力，以及如何调用工具。
|
||||
```
|
||||
|
||||
如果这个 Skill 需要真正调用系统能力，再去 [Skills/ToolRegistry.swift](./Skills/ToolRegistry.swift) 注册对应工具。
|
||||
需要调用原生 iOS API 的工具，在 `Skills/ToolRegistry.swift` 中注册即可。
|
||||
|
||||
## 关键目录
|
||||
|
||||
```text
|
||||
PhoneClaw/
|
||||
├── App/ # App 入口
|
||||
├── Agent/ # Agent 循环与多轮工具调用
|
||||
├── LLM/ # MLX 本地推理与 Prompt 构建
|
||||
├── Skills/ # Skill 解析、工具注册、数据模型
|
||||
├── UI/ # 聊天界面、Skill 管理、配置页面
|
||||
├── Models/ # 本地模型目录（默认不入库）
|
||||
├── PhoneClaw.xcworkspace
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## 实际执行链路
|
||||
|
||||
```text
|
||||
用户输入
|
||||
→ PromptBuilder 组装提示词
|
||||
→ Gemma 4 本地推理
|
||||
→ 需要能力时调用 load_skill
|
||||
→ 读取对应 SKILL.md
|
||||
→ 执行原生工具
|
||||
→ 返回最终中文结果
|
||||
```
|
||||
|
||||
## 常见问题
|
||||
|
||||
### 为什么安装后看不到权限弹窗？
|
||||
|
||||
通常是因为对应 Skill 还没有真正执行到系统 API。
|
||||
如果之前已经拒绝过一次，iOS 也不会反复弹框，需要到系统设置里手动开启。
|
||||
|
||||
### 为什么切模型后加载失败？
|
||||
|
||||
先确认：
|
||||
|
||||
- 模型目录名和代码里的 `availableModels` 一致
|
||||
- 该模型确实被打进了 App 包
|
||||
- 设备内存足够
|
||||
|
||||
### 为什么提醒事项创建失败？
|
||||
|
||||
最新代码会先尝试复用现有提醒列表；如果系统里没有可写列表，会再尝试自动创建一个 `PhoneClaw` 提醒列表。
|
||||
如果这一步仍失败，通常是系统提醒源本身不可写。
|
||||
|
||||
## 后续计划
|
||||
|
||||
PhoneClaw 接下来的方向，不只是“多加几个工具”，而是把它逐步做成一个真正可用的本地 iPhone Agent。
|
||||
|
||||
### 1. 扩展更多 iOS 原生 API
|
||||
|
||||
计划继续补充更多系统能力，例如：
|
||||
|
||||
- 文件与目录操作
|
||||
- 照片读取、整理、描述、检索
|
||||
- 备忘录 / Notes
|
||||
- 本地通知
|
||||
- 地图 / 位置相关能力
|
||||
- Safari / URL 打开与上下文传递
|
||||
- 更多通讯录、日历、提醒事项的读写能力
|
||||
|
||||
### 2. 扩展更多 Skill
|
||||
|
||||
后续会继续把能力拆成更清晰的 Skill，而不是把所有逻辑都堆在一个大 Prompt 里。
|
||||
|
||||
适合继续追加的 Skill 方向：
|
||||
|
||||
- 文件管理
|
||||
- 照片理解与整理
|
||||
- 日程规划
|
||||
- 个人信息管理
|
||||
- 本地知识库检索
|
||||
- 语音输入 / 语音播报
|
||||
|
||||
### 3. 串联更多本地模型
|
||||
|
||||
除了主聊天模型之外，后续适合接入的本地模型还包括：
|
||||
|
||||
- OCR 模型
|
||||
- 语音识别模型
|
||||
- 语音合成模型
|
||||
- Embedding / Reranker 模型
|
||||
- 更小的工具参数提取模型
|
||||
- 更强的规划模型或多模型协作链路
|
||||
|
||||
这会让 PhoneClaw 从“一个大模型做所有事”，逐渐演进成“多个本地模型协同工作”的架构。
|
||||
|
||||
### 4. 跨 App 自动化
|
||||
|
||||
这是很重要的方向，但会严格遵守 iOS 的安全边界。
|
||||
PhoneClaw 不会假设自己能像桌面系统那样任意操控所有 App，而是优先走 iOS 真正允许的能力：
|
||||
|
||||
- `App Intents`
|
||||
- `Shortcuts`
|
||||
- `URL Scheme / Deep Link`
|
||||
- `Share Sheet / 分享扩展`
|
||||
- 剪贴板中转
|
||||
- 系统通知与唤起
|
||||
|
||||
更现实的目标是：
|
||||
|
||||
- 在 App 之间传递内容
|
||||
- 拉起指定 App 到指定页面
|
||||
- 把本地理解能力和快捷指令工作流串起来
|
||||
- 把多步操作压缩成一条自然语言命令
|
||||
|
||||
### 5. 外部硬件与视觉扩展
|
||||
|
||||
除了手机内部能力之外，PhoneClaw 后续也会探索和外部硬件配合的形态。
|
||||
例如把外部视频输入、屏幕画面理解和本地模型串起来，让 PhoneClaw 不只是“在手机里回答问题”，而是逐步具备更强的现实世界感知与调度能力。
|
||||
|
||||
这部分暂时不展开太多，后面会留一些更有意思的玩法。
|
||||
|
||||
### 6. 我建议优先做的方向
|
||||
|
||||
如果按“最容易尽快做出体验差异”的顺序，我会建议优先做：
|
||||
|
||||
1. 文件 / 照片 / 备忘录 三类高频 API
|
||||
2. Shortcuts / App Intents 集成
|
||||
3. OCR + 语音识别
|
||||
4. 本地知识库检索
|
||||
5. 更细的自动化 Skill 编排
|
||||
|
||||
这样很快就能把 PhoneClaw 从“会聊天的本地模型”推进到“能真正帮用户做事的本地助手”。
|
||||
|
||||
## 参考链接
|
||||
|
||||
- Hugging Face CLI 文档: <https://huggingface.co/docs/huggingface_hub/guides/cli>
|
||||
- Hugging Face 下载文档: <https://huggingface.co/docs/huggingface_hub/en/guides/download>
|
||||
- Gemma 4 E2B MLX 模型: <https://huggingface.co/mlx-community/gemma-4-e2b-it-4bit>
|
||||
- Gemma 4 E4B MLX 模型: <https://huggingface.co/mlx-community/gemma-4-e4b-it-4bit>
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
MIT — 自由使用、修改和分发。
|
||||
+92
-270
@@ -1,196 +1,126 @@
|
||||
<div align="center">
|
||||
|
||||
# PhoneClaw
|
||||
|
||||
[简体中文](./README.md)
|
||||
**Your phone. Your AI. Entirely on-device.**
|
||||
|
||||
PhoneClaw is a local AI agent that runs directly on iPhone.
|
||||
It uses **Gemma 4 + MLX + Metal GPU** for on-device inference, so chats and tool calls stay on the device by default.
|
||||
[中文](README.md) · [Report an Issue](https://github.com/kellyvv/phoneclaw/issues) · [Request a Feature](https://github.com/kellyvv/phoneclaw/issues)
|
||||
|
||||
## What It Is
|
||||
</div>
|
||||
|
||||
PhoneClaw is an `LLM + SKILL` style local agent:
|
||||
---
|
||||
|
||||
- the LLM decides what the user wants
|
||||
- `SKILL.md` files describe how a capability should behave
|
||||
- native tools execute the actual device-side action
|
||||
PhoneClaw is a local AI assistant that runs entirely on your iPhone. No internet connection. No data uploads. No cloud dependency — all inference happens on-device.
|
||||
|
||||
Built-in native capabilities currently include:
|
||||
Say one sentence, and it can schedule a meeting, save a contact, and set a reminder — all at once, silently, with no confirmation dialogs.
|
||||
|
||||
- clipboard read/write
|
||||
- device info
|
||||
- text utilities
|
||||
- calendar event creation
|
||||
- reminders creation
|
||||
- contacts create/update
|
||||
## What It Can Do
|
||||
|
||||
## Highlights
|
||||
### 🗓️ Calendar · Reminders · Contacts
|
||||
|
||||
- fully offline by default
|
||||
- image input supported
|
||||
- file-driven skill system
|
||||
- multi-step tool calling
|
||||
- built-in permissions page, system prompt editor, and model switcher
|
||||
- memory-aware MLX runtime management for iPhone limits
|
||||
One sentence. Three things done:
|
||||
|
||||
## Requirements
|
||||
> "Tomorrow at 2pm, meet with Alex from Bytedance at Hightech Park. His number is 138xxxx. Also remind me tonight at 8 to send the architecture doc."
|
||||
|
||||
- macOS + Xcode 16 or newer
|
||||
- iOS 17.0 or newer
|
||||
- CocoaPods
|
||||
- an Apple developer signing identity for real-device testing
|
||||
PhoneClaw breaks it down and executes — a calendar event is created, a contact is saved silently, and a reminder fires at exactly 8pm.
|
||||
|
||||
Model guidance:
|
||||
### 📋 Clipboard · Device Info · Text Tools
|
||||
|
||||
- `Gemma 4 E2B`: better default choice for distribution
|
||||
- `Gemma 4 E4B`: stronger but heavier, better for high-end devices
|
||||
Read or write the clipboard, query your device specs, run text transformations — without leaving the app.
|
||||
|
||||
## Quick Start
|
||||
### 🖼️ Image Understanding
|
||||
|
||||
### 1. Install dependencies
|
||||
Take a photo or choose one from your library, then ask questions about it.
|
||||
|
||||
### 🧩 Custom Skills
|
||||
|
||||
Define new capabilities using a single Markdown file. No code changes. Hot-reload inside the app.
|
||||
|
||||
---
|
||||
|
||||
## Privacy Guarantee
|
||||
|
||||
| | PhoneClaw | Cloud AI |
|
||||
|--|--|--|
|
||||
| Network requests | ❌ Never | ✅ Every message |
|
||||
| Data uploads | ❌ Never | ✅ Always |
|
||||
| Works offline | ✅ Fully | ❌ No |
|
||||
| Your data belongs to | You | The vendor |
|
||||
|
||||
---
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Requirements
|
||||
|
||||
- iPhone with Apple Silicon (A16 or later; E4B model requires A17 Pro)
|
||||
- Xcode 16+, iOS 17.0+
|
||||
- CocoaPods: `gem install cocoapods`
|
||||
|
||||
### Step 1 — Download a Model
|
||||
|
||||
Place the model under the `Models/` folder in the project root. **You only need one** — place what you want to use.
|
||||
|
||||
**Recommended · E2B (~1.5 GB, works on all A16+ devices)**
|
||||
```
|
||||
Models/
|
||||
└── gemma-4-e2b-it-4bit/ ← Download mlx-community/gemma-4-2b-it-4bit from Hugging Face
|
||||
```
|
||||
|
||||
**Advanced · E4B (~3 GB, requires iPhone 15 Pro or later)**
|
||||
```
|
||||
Models/
|
||||
└── gemma-4-e4b-it-4bit/ ← Download mlx-community/gemma-4-4b-it-4bit from Hugging Face
|
||||
```
|
||||
|
||||
> `Models/` is gitignored. Model files are never committed to the repository.
|
||||
|
||||
### Step 2 — Install & Open
|
||||
|
||||
```bash
|
||||
pod install
|
||||
```
|
||||
|
||||
### 2. Download a model
|
||||
|
||||
PhoneClaw expects these exact folder names under `Models/`.
|
||||
|
||||
Install the Hugging Face CLI first:
|
||||
|
||||
```bash
|
||||
brew install hf
|
||||
```
|
||||
|
||||
or:
|
||||
|
||||
```bash
|
||||
pip install -U "huggingface_hub"
|
||||
```
|
||||
|
||||
Download only `E2B`:
|
||||
|
||||
```bash
|
||||
mkdir -p ./Models/gemma-4-e2b-it-4bit
|
||||
hf download mlx-community/gemma-4-e2b-it-4bit --local-dir ./Models/gemma-4-e2b-it-4bit
|
||||
```
|
||||
|
||||
Download only `E4B`:
|
||||
|
||||
```bash
|
||||
mkdir -p ./Models/gemma-4-e4b-it-4bit
|
||||
hf download mlx-community/gemma-4-e4b-it-4bit --local-dir ./Models/gemma-4-e4b-it-4bit
|
||||
```
|
||||
|
||||
Download both:
|
||||
|
||||
```bash
|
||||
mkdir -p ./Models/gemma-4-e2b-it-4bit
|
||||
mkdir -p ./Models/gemma-4-e4b-it-4bit
|
||||
hf download mlx-community/gemma-4-e2b-it-4bit --local-dir ./Models/gemma-4-e2b-it-4bit
|
||||
hf download mlx-community/gemma-4-e4b-it-4bit --local-dir ./Models/gemma-4-e4b-it-4bit
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `Models/` is already ignored by Git
|
||||
- the current Hugging Face repo sizes are roughly `3.58 GB` for `E2B` and `5.22 GB` for `E4B`
|
||||
- you can also download files manually from the model pages if you prefer
|
||||
|
||||
### 3. Open the workspace
|
||||
|
||||
```bash
|
||||
open PhoneClaw.xcworkspace
|
||||
```
|
||||
|
||||
Always open the workspace, not the project file.
|
||||
> ⚠️ Always open `.xcworkspace`, not `.xcodeproj`
|
||||
|
||||
### 4. Configure signing and run
|
||||
### Step 3 — Sign & Run
|
||||
|
||||
In Xcode:
|
||||
1. In Xcode: select the **PhoneClaw** target → **Signing & Capabilities**
|
||||
2. Set your **Team** (Apple ID)
|
||||
3. Change the **Bundle Identifier** (e.g. `com.yourname.phoneclaw`)
|
||||
4. Connect your iPhone via USB and press **⌘R**
|
||||
|
||||
1. select the `PhoneClaw` target
|
||||
2. open `Signing & Capabilities`
|
||||
3. choose your `Team`
|
||||
4. set a unique `Bundle Identifier`
|
||||
5. connect your iPhone
|
||||
6. press `⌘R`
|
||||
On first install, trust the certificate on your iPhone:
|
||||
**Settings → General → VPN & Device Management → Trust**
|
||||
|
||||
### 5. First launch
|
||||
|
||||
Inside the app:
|
||||
|
||||
- top-right puzzle icon: Skill manager
|
||||
- top-right sliders icon: model settings / system prompt / permissions
|
||||
|
||||
It is recommended to enable these permissions first:
|
||||
|
||||
- Calendar
|
||||
- Reminders
|
||||
- Contacts
|
||||
|
||||
Then try:
|
||||
|
||||
- `What is this phone's device information?`
|
||||
- `Remind me to send the file tonight at 8 PM`
|
||||
- `Save Wang Zong's number 13812345678`
|
||||
|
||||
## How to Ship Only One Model
|
||||
|
||||
This is the most practical setup for distribution.
|
||||
|
||||
### Option A: Ship only `E2B`
|
||||
|
||||
Keep:
|
||||
|
||||
```text
|
||||
Models/gemma-4-e2b-it-4bit
|
||||
```
|
||||
|
||||
Remove:
|
||||
|
||||
```text
|
||||
Models/gemma-4-e4b-it-4bit
|
||||
```
|
||||
|
||||
Then do both of the following:
|
||||
|
||||
1. remove the unused model folder reference in Xcode with `Remove Reference`
|
||||
2. verify `PhoneClaw > Build Phases > Copy Bundle Resources` only includes the model you want to ship
|
||||
|
||||
Finally, update [LLM/MLXLocalLLMService.swift](./LLM/MLXLocalLLMService.swift) so `availableModels` only contains the model(s) you actually distribute.
|
||||
Otherwise the configuration page will still show unavailable options.
|
||||
|
||||
### Option B: Ship both `E2B + E4B`
|
||||
|
||||
Keep both model folders and both Xcode resource references.
|
||||
Users can switch models from the in-app configuration page.
|
||||
---
|
||||
|
||||
## Built-in Skills
|
||||
|
||||
| Skill | Tools |
|
||||
| --- | --- |
|
||||
| Clipboard | `clipboard-read`, `clipboard-write` |
|
||||
| Device | `device-info`, `device-name`, `device-model`, `device-system-version`, `device-memory`, `device-processor-count` |
|
||||
| Text | `calculate-hash`, `text-reverse` |
|
||||
| Calendar | `calendar-create-event` |
|
||||
| Reminders | `reminders-create` |
|
||||
| Contacts | `contacts-upsert` |
|
||||
| Skill | Description |
|
||||
|-------|-------------|
|
||||
| 📅 Calendar | Create calendar events with title, time, and location |
|
||||
| ⏰ Reminders | Create reminders with push notifications at the due time |
|
||||
| 👤 Contacts | Create or update contacts, deduped by phone number |
|
||||
| 📋 Clipboard | Read and write the system clipboard |
|
||||
| 📱 Device Info | Query device name, OS version, memory, and more |
|
||||
| 🔤 Text Tools | Hash calculation, text reversal, and more |
|
||||
|
||||
## Custom Skills
|
||||
---
|
||||
|
||||
The smallest way to add a new capability is to add a new `SKILL.md`:
|
||||
## Adding Custom Skills
|
||||
|
||||
Create a `SKILL.md` file in the app's data directory and hot-reload in-app:
|
||||
|
||||
```text
|
||||
Application Support/PhoneClaw/skills/<skill-id>/SKILL.md
|
||||
```
|
||||
|
||||
Example:
|
||||
ApplicationSupport/PhoneClaw/skills/<skill-name>/SKILL.md
|
||||
```
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: MySkill
|
||||
name-zh: 我的能力
|
||||
description: What this skill does
|
||||
name: My Skill
|
||||
description: 'What this skill does'
|
||||
version: "1.0.0"
|
||||
icon: star
|
||||
disabled: false
|
||||
@@ -200,125 +130,17 @@ triggers:
|
||||
|
||||
allowed-tools:
|
||||
- my-tool-name
|
||||
|
||||
examples:
|
||||
- query: "How a user might ask for it"
|
||||
scenario: "When it should trigger"
|
||||
---
|
||||
|
||||
# Skill instructions
|
||||
# Instructions
|
||||
|
||||
Tell the model when to call tools, how to build arguments, and when to answer directly.
|
||||
Tell the AI when and how to use this skill and its tools.
|
||||
```
|
||||
|
||||
If the skill needs real native execution, register the tool in [Skills/ToolRegistry.swift](./Skills/ToolRegistry.swift).
|
||||
To call native iOS APIs, register your tool in `Skills/ToolRegistry.swift`.
|
||||
|
||||
## Key Directories
|
||||
|
||||
```text
|
||||
PhoneClaw/
|
||||
├── App/
|
||||
├── Agent/
|
||||
├── LLM/
|
||||
├── Skills/
|
||||
├── UI/
|
||||
├── Models/
|
||||
├── PhoneClaw.xcworkspace
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Runtime Flow
|
||||
|
||||
```text
|
||||
User input
|
||||
→ PromptBuilder
|
||||
→ local Gemma 4 inference
|
||||
→ load_skill when needed
|
||||
→ read SKILL.md
|
||||
→ execute native tool
|
||||
→ produce final answer
|
||||
```
|
||||
|
||||
## Useful Links
|
||||
|
||||
- Hugging Face CLI docs: <https://huggingface.co/docs/huggingface_hub/guides/cli>
|
||||
- Hugging Face download guide: <https://huggingface.co/docs/huggingface_hub/en/guides/download>
|
||||
- Gemma 4 E2B MLX model: <https://huggingface.co/mlx-community/gemma-4-e2b-it-4bit>
|
||||
- Gemma 4 E4B MLX model: <https://huggingface.co/mlx-community/gemma-4-e4b-it-4bit>
|
||||
|
||||
## Roadmap
|
||||
|
||||
PhoneClaw is meant to grow into a genuinely useful on-device iPhone agent, not just a chat UI with a couple of tools.
|
||||
|
||||
### 1. More iOS native APIs
|
||||
|
||||
Planned areas include:
|
||||
|
||||
- files and folders
|
||||
- photo reading, organization, and understanding
|
||||
- Notes integration
|
||||
- local notifications
|
||||
- maps and location-related tasks
|
||||
- Safari / URL handoff
|
||||
- broader calendar, reminders, and contacts workflows
|
||||
|
||||
### 2. More Skills
|
||||
|
||||
The long-term direction is to keep capabilities modular through Skills instead of pushing everything into one giant prompt.
|
||||
|
||||
Likely next skills:
|
||||
|
||||
- file management
|
||||
- photo understanding and organization
|
||||
- schedule planning
|
||||
- personal information management
|
||||
- local knowledge retrieval
|
||||
- voice input / voice output
|
||||
|
||||
### 3. More local models working together
|
||||
|
||||
Beyond the main chat model, good future additions include:
|
||||
|
||||
- OCR models
|
||||
- speech recognition models
|
||||
- text-to-speech models
|
||||
- embedding / reranker models
|
||||
- smaller argument-extraction models
|
||||
- stronger planners or multi-model orchestration
|
||||
|
||||
That would move PhoneClaw from “one model does everything” toward a more practical local multi-model agent stack.
|
||||
|
||||
### 4. Cross-app automation
|
||||
|
||||
This is a key direction, but it needs to stay realistic within iOS security limits.
|
||||
The project is not assuming unrestricted UI control over every app. Instead, the practical path is:
|
||||
|
||||
- `App Intents`
|
||||
- `Shortcuts`
|
||||
- `URL Schemes / Deep Links`
|
||||
- `Share Sheet / Share Extensions`
|
||||
- clipboard handoff
|
||||
- notification-driven flows
|
||||
|
||||
The goal is to make cross-app workflows feel natural while staying inside what iOS actually allows.
|
||||
|
||||
### 5. External hardware and visual expansion
|
||||
|
||||
Beyond the phone itself, PhoneClaw is also intended to explore workflows that involve external hardware.
|
||||
That may include combining external video input, on-device visual understanding, and local models so PhoneClaw can gradually move from “answering inside the phone” toward richer real-world perception and orchestration.
|
||||
|
||||
This part is intentionally kept a little vague for now.
|
||||
|
||||
### 6. Suggested priorities
|
||||
|
||||
If the goal is to improve real user value quickly, the strongest order is:
|
||||
|
||||
1. files / photos / notes
|
||||
2. Shortcuts / App Intents integration
|
||||
3. OCR + speech recognition
|
||||
4. local knowledge retrieval
|
||||
5. richer automation-oriented skill composition
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
MIT — free to use, modify, and distribute.
|
||||
Reference in new issue
Block a user