diff --git a/README.md b/README.md index aae9990..8802c3c 100644 --- a/README.md +++ b/README.md @@ -1,165 +1,369 @@ # PhoneClaw -An on-device AI Agent for iPhone, powered by **Gemma 4** and **MLX** (Metal GPU). Fully offline — no network, no cloud, no privacy leaks. +[English](./README_EN.md) -## Features +PhoneClaw 是一个运行在 iPhone 上的本地 AI Agent。 +它使用 **Gemma 4 + MLX + Metal GPU** 在设备端完成推理,不依赖云端,不上传聊天内容,适合做一个真正离线、可调用原生能力的私人助手。 -- **Fully Offline** — runs entirely on-device via MLX Metal GPU inference -- **Multimodal** — supports image input (vision language model) -- **File-driven Skill System** — define capabilities in plain `SKILL.md` files, no code changes needed -- **Multi-round Tool Chain** — agent can call multiple tools in sequence (up to 10 rounds) -- **Memory-safe** — dynamic history depth, GPU cache management for 6 GB jetsam limit -- **In-app Skill Editor** — view, edit, and hot-reload SKILL.md directly on device +## 这是什么 -## Project Structure +PhoneClaw 不是普通聊天壳,它是一个 `LLM + SKILL` 的本地 Agent: -``` -PhoneClaw/ -├── App/PhoneClawApp.swift ← Entry point -├── Agent/AgentEngine.swift ← Agent loop (tool detection, multi-round chain) -├── LLM/ -│ ├── LLMEngine.swift ← Protocol definition -│ ├── MLXLocalLLMService.swift ← MLX GPU inference (Gemma 4) -│ ├── PromptBuilder.swift ← Gemma 4 chat template + function calling -│ └── MLX/ ← Custom Gemma 4 VLM implementation (9 files) -├── Skills/ -│ ├── Skills.swift ← Data models -│ ├── SkillLoader.swift ← SKILL.md parser (YAML frontmatter + body) -│ └── ToolRegistry.swift ← Native iOS API tool registry -├── UI/ -│ ├── ContentView.swift ← Chat UI with skill progress cards -│ ├── ChatModels.swift ← UI data models -│ ├── SkillsManagerView.swift ← Skills management panel -│ ├── ConfigurationsView.swift ← Model parameter settings -│ └── Theme.swift ← Design system -├── Assets.xcassets/ -├── Info.plist -└── PhoneClaw.entitlements ← increased-memory-limit -``` +- LLM 负责理解用户意图 +- `SKILL.md` 负责描述某项能力该怎么做 +- 原生工具负责真正执行设备内操作 -## Requirements +当前已经打通的原生能力包括: -- Xcode 16+ -- iOS 17.0+ -- iPhone with Apple Silicon (A17 Pro or later recommended for Gemma 4 E2B) -- CocoaPods (`gem install cocoapods`) +- 剪贴板读写 +- 设备信息读取 +- 文本处理 +- 日历事件创建 +- 提醒事项创建 +- 通讯录联系人创建/更新 -## Model +## 项目特点 -PhoneClaw supports **Gemma 4 E2B** and **Gemma 4 E4B** (4-bit quantized, MLX format). -The app only shows models whose directory exists under `Models/` — **place only what you need**. +- 完全离线运行,默认不联网 +- 支持图片输入 +- 基于文件的 Skill 系统,新增能力不需要重写整个 Agent +- 支持多轮工具调用 +- 已内置权限管理、System Prompt 编辑、模型切换 +- 针对 iPhone 内存上限做了缓存清理和历史裁剪 -### Option A — E2B only (recommended, ~1.5 GB) +## 运行要求 -``` -Models/ -└── gemma-4-e2b-it-4bit/ -``` +- macOS + Xcode 16 或更新版本 +- iOS 17.0 或更新版本 +- CocoaPods +- 真机调试账号 -Download from Hugging Face: -``` -mlx-community/gemma-4-2b-it-4bit -``` +模型建议: -### Option B — E4B only (~3 GB, requires iPhone 15 Pro or later) +- `Gemma 4 E2B`:更稳,更适合默认分发 +- `Gemma 4 E4B`:效果更强,但更吃内存,建议高端机型使用 -``` -Models/ -└── gemma-4-e4b-it-4bit/ -``` +## 5 分钟快速开始 -Download from Hugging Face: -``` -mlx-community/gemma-4-4b-it-4bit -``` - -### Option C — Both models (user can switch in Settings) - -``` -Models/ -├── gemma-4-e2b-it-4bit/ -└── gemma-4-e4b-it-4bit/ -``` - -> The `Models/` directory is gitignored. Download model files separately and place them before building. - -## Getting Started - -### 1. Install dependencies +### 1. 安装依赖 ```bash pod install ``` -### 2. Open the workspace +### 2. 下载模型 + +先选择你要打包的模型。PhoneClaw 当前识别的目录名必须和下面保持一致。 + +推荐先安装 Hugging Face CLI: + +```bash +brew install hf +``` + +或者: + +```bash +pip install -U "huggingface_hub" +``` + +只下载 `E2B`: + +```bash +mkdir -p ./Models/gemma-4-e2b-it-4bit +hf download mlx-community/gemma-4-e2b-it-4bit --local-dir ./Models/gemma-4-e2b-it-4bit +``` + +只下载 `E4B`: + +```bash +mkdir -p ./Models/gemma-4-e4b-it-4bit +hf download mlx-community/gemma-4-e4b-it-4bit --local-dir ./Models/gemma-4-e4b-it-4bit +``` + +两个都下载: + +```bash +mkdir -p ./Models/gemma-4-e2b-it-4bit +mkdir -p ./Models/gemma-4-e4b-it-4bit +hf download mlx-community/gemma-4-e2b-it-4bit --local-dir ./Models/gemma-4-e2b-it-4bit +hf download mlx-community/gemma-4-e4b-it-4bit --local-dir ./Models/gemma-4-e4b-it-4bit +``` + +下载后目录大致如下: + +```text +Models/ +├── gemma-4-e2b-it-4bit/ +│ ├── config.json +│ ├── tokenizer.json +│ ├── processor_config.json +│ ├── chat_template.jinja +│ ├── model.safetensors +│ └── model.safetensors.index.json +└── gemma-4-e4b-it-4bit/ +``` + +说明: + +- `Models/` 已在 `.gitignore` 中忽略,不会被提交到仓库 +- 当前 Hugging Face 模型页显示的仓库体积大约是:`E2B 3.58 GB`、`E4B 5.22 GB` +- 如果你不想用 CLI,也可以直接从模型页面手动下载后放到对应目录 + +### 3. 打开工程 ```bash open PhoneClaw.xcworkspace ``` -> ⚠️ Always open `.xcworkspace`, not `.xcodeproj` +不要打开 `.xcodeproj`,请始终打开 `.xcworkspace`。 -### 3. Sign and run +### 4. 配置签名并运行 -1. In Xcode: select the **PhoneClaw** target → **Signing & Capabilities** -2. Set your **Team** (Apple ID) -3. Change **Bundle Identifier** to something unique (e.g. `com.yourname.phoneclaw`) -4. Connect your iPhone via USB -5. Press **⌘R** +在 Xcode 中: -First install requires trusting the certificate on iPhone: -**Settings → General → VPN & Device Management → Trust** +1. 选择 `PhoneClaw` target +2. 打开 `Signing & Capabilities` +3. 选择你的 `Team` +4. 把 `Bundle Identifier` 改成你自己的唯一值 +5. 连接 iPhone +6. 按 `⌘R` -## Built-in Skills +首次安装后,如果系统要求信任开发者证书,请在手机里完成信任。 -| Skill | Tools | -|-------|-------| -| Clipboard | `clipboard-read`, `clipboard-write` | -| Device | `device-info`, `device-name`, `device-model`, `device-system-version`, `device-memory`, `device-processor-count` | -| Text | `calculate-hash`, `text-reverse` | +### 5. 首次使用 -## Adding Custom Skills +打开 App 后: -Create a new directory under `ApplicationSupport/PhoneClaw/skills//SKILL.md`: +- 右上角拼图按钮:Skill 管理 +- 右上角滑杆按钮:模型设置 / 系统提示词 / 权限 + +建议先在权限页开启: + +- 日历 +- 提醒事项 +- 通讯录 + +然后直接试下面几句: + +- `这台手机的设备信息是什么` +- `提醒我今晚八点发文件` +- `帮我存一下王总的电话 13812345678` + +## 如何只打包一个模型 + +这是最常见的发布方式,尤其是你只想发 `E2B` 时。 + +### 方案 A:只打包 `E2B` + +保留: + +```text +Models/gemma-4-e2b-it-4bit +``` + +移除: + +```text +Models/gemma-4-e4b-it-4bit +``` + +然后做两步: + +1. 在 Xcode 的 Project Navigator 里删除不用的模型 folder reference,选择 `Remove Reference` +2. 在 `PhoneClaw > Build Phases > Copy Bundle Resources` 里确认只剩你要打包的模型 + +最后,修改 [LLM/MLXLocalLLMService.swift](./LLM/MLXLocalLLMService.swift) 里的 `availableModels`,只保留实际会随 App 分发的模型。 +否则配置页仍然会显示不存在的模型选项。 + +### 方案 B:同时打包 `E2B + E4B` + +保留两个目录和两个 Xcode 资源引用即可。 +用户可在 App 的 `模型设置 / Model Settings` 页里切换。 + +## 内置 Skill + +| Skill | 工具 | +| --- | --- | +| 剪贴板 | `clipboard-read`, `clipboard-write` | +| 设备 | `device-info`, `device-name`, `device-model`, `device-system-version`, `device-memory`, `device-processor-count` | +| 文本 | `calculate-hash`, `text-reverse` | +| 日历 | `calendar-create-event` | +| 提醒事项 | `reminders-create` | +| 通讯录 | `contacts-upsert` | + +## 自定义 Skill + +新增一个 Skill 的最小成本方式,是在应用目录里增加一个 `SKILL.md`。 + +示例结构: + +```text +Application Support/PhoneClaw/skills//SKILL.md +``` + +示例: ```yaml --- name: MySkill -description: 'What this skill does' +name-zh: 我的能力 +description: 这个 Skill 的作用 version: "1.0.0" icon: star disabled: false triggers: - - keyword1 + - 关键词1 allowed-tools: - my-tool-name examples: - - query: "example user query" - scenario: "what happens" + - query: "用户会怎么说" + scenario: "什么场景会触发" --- -# Skill Instructions +# Skill 指令 -Tell the model what to do and how to call the tools. +告诉模型何时调用工具、如何组织参数、何时直接回答。 ``` -Then register the native implementation in `ToolRegistry.swift`. +如果这个 Skill 需要真正调用系统能力,再去 [Skills/ToolRegistry.swift](./Skills/ToolRegistry.swift) 注册对应工具。 -## Architecture +## 关键目录 +```text +PhoneClaw/ +├── App/ # App 入口 +├── Agent/ # Agent 循环与多轮工具调用 +├── LLM/ # MLX 本地推理与 Prompt 构建 +├── Skills/ # Skill 解析、工具注册、数据模型 +├── UI/ # 聊天界面、Skill 管理、配置页面 +├── Models/ # 本地模型目录(默认不入库) +├── PhoneClaw.xcworkspace +└── README.md ``` -User Input - → PromptBuilder (Gemma 4 chat template) - → MLX GPU inference (streaming) - → Detect - ├── load_skill → inject SKILL.md body → re-infer - └── tool execution → ToolRegistry → iOS API → re-infer - → Final text response + +## 实际执行链路 + +```text +用户输入 +→ PromptBuilder 组装提示词 +→ Gemma 4 本地推理 +→ 需要能力时调用 load_skill +→ 读取对应 SKILL.md +→ 执行原生工具 +→ 返回最终中文结果 ``` +## 常见问题 + +### 为什么安装后看不到权限弹窗? + +通常是因为对应 Skill 还没有真正执行到系统 API。 +如果之前已经拒绝过一次,iOS 也不会反复弹框,需要到系统设置里手动开启。 + +### 为什么切模型后加载失败? + +先确认: + +- 模型目录名和代码里的 `availableModels` 一致 +- 该模型确实被打进了 App 包 +- 设备内存足够 + +### 为什么提醒事项创建失败? + +最新代码会先尝试复用现有提醒列表;如果系统里没有可写列表,会再尝试自动创建一个 `PhoneClaw` 提醒列表。 +如果这一步仍失败,通常是系统提醒源本身不可写。 + +## 后续计划 + +PhoneClaw 接下来的方向,不只是“多加几个工具”,而是把它逐步做成一个真正可用的本地 iPhone Agent。 + +### 1. 扩展更多 iOS 原生 API + +计划继续补充更多系统能力,例如: + +- 文件与目录操作 +- 照片读取、整理、描述、检索 +- 备忘录 / Notes +- 本地通知 +- 地图 / 位置相关能力 +- Safari / URL 打开与上下文传递 +- 更多通讯录、日历、提醒事项的读写能力 + +### 2. 扩展更多 Skill + +后续会继续把能力拆成更清晰的 Skill,而不是把所有逻辑都堆在一个大 Prompt 里。 + +适合继续追加的 Skill 方向: + +- 文件管理 +- 照片理解与整理 +- 日程规划 +- 个人信息管理 +- 本地知识库检索 +- 语音输入 / 语音播报 + +### 3. 串联更多本地模型 + +除了主聊天模型之外,后续适合接入的本地模型还包括: + +- OCR 模型 +- 语音识别模型 +- 语音合成模型 +- Embedding / Reranker 模型 +- 更小的工具参数提取模型 +- 更强的规划模型或多模型协作链路 + +这会让 PhoneClaw 从“一个大模型做所有事”,逐渐演进成“多个本地模型协同工作”的架构。 + +### 4. 跨 App 自动化 + +这是很重要的方向,但会严格遵守 iOS 的安全边界。 +PhoneClaw 不会假设自己能像桌面系统那样任意操控所有 App,而是优先走 iOS 真正允许的能力: + +- `App Intents` +- `Shortcuts` +- `URL Scheme / Deep Link` +- `Share Sheet / 分享扩展` +- 剪贴板中转 +- 系统通知与唤起 + +更现实的目标是: + +- 在 App 之间传递内容 +- 拉起指定 App 到指定页面 +- 把本地理解能力和快捷指令工作流串起来 +- 把多步操作压缩成一条自然语言命令 + +### 5. 外部硬件与视觉扩展 + +除了手机内部能力之外,PhoneClaw 后续也会探索和外部硬件配合的形态。 +例如把外部视频输入、屏幕画面理解和本地模型串起来,让 PhoneClaw 不只是“在手机里回答问题”,而是逐步具备更强的现实世界感知与调度能力。 + +这部分暂时不展开太多,后面会留一些更有意思的玩法。 + +### 6. 我建议优先做的方向 + +如果按“最容易尽快做出体验差异”的顺序,我会建议优先做: + +1. 文件 / 照片 / 备忘录 三类高频 API +2. Shortcuts / App Intents 集成 +3. OCR + 语音识别 +4. 本地知识库检索 +5. 更细的自动化 Skill 编排 + +这样很快就能把 PhoneClaw 从“会聊天的本地模型”推进到“能真正帮用户做事的本地助手”。 + +## 参考链接 + +- Hugging Face CLI 文档: +- Hugging Face 下载文档: +- Gemma 4 E2B MLX 模型: +- Gemma 4 E4B MLX 模型: + ## License MIT diff --git a/README_EN.md b/README_EN.md new file mode 100644 index 0000000..cd399ba --- /dev/null +++ b/README_EN.md @@ -0,0 +1,324 @@ +# PhoneClaw + +[简体中文](./README.md) + +PhoneClaw is a local AI agent that runs directly on iPhone. +It uses **Gemma 4 + MLX + Metal GPU** for on-device inference, so chats and tool calls stay on the device by default. + +## What It Is + +PhoneClaw is an `LLM + SKILL` style local agent: + +- the LLM decides what the user wants +- `SKILL.md` files describe how a capability should behave +- native tools execute the actual device-side action + +Built-in native capabilities currently include: + +- clipboard read/write +- device info +- text utilities +- calendar event creation +- reminders creation +- contacts create/update + +## Highlights + +- fully offline by default +- image input supported +- file-driven skill system +- multi-step tool calling +- built-in permissions page, system prompt editor, and model switcher +- memory-aware MLX runtime management for iPhone limits + +## Requirements + +- macOS + Xcode 16 or newer +- iOS 17.0 or newer +- CocoaPods +- an Apple developer signing identity for real-device testing + +Model guidance: + +- `Gemma 4 E2B`: better default choice for distribution +- `Gemma 4 E4B`: stronger but heavier, better for high-end devices + +## Quick Start + +### 1. Install dependencies + +```bash +pod install +``` + +### 2. Download a model + +PhoneClaw expects these exact folder names under `Models/`. + +Install the Hugging Face CLI first: + +```bash +brew install hf +``` + +or: + +```bash +pip install -U "huggingface_hub" +``` + +Download only `E2B`: + +```bash +mkdir -p ./Models/gemma-4-e2b-it-4bit +hf download mlx-community/gemma-4-e2b-it-4bit --local-dir ./Models/gemma-4-e2b-it-4bit +``` + +Download only `E4B`: + +```bash +mkdir -p ./Models/gemma-4-e4b-it-4bit +hf download mlx-community/gemma-4-e4b-it-4bit --local-dir ./Models/gemma-4-e4b-it-4bit +``` + +Download both: + +```bash +mkdir -p ./Models/gemma-4-e2b-it-4bit +mkdir -p ./Models/gemma-4-e4b-it-4bit +hf download mlx-community/gemma-4-e2b-it-4bit --local-dir ./Models/gemma-4-e2b-it-4bit +hf download mlx-community/gemma-4-e4b-it-4bit --local-dir ./Models/gemma-4-e4b-it-4bit +``` + +Notes: + +- `Models/` is already ignored by Git +- the current Hugging Face repo sizes are roughly `3.58 GB` for `E2B` and `5.22 GB` for `E4B` +- you can also download files manually from the model pages if you prefer + +### 3. Open the workspace + +```bash +open PhoneClaw.xcworkspace +``` + +Always open the workspace, not the project file. + +### 4. Configure signing and run + +In Xcode: + +1. select the `PhoneClaw` target +2. open `Signing & Capabilities` +3. choose your `Team` +4. set a unique `Bundle Identifier` +5. connect your iPhone +6. press `⌘R` + +### 5. First launch + +Inside the app: + +- top-right puzzle icon: Skill manager +- top-right sliders icon: model settings / system prompt / permissions + +It is recommended to enable these permissions first: + +- Calendar +- Reminders +- Contacts + +Then try: + +- `What is this phone's device information?` +- `Remind me to send the file tonight at 8 PM` +- `Save Wang Zong's number 13812345678` + +## How to Ship Only One Model + +This is the most practical setup for distribution. + +### Option A: Ship only `E2B` + +Keep: + +```text +Models/gemma-4-e2b-it-4bit +``` + +Remove: + +```text +Models/gemma-4-e4b-it-4bit +``` + +Then do both of the following: + +1. remove the unused model folder reference in Xcode with `Remove Reference` +2. verify `PhoneClaw > Build Phases > Copy Bundle Resources` only includes the model you want to ship + +Finally, update [LLM/MLXLocalLLMService.swift](./LLM/MLXLocalLLMService.swift) so `availableModels` only contains the model(s) you actually distribute. +Otherwise the configuration page will still show unavailable options. + +### Option B: Ship both `E2B + E4B` + +Keep both model folders and both Xcode resource references. +Users can switch models from the in-app configuration page. + +## Built-in Skills + +| Skill | Tools | +| --- | --- | +| Clipboard | `clipboard-read`, `clipboard-write` | +| Device | `device-info`, `device-name`, `device-model`, `device-system-version`, `device-memory`, `device-processor-count` | +| Text | `calculate-hash`, `text-reverse` | +| Calendar | `calendar-create-event` | +| Reminders | `reminders-create` | +| Contacts | `contacts-upsert` | + +## Custom Skills + +The smallest way to add a new capability is to add a new `SKILL.md`: + +```text +Application Support/PhoneClaw/skills//SKILL.md +``` + +Example: + +```yaml +--- +name: MySkill +name-zh: 我的能力 +description: What this skill does +version: "1.0.0" +icon: star +disabled: false + +triggers: + - keyword + +allowed-tools: + - my-tool-name + +examples: + - query: "How a user might ask for it" + scenario: "When it should trigger" +--- + +# Skill instructions + +Tell the model when to call tools, how to build arguments, and when to answer directly. +``` + +If the skill needs real native execution, register the tool in [Skills/ToolRegistry.swift](./Skills/ToolRegistry.swift). + +## Key Directories + +```text +PhoneClaw/ +├── App/ +├── Agent/ +├── LLM/ +├── Skills/ +├── UI/ +├── Models/ +├── PhoneClaw.xcworkspace +└── README.md +``` + +## Runtime Flow + +```text +User input +→ PromptBuilder +→ local Gemma 4 inference +→ load_skill when needed +→ read SKILL.md +→ execute native tool +→ produce final answer +``` + +## Useful Links + +- Hugging Face CLI docs: +- Hugging Face download guide: +- Gemma 4 E2B MLX model: +- Gemma 4 E4B MLX model: + +## Roadmap + +PhoneClaw is meant to grow into a genuinely useful on-device iPhone agent, not just a chat UI with a couple of tools. + +### 1. More iOS native APIs + +Planned areas include: + +- files and folders +- photo reading, organization, and understanding +- Notes integration +- local notifications +- maps and location-related tasks +- Safari / URL handoff +- broader calendar, reminders, and contacts workflows + +### 2. More Skills + +The long-term direction is to keep capabilities modular through Skills instead of pushing everything into one giant prompt. + +Likely next skills: + +- file management +- photo understanding and organization +- schedule planning +- personal information management +- local knowledge retrieval +- voice input / voice output + +### 3. More local models working together + +Beyond the main chat model, good future additions include: + +- OCR models +- speech recognition models +- text-to-speech models +- embedding / reranker models +- smaller argument-extraction models +- stronger planners or multi-model orchestration + +That would move PhoneClaw from “one model does everything” toward a more practical local multi-model agent stack. + +### 4. Cross-app automation + +This is a key direction, but it needs to stay realistic within iOS security limits. +The project is not assuming unrestricted UI control over every app. Instead, the practical path is: + +- `App Intents` +- `Shortcuts` +- `URL Schemes / Deep Links` +- `Share Sheet / Share Extensions` +- clipboard handoff +- notification-driven flows + +The goal is to make cross-app workflows feel natural while staying inside what iOS actually allows. + +### 5. External hardware and visual expansion + +Beyond the phone itself, PhoneClaw is also intended to explore workflows that involve external hardware. +That may include combining external video input, on-device visual understanding, and local models so PhoneClaw can gradually move from “answering inside the phone” toward richer real-world perception and orchestration. + +This part is intentionally kept a little vague for now. + +### 6. Suggested priorities + +If the goal is to improve real user value quickly, the strongest order is: + +1. files / photos / notes +2. Shortcuts / App Intents integration +3. OCR + speech recognition +4. local knowledge retrieval +5. richer automation-oriented skill composition + +## License + +MIT