PhoneClaw

Capability	Description
Image Understanding	Ask questions about photos or images directly, multimodal input
Clipboard	Read and write the system clipboard
Device Info	Query device name, OS version, memory, and more
Text Tools	Hash calculation, text reversal, and more
Calendar	Create calendar events
Reminders	Create reminders with push notifications at the due time
Contacts	Create or update contacts, deduped by phone number

Features

Fully offline, no network by default
Image input support (multimodal)
File-driven Skill system — add new capabilities without rewriting the Agent
Multi-round tool calling
Supports shell-only app install, then on-device E2B / E4B download
Built-in permission management, system prompt editor, and model switcher
Dynamic history trimming and GPU cache management for iPhone memory limits

Requirements

macOS + Xcode 16 or later
iOS 17.0 or later
CocoaPods
A real device with a developer account (Apple ID)

Model recommendation:

Model	Use case
Gemma 4 E2B	More stable, recommended for general distribution, A16 and above
Gemma 4 E4B	Stronger output, higher memory usage, recommended for iPhone 15 Pro and above

Quick Start

1. Clone the repository

git clone https://github.com/kellyvv/phoneclaw.git
cd phoneclaw

2. Install dependencies

pod install

3. Optional: pre-download a model locally

The default recommended flow is now:

Install the app shell to the iPhone from Xcode
Open the app
Go to Model Settings
Download Gemma 4 E2B or Gemma 4 E4B directly on the phone

You only need the Models/ directory on your Mac if you want to bundle a model inside the app itself.

Directory names must match exactly as shown below. Install the Hugging Face CLI first:

brew install hf
# or
pip install -U "huggingface_hub"

E2B only (recommended):

mkdir -p ./Models/gemma-4-e2b-it-4bit
hf download mlx-community/gemma-4-e2b-it-4bit --local-dir ./Models/gemma-4-e2b-it-4bit

E4B only:

mkdir -p ./Models/gemma-4-e4b-it-4bit
hf download mlx-community/gemma-4-e4b-it-4bit --local-dir ./Models/gemma-4-e4b-it-4bit

Both models:

mkdir -p ./Models/gemma-4-e2b-it-4bit ./Models/gemma-4-e4b-it-4bit
hf download mlx-community/gemma-4-e2b-it-4bit --local-dir ./Models/gemma-4-e2b-it-4bit
hf download mlx-community/gemma-4-e4b-it-4bit --local-dir ./Models/gemma-4-e4b-it-4bit

Expected directory structure after download:

Models/
├── gemma-4-e2b-it-4bit/
│   ├── config.json
│   ├── tokenizer.json
│   ├── processor_config.json
│   ├── chat_template.jinja
│   ├── model.safetensors
│   └── model.safetensors.index.json
└── gemma-4-e4b-it-4bit/

Models/ is gitignored and will not be committed. Approximate repository sizes on Hugging Face: E2B ~3.58 GB, E4B ~5.22 GB. You can also download manually from the model page and place files in the correct directory.

4. Open the workspace

open PhoneClaw.xcworkspace

Do not open .xcodeproj. Always open .xcworkspace.

5. Configure signing and run

In Xcode, select the PhoneClaw target
Open Signing & Capabilities
Set your Team
Change the Bundle Identifier to a unique value
Connect your iPhone and press ⌘R

On first install, if prompted to trust the developer certificate: Settings → General → VPN & Device Management → Trust

6. First use

After opening the app:

Top-right puzzle icon: Skill management
Top-right slider icon: Model settings / system prompt / permissions
If you installed a shell-only app, tap Download in the model settings page first

Download a model first, then enable Calendar, Reminders, and Contacts in the permissions page, then try:

What is this device's information?
Remind me tonight at 8 to send the file
Save Wang's phone number 13812345678

Default Install Flow and Model Bundling

Option A — Shell app + on-device model download

This is now the default recommended setup.

Advantages:

Much smaller install size from Xcode
Faster first-time app installation from the Mac
Users can choose E2B or E4B directly on the phone

By default, the project no longer bundles anything from Models/ into the app.

Option B — E2B only

Keep Models/gemma-4-e2b-it-4bit, remove Models/gemma-4-e4b-it-4bit
In Xcode's Project Navigator, delete the unused model folder reference and choose Remove Reference
In PhoneClaw > Build Phases > Copy Bundle Resources, manually add back the model you want to ship and confirm only that one remains
Edit availableModels in LLM/MLXLocalLLMService.swift to only include the models actually shipped (otherwise the settings page will show options that don't exist)

Option C — Both E2B and E4B

Keep both directories, then manually add both folder references back into Copy Bundle Resources. Users can switch in the app's model settings page.

Adding Custom Skills

Create a SKILL.md file in the app's data directory and hot-reload in-app:

Application Support/PhoneClaw/skills/<skill-id>/SKILL.md

---
name: MySkill
name-zh: My Skill
description: What this skill does
version: "1.0.0"
icon: star
disabled: false

triggers:
  - keyword1

allowed-tools:
  - my-tool-name

examples:
  - query: "How a user might phrase it"
    scenario: "What scenario triggers this"
---

# Skill Instructions

Tell the model when to call tools, how to structure arguments, and when to answer directly.

If this skill needs to call native iOS APIs, register the tool in Skills/ToolRegistry.swift.

FAQ

Why are there no permission dialogs after install? The corresponding Skill has likely not reached the system API call yet. If you previously denied permission, iOS will not prompt again — go to system Settings to re-enable.

Why does the model fail to load after switching? Verify that the model directory name matches availableModels in code, that the model has finished downloading on-device if you are using the shell-only install flow, or that it was actually included in the app bundle if you are shipping it built-in, and that the device has enough memory.

Why does creating a reminder fail? The latest code first attempts to reuse an existing writable reminder list. If none is found, it tries to automatically create a PhoneClaw list. If that also fails, the system reminder source itself is likely read-only.

Roadmap

1. More iOS native APIs

File and directory access
Photos — reading, organizing, describing, searching
Notes
Local notifications
Maps and location
Safari / URL opening and context passing
More read/write coverage for contacts, calendar, and reminders

2. More Skills

Continue breaking capabilities into focused Skills rather than embedding all logic in a single large prompt. Directions worth adding:

File management
Photo understanding and organization
Schedule planning
Personal information management
Local knowledge base search
Voice input / text-to-speech

3. More local models

Beyond the main chat model, suitable additions include:

OCR model
Speech recognition model
Speech synthesis model
Embedding / Reranker model
A smaller tool argument extraction model
A stronger planning model or multi-model pipeline

This moves PhoneClaw from "one big model doing everything" toward "multiple local models working together."

4. Cross-app automation

PhoneClaw will not assume desktop-style control over arbitrary apps. Instead it will use what iOS actually allows:

App Intents / Shortcuts
URL Scheme / Deep Link
Share Sheet extensions
Clipboard relay
System notifications and app launching

A realistic goal: pass content between apps, open a specific app to a specific screen, and compress multi-step operations into a single natural language command.

5. External hardware and visual input

Explore connecting external video input and screen understanding with local models, so PhoneClaw goes beyond answering questions in isolation and develops stronger real-world perception and scheduling capabilities.

Suggested priority order

If ordered by "fastest path to meaningful experience improvement":

Files / Photos / Notes — three high-frequency API categories
Shortcuts / App Intents integration
OCR + speech recognition
Local knowledge base search
Finer-grained automated Skill orchestration

References

License

MIT