PhoneClaw

Added voice input, with on-device audio analysis and recognition
Added Thinking Mode, available from the top-right corner in chat
Added chat history, with support for new sessions, switching, and deletion
Improved memory management and inference budgeting for long answers, multimodal requests, and model switching
Added a self-signable shell IPA, available from GitHub Releases

2026-04-06 Update

The default install flow is back to a shell app, with models downloaded on-device as needed
The settings page now includes model download, permission status, and bilingual display names
Contacts, reminders, calendar, device info, and clipboard flows have received a round of stability fixes

Core Features

Image Understanding (Multimodal): Take a photo or pick one from your library, then ask questions directly. Identify objects, read charts, describe scenes — all inference happens on your device, and your photos never leave your phone.

File-Driven Skill System: Each capability is defined by a single Markdown file (SKILL.md). Adding or modifying a skill requires no recompilation. Skills are language-agnostic — anyone can write and share them.

100% Offline & Private: All inference runs entirely on your iPhone. No network connections are made by default. Your conversations, images, and personal data are never uploaded or routed through any third-party server.

Flexible Model Management: Supports Gemma 4 E2B and E4B. Download models directly on your iPhone, or bundle them into the app at build time. Includes a built-in model switcher, system prompt editor, and automatic history trimming for iPhone memory constraints.

Built-in Skill Examples

Calendar: Create calendar events using natural language — title, time, and location all supported.

"Schedule a meeting at Hightech Park tomorrow at 2pm"

Reminders: Set time-based reminders that fire a system push notification exactly on schedule.

"Remind me tonight at 8 to send the file to my boss"

Contacts: Save or update contacts with name, phone, company, email, and notes. Automatically deduped by phone number.

"Save Wang's number 13812345678, he's from Bytedance"

Clipboard: Read and write the system clipboard. Useful as a data relay in multi-step tasks.

"Copy that text to the clipboard"

Device Info: Query device name, OS version, available memory, processor count, and more.

"What's the device info for this phone?"

Text Tools: Hash calculation, text reversal, and other basic text utilities.

"Calculate the MD5 of this text"

Requirements

macOS + Xcode 16 or later
iOS 17.0 or later
CocoaPods
A real device with a developer account (Apple ID)

Model recommendation:

Model	Use case
Gemma 4 E2B	More stable, recommended for general distribution, A16 and above
Gemma 4 E4B	Stronger output, higher memory usage, recommended for iPhone 15 Pro and above

Quick Start

If you are installing from source, you will need macOS + Xcode 16, iOS 17+, CocoaPods, a real device, and an Apple ID.

1. Direct install (no Mac required)

Download the latest shell IPA from GitHub Releases
Sign and install it with your own signing method
After the first launch, open Model Settings and download E2B or E4B on the phone

2. Clone the repository

git clone https://github.com/kellyvv/phoneclaw.git
cd phoneclaw

3. Install dependencies

pod install

4. Optional: pre-download a model locally

The default recommended flow is now:

Install the app shell to the iPhone from Xcode
Open the app
Go to Model Settings
Download Gemma 4 E2B or Gemma 4 E4B directly on the phone

You only need the Models/ directory on your Mac if you want to bundle a model inside the app itself.

Directory names must match exactly as shown below. Install the Hugging Face CLI first:

brew install hf
# or
pip install -U "huggingface_hub"

E2B only (recommended):

mkdir -p ./Models/gemma-4-e2b-it-4bit
hf download mlx-community/gemma-4-e2b-it-4bit --local-dir ./Models/gemma-4-e2b-it-4bit

E4B only:

mkdir -p ./Models/gemma-4-e4b-it-4bit
hf download mlx-community/gemma-4-e4b-it-4bit --local-dir ./Models/gemma-4-e4b-it-4bit

Both models:

mkdir -p ./Models/gemma-4-e2b-it-4bit ./Models/gemma-4-e4b-it-4bit
hf download mlx-community/gemma-4-e2b-it-4bit --local-dir ./Models/gemma-4-e2b-it-4bit
hf download mlx-community/gemma-4-e4b-it-4bit --local-dir ./Models/gemma-4-e4b-it-4bit

Expected directory structure after download:

Models/
├── gemma-4-e2b-it-4bit/
│   ├── config.json
│   ├── tokenizer.json
│   ├── processor_config.json
│   ├── chat_template.jinja
│   ├── model.safetensors
│   └── model.safetensors.index.json
└── gemma-4-e4b-it-4bit/

Models/ is gitignored and will not be committed. Approximate repository sizes on Hugging Face: E2B ~3.58 GB, E4B ~5.22 GB. You can also download manually from the model page and place files in the correct directory.

5. Open the workspace

open PhoneClaw.xcworkspace

Do not open .xcodeproj. Always open .xcworkspace.

6. Configure signing and run

In Xcode, select the PhoneClaw target
Open Signing & Capabilities
Set your Team
Change the Bundle Identifier to a unique value
Connect your iPhone and press ⌘R

On first install, if prompted to trust the developer certificate: Settings → General → VPN & Device Management → Trust

6. First use

After opening the app:

Top-right puzzle icon: Skill management
Top-right slider icon: Model settings / system prompt / permissions
If you installed a shell-only app, tap Download in the model settings page first

Download a model first, then enable Calendar, Reminders, and Contacts in the permissions page, then try:

What is this device's information?
Remind me tonight at 8 to send the file
Save Wang's phone number 13812345678

Default Install Flow and Model Bundling

Option A — Shell app + on-device model download

This is now the default recommended setup.

Advantages:

Much smaller install size from Xcode
Faster first-time app installation from the Mac
Users can choose E2B or E4B directly on the phone

By default, the project no longer bundles anything from Models/ into the app.

Option B — E2B only

Keep Models/gemma-4-e2b-it-4bit, remove Models/gemma-4-e4b-it-4bit
In Xcode's Project Navigator, delete the unused model folder reference and choose Remove Reference
In PhoneClaw > Build Phases > Copy Bundle Resources, manually add back the model you want to ship and confirm only that one remains
Edit availableModels in LLM/MLXLocalLLMService.swift to only include the models actually shipped (otherwise the settings page will show options that don't exist)

Option C — Both E2B and E4B

Download both models:

brew install hf
mkdir -p ./Models/gemma-4-e2b-it-4bit ./Models/gemma-4-e4b-it-4bit
hf download mlx-community/gemma-4-e2b-it-4bit --local-dir ./Models/gemma-4-e2b-it-4bit
hf download mlx-community/gemma-4-e4b-it-4bit --local-dir ./Models/gemma-4-e4b-it-4bit

Then add both folder references back into Xcode's Copy Bundle Resources.

Adding Custom Skills

Create a SKILL.md file in the app's data directory and hot-reload in-app:

Application Support/PhoneClaw/skills/<skill-id>/SKILL.md

---
name: MySkill
name-zh: My Skill
description: What this skill does
version: "1.0.0"
icon: star
disabled: false

triggers:
  - keyword1

allowed-tools:
  - my-tool-name

examples:
  - query: "How a user might phrase it"
    scenario: "What scenario triggers this"
---

# Skill Instructions

Tell the model when to call tools, how to structure arguments, and when to answer directly.

If this skill needs to call native iOS APIs, register the tool in Skills/ToolRegistry.swift.

FAQ

Why are there no permission dialogs after install? The corresponding Skill has likely not reached the system API call yet. If you previously denied permission, iOS will not prompt again — go to system Settings to re-enable.

Why does the model fail to load after switching? Verify that the model directory name matches availableModels in code, that the model has finished downloading on-device if you are using the shell-only install flow, or that it was actually included in the app bundle if you are shipping it built-in, and that the device has enough memory.

Why does creating a reminder fail? The latest code first attempts to reuse an existing writable reminder list. If none is found, it tries to automatically create a PhoneClaw list. If that also fails, the system reminder source itself is likely read-only.

Roadmap

1. More iOS native APIs

File and directory access
Photos — reading, organizing, describing, searching
Notes
Local notifications
Maps and location
Safari / URL opening and context passing
More read/write coverage for contacts, calendar, and reminders

2. More Skills

Continue breaking capabilities into focused Skills rather than embedding all logic in a single large prompt. Directions worth adding:

File management
Photo understanding and organization
Schedule planning
Personal information management
Local knowledge base search
Voice input / text-to-speech

3. More local models

Beyond the main chat model, suitable additions include:

OCR model
Speech recognition model
Speech synthesis model
Embedding / Reranker model
A smaller tool argument extraction model
A stronger planning model or multi-model pipeline

This moves PhoneClaw from "one big model doing everything" toward "multiple local models working together."

4. Cross-app automation

PhoneClaw will not assume desktop-style control over arbitrary apps. Instead it will use what iOS actually allows:

App Intents / Shortcuts
URL Scheme / Deep Link
Share Sheet extensions
Clipboard relay
System notifications and app launching

A realistic goal: pass content between apps, open a specific app to a specific screen, and compress multi-step operations into a single natural language command.

5. External hardware and visual input

Explore connecting external video input and screen understanding with local models, so PhoneClaw goes beyond answering questions in isolation and develops stronger real-world perception and scheduling capabilities.

Suggested priority order

If ordered by "fastest path to meaningful experience improvement":

Files / Photos / Notes — three high-frequency API categories
Shortcuts / App Intents integration
OCR + speech recognition
Local knowledge base search
Finer-grained automated Skill orchestration

References

License

Apache 2.0