Commit Graph
81 Commits
Author SHA1 Message Date
KellyvvandClaude Opus 4.6 c702e3cab0 docs: README update for 2026-04-08
- Add changelog entry covering ModelScope mirror and the memory-management
  rework (dynamic budget from headroom, dropped prompt-length subtraction).
- Fix stale path references after the MLX/Tools refactor:
  LLM/MLXLocalLLMService.swift → LLM/MLX/MLXLocalLLMService.swift
  Skills/ToolRegistry.swift → Tools/ToolRegistry.swift
- Custom skill snippet now shows the new \`type:\` field with device/content
  guidance, and notes the load-time allowed-tools validation.
- Add ModelScope mirror links to the reference list in both READMEs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 09:38:16 +08:00
KellyvvandClaude Opus 4.6 12eff570c8 chore: shell-only build, drop gemma-4-e4b-it-4bit from Resources phase
Default release build is the empty-shell IPA — model files live under
Models/ for local development but are no longer copied into the .app
bundle. Users still download Gemma weights at first launch via the
in-app downloader (ModelScope / hf-mirror / huggingface.co fallback).

Keep PBXFileReference + group children so Models/ stays browsable in
Xcode, but remove the BuildFile entry from PBXResourcesBuildPhase.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 09:30:34 +08:00
KellyvvandClaude Opus 4.6 be94b244d2 chore: remove DEVELOPMENT_TEAM from project.pbxproj
Xcode auto-injected the local team identifier when files were added to
the project. Strip it again so the repo stays signing-agnostic.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 09:25:09 +08:00
KellyvvandClaude Opus 4.6 779be78843 refactor: split MLX service, introduce skill type & runtime budgets
Major restructuring of the agent stack to support content-type skills,
data-driven runtime budgets, and the new translate skill.

LLM/MLX/
- Split MLXLocalLLMService.swift (1500 → 700 lines) into Core/, Gemma4/,
  Installation/ subdirectories
- New BundledModel: LinearBudgetFormula replaces tier tables, model
  parameters fully data-driven via ModelRuntimeProfile
- New RuntimeBudgets: pure functions for text/thinking/multimodal budgets,
  zero hardcoded model IDs
- Drop adjustedTextOutputTokens prepared-subtraction (chunked prefill made
  it obsolete and triggered false-positive truncation on long prompts)

Skills/
- Add SkillType (.device / .content) metadata, parsed from SKILL.md
  frontmatter
- New translate skill (type: content, no tools) with anaphora handling
- Trim translate (4563→1804), device (2357→1680), calendar (2169→1790)
- contacts SKILL.md adds multi-turn disambiguation + cancel guidance,
  semantics owned by body not framework
- SkillRegistry validates allowed-tools against ToolRegistry at load time

Agent/
- ToolCallParser becomes registry-aware: drops tool_call names not in
  ToolRegistry or protocol set (load_skill / list_skills); ToolRegistry
  is now the single white list, validation moves to parser boundary
- Router sticky routing uses semantic user-turn boundary instead of
  fixed suffix(4); correctly handles long agent loops
- AgentEngine system prompt template differentiates device vs content
  skills with explicit invocation rules
- SYSPROMPT.md auto-migration: old single-placeholder template backed up
  to .bak when new dual-placeholder is detected
- PromptBuilder.buildLoadedSkillPrompt receives availableTools list,
  explicitly tells model which tools (or none) are callable

Tools/
- ToolRegistry split into Helpers/Stores/AppPermissions/Handlers/* for
  separation of concerns
- Mark Tools/Handlers/* with explicit allowed argument signatures

UI/
- Add long-press copy contextMenu on user and assistant bubbles
- Enable .textSelection on user bubble

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 09:22:20 +08:00
Kellyvv 8872646858 chore: remove fixed development team from project 2026-04-07 16:29:46 +08:00
Kellyvv 446d216088 feat: improve on-device model download flow 2026-04-07 16:19:56 +08:00
Kellyvv 3d5c1ca67f docs: update release notes with ipa and chat history 2026-04-07 11:40:34 +08:00
Kellyvv 97e5c8993f docs: sync ipa install instructions in zh and en 2026-04-07 11:31:10 +08:00
Kellyvv 69c4b32248 feat: ship shell app and chat session polish 2026-04-07 11:13:35 +08:00
271ac8c1dc feat(ui): show download progress bar in model picker
Replace the plain "下载中 X/Y" badge with a small progress bar that
visualises file-count progress while a model is downloading. UI-only
change — does not touch the download path itself.

Inspired by @gclsoft's PR #3.

Co-Authored-By: Peter <gclsoft@qq.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 02:42:06 +08:00
Kellyvv 0d223ae373 feat: add audio input and stabilize gemma 4 inference 2026-04-07 01:50:29 +08:00
Kellyvv b6d3e54fc3 docs: add 2026-04-06 update notes to README 2026-04-06 16:11:43 +08:00
Kellyvv 3b30a77bbb Merge branch 'develop' 2026-04-06 16:07:12 +08:00
Kellyvv c81625aff8 Stabilize local skills and keep shell install default 2026-04-06 16:05:56 +08:00
Kellyvv 842cbe5c3c merge: develop → main 2026-04-06 12:09:47 +08:00
Kellyvv 63781c32c3 chore: switch license from MIT to Apache 2.0 2026-04-06 12:09:44 +08:00
Kellyvv ab216aabc1 merge: develop → main 2026-04-06 07:31:46 +08:00
Kellyvv be78c69f0d docs: restore E4B download commands in 方案 C / Option C 2026-04-06 07:31:44 +08:00
Kellyvv 2a0589a286 merge: develop → main 2026-04-06 07:06:02 +08:00
Kellyvv 5dde0d1b1a docs: sync English README — single video embed, no captions 2026-04-06 07:06:00 +08:00
Kellyvv 857e488e4e merge: develop → main 2026-04-06 07:05:18 +08:00
Kellyvv bc3c5e32c2 docs: simplify demo to single video, remove captions 2026-04-06 07:05:15 +08:00
Kellyvv 6a068e2731 merge: develop → main 2026-04-06 07:02:47 +08:00
Kellyvv 2a39f25732 docs: embed both demo videos side by side 2026-04-06 07:02:41 +08:00
Kellyvv 0f2752b7a6 merge: develop → main 2026-04-06 07:02:08 +08:00
Kellyvv 3cfb0a792c docs: embed demo video inline using GitHub user-attachments URL 2026-04-06 07:02:06 +08:00
Kellyvv 9563925d33 merge: develop → main 2026-04-06 06:54:36 +08:00
Kellyvv 265a280e4e docs: add demo video links section 2026-04-06 06:54:34 +08:00
Kellyvv 484608b0ff merge: develop → main 2026-04-06 06:49:40 +08:00
Kellyvv 316c0e6f62 docs: merge 运行要求 + 快速开始 + 安装方式 into one 快速开始 section 2026-04-06 06:49:38 +08:00
Kellyvv 777b917987 merge: develop → main 2026-04-06 06:45:10 +08:00
Kellyvv 696bafaf73 docs: add centered navigation links below header 2026-04-06 06:45:08 +08:00
Kellyvv 27a61107af merge: develop → main 2026-04-06 06:26:44 +08:00
Kellyvv bf7e3dea08 docs: rename section to '内置 Skill 示例 / Built-in Skill Examples' 2026-04-06 06:26:42 +08:00
Kellyvv d9546bba71 merge: develop → main 2026-04-06 06:25:39 +08:00
Kellyvv b5b40e6496 docs: replace 'local GPU' with 'on-device/手机端' in privacy section 2026-04-06 06:25:37 +08:00
Kellyvv b619b47510 merge: develop → main 2026-04-06 06:24:58 +08:00
Kellyvv 81d0ddd1ff docs: add usage examples to each Built-in Skill 2026-04-06 06:24:54 +08:00
Kellyvv 42d34247a3 merge: develop → main 2026-04-06 06:22:03 +08:00
Kellyvv 8a908c0377 docs: split Core Features and Built-in Skills into separate sections 2026-04-06 06:22:01 +08:00
Kellyvv ad482a6445 merge: develop → main 2026-04-06 06:19:01 +08:00
Kellyvv fd06a54f57 docs: rewrite features as descriptive paragraphs (Core Features style) 2026-04-06 06:18:59 +08:00
Kellyvv 954c5f9983 merge: develop → main 2026-04-06 06:15:36 +08:00
Kellyvv 116a716a35 docs: fix section order — description → 项目特点 → 能力表 2026-04-06 06:15:34 +08:00
Kellyvv 0cbbde8e30 merge: develop → main 2026-04-06 06:13:28 +08:00
Kellyvv 71c3fdbb80 docs: reorder — 项目特点 before capability table, description paragraph after 2026-04-06 06:13:23 +08:00
Kellyvv a3c53bf754 merge: develop → main 2026-04-06 06:10:15 +08:00
Kellyvv b304b40770 docs: remove title from README header (banner first); chore: clear Team ID 2026-04-06 06:10:12 +08:00
Kellyvv 3f84ae603a merge: develop → main 2026-04-06 05:51:21 +08:00
Kellyvv 2a3d14402d docs: remove app icon from README header 2026-04-06 05:51:18 +08:00
Kellyvv 2a462839d0 merge: develop → main 2026-04-06 05:49:01 +08:00
Kellyvv 67b2567e65 docs: centered README header — icon, banner, badges, nav links 2026-04-06 05:48:59 +08:00
Kellyvv 82964c8c1c merge: develop → main 2026-04-06 05:40:59 +08:00
Kellyvv 137e0fa5d1 docs: add banner image to README top, update model download flow (empty shell + on-device download) 2026-04-06 05:40:56 +08:00
Kellyvv ba1ed99a89 merge: develop → main 2026-04-06 05:05:57 +08:00
Kellyvv dd76a4718a docs: remove 关键目录/project structure section 2026-04-06 05:05:55 +08:00
Kellyvv 4675d72464 merge: develop → main 2026-04-06 05:04:47 +08:00
Kellyvv 342e7be66d docs: remove execution chain section 2026-04-06 05:03:42 +08:00
Kellyvv 9754635033 merge: develop → main (calendar/reminders/contacts skills, multi-model, README bilingual) 2026-04-06 04:29:04 +08:00
Kellyvv b1e486f21c docs: full sync README_EN with Chinese version — all sections, clean formatting 2026-04-06 04:28:28 +08:00
Kellyvv 33cc497789 docs: add image understanding to capabilities table 2026-04-06 04:26:43 +08:00
Kellyvv 1f5e6d33cd docs: add git clone as step 1 in quick start 2026-04-06 04:24:17 +08:00
Kellyvv 71dffe15cf docs: clean README — remove bold/icons/rules, drop 助手/LLM explanation paragraph 2026-04-06 04:23:12 +08:00
Kellyvv 9f960a79cc docs: fix README — single skill examples only, AI Agent positioning 2026-04-06 04:14:29 +08:00
Kellyvv 993cc6639d docs: AI Agent (not AI assistant) 2026-04-06 04:12:39 +08:00
Kellyvv e177bc06f3 docs: rewrite README — product-focused, bilingual (zh default / en) 2026-04-06 04:12:18 +08:00
Kellyvv 150964c660 chore: remove development team from tracked project settings 2026-04-06 04:05:40 +08:00
Kellyvv 4f469108de docs: rewrite README for quick start and roadmap 2026-04-06 04:03:54 +08:00
Kellyvv 9b579c91a3 docs: update README — model setup options (E2B only / E4B only / both) 2026-04-06 03:50:51 +08:00
Kellyvv 95553ee653 feat: major agent upgrade — multi-model, calendar/reminders/contacts skills
Agent:
- Rewrite load_skill follow-up: inject skill instructions into system
  block (buildLoadedSkillPrompt) instead of appending as user message
- Add auto tool-call shortcut for single no-arg skills
  (autoToolCallForLoadedSkills) — skips second LLM round
- Add Chinese date parsing (parseBasicChineseDate, detectDateInQuestion)
- Add heuristic argument inference (heuristicArgumentsForTool)
- Add looksLikePromptEcho / looksLikeStructuredIntermediateOutput filters
- Add renderToolResultLocally for zero-inference result formatting
- Add permission management API (AppPermissionKind, permissionStatuses,
  requestPermission)
- Add applyModelSelection() for runtime model switching

PromptBuilder:
- Replace buildFollowUp+buildForcedSkillContinuation with:
  buildLoadedSkillPrompt (skill in system block, cleaner prefill)
  buildToolAnswerPrompt (minimal tool-result answer prompt)
- Add extractSystemBlock / injectIntoSystemBlock helpers

MLXLocalLLMService:
- Add BundledModelOption struct
- Support multiple models (E2B + E4B) via availableModels array
- modelPath now dynamic based on selectedModel

ToolRegistry:
- Add calendar-create-event (EventKit, write-only access)
- Add reminders-create (EventKit, full access, EKAlarm for notifications)
  - ensureWritableReminderCalendar: auto-creates reminder list if needed
- Add contacts-upsert (Contacts, dedup by phone number)
- Fix: CNContacts .limited case treated as .granted
- Fix: reminders EKAlarm required for notification to fire

SkillLoader:
- Add calendar, reminders, contacts default SKILL.md entries

Info.plist:
- Add NSCalendarsWriteOnlyAccessUsageDescription
- Add NSCalendarsUsageDescription
- Add NSRemindersFullAccessUsageDescription
- Add NSRemindersUsageDescription
- Add NSContactsUsageDescription

ConfigurationsView:
- Add model picker (E2B / E4B)
- Add permissions panel (AppPermissionKind list with request support)
2026-04-06 03:49:21 +08:00
Kellyvv d5744708ba chore: remove all MediaPipe/E4B/LiteRT legacy references
- Delete doc/HOW_TO_BUILD_E4B_IOS_APP.md (entirely obsolete)
- Podfile: remove MediaPipeTasksGenAI/C pods (not used, code uses MLX SPM)
- pod install: regenerated Podfile.lock (1 pod: Yams only)
- README.md: remove E4B from hardware requirements
- LLMEngine.swift: clean MediaPipe from comments
- MLXLocalLLMService.swift: replace E4B → E2B in inline comments
- PromptBuilder.swift: replace E4B → E2B in inline comments
- AgentEngine.swift: replace LiteRT-LM comment with MLX
- doc/iphone_agent_permissions_*.md: replace E4B → Gemma 4
2026-04-06 02:24:50 +08:00
Kellyvv 649ad2162c revert: restore original PromptBuilder.swift completely 2026-04-06 02:18:04 +08:00
Kellyvv 80c49e39c4 revert: restore original kDefaultSystemPrompt
The 'optimization' broke model behavior — Gemma 4 needs
the literal <tool_call> example in system prompt to understand
the correct output format. Reverted to original.
2026-04-06 02:12:37 +08:00
Kellyvv 24d4098ce6 fix: optimize default system prompt
- Remove literal <tool_call> block from system turn
  (was being carried into buildFollowUp systemBlock, confusing
  the model into thinking it had already made a tool call)
- Condense instructions from ~110 tokens to ~55 tokens
  (halves prefill cost for the system block in every follow-up round)
- Rewrite as bullet rules instead of paragraph prose
2026-04-06 02:08:20 +08:00
Kellyvv 1469a4a795 fix: PromptBuilder — 0 token generation after load_skill
Root cause: <|turn>model trailing newline made Gemma 4 treat
the model turn as already completed, causing immediate EOS.

Fixes:
- Remove blank line after <|turn>model in buildFollowUp (isLoadSkill path)
- Remove blank line after <|turn>model in buildFollowUp (tool result path)
- Remove blank line after <|turn>model in buildForcedSkillContinuation
- Fix systemBlock range: ...upperBound → ..<upperBound (off-by-one)

Result: model now generates tool_call after skill is loaded
2026-04-06 02:06:47 +08:00
Kellyvv 4f113662f4 chore: local dev setup — restore Team ID, fix pbxproj structure
- Restore DEVELOPMENT_TEAM = ZP4QNJ35TH (needed for on-device build)
- Reorder PBXBuildFile / PBXGroup entries for Xcode compatibility
- Fix XCLocalSwiftPackageReference path for Packages/InferenceKit
2026-04-06 02:00:47 +08:00
Kellyvv 7e7f137726 refactor: rename mlx-swift-lm → Packages/InferenceKit 2026-04-06 01:53:57 +08:00
Kellyvv 475bf496e4 feat: vendor mlx-swift-lm as local SPM package (2.7MB, no .git)
Replaces remote XCRemoteSwiftPackageReference with local XCLocalSwiftPackageReference.
The mlx-swift-lm/ directory is now self-contained in this repo — anyone who
clones PhoneClaw gets MLXLMCommon, MLXLLM, MLXVLM without any network dependency.
2026-04-06 01:53:06 +08:00
Kellyvv abc25eb24f chore: open-source cleanup
- Remove personal Team ID (ZP4QNJ35TH) and Bundle ID (com.zxw.*)
  from project.pbxproj; replaced with empty team and com.example.phoneclaw
- Rewrite README.md for open-source (remove local paths, update to MLX architecture)
- Fix doc/HOW_TO_BUILD_E4B_IOS_APP.md: remove /Users/zxw/ hardcoded paths
- Delete LiteRT-LM/ (unused C API reference, never integrated)
- Delete _mediapipe-samples/ (reference only, not part of build)
- Delete _mlx-swift-lm/ (local clone, replaced with remote SPM reference)
- Convert _mlx-swift-lm from XCLocalSwiftPackageReference to remote
  XCRemoteSwiftPackageReference → https://github.com/ml-explore/mlx-swift-lm (branch: main)
- Update .gitignore to reflect removed directories
2026-04-06 01:44:55 +08:00
Kellyvv 0f146d4e8f refactor: flatten structure + remove dead code
- Move PhoneClaw/ contents to project root (no more nesting)
- Delete PhoneClawMac/ (nested SPM package, duplicated iOS code)
- Delete LLM/LocalLLMService.swift (dead MediaPipe implementation)
- Delete llm_inference_engine.h + PhoneClaw-Bridging-Header.h (unused)
- Delete Sources/ (empty dir), phoneclaw.code-workspace, TECH_PATH_ANALYSIS.md, project.yml
- Fix project.pbxproj: update INFOPLIST_FILE, CODE_SIGN_ENTITLEMENTS paths
- Fix project.pbxproj: remove SWIFT_OBJC_BRIDGING_HEADER setting
- Fix project.pbxproj: remove LocalLLMService.swift file reference
- Fix project.pbxproj: PhoneClaw group now uses name= instead of path=
2026-04-06 01:33:10 +08:00
Kellyvv d3f7dd78ca feat: initial commit — PhoneClaw on-device AI Agent
- AgentEngine: multi-round tool-chain (up to 10 rounds)
- MLX GPU inference with Gemma 4 E2B (4-bit quantized)
- VLM multimodal support (image input via <|image|> token)
- File-driven Skill system: SKILL.md (YAML frontmatter + body)
- Built-in Skills: clipboard, device, text
- ToolRegistry: iOS native API wrappers
- PromptBuilder: Gemma 4 chat template with function calling
- SwiftUI chat UI with Skill progress cards + thinking animation
- Memory-safe inference: dynamic history depth, GPU cache clearing
- PhoneClawMac: SPM package for macOS port
2026-04-06 01:26:23 +08:00